DOT/FAA/AM-17/17

Office of Aerospace Medicine

Washington, DC 20591

Failure to Follow Written 

Procedures

Colin G. Drury1

Catherine Drury Barnes1

Michelle R. Bryant2

1Applied Ergonomics Group, Inc.

Boulder, CO

2Civil Aerospace Medical Institute 

Federal Aviation Administration 

Oklahoma City, OK 73125

December 2017

Final Report
NOTICE

This document is disseminated under the sponsorship of 

the U.S. Department of Transportation in the interest of 

information exchange. The United States Government 

assumes no liability for the contents thereof.

___________

This publication and all Office of Aerospace Medicine 

technical reports are available in full-text from the 

Civil Aerospace Medical Institute’s publications website: 

http://www.faa.gov/go/oamtechreports
i

Technical Report Documentation Page

1. Report No. 2. Government Accession No. 3. Recipient's Catalog No.

DOT/FAA/AM-17/17

4. Title and Subtitle 5. Report Date

Failure to Follow Written Procedures December 2017

6. Performing Organization Code

7. Author(s) 8. Performing Organization Report No.

Drury CG,

1 Barnes CD,1 Bryant MR2

9. Performing Organization Name and Address 10. Work Unit No. (TRAIS)

1Applied Ergonomics Group, Inc., Boulder, CO

2FAA Civil Aerospace Medical Institute

P.O. Box 25082, Oklahoma City, OK 73125

11. Contract or Grant No.

12. Sponsoring Agency name and Address 13. Type of Report and Period Covered

Office of Aerospace Medicine

Federal Aviation Administration

800 Independence Ave., S.W.

Washington, DC 20591 14. Sponsoring Agency Code

15. Supplemental Notes

Work was accomplished under approved task AM-B-HRR-521

16. Abstract

Most tasks in aviation have a mandated written procedure to be followed specifically under the Code of 

Federal Regulations (CFR) Part 14, Section 43.13(a). However, the incidence of Failure to Follow Procedure 

(FFP) events continues to be a major issue in aviation maintenance. This report details the results of Phase 1 

of a two phase effort. In Phase 1 we used over 100 literature sources to identify human factors causes of FFP 

events, developed a classification system of FFP events (TAPES), categorized 154 FFP events reported in 

the Aviation Safety Reporting System (between 1999 and 2015) and 94 National Transportation Safety 

Board Accident reports (between 2005 and 2015), and identified the common challenges facing aviation 

maintenance operators. Results showed that the top three areas of concern were the validity and availability 

of the procedure documentation, the difficulty of the task being performed, and the organizations social 

rules/norms. Phase 2 of this work will apply the developed classification system to FFP events taken from 

semi-structured interviews of aviation maintenance personnel. Results of Phase 2 will identify best practices 

for mitigation FFP events in aviation maintenance environments.

17. Key Words 18. Distribution Statement

Failure to follow procedures, human factors, aviation 

maintenance

Document is available to the public through the 

Internet: http://www.faa.gov/go/oamtechreports/

19. Security Classif. (of this report) 20. Security Classif. (of this page) 21. No. of Pages 22. Price

Unclassified Unclassified 48

Form DOT F 1700.7 (8-72) Reproduction of completed page authorized
i

CONTENTS

List of Tables..........................................................................................................................................ii

Table of Figures......................................................................................................................................ii

List of Acronyms...................................................................................................................................iii

Executive Summary ..............................................................................................................................iv

Task 1 ................................................................................................................................................iv

Task 2 ................................................................................................................................................iv

Task 3 .................................................................................................................................................v

Failure to Follow Written Procedures ....................................................................................................1

Written Procedures in Aviation Maintenance Method...........................................................................1

Task 1 .................................................................................................................................................1

Understanding the Processes Involved in Procedures....................................................................1

Task 1 Method....................................................................................................................................2

Literature Relevant to Failure to Follow Written Procedures (FFP)..............................................2

Task 1 Results ....................................................................................................................................3

The Classification Scheme .............................................................................................................3

Task 2 .................................................................................................................................................5

Task 2 Method....................................................................................................................................5

Modifying the Classification Scheme.............................................................................................5

Selection of Databases....................................................................................................................6

Task 2 Results ....................................................................................................................................6

Classification of the ASRS Sample ................................................................................................6

Classification of the NTSB Sample................................................................................................8

Task 3 ...............................................................................................................................................10

Validating Databases....................................................................................................................10

Task 3 Method..................................................................................................................................10

Deriving Challenges and Best Practices.......................................................................................10

Task 3 Results ..................................................................................................................................10

Determination of Challenges and Best Practices..........................................................................10

Discussion ............................................................................................................................................14

Future Directions..................................................................................................................................14

Summary ..............................................................................................................................................14

References............................................................................................................................................15

Additional Resources ...........................................................................................................................19

Appendix A ........................................................................................................................................A-1

Appendix B.........................................................................................................................................B-1

Appendix C.........................................................................................................................................C-1

Appendix D ........................................................................................................................................D-1
ii

LIST OF TABLES

Table 1. Final TAPES Classification Schemes for CFs .........................................................................4

Table 2. Selection Criteria Used in the ASRS Database Queries...........................................................5

Table 3. Frequency of Operating Under each FAR for ASRS and NTSB Databases............................6

Table 4. Comparison of case and Contributing Factors for each FAR from ASRS. ..............................6

Table 5. Summaries of Frequencies of Each Event Pattern and Contributing Factor for ASRS............7

Table 6. Summaries of Frequencies of Each NTSB Event Pattern and Contributing Factor.................9

Table 7. Recommended Best Practices and Accompanying Challenges Compiled from All Sources.11

TABLE OF FIGURES

Figure 1. Logical decomposition of procedure errors with associated deviation categories..................4
iii

LIST OF ACRONYMS

CRM..................................................................................................................Crew Resource Management

FAA ............................................................................................................ Federal Aviation Administration
iv

EXECUTIVE SUMMARY

Most tasks in aviation have a mandated written procedure to be followed specifically under the Code 

of Federal Regulations (CFR) Part 14, Section 43.13(a). The rule is clear: use a manual for all work. 

However, the incidence of FFP events continues to be a major issue in aviation maintenance, despite 

considerable research over many years. A review of the issue by Drury and Johnson (2013) noted that 

“Procedure not followed” re-occurs with depressing regularity in incident and accident reports in 

aviation.” Recent Federal Aviation Administration (FAA) studies have confirmed this finding. For 

example, Banks, Wenzel, Drechsler, and Crayton (2013), reviewed over 650 reports from the National 

Aeronautics and Space Administration (NASA) Aviation Safety Reporting System (ASRS), and compiled 

and reviewed over 40 reports from both the public literature and from FAA Technical reports over the last 

25 years. 

In 2015, a Failure to Follow Procedures (FFP) research effort across aviation maintenance operators 

was commissioned by the FAA to (a) examine the primary and contributing factors associated with aviation 

maintenance operators failing to follow procedures (called Phase 1), and (b) develop mitigation strategies 

for reducing FFP events (called Phase 2). This report details the results of the Phase 1 effort, which 

consisted of three tasks.

Task 1 

Identify the human factors causes of FFP events. Task 1 resulted in the development of a classification 

system called TAPES, which was based on the 5 human factors categories most relevant to the classification 

of FFP events:

1. The maintenance task being performed (T)

2. The actor(s) who are performing the task (A)

3. The written or electronic procedure document used for the task (P)

4. The environmental context in which the task was performed (E)

5. The interactions between people in the broader organization beyond the immediate 

actor, called social (S).

Each of the 5 categories were then broken into different sub categories, which are documented in further 

detail within the body of this report.

Task 2

Categorize the human factors associated with FFP events reported in the ASRS (154 events between 

1999 and 2015) and National Transportation Safety Board (NTSB) databases ASRS (94 events between 

2005 and 2015). For the purposes of this research effort, a distinction was made between a willful disregard 

for following procedures and an actor’s attempt to follow procedures. While the former is reported, TAPES 

was used to classify only the latter. For the ASRS data base the top 3 categories were

1. Procedure Documentation (58%) - The documentation was not readily available, not up to date, 

or poorly written.

2. Task being performed (14%) - The actor was having difficulty with the maintenance task and 

an error was made.
v

3. Social (12%) - The organization’s inefficiencies in the management of work, norms governing 

how work is done, and the time pressures associated with work.

For the NTSB database the top 3 categories were

1. Task being performed (25%) - The actor was having difficulty with the maintenance task and 

an error was made.

2. Social (12%) - The organization’s inefficiencies in the management of work, norms governing 

how work is done, and the time pressures associated with work.

3. Procedure Documentation (6%) - The documentation was not readily available, not up to date, 

or poorly written.

However, 32% of the NTSB records contained insufficient information for the purposes of 

classification. This is in contrast to the ASRS data set, which had only 1% of records of the records 

containing insufficient information. Thus the ASRS results drove the search.

Task 3 

A list of the common challenges facing aviation maintenance operators was created in Task 3 as they 

related to the top 3 categories identified by TAPES (Table 7). An example from each category includes:

1. Procedure Documentation - Procedures may not be validated because of time constraints and 

perceived cost considerations. Also the task may not be available to observe while the 

procedure is being written.

2. Task being performed - Training may be cursory or poorly-implemented as it is not seen as a 

priority.

3. Social - As management changes and the maintenance becomes more competitive, there will 

always be pressures on mid-level managers and end-users to sacrifice the procedure following 

policy to a norm of expediency.

It is recommended that the results of Phase 1 be used in conjunction with the current Safety 

Management System requirements governing aviation maintenance operations. While larger businesses 

associated with providing aviation maintenance are likely to benefit from the integration of this effort along 

with other advances in maintenance research, it is unlikely that small business will be motivated to do so. 

Thus, the Phase 2 effort focused not only on the development of best practices for reducing FFP events but 

how to best communicate that information across a range of aviation maintenance operations.
1

FAILURE TO FOLLOW WRITTEN PROCEDURES

WRITTEN PROCEDURES IN AVIATION MAINTENANCE METHOD

Most tasks in aviation have a mandated written procedure to be followed specifically under the Code 

of Federal Regulations (CFRs) Part 14, Section 43.13(a). The rule is clear: use a manual for all work. 

However, as Drury and Johnson (2013) noted, “Procedure not followed re-occurs with depressing 

regularity in incident and accident reports in aviation.” Johnson and Avers (2014) listed Failure to Follow 

Procedures (FFP) as the number one cause of maintenance mishaps. An FAA study regarding major 

malfunctions that occurred within 90 days of a heavy maintenance check found that the number one reason 

for malfunction was the failure to comply with maintenance documentation (Johnson & Watson, 2001). As 

Rankin (2008) noted, failure to follow instructions was the primary cause of maintenance errors reported 

through Boeing’s Maintenance Error Decision Aid (MEDA). This non-following of written procedures is 

not limited to aviation maintenance, as Landry, Jacko, and Coulter (2006) showed for pilots. Walker (2005) 

reported similar findings in an analysis of offshore oil rig accidents. Overall, 31% of accidents involved 

maintenance procedures, 55% of those were classified as FFP. Clearly, FFP is a problem that is common 

across industries using procedures and a problem that is not decreasing with time.

With this background, it is not surprising that the FAA’s focus goes beyond merely listing FFP as a 

cause of accidents. Rather, detailed reasons beyond the causes are sought in order to properly address and 

mitigate such events. In Schroeder et al. (in press) a rationale was provided such that, a detailed review and 

analysis of the available literature on FFP could be conducted. This report is organized into three separate 

but related tasks. In Task 1, we developed a classification system for FFP events based on the extensive 

literature (See Task 1 for references) both within and beyond the aviation maintenance domain. In Task 2, 

we used this classification system to analyze two disparate databases of incidents/accidents to identify 

frequencies of the various factors contributing to FFP events. In Task 3, we integrated the database analyses, 

findings from the technical literature, and the classification scheme to derive a priority listing of CFs. These 

priorities and CFs were re-cast into Good Practices to be used to reduce Failure to Follow Procedure (FFP) 

events.

Task 1

Understanding the Processes Involved in Procedures

The purpose of Task 1 was to conduct a detailed review of the literature on FFPs to identify previously 

employed interventions, the timing of such interventions, and their resulting outcomes on aviation 

maintenance FFP events. 

The FAA’s early work on aviation maintenance and inspection errors following the Aloha Flight 243 

accident in 1988 generated considerable human factors research and applications through the Office of 

Aviation Medicine. The Aloha Boeing 737 lost part of its upper fuselage through a series of inspection and 

management failures, highlighting the possibility of airframe cracks that escaped maintenance inspection 

that resulted in catastrophic failure in the air. As part of this work Drury, Prabhu, and Gramopadhye (1990) 

performed a large number of task description/task analysis activities in the area of visual and nondestructive inspection, producing a generic task description of the inspection process. This was later 
2

expanded to include maintenance procedures as well as the inspection alone (Drury, 2005). However, the 

approach taken for the current work looked beyond the function/task description of the final performer to 

explore the entire context of how work is performed. In the current literature analysis, many of the errors, 

issues, CFs, Challenges, and Best Practices involve processes beyond the final use of a procedure by the 

inspector or AMT. The most common examples from the literature included:

 Procedures are not used because ...they are out-of-date 

 [Standard Operating Procedures] are typically written to fulfill an organizational requirement 

rather than to provide utility to technicians.

 ... [procedures] have not been written to cover the task at hand.

 ...[Organizational Policy] was unresponsive to employee’s suggestions

These, and many more, have been seen as CFs to FFP events, and thus need to be addressed in a 

comprehensive endeavor to reduce the incidence of adverse events. To do this, a literature review was 

conducted for the purpose of identifying CFs and developing a classification scheme for FFP event reports.

Task 1 Method

Literature Relevant to Failure to Follow Written Procedures (FFP)

Two main sources of literature were accessed for this task: 

 Literature previously compiled by the CAMI Human Factors team as part of the overalleffort 

on evaluating FFP events

 Literature compiled both from much of the prior work in the area by C. G. Drury at SUNY 

Buffalo, and direct searches based on Ergonomics Abstracts and Google Scholar. In addition, 

Applied Ergonomics Group Inc.’s (AEG’s) experience and contacts in the fields of chemical 

processes, chemical weapons demilitarization, nuclear power plants, and the UK’s Health and 

Safety Laboratories has provided additional reference material beyond AEG Inc.’s own 

experience in procedure and documentation design in aviation maintenance.

The combined literature review process yielded over 100 books, reports, published papers, conference 

proceedings papers and URL sites relevant to the issues and/or contributing conditions to FFP events.

The first analysis grouped information under the following headings:

 FFP Scenarios: Vignettes of Task, Operators, Machines, Environment and Social conditions 

(TOMES) that had led to actual FFP events, usually events with unwanted outcomes or for 

which recovery was required. A typical scenario is “User tries but fails to follow procedure as 

user misses performing one task.” 

 FFP CFs: Reasons the authors found, or postulated, for the occurrence of the FFP event. These 

could be at any level of depth of abstraction, or at multiple hierarchical levels. A typical 

Contributing Factor is “Un-validated/prototyped procedure.”

 FFP Good Questions: Where the main focus of literature was on evaluating or analyzing FFP 

events, the issues were typically expressed as investigative questions. Because these questions 

implied CFs, they could be re-written in Contributing Factor form. A typical Good Question is 

“Do users have an appropriate and known plan to improve or optimize procedures?”
3

 FFP Best Practices: Where a piece of literature made specific recommendations, these were 

classified as Good Practices, or the more common term Best Practices. They often implied CFs, 

but presented these in the form of solutions or mitigations. A typical Good Practice is 

“Procedures are available when needed and users can always find the correct procedure.”

 FFP Challenges: Possible reasons why good/best practices were not followed by the AMT. 

They represent the negative side of Best Practices. A typical challenge is “Need a process to 

improve response time for operator issues.”

Task 1 Results

The Classification Scheme

The next step was to organize and classify literature grouped into the FFP Scenarios heading into 

coherent patterns. For example, one set of reasons for FFP is that the procedures provided are not the best 

way to perform a task. Examples of this from four literature sources are:

 A procedure that is ponderous and is perceived as increasing workload, and/or interrupting 

smooth flow of tasks, will probably be ignored.

 Where rules are perceived as overly restrictive, skilled individuals may think they can violate 

the rules with little risk to their safety, and the resulting violations are likely to become routine.

Procedures not used because they are difficult to use.

 Unintentional violations arise from procedures that are either too difficult for people to follow, 

errors arising from not using the manual, or incorrectly applying the procedural steps (out of 

order, did not reference manual, etc.).

Therefore, all of the examples above represent a procedure that is difficult to follow, for various reasons, 

and can thus become part of an overarching scenario or Contributing Factor. 

Three classification schemes were tested on these data to determine which categories should be 

included in the developed schema. The first were the empirical set of scenarios from the UK’s Health and 

Safety laboratories, the second was the 5 Ps system developed by CAMI and discussed by Hollomon,

Drechsler, and Crayton (2015) , and expounded on in a 2017 report (Schroeder et al., in press), and the final 

was the characterization of Violations, originally found in Reason (1997) and seen in the Health and Safety 

Executive’s (HSE’s, 1995) Improving Compliance document and the website skybrary 

(www.skybrary.aero) in its document Assessing Procedures. From these, and specific examples of aviation 

maintenance FFP events, (e.g., Pearl and Drury, 1995, pp 127-165) noted, “Mechanics do not use workcards 

for frequently performed checks, i.e., A- checks and below. They have memorized these checks, ’gaining 

a feel for items to check’ through frequent repetition.” Boring, Gertman, and Le Blanc (2011, pp 1720-

1724) noted, “Performing steps or substeps in the wrong order...” the new classification schema was 

created that included both a bottom-up classification (e.g., HSE, 5 Ps schemes above) and top-down 

classification (e.g., Reason’s Violations) and can be seen in Table 1.
4

Table 1. Final TAPES Classification Schemes for CFs

TAPES Classification Definition

Task The actual task performed by the actor

Actor Those involved in conducting aviation maintenance 

tasks, including the AMT, MX supervisor, job inspector

Procedure Document The written or electronic document used by the actor to 

perform the task

Environment

The conditions surrounding the task to be performed, 

including physical layout, thermal / visual environment 

etc., excludes the documents used

Social The interactions between people in the broader 

organization beyond the immediate actor

As a guide to better understand the role each CF plays in an overall classification, factors were tabulated 

through classifying databases such as ASRS in much the same manner as Wenner and Drury (2000) did for 

ground damage incidents in aviation. Once completed, a combination of procedural deviations (also called 

scenario outcomes) and contributing factors were added to the developed classification schema (Figure 1).

Event 

Pattern (EP)

EP Contributing 

Factor

Classification of EP 

Contributing Factor

EP Scenario 

Outcome

Figure 1. Logical decomposition of procedure errors with associated deviation categories.
5

These groupings naturally lead to the groupings in Table 1 where the extensive list of CFs found in the 

literature were identified and consolidated. For each classification category, a Left-to-Right diagram was 

created (similar to Figure 1) where errors could be logically decomposed (see Appendix A). 

Task 2

Given the development of the TAPES classification system, it was necessary to apply it to practical 

scenarios to test its usefulness in aviation organizations. Both the ASRS and NTSB databases mentioned in 

Task 1 supplied the necessary components of error reporting such that Event Patterns could be identified 

and logically decomposed. The ASRS database was comprised of event reports of technical documentation 

and procedural errors occurring between January 1999 and December 2012 (Hollomon et al., 2015). This 

date range was chosen to maximize event reporting years, the number of events (prior to 1999 reports/events 

totaled less than 24), and the number of identifiable CFs (where events with limited information that did 

not provide enough detail to identify CFs were not included in the analysis). The iterative process for 

selecting the key variables for this database is shown in Table 2. One limitation of this approach was that 

this database query showed bias toward technical documentation. Since this limitation may have resulted 

in reports that overrepresented errors associated with technical documentation, it was important to ensure 

that other sources gathered did not also replicate this potential bias. Therefore, conclusions regarding the 

magnitude of influence that technical documentation may have in FFP events are not drawn in this research.

Table 2. Selection Criteria Used in the ASRS Database Queries

Reporter Organization Reporter Function

Air Carrier Inspector

Air Taxi Instructor

Contracted Service Lead Technician

Corporate Other/Unknown

FBO Parts/Stores Personnel

Fractional Quality Assurance/Audit

Government Technician

Trainee

Primary Problem Contributing Factors

Chart or Publication Chart or Publication

Logbook Entry Logbook Entry

Manuals Manuals

Procedure Procedure

The NTSB database included 871 procedure-related accidents that were investigated between January 

2005 and December 2014. These accidents involved maintenance as a causal, Contributing Factor, and/or 

significant finding by the NTSB.

Task 2 Method

Modifying the Classification Scheme 

To ensure that the ASRS events were coded consistently, two independent coders worked together 

extensively as has been done in other traditional inter-rater studies. Each ASRS report was coded for Event 

Pattern (i.e., final outcome error) and CF. Some reports had one Event Pattern and one CF, while others 
6

had multiple CF codes. This highlighted the flexibility in the use of the TAPES schema as, has been noted 

previously, errors rarely occur in a vacuum. The final coding scheme was presented as a set of hierarchical 

categories in Appendix B.

Selection of Databases

Sampling for event coding. Both the 871 procedure-related NTSB accidents and 650 ASRS reports 

were used to determine both the usefulness of the classification schema, and to identify CFs of FFP in the 

aviation maintenance environment. To ensure there were not differences between FAR parts, the reports 

were sorted for each (Table 3). 

Table 3. Frequency of Operating Under each FAR for ASRS and NTSB Databases

FAR Part Common Name NTSB (%) ASRS (%)

091 General Aviation 769 14

091K GA Fractional Owners 2 0

121 Scheduled Carriers 9 611

129 Foreign Sch. Carriers 1 0

133 Rotorcraft, Ext Load 8 0

135 Commuter Aircraft 36 16

137 Agricultural Ops. 40 0

145 MRO’s 0 2

Public Use 

Aircraft (PUBU) Assorted 7 0

As the aim of the present study was to develop and validate a classification scheme from which focused 

interventions could be derived, a sample of 150 events from each data base was examined such that the 

overlap between the two data sets (i.e., between Part 91 and Part 121) were maximized. Thus, the ASRS 

data was mainly Part 121, but all of the Part 91 and Part 135 cases were coded, for a total of 154 cases. For 

the NTSB data, depending on data quality (i.e., codable details), all Part 121 and Part 135 cases were 

sampled with the remainder coming from Part 91 and a few from Part 137. The overall findings were 

compared across data sets, with focused comparisons made for each specific FAR contributing to the data.

Task 2 Results

Classification of the ASRS Sample

As noted above, 154 cases were sampled, covering events between 1999 and 2014 representing each 

of the FAR parts (Table 4). 

Table 4. Comparison of case and Contributing Factors for each FAR from ASRS.

FAR Number of Cases Number of CF’s CF’s per Case

91 14 19 1.36

121 124 208 1.68

135 16 18 1.13
7

Results showed (Table 5) that No Error Made combined with the Procedure Followed, represented 95% 

of the errors within the ASRS sample. This finding makes sense for ASRS reports where reporters often 

report an incident that may have led to an accident but did not. Reports were overwhelmingly categorized 

as Procedure (58%) issues. This finding was not surprising given the search criteria were biased toward 

technical documentation issues. Task, Actor and Social categories were classified much less frequently. 

The fewest reports were classified as Environmental. This finding may reflect the view point that the ASRS 

database is not the appropriate tool to report environmental constraints. Therefore, it was determined that 

within the reports in our sample, AMTs typically tried to follow the procedure as written, but the physical 

document was usually referenced as the primary contributor to the event. 

Table 5. Summaries of Frequencies of Each Event Pattern and Contributing Factor for ASRS

Event Pattern Event Pattern Contributing Factor

Procedure Followed 69% Tries but fails to follow Procedure 60%

No Error Made 26%

Procedure Not Followed 5%

Wrong Procedure Chosen 7%

Procedure Followed 3%

Does not use any written Procedure 3%

Follows routine unapproved Procedure 1%

Procedure does not apply exactly <1%

TAPES Category TAPES Category Contributing Factor

Procedure 58%

Physical Document 38%

Document Writing and Production 18%

Document Revision 2%

Task 14%

Problems with Task as Found 7%

Errors Made During Task Performance 7%

Social 12%

Organizational Effectiveness 5%

Local Norms 4%

Organizational Pressures 2%

Organizational Policy 1%

Actor 9%

Actor Background 4%

Actor Current State 4%

Environment 7%

Physical Space for Task 1%

Thermal Environment 1%

Unexplained Error 1%
8

FAR Part analysis. The next comparisons were made to determine if differences existed between data 

from the three FAR types. That is, do specific FFP events occur based on the FAR Part that is conducting 

the maintenance? To test this question, first a Chi-Square analysis was conducted to determine whether 

AMTs reported following procedures more in one part than any other. This analysis found a significant 

difference between FAR Parts (χ2(2) = 7.5, p = 0.024). Therefore, by examining the means of Followed 

Procedures across Part, it could be seen that Part 91 AMTs reported following procedures significantly 

more than all other Parts. Second, to determine if certain categories in the TAPES classification schema 

contributed to FFP events differently by Part, a Chi-Square test was conducted. The results showed no 

difference between the three Parts (χ2(2) = 0.1, p = 0.95). Therefore, the overall conclusion from the FAR 

Part analysis of ASRS data was that there were differences in outcome error (EP Scenario Outcome, See 

Figure 1) but not in CFs. 

The primary purpose for creating a classification scheme was to determine what Challenges exist and 

the Best Practices for mitigating those challenges. Therefore, the CF frequency counts were used to 

determine the greatest contributor to FFP events. It was determined that the most frequent EP was “Tries 

but Fails to Follow Procedure” within the Procedures Category. Thus, the overall findings showed that the 

“Physical Document” and the “Document Writing and Production” were the greatest contributors to events 

reported. Again, this was not surprising considering the search criteria for the events coded within the ASRS 

database identified events specific to technical documentation (see Table 2).

Classification of the NTSB Sample

The NTSB maintenance-related accident database supplied by FAA/CAMI was sampled so as to 

maintain the greatest compatibility with the ASRS database sample described above. Thus, the sample 

comprised all of the 9 Part 121 cases, and 36 Part 135 cases to ensure maximum coverage of scheduled 

carriers. In addition to these, 49 Part 91 cases were sampled across a 10 year span from 2005 to 2015.1 As 

with the ASRS data, the numbers of CFs per case were found to have a similar distribution as the ASRS 

data. In contrast to the ASRS data, the NTSB data were significantly different from a Poisson distribution 

(χ2(2) = 15.0, p < .001). That is, there were more cases with only one contributing factor (more than 70 of 

94 cases). This outcome was to be expected given the goal of an NTSB investigation is to uncover a primary 

contributing factor and often does not continue to delve into lesser CFs.

All data were analyzed similar to the previously reported ASRS database results section (Table 6). 

These analyses found that “Procedure Followed” occurred at the highest rate (74%) with the CF 

Classification “Tries but Fails to Follow Procedure” occurring at the same rate as reported in ASRS reports. 

That is, those individuals who were performing the task had a procedure physically with them, and they 

attempted to follow it as written, but failed to fully perform the task as written. Within the ASRS data, there 

were more “Error not Classifiable” entries than expected in the NTSB reports. Similarly, there were more 

“Unexplained Error” findings in the CFs category than were expected.

 

1 Note that the full dockets prior to 2009 were no longer available on the FAA web site, so that the data on many 

cases were quite sparse, especially for contributing factors.
9

Table 6. Summaries of Frequencies of Each NTSB Event Pattern and Contributing Factor

Event Pattern Event Pattern 

Contributing Factor

Procedure Followed 74% Tries but fails to follow Procedure 60%

Error not Classifiable” 15%

Procedure Not Followed 11%

Wrong Procedure Chosen 14%

Procedure willfully ignored 7%

Does not use any written 

Procedure 2%

TAPES Category TAPES Category Contributing 

Factor

Procedure 6%

Document Writing and 

Production 3%

Physical Document 2%

Document Revision <1%

Task 26%

Problems with the task as found 21%

Errors Made During Task 

Performance 6%

Social 15%

Organizational Effectiveness 13%

Organizational Pressures <1%

Actor 3%

Actor Current State 2%

Actor Background <1%

Environment <1% Physical Space for Task 2%

Unexplained Error 32%

A Chi Square test was conducted to determine whether FAR Parts reported EPs differently. Results 

showed that EPs did, indeed, change based on FAR Part (χ2(2) = 15.0, p < .001). When examined further it 

was determined that this difference was likely due to the relatively large number of Unexplained Error for 

Part 91 as well as the relatively large Any Other factors for Part 121. However, these analyses underscored 

what was already known about the NTSB data; that Part 121 accidents generated more detailed reports with 

supporting CFs, than did the other two FAR Parts. Interestingly, Part 135 showed the greatest percentage 

of Unexplained Error.

Just as was conducted with the ASRS database, frequency counts were used to determine the greatest 

contributor to FFP events in the NTSB database. However, due to the general nature of the reports, or due 

to the format investigative report, the CFs were too general in nature to be specifically associated with 

particular EPs. Since the two databases provided differing levels of detail, it was determined that the ASRS 

reports (the more detailed database) would allow us to develop the most meaningful Challenges and Best 

Practices (shown in Table 4).
10

Task 3

Validating Databases

To generate a prioritized list of Challenges and Best Practices, the ASRS and NTSB databases were 

combined with the literature examined in Task 1 to derive a broadly-based prioritization of both Challenges 

and Best Practices. The coding for the TAPES scheme developed in Task 1 was used again for consistency 

(Table 1).

Task 3 Method

Deriving Challenges and Best Practices

Using the data from Task 2 to drive priority areas for Challenges, Best Practices, and Good Questions, 

each of the literature sources was summarized and relevant lists were placed into three tables: EPs/CFs,

Best Practices, Challenges, and Good Questions. As noted in Task 2, each source listed multiple CFs for 

events. Therefore, to structure the CFs, the TAPES classification scheme was also used to organize 

Challenges and Best Practices along with their reference source. Results provided a frequency count of each 

CF. However, it should be pointed out that the counts do not represent independent data points, since several 

papers from the same reference source on the same topic might mention the same items. As far as possible, 

potential bias was avoided by limiting multiple counts from the same authors or organization. Following 

this analysis, Challenges and Best Practices were derived.

Task 3 Results

Determination of Challenges and Best Practices

Findings showed that the Procedure category was the most prevalent issue related to events, followed 

by Social and then Actor. Task and Environment categories were considerably less frequent in the literature 

than the other three components. Bringing these findings together into an overview from the most frequent 

to least frequent, it was clear that the Procedure and its design/control/revision is a necessary first step to 

reducing the incidence of FFP events. Though the ASRS data base was biased toward technical 

documentation, it is natural to target the physical document by which AMTs complete their tasks in order 

to ensure they have the necessary instructions to complete their tasks accurately. This finding appeared in 

all analyses and was the largest CF in all except the NTSB incidents. Following Procedures in importance 

was the Social system surrounding the maintenance activities. Specifically, the primary issues included the 

inconsistency in enforcement of procedures policy, failure to provide a procedure for a task, and doing little 

to mitigate time pressures on actors performing the task. Actors were found to contribute less to FFP events 

than Procedures and Social. However, when they did, it was because they lacked training, experience, and 

showed a tendency to “cut corners.” The Task itself was less of a CF than the previous two categories, 

which may reflect a perspective that tasks are an inherent part of the job and therefore cannot necessarily 

contribute to incidents in the same way Procedures or Actors do. Finally, Environment was not often cited 

apart from the visual environment.

Table 7 shows the resulting Best Practices and Challenges derived from the analyses conducted in Tasks 

1 and 2. 
11

Table 7. Recommended Best Practices and Accompanying Challenges Compiled from All Sources

Best Practice Challenges Example

Procedures are 

technically accurate.

Current written procedures 

may not reflect the exact 

configuration of the aircraft 

due to less-than-perfect record 

keeping.

Inaccurate procedures lead at best to confusion and 

delay, at worst to FFP errors. It can be particularly 

difficult in Part 91 maintenance operations to ensure 

that each aircraft worked on has adequate records, 

specifically when maintenance has been performed 

elsewhere. For Part 121/135 operations, the sheer 

volume of procedures can be daunting.

Procedures are designed 

to conform to Human 

Factors guidelines for 

content, organization, 

readability, and 

graphics.

Changing formats does not 

have a high priority in 

maintenance organizations. 

Waiting until a planned 

automation system arrives can 

be a good excuse for inaction.

HF-based guidelines are readily available from many 

sources listed in the Reference section, e.g., Drury 

(2009). They have been proven to both reduce errors 

and be more acceptable to end-users. After so many 

controlled demonstrations of the benefits of using HF 

design best practices, there is no longer any reason 

not to use them.

Procedures incorporate 

explicit input from 

users, i.e., AMTs and 

inspectors with direct 

knowledge of the tasks.

Engineers producing 

procedures may assume that 

the OEM-provided 

information is adequate. They 

may also have little time to 

consult current users. Finally, 

they may not respect the 

expertise of on-the-ground 

knowledge provided by users.

Procedures are often derived from OEM sources, 

modified for local use. However, there is no guarantee 

that both the OEM and the engineers modifying the 

procedures have performed the task themselves. A 

current user who knows the task will be able to 

incorporate changes learned on-the-job that could be 

improvements if approved by engineering.

Procedures have been 

validated by observing 

their use in detail.

Many procedures are not 

validated because of time 

constraints and perceived cost 

considerations. Also the task 

may not be available to 

observe while the procedure is 

being written.

A procedure produced with validation is a theory, 

perhaps a good theory but that has not been 

determined until validated. It does take time and effort 

to find a user and follow a good validation process, 

but the cost of a single error or ambiguity far exceeds 

the costs involved in validation.

Procedures are kept upto-date.The procedure revision 

process adds to the workload 

of those engineers tasked with 

procedure production. When 

revisions are user-driven they 

may be seen as optional.

There are frequent complaints about the revision 

process, e.g., too prolonged, no feedback to users. 

Revision is often given cursory attention by engineers 

as this is the most expeditious way to reduce their 

workload. However, dissatisfaction with the revision 

process itself and its outcomes undermine trust and 

confidence in the procedure process as a whole, 

potentially leading to less-rigorous use of procedures.

Procedures are used 

only as needed, and at a 

suitable level for 

professional users.

Procedures are often seen as 

the automatic solution to errors 

in task performance, although 

As noted in Task 1 report, AMTs and inspectors can 

work at skill-based, rule-based, or knowledge-based 

levels. Rule-based implies procedures but is only 

effective when the conditions of use exactly match the 
12

Best Practice Challenges Example

other solutions need to be 

considered.

task at hand. Hence the number of “Existing Parts 

installed incorrectly” events. Consideration is already 

given to skill-based activities in procedures, e.g., not 

telling an AMT how to wire a castle nut. This means 

that consideration of skill-based performance and 

even knowledge-based performance must be given to 

highly professional employees such as AMTs and 

inspectors. This means writing for the range of users, 

perhaps at different levels for those new to, or 

returning to, a task.

Organizational policy 

on use of procedures is 

in place and is enforced 

by all levels of 

management as well as 

by peers. 

Although a high-level policy 

may have been written, it may 

not have been communicated 

to all levels. Also, the policy 

may be tempered in 

application by expediency of 

completing work to a deadline. 

Peer-to-peer communications 

may be awkward/unusual in 

strict policy enforcement.

A high-level policy does not guarantee compliance. 

As management changes and work becomes more 

competitive, there will always be pressures on midlevel managers and end-users to sacrifice the policy to 

a norm of expediency. Procedure users may be 

implicitly or explicitly rewarded for meeting 

deadlines despite the policy.

Procedures are 

available when needed 

and users can always 

find the correct 

procedure.

Procedures may not be 

available instantly due to 

technical or managerial 

problems, or even lack of 

responsible personnel on a 

night shift. The user may not 

be able to locate the procedure 

even if it is available.

In the day-to-day running of a facility, the procedure 

may not be instantly available to the user who is then

tempted to work without a procedure, use a similar 

but not identical procedure, or use a known outdated 

procedure rather than be the cause of dispatch delay.

Users are insulated 

from time/production 

pressures.

Time/production pressures are 

always important in an 

organization. As organizations 

become leaner, this pressure 

will increase.

Although time and production pressure exist, they 

should not be allowed to influence quality. The 

technical term is Speed-Accuracy Trade-Off (SATO), 

meaning that accuracy for many tasks decreases as 

speed increases. In high-reliability organizations 

(HRO’s) all actors have the ability to stop the process 

when an error is likely to occur. Aviation maintenance 

and inspection should be no exception as the 

consequences are high and well-known to all actors 

and the organization.

Users are trained 

appropriately, 

experienced, and 

knowledgeable.

Training may be cursory or 

poorly-implemented as it is not 

seen as a priority. Recurrent 

training is important. The test 

is knowledgeability, not 

Training and experience comprise fitting-the-actor-to 

the-task, a necessary second step after are fitting-thetask-to-the-actor, i.e., optimizing the task and its 

environment. They are not a substitute for good 

procedure design, good workplace design, or good 
13

Best Practice Challenges Example

merely a signed-off training 

experience. 

organization design. Most aviation maintenance 

organizations already have procedures in place to 

record training and experience, but occasional cornercutting occurs in the name of flexibility of task 

assignment. Zero tolerance in appropriate here.

Users have an 

appropriate and known 

plan to improve or 

optimize procedures

People using procedures often 

see a “better way” to perform 

them. If they have no simple 

process to officially optimize 

the procedure they will often 

go ahead and change the 

procedure without due 

oversight by engineering.

It is a natural instinct to try to improve the work 

system, and end-users of procedures are no exception. 

They will see ways to make the task better, quicker, 

easier by performing the task diligently. The danger is 

that this desired innovation meets frustration and 

perceived apathy when a change suggestion is made 

and not responded to in a timely manner thus 

encouraging the use of local norms that deviate from 

written procedures. This is not only technically 

dangerous but it also undermines confidence in the 

whole procedure system. Errors are the inevitable 

result.

Procedures are in a 

medium suitable for use 

at the working point.

If the procedure document is 

bulky or difficult to control at 

the work point, there is a 

natural tendency to leave it in 

a safe area and perform the 

task from memory.

Excessively long, bulky, or repetitive procedures are 

seen as tiresome by end users, leading to ignore the 

at-site use of the procedure, especially in constricted 

areas or adverse weather conditions. Better HF design 

of the document or medium, e.g., laminated cards or 

smart phones, have been shown to encourage use at 

the work point rather than working from memory, 

again resulting in reduced error rates. Improved 

compliance with procedures is accompanied by 

increased confidence in the procedure system.

There is a known policy 

to deal with incorrect or 

incorrectly installed 

parts on the aircraft.

It can be quicker and easier for 

the user to improvise a 

procedure from a knowledge 

base than to work through 

layers of management to try to 

solve the problem at hand.

An experienced AMT or inspector has a good 

knowledge of what to do if the procedure does not 

exactly match the situation as found. Using this 

knowledge instead of following the arduous process 

for locating a correct procedure is error-prone 

behavior. A known policy, consistently enforced, can 

prevent such seemingly expedient behavior.

A high-quality visual 

environment is 

provided, including aids 

for seeing inaccessible 

work points.

Better hangar and task lighting 

can be seen as adding costly 

frills to appease grumbling 

AMTs. Also, control of task 

lighting is cited as a problem 

of hoarding good flashlights or 

even misappropriating them.

Good lighting and good aids for seeing around 

intervening structures are effective in reducing errors 

and improving job satisfaction. There are known and 

proven techniques for lighting design in aviation 

maintenance, but users rarely express the need for 

appropriate lighting, seeing the current situation as a 

norm unlikely to be changed for the better.
14

The derived best practices and challenges were purposefully general in that we sought to create a set of 

guiding principles that could be validated through a data collection phase in future work. Another 

justification for the generality of the items was to ensure that they could transcend FAR Part. That is, 

providing a high-quality visual environment in practice would be different for Part 91 vs. Part 121 

maintenance operations.

DISCUSSION

The purpose of this study was to examine existing classification schemas for maintenance specific 

errors (commonly referred to as FFP events) and to add to their utility by incorporating reported data via 

ASRS and NTSB incident reports. The outcome of this work provides a classification system (TAPES, 

Appendix B) which is both broader and deeper than previously proposed schemas and that is applicable 

specifically to aviation maintenance events. 

The development and utilization of the TAPES system has two key benefits with regard to FFP events 

in maintenance operations. First, the literature and event reports concur that errors are due to more factors 

than incorrect manuals or inadequate guidance. Therefore, the TAPES classification system addresses the 

complicated patterns of contributing factors and their resulting errors, which can arise when real 

maintenance is performed on real aircraft (as opposed to the fictitious scenarios played out in training 

courses). By examining errors at this level, the aviation maintenance industry is supplied with evidence for 

possible FFP countermeasure strategies that address real CFs.

A second benefit to using the TAPES system is the ability to create Best Practices that address FFP 

events. When errors are classified appropriately, Challenges are identified and Best Practices naturally 

emerge. Thus, taking the time to examine what steps to take to reduce and prevent FFP events will directly 

result in cost savings for maintenance organizations due to reduced FFP events. This, in turn, promotes 

longer term Best Practices that may be facilitated industry-wide.

FUTURE DIRECTIONS

This report highlights the challenges and issues of FFP, which have been researched and uncovered 

spanning the past 30 years. It is unfortunate to note that these challenges and recommendations have not 

changed a great deal in that time period. This highlights a systemic problem with the way in which FFP 

events are addressed both within the literature and in the hangar. Therefore, it seems pertinent that the future 

of addressing FFP events center on the following: classifying errors such that practical Countermeasures 

and Best Practices may be derived, the effectiveness of derived countermeasures and Best Practices, and 

quantifying cost savings in an organization from implemented countermeasures and Best Practices. 

SUMMARY

The intent of the report was to review the previous literature on FFP events, integrate event reports that 

identify FFP as an outcome, and create a more specific classification system that categorizes root causes of 

FFP events within aviation maintenance. Within the scope of that effort, we used the classification system 

to identify CFs and possible Best Practices that can be used to target and mitigate FFP events industrywide. Finally, we recommend a plan for future research that transforms the TAPES classification system 

into a practical tool to derive countermeasures and best practices. Phase 2 of this research will utilize a wide 

range of first-hand accounts of FFP events and gather a practical assessment of the effectiveness of the Best 

Practices identified in the current report. The final products of this research will provide the tools, guidance, 

and information necessary to ensure practical application to the maintenance community. 
15

REFERENCES

Banks, J. O., Wenzel, B.M., Drechsler, G.K., & Crayton, L.G. (2013). Analysis of aviation maintenance 

failure to follow written procedures. Event Reports.

Boring, R. L., Gertman, D. I., & Le Blanc, K. (2011). Proceedings of the Human Factors and Ergonomics 

Society Annual Meeting. Human reliability analysis for computerized procedures. Santa Barbara, 

CA: Sage Publications. 

Bradley, W. E. (2009). Ability and Performance on Ill-Structured Problems: The Substitution Effect of 

Inductive Reasoning Ability. Behavioral Research in Accounting, 21(1), 19-35.

Center for Chemical Process Safety (1996). General References, in Guidelines for Writing Effective 

Operating and Maintenance Procedures. Hoboken, NJ: USA. John Wiley & Sons, Inc.

Chervak, S. C., & Drury, C. G. (2003). Effects of job instruction on maintenance task performance. 

Occupational Ergonomics, 3(2), 121-132.

Drury, C. G. (1998). Case study: error rates and paperwork design. Applied Ergonomics, 29(3), 213-216.

Drury, C. G. (2000). Development and Use of the Documentation Design Aid. Proceedings of the Human 

Factors and Ergonomics Society 44th Annual Meeting, Santa Monica, CA, 783-6.

Drury, C. G. (2005). Handbook of system reliability in airframe and engine Inspection. U.S. DOT, FAA, 

2005.

Drury, C. G. (2009). Chapter 6: Procedures and Technical Documentation. FAA Human Factors Guide 

for Aviation Maintenance and Inspection. Retrieved from http://www.2.hf.faa.gov/hfguide/

Drury, C. G., & Johnson, W. B. (2013). Proceedings of the Human Factors and Ergonomics Society 

Annual Meeting. Writing aviation maintenance procedures that people can / will follow. Santa 

Barbara, CA: Sage Publications. 

Drury, C. G. (2015). Human Factors and Ergonomics implications of big data analytics: Chartered 

Institute of Human Factors and Ergonomics annual lecture. Ergonomics, 58(5), 659-673.

Drury, C. G., Guy, K. & Wenner, C. A. (2010). Outsourcing Aviation Maintenance: Human Factors 

Implications, Specifically for Communications. The International Journal of Aviation Psychology, 

20(2), 124-143.

Drury, C. G., Paquet, V. & Kelly, H. (2015). Experimental Design and Analysis. In J. R. Wilson and S. 

Sharples (Eds.), Evaluation of Human Work (4th ed., pp 37-60). Boca Raton, FL: CRC Press.

Drury, C. G., Patel, S. C. & Prabhu, P. V. (2000). Relative advantage of portable computer-based 

workcards for aircraft inspection. International Journal of Industrial Ergonomics. 26(2), 163-176.

Drury, C. G. & Prabhu, P. (1996). Information requirements of aircraft inspection: framework and 

analysis. International Journal of Human-Computer Studies, 45, 679-695.

Drury, C.G., Prabhu, P., & Gramopadhye, A. (1990). Proceedings of the Human Factors Society 34th 

Annual Conference. Task analysis of aircraft inspection activities: methods and findings. Santa 

Barbara, CA: Sage Publications.
16

Eurocontrol (2010a). Operator’s guide to human factors in aviation: Assessing procedures. Retrieved 

from http://www.skybrary.aero/index.php/Assessing_Procedures_%28OGHFA_BN%29 

Eurocontrol (2010b). Operator’s guide to human factors in aviation: Adherence to sop’s. Retrieved from 

http://www.skybrary.aero/index.php/Adherence_to_SOPs_%28OGHFA_BN%29

Eurocontrol (2014). Operator’s guide to human factors in aviation procedure assessment tool (PAT). 

Retrieved from http://www.skybrary.aero/bookshelf/books/635.pdf

Federal Aviation Administration (2003). Standard operating procedures for flight deck crewmembers. US 

AC No: 120-71A. Washington, DC: US. Department of Transportation.

Federal Aviation Administration Safety Team (2009). Failure to Follow Procedures – Installation. 

Retrieved from 

https://www.faasafety.gov/files/gslac/courses/content/67/741/FFP%20Text%20Only%20Version.pdf

Federal Aviation Administration (2013). Aircraft Design—Original Equipment Manufacturer/Design 

Approval Holder Continuous Monitoring of Service History Best Practices Task Force. Final report 

of Commercial Aviation Safety Team SE-170 Task Force.

Green, M. (2004). Nursing error and human nature. Journal of Nursing Law, 9(4), 37-44.

Groner, N.E (2005). On Not Putting the Cart Before the Horse: Design Enables the Prediction of 

Decisions about Movement in Buildings at NIST SP 1032.

Hanson, D. C., Boucek, G. P., Smith, W. D. Hendrickson, J. F., Chikos, S. F., Howison, W. W., & 

Berson, B. L. (1982). Flight operations safety monitoring effects on the crew alerting system. In 

Proceedings of The Human Factors and Ergonomics Society Annual Meeting, 26, 783-787).

Hassall, M. E., Sanderson, P. M., & Cameron, I. T. (2010, September). Using cognitive work analysis 

techniques to identify human factor hazards. In Proceedings of the Human Factors and Ergonomics 

Society Annual Meeting, 54(4) 269-273. Sage Publications. 

Hazelhoff, Roelfzema, N. (2012). Procedure not followed. Retrieved from 

http://surveygizmolibrary.s3.amazonaws.com/library/173359/Procedurenotfollowed.pdf

Hollomon, M. Drechsler, G., & Crayton, L. (2015). Analysis of aviation maintenance failure to follow 

written procedures event reports. Presented at the 18th International Symposium on Aviation 

Psychology Conference. Dayton, OH. 

Health and Safety Executive (HSE) (1995). Improving compliance with safety procedures reducing 

industrial violations. Retrieved from 

http://www.hse.gov.uk/humanfactors/topics/improvecompliance.pdf

Health and Safety Executive (HSE). (1999). Reducing error and influencing behavior. London, UK: HSE 

Books.

Health and Safety Executive (HSE). (2000). Improving maintenance a guide to reducing human error. 

Retrieved from http://www.hseni.gov.uk/improving_maintenance_-

_a_guide_to_reducing_human_error.pdf

Health and Safety Executive (HSE). (2002). Revitalising health and safety in construction. Retrieved 

from http://www.hse.gov.uk/consult/disdocs/dde20.pdf
17

Health and Safety Executive (HSE). (2005). Supply chain issues for offshore accidents in the Southern 

North Sea. Retrieved from http://www.hse.gov.uk/research/rrpdf/rr352.pdf

Health and Safety Executive (HSE). (2006). Analysis of injuries to UK care assistants employed by local 

authorities in health and social care. Retrieved from 

http://www.hse.gov.uk/research/hsl_pdf/2006/hsl06116.pdf

Health and Safety Executive (HSE). (2009). Procedures audit tool (style/language/layout). Retrieved 

from http://www.hse.gov.uk/humanfactors/topics/procedures-audit-tool.pdf

Health and Safety Executive (HSE). (2012). Core topic 3: Identifying human failures. Retrieved from 

http://www.hseni.gov.uk/improving_maintenance_-_a_guide_to_reducing_human_error.pdf

Health and Safety Executive (HSE). (2012). Human factors that lead to non-compliance with standard 

operating procedures. Retrieved from http://www.hse.gov.uk/research/rrpdf/rr919.pdf

Hudson, P. (1999). The human factor in system reliability -Is human performance predictable? 

Proceedings from NATO RTO Human Factors and Medicine Panel Workshop, Siena, Italy. St. Joseph 

Ottawa/Hull.

International Civil Aviation Organization (ICAO) (1989). Human factors digest no 1, International Civil 

Aviation Organization. Circular No. 216-AN/131, Montreal, Canada, International Civil Aviation 

Organization, Reprinted as CAP 719, Fundamental Human Factors Concepts. CAA (UK), 2002. 

Jansen, S., Chaparro, A., Downs, D., Palmer, E., & Keebler, J. (2013, September). Visual and Cognitive 

Predictors of Visual Enhancement in Noisy Listening Conditions. In Proceedings of the Human 

Factors and Ergonomics Society Annual Meeting, 57(1) 1199-1203. SAGE Publications.

Johnson, W.B. (August 4, 2014). Improving Communication about Technical Publications: A Safety 

Enhancement Report. Aviation Pros. Retrieved from 

http://www.aviationpros.com/article/11521715/improving-communication-about-technicalpublications-a-safety-enhancement-report

Johnson, W.B. & Avers, K.A. (Eds.) (2014). The Operator’s Manual for Human Factors in Aviation

Maintenance. Washington, DC: Federal Aviation Administration, (See www.humanfactorsinfo.com)

Johnson, W. B. & Watson, J. (2001). Installation error in airline maintenance. Federal Aviation 

Administration Office of Aviation Medicine. Retrieved from http://hfskyway.faa.gov 

Landry, S. J., Jacko, J. A., & Coulter, W. H. (2006). Impact of the use of techniques and situation 

awareness on pilots' procedure compliance. Proceedings of the Human Factors and Ergonomics 

Society Annual Meeting. Santa Barbara, CA: Sage Publications.

Lindroos, Ola. (2009). Relationships between observed and perceived deviations from normative work 

procedures. Ergonomics 52(12), 1487-1500.

MCS. (2012). Risk Analysis of using a surface bop.

Merlin, P. W., Bendrick, G. A., & Holland, D. A. (2011). Breaking the Mishap Chain. Government 

Printing Office.
18

National Institute for Health Research Service Delivery and Organisation Programme. (2010). 

Understanding the dynamics of organizational culture change; creating safe places for patients and 

staff. Southampton, UK: National Coordinating Center for the Service Delivery and Organisation.

National Transportation Safety Board. (2013). Loss of Control, Sundance Helicopters, Inc., Eurocopter 

AS350-B2, N37SH, Near Las Vegas, Nevada, December 7, 2011. Aircraft Accident Report 

NTSB/AAR-13/01. Washington, DC.

Norman, D. A. (1997). The invisible computer. Cambridge, MA: MIT Press.

Ockerman, J.J., & Prichett, A. R. (2000). A review and reappraisal of task guidance aiding workers in 

procedure following. International Journal of Cognitive Ergonomics. 4(3), 191-212. 

Park, J, & Jung, W. (2003). The operators’ non-compliance behavior to conduct emergency operating 

procedures—comparing with the work experience and the complexity of procedural steps. Reliability 

Engineering and System Safety. 82(2), 115-131. doi: 0.1016/S0951-8320(03)00123-6

Park, J., Jung, W., Ha, J., & Shin, Y. (2004). Analysis of operators’ performance under emergencies using 

a training simulator of the nuclear power plant. Reliability Engineering & System Safety, 83(2), 179-

186.

Park, J., Joeng, K., & Jung, W. (2005). Identifying cognitive complexity factors affecting the complexity 

of procedural steps in emergency operating procedures of a nuclear power plant. Reliability 

Engineering & System Safety, 89(2), 121-136.

Patel, S., Drury, C.G., & Lofgren, J. (1994). Design of workcards for aircraft inspection. Applied 

Ergonomics, 25(5), 283-293.

Pearl, A., & Drury, C.G. (1996). Improving the reliability of maintenance checklists. Human Factors in 

Aviation Maintenance—Phase V progress report. (DOT/FAA/AM-96/2). Washington, DC: Office of 

Aerospace Medicine.

Pugliese G., Gosnell C., Bartley J.M., & Robinson S. (2010). Injection practices among clinicians in 

United States health care settings. American Journal of Infection Control, 38(10), 789-798.

Rankin, W. (2008, November). Safety management systems and Boeing-related safety activities.

Presented at the Safety Management System (SMS) Workshop for Air Transport Industry, Tokyo. 

Retrieved from www.atec.or.jp/03_symposium/H20_SMS/SMS_WS_Boeing.pdf

Rasmussen, J. (1983). Skill, rules and knowledge; signals, signs, and symbols, and other distinctions in 

human performance Models. IEEE Transactions on Systems, Man and Cybernetics, SMC-13(3), 257-

266.

Reason, J. T. (1997). Managing the risks of organizational accidents. Farnham, UK: Ashgate Publishing 

Company.

Reason, J. T. (2009). Human error: Models and management. BMJ, 320, 768-770.

Reece, W. J., & Hill, S. G. (1995, October). Human performance analysis of industrial radiography 

radiation exposure events. Proceedings of the Human Factors and Ergonomics Society Annual 

Meeting, 39(9), 491-495. SAGE Publications.
19

Reynolds, J., & Drury, C. G. (1993). An evaluation of the visual environment in aircraft inspection. 

Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting, 34-38, Seattle, 

WA. 

Schroeder, D. J., Newton, D. C., Bryant, M. R., & Drechsler (in press). Failure to follow procedures: A 

multi-level review. Civil Aerospace Medical Institute: Federal Aviation Administration. 

Shappell, S. A., & Weigman, D. A. (2000). The human factors analysis and classification ‘system–

HFACS, (DOT/FAA/AM-00/7). Washington, DC: Office of Aerospace Medicine.

Stein, E. S., & Buckley, E. (1994). Human factors at the FAA Technical Center: Bibliography 1958-1994

(DOT/FAA/CT-TN94/50). Atlantic City International Airport: Federal Aviation Administration 

Technical Center. 

Sutton, I. (2010). Process risk and reliability management: Operational integrity management. Elsevier, 

Oxford, UK. 

Suzuki, T., Von Thaden, T. L., & Geibel, W. D. (2008). Influence of time pressure on aircraft 

maintenance errors. University of Illinois at Urbana Champaign.

Wachter, J.K., & Yorio, P.L. (2013). Human performance tools that engage workers: the best defense 

against errors and their precursors. Professional Safety, February 2013.

Walker, D. (2005). Supply Chain Issues for offshore accidents in the Southern North Sea. (Report 352). 

Retrieved from http://www.hse.gov.uk/research/rrpdf/rr352.pdf

Walker, I. R. (2011). Reliability in scientific research: improving the dependability of measurements, 

calculations, equipment, and software. Cambridge University Press.

Wenner, C., & Drury, C. G. (2000). Analyzing human error in aircraft ground damage incidents.

International Journal of Industrial Ergonomics. 26(2), 177-199.

Werfelman, L. (2013). Internationally noncompliant. AeroSafety World. Retrieved from 

http://flightsafety.org/aerosafety-world-magazine/december-2013/intentionally-noncompliant

Wiseman, S., Cox, A. L., & Brumby, D. P. (2013). Designing devices with the task in mind: Which 

numbers are really used in hospitals? Human Factors, 55(1), 61–74.

Xu, S., Song, F., Li, Z., Zhao Q., Luo, W., He, X., & Salvendy G. (2008). An ergonomics study of 

computerized emergency operating procedures: presentation style, task complexity, and training level. 

Reliability Engineering and System Safety, 93(10), 1500-1511.

ADDITIONAL RESOURCES

Angros, R., Johnson, W.L., Rickel, J., Scholer, A. (2002). Learning Domain Knowledge for Teaching 

Procedural Skills. In Proceedings of the International Joint Conference on Autonomous Agents and 

Multiagent Systems.

Avers, K., Johnson, B., Banks, J., & Wenzel, B. (2012). Technical documentation challenges in aviation 

maintenance: a proceedings report. (DOT/FAA/AM-12/16). Washington, DC: Office of Aerospace 

Medicine.
20

Backinger, C., & Kingsley, P. (1993). Write it right: Recommendations for developing user instruction 

manuals for medical devices used in home health care. (HHS Pub. FDA 93-4258). Rockville, MD: 

U.S. Department of Health and Human Services, Food and Drug Administration.

Ball, G. (2011). Energy Institute, Human factors briefing note. Retrieved from 

http://publishing.energyinst.org/publication/ei-technical-publications/process-safety/managementsystems/human-factors-briefing-note-no.-6-safety-critical-procedures

Boyce, M. W., Duma, K. M., Hettinger, L. J., Malone, T. B., Wilson, D. P., & Lockett-Reynolds, J. 

(2011). Human Performance in Cybersecurity A Research Agenda. In Proceedings of the Human 

Factors and Ergonomics Society Annual Meeting, 55(1), 1115-1119. SAGE Publications.

Callan, J. R., Kelly, R. T., Cardinal, B. F., Gwynne, J. W., Tolbert, M. T., & Sawyer, C. R. (1991, 

September). Assessing User Compliance with Procedures for Soft Contact Lens Care. In Proceedings 

of the Human Factors and Ergonomics Society Annual Meeting, 35(10), 703-707. SAGE 

Publications.

Carthey, J., & Clarke, J. (2010). Implementing human factors into healthcare. Retrieved from 

http://www.patientsafetyfirst.nhs.uk/ashx/Asset.ashx?path=/Interventionsupport/Human+Factors+How-to+Guide+v1.2.pdf

Chaparro, A., & Groff, L. (2001). Human Factors Survey of Aviation Technical Manuals Phase 1 Report: 

Manual Development Procedures. (DOT/FAA/AR-01/43). Washington, DC: Office of Aviation 

Research .

Chaparro, A., & Groff, L. (2002). Survey of Aviation Technical Manuals Phase 2 Report: User 

Evaluation of Maintenance Documents. (DOT/FAA/AR-02/34). Washington, DC: Office of Aviation 

Research.

Chaparro, A., & Groff, L. (2002). Survey of Aviation Maintenance Technical Manuals Phase 3 Report: 

Final Report and Recommendations. (DOT/FAA/AR-02/123). Washington, DC: Office of Aviation 

Research .

Chaparro, A., & Groff, L. S. (2002). Human Factors Survey of Aviation Maintenance Technical Manuals. 

In Proceedings of the 16th Annual Human Factors in Aviation Maintenance Symposium.

Civil Aviation Authority (2003). Aviation Maintenance Human Factors (EASA/JAR145 Approved 

Organisations): Guidance Material on the UK CAA Interpretation of Part-145 Human Factors and 

Error Management Requirements (Chapter 3). West Sussex, UK.

de Brito, G. (2002). Towards a model for the study of written procedure following in dynamic 

environments. Reliability Engineering & System Safety, 75, 233-244.

Degani, A., Heymann, M., & Shafto, M. (1999, September). Formal aspects of procedures: The problem 

of sequential correctness. In Proceedings of the Human Factors and Ergonomics Society Annual 

Meeting, 43(20), 1113-1117. SAGE Publications.

Department of Health and Human Services (2002). On-site evaluation of IRB issues in university research 

program. Rockville, MD: Office for Human Resource Protection. 

Dhillon, B. S., & Liu, Y. (2006). Human error in maintenance: a review. Journal of Quality in 

Maintenance Engineering, 12(1), 21-36.
21

Diemand-Yauman. C., Oppenheimer, D. M.,Vaughan, E. B. (2010). Fortune Favors the Bold (and the 

Italicized): Effects of Disfluency on Educational, Cognition, 118(1) 111–115.

Dismukes, R. K. (2012). Prospective memory in workplace and everyday situations. Current Directions 

in Psychological Science, 21(4), 215-220.

Dreisbach, G. (2012). Mechanisms of Cognitive Control The Functional Role of Task Rules. Current 

Directions in Psychological Science, 21(4), 227-231.

International Air Transportation Association (IATA) (2011). International Air 

Transportation Association 2011 Annual Report. Singapore: Giovanni Bisignani.

Jolly, J. D., Hildebrand, E. A., & Branaghan, R. J. (2012). Better instructions for use to improve reusable 

medical equipment (RME) sterility. Human Factors: The Journal of the Human Factors and 

Ergonomics Society, doi: 0018720812456393.

Karwal, A.K., Verkaik, R., & Jansen, C. (2000). Non-Adherence to Procedures-Why Does it Happen. 

In Flight Safety Foundation 12th Annual European Aviation Safety Seminar. Amsterdam, 

Netherlands.

Kearns, S. K., & Sutton, J. E. (2013). Hangar Talk Survey Using Stories as a Naturalistic Method of 

Informing Threat and Error Management Training. Human Factors: The Journal of the Human 

Factors and Ergonomics Society, 55(2), 267-277.

Ma, J., Drury, C. G. & Marin, C. V. (2010). Language Error in Aviation Maintenance: Quantifying the 

Issues and Interventions in Four World Regions, The International Journal of Aviation Psychology, 

20(1), 25-47.

Manzey, D., Luz, M., Mueller, S., Dietz, A., Meixensberger, J., & Strauss, G. (2011). Automation in 

surgery: The impact of navigated-control assistance on performance workload situation awareness 

and acquisition of surgical skills. Human Factors, 53, 584.

Marx, D. (1998). Learning from our mistakes: A review of maintenance error investigation and analysis 

systems. Washington, DC: FAA.

McKeon, C. M., Fogarty, G. J., & Hegney, D. G. (2006). Organizational factors: impact on administration 

violations in rural nursing. Journal of Advanced Nursing, 55(1), 115-123.

Rankin, L. (2013). Failure to Follow Procedures: Airline Engineering & Maintenance Safety. Presented at 

Flightglobal and Flight International in Partnership with the Flight Safety Foundation Airline 

Engineering & Maintenance Safety. London, England.
A-1

APPENDIX A

Logical Groups for Contributing Factors Found in the Literature: 

Iteration 1 of TAPES Classification System

Figure A1. Hierarchical structure for Task Contributing Factors.
A-2

Figure A2. Hierarchical structure for Actor Contributing Factors.
A-3

Figure A3. Hierarchical structure for Procedure Document Contributing Factors
A-4

Figure A4. Hierarchical structure for Environment Contributing Factors
A-5

Figure A5. Hierarchical structure for Social Contributing Factors
B-1

APPENDIX B

Final Classification Schema for Event Patterns and Contributing Factors Specific to 

Maintenance Operations: Iteration 2 of TAPES Classification System

Figure B1. Procedure Event Pattern Classification Scheme.
B-2

Figure B2. Hierarchical structure for Task Contributing Factors.
B-3

Figure B3. Hierarchical structure for Actor Contributing Factors.
B-4

Figure B4. Hierarchical structure for Procedure Document Contributing Factors
B-5

Figure B5. Hierarchical structure for Environment Contributing Factors.
B-6

Figure B6. Hierarchical structure for Social Contributing Factors.
C-1

APPENDIX C

Summarized Sources for Event Patterns/Contributing Factors, 

Best Practices, Challenges, and Good Questions2

Table C1. Example from Event Patterns/Contributing Factors Table

Source Event Patterns / Contributing Factors

Revitalizing Procedures 

(HSE, 2009)

Procedures are not correct or out-of-date

Procedures are difficult to use or follow

Procedures are not readily available/portable

There are easier ways of performing the task

Pressure from peers

A failure to understand the risks

Table C2. Example from Best Practices Table

Source Best Practices

Revitalizing Procedures 

(HSE, 2009)

be accurate and complete

be clear and concise with an appropriate level of detail

be current and up to date

state necessary precautions for hazards

use familiar language

use consistent terminology

reflect how tasks are actually carried out

promote ownership by users

be in a suitable format; and be accessible

Design the job or task so that the correct procedure is hard 

to avoid (e.g., by engineering-out short cuts through 

equipment design or programmable logic controllers)

Base the procedure on how the task is actually performed. 

The operators may have devised an informal procedure that 

is quicker/easier and these methods should be incorporated 

into the formal procedure (as long as safety/quality issues 

are not compromised)

Identify incentives to take short cuts (such as work 

pressures) and address these directly

Adopt a control and review process to keep procedures 

relevant and up-to-date

Workcard Best Practices detail

 

2 Note that each table runs to several pages and only a few examples are presented here.
C-2

Table C3. Example from Challenges Table

Source Challenges

Improving Maintenance 

(HSE,2000)

Procedures that are needlessly too detailed or complicated 

Procedures that contain technical errors 

Clarity of instructions in procedures/manuals 

Speed with which information can be found in 

procedures/manuals

Procedures that are needlessly too detailed or complicated 

Johnson (2014) Need process to identify difficult tasks

Need process to report issues to OEM

Need process to improve response time for operator issues

Need process to clarify information requirements to 

promote change

Table C4. Example from Good Questions Table

Source Good Questions

Revitalizing Procedures 

(HSE, 2004)

Are your procedures accessible?

Are they actually followed by staff?

Do they always use the procedures? - Why not?

Are they written so that they can be understood and followed 

easily?

Do they reflect the tasks as they are actually carried out?

Do the procedures include key safety information?

Are they kept up to date and reviewed occasionally?

Are they of the right level of detail?

Do they include safety critical tasks?

Do they identify all the necessary warnings?

Are they easy to use?

Do operators use ‘black books’? - Why?

Ask them to explain how to do a specific task - Do they 

always do it that way? - How do they remember how to do it?

Were they involved in developing procedures?

Procedures audit tool (HSE, 

2009)

Make good use of open space; avoid clutter; remove 

unnecessary information.

Use margins; justify text to the left. 

Ensure font size is appropriate for all users and conditions 

(e.g., users with impaired eyesight; poor lighting; PPE with 

restricted visibility). 

Check that the use of colour is appropriate (availability & 

reliability of suitable printers; colour-blind personnel; 

contrast of text under artificial lighting etc.). 
C-3

Source Good Questions

Use consistent type-face and spacing. 

Use page-breaks to ensure steps are not split across pages. 

Number all steps (e.g., 1.1, 1.2, 1.2.1 ...). 

Differentiate clearly between steps (e.g., use a different 

tabular cell for each step). 

Have one action per procedural step. 

State who does what and when. 
D-1

APPENDIX D

Classifying Contributing Factors Noted by Sources

Table D1. Mid-Level Contributing Factors with Frequencies from the Literature

Mid-Level Code Title Count

100 Task Contributing Factors 0

110 Problems with the task as found 5

120 Errors made during task performance 21

200 Actors Contributing Factors 0

210 Actor Background 24

220 Actor Current State 39

300 Procedure Document Contributing Factors 3

310 Physical Document 88

320 Document writing and production 55

330 Document revision 17

400 Environmental Contributing Factors 4

410 Thermal Environment 1

420 Visual Environment 3

430 Auditory Environment 1

440 Physical Space for Task 10

500 Social Contributing Factors 0

510 Organizational Policy 16

520 Organizational Effectiveness 37

530 Local Norms 39

540 Organizational Pressures 21
D-2

Table D2. Original Task Contributing Factors with Frequencies from the Literature

Code Title Count

100 Task Contributing Factors

110 Problems with the task as found 1

111 Existing parts installed incorrectly 2

112 Prior tasks incorrectly performed

113 Prior tasks incorrectly signed off 1

114 Miscommunication at hand-off

115 A/C configuration not as specified in procedure

116 Equipment / tools not available 1

120 Errors made during task performance

121 Uses wrong procedure 3

122 Task performed from memory 10

123 Document not carried to work site 2

124 Task steps not performed in specified order 3

125 Missed task step

126 Parts installed incorrectly

127 Fails to inspect / sign off task 1

128 Fails to replace tools / equipment after use

129 Forgets to perform task 2
D-3

Table D3. Original Actor Contributing Factors with Frequencies from the Literature

Code Title Count

200 Actors Contributing Factors

210 Actor Background

211 Training 9

212 Experience 7

213 Knowledge 5

214 Qualification 2

215 Native Language vs. Documentation Language 1

216 Motivation/commitment to Follow Procedures

220 Actor Current State 1

221 Fatigued 5

222 Cutting Corners 10

223 Sees Procedure as too restrictive 6

224 Believes rules do not apply 3

225 Needs to explore boundaries 1

226 needs to make task more interesting 4

227 Task Familiarity 4

228 Workload 5
D-4

Table D4. Original Procedure Contributing Factors with Frequencies from the Literature

Code Title Count

300 Procedure Document Contributing Factors 3

310 Physical Document 15

311 Information Content - text / pagination issues 20

312 Information content illustration issues 4

313 Information readability / comprehension 27

314 Information organization 11

315 Emphasis on hazards and cautions 5

316 Physical handling of document 3

317 Media Type 3

320 Document writing and production 5

321 Incorrect procedure 18

322 Outdated procedure 7

323 Un-validated procedure 8

324 Procedure produced w/o task analysis 7

325 Procedure produced w/o end user input 1

326 Procedure writer has not seen task/workplace

327 Procedure refers to other manuals/procedures 1

328 Format consistency across procedures 5

329 Incompatibility between resources 3

330 Document revision 10

331 Revision process obscure to end users 1

332 Revision process too slow 2

333 Revision process ineffective 3

334 No feedback to end-user about revisions 1

335 Revision not validated
D-5

Table D5. Original Environment Contributing Factors with Frequencies from the Literature

Code Title Count

400 Environmental Contributing Factors 4

410 Thermal Environment

411 Working outside 1

412 Excessive heat

413 Excessive Cold

414 Excessive Humidity

415 Excessive Wind

420 Visual Environment 1

421 Working Outside 1

422 Poor General Lighting

423 Poor Task Lighting

424 Glare

425 Key point hidden from view 1

430 Auditory Environment 1

431 Excessive Noise

440 Physical Space for Task 1

441 Work in confined/Restricted Space 2

442 Design of Workplace 3

443 Condition of Workplace 1

444 Inadequate / Missing tools /equipment 3
D-6

Table D6. Original Social Contributing Factors with Frequencies from the Literature

Code Title Count

500 Social Contributing Factors

510 Organizational Policy 1

511 Alternatives to Procedures considered 6

512 Written rules to control maintenance activities

513 Interface between MRO and operator 1

514 Training Actors for Procedure awareness 7

515 Fair enforcement policy implemented locally 1

520 Organizational Effectiveness 2

521 Ineffective Procedure Policy enforcement 10

522 Ineffective guidance / coaching 3

523 Unresponsive to employee suggestions 1

524 Lack of resources for task performance 3

525 Lack of resources for procedures process 2

526 Negative organizational culture 4

527 Changes condoned w/o task updating procedures

528 Supervisors turn blind eye to non-compliance 3

529 Inadequate supervision 4

591 Reasons for procedures not understood by actors 3

592 Shift Handover ineffective 2

530 Local Norms 8

531 No procedures provided 19

532 Actor did not know current practice was wrong 1

533 It has been done that way for years 4

534 Violations ignored when they interfere with production 1

535 Actor believes procedure better if modified 2

536 Actor believes procedure not needed by experienced people 3

537 Actors believe they know what they are doing

538 Deviation demonstrates actors job knowledge 1

540 Organizational Pressures 2

541 Time / production Pressure 15

542 Interruptions tolerated 4

543 Distractions tolerated