Human Factors: FAA's Guidance and Oversight of Pilot Crew Resource Management Training Can Be Improved
Released: 1997-12-09T00:00:00

Letter

Background

Airline travel is one of the safest modes of public transportation in the United States. The current level of airline safety has been achieved, in part, because the airline industry and government regulatory agencies have implemented rigorous pilot training and evaluation programs. The major airlines have training programs for pilots that focus on, among other things, maintaining flying skills, qualifying to fly new types of aircraft, and acquiring skills in dealing with emergencies.

FAA’s original regulations for the airlines’ general training programs—referred to in this report as part 121—spell out the number of hours of training required in particular areas, such as the time spent practicing emergency procedures. Effective for 1996, FAA instituted a requirement for CRM training under part 121 that states the following: “After March 19, 1998, no certificate holder may use a person as a flight crewmember, and after March 19, 1999, no certificate holder may use a person as a flight attendant or aircraft dispatcher unless that person has completed approved crew resource management or dispatcher resource management initial training, as applicable, with that certificate holder or with another certificate holder.” FAA believes that this training should improve flight crews’ performance.

As an alternative to training under these regulations, airlines may apply to participate in AQP. Eight of the 10 major airlines have applied to, and been approved for participation in, AQP. Unlike traditional part 121 training, AQP specifies the criteria for the required level of performance in certain types of maneuvers, rather than hours of training, and it integrates CRM training with technical flying skills. The airlines are expected to fully implement AQP over a period of time, up to 8 years. Full implementation means that the airlines have trained their pilots for each type of aircraft they fly. Training, however, occurs only after the airline has gone through three other stages: (1) getting approval to participate in the program, (2) developing a training curriculum, and (3) training instructors. Continuing crew training, the last stage, is to occur annually.

Responsibility for AQP and traditional part 121 training rests with different FAA branches. The AQP Branch within the Office of Flight Standards Services oversees AQP, and the Branch expects to transfer many of its oversight responsibilities to inspectors in the field as each airline fully implements its AQP. The administration of traditional part 121 training is divided between the Air Carrier Training Branch, which sets training requirements, and the flight standards inspectors in the field, who are responsible for overseeing the training. FAA’s inspectors periodically review and approve airlines’ curricula and training materials and observe training.

CRM is a “human factors” approach for improving aviation safety by preventing or managing pilots’ errors. Human factors refers to a multidisciplinary effort to develop information about human capabilities and limitations and to apply this information to equipment, systems, facilities, procedures, jobs, environments, training, staffing, and personnel management for safe and effective human performance. Under this approach, pilots are trained to recognize potential mistakes in judgment or actions and to compensate for them to prevent accidents and incidents.

For example, in training for initial departure, CRM training has the captain practice briefing the crew about the actions to be taken if the takeoff must be aborted because of an emergency. CRM also teaches the crew to question orders when they believe they have information that indicates these orders are inappropriate. Similarly, CRM training teaches the crew to anticipate problems and make decisions that take these anticipated problems into account.

Airline Pilots’ Performance Was a Contributing Factor in Many Accidents and Incidents

About 30 percent of the 169 accidents and 18 percent of the 3,901 incidents that occurred from 1983 through 1995 were caused at least in part by pilots’ performance, according to our analysis of the National Transportation Safety Board’s (NTSB) and FAA’s data. Furthermore, the accident data indicate that nearly one-third of the accidents occurred because the pilots either did not follow, or did not correctly follow, CRM principles. The most frequently occurring accidents and incidents included collisions on the ground with objects and other airplanes, flights through turbulent weather that resulted in injuries, and deviations from flight paths that had the potential to cause an in-flight collision.

Data Show Pilots’ Performance Contributed to Accidents and Incidents

On the ground, pilot performance was associated most frequently with airplanes colliding with vehicles, buildings, other equipment, or animals. This was the case for both accidents (32 percent) and incidents (34 percent). Figure 1 shows the types of accidents and incidents on the ground reported from 1983 through 1995.

Loss of control on ground NTSB cited 62 events associated with pilots’ performance in 169 accidents.

FAA cited 446 events associated with pilots’ performance in 3,901 incident reports.

In the air, pilot performance was most frequently associated with injuries to passengers and flight attendants during turbulent weather—41 percent of accidents and 12 percent of incidents. Figure 2 shows the types of accidents and incidents in the air that were reported.

In addition to the accidents and incidents discussed above, FAA maintains data separately for those occasions on which pilots failed to comply with the air traffic controller’s instructions—such as not staying on the directed flight path and/or entering a runway without clearance. Of the 1,471 unauthorized maneuvers from 1987 through 1995, 80 percent occurred in the air, and most of these (73 percent) occurred when pilots did not maintain their assigned altitude levels. The unauthorized pilot maneuvers on the ground were most often (69 percent) associated with pilots’ moving airplanes onto runways without authorization from the air traffic control tower. These types of incidents have the potential to cause accidents. For example, the December 1990 crash at the Detroit Metropolitan Airport occurred when an airplane taxied onto a runway being used for takeoff by another airplane and collided with that airplane. Twelve people died. The first plane had not gotten permission from the control tower to enter this runway, as it should have. Figure 3 shows the most frequently reported unauthorized pilot maneuvers in the air and on the ground.

Deficiencies in CRM Contributed to Accidents

In our analysis of accidents, we found deficiencies in the airline pilots’ use of CRM in nearly one-third of all accidents involving pilots’ performance. Moreover, we found CRM deficiencies in half of the serious accidents in which there was at least one fatality. About 46 percent of these CRM deficiencies involved a lack of coordination among members of the cockpit crew, as well as the captain’s failure to assign tasks to other crew members and to effectively supervise the crew. Generally, these CRM deficiencies illustrated the importance of effective communication.

For example, in the Charlotte, North Carolina, crash in July 1994, communication among crew members did not occur, according to NTSB’s accident investigation report. NTSB believes that the captain, who was not flying the aircraft at the time and could not see the ground because of poor visibility, became disoriented and commanded the first officer, “down, push it down,” even though they were encountering windshear, which is a sudden change in wind direction. The first officer did not question the order, as he should have, according to NTSB, because the windshear was creating an unstable situation; the plane could not recover from the sudden downward shift in direction caused by following the captain’s order. The plane crashed nose down into the ground, and 37 people died.

Similarly, in a June 1984 accident in Detroit, Michigan, a lack of communication between the crew and air traffic controllers during a landing in a severe thunderstorm contributed to the accident, according to the NTSB report. The crew did not request clarification about the weather conditions or change its course of action to take these conditions into account. The winds associated with the storm forced the plane down precipitously, causing an emergency landing without the landing gear’s being fully extended. The plane skidded off the runway, causing serious damage to the aircraft and an emergency evacuation of the passengers. NTSB reported that the lack of CRM practices was a probable cause of the accident.

The National Aeronautics and Space Administration reported similar results in its analysis of pilot reports submitted to its voluntary reporting system. Nearly half of the reports cited deficiencies in the pilots’ use of CRM principles; about 53 percent of the CRM deficiencies concerned coordination among members, assignment of tasks, and crew supervision.

FAA’s Guidance and Oversight Do Not Ensure Effectiveness of CRM Training

For AQP training, FAA has specified the process airlines need to follow to develop and implement a curriculum that integrates CRM concepts with technical flying skills, but FAA’s guidance for CRM training under part 121 does not have the same degree of specificity. As a result, inspectors overseeing training under part 121 do not have standards they can use to evaluate airlines’ CRM training curriculum and the delivery of that training. Generally, inspectors could not use the guidance provided under AQP to evaluate part 121 training for the CRM curriculum because the curricula developed under the two programs differ significantly. As a result, airlines continue to need specific guidance for CRM under part 121—both those airlines that have opted not to enter AQP as well those that will continue to train at least some of their crews under part 121 until they have fully implemented AQP, which could take up to 8 years.

FAA’s Guidance for CRM Training Is Detailed Under AQP but Not Under Part 121

Once an airline elects to participate in AQP, it must follow SFAR 58 (the AQP regulation) for developing a formal curriculum—including assessing the skills pilots need to safely operate the aircraft they fly, developing curriculum objectives for teaching those skills, having measurable criteria for evaluating whether the pilots have achieved those objectives, and developing materials to teach those objectives. FAA must approve this curriculum. Furthermore, AQP requires all airlines to train their pilots in simulators so that they gain experience with a number of emergency situations. Finally, airlines must submit data to FAA demonstrating that their crews have mastered the skills they need to fly for those airlines.

In developing its AQP curriculum, an airline is required to integrate CRM training into every aspect of its crews’ training. As a result, the pilots trained under AQP are assessed on CRM principles as well as on technical flying skills. For example, when a pilot changes the aircraft’s altitude—a technical flying skill—CRM principles dictate that this pilot inform the other pilot by verbally announcing the new altitude while continually pointing to the altitude indicator until the other pilot also points to the altitude indicator and repeats the new altitude. This procedure is used to ensure that neither pilot will fail to maintain the appropriate altitude.

In contrast, FAA’s requirements for CRM training under part 121 do not require airlines to develop a curriculum for CRM training with measurable criteria or to integrate that curriculum with other aspects of part 121 training. For the CRM curriculum under part 121, FAA provides suggested training topics but does not clearly lay out how the airlines are to introduce these topics into their training programs, according to airline officials and FAA inspectors. For example, FAA recommends that airlines train crews in “workload management and situational awareness.” For this training, FAA suggests such topics as “preparation/planning/vigilance” and “workload distribution/distraction avoidance.” However, for those airlines that choose to integrate these topics with technical flying skills, FAA does not explain how the airlines are to do so.

The lack of specificity in FAA’s guidance for the development of a CRM curriculum under part 121 contrasts with the detailed guidance FAA provides for the development of a curriculum on technical flying skills. For example, FAA’s guidance on how pilots are to respond to windshear under part 121 directs them in a number of technical flying skills, such as how to handle the rudder, but it is silent on how to employ CRM principles in this situation. In contrast, under AQP, FAA’s guidance instructs the airlines to specify not only the technical skills but also the CRM principles that must be applied in a windshear situation.

Because FAA’s guidance on CRM training under part 121 is less specific, airlines vary in how they deliver their CRM training. While all the airlines provide classroom training in CRM principles under part 121 training, they may not integrate this training with technical flying skills. For example, airlines may (1) train pilots in technical flying skills in flight simulators without integrating CRM principles or (2) integrate CRM principles with technical flying skills in flight simulators. Generally, we found that CRM training had been integrated with technical flight training to a higher degree at those airlines that were in later phases of AQP implementation.

FAA’s Oversight of CRM Training Is Adequate for AQP but Not for Part 121

In developing AQP, FAA incorporated procedures for evaluating CRM training and developed a process for ensuring that FAA inspectors would have the criteria they need to conduct the evaluations for pilots’ training on different types of aircraft. Specifically, AQP provides a systematic way of identifying the tasks and subtasks involved in a particular phase of flight. Therefore, an inspector observing the training program can determine whether CRM principles are being invoked in a given flight situation. For example, when a crew is preparing for landing, AQP specifies that the first officer, if unsure of the planned course of action in the event of a missed approach, is to ask the captain to clarify the plan so that both have a full understanding of the actions they will take. Similarly, if a flight has to be diverted from one airport to another, the captain is to direct the first officer to (1) get out the maps for the alternate airport, (2) notify the flight attendants, and (3) make the announcement to the passengers. This delegation of tasks allows the captain time to handle radio contact with the airline’s dispatchers and air traffic controllers, obtain weather updates at the alternate airport, and fly the plane.

In the early stages of AQP implementation, the AQP Branch is evaluating airlines’ training. FAA will transfer this responsibility to inspectors in the field as airlines fully implement AQP. Field inspectors will be trained in evaluating the CRM training as an integral part of their evaluation of AQP training. The inspectors at those airlines that had progressed beyond the initial phases of AQP noted that they had received AQP training at the airlines for which they were responsible. Moreover, all of the inspectors we spoke with maintained that while certain facets of AQP were fixed, some parts were still evolving. As a result of the program’s flexibility and evolution, the inspectors pointed out that it was not possible to structure a training program for them that could cover every aspect of AQP at every airline. Despite this fluidity, these inspectors said that the AQP Branch Office made sure that the program’s standards were maintained across airlines.

While the evaluation of the delivery of CRM training is incorporated into the oversight process for AQP training, it is not under traditional part 121 training. Moreover, FAA has not provided its inspectors with any specific guidance or training for evaluating airlines’ CRM training under part 121. Although FAA inspectors may obtain some CRM training from a 3-hour computerized interactive course, this lack of guidance for evaluating CRM training under part 121 is troublesome to the inspectors we spoke with because of what they view as an inherent conflict between performance expectations for individuals under part 121 and crew performance expectations articulated in CRM principles. Under part 121, pilots are to master technical flying skills and perform these skills without reliance on any other crew member. In contrast, CRM principles and training teach pilots how to use to maximum effect the abilities and experience of other crew members, as well as their own technical flying skills.

Without formal FAA instructions, inspectors have developed their own approaches to this evaluation. For example, one inspector said that he based his approval on his belief that the airline for which he was responsible “had a good safety record” and “would probably establish a good program.” Another inspector said that in approving any training program, he sought guidance first from any applicable federal aviation regulation; the Inspector’s Handbook, applicable advisory circulars; and, finally, any other FAA publication, such as the Introduction to CRM Training. However, this inspector added that these sources did not provide the criteria he needed to evaluate CRM training. As a result, he looked for behaviors such as crew members’ “working together” to resolve problems, “catching errors,” or “dealing with the consequences resulting from uncaught errors.”

These ad hoc approaches to evaluating the delivery of CRM training are not satisfactory to FAA officials at headquarters or to officials for at least one airline. FAA officials told us that the agency needed additional CRM training for its inspectors conducting reviews under part 121. In addition, officials at one airline told us that the lack of specific guidance and training for FAA inspectors responsible for evaluating CRM training under part 121 has hampered FAA’s ability to review CRM programs. Furthermore, the problems FAA inspectors face in evaluating CRM training under part 121 will continue indefinitely in the absence of clearer guidance from FAA for those airlines that have decided not to enter AQP and for those airlines in the program that have not fully implemented it.

Because AQP is implemented by the type of aircraft the crew flies, even the airlines that have been accepted for AQP will continue to provide some CRM training under part 121. For the eight airlines implementing AQP, we estimate that only about one-third of their crews have begun to receive AQP training. Therefore, most crews are still receiving traditional training under part 121, and some will continue do so for up to 8 years. As of September 1997, the airlines’ estimated dates for completing the transition to AQP training ranged between 2000 and 2005. (See table 1.)

Conclusions

For the flying public, safety is the paramount issue, and FAA and the airlines have worked to provide rigorous training programs for pilots. Crew resource management, which focuses on making the best use of all available experience and skills in the cockpit, is increasingly seen as an important component of safe flights. FAA recognized the importance of crew resource management by requiring all airlines to include training in these principles and by incorporating crew resource management into its Advanced Qualification Program.

Pilots’ performance is not the only factor in airline accidents, but it is an important one. We identified pilots’ performance as the cause of about one-third of all the accidents and nearly one-fifth of the incidents for the 10 major airlines from 1983 through 1995.

Training for safer performance by pilots that teaches crew resource management can occur under either the Advanced Qualification Program or part 121. However, while FAA’s guidance for the implementation of the Advanced Qualification Program specifies a process for curriculum development that integrates this training with training in technical flying skills, FAA’s guidance for curriculum development under part 121 is ambiguous and does not provide standards that inspectors can use to evaluate and approve airlines’ training in crew resource management. As a result, FAA cannot be assured that airlines are developing a curriculum for teaching crew resource management that will effectively teach pilots how to best use all the skills and experience available to them in the cockpit. Furthermore, without specificity in the development of training for crew resource management under part 121 and without any guidance on how to evaluate this training under part 121, FAA inspectors are relying on their own experience in observing pilots or even on the belief that the airline “would probably establish a good program.” These problems are especially troublesome because pilots who have not completed FAA-approved crew resource management training by March 1998 may not fly for airlines.

Recommendations

To help ensure that airlines appropriately train pilots in CRM principles under part 121 and that FAA inspectors are able to uniformly evaluate this CRM training, we recommend that the Secretary of Transportation direct the Administrator of FAA to develop a process that airlines must follow for creating a CRM curriculum, with measurable criteria, under part 121 as it has for the Advanced Qualification Program.

Agency Comments and Our Evaluation

We provided a draft of this report to FAA for review and comment. We met with the Deputy Associate Administrator for Regulation and Certification, the Deputy Director of Flight Standards Services, the Managers for the Air Carrier Training Branch and the Advanced Qualification Program, and other officials. FAA commended our review of CRM training at the nation’s airlines. FAA accepted the report’s recommendation in part. FAA agreed that it should ensure that pilots are appropriately trained and noted that CRM training can provide desirable consequences in aviation safety. It further agreed that uniform evaluation of CRM training using measurable criteria is a commendable objective. However, FAA stated that science has not yet developed valid, reliable criteria for measuring CRM performance. FAA also agreed that more can be done to develop a process that airlines and inspectors can follow to create a CRM curriculum. FAA indicated that better guidance would be provided in a number of ways, such as updating Advisory Circular 120-51, Crew Resource Management Training, and supplemental guidance for inspectors included in the inspectors’ handbook and holding regional meetings with CRM specialists from Flight Standards Services and other organizations.

We concur with FAA that CRM training for pilots could improve aviation safety. However, we believe that before the contribution of CRM training to aviation safety can be measured, it is necessary to determine the extent to which the delivery of CRM training for pilots has occurred. We further concur with FAA that more should be done to develop processes for airlines and inspectors to follow in creating a CRM curriculum. We believe that until FAA establishes a process for CRM curriculum development that includes an assessment of the extent to which pilots have mastered that curriculum, it will not be possible to measure CRM’s performance in contributing to aviation safety.

Scope and Methodology

To determine the extent to which inadequate performance by pilots was a problem for the 10 major U.S. airlines, we examined the types and frequency of safety-threatening events—incidents and accidents—from 1983 through 1995.

To determine the adequacy of FAA’ s guidance for and oversight of pilots’ training, we reviewed FAA’s role in the airlines’ implementation of CRM. We focused primarily on CRM training because FAA has described the failure to apply CRM principles as a more important contributing factor in accidents than technical flying skills. We also compared FAA’s rules and regulations and other guidance for CRM training with that provided for other training programs, as well as interviewed FAA and airline officials. A detailed discussion of our methodology is presented in appendix I. Related GAO products are listed at the end of this report.

Our work was performed from October 1996 through October 1997 in accordance with generally accepted government auditing standards.

As arranged with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after the date of this letter. At that time, we will provide copies of the report to the Secretary of Transportation, appropriate congressional committees, and other interested parties. We will also make copies available to others upon request.

Please contact me at (202) 512-3650 if you or your staff have any questions about this report. Major contributors to this report are listed in appendix II.

Methodology

To identify the types and frequencies of accidents and incidents—safety-threatening events—related to pilot performance, we reviewed accident and incident data, including pilot deviations, contained in the National Transportation Board’s (NTSB) and the Federal Aviation Administration’s (FAA) electronic databases. We obtained these data from FAA’s National Aviation Safety Data Analysis Center. We limited our review to the reported events in accident and incident data sources involving the 10 major U.S. passenger airlines from 1983 through 1995. We did not independently verify these data.

To facilitate the comparison of accidents with incidents in our analysis of the types and frequencies of safety-threatening events, we made two adjustments to the data. First, because of differences in the way information is recorded in these databases, we matched the similar categories contained in both databases and used these categories in our analysis. For example, both NTSB’s and FAA’s databases contain the category “on ground collision with object,” which means an airplane struck an object, such as a vehicle or structure, while moving on the ground. Second, because the occurrences of events in accidents closely conform to those in incidents, we used the events that occurred in each of the 169 accidents as our unit of analysis. In our analysis of crew resource management (CRM) deficiencies, we used the accident as the unit of analysis because NTSB’s findings of CRM deficiencies were by accident and not by the individual events that occurred within accidents.

To characterize the prevalence of pilot performance as a factor in safety-threatening events over time and between airlines, we examined FAA’s incident and pilot deviation databases. We used these two databases because they are the only such sources with adequate numbers of observations to make such comparisons.

To determine the extent to which the inadequate use of CRM by pilots contributed to accidents and incidents, we performed a content analysis of the textual information found in the factual reports, briefs, and final reports of the 169 accidents investigated by NTSB from 1983 through 1995. We then classified CRM deficiencies according to the classification framework presented at a National Aeronautics and Space Administration (NASA)/Ames workshop in 1980. This framework groups CRM issues into five broad clusters: (1) Resource management—the application of specialized cognitive skills to effectively and efficiently utilize available resources, such as the ability to plan, organize, and communicate. (2) Organization processes—crew members’ actions and behaviors in the context of their assigned duties and expected responsibilities. (3) Personal factors—the knowledge, skills, abilities, and limitations that individual crew members bring with them to the cockpit. (4) Material resources internal to the aircraft—the cockpit crew’s appropriate, effective, and efficient use of instructional items, such as checklists, and navigational charts and equipment, such as on-board weather radar, navigational controls, and engine fire extinguisher. (5) Resources external to the aircraft—those people (air traffic controllers), entities (airports), and circumstances (emerging poor weather) that may affect pilots’ plans, decisions, and actions.

Table I.1 shows the classification framework used to categorize CRM issues.

To verify the results of our content analysis, we requested a similar analysis by NASA’s Aviation Safety Reporting System (ASRS) staff of voluntarily submitted pilot reports contained in the ASRS database. According to the aviation experts we consulted, ASRS incident reports provide the best source of information on deficiencies in CRM. Furthermore, because ASRS staff are most familiar with the data and have expertise in analyzing this free-form data, we concluded that it was more appropriate for them to perform this analysis.

To evaluate the adequacy of FAA’s oversight of airline pilot training, we obtained FAA’s training policies, requirements, guidance, and handbooks relevant to CRM training. We discussed training programs, including CRM, and training procedures with appropriate FAA officials, including officials in the Office of System Safety, the Office of Regulation and Certification’s Flight Standards Services, the Advanced Qualification Program Branch, the Office of Accident Investigation, and the Human Factors Division. In addition, we discussed airline training evaluation and approval processes and obtained training documents from FAA inspectors responsible for monitoring airline training. Finally, we contacted safety directors and trainers at the major airlines and obtained documents on their policies, procedures, research, and training curricula.

We requested comments from recognized experts in the field of human factors in academia and the aviation industry, pilots, and government officials from FAA, NTSB, and NASA. We incorporated their comments where appropriate and made adjustments to our methodology as warranted.

Major Contributors to This Report

Resources, Community, and Economic Development Division, Washington, D.C.

Office of General Counsel

Related GAO Products

Aviation Safety: New Airlines Illustrate Long-Standing Problems in FAA’s Inspection Program (GAO/RCED-97-2, Oct. 17, 1996).

Human Factors: Status of Efforts to Integrate Research on Human Factors Into FAA’s Activities (GAO/RCED-96-151, June 27, 1996).

Military Aircraft Safety: Significant Improvements Since 1975 (GAO/NSIAD-96-69BR, Feb. 1, 1996).

Aviation Safety: Data Problems Threaten FAA Strides on Safety Analysis System (GAO/AIMD-95-27, Feb. 8, 1995).

Aviation Safety: Unresolved Issues Involving U.S.-Registered Aircraft (GAO/RCED-93-135, June 18, 1993).

Aviation Safety: Changes Needed in FAA’s Service Difficulty Reporting Program (GAO/RCED-91-24, Mar. 21, 1991).

The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.