Aviation Safety: FAA's Safety Oversight System Is Effective but Could Benefit from Better Evaluation of Its Programs' Performance
Released: 2005-11-17T00:00:00

Letter

Background

FAA’s safety oversight system is made up of a number of programs for airlines and other entities. Safety oversight programs for airlines provide for their initial certification, periodic surveillance, and inspection. Since 1985, FAA has used National Work Program Guidelines (NPG), its traditional inspection program for airlines, as a primary means of ensuring that airlines comply with safety regulations. In NPG, an FAA committee of program managers identifies an annual minimum set of required inspections that are to be undertaken to ensure that airlines are in compliance with their operating certificates. In 1998, the agency implemented the Air Transportation Oversight System (ATOS), which currently oversees the nation’s largest 15 commercial airlines and cargo carriers, with the goal of eventually including all commercial passenger and cargo airlines in it. ATOS emphasizes a system safety approach that extends beyond periodically checking airlines for compliance with regulations to the use of technical and managerial skills to identify, analyze, and control hazards and risks. For example, under ATOS, inspectors develop surveillance plans for each airline, based on data analysis and assessment of risks, and adjust the plans periodically based on inspection results. However, the agency has been delayed in placing a significant number of other passenger airlines in ATOS, resulting in 99 passenger airlines, which we refer to as non-legacy airlines, continuing to be overseen through NPG, a process that is not risk-based or system safety oriented. In 2002, FAA added the Surveillance and Evaluation Program (SEP) to the NPG inspection program to incorporate principles of ATOS into its oversight of non-legacy passenger airlines. The two programs are used together to establish the number of annual inspections for non-legacy airlines. (Appendix 1 describes each inspection program.) Figure 1 illustrates some typical activities covered during inspections.

FAA’s safety oversight programs for other aspects of the aviation industry—including manufacturers of aircraft and aircraft parts, repair stations, flight schools, aviation maintenance technician schools, pilots, and mechanics—involve certification, surveillance, and inspection by FAA’s safety inspectors, engineers, flight surgeons, and designated representatives. FAA authorizes about 13,400 private individuals and about 180 organizations (called “designees”) to act as its representatives to conduct many safety certification activities, such as administering flight tests to pilots, inspecting repair work by maintenance facilities, conducting medical examinations of pilots, and approving designs for aircraft parts. These designees are grouped into 18 different programs and are overseen by three FAA offices—Flight Standards Service, Aerospace Medicine, and Aircraft Certification Service—all of which are under the Office of Aviation Safety (see fig. 2).

Since 1990, FAA has emphasized gaining compliance from the aviation industry through cooperative means by establishing industry partnership programs with the aviation community that allow participants, such as airlines and pilots, to self-report violations of safety regulations and help identify safety deficiencies, and potentially mitigate or avoid fines or other legal action. For example, the Voluntary Disclosure Program encourages the self-reporting of manufacturing problems and safety incidents by participants that can include air carriers and repair stations. Appendix II describes the industry partnership programs.

When violations of statutory and regulatory requirements are identified through inspections, through the partnership programs in certain cases, or through other methods, FAA has a variety of enforcement tools that it may use to respond to them, including administrative actions (such as issuing a warning notice or a letter of correction that includes the corrective actions the violator will take) and legal sanctions (such as levying a fine or suspending or revoking a pilot or other FAA-issued certificate).

FAA’s Safety Oversight System Focuses on Risk Identification and Mitigation Through System Safety, Leveraging of Resources, and Enforcement of Safety Regulations, but Benefits Are Not Being Fully Realized

In recent reports, we found that FAA’s safety oversight system has programs that focus on risk identification and mitigation through a system safety approach, the leveraging of resources, and enforcement of safety regulations, but that the benefits of these programs are not being fully realized. In our recent report on FAA’s oversight of non-legacy airlines, we found that the focus on risk identification through the addition of SEP has many strengths and allows for enhancing the efficiency of FAA’s oversight activities. Rather than relying on NPG’s customary method of conducting a set number of inspections of an airline’s operations, SEP emphasizes a system safety approach of using risk analysis techniques. SEP allows for the efficient use of inspection staff and resources by prioritizing workload based on areas of highest risk, and it includes a requirement that inspectors verify that corrective actions have occurred. For example, FAA has developed risk assessment worksheets for SEP that are aligned with key airline systems that guide inspectors through identifying and prioritizing risks. The worksheets guide inspectors to organize the results of their previous inspections and surveillance into a number of areas such as flight operations and personnel training in order to identify specific risks in each area and target the office’s resources to mitigating those risks. The development of a system safety approach addresses a long- standing concern by us that FAA did not have a methodology for assessing airline safety risks so that it could target limited inspection resources to high-risk conditions. Another strength of SEP, consistent with findings in our past reports, is that SEP relies on teams of inspectors, which are generally more effective than individual inspectors in their ability to collectively identify concerns.

However, the benefits of FAA’s system safety approach for the inspection of non-legacy airlines could be enhanced by a more complete implementation of SEP and addressing other challenges. The inspection workload for non-legacy airlines is still heavily oriented to the NPG’s non- risk based activities. For example, as shown in table 1, from fiscal years 2002 through 2004, 77 percent of inspection activities required for the top 25 non-legacy airlines in terms of the number of enplanements were identified through NPG, and the remaining percentage of inspection activities were identified based on risk through SEP. Although inspectors can replace NPG-identified activities with SEP-identified activities that they deem constitute a greater safety risk, we found that FAA inspectors interpret agency emphasis on NPG as discouraging this practice. In order to ensure that all inspectors who oversee non-legacy airlines have a complete and timely understanding of the agency’s policies relating to the inspection process, we recommended in September 2005 that FAA improve communication with and training of inspectors in this area.

Another way that FAA attempts to enhance the efficiency of its oversight activities is through its designee programs. We reported that FAA maximizes its resources by allowing designees to perform about 90 percent of certification-related activities, thus allowing FAA to better concentrate its limited staff resources on the most safety-critical functions. For example, while designees conduct routine certification functions, such as approvals of aircraft technologies that the agency and designees have had previous experience with, FAA focuses on new and complex aircraft designs or design changes. In addition, the use of designees expands FAA’s access to technical expertise within the aviation community. For the aviation industry, the designee programs enable individuals and organizations to obtain required FAA certifications—such as approvals of the design, production, and airworthiness of aircraft—in a timely manner, thus reducing delays and costs to the industry that might result from scheduling direct reviews by FAA staff. For example, officials from an aircraft manufacturer told us that the use of designees has added significantly to the company’s ability to enhance and improve daily operations by decreasing certification delivery time and increasing the flexibility and utilization of company resources. In addition, designees are convenient to the aviation industry due to their wide dispersal throughout the United States.

However, concerns about the consistency and adequacy of designee oversight that FAA field offices provide have been raised by experts and other individuals we interviewed. For example, designees and industry officials that we spoke with indicated that FAA’s level of oversight and interpretation of rules differ among regions and among offices within a region, which limits FAA’s assurance that designees’ work is performed uniformly in accordance with FAA’s standards and policy. Experts also ranked this issue as a top weakness. Table 2 shows the top five weaknesses identified by our experts. Experts also made a number of suggestions to strengthen the designee program, including clearly defining and following agency criteria for selecting designees and increasing penalties for designees found to violate standards or who do not exercise proper judgment. To improve management control of the designee programs, and thus increase assurance that designees meet FAA’s performance standards, we recommended that FAA develop mechanisms to improve the compliance of FAA program and field offices with existing policies and incorporate, as appropriate, suggestions from our expert panel. In response to our recommendations, FAA is planning, among other things, to form a team to identify and share best practices for overseeing designee programs.

FAA also leverages its resources through its industry partnership programs. These partnership programs are designed to assist the agency in receiving safety information, including reports of safety violations. According to FAA officials, the Aviation Safety Action Program, Aviation Safety Reporting Program, and Voluntary Disclosure Reporting Program augment FAA’s enforcement activities and allow FAA to be aware of many more safety incidents than are discovered during inspections and surveillance. In addition, the Flight Operational Quality Assurance Program provides safety information in the form of recorded flight data from participating airlines. FAA has established some management controls over its partnership programs, such as procedures to track actions taken to correct safety incidents reported under the programs, but the agency lacks management controls to measure and evaluate the performance of these programs, an issue that we will discuss later in the testimony.

FAA’s enforcement process, which is intended to ensure industry compliance with safety regulations, is another important element of its safety oversight system. FAA’s policy for assessing legal sanctions against entities or individuals that do not comply with aviation safety regulations is intended to deter future violations. FAA has established some management controls over its enforcement efforts, with procedures that provide guidance on identifying regulated entities and individuals that are subject to inspections or surveillance actions, determining workload priorities on the basis of the timing and type of inspection to be performed, detecting violations of safety regulations, tracking the actions that are taken by the entities and individuals to correct the violations and achieve compliance with regulations, and imposing punitive sanctions or remedial conditions on the violators. These procedures provide FAA inspectors, managers, and attorneys with a process to handle violations of safety regulations that are found during routine inspections.

However, we found that the effect of FAA’s legal sanctions on deterrence is unclear, and that recommendations for sanctions are sometimes changed on the basis of factors that are not associated with the merits of the case. We found that from fiscal years 1993 through 2003, attorneys in FAA’s Office of the Chief Counsel authorized a 52 percent reduction in the civil monetary penalties assessed from a total of $334 million to $162 million. FAA officials told us that the agency sometimes reduces sanctions in order to prioritize attorneys’ caseloads by closing the cases more quickly through negotiating a lower fine. Economic literature on deterrence suggests that although negative sanctions (such as fines and certificate suspensions) can deter violations, if the violator expects sanctions to be reduced, he or she may have less incentive to comply with regulations. In effect, the goal of preventing future violations is weakened when the penalties for present violations are lowered for reasons not related to the merits of the case. In addition, FAA lacks management controls to measure and evaluate its enforcement process, which we discuss later in this testimony.

FAA Has Made Training an Integral Part of Its Safety Oversight System but Several Actions Could Improve Results

FAA’s use of a risk-based system safety approach to inspections requires inspectors to apply data analysis and auditing skills to identify, analyze, assess, and control potential hazards and risks. Therefore, it is important that inspectors are well-trained in this approach and have sufficient knowledge of increasingly complex aircraft, aircraft parts, and systems to effectively identify safety risks. It is also important that FAA’s large cadre of designees is well-trained in federal aviation regulations and FAA policies. FAA has made training an integral part of its safety inspection system and has established mandatory training requirements for its workforce as well as designees. FAA provides inspectors with extensive training in federal aviation regulations; inspection and investigative techniques; and technical skills, such as flight training for operations inspectors. The agency provides its designees with an initial indoctrination that covers federal regulations and agency policies, and refresher training every 2 to 3 years.

We have reported that FAA has generally followed effective management practices for planning, developing, delivering, and assessing the impact of its technical training for safety inspectors, although some practices have yet to be fully implemented. In its planning activities for training, FAA has linked technical training efforts to its goal of safer air travel and has identified technical proficiencies needed to improve safety inspectors’ performance in meeting this goal. For example, FAA’s Offices of Flight Standards and Aircraft Certification have identified gaps in several of the competencies required to conduct system safety inspections, including risk assessment, data analysis, systems thinking, and designee oversight. According to FAA, it is working to correct these gaps. We have also identified gaps in the training provided to inspectors in the Office of Flight Standards who oversee non-legacy airlines, and have recommended that FAA improve inspectors’ training in areas such as system safety and risk management to ensure that these inspectors have a complete and timely understanding of FAA’s inspection policies. We have identified similar competency gaps related to designee oversight. For example, FAA does not require refresher training concerning designee oversight, which increases the risk that staff do not retain the information, skills, and competencies required to perform their oversight responsibilities. We recommended that FAA provide additional training for staff who directly oversee designees. We did not identify any specific gaps in the competencies of designees. In prioritizing funding for course development activities, FAA does not explicitly consider which projects are most critical. Figure 3 describes the extent to which FAA follows effective management practices in planning training.

In developing its training curriculum for inspectors, FAA also for the most part follows effective management practices, such as developing courses that support changes in inspection procedures resulting from regulatory changes or agency initiatives. On the other hand, FAA develops technical courses on an ad hoc basis rather than as part of an overall curriculum for each inspector specialty—such as air carrier operations, maintenance, and cabin safety—because the agency has not systematically identified the technical skills and competencies each type of inspector needs to effectively perform inspections. Figure 4 describes the extent to which FAA follows effective management practices in developing training.

In delivering training, FAA has also generally followed effective management practices. (See fig. 5.) For example, FAA has established clear accountability for ensuring that inspectors have access to technical training, developed a way for inspectors to choose courses that meet job needs and further professional development, and offers a wide array of technical and other courses. However, both FAA and its inspectors recognize the need for more timely selection of inspectors for technical training. In addition, FAA acknowledges the need to increase communication between inspectors and management with respect to the training program, especially to ensure that inspectors have bought into the system safety approach to inspections.

FAA offers numerous technical courses from which inspectors can select to meet job needs. However, from our survey of FAA’s inspectors, we estimate that only about half think that they have the technical knowledge needed for their jobs. FAA officials told us that inspectors’ negative views stem from their wanting to acquire proficiencies that are not as crucial in a system safety environment. We also found a disparity between inspectors and FAA concerning the receipt of requested training. We estimated that 28 percent of inspectors believe that they get the technical training that they request. However, FAA’s records show that FAA approves about 90 percent of these requests, and inspectors are making good progress in receiving training. Over half of the inspectors have completed at least 75 percent of technical training that FAA considers essential. FAA officials told us that inspectors’ negative views on their technical knowledge and the training they have received stem from their not accepting FAA’s move to a system safety approach. That is, the inspectors are concerned about acquiring individual technical proficiency that is not as crucial in a system safety environment. Given that it has not completed assessing whether training for each inspector specialty meets performance requirements, FAA is not in a position to make definitive conclusions concerning the adequacy of inspector technical training.

FAA also generally followed effective management practices in evaluating training. The agency requires that each training course receive a systematic evaluation every 3 years to determine if the course is up to date and relevant to inspectors’ jobs, although training officials noted that many courses have yet to undergo such an evaluation. However, FAA collects limited information on the effectiveness of training, and its evaluations have not measured the impact of training on FAA’s mission goals, such as reducing accidents. Training experts acknowledge that isolating performance improvements resulting from training programs is difficult for any organization. (See fig. 6.)

While FAA follows many effective management practices in its training program, the agency also recognizes the need for improvements, including (1) systematically assessing inspectors’ needs for technical and other training, (2) better timing of technical training so that inspectors receive it when it is needed to perform their jobs, and (3) better linking the training provided to achieving agency goals of improving aviation safety. FAA has begun to act in these areas, and we believe that if effectively implemented, the actions should improve the delivery of training and ultimately improve aviation safety. Therefore, it is important for FAA to follow through with its efforts. As a result, we recommended in September 2005, among other things, that in order to ensure that inspector technical training needs are identified and met in a timely manner, FAA systematically assess inspectors’ technical training needs, better align the timeliness of training to when inspectors need the training to do their jobs, and gain inspectors’ acceptance for changes made or planned to their training.

It is important that both FAA’s inspection workforce and FAA-certified aviation mechanics are knowledgeable about increasingly complex aircraft, aircraft parts, and systems. While we did not attempt to assess the technical proficiency that FAA’s workforce requires and will require in the near future, FAA officials said that inspectors do not need a substantial amount of technical training courses because inspectors are hired with a high degree of technical knowledge of aircraft and aircraft systems. They further indicated that inspectors can sufficiently keep abreast of many of the changes in aviation technology through FAA and industry training courses and on-the-job training. However, in its certification program for aviation mechanics, we found that FAA standards for minimum requirements for aviation courses at FAA-approved aviation maintenance technician schools and its requirements for FAA-issued mechanics certificates do not keep abreast with the latest technologies. In 2003, we reported that those standards had not been updated in more than 50 years. We recommended that FAA review the curriculum and certification requirements and update both. FAA plans to make changes in the curriculum for FAA approved aviation maintenance technicians that reflect up-to-date aviation technologies and finalize and distribute a revised Advisory Circular in March 2006 that describes the curriculum changes. FAA then plans to allow the aviation industry time to implement the recommended curriculum changes before changing the requirements for FAA-issued mechanics certificates.

FAA Has Evaluated Some Safety Programs, but the Lack of Evaluative Systems and Nationwide Data Impedes FAA’s Ability to Continuously Monitor Its Safety Programs

It is important for FAA to have effective evaluative processes and accurate nationwide data on its numerous safety oversight programs so that program managers and other officials have assurance that the safety programs are having their intended effect. Such processes and data are especially important because FAA’s workforce is so dispersed worldwide—with thousands of staff working out of more than 100 local offices—and because FAA’s use of a risk-based system safety approach represents a cultural shift from its traditional inspection program. Evaluation is important to understanding if the cultural shift has effectively occurred. Our most recent work has shown the lack of such processes and limitations with data for FAA’s inspection programs for non-legacy airlines, designee programs, industry partnership programs, and enforcement program. In response to recommendations that we have made regarding these programs, some improvements are being made. On the positive side, as we mentioned earlier, our most recent work found that FAA generally follows effective management practices in evaluating individual technical training courses.

FAA has not evaluated its inspection oversight programs for non-legacy airlines—which include SEP and NPG—to determine how the programs contribute to the agency’s mission and overall safety goals, and its nationwide inspection database lacks important information that could help it perform such evaluations—such as whether risks identified through SEP have been mitigated. In addition, the agency does not have a process to examine the nationwide implications of or trends in the risks that inspectors have identified through their risk assessments—information it would need to proactively determine risk trends at the national level on a continuous basis. FAA’s evaluation office instead conducts analyses of the types of inspections generated under SEP by airline and FAA region, according to FAA. We recommended that FAA develop a continuous evaluative process for activities under SEP and link SEP to the performance-related goals and measures developed by the agency, track performance toward these goals, and determine appropriate program changes. FAA is considering our recommendation, but its plan to place the remaining non-legacy airlines in the ATOS program by the end of fiscal year 2007 might make this recommendation unnecessary, according to the agency. Since FAA’s past efforts to move airlines to ATOS have experienced delays, we believe that this recommendation is still valid.

We also found that FAA lacked requirements or criteria for periodically evaluating its designee programs. In 2004, we reported that the agency had evaluated 6 of its 18 designee programs over the previous 7 years and had plans to evaluate 2 more, although it had no plans to evaluate the remaining 10 programs because of limited resources. FAA conducted these evaluations on an ad hoc basis usually at the request of headquarters directors or regional office managers. In addition, we found that FAA’s oversight of designees is hampered, in part, by the limited information on designees’ performance contained in the various designee databases. These databases contain descriptive information on designees, such as their types of designations and status (i.e., active or terminated). More complete information would allow the agency to gain a comprehensive picture of whether staff are carrying out their responsibilities to oversee designees. To improve management control of the designee programs, and thus increase assurance that designees meet the agency’s performance standards, we recommended that FAA establish a process to evaluate all designee programs and strengthen the effectiveness of its designee databases by improving the consistency and completeness of information in them. To address our recommendations, FAA expects to develop a plan to evaluate all designee programs on a recurring basis and intends to establish a team that will examine ways to improve automated information related to designees.

In addition, we found that FAA does not evaluate the effects of its industry partnership and enforcement programs to determine if stated program goals, such as deterrence of future violations, are being achieved. For example, little is known about nationwide trends in the types of violations reported under the partnership programs or whether systemic, nationwide causes of those violations are identified and addressed. Furthermore, FAA’s enforcement policy calls for inspectors and legal counsel staff to recommend or assess enforcement sanctions that would potentially deter future violations. However, without an evaluative process, it is not known whether the agency’s practice of generally closing cases with administrative actions rather than legal sanctions and at times reducing the amount of the fines, as mentioned earlier in this testimony, may weaken any deterrent effect that would be expected from sanctions.

FAA’s ability to evaluate the impact of its enforcement efforts is also hindered by the lack of useful nationwide data. FAA inspection offices maintain independent, site-specific databases because they do not find the nationwide enforcement database—the Enforcement Information System (EIS)—as useful as it could be because of missing or incomplete historical information about enforcement cases. As a result of incomplete data on individual cases, FAA inspectors lack the complete compliance history of violators when assessing sanctions. We recommended that FAA develop evaluative processes for its enforcement activities and partnership programs and use them to create performance goals, track performance towards those goals, and determine appropriate program changes. We also recommended that FAA take steps to improve the usefulness of the EIS database by enhancing the completeness of enforcement information. FAA expects to address some of these issues as it revises its enforcement policy, which is expected to be issued later in fiscal year 2006. In addition, FAA has established a database workgroup that is developing long- and short- term solutions to address the problems with EIS.

Recommendations We Have Made to Improve FAA’s Safety Oversight System

In order to help FAA fully realize the benefits from its safety oversight system, we have made a number of recommendations to address weaknesses that we identified in our reviews. These recommendations have not been fully implemented, although in some cases FAA has taken steps towards addressing them. Evaluative processes and relevant data are particularly important as FAA works to change its culture by incorporating a system safety approach into its oversight, and we have recommended that FAA develop continuous evaluative processes for its oversight programs for non-legacy airlines, its designee programs, and its industry partnership and enforcement programs, and systematically assess inspectors’ technical training needs. In addition, FAA’s nationwide databases are in need of improvements in their comprehensiveness and ease of use. Without comprehensive nationwide data, FAA does not have the information needed to evaluate its safety programs and have assurance that they are having the intended results. We have recommended that FAA improve the completeness of its designee and enforcement databases. Continuous improvements in these areas are critical to FAA’s ability to have a robust “early warning system” and maintain one of the safest aviation systems in the world.

Contacts and Acknowledgments

For further information on this testimony, please contact Dr. Gerald Dillingham at (202) 512-2834 or by email at dillinghamg@gao.gov. Individuals making key contributions to this testimony include Brad Dubbs, Phillis Riley, Teresa Spisak, and Alwynne Wilbur.

Appendix I: Description of FAA’s Inspection Programs

Table 3 describes the Federal Aviation Administration’s (FAA) three inspection processes for overseeing airlines: Air Transportation Oversight System (ATOS), National Work Program Guidelines (NPG), and Surveillance and Evaluation Program (SEP). Many of the elements of ATOS, such as the use of data to identify risks and the development of surveillance plans by inspectors, are incorporated in the SEP process. The NPG process, in contrast, is not focused on the use of data and relies on an established set of inspections that are not risk based.

Appendix II: Description of FAA’s Partnership Programs

Aviation Safety Action Program (ASAP)

Participation: Participants include employees of air carriers and repair stations that have entered into a memorandum of understanding with the Federal Aviation Administration (FAA). The memoranda can cover employee groups, such as pilots, maintenance employees, dispatchers, or flight attendants. Each employee group is covered by a separate memorandum of understanding. As of June 2004, FAA had accepted 54 memoranda of understanding and received over 80,000 ASAP reports, which may or may not include safety violations, according to FAA officials.

Purpose: ASAP seeks to improve aviation safety through the voluntary self-reporting of safety incidents under the procedures set forth in the memorandum of understanding. Under the program, FAA does not take enforcement action against employees who voluntarily self-reported safety violations for reports that are sole-source (the report is the only way FAA would have learned about the incident) and will pursue administrative action only for reports that are not sole-source. Incidents that involve alcohol, drugs, criminal activity, or an intentional disregard for safety are not eligible for self-reporting under ASAP.

Process: Each memorandum of understanding is a voluntary partnership between FAA, the airline, and an employee group. Although employee groups are not always included, FAA encourages their participation. The memorandum of understanding ensures that employees who voluntarily disclose FAA safety violations in accordance with the procedures and guidelines of ASAP will receive administrative action or no action in lieu of legal enforcement action.

Once a memorandum of understanding is approved, employees can begin reporting violations that fall under the agreement. When a violation occurs, an employee notifies the Event Review Committee, which includes representatives from FAA and the airline or the repair station and generally includes the appropriate employee association. The committee must be notified in writing within the time limit specified in the memorandum of understanding. The committee then determines whether to accept the report under the ASAP program. If the report is accepted (it meets the acceptance criteria in the memorandum and does not involve criminal activity, substance abuse, controlled substances, or alcohol), then the committee determines the action to take. That action may include remedial training or administrative action, but it will not include a legal sanction.

Results: FAA does not know the overall program results because it does not have a national, systematic process in place to evaluate the overall success of ASAP. However, FAA cites examples that describe ASAP’s contribution to enhanced aviation safety. These examples include identifying deficiencies in aircraft operations manuals, airport equipment, and runways. In July 2003, FAA’s Compliance and Enforcement Review recommended that FAA evaluate the use and effectiveness of this program.

Aviation Safety Reporting Program (ASRP)

Participation: Participants are all users of the national airspace system, including air traffic controllers and employees of air carriers and repair stations.

Purpose: The program is designed to improve aviation safety by offering limited immunity for individuals who voluntarily report safety incidents. ASRP was founded after TWA Flight 514 crashed on approach to landing in December 1974 after the crew misinterpreted information on the approach chart. This accident occurred only 6 weeks after another plane experienced the same error.

Process: The National Aeronautics and Space Administration (NASA) administers this program. When a safety incident occurs, a person may submit a form and incident report to NASA. There are four types of forms that can be submitted to NASA: (1) Air Traffic Control, (2) General Reports (includes Pilots), (3) Flight Attendants, and (4) Maintenance Personnel.

At least two aviation safety analysts read these forms and the incident reports that accompany them. The analysts at NASA screen the incident reports for urgent safety issues, which will be marked for immediate action to the appropriate FAA office or aviation authority. NASA analysts also edit the report’s narrative to eliminate any identifying information. In addition, each report has a tear-off portion, which is separated and returned to the individual who reported the incident as a receipt of the incident report’s acceptance into the ASRP. When a safety violation that has been previously reported under ASRP comes to the attention of FAA, the agency issues a legal sanction, which is then waived. Reports that would not be eligible to have a legal sanction waived include deliberate violations, violations involving a criminal offense, or accident; reports filed by participants who have committed a violation of federal aviation regulations or law within the last 5 years and reports filed later than 10 days following an incident.

Results: While FAA and NASA do not know the overall program results because they do not have a formal national evaluation program to measure the overall effectiveness of the program, the agencies widely disseminate information generated from the program to aircraft manufacturers and others. ASRP reports are compiled into a database known as the Aviation Safety Reporting System. When a potentially hazardous condition is reported, such as a defect in a navigational aid or a confusing procedure, NASA will send a safety alert to aircraft manufacturers, the FAA, airport representatives, and other aviation groups. The database is used for a monthly safety bulletin that includes excerpts from incident reports with supporting commentary by FAA safety experts. NASA officials estimate that the bulletin is read by over 150,000 people. In addition, individuals and organizations can request a search of the database for information on particular aircraft aviation safety subjects, including human performance errors and safety deficiencies. Further, NASA has used the database to analyze operational safety issues, such as general aviation incidents, pilot and controller communications, and runway incursions.

Flight Operational Quality Assurance (FOQA)

Participation: Participants include air carriers that equip their airplanes to record flight data. As of March 2004, 13 airlines had FAA-approved FOQA programs, and approximately 1,400 airplanes were equipped for the program.

Purpose: FOQA is designed to enhance aviation safety through the analysis of digital flight data generated during routine flights.

Process: Air carriers that participate in the program equip their aircraft with special acquisition devices or use the airplanes’ flight data recorders to collect data and determine if the aircraft are deviating from standard procedures. These data include engine temperatures, descent rate, and deviations from the flight path. When the aircraft lands, data are transmitted from the aircraft to the airline’s FOQA station, where they are analyzed for flight trends and possible safety problems.

Once the data are transmitted to the FOQA ground station, the data are extracted and analyzed by software programs. The FOQA data are combined with data from maintenance databases, weather conditions, and other safety reporting systems, such as ASAP, in order to identify trends in flight operations. The analysis typically focuses on events that fall outside normal boundaries specified by the manufacturer’s operational limitations and the air carrier’s operational standards.

FOQA data are collected and analyzed by individual air carriers. The data on safety trends are made available to FAA in an aggregated form with no identification of individual carriers. According to FAA officials, air carriers do not want to release this data to any outside party (including FAA) because of concerns that the data could then be publicly released. Air carriers pay for the special flight data recorders that can record FOQA data, which cost approximately $20,000 each. Although this can be an expensive investment for some air carriers, most newer aircraft models come with the data recorder built into the airplane. The International Civil Aviation Organization (ICAO) has recommended that airlines from member countries implement a FOQA program. FAA has notified ICAO that the program will remain voluntary in the United States.

Results: Although FAA has no formal national evaluation program to measure the overall results or effectiveness of FOQA programs, FAA cites examples that describe FOQA’s contribution to enhanced aviation safety. For example, one FOQA program highlighted a high rate of descent when airplanes land at a particular airport. On the basis of the information provided from FOQA, air traffic controllers at the airport were able to develop alternative approach procedures to decrease the rate of descent.

Voluntary Disclosure Reporting Program (VDRP)

Participation: Participants include air carriers, repair stations, and production approval holders.

Purpose: FAA initiated the program to promote aviation safety by encouraging the voluntary self-reporting of manufacturing, and quality control problems and safety incidents involving FAA requirements for maintenance, flight operations, drug and alcohol prevention programs, and security functions.

Process: Upon discovering a safety violation, participants can voluntarily disclose the violation to FAA within 24 hours. The initial notification should include a description of the violation, how and when the violation was discovered, and the corrective steps necessary to prevent repeat violations. Within 10 days of filing the initial notification to FAA, the entity is required to provide a written report that cites the regulations violated, describes how the violation was detected, provides an explanation of how the violation was inadvertent, and provides a description of the proposed comprehensive fix. FAA may pursue legal action if the participant discloses violations during, or in anticipation of, an FAA inspection.

The violation must be reported immediately after being detected, must be inadvertent, must not indicate that a certificate holder is unqualified, and must include the immediate steps that were taken to terminate the apparent violation. If these conditions are met, and the FAA inspector has approved the comprehensive fix, then the FAA inspector will prepare a letter of correction and the case is considered closed with the possibility of being reopened if the comprehensive fix is not completed.

Results: FAA does not know the overall program results because it does not have a process to measure the overall effectiveness of the program nationwide. A 2003 internal FAA report recommended that the agency evaluate the use and effectiveness of this program.

Selected GAO Reports on Aviation Safety

Aviation Safety: System Safety Approach Needs Further Integration into FAA’s Oversight of Airlines. GAO-05-726. Washington, D.C.: September 28, 2005.

Aviation Safety: FAA Management Practices for Technical Training Mostly Effective; Further Actions Could Enhance Results. GAO-05-728. Washington, D.C.: September 7, 2005.

Aviation Safety: Oversight of Foreign Code-Share Safety Program Should Be Strengthened. GAO-05-930. Washington, D.C.: August 5, 2005.

Aviation Safety: FAA Needs to Strengthen the Management of Its Designee Programs. GAO-05-40. Washington, D.C.: October 8, 2004.

Aviation Safety: Better Management Controls are Needed to Improve FAA’s Safety Enforcement and Compliance Efforts. GAO-04-646. Washington, D.C.: July 6, 2004.

Aviation Safety: Information on FAA’s Data on Operational Errors at Air Traffic Control Towers. GAO-03-1175R. Washington, D.C.: September 23, 2003.

Aviation Safety: FAA Needs to Update the Curriculum and Certification Requirements for Aviation Mechanics. GAO-03-317. Washington, D.C.: March 6, 2003.

Aviation Safety: FAA and DOD Response to Similar Safety Concerns. GAO-02-77. Washington. D.C.: January 22, 2002.

Aviation Safety: Safer Skies Initiative Has Taken Initial Steps to Reduce Accident Rates by 2007. GAO/RCED-00-111. Washington, D.C.: June 30, 2000.

Aviation Safety: FAA’s New Inspection System Offers Promise, but Problems Need to Be Addressed. GAO/RCED-99-183. Washington, D.C.: June 28, 1999.

Aviation Safety: Weaknesses in Inspection and Enforcement Limit FAA in Identifying and Responding to Risks. GAO/RCED-98-6. Washington, D.C.: February 27, 1998.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.