Aviation Safety: FAA's Safety Oversight System Is Effective but Could Benefit from Better Evaluation of Its Programs' Performance
Published: 2005-11-17T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The U.S. commercial aviation industry has an extraordinary safety record. However, when passenger airlines have accidents or serious incidents, regardless of their rarity, the consequences can be tragic. The Federal Aviation Administration (FAA) works to maintain a high level of safety through an effective safety oversight system. Keys to this system are to: (1) establish programs that focus resources on areas of highest safety risk and on mitigating risks; (2) provide training and communication to ensure that inspectors can consistently carry out the agency's oversight programs; and (3) have processes and data to continuously monitor, evaluate, and improve the numerous oversight programs that make up the safety oversight system. This statement focuses on these three key areas and is based on recent GAO reports on FAA's inspection oversight programs, industry partnership programs, enforcement program, and training program.
section_title: What GAO Found
paragraphs: FAA's safety oversight system includes programs that focus on risk identification and mitigation through a risk-based system safety approach, leveraging of resources through designee and partnership relationships, and enforcement of safety regulations, but the benefits of these programs are not being fully realized. For example, FAA's system safety approach includes the addition of a program that emphasizes risk identification to its traditional inspection program for overseeing some airlines, which is not based on risk. However, it is likely that the benefits of this approach could be enhanced if the inspection workload was not as heavily oriented to the traditional inspection program's non-risk based activities. FAA leverages its resources through its designee programs, in which designated individuals and organizations perform about 90 percent of certification-related activities, and through its industry partnership programs, which are designed to assist the agency in receiving safety information. An outgrowth of FAA's inspection process is its enforcement program, which is intended to ensure industry compliance with safety regulations. However, GAO has expressed concerns that this program may not be as effective as it could be in deterring violations. FAA has made training an integral part of its safety oversight system, but several actions could improve the results of its training efforts, including ensuring that inspectors are well-trained in FAA's system safety approach and have sufficient knowledge of increasingly complex aircraft and systems to effectively identify safety risks. FAA has established mandatory training requirements for its workforce and designees. We have reported that FAA has generally followed effective management practices for planning, developing, delivering, and assessing the impact of its technical training for safety inspectors. GAO has found inadequate evaluative processes and limitations with data for FAA's inspection programs, designee programs, industry partnership programs, and enforcement program. For example, FAA lacked requirements or criteria for evaluating its designee programs. In another example, FAA's nationwide enforcement database is not as useful as it could be because of missing or incomplete historical information about enforcement cases.

section_title: Letter
paragraphs: The U.S. commercial aviation industry, with less than one fatal accident  per 5 million flights from 2002 through 2005, has an extraordinary safety  record. This record is a result of the efforts of the Federal Aviation  Administration (FAA), airlines, manufacturers, and others in the aviation  industry to maintain one of the safest aviation systems in the world.  However, when passenger airlines have accidents or serious incidents,  regardless of their rarity, the consequences can be tragic, as a single  accident can result in hundreds of deaths. In order to maintain a high level  of safety, it is critical to have a safety oversight system that is  comprehensive, efficient, and effective and can provide an early warning  of hazards that can lead to accidents. It is equally important to have a  skilled, well-trained workforce to implement and monitor this safety  oversight system. FAA’s workforce of about 3,200 inspectors stationed at  more than 100 field offices throughout the world is responsible for  carrying out the agency’s processes to certify, inspect, and enforce safety  regulations for all aspects of the aviation industry, including the aircraft  and its component parts, over 100 commercial airlines, about 5,000 aircraft  repair stations, and hundreds of thousands of pilots and mechanics. FAA  augments its inspector workforce with nearly 13,600 designated  organizations and individuals (designees) that conduct the more routine  aspects of industry oversight, such as administering tests to pilots and  mechanics as part of their certification requirements, and augments the  safety information that it obtains from inspections with industry  partnership programs. Keys to making this safety oversight system work  are to: (1) establish programs that focus resources on areas of highest  safety risk and on mitigating risks; (2) provide training and other  communication to ensure that inspectors maintain the skills and  knowledge to consistently carry out the agency’s oversight programs; and  (3) have processes and data to continuously monitor, evaluate, and  improve the numerous oversight programs that make up the safety  oversight system. This statement focuses on these three key areas of  FAA’s “early warning system” and is based on our recent reports on FAA’s  inspection oversight programs, industry partnership programs,  enforcement program, and training program. We will also discuss our  recommendations that FAA has not fully addressed in these areas.
FAA’s aviation safety oversight system includes programs that focus on  risk identification and mitigation through a system safety approach, the  leveraging of resources, and enforcement of safety regulations, but the  benefits of these programs are not being fully realized. FAA’s system  safety approach has many strengths, including the addition of a program  that emphasizes risk identification to its traditional inspection program for  overseeing non-legacy airlines, which is not based on risk. However, it is  likely that the benefits of this approach could be enhanced if the  inspection workload for non-legacy airlines was not still heavily oriented  to the traditional inspection program’s non-risk based activities. FAA  leverages resources through its designee programs, in which designees  perform about 90 percent of certification-related activities, thus allowing  FAA to better concentrate its limited staff resources on the most safety- critical functions. However, concerns about the consistency and adequacy  of designee oversight that FAA field offices provide have been raised by  experts and other individuals we interviewed. FAA also leverages its  resources through industry partnership programs, which are designed to  assist the agency in receiving safety information. For example, FAA  encourages voluntary reports of safety violations by responding to them  by issuing a warning letter rather than a fine or other legal sanction. FAA’s  enforcement program, which is an outgrowth of its inspection process, is  intended to ensure industry compliance with safety regulations and is  another important element of its safety oversight system. FAA’s policy for  assessing legal sanctions against entities or individuals that do not comply  with aviation safety regulations is intended to deter future violations.  However, we found that recommendations for sanctions are sometimes  reduced on the basis of factors that are not associated with the merits of  the case, and the economic literature on deterrence suggests that the goal  of preventing future violations is weakened when the penalties for  violations are lowered for reasons not related to the merits of the case.
FAA has made training an integral part of its safety oversight system, but  several actions could improve the results of its training efforts. FAA’s use  of a risk-based system safety approach to inspections requires inspectors  to apply data analysis and auditing skills to identify and control potential  risks. Therefore, it is important that inspectors are well-trained in this  approach and have sufficient knowledge of increasingly complex aircraft  and systems to effectively identify safety risks. FAA has established  mandatory training requirements for its workforce as well as designees.
We have reported that FAA has generally followed effective management  practices for planning, developing, delivering, and assessing the impact of  its technical training for safety inspectors, although some practices have  yet to be fully implemented. For example, in developing its training  curriculum for inspectors, FAA followed effective management practices,  such as developing courses that support changes in inspection procedures  resulting from regulatory changes or agency initiatives. On the other hand,  FAA develops technical courses on an ad hoc basis rather than as part of  an overall curriculum for each type of inspector, such as inspectors of  operations or cabin safety, because the agency has not systematically  identified the technical skills and competencies each type of inspector  needs to effectively perform inspections. FAA has recognized the need for  improvements to its training program in this and other areas.
It is important for FAA to have effective evaluative processes and accurate  nationwide data for its numerous safety oversight programs so that  program managers and other officials have assurance that the safety  programs are having their intended effect. Such processes and data are  especially important because FAA’s workforce is so dispersed throughout  the world—with thousands of staff working out of more than 100 offices  worldwide—and because FAA’s use of a system safety approach  represents a cultural shift from its traditional inspection program.  Evaluation is important for understanding if the cultural shift has  effectively occurred. Our most recent work has shown the lack of  evaluative processes and limitations with data for FAA’s inspection  program for non-legacy airlines, designee programs, industry partnership  programs, and enforcement program. For example, we found that FAA  lacked requirements or criteria for evaluating its designee programs. In  another example, FAA’s enforcement policy calls for the assessment of  sanctions that would potentially deter future violations. However, FAA  lacks an evaluative process, so it is not known whether the agency’s  enforcement practices, such as at times reducing sanctions, may weaken  any deterrent effect that would be expected from such sanctions.  Furthermore, FAA’s ability to evaluate its programs is hindered by the lack  of useful nationwide data. For example, FAA’s nationwide enforcement  database is not as useful as it could be because of missing or incomplete  historical information about enforcement cases.
In order to help FAA fully realize the benefits of its safety oversight  system, we have made a number of recommendations to address the  weaknesses that we identified in our reviews. These recommendations  have not been fully implemented, although in some cases FAA has taken  steps towards addressing them. Evaluative processes and relevant data are  particularly important as FAA works to change its culture by incorporating  a system safety approach into its oversight, and we have recommended  that FAA develop continuous evaluative processes for its oversight of non- legacy airlines, its designee programs, and its enforcement program, and  systematically assess inspectors’ technical training needs. In addition,  FAA’s nationwide databases are in need of improvements in their  comprehensiveness and ease of use. We have recommended that FAA  improve the consistency and completeness of its designee and  enforcement databases. Continuous improvements in these areas are  critical to FAA’s ability to have a robust “early warning system” in order to  maintain one of the safest aviation systems in the world.
subsections: section_title: Background
paragraphs: FAA’s safety oversight system is made up of a number of programs for  airlines and other entities. Safety oversight programs for airlines provide  for their initial certification, periodic surveillance, and inspection. Since  1985, FAA has used National Work Program Guidelines (NPG), its  traditional inspection program for airlines, as a primary means of ensuring  that airlines comply with safety regulations. In NPG, an FAA committee of  program managers identifies an annual minimum set of required  inspections that are to be undertaken to ensure that airlines are in  compliance with their operating certificates. In 1998, the agency  implemented the Air Transportation Oversight System (ATOS), which  currently oversees the nation’s largest 15 commercial airlines and cargo  carriers, with the goal of eventually including all commercial passenger  and cargo airlines in it. ATOS emphasizes a system safety approach that  extends beyond periodically checking airlines for compliance with  regulations to the use of technical and managerial skills to identify,  analyze, and control hazards and risks. For example, under ATOS,  inspectors develop surveillance plans for each airline, based on data  analysis and assessment of risks, and adjust the plans periodically based  on inspection results. However, the agency has been delayed in placing a  significant number of other passenger airlines in ATOS, resulting in 99  passenger airlines, which we refer to as non-legacy airlines, continuing to  be overseen through NPG, a process that is not risk-based or system safety  oriented. In 2002, FAA added the Surveillance and Evaluation Program  (SEP) to the NPG inspection program to incorporate principles of ATOS  into its oversight of non-legacy passenger airlines. The two programs are  used together to establish the number of annual inspections for non-legacy  airlines. (Appendix 1 describes each inspection program.) Figure 1  illustrates some typical activities covered during inspections.
FAA’s safety oversight programs for other aspects of the aviation  industry—including manufacturers of aircraft and aircraft parts, repair  stations, flight schools, aviation maintenance technician schools, pilots,  and mechanics—involve certification, surveillance, and inspection by  FAA’s safety inspectors, engineers, flight surgeons, and designated  representatives. FAA authorizes about 13,400 private individuals and about  180 organizations (called “designees”) to act as its representatives to  conduct many safety certification activities, such as administering flight  tests to pilots, inspecting repair work by maintenance facilities,  conducting medical examinations of pilots, and approving designs for  aircraft parts. These designees are grouped into 18 different programs and  are overseen by three FAA offices—Flight Standards Service, Aerospace  Medicine, and Aircraft Certification Service—all of which are under the  Office of Aviation Safety (see fig. 2).
Since 1990, FAA has emphasized gaining compliance from the aviation  industry through cooperative means by establishing industry partnership  programs with the aviation community that allow participants, such as  airlines and pilots, to self-report violations of safety regulations and help  identify safety deficiencies, and potentially mitigate or avoid fines or other  legal action. For example, the Voluntary Disclosure Program encourages  the self-reporting of manufacturing problems and safety incidents by  participants that can include air carriers and repair stations. Appendix II  describes the industry partnership programs.
When violations of statutory and regulatory requirements are identified  through inspections, through the partnership programs in certain cases, or  through other methods, FAA has a variety of enforcement tools that it may  use to respond to them, including administrative actions (such as issuing a  warning notice or a letter of correction that includes the corrective actions  the violator will take) and legal sanctions (such as levying a fine or  suspending or revoking a pilot or other FAA-issued certificate).
section_title: FAA’s Safety Oversight System Focuses on Risk Identification and Mitigation Through System Safety, Leveraging of Resources, and Enforcement of Safety Regulations, but Benefits Are Not Being Fully Realized
paragraphs: In recent reports, we found that FAA’s safety oversight system has  programs that focus on risk identification and mitigation through a system  safety approach, the leveraging of resources, and enforcement of safety  regulations, but that the benefits of these programs are not being fully  realized. In our recent report on FAA’s oversight of non-legacy airlines, we  found that the focus on risk identification through the addition of SEP has  many strengths and allows for enhancing the efficiency of FAA’s oversight  activities. Rather than relying on NPG’s customary method of conducting  a set number of inspections of an airline’s operations, SEP emphasizes a  system safety approach of using risk analysis techniques. SEP allows for  the efficient use of inspection staff and resources by prioritizing workload  based on areas of highest risk, and it includes a requirement that  inspectors verify that corrective actions have occurred. For example, FAA  has developed risk assessment worksheets for SEP that are aligned with  key airline systems that guide inspectors through identifying and  prioritizing risks. The worksheets guide inspectors to organize the results  of their previous inspections and surveillance into a number of areas such  as flight operations and personnel training in order to identify specific  risks in each area and target the office’s resources to mitigating those  risks. The development of a system safety approach addresses a long- standing concern by us that FAA did not have a methodology for assessing  airline safety risks so that it could target limited inspection resources to  high-risk conditions. Another strength of SEP, consistent with findings in  our past reports, is that SEP relies on teams of inspectors, which are  generally more effective than individual inspectors in their ability to  collectively identify concerns.
However, the benefits of FAA’s system safety approach for the inspection  of non-legacy airlines could be enhanced by a more complete  implementation of SEP and addressing other challenges. The inspection  workload for non-legacy airlines is still heavily oriented to the NPG’s non- risk based activities. For example, as shown in table 1, from fiscal years  2002 through 2004, 77 percent of inspection activities required for the top  25 non-legacy airlines in terms of the number of enplanements were  identified through NPG, and the remaining percentage of inspection  activities were identified based on risk through SEP. Although inspectors  can replace NPG-identified activities with SEP-identified activities that  they deem constitute a greater safety risk, we found that FAA inspectors  interpret agency emphasis on NPG as discouraging this practice. In order  to ensure that all inspectors who oversee non-legacy airlines have a  complete and timely understanding of the agency’s policies relating to the  inspection process, we recommended in September 2005 that FAA  improve communication with and training of inspectors in this area.
Another way that FAA attempts to enhance the efficiency of its oversight  activities is through its designee programs. We reported that FAA  maximizes its resources by allowing designees to perform about 90  percent of certification-related activities, thus allowing FAA to better  concentrate its limited staff resources on the most safety-critical  functions. For example, while designees conduct routine certification  functions, such as approvals of aircraft technologies that the agency and  designees have had previous experience with, FAA focuses on new and  complex aircraft designs or design changes. In addition, the use of  designees expands FAA’s access to technical expertise within the aviation  community. For the aviation industry, the designee programs enable  individuals and organizations to obtain required FAA certifications—such  as approvals of the design, production, and airworthiness of aircraft—in a  timely manner, thus reducing delays and costs to the industry that might  result from scheduling direct reviews by FAA staff. For example, officials  from an aircraft manufacturer told us that the use of designees has added  significantly to the company’s ability to enhance and improve daily  operations by decreasing certification delivery time and increasing the  flexibility and utilization of company resources. In addition, designees are  convenient to the aviation industry due to their wide dispersal throughout  the United States.
However, concerns about the consistency and adequacy of designee  oversight that FAA field offices provide have been raised by experts and  other individuals we interviewed. For example, designees and industry  officials that we spoke with indicated that FAA’s level of oversight and  interpretation of rules differ among regions and among offices within a  region, which limits FAA’s assurance that designees’ work is performed  uniformly in accordance with FAA’s standards and policy. Experts also  ranked this issue as a top weakness. Table 2 shows the top five  weaknesses identified by our experts. Experts also made a number of  suggestions to strengthen the designee program, including clearly defining  and following agency criteria for selecting designees and increasing  penalties for designees found to violate standards or who do not exercise  proper judgment. To improve management control of the designee  programs, and thus increase assurance that designees meet FAA’s  performance standards, we recommended that FAA develop mechanisms  to improve the compliance of FAA program and field offices with existing  policies and incorporate, as appropriate, suggestions from our expert  panel. In response to our recommendations, FAA is planning, among other  things, to form a team to identify and share best practices for overseeing  designee programs.
FAA also leverages its resources through its industry partnership  programs. These partnership programs are designed to assist the agency in  receiving safety information, including reports of safety violations.  According to FAA officials, the Aviation Safety Action Program, Aviation  Safety Reporting Program, and Voluntary Disclosure Reporting Program  augment FAA’s enforcement activities and allow FAA to be aware of many  more safety incidents than are discovered during inspections and  surveillance. In addition, the Flight Operational Quality Assurance  Program provides safety information in the form of recorded flight data  from participating airlines. FAA has established some management  controls over its partnership programs, such as procedures to track  actions taken to correct safety incidents reported under the programs, but  the agency lacks management controls to measure and evaluate the  performance of these programs, an issue that we will discuss later in the  testimony.
FAA’s enforcement process, which is intended to ensure industry  compliance with safety regulations, is another important element of its  safety oversight system. FAA’s policy for assessing legal sanctions against  entities or individuals that do not comply with aviation safety regulations  is intended to deter future violations. FAA has established some  management controls over its enforcement efforts, with procedures that  provide guidance on identifying regulated entities and individuals that are  subject to inspections or surveillance actions, determining workload  priorities on the basis of the timing and type of inspection to be  performed, detecting violations of safety regulations, tracking the actions  that are taken by the entities and individuals to correct the violations and  achieve compliance with regulations, and imposing punitive sanctions or  remedial conditions on the violators. These procedures provide FAA  inspectors, managers, and attorneys with a process to handle violations of  safety regulations that are found during routine inspections.
However, we found that the effect of FAA’s legal sanctions on deterrence  is unclear, and that recommendations for sanctions are sometimes  changed on the basis of factors that are not associated with the merits of  the case. We found that from fiscal years 1993 through 2003, attorneys in  FAA’s Office of the Chief Counsel authorized a 52 percent reduction in the  civil monetary penalties assessed from a total of $334 million to $162  million. FAA officials told us that the agency sometimes reduces sanctions  in order to prioritize attorneys’ caseloads by closing the cases more  quickly through negotiating a lower fine. Economic literature on  deterrence suggests that although negative sanctions (such as fines and  certificate suspensions) can deter violations, if the violator expects  sanctions to be reduced, he or she may have less incentive to comply with  regulations. In effect, the goal of preventing future violations is weakened  when the penalties for present violations are lowered for reasons not  related to the merits of the case. In addition, FAA lacks management  controls to measure and evaluate its enforcement process, which we  discuss later in this testimony.
section_title: FAA Has Made Training an Integral Part of Its Safety Oversight System but Several Actions Could Improve Results
paragraphs: FAA’s use of a risk-based system safety approach to inspections requires  inspectors to apply data analysis and auditing skills to identify, analyze,  assess, and control potential hazards and risks. Therefore, it is important  that inspectors are well-trained in this approach and have sufficient  knowledge of increasingly complex aircraft, aircraft parts, and systems to  effectively identify safety risks. It is also important that FAA’s large cadre  of designees is well-trained in federal aviation regulations and FAA  policies. FAA has made training an integral part of its safety inspection  system and has established mandatory training requirements for its  workforce as well as designees. FAA provides inspectors with extensive  training in federal aviation regulations; inspection and investigative  techniques; and technical skills, such as flight training for operations  inspectors. The agency provides its designees with an initial indoctrination  that covers federal regulations and agency policies, and refresher training  every 2 to 3 years.
We have reported that FAA has generally followed effective management  practices for planning, developing, delivering, and assessing the impact of  its technical training for safety inspectors, although some practices have  yet to be fully implemented. In its planning activities for training, FAA has  linked technical training efforts to its goal of safer air travel and has  identified technical proficiencies needed to improve safety inspectors’  performance in meeting this goal. For example, FAA’s Offices of Flight  Standards and Aircraft Certification have identified gaps in several of the  competencies required to conduct system safety inspections, including  risk assessment, data analysis, systems thinking, and designee oversight.  According to FAA, it is working to correct these gaps. We have also  identified gaps in the training provided to inspectors in the Office of Flight  Standards who oversee non-legacy airlines, and have recommended that  FAA improve inspectors’ training in areas such as system safety and risk  management to ensure that these inspectors have a complete and timely  understanding of FAA’s inspection policies. We have identified similar  competency gaps related to designee oversight. For example, FAA does  not require refresher training concerning designee oversight, which  increases the risk that staff do not retain the information, skills, and  competencies required to perform their oversight responsibilities. We  recommended that FAA provide additional training for staff who directly  oversee designees. We did not identify any specific gaps in the  competencies of designees. In prioritizing funding for course  development activities, FAA does not explicitly consider which projects  are most critical. Figure 3 describes the extent to which FAA follows  effective management practices in planning training.
In developing its training curriculum for inspectors, FAA also for the most  part follows effective management practices, such as developing courses  that support changes in inspection procedures resulting from regulatory  changes or agency initiatives. On the other hand, FAA develops technical  courses on an ad hoc basis rather than as part of an overall curriculum for  each inspector specialty—such as air carrier operations, maintenance, and  cabin safety—because the agency has not systematically identified the  technical skills and competencies each type of inspector needs to  effectively perform inspections. Figure 4 describes the extent to which  FAA follows effective management practices in developing training.
In delivering training, FAA has also generally followed effective  management practices. (See fig. 5.) For example, FAA has established  clear accountability for ensuring that inspectors have access to technical  training, developed a way for inspectors to choose courses that meet job  needs and further professional development, and offers a wide array of  technical and other courses. However, both FAA and its inspectors  recognize the need for more timely selection of inspectors for technical  training. In addition, FAA acknowledges the need to increase  communication between inspectors and management with respect to the  training program, especially to ensure that inspectors have bought into the  system safety approach to inspections.
FAA offers numerous technical courses from which inspectors can select  to meet job needs. However, from our survey of FAA’s inspectors, we  estimate that only about half think that they have the technical knowledge  needed for their jobs. FAA officials told us that inspectors’ negative views  stem from their wanting to acquire proficiencies that are not as crucial in a  system safety environment. We also found a disparity between inspectors  and FAA concerning the receipt of requested training. We estimated that  28 percent of inspectors believe that they get the technical training that  they request. However, FAA’s records show that FAA approves about 90  percent of these requests, and inspectors are making good progress in  receiving training. Over half of the inspectors have completed at least 75  percent of technical training that FAA considers essential. FAA officials  told us that inspectors’ negative views on their technical knowledge and  the training they have received stem from their not accepting FAA’s move  to a system safety approach. That is, the inspectors are concerned about  acquiring individual technical proficiency that is not as crucial in a system  safety environment. Given that it has not completed assessing whether  training for each inspector specialty meets performance requirements,  FAA is not in a position to make definitive conclusions concerning the  adequacy of inspector technical training.
FAA also generally followed effective management practices in evaluating  training. The agency requires that each training course receive a  systematic evaluation every 3 years to determine if the course is up to date  and relevant to inspectors’ jobs, although training officials noted that  many courses have yet to undergo such an evaluation. However, FAA  collects limited information on the effectiveness of training, and its  evaluations have not measured the impact of training on FAA’s mission  goals, such as reducing accidents. Training experts acknowledge that  isolating performance improvements resulting from training programs is  difficult for any organization. (See fig. 6.)
While FAA follows many effective management practices in its training  program, the agency also recognizes the need for improvements, including  (1) systematically assessing inspectors’ needs for technical and other  training, (2) better timing of technical training so that inspectors receive it  when it is needed to perform their jobs, and (3) better linking the training  provided to achieving agency goals of improving aviation safety. FAA has  begun to act in these areas, and we believe that if effectively implemented,  the actions should improve the delivery of training and ultimately improve  aviation safety. Therefore, it is important for FAA to follow through with  its efforts. As a result, we recommended in September 2005, among other  things, that in order to ensure that inspector technical training needs are  identified and met in a timely manner, FAA systematically assess  inspectors’ technical training needs, better align the timeliness of training  to when inspectors need the training to do their jobs, and gain inspectors’  acceptance for changes made or planned to their training.
It is important that both FAA’s inspection workforce and FAA-certified  aviation mechanics are knowledgeable about increasingly complex  aircraft, aircraft parts, and systems. While we did not attempt to assess the  technical proficiency that FAA’s workforce requires and will require in the  near future, FAA officials said that inspectors do not need a substantial  amount of technical training courses because inspectors are hired with a  high degree of technical knowledge of aircraft and aircraft systems. They  further indicated that inspectors can sufficiently keep abreast of many of  the changes in aviation technology through FAA and industry training  courses and on-the-job training. However, in its certification program for  aviation mechanics, we found that FAA standards for minimum  requirements for aviation courses at FAA-approved aviation maintenance  technician schools and its requirements for FAA-issued mechanics  certificates do not keep abreast with the latest technologies. In 2003, we  reported that those standards had not been updated in more than 50  years. We recommended that FAA review the curriculum and  certification requirements and update both. FAA plans to make changes in  the curriculum for FAA approved aviation maintenance technicians that  reflect up-to-date aviation technologies and finalize and distribute a  revised Advisory Circular in March 2006 that describes the curriculum  changes. FAA then plans to allow the aviation industry time to implement  the recommended curriculum changes before changing the requirements  for FAA-issued mechanics certificates.
section_title: FAA Has Evaluated Some Safety Programs, but the Lack of Evaluative Systems and Nationwide Data Impedes FAA’s Ability to Continuously Monitor Its Safety Programs
paragraphs: It is important for FAA to have effective evaluative processes and accurate  nationwide data on its numerous safety oversight programs so that  program managers and other officials have assurance that the safety  programs are having their intended effect. Such processes and data are  especially important because FAA’s workforce is so dispersed  worldwide—with thousands of staff working out of more than 100 local  offices—and because FAA’s use of a risk-based system safety approach  represents a cultural shift from its traditional inspection program.  Evaluation is important to understanding if the cultural shift has  effectively occurred. Our most recent work has shown the lack of such  processes and limitations with data for FAA’s inspection programs for  non-legacy airlines, designee programs, industry partnership programs,  and enforcement program. In response to recommendations that we have  made regarding these programs, some improvements are being made. On  the positive side, as we mentioned earlier, our most recent work found  that FAA generally follows effective management practices in evaluating  individual technical training courses.
FAA has not evaluated its inspection oversight programs for non-legacy  airlines—which include SEP and NPG—to determine how the programs  contribute to the agency’s mission and overall safety goals, and its  nationwide inspection database lacks important information that could  help it perform such evaluations—such as whether risks identified through  SEP have been mitigated. In addition, the agency does not have a process  to examine the nationwide implications of or trends in the risks that  inspectors have identified through their risk assessments—information it  would need to proactively determine risk trends at the national level on a  continuous basis. FAA’s evaluation office instead conducts analyses of the  types of inspections generated under SEP by airline and FAA region,  according to FAA. We recommended that FAA develop a continuous  evaluative process for activities under SEP and link SEP to the  performance-related goals and measures developed by the agency, track  performance toward these goals, and determine appropriate program  changes. FAA is considering our recommendation, but its plan to place the  remaining non-legacy airlines in the ATOS program by the end of fiscal  year 2007 might make this recommendation unnecessary, according to the  agency. Since FAA’s past efforts to move airlines to ATOS have  experienced delays, we believe that this recommendation is still valid.
We also found that FAA lacked requirements or criteria for periodically  evaluating its designee programs. In 2004, we reported that the agency had  evaluated 6 of its 18 designee programs over the previous 7 years and had  plans to evaluate 2 more, although it had no plans to evaluate the  remaining 10 programs because of limited resources. FAA conducted  these evaluations on an ad hoc basis usually at the request of headquarters  directors or regional office managers. In addition, we found that FAA’s  oversight of designees is hampered, in part, by the limited information on  designees’ performance contained in the various designee databases.  These databases contain descriptive information on designees, such as  their types of designations and status (i.e., active or terminated). More  complete information would allow the agency to gain a comprehensive  picture of whether staff are carrying out their responsibilities to oversee  designees. To improve management control of the designee programs, and  thus increase assurance that designees meet the agency’s performance  standards, we recommended that FAA establish a process to evaluate all  designee programs and strengthen the effectiveness of its designee  databases by improving the consistency and completeness of information  in them. To address our recommendations, FAA expects to develop a plan  to evaluate all designee programs on a recurring basis and intends to  establish a team that will examine ways to improve automated information  related to designees.
In addition, we found that FAA does not evaluate the effects of its industry  partnership and enforcement programs to determine if stated program  goals, such as deterrence of future violations, are being achieved. For  example, little is known about nationwide trends in the types of violations  reported under the partnership programs or whether systemic, nationwide  causes of those violations are identified and addressed. Furthermore,  FAA’s enforcement policy calls for inspectors and legal counsel staff to  recommend or assess enforcement sanctions that would potentially deter  future violations. However, without an evaluative process, it is not known  whether the agency’s practice of generally closing cases with  administrative actions rather than legal sanctions and at times reducing  the amount of the fines, as mentioned earlier in this testimony, may  weaken any deterrent effect that would be expected from sanctions.
FAA’s ability to evaluate the impact of its enforcement efforts is also  hindered by the lack of useful nationwide data. FAA inspection offices  maintain independent, site-specific databases because they do not find the  nationwide enforcement database—the Enforcement Information System  (EIS)—as useful as it could be because of missing or incomplete historical  information about enforcement cases. As a result of incomplete data on  individual cases, FAA inspectors lack the complete compliance history of  violators when assessing sanctions. We recommended that FAA develop  evaluative processes for its enforcement activities and partnership  programs and use them to create performance goals, track performance  towards those goals, and determine appropriate program changes. We also  recommended that FAA take steps to improve the usefulness of the EIS  database by enhancing the completeness of enforcement information. FAA  expects to address some of these issues as it revises its enforcement  policy, which is expected to be issued later in fiscal year 2006. In addition,  FAA has established a database workgroup that is developing long- and  short- term solutions to address the problems with EIS.
section_title: Recommendations We Have Made to Improve FAA’s Safety Oversight System
paragraphs: In order to help FAA fully realize the benefits from its safety oversight  system, we have made a number of recommendations to address  weaknesses that we identified in our reviews. These recommendations  have not been fully implemented, although in some cases FAA has taken  steps towards addressing them. Evaluative processes and relevant data are  particularly important as FAA works to change its culture by incorporating  a system safety approach into its oversight, and we have recommended  that FAA develop continuous evaluative processes for its oversight  programs for non-legacy airlines, its designee programs, and its industry  partnership and enforcement programs, and systematically assess  inspectors’ technical training needs. In addition, FAA’s nationwide  databases are in need of improvements in their comprehensiveness and  ease of use. Without comprehensive nationwide data, FAA does not have  the information needed to evaluate its safety programs and have assurance  that they are having the intended results. We have recommended that FAA  improve the completeness of its designee and enforcement databases.  Continuous improvements in these areas are critical to FAA’s ability to  have a robust “early warning system” and maintain one of the safest  aviation systems in the world.
section_title: Contacts and Acknowledgments
paragraphs: For further information on this testimony, please contact Dr. Gerald  Dillingham at (202) 512-2834 or by email at dillinghamg@gao.gov.  Individuals making key contributions to this testimony include Brad  Dubbs, Phillis Riley, Teresa Spisak, and Alwynne Wilbur.
section_title: Appendix I: Description of FAA’s Inspection Programs
paragraphs: Table 3 describes the Federal Aviation Administration’s (FAA) three  inspection processes for overseeing airlines: Air Transportation Oversight  System (ATOS), National Work Program Guidelines (NPG), and  Surveillance and Evaluation Program (SEP). Many of the elements of  ATOS, such as the use of data to identify risks and the development of  surveillance plans by inspectors, are incorporated in the SEP process. The  NPG process, in contrast, is not focused on the use of data and relies on an  established set of inspections that are not risk based.
section_title: Appendix II: Description of FAA’s Partnership Programs
subsections: section_title: Aviation Safety Action Program (ASAP)
paragraphs: Participation: Participants include employees of air carriers and repair  stations that have entered into a memorandum of understanding with the  Federal Aviation Administration (FAA). The memoranda can cover  employee groups, such as pilots, maintenance employees, dispatchers, or  flight attendants. Each employee group is covered by a separate  memorandum of understanding. As of June 2004, FAA had accepted 54  memoranda of understanding and received over 80,000 ASAP reports,  which may or may not include safety violations, according to FAA  officials.
Purpose: ASAP seeks to improve aviation safety through the voluntary  self-reporting of safety incidents under the procedures set forth in the  memorandum of understanding. Under the program, FAA does not take  enforcement action against employees who voluntarily self-reported safety  violations for reports that are sole-source (the report is the only way FAA  would have learned about the incident) and will pursue administrative  action only for reports that are not sole-source. Incidents that involve  alcohol, drugs, criminal activity, or an intentional disregard for safety are  not eligible for self-reporting under ASAP.
Process: Each memorandum of understanding is a voluntary partnership  between FAA, the airline, and an employee group. Although employee  groups are not always included, FAA encourages their participation. The  memorandum of understanding ensures that employees who voluntarily  disclose FAA safety violations in accordance with the procedures and  guidelines of ASAP will receive administrative action or no action in lieu  of legal enforcement action.
Once a memorandum of understanding is approved, employees can begin  reporting violations that fall under the agreement. When a violation  occurs, an employee notifies the Event Review Committee, which includes  representatives from FAA and the airline or the repair station and  generally includes the appropriate employee association. The committee  must be notified in writing within the time limit specified in the  memorandum of understanding. The committee then determines whether  to accept the report under the ASAP program. If the report is accepted (it  meets the acceptance criteria in the memorandum and does not involve  criminal activity, substance abuse, controlled substances, or alcohol), then  the committee determines the action to take. That action may include  remedial training or administrative action, but it will not include a legal  sanction.
Results: FAA does not know the overall program results because it does  not have a national, systematic process in place to evaluate the overall  success of ASAP. However, FAA cites examples that describe ASAP’s  contribution to enhanced aviation safety. These examples include  identifying deficiencies in aircraft operations manuals, airport equipment,  and runways. In July 2003, FAA’s Compliance and Enforcement Review  recommended that FAA evaluate the use and effectiveness of this  program.
section_title: Aviation Safety Reporting Program (ASRP)
paragraphs: Participation: Participants are all users of the national airspace system,  including air traffic controllers and employees of air carriers and repair  stations.
Purpose: The program is designed to improve aviation safety by offering  limited immunity for individuals who voluntarily report safety incidents.  ASRP was founded after TWA Flight 514 crashed on approach to landing  in December 1974 after the crew misinterpreted information on the  approach chart. This accident occurred only 6 weeks after another plane  experienced the same error.
Process: The National Aeronautics and Space Administration (NASA)  administers this program. When a safety incident occurs, a person may  submit a form and incident report to NASA. There are four types of forms  that can be submitted to NASA: (1) Air Traffic Control, (2) General  Reports (includes Pilots), (3) Flight Attendants, and (4) Maintenance  Personnel.
At least two aviation safety analysts read these forms and the incident  reports that accompany them. The analysts at NASA screen the incident  reports for urgent safety issues, which will be marked for immediate  action to the appropriate FAA office or aviation authority. NASA analysts  also edit the report’s narrative to eliminate any identifying information. In  addition, each report has a tear-off portion, which is separated and  returned to the individual who reported the incident as a receipt of the  incident report’s acceptance into the ASRP. When a safety violation that  has been previously reported under ASRP comes to the attention of FAA,  the agency issues a legal sanction, which is then waived. Reports that  would not be eligible to have a legal sanction waived include deliberate  violations, violations involving a criminal offense, or accident; reports filed  by participants who have committed a violation of federal aviation  regulations or law within the last 5 years and reports filed later than 10  days following an incident.
Results: While FAA and NASA do not know the overall program results  because they do not have a formal national evaluation program to measure  the overall effectiveness of the program, the agencies widely disseminate  information generated from the program to aircraft manufacturers and  others. ASRP reports are compiled into a database known as the Aviation  Safety Reporting System. When a potentially hazardous condition is  reported, such as a defect in a navigational aid or a confusing procedure,  NASA will send a safety alert to aircraft manufacturers, the FAA, airport  representatives, and other aviation groups. The database is used for a  monthly safety bulletin that includes excerpts from incident reports with  supporting commentary by FAA safety experts. NASA officials estimate  that the bulletin is read by over 150,000 people. In addition, individuals and  organizations can request a search of the database for information on  particular aircraft aviation safety subjects, including human performance  errors and safety deficiencies. Further, NASA has used the database to  analyze operational safety issues, such as general aviation incidents, pilot  and controller communications, and runway incursions.
section_title: Flight Operational Quality Assurance (FOQA)
paragraphs: Participation: Participants include air carriers that equip their airplanes  to record flight data. As of March 2004, 13 airlines had FAA-approved  FOQA programs, and approximately 1,400 airplanes were equipped for the  program.
Purpose: FOQA is designed to enhance aviation safety through the analysis  of digital flight data generated during routine flights.
Process: Air carriers that participate in the program equip their aircraft  with special acquisition devices or use the airplanes’ flight data recorders  to collect data and determine if the aircraft are deviating from standard  procedures. These data include engine temperatures, descent rate, and  deviations from the flight path. When the aircraft lands, data are  transmitted from the aircraft to the airline’s FOQA station, where they are  analyzed for flight trends and possible safety problems.
Once the data are transmitted to the FOQA ground station, the data are  extracted and analyzed by software programs. The FOQA data are  combined with data from maintenance databases, weather conditions, and  other safety reporting systems, such as ASAP, in order to identify trends in  flight operations. The analysis typically focuses on events that fall outside  normal boundaries specified by the manufacturer’s operational limitations  and the air carrier’s operational standards.
FOQA data are collected and analyzed by individual air carriers. The data  on safety trends are made available to FAA in an aggregated form with no  identification of individual carriers. According to FAA officials, air carriers  do not want to release this data to any outside party (including FAA)  because of concerns that the data could then be publicly released. Air  carriers pay for the special flight data recorders that can record FOQA  data, which cost approximately $20,000 each. Although this can be an  expensive investment for some air carriers, most newer aircraft models  come with the data recorder built into the airplane. The International Civil  Aviation Organization (ICAO) has recommended that airlines from  member countries implement a FOQA program. FAA has notified ICAO  that the program will remain voluntary in the United States.
Results: Although FAA has no formal national evaluation program to  measure the overall results or effectiveness of FOQA programs, FAA cites  examples that describe FOQA’s contribution to enhanced aviation safety.  For example, one FOQA program highlighted a high rate of descent when  airplanes land at a particular airport. On the basis of the information  provided from FOQA, air traffic controllers at the airport were able to  develop alternative approach procedures to decrease the rate of descent.
section_title: Voluntary Disclosure Reporting Program (VDRP)
paragraphs: Participation: Participants include air carriers, repair stations, and  production approval holders.
Purpose: FAA initiated the program to promote aviation safety by  encouraging the voluntary self-reporting of manufacturing, and quality  control problems and safety incidents involving FAA requirements for  maintenance, flight operations, drug and alcohol prevention programs, and  security functions.
Process: Upon discovering a safety violation, participants can voluntarily  disclose the violation to FAA within 24 hours. The initial notification  should include a description of the violation, how and when the violation  was discovered, and the corrective steps necessary to prevent repeat  violations. Within 10 days of filing the initial notification to FAA, the entity  is required to provide a written report that cites the regulations violated,  describes how the violation was detected, provides an explanation of how  the violation was inadvertent, and provides a description of the proposed  comprehensive fix. FAA may pursue legal action if the participant  discloses violations during, or in anticipation of, an FAA inspection.
The violation must be reported immediately after being detected, must be  inadvertent, must not indicate that a certificate holder is unqualified, and  must include the immediate steps that were taken to terminate the  apparent violation. If these conditions are met, and the FAA inspector has  approved the comprehensive fix, then the FAA inspector will prepare a  letter of correction and the case is considered closed with the possibility  of being reopened if the comprehensive fix is not completed.
Results: FAA does not know the overall program results because it does  not have a process to measure the overall effectiveness of the program  nationwide. A 2003 internal FAA report recommended that the agency  evaluate the use and effectiveness of this program.
section_title: Selected GAO Reports on Aviation Safety
paragraphs: Aviation Safety: System Safety Approach Needs Further Integration into  FAA’s Oversight of Airlines. GAO-05-726. Washington, D.C.: September 28,  2005.
Aviation Safety: FAA Management Practices for Technical Training  Mostly Effective; Further Actions Could Enhance Results. GAO-05-728.  Washington, D.C.: September 7, 2005.
Aviation Safety: Oversight of Foreign Code-Share Safety Program Should  Be Strengthened. GAO-05-930. Washington, D.C.: August 5, 2005.
Aviation Safety: FAA Needs to Strengthen the Management of Its  Designee Programs. GAO-05-40. Washington, D.C.: October 8, 2004.
Aviation Safety: Better Management Controls are Needed to Improve  FAA’s Safety Enforcement and Compliance Efforts. GAO-04-646.  Washington, D.C.: July 6, 2004.
Aviation Safety: Information on FAA’s Data on Operational Errors at  Air Traffic Control Towers. GAO-03-1175R. Washington, D.C.: September  23, 2003.
Aviation Safety: FAA Needs to Update the Curriculum and Certification  Requirements for Aviation Mechanics. GAO-03-317. Washington, D.C.:  March 6, 2003.
Aviation Safety: FAA and DOD Response to Similar Safety Concerns.  GAO-02-77. Washington. D.C.: January 22, 2002.
Aviation Safety: Safer Skies Initiative Has Taken Initial Steps to Reduce  Accident Rates by 2007. GAO/RCED-00-111. Washington, D.C.: June 30,  2000.
Aviation Safety: FAA’s New Inspection System Offers Promise, but  Problems Need to Be Addressed. GAO/RCED-99-183. Washington, D.C.:  June 28, 1999.
Aviation Safety: Weaknesses in Inspection and Enforcement Limit FAA  in Identifying and Responding to Risks. GAO/RCED-98-6. Washington,  D.C.: February 27, 1998.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. It may be reproduced and distributed in its entirety without further  permission from GAO. However, because this work may contain copyrighted images or  other material, permission from the copyright holder may be necessary if you wish to  reproduce this material separately.