Aviation Safety: System Safety Approach Needs Further Integration into FAA's Oversight of Airlines
Published: 2005-09-28T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Federal Aviation Administration (FAA) uses the Air Transportation Oversight System (ATOS), which was developed around the principles of system safety, to oversee seven "legacy airlines" and nine other airlines. In this report, we refer to airlines that are not in ATOS as non-legacy airlines. Two other processes are used to oversee 99 non-legacy passenger airlines, which represent a fast-growing segment of the commercial aviation passenger industry and carried about 200 million passengers in 2004. The National Work Program Guidelines (NPG) establishes a set of inspection activities for non-legacy airlines. The Surveillance and Evaluation Program (SEP) uses principles of system safety to identify additional risk-based inspections for those airlines. GAO's objective was to assess the processes used by FAA to ensure the safety of non-legacy passenger airlines. GAO reviewed the strengths of FAA's inspection oversight for non-legacy passenger airlines and the issues that hinder its effectiveness.
section_title: What GAO Found
paragraphs: A key strength of FAA's inspection oversight of non-legacy airlines is the introduction of system safety concepts to some inspections, which FAA accomplished by adding SEP to its traditional inspection process, NPG. Although NPG has risk-based elements, it lacks the structured approach to risk identification found in SEP. Under SEP, data are used to help determine trends or problems. The SEP process uses a team of inspectors to identify inspection activities, which we have previously reported is generally more effective than the use of individuals due to their collective ability to identify risks. Under SEP, inspectors also ascertain risks internal to FAA, such as staffing shortages. FAA's oversight of non-legacy airlines further incorporates processes to ensure that inspectors follow up on airline actions taken in response to inspection findings. These efforts address several past GAO concerns, including that NPG did not allow FAA to identify risks and allocate inspection resources accordingly. The full potential of FAA's inspection program for non-legacy airlines, however, is not being realized due to incomplete implementation of its system safety approach and other challenges. The inspection workload is still heavily oriented to nonrisk-based activities, with 77 percent of inspection activities being identified through the NPG and the remaining relatively small percentage identified through SEP. The emphasis on NPG, including FAA's guidance that inspectors must complete NPG-required inspection activities, acts as a disincentive to identifying further inspection activities through SEP. Inspectors face workload challenges as staff lost through attrition may not be replaced due to a hiring freeze. FAA estimates that over 1,100 inspectors of non-legacy airlines will leave the agency in fiscal years 2005 to 2010. In addition, some FAA inspectors indicated that a lack of technical training on airline systems and equipment posed potential risks to the agency's oversight process. Finally, FAA lacks a process to communicate information to inspectors on how certain internal risks identified through SEP are being resolved. Moreover, FAA has not established a process to evaluate the effectiveness of SEP.

section_title: Letter
paragraphs: The U.S. commercial aviation industry, with less than one fatal accident per  5 million flights from 2002 through 2004, has an extraordinary safety  record. However, when passenger airlines have accidents or serious  incidents, regardless of their rarity, the consequences can be tragic. In  order to maintain a high level of aviation safety, it is critical to have well- established, efficient, and effective systems in place to provide an early  warning of hazards that can lead to accidents. The Federal Aviation  Administration (FAA) has established a number of systems and processes  to inspect and oversee various aspects of passenger airline safety, such as  aircraft maintenance and flight operations. About 585 of FAA’s  approximately 3,200 inspectors are dedicated to overseeing the largest  commercial passenger airlines, including the seven “legacy airlines,” and  nine other airlines through FAA’s Air Transportation Oversight System  (ATOS), which was developed around the principles of system safety.  System safety involves the continual evaluation of all of an airline’s  operations for the purpose of identifying and mitigating risks.  Approximately 1,100 inspectors oversee other entities and individuals,  including 99 smaller commercial passenger airlines—which we refer to as  non-legacy passenger airlines—about 5,200 aircraft repair stations, and  approximately 625,000 pilots. Non-legacy passenger airlines, a fast-growing  segment of the commercial aviation passenger industry, carried about 200  million passengers in 2004.
FAA’s inspection process for airlines not covered by ATOS has two  components.  The National Work Program Guidelines (NPG) is the baseline  component of the oversight program for these airlines. In 2002, FAA added  another component, the Surveillance and Evaluation Program (SEP), to the  inspection process to incorporate principles of ATOS into its oversight of  non-legacy passenger airlines. The two components are used together to  establish the number of annual inspections for non-legacy airlines.  Inspections can encompass many different activities, such as visually spot- checking an airplane at a gate, monitoring procedures on a scheduled  flight, or observing maintenance being performed on an aircraft. Each year,  FAA headquarters establishes baseline inspections for each airline through  NPG, while through SEP, teams of FAA inspectors analyze the results of an  airline’s prior inspections at periodic meetings and, based on their  assessment of specific risks, establish other inspections that may be  needed.
In response to your request, we assessed FAA’s processes for ensuring the  safety of non-legacy passenger airlines. Specifically, we addressed the  following questions: (1) What are the strengths of FAA’s inspection  approach for non-legacy passenger airlines? and (2) What issues hinder the  effectiveness of FAA’s inspection approach?
To address these questions, we obtained and analyzed information from a  variety of sources. We examined FAA documents about SEP, NPG, and  ATOS. We also reviewed prior reports prepared by us and others on SEP  and NPG to determine significant issues involving those programs and how  they were resolved. In addition, we surveyed a statistical sample of FAA  safety inspectors to obtain their views about the training they receive. We  had no practical way to assess information on the amount of training  necessary for inspector proficiency or the timeliness of the training  provided. We conducted semistructured interviews and analyzed relevant  documents from FAA headquarters officials, and from field managers and  inspectors in 7 regional and 13 field offices, which were selected because  they oversee the top 25 non-legacy airlines ranked by the number of  enplanements in 2004. At these locations, we collected information on the  inspection process and inspector staffing levels, workload, and training. In  addition, we conducted semistructured interviews with safety officials at  16 of those top 25 non-legacy airlines, and with officials at seven industry  organizations that represent airlines, inspectors, pilots, mechanics, and  maintenance facilities. We analyzed data on required NPG and SEP  inspections for fiscal years 2002 through 2004 from FAA’s nationwide  inspection database—the Program Tracking and Reporting Subsystem  (PTRS). We assessed the reliability of the database and found the data  sufficiently reliable for the types of analyses that we conducted for this  report. We also tested for the presence of several management controls,  including the processes for verifying inspection results, establishing a  process to evaluate FAA’s inspection oversight process, managing the PTRS  database, and communicating among managers and inspectors relating to  NPG and SEP. In addition, we reviewed literature on system safety and  compared FAA’s system safety framework with that presented in the  literature. We conducted our work from August 2004 through September  2005 in accordance with generally accepted government auditing  standards. Additional information on our methodology is found in  appendix I.
subsections: section_title: Background
paragraphs: Non-legacy airlines represent a fast-growing segment of the passenger  airline industry. From 2002 through 2004, the annual enplanements for  these airlines grew from 122 million to about 200 million passengers, or  from about 20 percent to 28 percent of all passenger air travel. During this  same period, the percentage of people flying on legacy airlines declined  from 80 percent to about 72 percent of all passengers flown (see fig. 1).
subsections: section_title: FAA’s Inspector Workforce Carries Out Oversight
paragraphs: FAA’s safety oversight of non-legacy airlines is carried out by inspectors  located at 109 field offices throughout the world that are part of 9 regional  offices. For each airline, FAA puts together a team led by principal  inspectors who maintain primary responsibility for managing the airline’s  certificate requirements and focus on one of three disciplines: avionics,  maintenance, or operations. Additional team members include those based  at the FAA office that holds the airline’s operating certificate—typically an  aircraft dispatch inspector, a cabin safety inspector, and assistants. FAA  locates its principal inspectors close to their respective airlines’ primary  operational base. For example, the principal inspectors for Independence  Air (formerly Atlantic Coast Airlines) are located at FAA’s field office at  Dulles International Airport, where the airline has its headquarters. In  addition, FAA has geographic inspectors based at the 109 field offices to  conduct additional inspections. Rather than being designated to particular  airlines, geographic inspectors may conduct inspections of aircraft of any  non-legacy airlines that land in their area.
FAA’s safety inspector workforce for all commercial airlines has remained  steady, averaging about 1,780 inspectors over fiscal years 2002 through  2004. FAA cannot determine how many of these inspectors were assigned  to non-legacy passenger airlines during those 3 years, because the agency  did not collect that information, according to an FAA headquarters’ official.  FAA expects the number of inspectors assigned to non-legacy airlines to  remain steady for fiscal years 2005 and 2006 and decline slightly in fiscal  year 2007, as shown in figure 2. Inspectors record information about these  inspections in PTRS—a nationwide computerized database that maintains  such information as inspector findings and airline activities in response to  the findings.
section_title: FAA Uses NPG and SEP to Oversee Non-legacy Airlines
paragraphs: NPG and SEP are the main inspection processes that FAA uses to oversee  the safety of non-legacy airlines. Since 1985, FAA has used NPG, which  includes both required and planned inspections, as its primary means of  ensuring that airlines comply with safety regulations. In NPG, an FAA  committee of program managers identifies an annual minimum set of  required inspections that are to be undertaken to ensure that airlines are in  compliance with their operating certificates. In addition, inspectors  determine annual sets of planned inspections based on their knowledge  and experience with the particular airlines they oversee. Typically,  inspections would include ramp inspections, in which inspectors examine  an aircraft while it is parked at the airport, and maintenance inspections.  However, we found problems with NPG throughout the 1990s, including:  (1) FAA’s routine inspections were ineffective in identifying serious safety  problems, (2) critical airline inspections had not been conducted, (3) FAA’s  follow-up actions often did not ensure that problems were corrected once  identified, and (4) FAA did not have a methodology for estimating airline  safety risks so that it could target limited inspection resources to high-risk  inspections. We also found that FAA’s inspection database was of limited  use in providing early warning of potential risks or targeting inspection  resources.
In response to these findings, and in the aftermath of the 1996 ValuJet  crash, an FAA task force reviewed the agency’s safety inspection process  and recommended in part that the agency initiate a project to make  surveillance of airlines more systematic and targeted to deal with identified  risks. This recommendation resulted in the agency’s development and  implementation of ATOS in 1998 at the nation’s 10 largest commercial  passenger airlines, with the goal of eventually including all commercial  passenger and cargo airlines. ATOS emphasizes a system safety approach  that extends beyond periodically checking airlines for compliance with  regulations to the use of technical and managerial skills to identify, analyze,  and control hazards and risks. The goal of ATOS is to identify safety  trends in order to spot and correct problems at their root cause before an  accident occurs. This program allows FAA inspectors to look at an airline  as a whole, to see how the many elements of its operations, including  aircraft, pilots, maintenance facilities, flight operations, and cabin safety,  interact to meet federal standards. Collectively, the airlines under ATOS  had a dedicated inspector staff of 585 inspectors as of July 2005. The  number assigned to each airline depends on the size of the airline’s  operations. ATOS uses special checklists and databases that are intended  to cover all areas of airline operations. Part of FAA’s oversight is expected  to include an in-depth look at an airline’s policies and procedures and  whether the airline is following them. There are 16 airlines in the ATOS  program, including two cargo carriers as of September 2005. Due to  resource constraints, FAA determined it would not be able to immediately  place the remaining passenger airlines in the ATOS program. FAA  developed SEP as a bridge to introduce safety risk concepts used in ATOS  into the oversight process for non-legacy airlines in order to facilitate the  ultimate transition of these airlines to ATOS. As of September 2005, FAA  estimates that it will move the remaining non-legacy airlines to ATOS by the  end of fiscal year 2007.
section_title: SEP Implemented as a Transition to ATOS
paragraphs: Since the introduction of ATOS, FAA has been moving toward integrating  system safety into its oversight activities. However, the agency has been  delayed in placing a significant number of non-legacy airlines in the ATOS  program, resulting in those airlines continuing to be overseen through  NPG, a process that is not system safety oriented.  FAA agrees that NPG is  not a system safety process, but explained that the planned NPG  inspections, which inspectors identify based on their expertise, provide a  risk-based element to the process. To address the delay in moving airlines  to ATOS, FAA, in 2002, added SEP to NPG as a way to introduce airlines to  a system safety oversight process until those airlines were transitioned to  ATOS. Although originally envisioned by FAA as a transitional program  for airlines that were awaiting placement in ATOS, SEP has become a more  permanent oversight process as budget and staff constraints have  prevented the agency from moving all airlines to ATOS.
SEP is complementary to NPG and permits the risk-based, data-driven  alteration of required NPG inspections. SEP provides a formal structure to  risk identification that is absent from the NPG process. Under SEP, the  principal FAA inspectors for each airline meet periodically during the year  to discuss the results of their inspections and identify risks using the same risk assessment principles and checklists used in ATOS. An outcome of  the meetings is the identification of inspections to augment the NPG  baseline of required inspections, which FAA views as the minimum number  of inspections that need to occur to ensure that certain areas of all airlines  are reviewed. During the SEP process, FAA headquarters allows principal  inspectors to replace NPG-required inspections with inspections targeting  higher-risk areas identified through the SEP meetings. This process is  known as retargeting. At the end of the meeting, entries are created in the  PTRS database that indicate all inspections to be performed for that airline.  SEP-initiated inspections that are designated as priorities are required to be  completed. See appendix II for additional information on SEP, ATOS, and  NPG.
section_title: FAA Also Uses Industry Partnership Programs to Provide Safety Oversight of Airlines
paragraphs: FAA also oversees the safety of both legacy and non-legacy passenger  airlines through participation in industry partnership programs. Two of  these programs—the Aviation Safety Action Program and Voluntary  Disclosure Reporting Program—encourage certain airline employees, such  as pilots and mechanics, or airlines to voluntarily report safety information  that might be critical to identifying potential precursors to accidents  without fear that FAA or their companies will use reports accepted under  the programs to take legal enforcement or disciplinary actions against  them. The Aviation Safety Action Program provides for the voluntary self- reporting of safety incidents under procedures set out in memorandums of  understanding between FAA, airlines, and participating employee groups  such as aircraft mechanics. As of June 2005, 20 of the top 25 non-legacy  airlines (in terms of the number of enplanements) were participating in the  Aviation Safety Action Program. The Voluntary Disclosure Reporting  Program allows airlines to voluntarily report safety incidents to FAA.  However, our prior work found that FAA has not analyzed violation data  derived from these two programs to monitor national trends in airline  operations, so that it can target resources to address operational risks. A  third program, the Internal Evaluation Program, requires airlines to  continuously monitor and evaluate their practices and procedures.
section_title: FAA’s Oversight Process Uses System Safety, Which Is Designed to Identify and Control Risks and Improve Resource Utilization
paragraphs: The incorporation of system safety into FAA’s oversight presents a shift in  concept from the agency’s traditional oversight method, which relied upon  periodic inspections, to an approach that allows for the prioritization of  inspections based on areas of highest risks. The development of SEP, a key  element in FAA’s system safety oversight process, allows for the efficient  use of inspection staff and resources by incorporating this risk-based  approach. FAA’s incorporation of SEP into its oversight addresses a past  concern by us that the agency was not prioritizing inspectors’ workload.  Our review of literature by government and private organizations  supported FAA’s concept of system safety because it makes apparent the  risks that are the basis for changes. In addition, FAA’s oversight process  includes a requirement that inspectors verify that corrective actions have  occurred, providing a level of assurance that the safety problem has been  mitigated.
subsections: section_title: FAA Has Provided System Safety Training for Inspectors
paragraphs: To facilitate the implementation of a system safety approach, FAA has  made an effort to train its staff in system safety, which is a strength of its  oversight approach. FAA has recognized that inspectors need training in  the system safety concept in order to effectively incorporate this approach  into airline oversight. Specifically, FAA believed that its inspectors have  gaps in their needed competencies and skills that could affect the agency’s  system safety approach to inspections. Among the largest training gaps for  inspectors, according to FAA, were training in risk analyses and systems  thinking. As a result of these findings, FAA has undertaken training  activities to improve inspector competencies in system safety. According  to FAA, between fiscal years 2002 and 2004, it provided basic system safety  training to almost 3,675 staff, including non-legacy and ATOS inspectors.
section_title: SEP Incorporates System Safety by Focusing on the Identification and Prioritization of Inspections Based on Risks
paragraphs: By incorporating risk assessment into the inspection process, SEP  addresses a long-standing problem. Since the mid-1980s, our reports have  shown that FAA did not have a methodology for assessing airline safety  risks so that it could target limited inspection resources to high-risk  conditions. SEP’s system safety concept of reducing risk through the  identification, analysis, and control of hazards is also consistent with the  views presented in literature of other government and private  organizations, such as the Department of Defense, Canada’s civil aviation  authority (Transport Canada), and the Flight Safety Foundation. For  example, DOD looks at system safety as a means of reducing risk through  early identification, analysis, elimination, and control of hazards. For the  airlines it oversees, Transport Canada requests that each airline  incorporate system safety into its operations by integrating safety into its  policies, management and employee practices, and operating procedures.  FAA incorporates such concepts in its system safety oversight approach.
SEP’s system safety process has a number of strengths. At periodic  meetings, the principal inspectors of an airline identify potential risks that  they believe should be addressed. These inspectors consider a variety of  information that they have obtained through personal observation and  from reports prepared by geographic inspectors and located in the PTRS  database. A strength of this approach, consistent with findings in our past  reports, is that teams of inspectors are generally more effective than  individual inspectors in their ability to collectively identify concerns.  Another strength is that FAA has developed risk assessment worksheets  aligned with key airline systems that guide inspectors through identifying  and prioritizing risks. The worksheets guide inspectors to organize the  results of their previous inspections and surveillance into a number of  areas, such as flight operations, personnel training, and cabin safety, in  order to identify specific risks in each area and target the office’s resources  to mitigating those risks. (See app. II for more details on the risk  assessment process.)
section_title: SEP Also Allows for the Identification of Internal Risks That Might Hinder Oversight
paragraphs: During their periodic meetings, principal inspectors also have the  opportunity to identify risks in FAA’s internal operations that could  adversely affect the inspectors’ ability to conduct safety oversight of non- legacy airlines, which is another strength of FAA’s oversight approach. FAA  inspectors and managers agreed that identifying internal risks is helpful, as  they considered it an efficient and effective way of identifying needed  program improvements. Inspectors can identify deficiencies in a number of  categories, including inspector staffing, training, availability of geographic  inspectors, and resources such as travel funds. In addition, the inspectors  can quantify the degree of the risk that each of these categories poses.  Being able to identify and make officials aware of situations that can hinder  an agency’s ability to perform its mission is a key management control.
This process has resulted in FAA headquarters receiving significant  information on areas that inspectors believe are deficient. For fiscal years  2002 through 2004, inspectors identified about 560 risks that, in their  opinion, could have an impact on how FAA managed the inspection  process of airlines, as shown in table 1. Inadequate training (aircraft- specific training and other training) was the most frequently identified  concern of the inspectors, accounting for about 40 percent of the total risks  identified during the 3-year period—27 percent of the risks were related to  inadequate or nonexistent aircraft-specific training and 15 percent were  related to the lack of other training. We discuss this issue in detail later in  this report. The total number of risks identified by FAA inspectors declined  over the 3 years. Inspectors from two FAA field offices opined that the  decline may be due to better staff understanding of risk identification and  to the risks being addressed expeditiously so that inspectors did not need  to restate them the following year.
The process not only allows inspectors to identify the risk, but also to  describe the severity of the risk in order to assist FAA headquarters in  prioritizing its consideration of internal risks. Inspectors rate each risk they  identify in terms of severity (negligible to catastrophic) and likelihood of  occurrence (improbable to frequent). Risks are then categorized using a  scale of high, medium, or low depending on the severity and likelihood of  occurrences. For example, risks rated high are those that have high levels  of likelihood and severity. As shown in table 2, over a 3-year period, 125 out  of 557 risks were rated as high severity, which could help FAA determine  which issues most need targeting. Among the risks rated as high, training  and inadequate staffing were identified most frequently, accounting for 34  percent and 31 percent, respectively. As of May 2005, about 87 percent of  identified risks had been closed, including about 90 percent of the risks  identified as high. Risks are closed either because action has been taken to  mitigate the risk or FAA has determined that the level of risk is acceptable  and no action is warranted.
An additional strength of the internal risk identification process is that FAA  has established a chain of command for responding to these internal risks  at the appropriate level. For example, local training issues are likely to be  handled by field offices, geographic support issues would normally be  handled by regional offices, and issues that cannot be resolved by these  field and regional offices would be referred to headquarters.
section_title: FAA’s Oversight Process Includes Monitoring to Follow Up on Airline Actions Taken in Response to Findings
paragraphs: Another strength of FAA’s oversight of non-legacy airlines is that inspectors  monitor the actions that those airlines have taken in response to inspection  findings through subsequent inspections and participation in safety  partnership programs with the airlines. However, the inspectors’  monitoring process could be improved by better data, as we discuss later in  this report. SEP requires inspectors to monitor the actions taken by non- legacy airlines that will address the risks identified during an inspection  and to verify that the actions taken have resulted or will result in resolution  of the problems. This requirement addresses a concern of ours that the  NPG inspection program did not often ensure that the agency follow-up  actions corrected problems once they were identified. In addition, FAA’s  requirement for monitoring and verifying corrective action conforms to a  management control standard on the need for federal agencies to have  monitoring and verification policies or procedures for ensuring that the  findings of audits and other reviews are promptly resolved. FAA inspectors  monitor and confirm that corrective action has been taken by either  conducting a special inspection of the area, following up during a  subsequent inspection, or reviewing airline documentation that addresses  the deficiency. According to inspectors and field office managers, the  verification method used is based in many instances on the severity of the  deficiency, with special inspections conducted for areas that could affect  air safety.
According to some FAA inspectors and airline officials, the regular  meetings held to discuss concerns about airlines have opened up  communications both within FAA and between FAA and the airlines. Lack  of communication among the airlines and FAA was, according to the  director of FAA’s Flight Standards Service, historically a significant cause  of airline safety problems. FAA inspectors and managers noted that they  maintain an open line of communication with airline management by  meeting regularly with key airline officials (such as directors of operations  and directors of maintenance) to discuss the results of the SEP risk  analyses and the airline’s proposed resolutions.
section_title: FAA’s Oversight Process Is Hindered by Certain Program Weaknesses and Human Capital Management Challenges
paragraphs: FAA’s oversight of non-legacy airlines is hindered by the incomplete  implementation of its system safety approach and other challenges. FAA’s  incorporation of system safety into its oversight process is incomplete, as  the agency continues to emphasize a nonsafety system-based process,  NPG, to identify most of its inspection activities. Human capital  management challenges also impede FAA’s oversight. These challenges  include a hiring freeze that began in January 2005; attrition; in certain  cases, the reallocation of inspectors’ duties as airlines are moved to the  ATOS program; insufficient training for inspectors on risk management, a  key element of SEP; and insufficient communication between FAA  headquarters and field offices. In addition, FAA lacks a process to  continuously evaluate SEP. Finally, the agency’s PTRS database lacks some  important inspection-related information, such as whether risks have been  mitigated, that would aid in targeting further oversight activities as well as  a nationwide analysis of inspection results.
section_title: Most Inspection Activities Are Not Prioritized Based on a Structured Risk Assessment Process
paragraphs: We found that NPG remains the primary basis for FAA’s inspection of non- legacy airlines. For fiscal years 2002 through 2004, about 77 percent of the  required inspection activities for the top 25 non-legacy airlines were  initiated through the NPG annual planning session, compared with 23  percent that were SEP-initiated. As a result, most of the required inspection  activities are not prioritized based on risk. (See table 3.) In addition, the  total number of required inspections for non-legacy airlines declined during  the 3-year period, while the number of passengers on those airlines grew  from 122 million to 200 million annual enplanements. The decline in  inspections during this period may be due, in part, to the movement of  three airlines from SEP to ATOS. FAA headquarters officials did not know  why this decline occurred, but told us that it may be due to agency efforts  to eliminate unnecessary activities and shift from a “quality control”  oversight approach, in which individual products or outputs are inspected  to determine if they meet specifications, to a “quality assurance” approach,  in which airlines’ processes are inspected. While NPG also includes  planned inspections, which have a risk-based element, these inspections  lack the structured approach to risk identification that SEP has. FAA  officials acknowledged that the NPG planned inspections would benefit  from the use of risk management tools.
According to FAA officials, there are no minimum numbers or percentages  of required SEP inspections. Therefore, a large percentage of NPG-initiated  activities is not unexpected or negative in their view. FAA officials said  NPG-initiated activities are an important part of the inspection process  because they provide information to the agency on an annual basis for  certain types of inspections and enable periodic assessments of key  programs in an airline’s operations. According to FAA, NPG is organized to  systematically validate on an annual basis the performance of all of an  airline’s safety-critical programs. FAA considers this type of recurring  assessment to be an important system safety principle and believes that it  provides an opportunity to discover and correct latent failures before they  cause safety problems. In addition, according to FAA officials, NPG  identifies a minimum number of inspections that they believe need to occur  to ensure that certain areas of an airline are reviewed, unlike SEP, which  does not provide the built-in assurance that these areas will be inspected  eventually.
We agree with FAA officials that the agency should not establish minimum  numbers or expected percentages of SEP inspections. However, we found  that the large percentage of required NPG activities compared with SEP  activities occurs, in part, because FAA’s guidance places a greater emphasis  on completing required NPG activities. This situation, along with other  factors we discuss below, may deter inspectors from identifying additional  inspections through SEP. FAA’s guidance includes the following:  Field offices must complete 100 percent of the required NPG-identified  activities by the end of each fiscal year (unless they are retargeted).
Field offices are not required to complete SEP-initiated activities by the  end of the fiscal year and may reschedule them to the following fiscal  year.
In addition, the budget given to field offices for inspections is based on  required NPG-initiated activities, and funding to cover SEP-initiated  activities must come out of that budget. Inspectors have indicated concern  regarding resources for inspections; for example, a lack of resources,  including a lack of travel funds to conduct inspection activities, made up  about 10 percent of the internal risks identified by inspectors through SEP  during fiscal years 2002 through 2004. According to headquarters officials,  requests for funding high-risk SEP-identified inspections are always  approved; however, some noncritical activities may be delayed due to  budget constraints. For example, they said that some certification work— such as certifying that new aircraft or aircraft parts meet FAA standards— is being delayed until an inspector travels to the airline to perform another  activity and can do the certification at little or no extra cost.
Under FAA’s guidance, if inspectors identify a risk under SEP that they wish  to mitigate through an additional inspection activity, they must either add  the SEP-initiated activity to the list of required NPG-initiated activities or  retarget a required NPG activity and replace it with the SEP-initiated  activity. Adding SEP-initiated activities to the required NPG-identified  activities may increase the overall workload of inspectors. This situation  may affect the number of additional activities that inspectors identify  during the SEP meetings. In fact, we found that in some cases, no SEP- initiated activities were included with NPG-initiated activities as part of an  airline’s overall required inspections. For example, for one airline overseen  by FAA’s Central Region, PTRS data show that, in fiscal year 2002,  inspectors conducted about 400 required NPG-initiated activities and no  SEP-initiated activities. According to FAA headquarters officials, many  inspectors view SEP-initiated activities as additional work. This view was  also held by some inspectors we spoke with.
Moreover, our analysis of PTRS data indicates that retargeting occurs  infrequently. In fiscal year 2002, about 5 percent of all NPG-required  inspections were retargeted by inspectors and 3 percent were retargeted in  fiscal years 2003 and 2004. This low rate occurs, in part, because field  offices have interpreted FAA’s emphasis on NPG inspections as  discouraging retargeting. For example, three of the principal inspection  teams for the 16 airlines told us they do not retarget any NPG activities.  Further, in 5 of the 13 field offices we visited, we were told by both  managers and inspectors that they believe NPG activities take priority over  all but the most risky situations identified through SEP. Similarly, an official  with the Professional Airways Systems Specialists, a union that represents  many FAA safety inspectors, told us that its members believe that FAA’s  management does not give as much attention and priority to SEP as it does  to ATOS.
FAA’s emphasis on required NPG-initiated inspections is also shown  through its data-tracking efforts. According to FAA officials, for fiscal years  2002 through 2004, all required NPG activities had been completed by the  end of the year. FAA did not know how many completed SEP-initiated  activities were mitigated or required further inspection activities that were  then carried over to the next fiscal year because it does not collect such  information nationwide in PTRS. Moreover, field office managers also rely  on PTRS, which lacks information on the number of activities that are  carried over. For example, one field office supervisor told us that one  inspector had carried over 19 activities from fiscal year 2004 to fiscal year  2005. However, the field office supervisor said it was difficult to determine  how many additional activities were carried over because PTRS does not  track this information.
subsections: section_title: Shifts in Inspectors’ Workload Present a Challenge to FAA’s Inspection Oversight Process
paragraphs: The number of FAA inspectors available to oversee non-legacy airlines will  be affected by recent and anticipated trends in attrition and a hiring freeze.  First, for fiscal year 2004, the number of inspectors (those assigned to non- legacy airlines, ATOS, and general aviation) who left FAA was greater than  the number of inspectors hired, resulting in a net loss of 67 inspectors.  Second, the number of inspectors of commercial airlines (both ATOS and  non-legacy) is expected to continue to decline from 1,810 at the end of  fiscal year 2004 to 1,704 at the end of fiscal year 2005. Third, from fiscal  years 2006 to 2007, the agency anticipates the number of inspectors of non- legacy airlines will decline from 1,119 to 1,088. Fourth, for fiscal years 2005  through 2010, FAA estimated that over 1,100 inspectors of non-legacy  airlines and general aviation will leave the agency, with an average loss due  to attrition of about 195 inspectors per year.
FAA will have to hire inspectors to offset the anticipated losses. However,  the agency put a hiring freeze in place in January 2005 for budgetary  reasons, which prevents the agency from hiring inspectors to replace those  lost due to attrition unless they are deemed critical.  Furthermore, in order  to absorb a budget reduction in fiscal year 2005, FAA expects to eliminate  about 200 inspector positions in the Office of Aviation Safety. These  reductions will be accomplished through attrition. In fiscal year 2006, the  office expects the staffing level to increase by about 80, which would still  be below the fiscal year 2004 staffing level. FAA is filling safety-critical  positions that become vacant through internal appointments. According to  FAA officials, principal inspectors and managers are considered safety  critical. As other safety inspectors, including geographic inspectors, leave  the agency or are appointed to safety-critical positions, they are not being  replaced at this time, according to those officials. Moreover, their workload  is being divided among the remaining inspectors, thereby increasing the  inspectors’ workload, which has resulted in other, less critical, work being  delayed or deferred. In 11 of the 13 FAA field offices we contacted, officials  indicated a shortage of different types of inspectors needed to oversee the  non-legacy airlines. For example, the Washington, D.C., field office did not  have a principal avionics inspector. Officials with FAA field and regional  offices said they have developed ways to deal with the inadequate staffing  of inspectors in their offices. For example, the Southern Regional Office  told us that it uses geographic inspectors more frequently than in the past  to help conduct inspections rather than having the inspectors assigned to  the non-legacy airline travel to conduct them. However, two field offices—- Chicago and Minneapolis—-indicated the need for more geographic  inspectors. Another way that FAA offices are dealing with inspector  shortages is by having an inspector perform inspections for several offices.  For example, the Atlanta field office has been without a cabin safety  inspector for a year and uses the inspector assigned to the Southern  Regional Office. According to an FAA inspector, some inspections are  delayed due to lack of staff availability. In addition, according to our  analysis of FAA data (discussed earlier in this report), for fiscal years 2002  through 2004, principal inspectors identified lack of staff for the airlines’  certificate management teams as among the top two risks for each year,  and overall, more than one-third of the risks identified were related to lack  of inspectors.
The move of additional airlines into ATOS, which requires more inspectors  per airline than SEP, also affects the workload of inspectors for non-legacy  airlines. In addition, unlike SEP inspectors, ATOS inspectors are dedicated  to an airline and generally cannot be used to conduct inspections of other  entities. When four airlines were recently transitioned into ATOS, the total  size of the four inspection teams increased 30 percent, from 73 to 95  inspectors, as shown in table 4. As a result, the number of inspectors  available to oversee non-legacy airlines was reduced not only by those 73  who had covered the four airlines while they were under the NPG and SEP  processes (who would have also had responsibilities for other entities) but  by an additional 22 inspectors as well.
FAA is aware that transitioning additional inspectors to airlines in ATOS is  putting a burden on the workload of those inspectors who continue to  inspect non-legacy airlines, and the agency has, in one case, revised its  procedures for dedicating staff to individual ATOS airlines. When  Champion Airlines became part of ATOS in January 2005, FAA changed  ATOS policy to allow the Northwest Airlines inspection team to share its  data analyst and manager with the Champion inspection team. FAA  officials believe that sharing staff resources will address the issue of staff  shortages.
All of the inspectors at the offices we contacted had other duties in  addition to overseeing non-legacy airlines. The inspectors performed  activities such as certifying and approving new aircraft types; overseeing  repair stations and aviation schools; reviewing FAA directives and aircraft  updates; responding to complaints; investigating accidents, incidents, and  complaints; overseeing designees; and attending training. Although  primarily responsible for a particular airline, some principal inspectors  conducted inspections of other airlines and entities. For example, a  principal inspector in the Great Lakes Region was responsible for  overseeing three repair stations in addition to a non-legacy airline. In  addition, a principal operations inspector in the Southern Region was also  responsible for overseeing designees such as pilot examiners. Our analysis  of the inspectors’ workload contained in the PTRS database showed that  for fiscal years 2002 through 2004, about 75 percent of FAA inspectors had  responsibility for more than 3 entities, and about half had responsibility for  more than 15. While the size and complexity of the various entities will  dictate the inspector workload, having responsibility for a large number of  entities can increase an inspector’s workload because of the need to be  familiar with the operating procedures of each entity as well as to spend  time physically inspecting each entity.
section_title: Inspectors Are Concerned That Training Limitations Present a Challenge to Inspection Oversight Process
paragraphs: While FAA provided system safety training to its inspector workforce, as  mentioned earlier in this report, the lack of sufficient training in other areas  was identified by principal inspectors during their risk analysis meetings  and by inspectors we spoke with. Our analysis of FAA’s database of  internal risks identified by principal inspectors of non-legacy airlines  indicated that 42 percent (233 out of 558) of the internal risks identified for  fiscal years 2002 through 2004 related to training. Although FAA policy  requires that principal inspectors assigned to non-legacy airlines complete  technical training on the airline’s specific policies, procedures, and  equipment before they can conduct airline inspection activities, principal  inspectors identified risks associated with the lack of such training. For  example, our analysis of the internal risk database found that principal  inspectors identified a lack of technical training for themselves or other  inspectors on their team on specific aircraft, engine types, navigation  equipment, and avionics that pertained to the airlines they were  overseeing. Others cited a lack of training in cargo loading, hazardous  material identification, or weight and balance. Still others indicated the  need for training in FAA’s systems or processes. For example, several  indicated that they had not received the basic courses for principal  inspectors (such as “foundation for principal inspectors”). As another  example, several principal inspectors reported that certain inspectors on  their team lacked training in using the computer system that analyzes  information collected in PTRS, noting that without this training they were  unable to directly access the system. Even though inspectors expressed  these concerns, approximately 87 percent of identified internal risks have  been closed by FAA, according to our analysis of FAA data. However, we  could not determine how many of those risks were actually mitigated, as  FAA closes risks either when action has been taken to mitigate the risk or  because the agency has determined that the level of risk is acceptable and  no action is warranted.
During our field office visits, inspectors also spoke about the lack of  technical training in certain areas. For example, inspectors told us they had  not received training on some newer aircraft they were responsible for  inspecting and that they were not always able to attend free training  provided by the manufacturers, which they felt would help them obtain  needed technical knowledge. We found instances where management  acknowledged that such training was needed. In the past, concerns had  been raised that FAA’s acceptance of free training from a regulated entity  might have the appearance of a conflict of interest. In September 2005, we  reported that FAA accepts such training under limited circumstances.  FAA has established safeguards to help preclude the appearance of a conflict of interest when FAA accepts training from the entities it regulates  at no cost to the agency or in exchange for an in-kind service.
In addition, the principal inspectors we spoke with indicated that some  geographic inspectors who assist them in providing oversight of non-legacy  airlines lack the technical and SEP-specific procedural training to  maximize their usefulness. Several field office managers and principal  inspectors told us that geographic inspectors needed to be knowledgeable  about the operations of airlines they inspect in order to provide effective  assistance. Several principal inspectors indicated that misperceptions  about an airline’s operating requirements are not uncommon because  geographic inspectors often lack training on the aircraft they inspect. As a  result, the inspectors indicated that geographic inspectors had at times  incorrectly coded inspection information in PTRS and erroneously  identified risks, which resulted in principal inspectors having to spend time  determining that a problem did not actually exist. In addition, several  airline officials told us that geographic inspectors had incorrectly identified  problems and appeared to not understand airlines’ operational procedures.  For example, we were told that a geographic inspector improperly cited an  airline for not complying with a deicing program that the airline had  previously informed FAA inspectors it was changing. FAA headquarters  officials told us that it is not feasible to train all geographic inspectors on  all airlines they may encounter. Further, they noted it would be good for  geographic inspectors to contact principal inspectors when they are unsure  of whether an action or condition they observe is a problem. However, FAA  headquarters has not provided that instruction to the geographic  inspectors. Industry representatives also told us this was a problem that  could be alleviated by inspectors having access to airlines’ operating  manuals online.
Another issue identified as a problem by principal inspectors we spoke  with is that not all geographic inspectors are provided training in SEP,  including how to use the uniform coding scheme that was developed for  SEP to facilitate the identification and analysis of risks. Geographic  inspectors may therefore note the results of their inspections through  narrative in the comments section of PTRS rather than through SEP codes,  so that the principal inspectors must read through all the comments in  order to identify geographic inspectors’ concerns. As PTRS may contain  thousands of entries on one airline each year, reading every narrative  comment can be time consuming and difficult to interpret. The lack of  coding of geographic inspectors’ findings could therefore result in principal  inspectors missing problems or risks identified by geographic inspectors.  We do not know how often this problem occurs, as FAA has not assessed  the reliability of this data in PTRS. However, several FAA staff brought this  issue to our attention. One field office manager and SEP coordinator told  us that PTRS entries by geographic inspectors are not helpful because they  do not provide meaningful comments for FAA’s risk assessments, and  several principal inspectors told us that training for geographic inspectors  on SEP coding would help them more easily identify risks during their  periodic meetings.
These training issues may arise in part because FAA has not systematically  identified technical training needs for principal or other inspectors. Our  previous report found that FAA has not systematically identified overall  training needs of its inspectors to ensure that the curriculum addresses the  unique training needs of each type of inspector. Instead, course  development focuses on individual courses. FAA said it recognizes that it  manages courses as individual components of an overall curriculum that is  only loosely defined and that it needs to develop courses and address  training needs as part of an overall curriculum. It has established a  curriculum transformation plan that it estimates it will fully implement in  2008.
In responding to our survey, inspectors had mixed views on the usefulness  of technical training. For about three-quarters of the responses, roughly  equal percentages of inspectors of non-legacy airlines responded that the  technical training they received in the last 2 years helped them do their  current jobs to some extent, to a moderate extent, or to a great extent. On  the other hand, 6 percent indicated that the technical training had not  helped them do their jobs and another 26 percent indicated it had only  helped to some extent. (See fig. 3.)
Further, the timeliness of certain training was identified as a concern by  both principal inspectors we spoke with and inspectors responding to our  survey. Specifically, some principal inspectors we spoke with indicated a  lack of timeliness for SEP training. Since FAA’s introduction of the SEP  training for non-legacy inspectors in 2001, a total of 700 inspectors have  received the training—-just over 300 inspectors between 2001 and 2002 and  almost 400 inspectors between 2003 and 2004. This means that less than 40  percent of the approximately 1,800 inspectors of both ATOS and non- legacy airlines (as of fiscal year 2004) could have received SEP training.  FAA requires all principal inspectors and other inspectors who are part of  certificate management teams to receive SEP training. Geographic  inspectors are not part of those teams and may not receive that training.  While all of the principal inspectors who were on staff when SEP was  implemented in 2002 received initial SEP training, according to FAA, since  that time, newly assigned inspectors may not have received the training  before beginning their responsibilities. Four principal inspectors who were  assigned relatively recently to non-legacy airlines and were not provided  the initial SEP training told us they had received subsequent training given  to all inspectors to explain revisions to the SEP program. They said that the  lack of initial SEP training hindered their ability to understand SEP terms  and processes and participate fully in the periodic meetings to analyze  information from inspections to identify risks. Those inspectors also  believed that the subsequent SEP training was not as helpful without the  initial SEP training.
The views of the inspectors we spoke with are consistent with our survey  results, which indicated that many inspectors of non-legacy airlines are not  highly satisfied with the timeliness of technical training they say they need  to do their jobs. Our survey found that only about 16 percent of non-legacy  inspectors believe to a great or very great extent that they have received  technical training in a timely manner to do their jobs. Moreover, about the  same percentage (15 percent) of the inspectors reported that the timing of  their training had not been useful to their jobs at all. (See fig. 4.)
About one-third of the inspectors in our survey indicated that the training  they received in the automated systems used in the inspection process— including PTRS—was provided in a timely manner to a moderate extent.  Another 30 percent thought the training was only somewhat timely. (See fig.  5.) Timely training on automated systems is important in order to have  accurate data. Inaccurate PTRS data hinder the identification of risks by  principal inspectors, as we discussed previously in this report.
section_title: FAA Lacks Effective Communication in How Internal Risks Are Resolved
paragraphs: Some inspectors told us that when they identified internal risks to FAA,  they were able to see the results of actions taken by their field and regional  offices to address the issues, but that they did not know what actions, if  any, were taken by FAA headquarters. This situation indicates the lack of a  key management control that calls for information to be recorded and  communicated to individuals within an entity who need the information in  a form and within a time frame that enable them to carry out their  responsibilities. However, unless the inspectors have access to a secure  intra-agency Web site and know how to locate the issue they submitted,  they do not know how FAA headquarters has responded to the risks  because this information is not directly provided to them by headquarters.  Headquarters officials acknowledged that there is no formal feedback loop  by which to inform the inspectors about the issues that they raised. In prior  work, we found that a lack of communication with inspector staff on the  resolution of enforcement actions that resulted from inspections is  frustrating to inspectors and might be a disincentive to reporting violations they find during inspections. This suggests that a similar lack of  communication could reduce inspectors’ interest in identifying internal  risks to FAA. We found that the overall number of times internal risks were  identified by inspectors declined for fiscal years 2002 through 2004.
section_title: PTRS Data Are Missing Elements That Would Make the Data More Useful
paragraphs: The PTRS database has limitations that reduce the usefulness of the data in  helping inspectors ensure that they are effectively conducting inspections.  PTRS does not facilitate FAA’s ability to monitor whether risks identified  through SEP have been mitigated. While the SEP process generates  required inspection activities based on risks that have been identified  through analysis of prior inspections, inspectors record these inspection  activities as closed in the PTRS database once they have completed the  inspection, whether or not the risks identified in SEP have been mitigated,  according to some principal inspectors we spoke with. While inspectors  are expected to use the comments section in PTRS to record additional  information, such as whether identified risks have been mitigated, they do  so infrequently. Our analysis of about 1.8 million PTRS records showed that  fewer than 400 records included any form of the words “mitigate,” “fix,” or  “resolve.” Moreover, such analysis is time consuming to conduct because a  uniform coding scheme is not employed for entering the information. As a  result, managers are not easily able to use PTRS data to monitor whether  risks identified through SEP have been mitigated and lack data that would  be useful in evaluating SEP. It is left to inspectors to determine  subsequently if the risks have been mitigated, usually through follow-up on- site inspections.
In addition, while a specific risk identified through SEP might result in  multiple inspections, there is no clear linkage between these inspections  when they are recorded in PTRS, according to a regional SEP manager. For  example, if a risk was identified through SEP-generated inspection  activities that included looking at manuals, checking aircraft records, and  performing a ramp check, inspectors would have to review all comment  records to identify the resulting activities in PTRS that are related to that  particular risk. This lack of linkage in PTRS may reduce inspectors’ overall  understanding of an airline’s situation. Inspectors who do not know why  activities have been generated may not target or fit their work to assess  these identified risks, while a greater understanding of how inspection  activities are connected to identified risks could help ensure that  inspections are conducted in a way that maximizes the benefits of a risk- based approach.
section_title: FAA Has Not Evaluated Its Inspection Oversight Process or Linked It to Agency Goals
paragraphs: FAA has not evaluated its inspection oversight process for non-legacy  airlines to determine how the process contributes to the agency’s mission  and overall safety goals. In addition, FAA has not explicitly linked SEP to  the overall safety goals. According to management control standards for  federal agencies and our prior reports on results-oriented management,  federal agencies should establish measurable performance goals for their  programs and operations. Agencies should also have an evaluation process  for their programs so that agency officials, Members of Congress, and  others will be able to determine whether goals are being achieved.
While FAA has not established a specific goal for SEP, it has an overall goal  to achieve the lowest possible accident rate and constantly improve safety.  To address this goal, FAA’s strategy includes expanding cost-effective  safety oversight and surveillance by targeting its inspection resources  better. However, FAA’s strategy does not explicitly show how SEP  contributes to the safety goal. In addition, FAA has not yet evaluated SEP to  determine if it is achieving the agency’s goal. As a consequence of not  having an evaluative process for SEP, FAA does not have the information it  needs to determine what changes should be made to improve its system  safety process. Also, FAA does not have a process to examine the  nationwide implications of or trends in the risks that inspectors have  identified through their risk assessments. Consequently, FAA does not have  the information it needs to proactively determine on a continuous basis  risk trends at the national level. While FAA has an evaluation office, this  office is not doing this type of analysis; rather, the office conducts analyses  of the types of inspections generated under SEP by airline and FAA region,  according to the manager of that office. One FAA field office we contacted  had taken the initiative to analyze risks identified during the SEP meetings  for trends. In FAA’s Eastern Region, a manager routinely analyzes trends in  types of findings within an airline and across airlines and provides this  information to the inspectors. This is an informal process, and it is not  certain that it will continue since the manager has been reassigned to  another office. Moreover, such analyses at the local or regional level do not  fulfill the management control to have a process in place to determine if  program goals are being achieved.
In addition, we found that while FAA has created specific national goals for  NPG that call for completing all required inspections each year, in some  cases these goals impede the agency’s ability to effectively implement its  system safety approach. For example, in order to meet the annual goal, the  regional offices have established interim goals to complete 25 percent of  the required inspections each quarter. Inspectors in one field office told us  that this situation created a disincentive for identifying additional activities  under SEP due to concerns about completing the quarterly requirements.  Those inspectors also noted that these goals encouraged them to prioritize  their work based on what they can most easily accomplish, rather than on  what represents the most significant risks—the antithesis of a system  safety approach.
section_title: Conclusions
paragraphs: Our review of FAA’s oversight of non-legacy airlines suggests that the full  benefits of a system safety approach can be realized only if the approach is  more fully implemented and utilized. FAA’s articulation of its system safety  approach, its application of system safety principles to its oversight  process through SEP, and its widely distributed training on system safety  are positive steps toward improving oversight by using the advantages of  system safety—particularly the ability to identify and prioritize inspections  based on risk. Fully developing SEP is important since FAA has been  unable to move significant numbers of airlines to ATOS and has only  recently established the goal to do so by the end of fiscal year 2007. A  process such as SEP is needed to identify risks in and among non-legacy  airlines and system-wide problems. However, the usefulness of FAA’s  system safety approach is reduced by limitations in the implementation of  SEP—such as FAA headquarters’ predominant focus on NPG, which has  led to only a small percentage of inspection activities being SEP-initiated;  the lack of training of geographic inspectors on SEP codes that could make  their inspections more useful to identifying risks during the SEP process;  and the lack of linkage to national goals and evaluations of SEP. Until SEP  is more fully implemented, it is clear that FAA’s approach to overseeing  non-legacy airlines is not largely risk based. Unless SEP is more thoroughly  integrated into its oversight, FAA may not be fully maximizing the  efficiency of its inspections in identifying and mitigating risks in order to  ensure the safety of non-legacy airlines. With FAA operating under a hiring  freeze and the number of inspectors available for non-legacy airlines  possibly further reduced by attrition and the move of inspection staff to the  ATOS program, the need to maximize the effectiveness of inspection  activities in ensuring the safety of non-legacy airlines is even more critical.
section_title: Recommendations for Executive Action
paragraphs: To improve the effectiveness of the agency’s oversight of non-legacy  airlines, we recommend that the Secretary of Transportation direct the FAA  Administrator to implement the following four recommendations:  To improve its safety oversight of airlines, FAA should (1) develop a  continuous evaluative process for its activities under SEP and link SEP  to the performance-related goals and measures developed by the  agency, track performance towards agency goals, and determine  appropriate program changes. The evaluation should include an analysis  of inspection findings to identify trends and risks at the national level.
In order to ensure that all regional and field offices have a complete and  timely understanding of FAA’s policies relating to the inspection  process, FAA needs to (2) improve communication in areas such as  whether and how internal risks identified by inspectors have been  resolved and (3) improve training in areas such as risk management,  coding items in the PTRS database, and how and under what  circumstances SEP-identified activities can replace NPG-identified  activities through retargeting.
To better utilize geographic inspectors’ support, FAA needs to (4)  improve the geographic inspectors’ understanding of the system safety  approach and operations of the airlines they inspect. FAA should  consider actions such as additional training, additional oversight in  particular areas, having airlines’ operating manuals available online for  review by inspectors, and improving communication between  geographic inspectors and principal inspectors on issues related to  identifying safety violations.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DOT for review and comment. We  obtained oral comments from FAA and DOT officials, including FAA’s  Deputy Associate Administrator for Aviation Safety. In particular, FAA  officials made the point that planned NPG activities have an element of risk  identification. We agree and revised the report. The FAA officials generally  agreed with our recommendations to improve communication, training,  and geographic inspectors’ understanding of the system safety approach.  The officials said that FAA would consider our recommendation to develop  a continuous evaluative process for SEP and link it to the agency’s goals,  but that its plan to eventually place the remaining non-legacy airlines in the  ATOS program might make this recommendation unnecessary. In the past,  FAA’s efforts to move airlines to ATOS have experienced delays, therefore,  we retained this recommendation. FAA officials also provided clarifying  comments and technical corrections, which we incorporated as  appropriate.
As agreed with your office, unless you announce the contents of this report  earlier, we plan no further distribution until 14 days from the report date. At  that time, we will send copies of this report to other congressional  committees, the Secretary of Transportation, and the Administrator, FAA.  We will also make copies available to others upon request. In addition, the  report will be available at no cost on GAO’s Web site at www.gao.gov.
If you or your staff have any questions about this report, please contact me  at (202) 512-2834 or dillinghamg@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix III.
section_title: Objectives, Scope, and Methodology
paragraphs: The objective of this report is to assess the Federal Aviation  Administration’s (FAA) processes for ensuring the safety of a fast-growing  portion of the commercial airline industry—the non-legacy passenger  airlines. Specifically, this report responds to the following questions: (1) What are the strengths of FAA’s inspection oversight of non-legacy  passenger airlines? (2) What issues hinder the effectiveness of FAA’s  inspection approach?
The scope of our review included FAA’s oversight activities from fiscal year  2002—when FAA implemented a new process (the Surveillance and  Evaluation Program or SEP) to its oversight of non-legacy airlines— through fiscal year 2004. To address the two questions, we obtained and  analyzed information about FAA’s oversight of the top 25 non-legacy  airlines ranked according to the number of enplanements in 2004. Those 25  airlines are identified in table 5, and each was covered by SEP during the  entire 3-year period or a portion of the period. Together, these 25 airlines  accounted for about 90 percent of all non-legacy passenger enplanements  in 2004.
To determine FAA’s legal oversight responsibility for commercial passenger  airlines, we obtained and analyzed regulations that govern FAA oversight of  these airlines. We also obtained and reviewed FAA handbooks, procedures,  and orders that describe the role and responsibilities of FAA managers and inspectors in implementing the two inspection oversight processes for  non-legacy airlines—SEP and the National Work Program Guidelines  (NPG). We conducted literature searches and reviewed prior reports and  articles on the oversight processes, including those prepared by the  Department of Transportation’s Inspector General and us. We reviewed  documentation provided by FAA, and contacted officials there and at GAO,  to determine whether findings identified in those reports have been  addressed. We also reviewed literature on system safety and compared it  with FAA’s system safety framework, which is incorporated in SEP.
We interviewed FAA headquarters officials from the Office of Aviation  Safety and its Flight Standards Service to obtain descriptions of NPG and  SEP. To understand how these processes were implemented, we collected  information from 7 regional offices and 13 field and certificate management  offices. We selected these offices because they have oversight  responsibility for most of the top 25 non-legacy airlines ranked by the  number of enplanements in 2004. We also interviewed officials from 16 of  the 25 non-legacy airlines. The regional offices, field and certificate  management offices, and airlines we interviewed are shown in table 6. For  each FAA office, we conducted semistructured interviews with managers,  SEP coordinators, and inspectors to obtain information on how they  implemented NPG and SEP. We also collected information on inspector  staffing levels, workload, and training. In addition, we reviewed whether  management controls have been established for NPG and SEP, and  determined if they were linked to FAA’s overall safety performance goals.  These controls included the (1) establishment of performance goals, (2)  verification of inspection results, (3) management of the inspection  database (Program Tracking and Reporting Subsystem or PTRS), and (4)  communication among managers and inspectors relating to NPG and SEP.
We also interviewed airline safety officials to obtain their views on FAA’s  inspection and surveillance activities and their participation in FAA- industry partnership programs.
We also conducted interviews with seven industry organizations to obtain  their views on FAA’s oversight processes. (See table 7.) We selected these  organizations because they represented diverse segments of the aviation  industry. In addition, we selected organizations that met one or more of the  following criteria: membership inspected by FAA, familiarity with FAA’s  safety oversight of airlines, familiarity with FAA’s SEP or Air Transportation  Oversight System processes, and familiarity with inspector training.
We analyzed information about FAA’s inspection activities for the top 25  non-legacy airlines using the agency’s PTRS database for fiscal years 2002  through 2004. Specifically, we analyzed information on the number of  required NPG and SEP inspections, inspector workload (i.e., number and  type of entities inspected), and number of NPG activities that were  retargeted (i.e., NPG-identified activities that were replaced with SEP- identified activities). This analysis excluded enroute activities, which take  place when an inspector monitors an aircraft as it travels from one  destination to another. These inspections were excluded because  inspectors often use these as secondary inspections as a means to travel to  a location where their primary inspection is such as a repair station. In  addition, we analyzed data that FAA maintains on the internal risks  identified by inspectors through regular SEP meetings. For fiscal years  2002 through 2004, we examined the type, severity, and closure status of  the identified risks. We also analyzed comment fields in that data to  determine if they indicated that the risks had been mitigated.
To assess the reliability of the data used in this report, we interviewed  knowledgeable agency officials about the data, performed electronic  testing of relevant data fields for obvious errors in accuracy and  completeness, and collected and reviewed documentation from data  system managers about the data and the systems that produced them. We  determined that the data were sufficiently reliable for the purposes of this  report.
To gather information about inspectors’ perspectives on the technical  training available to them, we conducted a Web-based survey of a  representative sample of FAA safety inspectors. The survey asked a  combination of questions concerning the amount and timeliness of training  that allowed for open-ended and close-ended responses. We had no  practical way to assess information on the amount of training necessary for  inspector proficiency or the timeliness of the training provided. We drew a  stratified random probability sample of 496 inspectors from the population  of 2,989 FAA aviation safety inspectors. We stratified the population into  12 groups on the basis of the type of work the inspector performed. Each  sample element was subsequently weighted in the analysis to account  statistically for all members of the population. For this report, we used a  subgroup sample of 205 non-legacy safety inspectors.
Because we followed a probability procedure based on random selections,  our sample is only one of a large number of samples that we might have  drawn. Since each sample could have provided different estimates, we  express our confidence in the precision of our overall sample’s results as a  95 percent confidence interval (e.g., plus or minus 7.1 percent). This is the  interval that would contain the actual population value for 95 percent of the  samples we could have drawn. As a result, we are 95 percent confident that  each of the confidence intervals in this report will include the true values in  the study population. The percentage estimates for the non-legacy  subgroup has a margin of error of plus or minus 7.1 percent or less. Survey  estimates presented as comparisons between groups are statistically  significant when the 95 percent confidence intervals do not overlap.
The survey was conducted using self-administered electronic  questionnaires accessible on the Internet through a secure Web browser.  We sent e-mail notifications to 496 inspectors (205 of which were in our  subgroup of inspectors of non-legacy airlines) beginning on December 6,  2004. We then sent each potential respondent a unique password and user  name to ensure that only members of the target population could  participate in the survey. To encourage respondents to complete the  questionnaire, we sent a subsequent e-mail message to further prompt each  nonrespondent approximately 2 weeks after the initial e-mail message. We  sent nonrespondents two more notices and closed the survey on February  4, 2005. Of the 496 inspectors we surveyed, we received 392 usable  responses (79 percent).  Among our subgroup of 205 non-legacy inspectors,  we received 161 usable responses (79 percent).
In addition to sampling errors, the practical difficulties in conducting  surveys of this type may introduce other types of errors, commonly  referred to as nonsampling errors. For example, questions may be  misinterpreted, or the respondents’ answers may differ from those of the  inspectors who did not respond. We took steps to reduce these errors.
Finally, we pretested the content and format of the questionnaire with  safety inspectors at local FAA offices in Baltimore, Los Angeles, and  Seattle. During the pretests we asked the inspectors questions to determine  whether (1) the survey questions were clear, (2) the terms used were  precise, (3) the questionnaire placed an undue burden on the respondents,  and (4) the questions were unbiased. We made changes to the content and  format of the final questionnaire based on the pretest results.
We conducted our work from August 2004 through September 2005 in  accordance with generally accepted government auditing standards.
section_title: FAA’s Surveillance and Evaluation Process
subsections: section_title: SEP Incorporates System Safety into FAA’s Inspection Oversight of Non- legacy Airlines
paragraphs: The Surveillance and Evaluation Program (SEP) is a process designed to  introduce a data driven risk analysis system for non-legacy airlines and is  guided by a Surveillance and Evaluation Assessment Tool (SEAT).  SEP  models itself on a system safety approach that is incorporated in the  oversight of legacy airlines through the Air Transportation Oversight  System (ATOS).  SEP allows teams of inspectors to identify suspected  trends through a data based system rather than relying totally on past  experiences.  A certificate management team oversees each of the 99  airlines under SEP.  Each team is led by three principal inspectors, one for  each major area of inspections (operations, maintenance, and avionics).   Additional team members include those based at the FAA office that holds  the airline’s operating certificate—typically an aircraft dispatch inspector, a  cabin safety inspector, and assistants.
The principal inspectors are required to meet as a team at least twice a year  to assess the safety risks associated with the airlines they oversee and  develop a surveillance plan for the airline, including designating inspection  duties to other FAA offices at locations to which the airline flies or  conducts business.  In some field offices, teams meet each quarter.  The  teams use a planning tool—SEAT—to assess the systems in place at an  airline and to identify any potential internal and external risks.  Using  SEAT, the team analyzes 10 systems, shown in table 8.
To complete SEAT, the principal inspectors rely on their knowledge of the  airline and on the data available through FAA’s Safety Performance  Analysis System database, which contains the Program Tracking and  Reporting Subsystem and 12 additional databases with safety and  performance information.  SEAT includes a set of risk indicators for each  of the elements (often in the form of a question) to be discussed by the  team to indicate concerns about any real or potential problem that could  contribute to the failure of one of the airline’s elements, subsystems, or  systems.  Inspectors rate each risk they identify in terms of severity  (negligible to catastrophic) and likelihood of occurrence (improbable to  frequent), and SEAT calculates an overall risk rating.  Risks rated high have  high levels of likelihood and severity.
Using this information, the team identifies a set of required inspections that  must be completed during that fiscal year and planned inspections.  The  required inspections are added to the NPG-identified inspections, and the  combined list becomes the annual surveillance plan for the airline.  During  the SEAT process, principal inspectors are allowed to substitute NPG- required inspections with SEP-identified inspections targeting higher-risk  areas.
section_title: SEP Incorporates Some Elements of ATOS, While NPG Relies on a Set Number of Inspections
paragraphs: Table 9 describes FAA’s three inspection processes for overseeing legacy  and non-legacy airlines: ATOS, NPG, and SEP.  Many of the elements of  ATOS, such as the use of data to identify risks and the development of  surveillance plans by inspectors, are incorporated in the SEP process.  The  NPG process, in contrast, is not focused on the use of data and relies on an  established set of inspections that are not risk based.
section_title: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the above, Teresa Spisak, Assistant Director; Carl Barden;  Bari Bendell; Curtis Groves; Bert Japikse; Catherine Kim; Minette  Richardson; Phillis Riley; Larry Thomas; Alwynne Wilbur; and Matt Zisman  made key contributions to this report.