Aviation Safety: System Safety Approach Needs Further Integration into FAA's Oversight of Airlines
Released: 2005-10-26T00:00:00

Letter

Background

Non-legacy airlines represent a fast-growing segment of the passenger airline industry. From 2002 through 2004, the annual enplanements for these airlines grew from 122 million to about 200 million passengers, or from about 20 percent to 28 percent of all passenger air travel. During this same period, the percentage of people flying on legacy airlines declined from 80 percent to about 72 percent of all passengers flown (see fig. 1).

FAA’s Inspector Workforce Carries Out Oversight

FAA’s safety oversight of non-legacy airlines is carried out by inspectors located at 109 field offices throughout the world that are part of 9 regional offices. For each airline, FAA puts together a team led by principal inspectors who maintain primary responsibility for managing the airline’s certificate requirements and focus on one of three disciplines: avionics, maintenance, or operations. Additional team members include those based at the FAA office that holds the airline’s operating certificate—typically an aircraft dispatch inspector, a cabin safety inspector, and assistants. FAA locates its principal inspectors close to their respective airlines’ primary operational base. For example, the principal inspectors for Independence Air (formerly Atlantic Coast Airlines) are located at FAA’s field office at Dulles International Airport, where the airline has its headquarters. In addition, FAA has geographic inspectors based at the 109 field offices to conduct additional inspections. Rather than being designated to particular airlines, geographic inspectors may conduct inspections of aircraft of any non-legacy airlines that land in their area.

FAA’s safety inspector workforce for all commercial airlines has remained steady, averaging about 1,780 inspectors over fiscal years 2002 through 2004. FAA cannot determine how many of these inspectors were assigned to non-legacy passenger airlines during those 3 years, because the agency did not collect that information, according to an FAA headquarters’ official. FAA expects the number of inspectors assigned to non-legacy airlines to remain steady for fiscal years 2005 and 2006 and decline slightly in fiscal year 2007, as shown in figure 2. Inspectors record information about these inspections in PTRS—a nationwide computerized database that maintains such information as inspector findings and airline activities in response to the findings.

FAA Uses NPG and SEP to Oversee Non-legacy Airlines

NPG and SEP are the main inspection processes that FAA uses to oversee the safety of non-legacy airlines. Since 1985, FAA has used NPG, which includes both required and planned inspections, as its primary means of ensuring that airlines comply with safety regulations. In NPG, an FAA committee of program managers identifies an annual minimum set of required inspections that are to be undertaken to ensure that airlines are in compliance with their operating certificates. In addition, inspectors determine annual sets of planned inspections based on their knowledge and experience with the particular airlines they oversee. Typically, inspections would include ramp inspections, in which inspectors examine an aircraft while it is parked at the airport, and maintenance inspections. However, we found problems with NPG throughout the 1990s, including: (1) FAA’s routine inspections were ineffective in identifying serious safety problems, (2) critical airline inspections had not been conducted, (3) FAA’s follow-up actions often did not ensure that problems were corrected once identified, and (4) FAA did not have a methodology for estimating airline safety risks so that it could target limited inspection resources to high-risk inspections. We also found that FAA’s inspection database was of limited use in providing early warning of potential risks or targeting inspection resources.

In response to these findings, and in the aftermath of the 1996 ValuJet crash, an FAA task force reviewed the agency’s safety inspection process and recommended in part that the agency initiate a project to make surveillance of airlines more systematic and targeted to deal with identified risks. This recommendation resulted in the agency’s development and implementation of ATOS in 1998 at the nation’s 10 largest commercial passenger airlines, with the goal of eventually including all commercial passenger and cargo airlines. ATOS emphasizes a system safety approach that extends beyond periodically checking airlines for compliance with regulations to the use of technical and managerial skills to identify, analyze, and control hazards and risks. The goal of ATOS is to identify safety trends in order to spot and correct problems at their root cause before an accident occurs. This program allows FAA inspectors to look at an airline as a whole, to see how the many elements of its operations, including aircraft, pilots, maintenance facilities, flight operations, and cabin safety, interact to meet federal standards. Collectively, the airlines under ATOS had a dedicated inspector staff of 585 inspectors as of July 2005. The number assigned to each airline depends on the size of the airline’s operations. ATOS uses special checklists and databases that are intended to cover all areas of airline operations. Part of FAA’s oversight is expected to include an in-depth look at an airline’s policies and procedures and whether the airline is following them. There are 16 airlines in the ATOS program, including two cargo carriers as of September 2005. Due to resource constraints, FAA determined it would not be able to immediately place the remaining passenger airlines in the ATOS program. FAA developed SEP as a bridge to introduce safety risk concepts used in ATOS into the oversight process for non-legacy airlines in order to facilitate the ultimate transition of these airlines to ATOS. As of September 2005, FAA estimates that it will move the remaining non-legacy airlines to ATOS by the end of fiscal year 2007.

SEP Implemented as a Transition to ATOS

Since the introduction of ATOS, FAA has been moving toward integrating system safety into its oversight activities. However, the agency has been delayed in placing a significant number of non-legacy airlines in the ATOS program, resulting in those airlines continuing to be overseen through NPG, a process that is not system safety oriented. FAA agrees that NPG is not a system safety process, but explained that the planned NPG inspections, which inspectors identify based on their expertise, provide a risk-based element to the process. To address the delay in moving airlines to ATOS, FAA, in 2002, added SEP to NPG as a way to introduce airlines to a system safety oversight process until those airlines were transitioned to ATOS. Although originally envisioned by FAA as a transitional program for airlines that were awaiting placement in ATOS, SEP has become a more permanent oversight process as budget and staff constraints have prevented the agency from moving all airlines to ATOS.

SEP is complementary to NPG and permits the risk-based, data-driven alteration of required NPG inspections. SEP provides a formal structure to risk identification that is absent from the NPG process. Under SEP, the principal FAA inspectors for each airline meet periodically during the year to discuss the results of their inspections and identify risks using the same risk assessment principles and checklists used in ATOS. An outcome of the meetings is the identification of inspections to augment the NPG baseline of required inspections, which FAA views as the minimum number of inspections that need to occur to ensure that certain areas of all airlines are reviewed. During the SEP process, FAA headquarters allows principal inspectors to replace NPG-required inspections with inspections targeting higher-risk areas identified through the SEP meetings. This process is known as retargeting. At the end of the meeting, entries are created in the PTRS database that indicate all inspections to be performed for that airline. SEP-initiated inspections that are designated as priorities are required to be completed. See appendix II for additional information on SEP, ATOS, and NPG.

FAA Also Uses Industry Partnership Programs to Provide Safety Oversight of Airlines

FAA also oversees the safety of both legacy and non-legacy passenger airlines through participation in industry partnership programs. Two of these programs—the Aviation Safety Action Program and Voluntary Disclosure Reporting Program—encourage certain airline employees, such as pilots and mechanics, or airlines to voluntarily report safety information that might be critical to identifying potential precursors to accidents without fear that FAA or their companies will use reports accepted under the programs to take legal enforcement or disciplinary actions against them. The Aviation Safety Action Program provides for the voluntary self- reporting of safety incidents under procedures set out in memorandums of understanding between FAA, airlines, and participating employee groups such as aircraft mechanics. As of June 2005, 20 of the top 25 non-legacy airlines (in terms of the number of enplanements) were participating in the Aviation Safety Action Program. The Voluntary Disclosure Reporting Program allows airlines to voluntarily report safety incidents to FAA. However, our prior work found that FAA has not analyzed violation data derived from these two programs to monitor national trends in airline operations, so that it can target resources to address operational risks. A third program, the Internal Evaluation Program, requires airlines to continuously monitor and evaluate their practices and procedures.

FAA’s Oversight Process Uses System Safety, Which Is Designed to Identify and Control Risks and Improve Resource Utilization

The incorporation of system safety into FAA’s oversight presents a shift in concept from the agency’s traditional oversight method, which relied upon periodic inspections, to an approach that allows for the prioritization of inspections based on areas of highest risks. The development of SEP, a key element in FAA’s system safety oversight process, allows for the efficient use of inspection staff and resources by incorporating this risk-based approach. FAA’s incorporation of SEP into its oversight addresses a past concern by us that the agency was not prioritizing inspectors’ workload. Our review of literature by government and private organizations supported FAA’s concept of system safety because it makes apparent the risks that are the basis for changes. In addition, FAA’s oversight process includes a requirement that inspectors verify that corrective actions have occurred, providing a level of assurance that the safety problem has been mitigated.

FAA Has Provided System Safety Training for Inspectors

To facilitate the implementation of a system safety approach, FAA has made an effort to train its staff in system safety, which is a strength of its oversight approach. FAA has recognized that inspectors need training in the system safety concept in order to effectively incorporate this approach into airline oversight. Specifically, FAA believed that its inspectors have gaps in their needed competencies and skills that could affect the agency’s system safety approach to inspections. Among the largest training gaps for inspectors, according to FAA, were training in risk analyses and systems thinking. As a result of these findings, FAA has undertaken training activities to improve inspector competencies in system safety. According to FAA, between fiscal years 2002 and 2004, it provided basic system safety training to almost 3,675 staff, including non-legacy and ATOS inspectors.

SEP Incorporates System Safety by Focusing on the Identification and Prioritization of Inspections Based on Risks

By incorporating risk assessment into the inspection process, SEP addresses a long-standing problem. Since the mid-1980s, our reports have shown that FAA did not have a methodology for assessing airline safety risks so that it could target limited inspection resources to high-risk conditions. SEP’s system safety concept of reducing risk through the identification, analysis, and control of hazards is also consistent with the views presented in literature of other government and private organizations, such as the Department of Defense, Canada’s civil aviation authority (Transport Canada), and the Flight Safety Foundation. For example, DOD looks at system safety as a means of reducing risk through early identification, analysis, elimination, and control of hazards. For the airlines it oversees, Transport Canada requests that each airline incorporate system safety into its operations by integrating safety into its policies, management and employee practices, and operating procedures. FAA incorporates such concepts in its system safety oversight approach.

SEP’s system safety process has a number of strengths. At periodic meetings, the principal inspectors of an airline identify potential risks that they believe should be addressed. These inspectors consider a variety of information that they have obtained through personal observation and from reports prepared by geographic inspectors and located in the PTRS database. A strength of this approach, consistent with findings in our past reports, is that teams of inspectors are generally more effective than individual inspectors in their ability to collectively identify concerns. Another strength is that FAA has developed risk assessment worksheets aligned with key airline systems that guide inspectors through identifying and prioritizing risks. The worksheets guide inspectors to organize the results of their previous inspections and surveillance into a number of areas, such as flight operations, personnel training, and cabin safety, in order to identify specific risks in each area and target the office’s resources to mitigating those risks. (See app. II for more details on the risk assessment process.)

SEP Also Allows for the Identification of Internal Risks That Might Hinder Oversight

During their periodic meetings, principal inspectors also have the opportunity to identify risks in FAA’s internal operations that could adversely affect the inspectors’ ability to conduct safety oversight of non- legacy airlines, which is another strength of FAA’s oversight approach. FAA inspectors and managers agreed that identifying internal risks is helpful, as they considered it an efficient and effective way of identifying needed program improvements. Inspectors can identify deficiencies in a number of categories, including inspector staffing, training, availability of geographic inspectors, and resources such as travel funds. In addition, the inspectors can quantify the degree of the risk that each of these categories poses. Being able to identify and make officials aware of situations that can hinder an agency’s ability to perform its mission is a key management control.

This process has resulted in FAA headquarters receiving significant information on areas that inspectors believe are deficient. For fiscal years 2002 through 2004, inspectors identified about 560 risks that, in their opinion, could have an impact on how FAA managed the inspection process of airlines, as shown in table 1. Inadequate training (aircraft- specific training and other training) was the most frequently identified concern of the inspectors, accounting for about 40 percent of the total risks identified during the 3-year period—27 percent of the risks were related to inadequate or nonexistent aircraft-specific training and 15 percent were related to the lack of other training. We discuss this issue in detail later in this report. The total number of risks identified by FAA inspectors declined over the 3 years. Inspectors from two FAA field offices opined that the decline may be due to better staff understanding of risk identification and to the risks being addressed expeditiously so that inspectors did not need to restate them the following year.

The process not only allows inspectors to identify the risk, but also to describe the severity of the risk in order to assist FAA headquarters in prioritizing its consideration of internal risks. Inspectors rate each risk they identify in terms of severity (negligible to catastrophic) and likelihood of occurrence (improbable to frequent). Risks are then categorized using a scale of high, medium, or low depending on the severity and likelihood of occurrences. For example, risks rated high are those that have high levels of likelihood and severity. As shown in table 2, over a 3-year period, 125 out of 557 risks were rated as high severity, which could help FAA determine which issues most need targeting. Among the risks rated as high, training and inadequate staffing were identified most frequently, accounting for 34 percent and 31 percent, respectively. As of May 2005, about 87 percent of identified risks had been closed, including about 90 percent of the risks identified as high. Risks are closed either because action has been taken to mitigate the risk or FAA has determined that the level of risk is acceptable and no action is warranted.

An additional strength of the internal risk identification process is that FAA has established a chain of command for responding to these internal risks at the appropriate level. For example, local training issues are likely to be handled by field offices, geographic support issues would normally be handled by regional offices, and issues that cannot be resolved by these field and regional offices would be referred to headquarters.

FAA’s Oversight Process Includes Monitoring to Follow Up on Airline Actions Taken in Response to Findings

Another strength of FAA’s oversight of non-legacy airlines is that inspectors monitor the actions that those airlines have taken in response to inspection findings through subsequent inspections and participation in safety partnership programs with the airlines. However, the inspectors’ monitoring process could be improved by better data, as we discuss later in this report. SEP requires inspectors to monitor the actions taken by non- legacy airlines that will address the risks identified during an inspection and to verify that the actions taken have resulted or will result in resolution of the problems. This requirement addresses a concern of ours that the NPG inspection program did not often ensure that the agency follow-up actions corrected problems once they were identified. In addition, FAA’s requirement for monitoring and verifying corrective action conforms to a management control standard on the need for federal agencies to have monitoring and verification policies or procedures for ensuring that the findings of audits and other reviews are promptly resolved. FAA inspectors monitor and confirm that corrective action has been taken by either conducting a special inspection of the area, following up during a subsequent inspection, or reviewing airline documentation that addresses the deficiency. According to inspectors and field office managers, the verification method used is based in many instances on the severity of the deficiency, with special inspections conducted for areas that could affect air safety.

According to some FAA inspectors and airline officials, the regular meetings held to discuss concerns about airlines have opened up communications both within FAA and between FAA and the airlines. Lack of communication among the airlines and FAA was, according to the director of FAA’s Flight Standards Service, historically a significant cause of airline safety problems. FAA inspectors and managers noted that they maintain an open line of communication with airline management by meeting regularly with key airline officials (such as directors of operations and directors of maintenance) to discuss the results of the SEP risk analyses and the airline’s proposed resolutions.

FAA’s Oversight Process Is Hindered by Certain Program Weaknesses and Human Capital Management Challenges

FAA’s oversight of non-legacy airlines is hindered by the incomplete implementation of its system safety approach and other challenges. FAA’s incorporation of system safety into its oversight process is incomplete, as the agency continues to emphasize a nonsafety system-based process, NPG, to identify most of its inspection activities. Human capital management challenges also impede FAA’s oversight. These challenges include a hiring freeze that began in January 2005; attrition; in certain cases, the reallocation of inspectors’ duties as airlines are moved to the ATOS program; insufficient training for inspectors on risk management, a key element of SEP; and insufficient communication between FAA headquarters and field offices. In addition, FAA lacks a process to continuously evaluate SEP. Finally, the agency’s PTRS database lacks some important inspection-related information, such as whether risks have been mitigated, that would aid in targeting further oversight activities as well as a nationwide analysis of inspection results.

Most Inspection Activities Are Not Prioritized Based on a Structured Risk Assessment Process

We found that NPG remains the primary basis for FAA’s inspection of non- legacy airlines. For fiscal years 2002 through 2004, about 77 percent of the required inspection activities for the top 25 non-legacy airlines were initiated through the NPG annual planning session, compared with 23 percent that were SEP-initiated. As a result, most of the required inspection activities are not prioritized based on risk. (See table 3.) In addition, the total number of required inspections for non-legacy airlines declined during the 3-year period, while the number of passengers on those airlines grew from 122 million to 200 million annual enplanements. The decline in inspections during this period may be due, in part, to the movement of three airlines from SEP to ATOS. FAA headquarters officials did not know why this decline occurred, but told us that it may be due to agency efforts to eliminate unnecessary activities and shift from a “quality control” oversight approach, in which individual products or outputs are inspected to determine if they meet specifications, to a “quality assurance” approach, in which airlines’ processes are inspected. While NPG also includes planned inspections, which have a risk-based element, these inspections lack the structured approach to risk identification that SEP has. FAA officials acknowledged that the NPG planned inspections would benefit from the use of risk management tools.

According to FAA officials, there are no minimum numbers or percentages of required SEP inspections. Therefore, a large percentage of NPG-initiated activities is not unexpected or negative in their view. FAA officials said NPG-initiated activities are an important part of the inspection process because they provide information to the agency on an annual basis for certain types of inspections and enable periodic assessments of key programs in an airline’s operations. According to FAA, NPG is organized to systematically validate on an annual basis the performance of all of an airline’s safety-critical programs. FAA considers this type of recurring assessment to be an important system safety principle and believes that it provides an opportunity to discover and correct latent failures before they cause safety problems. In addition, according to FAA officials, NPG identifies a minimum number of inspections that they believe need to occur to ensure that certain areas of an airline are reviewed, unlike SEP, which does not provide the built-in assurance that these areas will be inspected eventually.

We agree with FAA officials that the agency should not establish minimum numbers or expected percentages of SEP inspections. However, we found that the large percentage of required NPG activities compared with SEP activities occurs, in part, because FAA’s guidance places a greater emphasis on completing required NPG activities. This situation, along with other factors we discuss below, may deter inspectors from identifying additional inspections through SEP. FAA’s guidance includes the following: Field offices must complete 100 percent of the required NPG-identified activities by the end of each fiscal year (unless they are retargeted).

Field offices are not required to complete SEP-initiated activities by the end of the fiscal year and may reschedule them to the following fiscal year.

In addition, the budget given to field offices for inspections is based on required NPG-initiated activities, and funding to cover SEP-initiated activities must come out of that budget. Inspectors have indicated concern regarding resources for inspections; for example, a lack of resources, including a lack of travel funds to conduct inspection activities, made up about 10 percent of the internal risks identified by inspectors through SEP during fiscal years 2002 through 2004. According to headquarters officials, requests for funding high-risk SEP-identified inspections are always approved; however, some noncritical activities may be delayed due to budget constraints. For example, they said that some certification work— such as certifying that new aircraft or aircraft parts meet FAA standards— is being delayed until an inspector travels to the airline to perform another activity and can do the certification at little or no extra cost.

Under FAA’s guidance, if inspectors identify a risk under SEP that they wish to mitigate through an additional inspection activity, they must either add the SEP-initiated activity to the list of required NPG-initiated activities or retarget a required NPG activity and replace it with the SEP-initiated activity. Adding SEP-initiated activities to the required NPG-identified activities may increase the overall workload of inspectors. This situation may affect the number of additional activities that inspectors identify during the SEP meetings. In fact, we found that in some cases, no SEP- initiated activities were included with NPG-initiated activities as part of an airline’s overall required inspections. For example, for one airline overseen by FAA’s Central Region, PTRS data show that, in fiscal year 2002, inspectors conducted about 400 required NPG-initiated activities and no SEP-initiated activities. According to FAA headquarters officials, many inspectors view SEP-initiated activities as additional work. This view was also held by some inspectors we spoke with.

Moreover, our analysis of PTRS data indicates that retargeting occurs infrequently. In fiscal year 2002, about 5 percent of all NPG-required inspections were retargeted by inspectors and 3 percent were retargeted in fiscal years 2003 and 2004. This low rate occurs, in part, because field offices have interpreted FAA’s emphasis on NPG inspections as discouraging retargeting. For example, three of the principal inspection teams for the 16 airlines told us they do not retarget any NPG activities. Further, in 5 of the 13 field offices we visited, we were told by both managers and inspectors that they believe NPG activities take priority over all but the most risky situations identified through SEP. Similarly, an official with the Professional Airways Systems Specialists, a union that represents many FAA safety inspectors, told us that its members believe that FAA’s management does not give as much attention and priority to SEP as it does to ATOS.

FAA’s emphasis on required NPG-initiated inspections is also shown through its data-tracking efforts. According to FAA officials, for fiscal years 2002 through 2004, all required NPG activities had been completed by the end of the year. FAA did not know how many completed SEP-initiated activities were mitigated or required further inspection activities that were then carried over to the next fiscal year because it does not collect such information nationwide in PTRS. Moreover, field office managers also rely on PTRS, which lacks information on the number of activities that are carried over. For example, one field office supervisor told us that one inspector had carried over 19 activities from fiscal year 2004 to fiscal year 2005. However, the field office supervisor said it was difficult to determine how many additional activities were carried over because PTRS does not track this information.

Shifts in Inspectors’ Workload Present a Challenge to FAA’s Inspection Oversight Process

The number of FAA inspectors available to oversee non-legacy airlines will be affected by recent and anticipated trends in attrition and a hiring freeze. First, for fiscal year 2004, the number of inspectors (those assigned to non- legacy airlines, ATOS, and general aviation) who left FAA was greater than the number of inspectors hired, resulting in a net loss of 67 inspectors. Second, the number of inspectors of commercial airlines (both ATOS and non-legacy) is expected to continue to decline from 1,810 at the end of fiscal year 2004 to 1,704 at the end of fiscal year 2005. Third, from fiscal years 2006 to 2007, the agency anticipates the number of inspectors of non- legacy airlines will decline from 1,119 to 1,088. Fourth, for fiscal years 2005 through 2010, FAA estimated that over 1,100 inspectors of non-legacy airlines and general aviation will leave the agency, with an average loss due to attrition of about 195 inspectors per year.

FAA will have to hire inspectors to offset the anticipated losses. However, the agency put a hiring freeze in place in January 2005 for budgetary reasons, which prevents the agency from hiring inspectors to replace those lost due to attrition unless they are deemed critical. Furthermore, in order to absorb a budget reduction in fiscal year 2005, FAA expects to eliminate about 200 inspector positions in the Office of Aviation Safety. These reductions will be accomplished through attrition. In fiscal year 2006, the office expects the staffing level to increase by about 80, which would still be below the fiscal year 2004 staffing level. FAA is filling safety-critical positions that become vacant through internal appointments. According to FAA officials, principal inspectors and managers are considered safety critical. As other safety inspectors, including geographic inspectors, leave the agency or are appointed to safety-critical positions, they are not being replaced at this time, according to those officials. Moreover, their workload is being divided among the remaining inspectors, thereby increasing the inspectors’ workload, which has resulted in other, less critical, work being delayed or deferred. In 11 of the 13 FAA field offices we contacted, officials indicated a shortage of different types of inspectors needed to oversee the non-legacy airlines. For example, the Washington, D.C., field office did not have a principal avionics inspector. Officials with FAA field and regional offices said they have developed ways to deal with the inadequate staffing of inspectors in their offices. For example, the Southern Regional Office told us that it uses geographic inspectors more frequently than in the past to help conduct inspections rather than having the inspectors assigned to the non-legacy airline travel to conduct them. However, two field offices—- Chicago and Minneapolis—-indicated the need for more geographic inspectors. Another way that FAA offices are dealing with inspector shortages is by having an inspector perform inspections for several offices. For example, the Atlanta field office has been without a cabin safety inspector for a year and uses the inspector assigned to the Southern Regional Office. According to an FAA inspector, some inspections are delayed due to lack of staff availability. In addition, according to our analysis of FAA data (discussed earlier in this report), for fiscal years 2002 through 2004, principal inspectors identified lack of staff for the airlines’ certificate management teams as among the top two risks for each year, and overall, more than one-third of the risks identified were related to lack of inspectors.

The move of additional airlines into ATOS, which requires more inspectors per airline than SEP, also affects the workload of inspectors for non-legacy airlines. In addition, unlike SEP inspectors, ATOS inspectors are dedicated to an airline and generally cannot be used to conduct inspections of other entities. When four airlines were recently transitioned into ATOS, the total size of the four inspection teams increased 30 percent, from 73 to 95 inspectors, as shown in table 4. As a result, the number of inspectors available to oversee non-legacy airlines was reduced not only by those 73 who had covered the four airlines while they were under the NPG and SEP processes (who would have also had responsibilities for other entities) but by an additional 22 inspectors as well.

FAA is aware that transitioning additional inspectors to airlines in ATOS is putting a burden on the workload of those inspectors who continue to inspect non-legacy airlines, and the agency has, in one case, revised its procedures for dedicating staff to individual ATOS airlines. When Champion Airlines became part of ATOS in January 2005, FAA changed ATOS policy to allow the Northwest Airlines inspection team to share its data analyst and manager with the Champion inspection team. FAA officials believe that sharing staff resources will address the issue of staff shortages.

All of the inspectors at the offices we contacted had other duties in addition to overseeing non-legacy airlines. The inspectors performed activities such as certifying and approving new aircraft types; overseeing repair stations and aviation schools; reviewing FAA directives and aircraft updates; responding to complaints; investigating accidents, incidents, and complaints; overseeing designees; and attending training. Although primarily responsible for a particular airline, some principal inspectors conducted inspections of other airlines and entities. For example, a principal inspector in the Great Lakes Region was responsible for overseeing three repair stations in addition to a non-legacy airline. In addition, a principal operations inspector in the Southern Region was also responsible for overseeing designees such as pilot examiners. Our analysis of the inspectors’ workload contained in the PTRS database showed that for fiscal years 2002 through 2004, about 75 percent of FAA inspectors had responsibility for more than 3 entities, and about half had responsibility for more than 15. While the size and complexity of the various entities will dictate the inspector workload, having responsibility for a large number of entities can increase an inspector’s workload because of the need to be familiar with the operating procedures of each entity as well as to spend time physically inspecting each entity.

Inspectors Are Concerned That Training Limitations Present a Challenge to Inspection Oversight Process

While FAA provided system safety training to its inspector workforce, as mentioned earlier in this report, the lack of sufficient training in other areas was identified by principal inspectors during their risk analysis meetings and by inspectors we spoke with. Our analysis of FAA’s database of internal risks identified by principal inspectors of non-legacy airlines indicated that 42 percent (233 out of 558) of the internal risks identified for fiscal years 2002 through 2004 related to training. Although FAA policy requires that principal inspectors assigned to non-legacy airlines complete technical training on the airline’s specific policies, procedures, and equipment before they can conduct airline inspection activities, principal inspectors identified risks associated with the lack of such training. For example, our analysis of the internal risk database found that principal inspectors identified a lack of technical training for themselves or other inspectors on their team on specific aircraft, engine types, navigation equipment, and avionics that pertained to the airlines they were overseeing. Others cited a lack of training in cargo loading, hazardous material identification, or weight and balance. Still others indicated the need for training in FAA’s systems or processes. For example, several indicated that they had not received the basic courses for principal inspectors (such as “foundation for principal inspectors”). As another example, several principal inspectors reported that certain inspectors on their team lacked training in using the computer system that analyzes information collected in PTRS, noting that without this training they were unable to directly access the system. Even though inspectors expressed these concerns, approximately 87 percent of identified internal risks have been closed by FAA, according to our analysis of FAA data. However, we could not determine how many of those risks were actually mitigated, as FAA closes risks either when action has been taken to mitigate the risk or because the agency has determined that the level of risk is acceptable and no action is warranted.

During our field office visits, inspectors also spoke about the lack of technical training in certain areas. For example, inspectors told us they had not received training on some newer aircraft they were responsible for inspecting and that they were not always able to attend free training provided by the manufacturers, which they felt would help them obtain needed technical knowledge. We found instances where management acknowledged that such training was needed. In the past, concerns had been raised that FAA’s acceptance of free training from a regulated entity might have the appearance of a conflict of interest. In September 2005, we reported that FAA accepts such training under limited circumstances. FAA has established safeguards to help preclude the appearance of a conflict of interest when FAA accepts training from the entities it regulates at no cost to the agency or in exchange for an in-kind service.

In addition, the principal inspectors we spoke with indicated that some geographic inspectors who assist them in providing oversight of non-legacy airlines lack the technical and SEP-specific procedural training to maximize their usefulness. Several field office managers and principal inspectors told us that geographic inspectors needed to be knowledgeable about the operations of airlines they inspect in order to provide effective assistance. Several principal inspectors indicated that misperceptions about an airline’s operating requirements are not uncommon because geographic inspectors often lack training on the aircraft they inspect. As a result, the inspectors indicated that geographic inspectors had at times incorrectly coded inspection information in PTRS and erroneously identified risks, which resulted in principal inspectors having to spend time determining that a problem did not actually exist. In addition, several airline officials told us that geographic inspectors had incorrectly identified problems and appeared to not understand airlines’ operational procedures. For example, we were told that a geographic inspector improperly cited an airline for not complying with a deicing program that the airline had previously informed FAA inspectors it was changing. FAA headquarters officials told us that it is not feasible to train all geographic inspectors on all airlines they may encounter. Further, they noted it would be good for geographic inspectors to contact principal inspectors when they are unsure of whether an action or condition they observe is a problem. However, FAA headquarters has not provided that instruction to the geographic inspectors. Industry representatives also told us this was a problem that could be alleviated by inspectors having access to airlines’ operating manuals online.

Another issue identified as a problem by principal inspectors we spoke with is that not all geographic inspectors are provided training in SEP, including how to use the uniform coding scheme that was developed for SEP to facilitate the identification and analysis of risks. Geographic inspectors may therefore note the results of their inspections through narrative in the comments section of PTRS rather than through SEP codes, so that the principal inspectors must read through all the comments in order to identify geographic inspectors’ concerns. As PTRS may contain thousands of entries on one airline each year, reading every narrative comment can be time consuming and difficult to interpret. The lack of coding of geographic inspectors’ findings could therefore result in principal inspectors missing problems or risks identified by geographic inspectors. We do not know how often this problem occurs, as FAA has not assessed the reliability of this data in PTRS. However, several FAA staff brought this issue to our attention. One field office manager and SEP coordinator told us that PTRS entries by geographic inspectors are not helpful because they do not provide meaningful comments for FAA’s risk assessments, and several principal inspectors told us that training for geographic inspectors on SEP coding would help them more easily identify risks during their periodic meetings.

These training issues may arise in part because FAA has not systematically identified technical training needs for principal or other inspectors. Our previous report found that FAA has not systematically identified overall training needs of its inspectors to ensure that the curriculum addresses the unique training needs of each type of inspector. Instead, course development focuses on individual courses. FAA said it recognizes that it manages courses as individual components of an overall curriculum that is only loosely defined and that it needs to develop courses and address training needs as part of an overall curriculum. It has established a curriculum transformation plan that it estimates it will fully implement in 2008.

In responding to our survey, inspectors had mixed views on the usefulness of technical training. For about three-quarters of the responses, roughly equal percentages of inspectors of non-legacy airlines responded that the technical training they received in the last 2 years helped them do their current jobs to some extent, to a moderate extent, or to a great extent. On the other hand, 6 percent indicated that the technical training had not helped them do their jobs and another 26 percent indicated it had only helped to some extent. (See fig. 3.)

Further, the timeliness of certain training was identified as a concern by both principal inspectors we spoke with and inspectors responding to our survey. Specifically, some principal inspectors we spoke with indicated a lack of timeliness for SEP training. Since FAA’s introduction of the SEP training for non-legacy inspectors in 2001, a total of 700 inspectors have received the training—-just over 300 inspectors between 2001 and 2002 and almost 400 inspectors between 2003 and 2004. This means that less than 40 percent of the approximately 1,800 inspectors of both ATOS and non- legacy airlines (as of fiscal year 2004) could have received SEP training. FAA requires all principal inspectors and other inspectors who are part of certificate management teams to receive SEP training. Geographic inspectors are not part of those teams and may not receive that training. While all of the principal inspectors who were on staff when SEP was implemented in 2002 received initial SEP training, according to FAA, since that time, newly assigned inspectors may not have received the training before beginning their responsibilities. Four principal inspectors who were assigned relatively recently to non-legacy airlines and were not provided the initial SEP training told us they had received subsequent training given to all inspectors to explain revisions to the SEP program. They said that the lack of initial SEP training hindered their ability to understand SEP terms and processes and participate fully in the periodic meetings to analyze information from inspections to identify risks. Those inspectors also believed that the subsequent SEP training was not as helpful without the initial SEP training.

The views of the inspectors we spoke with are consistent with our survey results, which indicated that many inspectors of non-legacy airlines are not highly satisfied with the timeliness of technical training they say they need to do their jobs. Our survey found that only about 16 percent of non-legacy inspectors believe to a great or very great extent that they have received technical training in a timely manner to do their jobs. Moreover, about the same percentage (15 percent) of the inspectors reported that the timing of their training had not been useful to their jobs at all. (See fig. 4.)

About one-third of the inspectors in our survey indicated that the training they received in the automated systems used in the inspection process— including PTRS—was provided in a timely manner to a moderate extent. Another 30 percent thought the training was only somewhat timely. (See fig. 5.) Timely training on automated systems is important in order to have accurate data. Inaccurate PTRS data hinder the identification of risks by principal inspectors, as we discussed previously in this report.

FAA Lacks Effective Communication in How Internal Risks Are Resolved

Some inspectors told us that when they identified internal risks to FAA, they were able to see the results of actions taken by their field and regional offices to address the issues, but that they did not know what actions, if any, were taken by FAA headquarters. This situation indicates the lack of a key management control that calls for information to be recorded and communicated to individuals within an entity who need the information in a form and within a time frame that enable them to carry out their responsibilities. However, unless the inspectors have access to a secure intra-agency Web site and know how to locate the issue they submitted, they do not know how FAA headquarters has responded to the risks because this information is not directly provided to them by headquarters. Headquarters officials acknowledged that there is no formal feedback loop by which to inform the inspectors about the issues that they raised. In prior work, we found that a lack of communication with inspector staff on the resolution of enforcement actions that resulted from inspections is frustrating to inspectors and might be a disincentive to reporting violations they find during inspections. This suggests that a similar lack of communication could reduce inspectors’ interest in identifying internal risks to FAA. We found that the overall number of times internal risks were identified by inspectors declined for fiscal years 2002 through 2004.

PTRS Data Are Missing Elements That Would Make the Data More Useful

The PTRS database has limitations that reduce the usefulness of the data in helping inspectors ensure that they are effectively conducting inspections. PTRS does not facilitate FAA’s ability to monitor whether risks identified through SEP have been mitigated. While the SEP process generates required inspection activities based on risks that have been identified through analysis of prior inspections, inspectors record these inspection activities as closed in the PTRS database once they have completed the inspection, whether or not the risks identified in SEP have been mitigated, according to some principal inspectors we spoke with. While inspectors are expected to use the comments section in PTRS to record additional information, such as whether identified risks have been mitigated, they do so infrequently. Our analysis of about 1.8 million PTRS records showed that fewer than 400 records included any form of the words “mitigate,” “fix,” or “resolve.” Moreover, such analysis is time consuming to conduct because a uniform coding scheme is not employed for entering the information. As a result, managers are not easily able to use PTRS data to monitor whether risks identified through SEP have been mitigated and lack data that would be useful in evaluating SEP. It is left to inspectors to determine subsequently if the risks have been mitigated, usually through follow-up on- site inspections.

In addition, while a specific risk identified through SEP might result in multiple inspections, there is no clear linkage between these inspections when they are recorded in PTRS, according to a regional SEP manager. For example, if a risk was identified through SEP-generated inspection activities that included looking at manuals, checking aircraft records, and performing a ramp check, inspectors would have to review all comment records to identify the resulting activities in PTRS that are related to that particular risk. This lack of linkage in PTRS may reduce inspectors’ overall understanding of an airline’s situation. Inspectors who do not know why activities have been generated may not target or fit their work to assess these identified risks, while a greater understanding of how inspection activities are connected to identified risks could help ensure that inspections are conducted in a way that maximizes the benefits of a risk- based approach.

FAA Has Not Evaluated Its Inspection Oversight Process or Linked It to Agency Goals

FAA has not evaluated its inspection oversight process for non-legacy airlines to determine how the process contributes to the agency’s mission and overall safety goals. In addition, FAA has not explicitly linked SEP to the overall safety goals. According to management control standards for federal agencies and our prior reports on results-oriented management, federal agencies should establish measurable performance goals for their programs and operations. Agencies should also have an evaluation process for their programs so that agency officials, Members of Congress, and others will be able to determine whether goals are being achieved.

While FAA has not established a specific goal for SEP, it has an overall goal to achieve the lowest possible accident rate and constantly improve safety. To address this goal, FAA’s strategy includes expanding cost-effective safety oversight and surveillance by targeting its inspection resources better. However, FAA’s strategy does not explicitly show how SEP contributes to the safety goal. In addition, FAA has not yet evaluated SEP to determine if it is achieving the agency’s goal. As a consequence of not having an evaluative process for SEP, FAA does not have the information it needs to determine what changes should be made to improve its system safety process. Also, FAA does not have a process to examine the nationwide implications of or trends in the risks that inspectors have identified through their risk assessments. Consequently, FAA does not have the information it needs to proactively determine on a continuous basis risk trends at the national level. While FAA has an evaluation office, this office is not doing this type of analysis; rather, the office conducts analyses of the types of inspections generated under SEP by airline and FAA region, according to the manager of that office. One FAA field office we contacted had taken the initiative to analyze risks identified during the SEP meetings for trends. In FAA’s Eastern Region, a manager routinely analyzes trends in types of findings within an airline and across airlines and provides this information to the inspectors. This is an informal process, and it is not certain that it will continue since the manager has been reassigned to another office. Moreover, such analyses at the local or regional level do not fulfill the management control to have a process in place to determine if program goals are being achieved.

In addition, we found that while FAA has created specific national goals for NPG that call for completing all required inspections each year, in some cases these goals impede the agency’s ability to effectively implement its system safety approach. For example, in order to meet the annual goal, the regional offices have established interim goals to complete 25 percent of the required inspections each quarter. Inspectors in one field office told us that this situation created a disincentive for identifying additional activities under SEP due to concerns about completing the quarterly requirements. Those inspectors also noted that these goals encouraged them to prioritize their work based on what they can most easily accomplish, rather than on what represents the most significant risks—the antithesis of a system safety approach.

Conclusions

Our review of FAA’s oversight of non-legacy airlines suggests that the full benefits of a system safety approach can be realized only if the approach is more fully implemented and utilized. FAA’s articulation of its system safety approach, its application of system safety principles to its oversight process through SEP, and its widely distributed training on system safety are positive steps toward improving oversight by using the advantages of system safety—particularly the ability to identify and prioritize inspections based on risk. Fully developing SEP is important since FAA has been unable to move significant numbers of airlines to ATOS and has only recently established the goal to do so by the end of fiscal year 2007. A process such as SEP is needed to identify risks in and among non-legacy airlines and system-wide problems. However, the usefulness of FAA’s system safety approach is reduced by limitations in the implementation of SEP—such as FAA headquarters’ predominant focus on NPG, which has led to only a small percentage of inspection activities being SEP-initiated; the lack of training of geographic inspectors on SEP codes that could make their inspections more useful to identifying risks during the SEP process; and the lack of linkage to national goals and evaluations of SEP. Until SEP is more fully implemented, it is clear that FAA’s approach to overseeing non-legacy airlines is not largely risk based. Unless SEP is more thoroughly integrated into its oversight, FAA may not be fully maximizing the efficiency of its inspections in identifying and mitigating risks in order to ensure the safety of non-legacy airlines. With FAA operating under a hiring freeze and the number of inspectors available for non-legacy airlines possibly further reduced by attrition and the move of inspection staff to the ATOS program, the need to maximize the effectiveness of inspection activities in ensuring the safety of non-legacy airlines is even more critical.

Recommendations for Executive Action

To improve the effectiveness of the agency’s oversight of non-legacy airlines, we recommend that the Secretary of Transportation direct the FAA Administrator to implement the following four recommendations: To improve its safety oversight of airlines, FAA should (1) develop a continuous evaluative process for its activities under SEP and link SEP to the performance-related goals and measures developed by the agency, track performance towards agency goals, and determine appropriate program changes. The evaluation should include an analysis of inspection findings to identify trends and risks at the national level.

In order to ensure that all regional and field offices have a complete and timely understanding of FAA’s policies relating to the inspection process, FAA needs to (2) improve communication in areas such as whether and how internal risks identified by inspectors have been resolved and (3) improve training in areas such as risk management, coding items in the PTRS database, and how and under what circumstances SEP-identified activities can replace NPG-identified activities through retargeting.

To better utilize geographic inspectors’ support, FAA needs to (4) improve the geographic inspectors’ understanding of the system safety approach and operations of the airlines they inspect. FAA should consider actions such as additional training, additional oversight in particular areas, having airlines’ operating manuals available online for review by inspectors, and improving communication between geographic inspectors and principal inspectors on issues related to identifying safety violations.

Agency Comments and Our Evaluation

We provided a draft of this report to DOT for review and comment. We obtained oral comments from FAA and DOT officials, including FAA’s Deputy Associate Administrator for Aviation Safety. In particular, FAA officials made the point that planned NPG activities have an element of risk identification. We agree and revised the report. The FAA officials generally agreed with our recommendations to improve communication, training, and geographic inspectors’ understanding of the system safety approach. The officials said that FAA would consider our recommendation to develop a continuous evaluative process for SEP and link it to the agency’s goals, but that its plan to eventually place the remaining non-legacy airlines in the ATOS program might make this recommendation unnecessary. In the past, FAA’s efforts to move airlines to ATOS have experienced delays, therefore, we retained this recommendation. FAA officials also provided clarifying comments and technical corrections, which we incorporated as appropriate.

As agreed with your office, unless you announce the contents of this report earlier, we plan no further distribution until 14 days from the report date. At that time, we will send copies of this report to other congressional committees, the Secretary of Transportation, and the Administrator, FAA. We will also make copies available to others upon request. In addition, the report will be available at no cost on GAO’s Web site at www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-2834 or dillinghamg@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III.

Objectives, Scope, and Methodology

The objective of this report is to assess the Federal Aviation Administration’s (FAA) processes for ensuring the safety of a fast-growing portion of the commercial airline industry—the non-legacy passenger airlines. Specifically, this report responds to the following questions: (1) What are the strengths of FAA’s inspection oversight of non-legacy passenger airlines? (2) What issues hinder the effectiveness of FAA’s inspection approach?

The scope of our review included FAA’s oversight activities from fiscal year 2002—when FAA implemented a new process (the Surveillance and Evaluation Program or SEP) to its oversight of non-legacy airlines— through fiscal year 2004. To address the two questions, we obtained and analyzed information about FAA’s oversight of the top 25 non-legacy airlines ranked according to the number of enplanements in 2004. Those 25 airlines are identified in table 5, and each was covered by SEP during the entire 3-year period or a portion of the period. Together, these 25 airlines accounted for about 90 percent of all non-legacy passenger enplanements in 2004.

To determine FAA’s legal oversight responsibility for commercial passenger airlines, we obtained and analyzed regulations that govern FAA oversight of these airlines. We also obtained and reviewed FAA handbooks, procedures, and orders that describe the role and responsibilities of FAA managers and inspectors in implementing the two inspection oversight processes for non-legacy airlines—SEP and the National Work Program Guidelines (NPG). We conducted literature searches and reviewed prior reports and articles on the oversight processes, including those prepared by the Department of Transportation’s Inspector General and us. We reviewed documentation provided by FAA, and contacted officials there and at GAO, to determine whether findings identified in those reports have been addressed. We also reviewed literature on system safety and compared it with FAA’s system safety framework, which is incorporated in SEP.

We interviewed FAA headquarters officials from the Office of Aviation Safety and its Flight Standards Service to obtain descriptions of NPG and SEP. To understand how these processes were implemented, we collected information from 7 regional offices and 13 field and certificate management offices. We selected these offices because they have oversight responsibility for most of the top 25 non-legacy airlines ranked by the number of enplanements in 2004. We also interviewed officials from 16 of the 25 non-legacy airlines. The regional offices, field and certificate management offices, and airlines we interviewed are shown in table 6. For each FAA office, we conducted semistructured interviews with managers, SEP coordinators, and inspectors to obtain information on how they implemented NPG and SEP. We also collected information on inspector staffing levels, workload, and training. In addition, we reviewed whether management controls have been established for NPG and SEP, and determined if they were linked to FAA’s overall safety performance goals. These controls included the (1) establishment of performance goals, (2) verification of inspection results, (3) management of the inspection database (Program Tracking and Reporting Subsystem or PTRS), and (4) communication among managers and inspectors relating to NPG and SEP.

We also interviewed airline safety officials to obtain their views on FAA’s inspection and surveillance activities and their participation in FAA- industry partnership programs.

We also conducted interviews with seven industry organizations to obtain their views on FAA’s oversight processes. (See table 7.) We selected these organizations because they represented diverse segments of the aviation industry. In addition, we selected organizations that met one or more of the following criteria: membership inspected by FAA, familiarity with FAA’s safety oversight of airlines, familiarity with FAA’s SEP or Air Transportation Oversight System processes, and familiarity with inspector training.

We analyzed information about FAA’s inspection activities for the top 25 non-legacy airlines using the agency’s PTRS database for fiscal years 2002 through 2004. Specifically, we analyzed information on the number of required NPG and SEP inspections, inspector workload (i.e., number and type of entities inspected), and number of NPG activities that were retargeted (i.e., NPG-identified activities that were replaced with SEP- identified activities). This analysis excluded enroute activities, which take place when an inspector monitors an aircraft as it travels from one destination to another. These inspections were excluded because inspectors often use these as secondary inspections as a means to travel to a location where their primary inspection is such as a repair station. In addition, we analyzed data that FAA maintains on the internal risks identified by inspectors through regular SEP meetings. For fiscal years 2002 through 2004, we examined the type, severity, and closure status of the identified risks. We also analyzed comment fields in that data to determine if they indicated that the risks had been mitigated.

To assess the reliability of the data used in this report, we interviewed knowledgeable agency officials about the data, performed electronic testing of relevant data fields for obvious errors in accuracy and completeness, and collected and reviewed documentation from data system managers about the data and the systems that produced them. We determined that the data were sufficiently reliable for the purposes of this report.

To gather information about inspectors’ perspectives on the technical training available to them, we conducted a Web-based survey of a representative sample of FAA safety inspectors. The survey asked a combination of questions concerning the amount and timeliness of training that allowed for open-ended and close-ended responses. We had no practical way to assess information on the amount of training necessary for inspector proficiency or the timeliness of the training provided. We drew a stratified random probability sample of 496 inspectors from the population of 2,989 FAA aviation safety inspectors. We stratified the population into 12 groups on the basis of the type of work the inspector performed. Each sample element was subsequently weighted in the analysis to account statistically for all members of the population. For this report, we used a subgroup sample of 205 non-legacy safety inspectors.

Because we followed a probability procedure based on random selections, our sample is only one of a large number of samples that we might have drawn. Since each sample could have provided different estimates, we express our confidence in the precision of our overall sample’s results as a 95 percent confidence interval (e.g., plus or minus 7.1 percent). This is the interval that would contain the actual population value for 95 percent of the samples we could have drawn. As a result, we are 95 percent confident that each of the confidence intervals in this report will include the true values in the study population. The percentage estimates for the non-legacy subgroup has a margin of error of plus or minus 7.1 percent or less. Survey estimates presented as comparisons between groups are statistically significant when the 95 percent confidence intervals do not overlap.

The survey was conducted using self-administered electronic questionnaires accessible on the Internet through a secure Web browser. We sent e-mail notifications to 496 inspectors (205 of which were in our subgroup of inspectors of non-legacy airlines) beginning on December 6, 2004. We then sent each potential respondent a unique password and user name to ensure that only members of the target population could participate in the survey. To encourage respondents to complete the questionnaire, we sent a subsequent e-mail message to further prompt each nonrespondent approximately 2 weeks after the initial e-mail message. We sent nonrespondents two more notices and closed the survey on February 4, 2005. Of the 496 inspectors we surveyed, we received 392 usable responses (79 percent). Among our subgroup of 205 non-legacy inspectors, we received 161 usable responses (79 percent).

In addition to sampling errors, the practical difficulties in conducting surveys of this type may introduce other types of errors, commonly referred to as nonsampling errors. For example, questions may be misinterpreted, or the respondents’ answers may differ from those of the inspectors who did not respond. We took steps to reduce these errors.

Finally, we pretested the content and format of the questionnaire with safety inspectors at local FAA offices in Baltimore, Los Angeles, and Seattle. During the pretests we asked the inspectors questions to determine whether (1) the survey questions were clear, (2) the terms used were precise, (3) the questionnaire placed an undue burden on the respondents, and (4) the questions were unbiased. We made changes to the content and format of the final questionnaire based on the pretest results.

We conducted our work from August 2004 through September 2005 in accordance with generally accepted government auditing standards.

FAA’s Surveillance and Evaluation Process

SEP Incorporates System Safety into FAA’s Inspection Oversight of Non- legacy Airlines

The Surveillance and Evaluation Program (SEP) is a process designed to introduce a data driven risk analysis system for non-legacy airlines and is guided by a Surveillance and Evaluation Assessment Tool (SEAT). SEP models itself on a system safety approach that is incorporated in the oversight of legacy airlines through the Air Transportation Oversight System (ATOS). SEP allows teams of inspectors to identify suspected trends through a data based system rather than relying totally on past experiences. A certificate management team oversees each of the 99 airlines under SEP. Each team is led by three principal inspectors, one for each major area of inspections (operations, maintenance, and avionics). Additional team members include those based at the FAA office that holds the airline’s operating certificate—typically an aircraft dispatch inspector, a cabin safety inspector, and assistants.

The principal inspectors are required to meet as a team at least twice a year to assess the safety risks associated with the airlines they oversee and develop a surveillance plan for the airline, including designating inspection duties to other FAA offices at locations to which the airline flies or conducts business. In some field offices, teams meet each quarter. The teams use a planning tool—SEAT—to assess the systems in place at an airline and to identify any potential internal and external risks. Using SEAT, the team analyzes 10 systems, shown in table 8.

To complete SEAT, the principal inspectors rely on their knowledge of the airline and on the data available through FAA’s Safety Performance Analysis System database, which contains the Program Tracking and Reporting Subsystem and 12 additional databases with safety and performance information. SEAT includes a set of risk indicators for each of the elements (often in the form of a question) to be discussed by the team to indicate concerns about any real or potential problem that could contribute to the failure of one of the airline’s elements, subsystems, or systems. Inspectors rate each risk they identify in terms of severity (negligible to catastrophic) and likelihood of occurrence (improbable to frequent), and SEAT calculates an overall risk rating. Risks rated high have high levels of likelihood and severity.

Using this information, the team identifies a set of required inspections that must be completed during that fiscal year and planned inspections. The required inspections are added to the NPG-identified inspections, and the combined list becomes the annual surveillance plan for the airline. During the SEAT process, principal inspectors are allowed to substitute NPG- required inspections with SEP-identified inspections targeting higher-risk areas.

SEP Incorporates Some Elements of ATOS, While NPG Relies on a Set Number of Inspections

Table 9 describes FAA’s three inspection processes for overseeing legacy and non-legacy airlines: ATOS, NPG, and SEP. Many of the elements of ATOS, such as the use of data to identify risks and the development of surveillance plans by inspectors, are incorporated in the SEP process. The NPG process, in contrast, is not focused on the use of data and relies on an established set of inspections that are not risk based.

GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the above, Teresa Spisak, Assistant Director; Carl Barden; Bari Bendell; Curtis Groves; Bert Japikse; Catherine Kim; Minette Richardson; Phillis Riley; Larry Thomas; Alwynne Wilbur; and Matt Zisman made key contributions to this report.