Aviation Safety: Improved Data Quality and Analysis Capabilities Are Needed as FAA Plans a Risk-Based Approach to Safety Oversight
Released: 2010-06-10T00:00:00

Letter

Background

Federal agencies and aviation industry stakeholders gather and analyze aviation data for a variety of purposes. Federal agencies gather and analyze aviation data primarily to improve safety. To oversee aviation safety across the national airspace system, FAA maintains data on various aviation sectors, including passenger airlines, air cargo carriers, general aviation, and air ambulance operators. FAA also gathers and analyzes data on industry performance through its inspection and certification programs and uses these data to ensure that the industry complies with its safety regulations. In addition, FAA obtains information on safety events and incidents collected by other federal agencies, including NTSB, NASA, and USDA.

The aviation industry gathers quantitative and narrative data on the performance of flights and analyzes these data to increase safety, efficiency, and profitability. Industry stakeholders also maintain historical data on equipment and maintenance issues. These stakeholders are required to report some data to FAA, such as data on accidents, engine failures, and near midair collisions, and they have agreements with FAA and other agencies to share other data voluntarily. The voluntarily shared data include both electronically recorded data from aircraft equipment under the Flight Operational Quality Assurance program (FOQA) and information on violations of federal regulations or on safety events self- reported by pilots, mechanics, and other airmen under three programs— Aviation Safety Action Program (ASAP), Aviation Safety Reporting System (ASRS), and Voluntary Disclosure Reporting Program (VDRP). FAA also recently established the Air Traffic Safety Action Program (ATSAP), modeled after the airlines’ ASAP program. It is a confidential, voluntary reporting system available to FAA’s approximately 17,000 air traffic control personnel, who can use the program to identify and report safety and operational concerns.

Table 1 describes the 13 aviation safety databases that we reviewed.

FAA Analyzes Data on Past Safety Events to Prevent Their Recurrence and Plans to Use Data to Support a More Proactive Approach to Managing Risks

FAA Is Shifting from a Reactive to a Proactive Approach to Using Data to Manage Risk

For decades, the aviation industry and federal regulators, including FAA, have used data reactively to identify the causes of aviation accidents and incidents and take actions to prevent their recurrence. Since 1998, for example, FAA has partnered with the airline industry through CAST with the goal of continuously improving aviation safety. Over the years, CAST has looked at the causes of past accidents—such as controlled flight into terrain—and various safety events—such as turbulence or runway incursions. CAST analyzes past instances of such accidents and events to identify precipitating conditions and causes. CAST then uses its analysis to formulate an intervention strategy designed to reduce the likelihood of a recurrence and validate the effectiveness of the intervention. According to CAST, its work has helped to decrease commercial airline fatalities— exceeding its goal to reduce fatal commercial accidents by 80 percent by 2007—and is an important aspect of FAA’s efforts to increase aviation safety by sharing and analyzing data. Table 2 provides examples of how FAA and industry have used CAST’s work. (Recent work by CAST to work with FAA’s Aviation Safety Information Analysis and Sharing initiative to develop safety enhancements and mitigate future threats is discussed later in this report.)

Besides analyzing data on past safety events to develop intervention strategies, FAA staff perform such analyses to inform changes in agency policies. For example, to inform the rule-making requirement that the costs and benefits of a proposed regulation be determined under Executive Order 12866, FAA analysts identified the number of aircraft that could be certified as “light sport” aircraft and were involved in accidents.

In some cases, FAA program managers request specific analyses to inform policy changes. For instance, in response to a 2007 recommendation by NTSB and a petition from Hawaiian air tour operators, FAA program managers requested an analysis of aircraft crashes associated with FAA’s requirement for tour aircraft to maintain a 1,500-foot separation from the ground. After analyzing air crashes involving Hawaiian tour aircraft 13 years before and 13 years after the requirement was implemented, FAA concluded that the requirement helped to reduce the number of crashes and significantly improved safety.

While FAA will continue to use data to analyze past safety events, it is also working to use data proactively to search for risks and take actions to mitigate them before they result in accidents. FAA’s emphasis is shifting to a proactive approach to data analysis because as accidents have become increasingly rare, less information is available for reactive analyses of their causes. As a result, according to a study of FAA’s safety oversight by a 2008 independent review team, information that can be used to help identify accident and incident precursors, such as voluntarily reported data, has become more critical for accident prevention. In addition, several experts we spoke with said that proactively identifying risks is necessary to maintain the current level of safety and possibly achieve an even higher level of safety in the future. FAA is undertaking this transition in coordination with the international aviation community, working with the International Civil Aviation Organization (ICAO) to adopt applicable global standards for safety management. Senior FAA officials and ICAO agree that effective safety management is data driven and that data are essential to identifying emerging risks. Figure 1 illustrates the type of transition FAA plans as the agency shifts its emphasis to a proactive assessment of emerging safety risks, according to FAA officials.

The new technologies and procedures that FAA will implement for NextGen, which are intended to increase the safety, efficiency, and capacity of the national airspace system, could also lead to consequences that have unintended effects on system safety. For example, NextGen changes to landing procedures, which are designed to allow more frequent landings, could reduce congestion in the air and improve fuel efficiency, but might have the unintended effect of increasing congestion and safety risks on airport taxiways. To avoid such unintended consequences, FAA plans, as it improves its ability to integrate data and analyze trends, to model the impact of changes planned for NextGen. To do so, it has begun to develop a baseline of current conditions and then expects to analyze how NextGen changes will affect those conditions, according to a senior FAA official. FAA is in the process of designing tools that will allow it to model the changes. For example, a project called the National Level System Safety Assessment is designed to allow FAA to assess risks across the national airspace system. Currently, FAA assesses risks for specific NextGen procedures and technologies, but cannot model the risks across the national airspace system in a comprehensive manner. This project will integrate data on past safety events from a number of FAA offices and external sources to proactively identify risks that might emerge with the introduction of changes planned for NextGen. FAA has begun to obtain some operational data for the project and has contracted with the Volpe National Transportation Systems Center, which will be responsible for integrating airport runway surface data, including surface radar, weather, aircraft, and other data. A senior FAA official told us that although safety assessments had been conducted on individual NextGen technologies, until the agency has finalized this modeling project, it cannot begin systemwide assessments of the safety of NextGen technologies and procedures that are already being deployed, including 700 new navigational procedures that had been deployed as of October 2009.

SMS and Its Use of Data Form the Linchpin of FAA’s Risk-Based Approach, but Full Implementation of SMS Could Be Years Away

SMS is an integrated, data-driven approach to managing safety risk that FAA expects will help it continuously improve aviation safety. FAA stated that successfully implementing SMS is critical to meeting the challenges of a rapidly changing and expanding aviation system. As stated earlier in this report, FAA’s traditional approach is to analyze data to determine the causes of an accident or incident after the fact. To achieve the next level of safety, FAA says, it now requires a more forward-thinki ng approach, which SMS provides, to identify systemwide trends and manage emerging risks before they result in incidents or accidents. To identify emerging risks, FAA plans to collect and analyze safety data, and it can then use the results of its analyses to make data-driven decisions about how to address safety risks. Issued in September 2008, FAA’s guidance on implementing SMS explains the importance of data collection and a to the execution of SMS. SMS: safety policy and objectives, safety risk management, safety assurance, and safety promotion. First, safety policy and objectives describe an organization’s requirements and oversight responsibilities f aviation activities. Second, safety risk management describes how an organization will identify hazards and safety risks in aviation operations, nalysis This guidance defines four main components for or including how it will develop rules, regulations, and safety performance measures. Third, the safety assurance component of SMS uses data analyses to discover emerging risks and to model the impact of safety changes. Fourth, safety promotion is an organization’s plan training, communication, and dissemination of safety information take place. According to FAA, SMS with these components will enable the agency to provide the air transportation system and the public at large with enhanced safety.

FAA’s goal is for the Office of Aviation Safety to have initial operating capabilities in place for SMS by the end of fiscal year 2010. According to FAA officials, these initial operating capabilities include training employees and defining how to apply SMS to the agency’s overall oversight activities. To accomplish this, FAA’s guidance for implementing SMS requires the Office of Aviation Safety, the Office of Airports, and the Air Traffic Organization (ATO) each to develop implementation plans that include schedules, procedures for acquiring and analyzing data, and measures to track implementation progress. ATO has issued its SMS implementation plan and has also created an SMS manual that provides specific operational information and guidance regarding the daily activities of ATO employees. In addition, it met its target to implement its initial SMS operating capabilities in March 2010. The Offices of Airports and Aviation Safety have yet to issue their implementation plans. However, the Office of Aviation Safety has issued guidance and safety documents that provide a general discussion of SMS, but they do not include a schedule of specific activities or time frames for completion as called for in the agencywide SMS guidance. Senior FAA officials told us Aviation Safety has formed working groups and expects those groups that are charged with defining the various SMS activities to meet the guidance requirements. In addition, according to FAA officials, various offices within Aviation Safety will be responsible for implementing processes to fulfill SMS requirements.

Additionally, FAA’s guidance for implementing SMS requires the formation of an agencywide SMS committee, which was chartered in July 2009. The committee includes an executive council—whose members are the FAA associate administrators from the offices of Aviation Safety, Airports, and Commercial Space Transportation, and the Chief Operating Officer of ATO—and a committee composed of SMS professionals from each of those FAA offices. Chaired by the Office of Aviation Safety, the agencywide committee is tasked with recommending policy and guidance to the implementing FAA organizations and management.

Because SMS relies on data to identify emerging risks, FAA has an effort under way to enhance its access to industry data and improve its analysis capability. The ASIAS initiative is a collaborative government-industry effort to share and analyze data. The Office of Aviation Safety’s SMS implementation plan reiterates that data exchanges between ASIAS and other sources are crucial elements of emerging risk analysis. FAA’s draft plan for ASIAS notes that this effort will require access to existing and previously unattainable data sources, enhanced analytical methodologies, and technical advancements to support safety risk analysis that are not achievable with current databases and analytical strategies. The draft plan would cover ASIAS activities through 2022. FAA did not confirm when a final plan would be completed.

While FAA has issued agencywide guidance on implementing SMS and has some efforts such as ASIAS under way, it does not have a way to measure or specific times to indicate full implementation. FAA officials told us that the current efforts would provide a foundation for the full implementation of SMS. But without a clear description of the activities to be completed and time frames for their completion, it may be years before SMS is fully implemented and its benefits are realized. In commenting on a draft of this report, FAA officials noted that even with a clear description of the activities to be completed and time frames for their completion, it will be years before SMS is fully implemented and its benefits are realized. We agree with FAA and note that specific time frames establish expectations for FAA’s implementation of SMS and provide a means of accountability for meeting those expectations.

FAA Has Identified and Is Addressing Challenges to Using Data for Safety Oversight

FAA has efforts under way to address two key challenges to using data more effectively to manage risk. First, data are not coded to permit electronic integration, analysis, and sharing. Second, data from two voluntary reporting programs lack identifying details needed for some types of analysis, and the data do not remain available for long-term analysis.

Data Are Not Coded to Permit Automated Analysis

FAA and the other federal agencies that gather the data FAA uses for analysis lack standard definitions, common identifiers, and common classification schemes for both quantitative and narrative data. For example, according to an NTSB official, databases used in ASIAS were not designed for searches of specific products, manufacturers, or airlines. As a result, FAA has had to develop common identifiers and standard classification schemes so that automation can be used to quickly integrate and analyze data from multiple external and internal sources. In addition, narrative information, in particular, poses challenges. Narrative reports from NTSB investigators and FAA inspectors, as well as narrative reports submitted voluntarily by pilots, air traffic controllers, mechanics, and others, constitute a rich source of information about safety events, but could not be coded to permit automated analysis until the ASIAS text analysis process was developed. Consequently, analysts are now able to automate coding, integration, and analysis of data from different databases—a time-consuming and costly process when manually performed. According to a senior FAA official, this automated analysis process is unique and pathbreaking and will allow for more efficient safety analyses. When FAA studied wrong runway departures, for example, data from six different databases—from NTSB, NASA, and its own offices— were extracted for analysis. FAA compared the cost and time needed to extract and integrate data from multiple databases using an automated versus a manual process for coding the data. FAA found that using the automated coding process would cost about $6,000 and take 52 hours, while the manual process was estimated to cost over $750,000 and take more than 6,500 hours to complete. However, NASA noted that by using a “data mapping” approach, it can conduct an automated analysis of ASRS and other data sources in about 2 to 8 hours, depending on the complexity of the analysis.

Data from Voluntary Reporting Programs Have Characteristics That Impede Analysis

Details of reported incidents are redacted from ASAP and FOQA data before an FAA contractor analyzes the data. These details include the date, time and flight number, and the names of the carrier or individuals involved. We previously reported that these identifying details are redacted to safeguard the participants in ASAP from enforcement or disciplinary action and participants in FOQA from public release of the data. Additionally, ASAP and FOQA data are retained for only 3 years. Without identifying details and without maintaining data for longer periods, opportunities for some analyses are limited. To allow the contractor to perform more detailed analyses for FAA, the agency and the industry have agreed on a process through which the ASIAS Executive Board provides permission for the contractor to perform a specific, defined analysis and to use data with the identifying details needed for that particular analysis. At the conclusion of the analysis and with the approval of the ASIAS Executive Board, a final report containing detailed results is given to CAST to develop safety enhancements to mitigate the identified safety issues. As of March 2010, three such analyses had been completed.

The Department of Transportation (DOT) Inspector General (IG) and the 2008 independent review team found that FAA could improve its use of ASAP data by analyzing these data for national trends, noting that by not doing so, the agency is missing opportunities to help reduce recurrences of safety events or to identify national patterns indicative of risks. FAA concurred with this finding and said that it already has the ability to conduct national trend analyses. The IG reported, however, that FAA receives quarterly summaries of ASAP information from carriers of how many ASAP reports they received each month, and these summaries do not provide sufficient detail about ASAP events or corrective actions. According to the IG, these reports generally contain information about the number but not nature of ASAP submissions for that quarter and any resulting safety enhancement.

While FAA’s contractor loses access to ASAP reports after 3 years, about 62 percent of ASAP reports appear in ASRS, along with other reports voluntarily submitted by industry personnel, according to a NASA official. NASA, which maintains ASRS for FAA, has access to the identifying details in the submitted reports for no more than 90 days so it can follow up as necessary to clarify any questions that the reports raise. After that time, NASA removes the identifying details and incorporates ASAP and ASRS reports into ASRS. ASRS is accessible to the public, and NASA performs special analyses of ASRS data for FAA at its request. NASA also publishes an online newsletter every month with summary statistics and examples of a few reports selected to illustrate certain safety issues, such as general aviation pilots’ use of new cockpit technology, and issues safety alerts regarding significant safety events identified by ASRS reports. However, NASA does not comment on these reports or respond to any questions or issues raised by the authors, since its role is to act as an “honest broker” as it collects and analyzes the data. By comparison, in the United Kingdom (UK), an advisory board reviews a selection of reports received, commenting on the appropriateness of the action taken and answering questions. These responses are useful to the aviation community because they communicate on commonly experienced safety issues. In commenting on a draft of this report, NASA indicated that the UK model would not be practicable for ASRS data because the UK receives only about 300 reports annually compared with about 50,000 reports for ASRS. However, NASA noted that in the past it had an ASRS advisory committee that had provided a forum for FAA and industry to discuss corrective action. The agency acknowledged the need to reestablish this committee. In addition, NASA noted that it receives feedback on its safety alerts, which indicated positive corrective actions for about 60 percent of the alerts.

FAA Is Developing a Plan for Using Data under SMS, but the Plan Does Not Fully Address Data, Analysis, and Staffing Requirements

As part of its efforts to develop initial SMS capabilities, FAA expects to address how data and analysis will help it identify emerging aviation safety risks. Specifically, FAA has a draft plan—which it calls a safety management plan—that defines the agency’s analytical requirements and the role of safety analysis in improving safety, especially as NextGen is implemented. FAA’s draft plan—which covers the risk management and safety assurance components of SMS discussed earlier in this report— recognizes the agency’s future need for data and analysis, but does not specify requirements for them. For example, the draft safety management plan does not define the level of accuracy and completeness needed for the data, indicate what metrics and processes FAA will use to assess the data, or identify any specific data. According to senior FAA officials, as of February 2010, there was no specific date for finalizing the draft plan.

To meet its data challenges and develop needed analytical approaches, FAA will also have to identify staff with both aviation operational experience and statistical expertise to effectively analyze aviation safety data in the future. While aviation safety experts we interviewed were generally satisfied with the qualifications of current FAA analysts, they expressed concerns about FAA’s capacity to meet future needs. Several experts we interviewed said FAA needs additional qualified analysts, but the draft safety management plan does not mention staffing requirements for implementing FAA’s analysis strategy. In our 2009 report on FAA’s human capital system, we made several recommendations to FAA on how it can help ensure the continued hiring, recruitment, and retention of staff needed to operate the national airspace system. According to FAA, it has not yet determined how many additional analysts it will need; however, the Office of Accident Investigation and Prevention has been approved to hire 11 additional analysts in fiscal year 2010. Without a plan that includes data and analysis requirements and staffing needs, the agency will not be able to link the resources it needs to the data capabilities it requires for its risk-based approach.

FAA Has Limited Access to Some Voluntarily Reported Data and Lacks Key Data to Assess the Safety Performance of Certain Industry Sectors

Voluntary Reporting Programs Generate Safety Information That Is Not Available from Other Sources, although the Data Have Some Limitations

prolem; withot them, the rik level wold go p.

FAA’s voluntary reporting programs—ASAP, ASRS, FOQA, and VDRP— generate safety information that FAA does not identify through other means. Whereas data from other sources are derived from inspections, audits, and other agency reports, these voluntary programs rely solely on cooperation between FAA and industry personnel. To obtain voluntarily reported information that can be used to improve safety, FAA agrees not to take enforcement action against carriers or industry personnel who self- report violations through ASAP, ASRS, and VDRP. Similarly, carriers that operate ASAPs agree not to take disciplinary action against personnel who self-report violations of FAA regulations or carriers’ operating procedures. In both cases, this agreement holds only for actions that are reported in a timely manner, were not intentional or criminal, did not involve drugs or alcohol, did not result in accidents, and have not already been detected by FAA. Conversely, personnel who do not voluntarily report violations within the specified time face the threat of enforcement or disciplinary action if the violations are discovered later. This combination of promised immunity for self-reporting and threat of enforcement and disciplinary action for remaining silent creates an incentive for industry personnel to participate in the voluntary reporting programs.

Through ASAP, ASRS, and VDRP, airspace users, including air carriers, air operators, and employees, self-report safety events and violations of their operating certificates or company procedures. For example, under ASAP, an employee reports an incident or event, which an event review committee, composed of representatives from FAA, the carrier, and the applicable employee group, then reviews. The ASAP event review committee assesses a report to determine (1) if it meets the criteria previously mentioned for inclusion in the program and (2), if included, what follow-up actions, enhancements, or mitigations should be implemented to address the safety concern.

While ASAP, ASRS, and VDRP rely on individuals or the air carrier to file reports, FOQA data are generated by electronic equipment on aircraft, which continuously records more than a thousand parameters of data for individual flights. Vendors collect the data from carriers and can then, on their behalf, analyze the data and transfer files to the carriers’ internal analysis teams or forward the data files to an FAA contractor for inclusion in ASIAS and subsequent analysis if a carrier is partnering with FAA. The FAA contractor also receives ASAP reports approved by event review committees. The contractor aggregates and analyzes the data from participating carriers and, starting in 2010, will brief the ASIAS Executive Board on a quarterly basis. These briefings will consist of the status reports on directed studies, the number of FOQA and ASAP records, and industry benchmarks that will enable carriers to compare their individual safety performance relative to the national trends and prioritize their internal safety initiatives. According to FAA, 28 carriers participate in ASIAS. All ASIAS participants share ASAP data, and 13 carriers also share FOQA data. FAA estimates that airlines contributing ASAP data to ASIAS account for 80 percent of the flights of all commercial airlines with FAA- approved ASAP programs.

FAA and industry officials, as well as experts we talked to, agreed that voluntarily reported data are critical to improving aviation safety. Moreover, according to ICAO, FAA officials, and safety experts we interviewed, voluntary reporting by operational personnel is a cornerstone of SMS, because, as ICAO has stated, operational personnel are in the best position to report the existence of safety hazards and to attest to what works and does not work during everyday operations. In 2007, NTSB reported that FOQA and ASAP programs are relevant to the safety assurance component of SMS because they provide a direct means for air carriers to evaluate the quality of their training and operations. Further, these programs can also be used in the safety risk management component of SMS because they can help identify emerging risks. In addition, FAA and industry officials agreed that voluntary programs help promote a healthy reporting culture and an increased awareness of safety by industry personnel. Furthermore, FAA officials told us they believe that the voluntary programs, such as ASAP, gather safety information that would not be discovered 95 percent of the time. In addition, in commenting on a draft of this report, NASA noted that voluntarily reported data are valuable in learning why an event occurred.

Industry officials told us how their companies have used voluntarily reported data to implement changes that respond to safety concerns. For example, one airline analyzed ASAP data to decrease the number of unstable approaches. Safety officials from another airline told us that ASAP reports and FOQA data helped them to identify potential pilot issues, suggest additional training, and adjust processes and checklists based on human factors issues. They further commented that if pilots do not self-disclose potential safety issues, airlines may be limited in their ability to identify emerging safety trends.

Because of the importance of voluntarily reported data to proactive safety analysis, NTSB and the DOT IG have also recommended that FAA further encourage participation in voluntary programs. For example, in 2010, NTSB recommended that FAA require the establishment of FOQA programs by carriers regulated under part 121. As another example, the DOT IG reported in May 2009 that FAA was not realizing the full benefits of ASAP and recommended that the agency develop a central database for all air carriers’ ASAP reports to be used for nationwide trend analysis.

While voluntarily reported data have been used to enhance safety, they also have some limitations. First, the completeness of the data is unknown, since reporting is voluntary, and there is no way to know how many violations or safety situations are not reported. For example, according to FAA, factors such as an individual’s awareness of ASRS, motivation to report a situation, and perception of an incident’s severity may influence the decision to submit an ASRS report. Second, the completeness of the data is further limited because participation varies among airlines, with 73 airlines participating in ASAP and 36 participating in FOQA. Third, as discussed later in this report, inadequate data quality controls can also limit the completeness of the data. For example, controls may not be adequate to ensure that the data entered into a database have been accurately compiled or processed. Fourth, the accuracy of voluntarily reported data cannot always be verified. Voluntarily reported data are subjective and are not always accompanied by supporting documentation, such as statistics, measurements, or other quantifiable information related to the reported events. According to an FAA analyst, a variety of factors can influence the accuracy of the data, including the reporter’s experience, visibility conditions, the duration of the event, and any trauma experienced by the reporter. FAA notes, for example, that even senior pilots’ estimates of how far aircraft descend during encounters with turbulence often differ considerably from the actual distances recorded on aircraft flight data recorders. Acknowledging this limitation, NASA notes that the information is nonetheless valuable, since a reporter is providing information on how it perceived the situation, which in large part determines its reactions. Fifth, electronically collected data from FOQA also have limitations. Vendors that process FOQA data explained that software bugs and inaccurate sensors can affect data results. To mitigate these problems, vendors review anomaly reports and validate data prior to analysis.

In an effort to address the limitations of voluntarily reported data, NASA developed a survey methodology project—NAOMS—in 1997 to systematically collect information on safety events by conducting telephone interviews with randomly selected airspace users such as pilots. In our 2009 assessment of the survey, we concluded that NAOMS was a successful proof of concept and that a similar project, adequately funded and appropriately planned, could enhance FAA’s safety knowledge. However, FAA maintained that FOQA provides better data by providing precise rates of occurrence on multiple parameters collected by flight data recorders that could obviate the benefits from NAOMS data. Nonetheless, we concluded that the NAOMS survey could be useful in complementing other databases, such as ASRS. The survey data, when properly analyzed, could be used to call attention to low-risk events that could serve as potential indicators for further investigation in conjunction with other data sources. Furthermore, in commenting on a draft of this report, both NTSB and NASA agreed on the usefulness of a survey similar to NAOMS in complementing other data. NASA pointed out that NAOMS-type data could provide the data for trends and explain “what” is happening in the system while ASRS provides “why” it is happening. NTSB further noted that its investigations of numerous serious incidents and accidents found that FOQA data gave no indication of underlying problems.

Participants’ Concerns Limit FAA’s Access to Voluntarily Reported Data

Although FAA, carriers, and experts we interviewed agreed that voluntarily reported data are an important source of information for understanding and addressing safety issues, some carriers and industry personnel are not participating in FAA’s ASAP and FOQA voluntary reporting programs. According to airline and FAA officials, two factors have primarily affected participation: (1) the fears of employees that their employers will take disciplinary action to address self-reported violations and (2) the costs to the airlines of purchasing and installing FOQA equipment and analyzing the data.

Specifically, we found that, partly because of employees’ fears of disciplinary action, from 2006 through 2008, four large air carriers and their pilot unions suspended their ASAP. According to safety officials at one airline, pilots had raised concerns about letters of reprimand or unpaid time off that directly resulted from ASAP reports. Pilot unions and air carriers that we spoke with agreed on the importance of confidentiality. However, pilot union officials we interviewed also expressed concern about whether airlines were ensuring the confidentiality of ASAP reports and suggested that, for an airline’s safety culture to improve, pilots, in particular, must be able report certain events without fear of reprisal. Despite these concerns, as of June 2009, the four carriers and pilot unions that had suspended their ASAP had restarted or were restarting their participation in the program.

Additionally, according to industry officials and experts we interviewed, the costs of purchasing and installing equipment and analyzing data have deterred participation in FOQA, especially for smaller carriers. Several large carriers we interviewed said more than 50 percent of their fleets already have FOQA equipment and they plan to expand their fleets’ capability by retrofitting aircraft or ensuring that new aircraft include the equipment. By contrast, officials from smaller carriers were concerned about costs and estimated that installing FOQA equipment would cost an average of $12,000 for each new aircraft and up to $35,000 for retrofits of older aircraft models. As a result of cost concerns, according to airline officials, only 11 of 65 smaller carriers have approved FOQA programs, and according to FAA, an additional small carrier FOQA program is pending. Officials from these smaller carriers said that incentives to cover equipment costs, which are not currently available, would help increase participation. FAA officials noted that as carrier fleets age, newer replacement aircraft will already be fitted for FOQA equipment and, therefore, costs for participating will continue to decrease. However, the life span of an aircraft is usually at least 30 years for large carriers, so the transition to a fully equipped fleet could take decades.

Currently, large carriers are the principal participants in FOQA and ASAP, and they provide service for the majority of passengers on domestic and international flights. Nonetheless, we found that 4 of the 25 large carriers do not have active FOQA programs and 1 large carrier did not have an ASAP. According to FAA officials, an additional 4 carriers have FOQA programs pending. To the extent that operators do not participate in the programs, they do not obtain information that they could use to monitor and improve the safety performance of their aircraft, related equipment, and personnel, and to the extent that they do not partner with FAA, opportunities to identify nationwide safety trends and improvements are lost. To encourage greater participation in FOQA and ASAP, FAA provides training to smaller carriers on how to develop versions of both programs that do not require as much capital investment but do allow the carriers to collect unique safety data. However, FAA lacks carrier-specific information on why air carriers are not participating in voluntary reporting programs. Having such information would allow FAA to identify further actions to encourage participation.

FAA Lacks Data to Assess the Safety of Certain Industry Sectors

FAA’s ability to monitor and manage risk for certain industry sectors, such as general aviation, air ambulance operators, and air cargo carriers, is limited by incomplete data. While FAA collects data on actual flight hours and numbers of departures for large air carriers that operate under part 121 and scheduled flights with fewer than 10 seats that operate under part 135, it does not collect actual flight activity data for smaller air carriers that provide on-demand service, such as air taxis and air ambulances, and general aviation operators—sectors that have had a higher number of fatal accidents in recent years than large air carriers. For instance, in 2008, large air carriers providing scheduled service had 20 accidents, none of which were fatal. By comparison, during that same year, there were 1,559 general aviation accidents, including 275 fatal accidents involving 495 fatalities. (Fig. 2 shows the trends in general aviation accidents.) Without information on the number of general aviation flights, FAA cannot compare the safety performance among industry sectors or assess trends.

Additionally, the number of accidents for air ambulance and air cargo operators points to safety vulnerabilities in these areas. From 1998 through 2008, the air ambulance sector averaged 13 accidents per year.

While the total number of air ambulance accidents peaked at 19 in 2003, the number of fatal accidents peaked in 2008, when 9 fatal accidents occurred. Without data on the number of flights or flight hours, FAA and the air ambulance industry are unable to determine whether the increased number of accidents has resulted in an increased accident rate, or whether it is a reflection of growth in the industry. According to FAA, it annually surveys a sample of potentially active general aviation aircraft, and in the latest survey in 2008, it surveyed all air ambulance operators. However, we noted in our April 2009 testimony that less than 40 percent of the air ambulance operators responded, raising questions about the reliability of the information collected. Similarly, our review of air cargo safety found that small cargo carriers had more accidents and fatal accidents than large cargo carriers, but the available information was not sufficient to assess the significance of this difference. We found that smaller air cargo carriers averaged 29 accidents per year from 1997 through 2008, while large cargo carriers averaged 8 accidents each year during this period. However, a lack of operations data for small cargo carriers makes it difficult for FAA to prioritize risks and better target safety improvements and oversight to the areas of highest risk.

To address the lack of data, we previously recommended that FAA identify the data necessary to better understand the air ambulance industry and develop a systematic approach for gathering and using these data. Similarly, we recommended that FAA gather comprehensive and accurate data on smaller air cargo operations (those covered under part 135) to gain a better understanding of air cargo accident rates and better target safety initiatives. FAA agreed with both recommendations, but has not fully addressed either. In response to our recommendation on air ambulance data, FAA has surveyed all helicopter air ambulance operators to collect flight activity data. However, as mentioned earlier in this report, FAA’s survey response rate was low, raising questions about whether this information can serve as an accurate measure or indicator of flight activity. FAA plans to evaluate ways to collect the air cargo data over the long term.

FAA Has Various Processes in Place to Help Ensure Data Quality, but Weaknesses Still Exist

FAA Has Taken a Number of Steps to Help Ensure Data Quality

FAA, along with the international aviation community, recognizes that high-quality data—that is, reliable, valid data—are essential to the effectiveness of a data-driven approach to safety, such as SMS. To help ensure data quality, FAA has issued guidance, established procedures, and implemented controls. For example, FAA has issued an order that establishes an agencywide policy on data management. This policy applies to all information from FAA and other sources used to perform the agency’s mission. In accordance with the data management order, FAA’s Office of Aviation Safety has established a data management framework that includes a four-step process for importing data from other FAA offices and external sources. This process includes data acquisition—obtaining information from various data owners, data standardization—validating data by comparing a new data set with previous data sets to identify inconsistencies, data integration—translating data values into plain English and correcting data loading—importing data into the agency’s own systems.

This four-step process applies to 10 of the 13 databases we reviewed— 8 maintained by FAA offices and 2 maintained by NTSB and USDA. (The process applies to 2 of the 4 voluntary reporting databases.) NTSB and USDA also have data quality assurance processes that apply to their databases. For example, NTSB conducts annual reviews of aircraft accident data, and according to USDA, airport managers and wildlife biologists are asked to check data from their respective airports and report errors.

In addition, to help ensure the quality of its data, FAA applies various quality controls, such as validation and verification processes, to better ensure accuracy and completeness. We have identified some standard quality controls that an agency should employ to achieve these high- quality results. For example, agencies should have managers review data, have procedures in place to verify that data are complete and accurate, and correct erroneous data. We assessed 12 databases against these standard quality controls and found that the extent to which such controls were applied varied. (See fig. 3.) While NTSB’s aviation accident and incident database (NTSB), VDRP, and USDA’s wildlife strike database applied all five of the quality control activities we considered for this analysis to some extent, the remaining 9 databases lacked one or more quality control activity. In addition, we found that all of the databases we reviewed fully or to some extent had procedures in place to validate and edit data to help ensure that accurate data are entered into electronic systems and to help ensure that erroneous data are identified, reported, and corrected. To the extent that the databases lack various controls, FAA lacks assurance that the information it uses for oversight is accurate and complete.

While the databases we reviewed varied in the extent to which they had standard quality controls, FAA has other data quality controls in place for some databases that we consider good practices for handling data, as shown in table 3.

Furthermore, other data quality controls apply to the voluntary reporting systems we reviewed. For example, as previously discussed, an event review committee at each participating carrier is tasked with reviewing and analyzing reports submitted under ASAP. This committee determines whether such reports qualify for inclusion in the program, identifies and proposes solutions for actual or potential problems with information contained in the reports, and annually reviews the ASAP database to determine whether corrective actions have been effective in preventing or reducing the recurrence of targeted safety-related events.

For ASRS, NASA officials told us that each individual ASRS report is reviewed by two expert analysts within 3 days of receipt. Each report captures data for seven criteria and data fields, which are screened to ensure accuracy. The analysts also evaluate the database to ensure that publicly released ASRS data do not include information that might identify the reporter. In addition, for FOQA, vendors have quality assurance procedures in place. For instance, one vendor’s procedures include automated checks and tests of each flight data file to detect parameter problems (for example, anomalies in the flight data attributable to faulty sensors), reports of anomalies created for the affected airline, and manual reviews of any data indicating warning-level risk events. These controls are designed to monitor the reliability and validity of FOQA data and to identify technical problems that affect data quality and need to be corrected. However, according to the vendor, some data will always be missing because of data-recording equipment failures or lost flight data cards, but such missing data do not affect the statistical validity of the large FOQA data set.

FAA Is Taking Steps to Address Data Quality Weaknesses Identified by Its Analysts and Others

FAA is taking steps to address data weaknesses identified by its analysts, the DOT IG, and us. For example, FAA officials told us that to mitigate problems from external data sources, they combine data from various sources to validate analysis results. According to FAA analysts, they typically combine AIDS data with wildlife strike, ASRS, or PDS data using a manual process to verify study findings. As another example, we previously reported the importance of aggregating data from multiple sources to understand icing-related incidents. We reported that the AIDS database included 200 icing-related incidents involving large commercial airplanes that occurred from 1998 through 2007. During this same time period, ASRS received over 600 icing- and winter weather-related incident reports involving large commercial airplanes. These incidents revealed a variety of safety issues such as runways contaminated by snow or ice, ground deicing problems, and in-flight icing encounters. This suggested that risks from icing and other winter weather operating conditions may be greater than indicated by the AIDS database.

FAA analysts also communicate with other data providers and experts about quality concerns or sometimes make independent corrections to data. For example, one FAA official told us that analysts communicate with NTSB to report incorrect data in a field and then rely on NTSB to correct the data in its database. FAA analysts said that they retrieve data from public Web sites and then collaborate with subject matter experts to identify and correct any errors in those data. They make such corrections based on their knowledge of the data’s reliability and their own expertise in working with such data. Analysts also said they use information from the narrative sections of a report to correct data fields.

In addition, the IG has identified weaknesses in the quality of specific data, which FAA is working to address. For instance, according to the IG, ATOS data are inconsistent and incomplete because the database has undergone multiple revisions since it was introduced in 1998 and some data fields have changed from one year to another. During these revisions, some data have been lost. Though designed to improve ATOS’s value and usability as an inspection tool, the revisions limit opportunities for analysis of long- term trends to the extent that data fields have changed over time. The revisions do not, however, affect FAA’s ability to analyze the data at a particular point in time. In addition, the process for reporting inspection findings is time-consuming and creates an incentive for inspectors to underreport their inspection results. To report, inspectors must complete a Yes/No checklist and, for every No check, provide a narrative explanation. According to the 2008 independent review team, inspectors have an incentive to check Yes so they can complete their reports in a timely manner. Consequently, the system may underreport problems that inspectors have identified but not taken the time to report.

The IG has also found, and FAA agreed, that OEDS has some missing and incorrect data on operational errors and pilot deviations because personnel have intentionally or unintentionally misclassified these events. Such misclassification is problematic because it can lead to errors in FAA’s assessment and reporting of how well the agency is meeting its annual performance targets for operational errors and pilot deviations. In 2007, the IG investigated operational errors at the Dallas-Fort Worth terminal radar approach control facility and found that FAA air traffic managers had intentionally misclassified operational errors as either pilot deviations or nonoccurrences. On the basis of this finding, FAA agreed with the IG’s recommendation that the agency establish a follow-up mechanism to ensure compliance with guidance for investigating pilot deviations.

Finally, over the years, we have identified weaknesses in the quality of aviation safety data that hinder FAA’s ability to oversee the industry. In response, the agency has taken steps to address many of the problems that we have identified. For example, in our 2007 review of runway safety, we found that FAA’s categorization of the severity of runway incursions involves a level of subjectivity, raising questions about the accuracy of the data. We reported that an internal FAA audit of 2006 runway incursion data found that the subjectivity of the severity classifications has the potential to affect the accuracy of the classifications. We also found that FAA did not systematically collect data on the number of runway overruns that do not result in damage or injury that could be used for analytical purposes to study trends in and causes of these incidents. In July 2009, FAA indicated that it was working to establish procedures that will ensure that all runway overruns and other excursions are reported.

Conclusions

Aviation safety data are critical to FAA’s safety oversight and its planned implementation of SMS. To its credit, FAA has taken steps to help ensure the quality of the data it uses, such as implementing quality controls to help ensure that errors are identified, reported, and corrected, but these procedures are not applied consistently across all databases. Although FAA is developing a plan that will address how data fit into its new oversight approach, that plan lacks a description of the data that will be required to conduct proactive data analyses, an inventory of the skills personnel will need to perform such analyses and help ensure data quality, and a description of the steps needed to address continuing data quality problems. Unless the plan links FAA’s data requirements and staffing needs to the analyses that will drive its proactive safety management system and addresses the agency’s data quality problems, available data may not be as reliable and useful as they could be to support SMS.

While NextGen technologies and procedures are intended to increase the safety, efficiency, and capacity of the national airspace system, their introduction could have unintended effects on system safety if not done in a comprehensive manner. As FAA improves its ability to integrate and analyze data from multiple sources, it plans to increase its capacity to model the impact of NextGen changes and to identify and manage risks. Because some NextGen changes are already taking place, it is urgent that FAA move with all deliberate speed to advance its analytical capability.

The data that FAA obtains through voluntary reporting programs afford insights into safety events that are not available from other sources and are critical to improving aviation safety, but participation in these programs has been limited by concerns about the impact of disclosure and, especially in the case of smaller carriers, by cost considerations. Efforts such as the training FAA provides to smaller carriers on how to develop programs that require less capital investment have the potential to increase participation and improve safety. However, without carrier- specific information on why air carriers are not participating in these programs, FAA cannot determine if its efforts to increase participation are sufficient.

Recommendations for Executive Action

To help improve and expand FAA’s capability to use data for aviation safety oversight, we recommend that the Secretary of Transportation direct the FAA Administrator to take the following four actions: develop and implement a comprehensive plan that addresses how data fit into FAA’s implementation of a proactive approach to safety oversight and ensure that this plan fully describes the relevant data challenges (such as ensuring data quality and continued access to voluntarily reported safety data), analytical approaches, and staffing requirements and integrates efforts to address them; given the importance of high-quality data, extend standard quality controls, as appropriate, to the databases that support aviation safety oversight to ensure that the data are as reliable and valid as possible; proceed with all deliberate speed to develop the capability to model the impact of NextGen changes on the national airspace system and manage any risks emerging from these changes; and systematically identify the reasons that carriers are not participating in voluntary reporting programs, such as through a survey, and identify and implement further steps to encourage greater program participation, especially by smaller carriers.

Agency Comments

We provided copies of a draft of this report to DOT, NASA, USDA, and NTSB for their review and comment. DOT agreed to consider our recommendations. DOT and NASA provided technical corrections and clarifications, which we incorporated as appropriate. USDA had no comments. NTSB generally agreed with our findings and recommendations to FAA and provided several comments. First, NTSB noted that our use of the terms “reactive” and “proactive” implied a new approach to aviation safety data analysis that is different from past analyses of accidents and incidents to improve safety. The agency noted that a more efficient, effective approach to safety analysis should continue to include FAA’s previous reactive approach as well as new, more predictive capabilities. We agree with NTSB’s comment and note that our report indicates that FAA plans to continue to use data to analyze past safety events as it also works to use data more proactively. NTSB further noted that the success of SMS will depend on the maturation of FAA’s data analysis capabilities.

Second, NTSB agreed with our finding that the lack of a final plan for ASIAS and for SMS implementation, which are key elements of FAA’s planned proactive safety analysis capability, was a cause for concern. The agency noted that it had made several recommendations to FAA to require SMS programs for part 121, part 135, and part 91 carriers and that FAA had not yet taken action to require these programs.

Third, regarding FAA’s access to voluntarily reported data, NTSB agreed with our finding that the redaction of flight details from ASAP and FOQA analyses is a serious constraint on the thoroughness and potential utility of ASIAS and other assessments of safety. If FAA does not address these data limitations, NTSB observed, such constraints are likely to pose serious and continuing threats to the broader use of voluntary reporting programs to support safety analysis. In NTSB’s view, our recommendations to FAA do not go far enough to recommend mechanisms besides redaction, such as statutory exemptions from disclosure, to protect these data from enforcement and disciplinary uses or public release. We did not revise our recommendations to FAA to include these issues because, while we found that participation was temporarily affected, in part, by employees’ fears of disciplinary action by their employers, we did not find evidence that participation was inhibited by the fear of enforcement action by FAA or public disclosure. In addition, our work indicated that the current mechanisms to protect the data appeared to be working.

Fourth, regarding FAA’s access to data on various safety events, NTSB noted the importance of FAA collecting the necessary data to support its new approaches to data analysis rather than simply combining existing data sources into an analysis program. NTSB also agreed with our finding that independent survey efforts like NAOMS could provide a useful complement to other data sources, including FOQA, in providing improved data quality and analysis capabilities.

Finally, NTSB agreed with our finding that the availability of operations data for sectors other than large commercial carriers (i.e., part 121 operators) is severely limited. The agency noted that accurate flight activity data are not available for most of these operations and must be estimated from FAA’s annual survey of a sample of active general aviation aircraft. NTSB also pointed out that FAA does not require reporting for the majority of equipment reliability or maintenance related events. To address these shortcomings, NTSB noted its recent recommendation to FAA to take steps to increase general aviation reporting to FAA’s service difficulty reporting system. To correct these and other data deficiencies, NTSB believes that FAA should explore the development of new aviation safety data collection techniques or methods to supplement current areas of data deficiency.

As arranged with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after the date of this letter. At that time, we will send copies of this report to relevant congressional committees, the Secretaries of Transportation and Agriculture, the Administrator of the Federal Aviation Administration, the Chairman of the National Transportation Safety Board, and the Administrator of the National Aeronautics and Space Administration. We will also make copies available to others on request. In addition, this report will be made available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-2834 or dillinghamg@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix II.

Gerald L. Dillingham, Ph.D Director, Physical Infrastructure Issues .

Appendix I: Objectives, Scope, and Methodology

In this report, we assessed the Federal Aviation Administration’s (FAA) capacity to use available data to oversee aviation safety. To do so, we addressed the following questions: (1) How does FAA use data to oversee aviation safety, and what changes, if any, has it planned? (2) To what extent does FAA have access to data for monitoring aviation safety and the safety of various aviation industry sectors? (3) What does FAA do to help ensure the quality of the data it uses to oversee aviation safety?

To perform our review, we selected 10 safety events that were among those previously identified as key by the National Aeronautics and Space Administration’s (NASA) National Aviation Operations Monitoring Service (NAOMS) or by the FAA-industry Commercial Aviation Safety Team (CAST). This selection allowed us to focus our review on a manageable subset of FAA oversight activities and data sources. (See table 4.) We then identified 13 databases available to FAA that contained data on these safety events and reviewed these databases. The databases are described in the background section of this report.

To determine how FAA uses data to oversee aviation safety, we reviewed reports by FAA, the International Civil Aviation Organization, and industry and other published documents. In addition, we interviewed FAA, industry associations, and other selected industry groups.

To determine the extent to which FAA has access to data for monitoring aviation safety and the safety performance of various aviation industry sectors, we interviewed FAA data analysts, contractors, and other officials responsible for data management. We also reviewed previous GAO reports on FAA’s access to data on certain aviation sectors, such as air ambulances and air cargo operations.

To determine how FAA ensures the quality of its data, we interviewed FAA and industry officials (see table 5). In addition, we reviewed assessments of selected FAA data by the Department of Transportation Inspector General and an independent review team appointed by the Secretary of Transportation in 2008. We also derived a number of data quality principles from our previous work on internal controls, and we assessed the quality of 12 of the 13 selected aviation safety databases by comparing our data quality principles with FAA’s practices for ensuring data quality. These principles include ensuring that the data are complete and accurate, measure intended safety concerns, and are useful for their intended oversight purposes. To measure the extent to which FAA’s practices were consistent with these principles, we evaluated information and all other materials regarding the databases using a three-point scale. To validate the results, multiple reviewers independently scored each principle. When the initial scores differed, the reviewers collectively agreed on a final score for each principle. Further, we used the results of GAO studies that considered the availability, quality, and use of data in aviation safety oversight.

In addition, to address all three research questions, we individually interviewed 10 aviation safety experts and asked them to identify challenges to using data for overseeing aviation safety, the reasonableness of FAA’s current and planned efforts to use aviation safety data, and ways that FAA could enhance its data collection and analysis processes to improve its oversight capabilities. We selected experts who represent a cross section of aviation stakeholders, including persons with general knowledge of aviation safety, aircraft operations, human factors, aircraft maintenance, and air traffic control. The experts have operational, academic, or other professional expertise in these areas. Those experts are Mr. Basil Barimo, Vice President, Safety and Operations, Air Transport Mr. James Burin, Director of Technical Plans and Programs, Flight Safety Kim Cardosi, Ph.D., U.S. Department of Transportation, Volpe Center; Todd Curtis, Ph.D., Director, The Airsafe.com Foundation; Mr. John Goglia, Senior Vice President for Aviation Operations and Safety Programs, JDA Aviation Technology Solutions, former board member of the National Transportation Safety Board; Mr. Keith Hagy, Director, Engineering and Air Safety, Air Line Pilots Brigadier General Leon Johnson (Air Force, retired), former Flight Operations Manager, United Parcel Service; Mr. Bruce Landsberg, Executive Director, Aircraft Owners and Pilots Thomas Weitzel, Ed.D., Associate Professor, Embry Riddle Aeronautical Mr. Dale Wright, Director, Safety and Technology, National Air Traffic Controllers Association.

Appendix II: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

In addition to the person named above, Teresa Spisak, Assistant Director; Elizabeth Eisenstadt; N’Kenge Gibson; H. Brandon Haller; Erica Miles; and Richard Scott made key contributions to this report.