Aviation Safety: FAA Is Taking Steps to Improve Data, but Challenges for Managing Safety Risks Remain
Released: 2012-04-25T00:00:00

Letter

FAA Uses Reactive and Proactive Data Analysis to Prevent Accidents and Manage Risk

For decades, FAA, other federal regulators, and the aviation industry have used data in a reactive fashion—that is, to identify the causes of aviation accidents and incidents and take actions to prevent their recurrence. Aviation accident data are collected by NTSB, but FAA also collects some accident data and uses various databases and voluntary reporting programs to collect incident data, such as for runway incursions—the unauthorized presence of an aircraft, vehicle, or person on a runway. FAA also gathers and analyzes data through its inspection and certification programs to ensure industry compliance with safety regulations. (App. I provides more information on the databases discussed in this statement.)

Since 1998, FAA has partnered with the airline industry through the Commercial Aviation Safety Team (CAST) to identify precursors and contributing factors and ensure that efforts to improve safety focus on the most prevalent categories of accidents. CAST has reduced the risk in commercial aviation by focusing on areas such as controlled flight into terrain, loss of control, and runway incursions. CAST analyzes accident and incident data to identify precipitating conditions and causes, and then formulates an intervention strategy designed to reduce the likelihood of a recurrence. According to CAST, its work—along with new aircraft, regulations, and other activities—reduced the commercial aviation fatal accident rate by 83 percent from 1998 to 2008 and is an important aspect of FAA’s efforts to improve aviation safety by sharing and analyzing data. (Fig. 1 illustrates the number of fatal and nonfatal commercial air carrier accidents from 1998 through 2010.)

Similarly, FAA analyzes data on incidents, such as those where there is a risk of a catastrophic accident. For example, data on runway incursions, other surface incidents, and airborne incidents are collected at airports that have air traffic control towers. FAA analyzes those data to categorize incidents according to the actions or inactions of air traffic controllers, pilots, or others, such as pedestrians or vehicle operators, and determines the severity of those incidents. These data are then used to assess the root causes of incidents to identify potential remedies. Using this process, FAA has taken steps to improve safety in the terminal area since 2007 and has both reduced the number of serious runway incursions—where collisions are narrowly avoided or where there was a significant potential for a collision—and undertaken successful efforts to increase the reporting of incidents.

Although FAA will continue using data in a reactive manner to understand the causes of accidents and incidents, it is shifting emphasis to a proactive approach in which it analyzes data to identify and mitigate risks to prevent future accidents as part of its adoption of SMS. As a result, data that can be used to help identify accident and incident precursors— such as data on an incident that is voluntarily reported by pilots, air traffic controllers, or others to FAA or air carriers—have become more critical for accident prevention, according to the independent review of FAA’s safety oversight in 2008. CAST is also now moving beyond the forensic approach of examining past accident data to a more proactive approach that will focus on risk prediction and mitigation strategies and aims to reduce the U.S. commercial fatality risk by 50 percent from 2010 to 2025. FAA’s effort to integrate aviation safety data—the Aviation Safety Information Analysis and Sharing (ASIAS) system, which connects 46 safety databases across the industry and has 45 participating airlines—is integrated into the CAST process. ASIAS enables better safety information management and data sharing as it proactively extracts from public and non-public data sources, including accidents, incidents, and voluntary reporting. FAA has demonstrated the potential of using integrated safety data to better understand the causes of certain safety events and identify mitigating strategies. For example, FAA fused data from government and industry sources to identify underlying factors contributing to numerous nuisance warnings pilots were receiving from their terrain awareness warning systems (TAWS). By combining all of the data, FAA was able to identify needed changes in the way it handles air traffic as well as improvements in the design of the TAWS software. FAA also plans to use data proactively to model the impact of the Next Generation Air Transportation System (NextGen) on the safety of the national airspace system, to proactively identify risks that might emerge with the introduction of NextGen changes. Figure 2 illustrates the type of transition FAA plans as the agency shifts its emphasis to a proactive assessment of emerging safety risks.

As part of its oversight system for commercial air carriers, FAA collects and analyzes data to ensure that the industry complies with safety regulations. FAA uses the Air Transportation Oversight System (ATOS), a risk-based data-driven system, to oversee maintenance and operations at all air carriers. Under the ATOS concept, FAA inspectors use data analysis to focus their inspections on areas that pose the greatest risk. ATOS also permits inspectors to shift the focus of their inspections in response to changing conditions within air carriers’ operations. In contrast, FAA’s oversight program for the remaining operators (i.e., air taxi, general aviation, etc.) focuses on inspectors completing a prescribed number of inspection activities annually and is primarily based on checking operator compliance with regulations.

FAA Has Various Processes in Place to Help Ensure Data Quality

Implementing systems and processes that capture accurate and complete data is critical for FAA to determine the magnitude of safety issues, assess their potential impacts, identify their root causes, and effectively address and mitigate them. As such, FAA has various processes in place to help ensure data quality and is taking steps to address remaining weaknesses. For example, FAA established an agency-wide order on data management that specifies the roles and associated responsibilities for data management within the agency. This order applies to all sharable information from FAA and other sources used to perform the agency’s mission.

In accordance with the data management order, FAA’s Office of Aviation Safety established a data management framework that includes a four- step process for importing data from other FAA offices and external sources. This process includes data acquisition—obtaining information from various data owners, data standardization—validating data by comparing a new data set with previous data sets to identify inconsistencies, data integration—translating data values into plain English and correcting data errors, and data loading—importing data into the agency’s own systems.

FAA has furthermore put in place data quality controls that we consider good practices for handling data, although weaknesses remain in some areas. For example, FAA has developed training for users on data systems and restricted access to the data. The FAA databases we reviewed in 2010 also had at least some controls in place to ensure that erroneous data are identified, reported, and corrected. However, several of the databases lacked an important control in that managers do not review the data prior to entry into the system. This quality control is important because it could affect accuracy and completeness.controls in place and is taking steps to address its data weaknesses; FAA has however, vulnerabilities remain that potentially limit the usefulness of FAA’s data for some of the safety analyses planned to support SMS. In 2010, we made several recommendations to FAA to help improve and expand its capability to use data for aviation safety oversight. For example, we recommended that FAA extend standard quality controls, as appropriate, to the databases that support aviation safety oversight. Although FAA concurred with our recommendations, it has not fully implemented them.

Data Limitations and Lack of Data Challenge FAA’s Ability to Manage Safety Risks

FAA has put in place various quality controls for its data, but it continues to experience data challenges—including limitations with the analysis it conducts and data it collects, as well as the absence of data in some areas. Some of these limitations hinder the agency’s ability to comprehensively and accurately assess and manage risk, as seen in the following examples:

FAA’s changes to reporting policies impact its ability to accurately determine operational error trends. The rate and number of airborne operational errors—errors made by air traffic controllers—have increased considerably in recent years, with the rate nearly doubling for errors in the terminal area from 2008 to 2011. Multiple changes to reporting policies and processes during this time make it difficult to know the extent to which the recent increases in operational errors are due to more accurate reporting, an increase in the occurrence of safety incidents, or both. For example, FAA removed air traffic controller names from reports in the Air Traffic Quality Assurance (ATQA) database, which may encourage controllers to share more information about incidents. (See fig. 3.) Without determining the potential impact of these policy changes, FAA cannot ensure accurate and consistent measures of operational errors and cannot assess the risks posed over time. FAA believes that these changes likely caused the increases in operational errors but has not yet conducted an analysis to validate the linkage.

Incomplete data on operational errors are assessed, making it difficult to account for all potential risk. Operational errors can be captured in multiple reporting systems. For instance, an air traffic controller’s failure to maintain minimum separation between two aircraft—a loss of separation—could be reported to the ATQA database by a supervisor and will also be captured automatically by airplane tracking technology—the Traffic Analysis and Review Program (TARP)—if it is in use at the relevant facility. However, FAA’s current process for analyzing data on losses of separation captured by these systems only assesses those incidents that occur between two or more radar-tracked aircraft. By excluding incidents such as those that occur between the aircraft and terrain or aircraft and protected airspace, FAA is not considering the systemic risks associated with many other airborne incidents. We recommended last year that FAA expand its current risk assessment process,that it will include these incidents in its risk assessment process before the end of 2013.

Lack of coordination among data systems may affect FAA’s ability to conduct comprehensive data analyses. As previously mentioned, multiple programs and systems capture safety data. Some of these programs—including the Air Traffic Safety Action Program (ATSAP), ATQA, and the Risk Analysis Process (RAP) that considers ATQA and TARP data—also assign contributing factors to the incidents they review. (See fig. 4.) Though both ATSAP and RAP look at some of the same types of incidents (e.g., airborne losses of separation), they had not coordinated on a common set of contributing factors to describe and analyze the incidents. As a result, it is difficult to compare the data and conduct comprehensive analyses. According to FAA officials, they are currently developing a common set of contributing factors for ATSAP and RAP, as well as a translation capability that will allow for the inclusion of historical data on contributing factors in future analyses.

Lack of a robust, complete, and secure data repository of pilot records raises questions about data reliability. Because the training and experience of some pilots have been factors in several commercial aviation accidents, there have been efforts to increase the amount of information airlines have before hiring pilots. The Pilot Records Improvement Act of 1996 requires airlines to conduct background checks on pilots before hiring them, and the Airline Safety Act requires that FAA develop a centralized pilot records database that air carriers must access to review pilot qualifications and past performance data before hiring pilots. According to the Department of Transportation Inspector General (IG), FAA met the act’s initial milestone in developing a centralized electronic pilot records database that will include records previously maintained by air carriers. However, the IG indicated that FAA needs to address the level of detail that should be captured from air carrier pilot training records— such as determining whether recurrent flight training will be included, determining how to transition from the current practices to the new database without disrupting information flow, and deciding how to ensure the reliability of data. The IG also noted that FAA lacks a centralized process to receive and respond to carriers’ requests for pilot records.

Lack of ramp incident data means FAA is unable to assess the risk of catastrophic accidents in this area. In 2007, we reported that efforts to address the occurrence of safety incidents in ramp areas were hindered by the lack of data on the nature, extent, and cost of ramp incidents and accidents. FAA still collects no comprehensive data on incidents in the ramp area and NTSB does not routinely collect data on ramp accidents unless they result in serious injury or substantial aircraft damage. The Occupational Safety and Health Administration (OSHA), the primary source of ramp fatality data, collects only data from accidents involving an employee death or the hospitalization of at least three employees. The lack of ramp incident data will pose a challenge as airports move to implement SMS. We recommended in FAA agreed with 2011 that FAA extend oversight to the ramp areas.our recommendation but noted that it already oversees ramps through its oversight of airlines. FAA expects to further enhance that oversight through its proposed ruling to require airports with air carrier operations to establish a safety management system.

Lack of a process to track and assess runway excursions denies FAA the ability to assess the risks of these incidents. Runway excursions can be as dangerous as incursions; according to the Flight Safety Foundation, excursions have resulted in more fatalities than incursions globally. (Fig. 5 illustrates the difference between runway incursions and excursions.) FAA does not have a process to track excursions, unlike for runway incursions. We recommended in 2011 that FAA develop and implement plans to track and assess runway excursions. FAA agreed and will be developing a program to collect and analyze runway excursion data and is drafting an order to set out the definitions and risk assessment processes for categorizing and analyzing the data. However, according to our review of FAA’s plans, it will be several years before FAA has obtained enough detailed information about these incidents in order to assess risks.

Lack of complete data on active pilot schools and pilot examiners makes it difficult for FAA to ensure that safety standards are being met. Inspections are a part of FAA’s oversight of Part 141 pilot schools and pilot examiners— gatekeepers for the initial pilot training process. However, it was unclear from our analysis of FAA inspection data for pilot schools and pilot examiners whether FAA met its oversight requirements because we could not determine the number of active entities that should have been inspected each year. FAA does not maintain a historical listing of pilot schools and examiners, and, thus, we could not define the universe of active entities that was required to be inspected. Because of this data limitation, we could not determine the completion percentage of the inspections for either group. In November 2011, we recommended that FAA develop a comprehensive system for measuring its performance in meeting its inspection requirements for pilot schools and examiners.that (1) it needed to clarify its inspection requirements for pilot schools in the revision of its national oversight policy guidelines, and (2) its new designee management system, which would include oversight of pilot examiners, will provide more comprehensive data once it is developed.

FAA acknowledged our recommendation and noted In closing, Mr. Chairman, FAA regulates one of the safest aviation systems in the world. To its credit, FAA continues to strive for even higher levels of safety. Shifting to a data-driven, risk-based safety oversight approach means that FAA needs data that are appropriate, complete, and accurate to be able to identify system-wide trends and manage emerging risks. Furthermore, when implementing changes in safety data reporting systems, or processes used to assess and analyze data to determine risk, FAA needs to take into account how such changes might impact trend analysis. Today, I have pointed out some of the challenges FAA faces in improving its data. While FAA is working diligently to improve its data in some instances, more work remains to address limitations and to collect additional data where necessary.

Chairman Petri, Ranking Member Costello, and Members of the Subcommittee, this concludes my prepared statement. I would be pleased to answer any questions at this time.

GAO Contact and Staff Acknowledgments

For further information on this testimony, please contact Gerald L. Dillingham, Ph.D., at (202) 512-2834 or dillinghamg@gao.gov. In addition, contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals making key contributions to this testimony include Brandon Haller (Assistant Director), Pamela Vines, Martha Chow, Vashun Cole, Kevin Egan, Colin Fallon, Molly Laster, Brooke Leary, Erica Miles, Richard Scott, Teresa Spisak, and Jessica Wintfeld.

Appendix I: Summary of Databases Referenced in Statement

Responsible entity FAA

Quantitative This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.