Aviation Safety: FAA Is Taking Steps to Improve Data, but Challenges for Managing Safety Risks Remain
Published: 2012-04-25T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The U.S. aviation system is one of the safest in the world, but fatal accidents, though rare, continue to occur. As a result of recent accidents and related NTSB findings, FAA announced a Call to Action Plan in June 2009 to, among other things, increase air carrier participation in voluntary safety programs. In 2010, Congress passed the Airline Safety and Federal Aviation Administration Extension Act, which, in part, called for FAA to better manage safety risks. As a result, FAA developed a concerted strategy to implement new safety programs, including increasing air carrier use of voluntary safety programs and advancing the use of SMS.
FAA is implementing SMSa data-driven, risk-based safety approach that involves establishing the necessary organizational structures, accountabilities, policies, and procedures. The implementation of SMS heightens the importance of obtaining and using high-quality aviation safety data.
This statement is based on GAOs previous work and focuses on (1) how FAA uses data to manage safety risks, (2) how FAA ensures it has quality data to manage risk, and (3) the challenges FAA faces in using data to better manage safety risks.
section_title: What GAO Found
paragraphs: The Federal Aviation Administration (FAA) uses data reactively and proactively to prevent accidents and manage safety risks. For instance, since 1998, FAA has partnered with the airline industry to identify precursors and contributing factors, and ensure that efforts to improve safety focus on the most prevalent categories of accidents and formulate an intervention strategy designed to reduce recurrences. Although FAA plans to continue using data reactively to understand the causes of accidents and incidents, as part of its adoption of Safety Management Systems (SMS), it is shifting to a proactive approach in which it analyzes data to identify and mitigate risks before they result in accidents.
Implementing systems and processes that capture accurate and complete data are critical for FAA to determine the magnitude of safety issues, assess their potential impacts, identify their root causes, and effectively address and mitigate them. Though FAA has put in place data quality controls, weaknesses remain in some areas. In particular, several FAA databases GAO reviewed in 2010 did not have a managerial review process prior to data entryan important control that helps ensure data accuracy and completeness. In response to GAOs recommendations, FAA is taking steps to address its data weaknesses, but vulnerabilities that remain potentially limit the datas usefulness for safety analysis.
FAA also continues to experience data-related challenges, including limitations with the analysis it conducts and the data it collects and the absence of data in some areas. For example, FAA does not have a process to track or assess runway excursions, which occur when an aircraft veers off or overruns a runway. Runway excursions can be as dangerous as runway incursions, which occur when an unauthorized aircraft, vehicle, or person is on a runway, and FAA has tracked runway incursions for years. GAO previously recommended that FAA develop and implement plans to track and assess runway excursions. FAA agreed and is currently developing a program to collect and analyze runway excursion data and is drafting an order to set out the definitions and risk assessment processes for categorizing and analyzing the data. However, according to GAOs review of FAAs plans, it will be several years before FAA has obtained enough detailed information about these incidents to assess risks. Similarly, GAO has found that efforts to address the occurrence of safety incidents in ramp areas were hindered by the lack of data on the nature, extent, and cost of such incidents and accidents. FAA collects no comprehensive data on incidents in ramp areas, and the National Transportation Safety Board (NTSB) does not routinely collect data on ramp accidents unless they result in serious injury or substantial aircraft damage. FAAs lack of ramp incident data means that FAA is unable to assess the risk of catastrophic accidents in this area. FAA agreed with GAOs recommendation to extend oversight to ramp areas but noted that it already provides oversight through its oversight of airlines. FAA expects to further enhance that oversight through its proposed ruling to require airports with air carrier operations to establish a safety management system.
section_title: What GAO Recommends
paragraphs: GAO has made a number of recommendations to address data quality weaknesses. FAA concurred with most of these recommendations and in some cases has taken steps toward addressing them.

section_title: Letter
paragraphs: Thank you for the opportunity to testify before you today on the safety of  U.S. commercial aviation. The U.S. aviation system is one of the safest in  the world, but fatal accidents, though rare, continue to occur. The last  fatal commercial aviation accident occurred in Buffalo, New York, on  February 12, 2009, when 50 people perished in a Colgan Air crash. In  response to this accident, and National Transportation Safety Board  (NTSB) findings that pilot training and lack of qualifications were  potentially contributing factors, the Federal Aviation Administration (FAA)  began a Call to Action Plan in June 2009 to, among other things, increase  air carrier participation in voluntary safety programs. In 2010, Congress  passed the Airline Safety and Federal Aviation Administration Extension  Act (Airline Safety Act), which, in part, called for FAA to better manage  safety risks. As a result, FAA developed a concerted strategy to  implement new safety programs required by the Airline Safety Act,  including establishing better processes for managing safety risks and  advancing Safety Management Systems (SMS).
SMS is an integrated, data-driven approach to managing safety risk that  involves establishing the necessary organizational structures,  accountabilities, policies, and procedures to enhance safety. SMS  introduces an evolutionary structured process in system safety and safety  management that obligates organizations to manage safety with the same  level of priority that other core business processes are managed. This  applies to both internal FAA operations and external aviation industry  organizations. The International Civil Aviation Organization (ICAO), of  which the U.S. is a member state, requires SMS for the management of  safety risk in air operations, maintenance, air traffic services, and airports.  SMS is becoming a worldwide standard throughout the aviation industry,  and SMS concepts have generated widespread support as an effective  approach that can deliver safety benefits. The implementation of SMS  heightens the importance of obtaining and using high-quality aviation  safety data. Further, according to a 2008 independent review team  chartered by the Secretary of Transportation,accidents have become increasingly rare, information that can be used to  help identify accident and incident precursors has become more critical  for accident prevention.
My testimony today focuses on (1) how FAA uses data to manage safety  risks, (2) how FAA ensures it has quality data to manage risk, and (3) the  challenges FAA faces in using data to better manage safety risks. This  statement is based on our previous work, including our May 2010 report  on aviation data quality, our October 2011 report on terminal area safety,  and our November 2011 report and March 2012 statement for the record  on initial pilot training. such as the status of our recommendations and programs or initiatives  FAA planned to implement—as necessary during March and April 2012.  We also conferred with FAA officials on the new information included in  this statement. The GAO publications cited in this statement contain  detailed explanations of the methods used to conduct our work, which we  performed in accordance with generally accepted government auditing  standards.
See GAO, Aviation Safety: Improved Data Quality and Analysis Capabilities Are Needed  as FAA Plans a Risk-Based Approach to Safety Oversight, GAO-10-414 (Washington,  D.C.: May 6, 2010); Aviation Safety: Enhanced Oversight and Improved Availability of  Risk-Based Data Could Further Improve Safety, GAO-12-24 (Washington, D.C.: Oct. 5,  2011); Initial Pilot Training: Better Management Controls Are Needed to Improve FAA  Oversight, GAO-12-117 (Washington, D.C.: Nov. 4, 2011); and Aviation Safety: FAA Has  An Opportunity to Enhance Safety and Improve Oversight of Initial Pilot Training,  GAO-12-537T (Washington, D.C.: Mar. 20, 2012).
subsections: section_title: FAA Uses Reactive and Proactive Data Analysis to Prevent Accidents and Manage Risk
paragraphs: For decades, FAA, other federal regulators, and the aviation industry  have used data in a reactive fashion—that is, to identify the causes of  aviation accidents and incidents and take actions to prevent their  recurrence. Aviation accident data are collected by NTSB, but FAA also  collects some accident data and uses various databases and voluntary  reporting programs to collect incident data, such as for runway  incursions—the unauthorized presence of an aircraft, vehicle, or person  on a runway. FAA also gathers and analyzes data through its inspection  and certification programs to ensure industry compliance with safety  regulations. (App. I provides more information on the databases  discussed in this statement.)
Since 1998, FAA has partnered with the airline industry through the  Commercial Aviation Safety Team (CAST) to identify precursors and  contributing factors and ensure that efforts to improve safety focus on the  most prevalent categories of accidents. CAST has reduced the risk in  commercial aviation by focusing on areas such as controlled flight into  terrain, loss of control, and runway incursions. CAST analyzes accident  and incident data to identify precipitating conditions and causes, and then  formulates an intervention strategy designed to reduce the likelihood of a  recurrence. According to CAST, its work—along with new aircraft,  regulations, and other activities—reduced the commercial aviation fatal  accident rate by 83 percent from 1998 to 2008 and is an important aspect  of FAA’s efforts to improve aviation safety by sharing and analyzing data.  (Fig. 1 illustrates the number of fatal and nonfatal commercial air carrier  accidents from 1998 through 2010.)
Similarly, FAA analyzes data on incidents, such as those where there is a  risk of a catastrophic accident. For example, data on runway incursions,  other surface incidents, and airborne incidents are collected at airports  that have air traffic control towers. FAA analyzes those data to categorize  incidents according to the actions or inactions of air traffic controllers,  pilots, or others, such as pedestrians or vehicle operators, and  determines the severity of those incidents. These data are then used to  assess the root causes of incidents to identify potential remedies. Using  this process, FAA has taken steps to improve safety in the terminal area  since 2007 and has both reduced the number of serious runway  incursions—where collisions are narrowly avoided or where there was a  significant potential for a collision—and undertaken successful efforts to  increase the reporting of incidents.
Although FAA will continue using data in a reactive manner to understand  the causes of accidents and incidents, it is shifting emphasis to a  proactive approach in which it analyzes data to identify and mitigate risks  to prevent future accidents as part of its adoption of SMS. As a result,  data that can be used to help identify accident and incident precursors— such as data on an incident that is voluntarily reported by pilots, air traffic  controllers, or others to FAA or air carriers—have become more critical  for accident prevention, according to the independent review of FAA’s  safety oversight in 2008. CAST is also now moving beyond the forensic  approach of examining past accident data to a more proactive approach  that will focus on risk prediction and mitigation strategies and aims to  reduce the U.S. commercial fatality risk by 50 percent from 2010 to 2025.  FAA’s effort to integrate aviation safety data—the Aviation Safety  Information Analysis and Sharing (ASIAS) system, which connects 46  safety databases across the industry and has 45 participating airlines—is  integrated into the CAST process. ASIAS enables better safety  information management and data sharing as it proactively extracts from  public and non-public data sources, including accidents, incidents, and  voluntary reporting. FAA has demonstrated the potential of using  integrated safety data to better understand the causes of certain safety  events and identify mitigating strategies. For example, FAA fused data  from government and industry sources to identify underlying factors  contributing to numerous nuisance warnings pilots were receiving from  their terrain awareness warning systems (TAWS). By combining all of the  data, FAA was able to identify needed changes in the way it handles air  traffic as well as improvements in the design of the TAWS software. FAA  also plans to use data proactively to model the impact of the Next  Generation Air Transportation System (NextGen) on the safety of the  national airspace system, to proactively identify risks that might emerge  with the introduction of NextGen changes. Figure 2 illustrates the type of  transition FAA plans as the agency shifts its emphasis to a proactive  assessment of emerging safety risks.
As part of its oversight system for commercial air carriers, FAA collects  and analyzes data to ensure that the industry complies with safety  regulations. FAA uses the Air Transportation Oversight System (ATOS), a  risk-based data-driven system, to oversee maintenance and operations at  all air carriers. Under the ATOS concept, FAA inspectors use data  analysis to focus their inspections on areas that pose the greatest risk.  ATOS also permits inspectors to shift the focus of their inspections in  response to changing conditions within air carriers’ operations. In  contrast, FAA’s oversight program for the remaining operators (i.e., air  taxi, general aviation, etc.) focuses on inspectors completing a prescribed  number of inspection activities annually and is primarily based on  checking operator compliance with regulations.
section_title: FAA Has Various Processes in Place to Help Ensure Data Quality
paragraphs: Implementing systems and processes that capture accurate and complete  data is critical for FAA to determine the magnitude of safety issues,  assess their potential impacts, identify their root causes, and effectively  address and mitigate them. As such, FAA has various processes in place  to help ensure data quality and is taking steps to address remaining  weaknesses. For example, FAA established an agency-wide order on  data management that specifies the roles and associated responsibilities  for data management within the agency. This order applies to all  sharable information from FAA and other sources used to perform the  agency’s mission.
In accordance with the data management order, FAA’s Office of Aviation  Safety established a data management framework that includes a four- step process for importing data from other FAA offices and external  sources. This process includes data acquisition—obtaining information from various data owners, data standardization—validating data by comparing a new data set  with previous data sets to identify inconsistencies, data integration—translating data values into plain English and  correcting data errors, and data loading—importing data into the agency’s own systems.
FAA has furthermore put in place data quality controls that we consider  good practices for handling data, although weaknesses remain in some  areas. For example, FAA has developed training for users on data  systems and restricted access to the data. The FAA databases we  reviewed in 2010 also had at least some controls in place to ensure that  erroneous data are identified, reported, and corrected. However, several  of the databases lacked an important control in that managers do not  review the data prior to entry into the system. This quality control is  important because it could affect accuracy and completeness.controls in place and is taking steps to address its data weaknesses;   FAA has  however, vulnerabilities remain that potentially limit the usefulness of  FAA’s data for some of the safety analyses planned to support SMS. In  2010, we made several recommendations to FAA to help improve and  expand its capability to use data for aviation safety oversight. For  example, we recommended that FAA extend standard quality controls, as  appropriate, to the databases that support aviation safety oversight.  Although FAA concurred with our recommendations, it has not fully  implemented them.
section_title: Data Limitations and Lack of Data Challenge FAA’s Ability to Manage Safety Risks
paragraphs: FAA has put in place various quality controls for its data, but it continues  to experience data challenges—including limitations with the analysis it  conducts and data it collects, as well as the absence of data in some  areas. Some of these limitations hinder the agency’s ability to  comprehensively and accurately assess and manage risk, as seen in the  following examples:
FAA’s changes to reporting policies impact its ability to accurately  determine operational error trends. The rate and number of airborne  operational errors—errors made by air traffic controllers—have  increased considerably in recent years, with the rate nearly doubling  for errors in the terminal area from 2008 to 2011. Multiple changes to  reporting policies and processes during this time make it difficult to  know the extent to which the recent increases in operational errors  are due to more accurate reporting, an increase in the occurrence of  safety incidents, or both. For example, FAA removed air traffic  controller names from reports in the Air Traffic Quality Assurance  (ATQA) database, which may encourage controllers to share more  information about incidents. (See fig. 3.) Without determining the  potential impact of these policy changes, FAA cannot ensure accurate  and consistent measures of operational errors and cannot assess the  risks posed over time. FAA believes that these changes likely caused  the increases in operational errors but has not yet conducted an  analysis to validate the linkage.
Incomplete data on operational errors are assessed, making it difficult  to account for all potential risk. Operational errors can be captured in  multiple reporting systems. For instance, an air traffic controller’s  failure to maintain minimum separation between two aircraft—a loss  of separation—could be reported to the ATQA database by a  supervisor and will also be captured automatically by airplane  tracking technology—the Traffic Analysis and Review Program  (TARP)—if it is in use at the relevant facility. However, FAA’s current  process for analyzing data on losses of separation captured by these  systems only assesses those incidents that occur between two or  more radar-tracked aircraft. By excluding incidents such as those that  occur between the aircraft and terrain or aircraft and protected  airspace, FAA is not considering the systemic risks associated with  many other airborne incidents. We recommended last year that FAA  expand its current risk assessment process,that it will include these incidents in its risk assessment process  before the end of 2013.
Lack of coordination among data systems may affect FAA’s ability to  conduct comprehensive data analyses. As previously mentioned,  multiple programs and systems capture safety data. Some of these  programs—including the Air Traffic Safety Action Program (ATSAP),  ATQA, and the Risk Analysis Process (RAP) that considers ATQA  and TARP data—also assign contributing factors to the incidents they  review. (See fig. 4.) Though both ATSAP and RAP look at some of the  same types of incidents (e.g., airborne losses of separation), they had  not coordinated on a common set of contributing factors to describe  and analyze the incidents. As a result, it is difficult to compare the  data and conduct comprehensive analyses. According to FAA  officials, they are currently developing a common set of contributing  factors for ATSAP and RAP, as well as a translation capability that will  allow for the inclusion of historical data on contributing factors in  future analyses.
Lack of a robust, complete, and secure data repository of pilot records  raises questions about data reliability. Because the training and  experience of some pilots have been factors in several commercial  aviation accidents, there have been efforts to increase the amount of  information airlines have before hiring pilots. The Pilot Records  Improvement Act of 1996 requires airlines to conduct background  checks on pilots before hiring them, and the Airline Safety Act  requires that FAA develop a centralized pilot records database that air  carriers must access to review pilot qualifications and past  performance data before hiring pilots. According to the Department of  Transportation Inspector General (IG), FAA met the act’s initial  milestone in developing a centralized electronic pilot records database  that will include records previously maintained by air carriers.  However, the IG indicated that FAA needs to address the level of  detail that should be captured from air carrier pilot training records— such as determining whether recurrent flight training will be included,  determining how to transition from the current practices to the new  database without disrupting information flow, and deciding how to  ensure the reliability of data. The IG also noted that FAA lacks a  centralized process to receive and respond to carriers’ requests for  pilot records.
Lack of ramp incident data means FAA is unable to assess the risk of  catastrophic accidents in this area. In 2007, we reported that efforts to  address the occurrence of safety incidents in ramp areas were  hindered by the lack of data on the nature, extent, and cost of ramp  incidents and accidents. FAA still collects no comprehensive data on  incidents in the ramp area and NTSB does not routinely collect data  on ramp accidents unless they result in serious injury or substantial  aircraft damage. The Occupational Safety and Health Administration  (OSHA), the primary source of ramp fatality data, collects only data  from accidents involving an employee death or the hospitalization of  at least three employees. The lack of ramp incident data will pose a  challenge as airports move to implement SMS. We recommended in   FAA agreed with  2011 that FAA extend oversight to the ramp areas.our recommendation but noted that it already oversees ramps through  its oversight of airlines. FAA expects to further enhance that oversight  through its proposed ruling to require airports with air carrier  operations to establish a safety management system.
Lack of a process to track and assess runway excursions denies FAA  the ability to assess the risks of these incidents. Runway excursions  can be as dangerous as incursions; according to the Flight Safety  Foundation, excursions have resulted in more fatalities than  incursions globally. (Fig. 5 illustrates the difference between runway  incursions and excursions.) FAA does not have a process to track  excursions, unlike for runway incursions. We recommended in 2011  that FAA develop and implement plans to track and assess runway  excursions. FAA agreed and will be developing a program to collect  and analyze runway excursion data and is drafting an order to set out  the definitions and risk assessment processes for categorizing and  analyzing the data. However, according to our review of FAA’s plans,  it will be several years before FAA has obtained enough detailed  information about these incidents in order to assess risks.
Lack of complete data on active pilot schools and pilot examiners  makes it difficult for FAA to ensure that safety standards are being  met. Inspections are a part of FAA’s oversight of Part 141 pilot  schools and pilot examiners— gatekeepers for the initial pilot  training process. However, it was unclear from our analysis of FAA  inspection data for pilot schools and pilot examiners whether FAA met  its oversight requirements because we could not determine the  number of active entities that should have been inspected each year.  FAA does not maintain a historical listing of pilot schools and  examiners, and, thus, we could not define the universe of active  entities that was required to be inspected. Because of this data  limitation, we could not determine the completion percentage of the  inspections for either group. In November 2011, we recommended  that FAA develop a comprehensive system for measuring its  performance in meeting its inspection requirements for pilot schools  and examiners.that (1) it needed to clarify its inspection requirements for pilot schools  in the revision of its national oversight policy guidelines, and (2) its  new designee management system, which would include oversight of  pilot examiners, will provide more comprehensive data once it is  developed.
FAA acknowledged our recommendation and noted  In closing, Mr. Chairman, FAA regulates one of the safest aviation  systems in the world. To its credit, FAA continues to strive for even higher  levels of safety. Shifting to a data-driven, risk-based safety oversight  approach means that FAA needs data that are appropriate, complete, and  accurate to be able to identify system-wide trends and manage emerging  risks. Furthermore, when implementing changes in safety data reporting  systems, or processes used to assess and analyze data to determine  risk, FAA needs to take into account how such changes might impact  trend analysis. Today, I have pointed out some of the challenges FAA  faces in improving its data. While FAA is working diligently to improve its  data in some instances, more work remains to address limitations and to  collect additional data where necessary.
Chairman Petri, Ranking Member Costello, and Members of the  Subcommittee, this concludes my prepared statement. I would be  pleased to answer any questions at this time.
section_title: GAO Contact and Staff Acknowledgments
paragraphs: For further information on this testimony, please contact Gerald L.  Dillingham, Ph.D., at (202) 512-2834 or dillinghamg@gao.gov. In  addition, contact points for our Offices of Congressional Relations and  Public Affairs may be found on the last page of this statement. Individuals  making key contributions to this testimony include Brandon Haller  (Assistant Director), Pamela Vines, Martha Chow, Vashun Cole, Kevin  Egan, Colin Fallon, Molly Laster, Brooke Leary, Erica Miles, Richard  Scott, Teresa Spisak, and Jessica Wintfeld.
section_title: Appendix I: Summary of Databases Referenced in Statement
subsections: section_title: Responsible entity FAA
paragraphs: Quantitative  This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.