Aviation Safety: Improved Data Quality and Analysis Capabilities Are Needed as FAA Plans a Risk-Based Approach to Safety Oversight
Published: 2010-05-06T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: To improve aviation safety, the Federal Aviation Administration (FAA) plans to have in place the initial capabilities of a risk-based approach to safety oversight, known as a safety management system (SMS), by the end of fiscal year 2010. FAA is also implementing new procedures and technologies to enhance the safety, capacity, and efficiency of the national airspace system. Data are central to SMS and FAA's ability to test the impact of these changes on safety. This congressionally requested report addresses FAA's (1) current and planned use of data to oversee aviation safety, (2) access to data for monitoring aviation safety and the safety performance of various industry sectors, and (3) efforts to help ensure data quality. To perform this work, GAO reviewed 13 databases that contain data on key aviation safety events, assessed data quality controls for the databases, and interviewed agency and industry officials, as well as 10 experts in aviation safety and data.
section_title: What GAO Found
paragraphs: FAA analyzes data on past safety events, such as engine failures, to prevent their recurrence and plans to use data to support a more proactive approach to managing risk. For example, weather and air traffic control data helped identify factors associated with injuries from turbulence. As part of SMS, FAA plans to analyze data proactively to support a risk-based approach to safety oversight. For example, FAA plans to use data to model the impact of proposed changes in procedures and technologies on the safety of the national airspace system. Experts said that identifying risks is necessary to maintain the current level of safety and possibly achieve a higher level of safety in the future. Because SMS relies on data to identify emerging risks, FAA has an effort under way to enhance its access to industry data and improve its capability for automated analysis of multiple databases. According to FAA, this effort will allow for more efficient safety analyses. FAA is also developing a plan for managing data under SMS, but the plan does not fully address data, analysis, or staffing requirements. Without such requirements, the plan will not provide timely guidance for implementing SMS. FAA has access to some voluntarily reported data, which are important for SMS, but not all carriers and aviation personnel participate in FAA's voluntary reporting programs. Such data are gathered electronically by equipment on aircraft or reported by aviation personnel or carriers following noncriminal, unintentional violations or safety events. Industry personnel have some incentives to participate in voluntary programs, such as promised immunity from disciplinary action, but concerns about sanctions and the cost of equipment have deterred full participation, especially by smaller carriers. While FAA has some information on reasons for nonparticipation and has taken some steps to promote greater participation, it lacks carrier-specific information on why air carriers are not participating. FAA also lacks data to assess the safety performance of certain industry sectors, such as air cargo and air ambulance operators. GAO has previously made recommendations to address this lack of data. FAA concurred with GAO's prior recommendations and is taking actions to address them. To help ensure data quality--that is, data that are reliable (complete and accurate) and valid (measure what is intended)--FAA has implemented a number of data quality controls that are consistent with GAO's standards for data quality, but some weaknesses exit. For example, all the databases GAO reviewed had at least some controls in place to ensure that erroneous data are identified, reported, and corrected. However, about half the databases lack an important control--managers do not review the data prior to entry into the data system. FAA is taking steps to address its data weaknesses, but vulnerabilities remain, potentially limiting the usefulness of FAA's data for the safety analyses planned to support SMS.

section_title: Letter
paragraphs: To increase the safety, capacity, and efficiency of the national airspace  system, the Federal Aviation Administration (FAA), in cooperation with  aviation industry stakeholders and other federal agencies, is developing a  new, technologically advanced air traffic management system—the Next  Generation Air Transportation System (NextGen). At the same time, FAA  is attempting to further enhance aviation safety by shifting to a new data- driven, risk-based safety oversight approach, which industry and  international air traffic management organizations are also adopting.  Under this new approach, called a safety management system (SMS)  approach, FAA will continue to use data on aviation accidents and  incidents—referred to collectively as safety events—to identify and  address their causes. In addition, under SMS, FAA plans to use aviation  safety data to identify conditions that could lead to safety events and to  address them through changes in organizational processes, management,  and culture. Furthermore, SMS will allow FAA to test the impact of  NextGen changes on aviation safety and to identify safety vulnerabilities  and mitigating measures.
You asked us to assess FAA’s capacity to use available data to oversee  aviation safety. To do so, we addressed the following questions:  1.  How does FAA use data to oversee aviation safety and what changes, if  any, has it planned?  2.  To what extent does FAA have access to data for monitoring aviation  safety and the safety of various aviation industry sectors?  3.  What does FAA do to help ensure the quality of the data it uses to  oversee aviation safety?
To answer these questions, we reviewed 13 aviation safety databases  maintained by FAA, the National Aeronautics and Space Administration  (NASA), the National Transportation Safety Board (NTSB), and the U.S.  Department of Agriculture (USDA). We selected these databases because  they covered key safety events identified by an FAA-industry effort, the  Commercial Aviation Safety Team (CAST), and NASA’s National Aviation  Operations Monitoring Service (NAOMS) project—a research and  development effort that examined the feasibility of using surveys of  randomly selected pilots to identify accident precursors and potential  safety issues. Those safety events are    equipment problems, including engine shutdowns and cargo shifts;    weather events while airborne, including icing and wind shear;    passenger-related events, including medical emergencies;  flight crew problems, such as pilot fatigue;    airborne conflicts, including bird strikes;    ground events, including near collisions on the runway;    aircraft handling-related events, including near collisions with terrain,  deviations from assigned routes, and hard landings;    adverse interactions with air traffic control, including the inability to  communicate with air traffic controllers.
We also selected these databases because they contain data from a range  of internal FAA and external sources. To determine the quality of these  data, we collected and compared information on the data quality controls  used by FAA and others with GAO-identified practices for helping to  ensure data reliability. These practices include ensuring that the data are  complete and accurate, measure intended safety concerns, and are useful  for their intended oversight purposes. FAA and industry gather many other  types of data—such as FAA’s air carriers’ operations and maintenance  data—that we did not assess as part of this review.
We also interviewed 10 aviation safety and data experts to identify critical  data-related issues facing FAA and, where applicable, to assess the  agency’s approaches for addressing data-related issues. We selected  experts in aircraft operations, human factors, aircraft maintenance, and air  traffic control. These experts have general knowledge about aviation  safety and represent a cross section of the aviation industry. Throughout  this report we have highlighted key comments of these stakeholders as  sidebars. In addition, we analyzed documents and interviewed officials  from FAA, NASA, NTSB, USDA, and air carriers; government contractors  that analyze or help collect selected data; aviation industry organizations;  data vendors; and air carrier employee groups. We also used the results of  GAO studies that considered the availability, quality, and use of data in  aviation safety oversight. We conducted this performance audit from  August 2008 through May 2010 in accordance with generally accepted  government auditing standards. Those standards require that we plan and  perform the audit to obtain sufficient, appropriate evidence to provide a  reasonable basis for our findings and conclusions based on our audit  objectives. We believe that the evidence obtained provides a reasonable  basis for our findings and conclusions based on our audit objectives. (See  app. I for additional information on our methods.)
subsections: section_title: Background
paragraphs: Federal agencies and aviation industry stakeholders gather and analyze  aviation data for a variety of purposes. Federal agencies gather and  analyze aviation data primarily to improve safety. To oversee aviation  safety across the national airspace system, FAA maintains data on various  aviation sectors, including passenger airlines, air cargo carriers, general  aviation, and air ambulance operators. FAA also gathers and analyzes data  on industry performance through its inspection and certification programs  and uses these data to ensure that the industry complies with its safety  regulations. In addition, FAA obtains information on safety events and  incidents collected by other federal agencies, including NTSB, NASA, and  USDA.
The aviation industry gathers quantitative and narrative data on the  performance of flights and analyzes these data to increase safety,  efficiency, and profitability. Industry stakeholders also maintain historical  data on equipment and maintenance issues. These stakeholders are  required to report some data to FAA, such as data on accidents, engine  failures, and near midair collisions, and they have agreements with FAA  and other agencies to share other data voluntarily. The voluntarily shared  data include both electronically recorded data from aircraft equipment  under the Flight Operational Quality Assurance program (FOQA) and  information on violations of federal regulations or on safety events self- reported by pilots, mechanics, and other airmen under three programs— Aviation Safety Action Program (ASAP), Aviation Safety Reporting System  (ASRS), and Voluntary Disclosure Reporting Program (VDRP). FAA also  recently established the Air Traffic Safety Action Program (ATSAP),  modeled after the airlines’ ASAP program. It is a confidential, voluntary  reporting system available to FAA’s approximately 17,000 air traffic  control personnel, who can use the program to identify and report safety  and operational concerns.
Table 1 describes the 13 aviation safety databases that we reviewed.
section_title: FAA Analyzes Data on Past Safety Events to Prevent Their Recurrence and Plans to Use Data to Support a More Proactive Approach to Managing Risks
subsections: section_title: FAA Is Shifting from a Reactive to a Proactive Approach to Using Data to Manage Risk
paragraphs: For decades, the aviation industry and federal regulators, including FAA,  have used data reactively to identify the causes of aviation accidents and  incidents and take actions to prevent their recurrence. Since 1998, for  example, FAA has partnered with the airline industry through CAST with  the goal of continuously improving aviation safety. Over the years, CAST  has looked at the causes of past accidents—such as controlled flight into  terrain—and various safety events—such as turbulence or runway  incursions. CAST analyzes past instances of such accidents and events to  identify precipitating conditions and causes. CAST then uses its analysis to  formulate an intervention strategy designed to reduce the likelihood of a  recurrence and validate the effectiveness of the intervention. According to  CAST, its work has helped to decrease commercial airline fatalities— exceeding its goal to reduce fatal commercial accidents by 80 percent by  2007—and is an important aspect of FAA’s efforts to increase aviation  safety by sharing and analyzing data. Table 2 provides examples of how  FAA and industry have used CAST’s work. (Recent work by CAST to work  with FAA’s Aviation Safety Information Analysis and Sharing   initiative to develop safety enhancements and mitigate future threats is  discussed later in this report.)
Besides analyzing data on past safety events to develop intervention  strategies, FAA staff perform such analyses to inform changes in agency  policies. For example, to inform the rule-making requirement that the  costs and benefits of a proposed regulation be determined under  Executive Order 12866, FAA analysts identified the number of aircraft that  could be certified as “light sport” aircraft and were involved in accidents.
In some cases, FAA program managers request specific analyses to inform  policy changes. For instance, in response to a 2007 recommendation by  NTSB and a petition from Hawaiian air tour operators, FAA program  managers requested an analysis of aircraft crashes associated with FAA’s  requirement for tour aircraft to maintain a 1,500-foot separation from the  ground. After analyzing air crashes involving Hawaiian tour aircraft 13  years before and 13 years after the requirement was implemented, FAA  concluded that the requirement helped to reduce the number of crashes  and significantly improved safety.
While FAA will continue to use data to analyze past safety events, it is also  working to use data proactively to search for risks and take actions to  mitigate them before they result in accidents. FAA’s emphasis is shifting to  a proactive approach to data analysis because as accidents have become  increasingly rare, less information is available for reactive analyses of their  causes. As a result, according to a study of FAA’s safety oversight by a  2008 independent review team, information that can be used to help  identify accident and incident precursors, such as voluntarily reported  data, has become more critical for accident prevention. In addition,  several experts we spoke with said that proactively identifying risks is  necessary to maintain the current level of safety and possibly achieve an  even higher level of safety in the future. FAA is undertaking this transition  in coordination with the international aviation community, working with  the International Civil Aviation Organization (ICAO) to adopt applicable  global standards for safety management. Senior FAA officials and ICAO  agree that effective safety management is data driven and that data are  essential to identifying emerging risks. Figure 1 illustrates the type of  transition FAA plans as the agency shifts its emphasis to a proactive  assessment of emerging safety risks, according to FAA officials.
The new technologies and procedures that FAA will implement for  NextGen, which are intended to increase the safety, efficiency, and  capacity of the national airspace system, could also lead to consequences  that have unintended effects on system safety. For example, NextGen  changes to landing procedures, which are designed to allow more frequent  landings, could reduce congestion in the air and improve fuel efficiency,  but might have the unintended effect of increasing congestion and safety  risks on airport taxiways. To avoid such unintended consequences, FAA  plans, as it improves its ability to integrate data and analyze trends, to  model the impact of changes planned for NextGen. To do so, it has begun  to develop a baseline of current conditions and then expects to analyze  how NextGen changes will affect those conditions, according to a senior  FAA official. FAA is in the process of designing tools that will allow it to  model the changes. For example, a project called the National Level  System Safety Assessment is designed to allow FAA to assess risks across  the national airspace system. Currently, FAA assesses risks for specific  NextGen procedures and technologies, but cannot model the risks across  the national airspace system in a comprehensive manner. This project will  integrate data on past safety events from a number of FAA offices and  external sources to proactively identify risks that might emerge with the  introduction of changes planned for NextGen. FAA has begun to obtain  some operational data for the project and has contracted with the Volpe  National Transportation Systems Center, which will be responsible for  integrating airport runway surface data, including surface radar, weather,  aircraft, and other data. A senior FAA official told us that although safety  assessments had been conducted on individual NextGen technologies,  until the agency has finalized this modeling project, it cannot begin  systemwide assessments of the safety of NextGen technologies and  procedures that are already being deployed, including 700 new  navigational procedures that had been deployed as of October 2009.
section_title: SMS and Its Use of Data Form the Linchpin of FAA’s Risk-Based Approach, but Full Implementation of SMS Could Be Years Away
paragraphs: SMS is an integrated, data-driven approach to managing safety risk that  FAA expects will help it continuously improve aviation safety. FAA  stated that successfully implementing SMS is critical to meeting the  challenges of a rapidly changing and expanding aviation system. As stated  earlier in this report, FAA’s traditional approach is to analyze data to  determine the causes of an accident or incident after the fact. To achieve  the next level of safety, FAA says, it now requires a more forward-thinki ng  approach, which SMS provides, to identify systemwide trends and manage  emerging risks before they result in incidents or accidents. To identify  emerging risks, FAA plans to collect and analyze safety data, and it can  then use the results of its analyses to make data-driven decisions about  how to address safety risks. Issued in September 2008, FAA’s guidance on implementing SMS explains the importance of data collection and a to the execution of SMS. SMS: safety policy and objectives, safety risk management, safety  assurance, and safety promotion. First, safety policy and objectives  describe an organization’s requirements and oversight responsibilities f aviation activities. Second, safety risk management describes how an  organization will identify hazards and safety risks in aviation operations,  nalysis   This guidance defines four main components for or  including how it will develop rules, regulations, and safety performance  measures. Third, the safety assurance component of SMS uses data  analyses to discover emerging risks and to model the impact of safety  changes. Fourth, safety promotion is an organization’s plan  training, communication, and dissemination of safety information take  place. According to FAA, SMS with these components will enable the  agency to provide the air transportation system and the public at large  with enhanced safety.
FAA’s goal is for the Office of Aviation Safety to have initial operating  capabilities in place for SMS by the end of fiscal year 2010. According to  FAA officials, these initial operating capabilities include training  employees and defining how to apply SMS to the agency’s overall  oversight activities. To accomplish this, FAA’s guidance for implementing  SMS requires the Office of Aviation Safety, the Office of Airports, and the  Air Traffic Organization (ATO) each to develop implementation plans that  include schedules, procedures for acquiring and analyzing data, and  measures to track implementation progress. ATO has issued its SMS  implementation plan and has also created an SMS manual that provides  specific operational information and guidance regarding the daily  activities of ATO employees. In addition, it met its target to implement its  initial SMS operating capabilities in March 2010. The Offices of Airports  and Aviation Safety have yet to issue their implementation plans. However,  the Office of Aviation Safety has issued guidance and safety documents  that provide a general discussion of SMS, but they do not include a  schedule of specific activities or time frames for completion as called for  in the agencywide SMS guidance. Senior FAA officials told us Aviation  Safety has formed working groups and expects those groups that are  charged with defining the various SMS activities to meet the guidance  requirements. In addition, according to FAA officials, various offices  within Aviation Safety will be responsible for implementing processes to  fulfill SMS requirements.
Additionally, FAA’s guidance for implementing SMS requires the formation  of an agencywide SMS committee, which was chartered in July 2009. The  committee includes an executive council—whose members are the FAA  associate administrators from the offices of Aviation Safety, Airports, and  Commercial Space Transportation, and the Chief Operating Officer of  ATO—and a committee composed of SMS professionals from each of  those FAA offices. Chaired by the Office of Aviation Safety, the  agencywide committee is tasked with recommending policy and guidance  to the implementing FAA organizations and management.
Because SMS relies on data to identify emerging risks, FAA has an effort  under way to enhance its access to industry data and improve its analysis  capability. The ASIAS initiative is a collaborative government-industry  effort to share and analyze data. The Office of Aviation Safety’s SMS  implementation plan reiterates that data exchanges between ASIAS and  other sources are crucial elements of emerging risk analysis. FAA’s draft  plan for ASIAS notes that this effort will require access to existing and  previously unattainable data sources, enhanced analytical methodologies,  and technical advancements to support safety risk analysis that are not  achievable with current databases and analytical strategies. The draft plan  would cover ASIAS activities through 2022. FAA did not confirm when a  final plan would be completed.
While FAA has issued agencywide guidance on implementing SMS and has  some efforts such as ASIAS under way, it does not have a way to measure  or specific times to indicate full implementation. FAA officials told us that  the current efforts would provide a foundation for the full implementation  of SMS. But without a clear description of the activities to be completed  and time frames for their completion, it may be years before SMS is fully  implemented and its benefits are realized. In commenting on a draft of this  report, FAA officials noted that even with a clear description of the  activities to be completed and time frames for their completion, it will be  years before SMS is fully implemented and its benefits are realized. We  agree with FAA and note that specific time frames establish expectations  for FAA’s implementation of SMS and provide a means of accountability  for meeting those expectations.
section_title: FAA Has Identified and Is Addressing Challenges to Using Data for Safety Oversight
paragraphs: FAA has efforts under way to address two key challenges to using data  more effectively to manage risk. First, data are not coded to permit  electronic integration, analysis, and sharing. Second, data from two  voluntary reporting programs lack identifying details needed for some  types of analysis, and the data do not remain available for long-term  analysis.
subsections: section_title: Data Are Not Coded to Permit Automated Analysis
paragraphs: FAA and the other federal agencies that gather the data FAA uses for  analysis lack standard definitions, common identifiers, and common  classification schemes for both quantitative and narrative data. For  example, according to an NTSB official, databases used in ASIAS were not  designed for searches of specific products, manufacturers, or airlines. As a  result, FAA has had to develop common identifiers and standard  classification schemes so that automation can be used to quickly integrate  and analyze data from multiple external and internal sources. In addition,  narrative information, in particular, poses challenges. Narrative reports  from NTSB investigators and FAA inspectors, as well as narrative reports  submitted voluntarily by pilots, air traffic controllers, mechanics, and  others, constitute a rich source of information about safety events, but  could not be coded to permit automated analysis until the ASIAS text  analysis process was developed. Consequently, analysts are now able to  automate coding, integration, and analysis of data from different  databases—a time-consuming and costly process when manually  performed. According to a senior FAA official, this automated analysis  process is unique and pathbreaking and will allow for more efficient safety  analyses. When FAA studied wrong runway departures, for example, data  from six different databases—from NTSB, NASA, and its own offices— were extracted for analysis. FAA compared the cost and time needed to  extract and integrate data from multiple databases using an automated  versus a manual process for coding the data. FAA found that using the  automated coding process would cost about $6,000 and take 52 hours,  while the manual process was estimated to cost over $750,000 and take  more than 6,500 hours to complete. However, NASA noted that by using a  “data mapping” approach, it can conduct an automated analysis of ASRS  and other data sources in about 2 to 8 hours, depending on the complexity  of the analysis.
section_title: Data from Voluntary Reporting Programs Have Characteristics That Impede Analysis
paragraphs: Details of reported incidents are redacted from ASAP and FOQA data  before an FAA contractor analyzes the data. These details include the date,  time and flight number, and the names of the carrier or individuals  involved. We previously reported that these identifying details are  redacted to safeguard the participants in ASAP from enforcement or  disciplinary action and participants in FOQA from public release of the  data. Additionally, ASAP and FOQA data are retained for only 3 years.  Without identifying details and without maintaining data for longer  periods, opportunities for some analyses are limited. To allow the  contractor to perform more detailed analyses for FAA, the agency and the  industry have agreed on a process through which the ASIAS Executive  Board provides permission for the contractor to perform a specific,  defined analysis and to use data with the identifying details needed for  that particular analysis. At the conclusion of the analysis and with the  approval of the ASIAS Executive Board, a final report containing detailed  results is given to CAST to develop safety enhancements to mitigate the  identified safety issues. As of March 2010, three such analyses had been  completed.
The Department of Transportation (DOT) Inspector General (IG) and the  2008 independent review team found that FAA could improve its use of  ASAP data by analyzing these data for national trends, noting that by not  doing so, the agency is missing opportunities to help reduce recurrences  of safety events or to identify national patterns indicative of risks. FAA  concurred with this finding and said that it already has the ability to  conduct national trend analyses. The IG reported, however, that FAA  receives quarterly summaries of ASAP information from carriers of how  many ASAP reports they received each month, and these summaries do  not provide sufficient detail about ASAP events or corrective actions.  According to the IG, these reports generally contain information about the  number but not nature of ASAP submissions for that quarter and any  resulting safety enhancement.
While FAA’s contractor loses access to ASAP reports after 3 years, about  62 percent of ASAP reports appear in ASRS, along with other reports  voluntarily submitted by industry personnel, according to a NASA  official. NASA, which maintains ASRS for FAA, has access to the  identifying details in the submitted reports for no more than 90 days so it  can follow up as necessary to clarify any questions that the reports raise.  After that time, NASA removes the identifying details and incorporates  ASAP and ASRS reports into ASRS. ASRS is accessible to the public, and  NASA performs special analyses of ASRS data for FAA at its request.  NASA also publishes an online newsletter every month with summary  statistics and examples of a few reports selected to illustrate certain safety  issues, such as general aviation pilots’ use of new cockpit technology, and  issues safety alerts regarding significant safety events identified by ASRS  reports. However, NASA does not comment on these reports or respond to  any questions or issues raised by the authors, since its role is to act as an  “honest broker” as it collects and analyzes the data. By comparison, in the  United Kingdom (UK), an advisory board reviews a selection of reports  received, commenting on the appropriateness of the action taken and  answering questions. These responses are useful to the aviation  community because they communicate on commonly experienced safety  issues. In commenting on a draft of this report, NASA indicated that the  UK model would not be practicable for ASRS data because the UK  receives only about 300 reports annually compared with about 50,000  reports for ASRS. However, NASA noted that in the past it had an ASRS  advisory committee that had provided a forum for FAA and industry to  discuss corrective action. The agency acknowledged the need to  reestablish this committee. In addition, NASA noted that it receives  feedback on its safety alerts, which indicated positive corrective actions  for about 60 percent of the alerts.
section_title: FAA Is Developing a Plan for Using Data under SMS, but the Plan Does Not Fully Address Data, Analysis, and Staffing Requirements
paragraphs: As part of its efforts to develop initial SMS capabilities, FAA expects to  address how data and analysis will help it identify emerging aviation safety  risks. Specifically, FAA has a draft plan—which it calls a safety  management plan—that defines the agency’s analytical requirements and  the role of safety analysis in improving safety, especially as NextGen is  implemented. FAA’s draft plan—which covers the risk management and  safety assurance components of SMS discussed earlier in this report— recognizes the agency’s future need for data and analysis, but does not  specify requirements for them. For example, the draft safety management  plan does not define the level of accuracy and completeness needed for  the data, indicate what metrics and processes FAA will use to assess the  data, or identify any specific data. According to senior FAA officials, as of  February 2010, there was no specific date for finalizing the draft plan.
To meet its data challenges and develop needed analytical approaches,  FAA will also have to identify staff with both aviation operational  experience and statistical expertise to effectively analyze aviation safety  data in the future. While aviation safety experts we interviewed were  generally satisfied with the qualifications of current FAA analysts, they  expressed concerns about FAA’s capacity to meet future needs. Several  experts we interviewed said FAA needs additional qualified analysts, but  the draft safety management plan does not mention staffing requirements  for implementing FAA’s analysis strategy. In our 2009 report on FAA’s  human capital system, we made several recommendations to FAA on how  it can help ensure the continued hiring, recruitment, and retention of staff  needed to operate the national airspace system. According to FAA, it has  not yet determined how many additional analysts it will need; however,  the Office of Accident Investigation and Prevention has been approved to  hire 11 additional analysts in fiscal year 2010. Without a plan that includes  data and analysis requirements and staffing needs, the agency will not be  able to link the resources it needs to the data capabilities it requires for its  risk-based approach.
section_title: FAA Has Limited Access to Some Voluntarily Reported Data and Lacks Key Data to Assess the Safety Performance of Certain Industry Sectors
subsections: section_title: Voluntary Reporting Programs Generate Safety Information That Is Not Available from Other Sources, although the Data Have Some Limitations
paragraphs: prolem; withot them, the rik level wold  go p.
FAA’s voluntary reporting programs—ASAP, ASRS, FOQA, and VDRP— generate safety information that FAA does not identify through other  means. Whereas data from other sources are derived from inspections,  audits, and other agency reports, these voluntary programs rely solely on  cooperation between FAA and industry personnel. To obtain voluntarily  reported information that can be used to improve safety, FAA agrees not  to take enforcement action against carriers or industry personnel who self- report violations through ASAP, ASRS, and VDRP. Similarly, carriers that  operate ASAPs agree not to take disciplinary action against personnel who  self-report violations of FAA regulations or carriers’ operating procedures.  In both cases, this agreement holds only for actions that are reported in a  timely manner, were not intentional or criminal, did not involve drugs or  alcohol, did not result in accidents, and have not already been detected by  FAA. Conversely, personnel who do not voluntarily report violations  within the specified time face the threat of enforcement or disciplinary  action if the violations are discovered later. This combination of promised  immunity for self-reporting and threat of enforcement and disciplinary  action for remaining silent creates an incentive for industry personnel to  participate in the voluntary reporting programs.
Through ASAP, ASRS, and VDRP, airspace users, including air carriers, air  operators, and employees, self-report safety events and violations of their  operating certificates or company procedures. For example, under ASAP,  an employee reports an incident or event, which an event review  committee, composed of representatives from FAA, the carrier, and the  applicable employee group, then reviews. The ASAP event review  committee assesses a report to determine (1) if it meets the criteria  previously mentioned for inclusion in the program and (2), if included,  what follow-up actions, enhancements, or mitigations should be  implemented to address the safety concern.
While ASAP, ASRS, and VDRP rely on individuals or the air carrier to file  reports, FOQA data are generated by electronic equipment on aircraft,  which continuously records more than a thousand parameters of data for  individual flights. Vendors collect the data from carriers and can then, on  their behalf, analyze the data and transfer files to the carriers’ internal  analysis teams or forward the data files to an FAA contractor for inclusion  in ASIAS and subsequent analysis if a carrier is partnering with FAA. The  FAA contractor also receives ASAP reports approved by event review  committees. The contractor aggregates and analyzes the data from  participating carriers and, starting in 2010, will brief the ASIAS Executive  Board on a quarterly basis. These briefings will consist of the status  reports on directed studies, the number of FOQA and ASAP records, and  industry benchmarks that will enable carriers to compare their individual  safety performance relative to the national trends and prioritize their  internal safety initiatives. According to FAA, 28 carriers participate in  ASIAS. All ASIAS participants share ASAP data, and 13 carriers also share  FOQA data. FAA estimates that airlines contributing ASAP data to ASIAS  account for 80 percent of the flights of all commercial airlines with FAA- approved ASAP programs.
FAA and industry officials, as well as experts we talked to, agreed that  voluntarily reported data are critical to improving aviation safety.  Moreover, according to ICAO, FAA officials, and safety experts we  interviewed, voluntary reporting by operational personnel is a cornerstone  of SMS, because, as ICAO has stated, operational personnel are in the best  position to report the existence of safety hazards and to attest to what  works and does not work during everyday operations. In 2007, NTSB  reported that FOQA and ASAP programs are relevant to the safety  assurance component of SMS because they provide a direct means for air  carriers to evaluate the quality of their training and operations. Further,  these programs can also be used in the safety risk management  component of SMS because they can help identify emerging risks. In  addition, FAA and industry officials agreed that voluntary programs help  promote a healthy reporting culture and an increased awareness of safety  by industry personnel. Furthermore, FAA officials told us they believe that  the voluntary programs, such as ASAP, gather safety information that  would not be discovered 95 percent of the time. In addition, in  commenting on a draft of this report, NASA noted that voluntarily  reported data are valuable in learning why an event occurred.
Industry officials told us how their companies have used voluntarily  reported data to implement changes that respond to safety concerns. For  example, one airline analyzed ASAP data to decrease the number of  unstable approaches. Safety officials from another airline told us that  ASAP reports and FOQA data helped them to identify potential pilot  issues, suggest additional training, and adjust processes and checklists  based on human factors issues. They further commented that if pilots do  not self-disclose potential safety issues, airlines may be limited in their  ability to identify emerging safety trends.
Because of the importance of voluntarily reported data to proactive safety  analysis, NTSB and the DOT IG have also recommended that FAA further  encourage participation in voluntary programs. For example, in 2010,  NTSB recommended that FAA require the establishment of FOQA  programs by carriers regulated under part 121. As another example, the  DOT IG reported in May 2009 that FAA was not realizing the full benefits  of ASAP and recommended that the agency develop a central database for  all air carriers’ ASAP reports to be used for nationwide trend analysis.
While voluntarily reported data have been used to enhance safety, they  also have some limitations. First, the completeness of the data is  unknown, since reporting is voluntary, and there is no way to know how  many violations or safety situations are not reported. For example,  according to FAA, factors such as an individual’s awareness of ASRS,  motivation to report a situation, and perception of an incident’s severity  may influence the decision to submit an ASRS report. Second, the  completeness of the data is further limited because participation varies  among airlines, with 73 airlines participating in ASAP and 36 participating  in FOQA. Third, as discussed later in this report, inadequate data quality  controls can also limit the completeness of the data. For example, controls  may not be adequate to ensure that the data entered into a database have  been accurately compiled or processed. Fourth, the accuracy of  voluntarily reported data cannot always be verified. Voluntarily reported  data are subjective and are not always accompanied by supporting  documentation, such as statistics, measurements, or other quantifiable  information related to the reported events. According to an FAA analyst, a  variety of factors can influence the accuracy of the data, including the  reporter’s experience, visibility conditions, the duration of the event, and  any trauma experienced by the reporter. FAA notes, for example, that  even senior pilots’ estimates of how far aircraft descend during encounters  with turbulence often differ considerably from the actual distances  recorded on aircraft flight data recorders. Acknowledging this limitation,  NASA notes that the information is nonetheless valuable, since a reporter  is providing information on how it perceived the situation, which in large  part determines its reactions. Fifth, electronically collected data from  FOQA also have limitations. Vendors that process FOQA data explained  that software bugs and inaccurate sensors can affect data results. To  mitigate these problems, vendors review anomaly reports and validate  data prior to analysis.
In an effort to address the limitations of voluntarily reported data, NASA  developed a survey methodology project—NAOMS—in 1997 to  systematically collect information on safety events by conducting  telephone interviews with randomly selected airspace users such as pilots.  In our 2009 assessment of the survey, we concluded that NAOMS was a  successful proof of concept and that a similar project, adequately funded  and appropriately planned, could enhance FAA’s safety knowledge.  However, FAA maintained that FOQA provides better data by providing  precise rates of occurrence on multiple parameters collected by flight data  recorders that could obviate the benefits from NAOMS data. Nonetheless,  we concluded that the NAOMS survey could be useful in complementing  other databases, such as ASRS. The survey data, when properly analyzed,  could be used to call attention to low-risk events that could serve as  potential indicators for further investigation in conjunction with other data  sources. Furthermore, in commenting on a draft of this report, both NTSB  and NASA agreed on the usefulness of a survey similar to NAOMS in  complementing other data. NASA pointed out that NAOMS-type data could  provide the data for trends and explain “what” is happening in the system  while ASRS provides “why” it is happening. NTSB further noted that its  investigations of numerous serious incidents and accidents found that  FOQA data gave no indication of underlying problems.
section_title: Participants’ Concerns Limit FAA’s Access to Voluntarily Reported Data
paragraphs: Although FAA, carriers, and experts we interviewed agreed that  voluntarily reported data are an important source of information for  understanding and addressing safety issues, some carriers and industry  personnel are not participating in FAA’s ASAP and FOQA voluntary  reporting programs. According to airline and FAA officials, two factors  have primarily affected participation: (1) the fears of employees that their  employers will take disciplinary action to address self-reported violations  and (2) the costs to the airlines of purchasing and installing FOQA  equipment and analyzing the data.
Specifically, we found that, partly because of employees’ fears of  disciplinary action, from 2006 through 2008, four large air carriers and  their pilot unions suspended their ASAP. According to safety officials at  one airline, pilots had raised concerns about letters of reprimand or  unpaid time off that directly resulted from ASAP reports. Pilot unions and  air carriers that we spoke with agreed on the importance of  confidentiality. However, pilot union officials we interviewed also  expressed concern about whether airlines were ensuring the  confidentiality of ASAP reports and suggested that, for an airline’s safety  culture to improve, pilots, in particular, must be able report certain events  without fear of reprisal. Despite these concerns, as of June 2009, the four  carriers and pilot unions that had suspended their ASAP had restarted or  were restarting their participation in the program.
Additionally, according to industry officials and experts we interviewed,  the costs of purchasing and installing equipment and analyzing data have  deterred participation in FOQA, especially for smaller carriers. Several  large carriers we interviewed said more than 50 percent of their fleets  already have FOQA equipment and they plan to expand their fleets’  capability by retrofitting aircraft or ensuring that new aircraft include the  equipment. By contrast, officials from smaller carriers were concerned  about costs and estimated that installing FOQA equipment would cost an  average of $12,000 for each new aircraft and up to $35,000 for retrofits of  older aircraft models. As a result of cost concerns, according to airline  officials, only 11 of 65 smaller carriers have approved FOQA programs,  and according to FAA, an additional small carrier FOQA program is  pending. Officials from these smaller carriers said that incentives to cover  equipment costs, which are not currently available, would help increase  participation. FAA officials noted that as carrier fleets age, newer  replacement aircraft will already be fitted for FOQA equipment and,  therefore, costs for participating will continue to decrease. However, the  life span of an aircraft is usually at least 30 years for large carriers, so the  transition to a fully equipped fleet could take decades.
Currently, large carriers are the principal participants in FOQA and ASAP,  and they provide service for the majority of passengers on domestic and  international flights. Nonetheless, we found that 4 of the 25 large carriers  do not have active FOQA programs and 1 large carrier did not have an  ASAP. According to FAA officials, an additional 4 carriers have FOQA  programs pending. To the extent that operators do not participate in the  programs, they do not obtain information that they could use to monitor  and improve the safety performance of their aircraft, related equipment,  and personnel, and to the extent that they do not partner with FAA,  opportunities to identify nationwide safety trends and improvements are  lost. To encourage greater participation in FOQA and ASAP, FAA provides  training to smaller carriers on how to develop versions of both programs  that do not require as much capital investment but do allow the carriers to  collect unique safety data. However, FAA lacks carrier-specific  information on why air carriers are not participating in voluntary reporting  programs. Having such information would allow FAA to identify further  actions to encourage participation.
section_title: FAA Lacks Data to Assess the Safety of Certain Industry Sectors
paragraphs: FAA’s ability to monitor and manage risk for certain industry sectors, such  as general aviation, air ambulance operators, and air cargo carriers, is  limited by incomplete data. While FAA collects data on actual flight hours  and numbers of departures for large air carriers that operate under part  121 and scheduled flights with fewer than 10 seats that operate under part  135, it does not collect actual flight activity data for smaller air carriers  that provide on-demand service, such as air taxis and air ambulances, and  general aviation operators—sectors that have had a higher number of fatal  accidents in recent years than large air carriers. For instance, in 2008,  large air carriers providing scheduled service had 20 accidents, none of  which were fatal. By comparison, during that same year, there were 1,559  general aviation accidents, including 275 fatal accidents involving 495  fatalities. (Fig. 2 shows the trends in general aviation accidents.) Without  information on the number of general aviation flights, FAA cannot  compare the safety performance among industry sectors or assess trends.
Additionally, the number of accidents for air ambulance and air cargo  operators points to safety vulnerabilities in these areas. From 1998  through 2008, the air ambulance sector averaged 13 accidents per year.
While the total number of air ambulance accidents peaked at 19 in 2003,  the number of fatal accidents peaked in 2008, when 9 fatal accidents  occurred. Without data on the number of flights or flight hours, FAA and  the air ambulance industry are unable to determine whether the increased  number of accidents has resulted in an increased accident rate, or whether  it is a reflection of growth in the industry. According to FAA, it annually  surveys a sample of potentially active general aviation aircraft, and in the  latest survey in 2008, it surveyed all air ambulance operators. However, we  noted in our April 2009 testimony that less than 40 percent of the air  ambulance operators responded, raising questions about the reliability of  the information collected. Similarly, our review of air cargo safety found  that small cargo carriers had more accidents and fatal accidents than large  cargo carriers, but the available information was not sufficient to assess  the significance of this difference. We found that smaller air cargo carriers  averaged 29 accidents per year from 1997 through 2008, while large cargo  carriers averaged 8 accidents each year during this period. However, a  lack of operations data for small cargo carriers makes it difficult for FAA  to prioritize risks and better target safety improvements and oversight to  the areas of highest risk.
To address the lack of data, we previously recommended that FAA identify  the data necessary to better understand the air ambulance industry and  develop a systematic approach for gathering and using these data.  Similarly, we recommended that FAA gather comprehensive and accurate  data on smaller air cargo operations (those covered under part 135) to  gain a better understanding of air cargo accident rates and better target  safety initiatives. FAA agreed with both recommendations, but has not  fully addressed either. In response to our recommendation on air  ambulance data, FAA has surveyed all helicopter air ambulance operators  to collect flight activity data. However, as mentioned earlier in this report,  FAA’s survey response rate was low, raising questions about whether this  information can serve as an accurate measure or indicator of flight  activity. FAA plans to evaluate ways to collect the air cargo data over the  long term.
section_title: FAA Has Various Processes in Place to Help Ensure Data Quality, but Weaknesses Still Exist
subsections: section_title: FAA Has Taken a Number of Steps to Help Ensure Data Quality
paragraphs: FAA, along with the international aviation community, recognizes that  high-quality data—that is, reliable, valid data—are essential to the  effectiveness of a data-driven approach to safety, such as SMS. To help  ensure data quality, FAA has issued guidance, established procedures, and  implemented controls. For example, FAA has issued an order that  establishes an agencywide policy on data management. This policy  applies to all information from FAA and other sources used to perform the  agency’s mission. In accordance with the data management order, FAA’s  Office of Aviation Safety has established a data management framework  that includes a four-step process for importing data from other FAA  offices and external sources. This process includes    data acquisition—obtaining information from various data owners,    data standardization—validating data by comparing a new data set with  previous data sets to identify inconsistencies,    data integration—translating data values into plain English and correcting    data loading—importing data into the agency’s own systems.
This four-step process applies to 10 of the 13 databases we reviewed— 8  maintained by FAA offices and 2 maintained by NTSB and USDA. (The  process applies to 2 of the 4 voluntary reporting databases.) NTSB and  USDA also have data quality assurance processes that apply to their  databases. For example, NTSB conducts annual reviews of aircraft  accident data, and according to USDA, airport managers and wildlife  biologists are asked to check data from their respective airports and  report errors.
In addition, to help ensure the quality of its data, FAA applies various  quality controls, such as validation and verification processes, to better  ensure accuracy and completeness. We have identified some standard  quality controls that an agency should employ to achieve these high- quality results. For example, agencies should have managers review data,  have procedures in place to verify that data are complete and accurate,  and correct erroneous data. We assessed 12 databases against these  standard quality controls and found that the extent to which such controls  were applied varied. (See fig. 3.) While NTSB’s aviation accident and  incident database (NTSB), VDRP, and USDA’s wildlife strike database  applied all five of the quality control activities we considered for this  analysis to some extent, the remaining 9 databases lacked one or more  quality control activity. In addition, we found that all of the databases we  reviewed fully or to some extent had procedures in place to validate and  edit data to help ensure that accurate data are entered into electronic  systems and to help ensure that erroneous data are identified, reported,  and corrected. To the extent that the databases lack various controls, FAA  lacks assurance that the information it uses for oversight is accurate and  complete.
While the databases we reviewed varied in the extent to which they had  standard quality controls, FAA has other data quality controls in place for  some databases that we consider good practices for handling data, as  shown in table 3.
Furthermore, other data quality controls apply to the voluntary reporting  systems we reviewed. For example, as previously discussed, an event  review committee at each participating carrier is tasked with reviewing  and analyzing reports submitted under ASAP. This committee determines  whether such reports qualify for inclusion in the program, identifies and  proposes solutions for actual or potential problems with information  contained in the reports, and annually reviews the ASAP database to  determine whether corrective actions have been effective in preventing or  reducing the recurrence of targeted safety-related events.
For ASRS, NASA officials told us that each individual ASRS report is  reviewed by two expert analysts within 3 days of receipt. Each report  captures data for seven criteria and data fields, which are screened to  ensure accuracy. The analysts also evaluate the database to ensure that  publicly released ASRS data do not include information that might identify  the reporter. In addition, for FOQA, vendors have quality assurance  procedures in place. For instance, one vendor’s procedures include  automated checks and tests of each flight data file to detect parameter  problems (for example, anomalies in the flight data attributable to faulty  sensors), reports of anomalies created for the affected airline, and manual  reviews of any data indicating warning-level risk events. These controls  are designed to monitor the reliability and validity of FOQA data and to  identify technical problems that affect data quality and need to be  corrected. However, according to the vendor, some data will always be  missing because of data-recording equipment failures or lost flight data  cards, but such missing data do not affect the statistical validity of the  large FOQA data set.
section_title: FAA Is Taking Steps to Address Data Quality Weaknesses Identified by Its Analysts and Others
paragraphs: FAA is taking steps to address data weaknesses identified by its analysts,  the DOT IG, and us. For example, FAA officials told us that to mitigate  problems from external data sources, they combine data from various  sources to validate analysis results. According to FAA analysts, they  typically combine AIDS data with wildlife strike, ASRS, or PDS data using  a manual process to verify study findings. As another example, we  previously reported the importance of aggregating data from multiple  sources to understand icing-related incidents. We reported that the AIDS  database included 200 icing-related incidents involving large commercial  airplanes that occurred from 1998 through 2007. During this same time  period, ASRS received over 600 icing- and winter weather-related incident  reports involving large commercial airplanes. These incidents revealed a  variety of safety issues such as runways contaminated by snow or ice,  ground deicing problems, and in-flight icing encounters. This suggested  that risks from icing and other winter weather operating conditions may  be greater than indicated by the AIDS database.
FAA analysts also communicate with other data providers and experts  about quality concerns or sometimes make independent corrections to  data. For example, one FAA official told us that analysts communicate  with NTSB to report incorrect data in a field and then rely on NTSB to  correct the data in its database. FAA analysts said that they retrieve data  from public Web sites and then collaborate with subject matter experts to  identify and correct any errors in those data. They make such corrections  based on their knowledge of the data’s reliability and their own expertise  in working with such data. Analysts also said they use information from  the narrative sections of a report to correct data fields.
In addition, the IG has identified weaknesses in the quality of specific data,  which FAA is working to address. For instance, according to the IG, ATOS  data are inconsistent and incomplete because the database has undergone  multiple revisions since it was introduced in 1998 and some data fields  have changed from one year to another. During these revisions, some data  have been lost. Though designed to improve ATOS’s value and usability as  an inspection tool, the revisions limit opportunities for analysis of long- term trends to the extent that data fields have changed over time. The  revisions do not, however, affect FAA’s ability to analyze the data at a  particular point in time. In addition, the process for reporting inspection  findings is time-consuming and creates an incentive for inspectors to  underreport their inspection results. To report, inspectors must complete  a Yes/No checklist and, for every No check, provide a narrative  explanation. According to the 2008 independent review team, inspectors  have an incentive to check Yes so they can complete their reports in a  timely manner. Consequently, the system may underreport problems that  inspectors have identified but not taken the time to report.
The IG has also found, and FAA agreed, that OEDS has some missing and  incorrect data on operational errors and pilot deviations because  personnel have intentionally or unintentionally misclassified these events.  Such misclassification is problematic because it can lead to errors in  FAA’s assessment and reporting of how well the agency is meeting its  annual performance targets for operational errors and pilot deviations. In  2007, the IG investigated operational errors at the Dallas-Fort Worth  terminal radar approach control facility and found that FAA air traffic  managers had intentionally misclassified operational errors as either pilot  deviations or nonoccurrences. On the basis of this finding, FAA agreed  with the IG’s recommendation that the agency establish a follow-up  mechanism to ensure compliance with guidance for investigating pilot  deviations.
Finally, over the years, we have identified weaknesses in the quality of  aviation safety data that hinder FAA’s ability to oversee the industry. In  response, the agency has taken steps to address many of the problems that  we have identified. For example, in our 2007 review of runway safety, we  found that FAA’s categorization of the severity of runway incursions  involves a level of subjectivity, raising questions about the accuracy of the  data. We reported that an internal FAA audit of 2006 runway incursion  data found that the subjectivity of the severity classifications has the  potential to affect the accuracy of the classifications. We also found that  FAA did not systematically collect data on the number of runway overruns  that do not result in damage or injury that could be used for analytical  purposes to study trends in and causes of these incidents. In July 2009,  FAA indicated that it was working to establish procedures that will ensure  that all runway overruns and other excursions are reported.
section_title: Conclusions
paragraphs: Aviation safety data are critical to FAA’s safety oversight and its planned  implementation of SMS. To its credit, FAA has taken steps to help ensure  the quality of the data it uses, such as implementing quality controls to  help ensure that errors are identified, reported, and corrected, but these  procedures are not applied consistently across all databases. Although  FAA is developing a plan that will address how data fit into its new  oversight approach, that plan lacks a description of the data that will be  required to conduct proactive data analyses, an inventory of the skills  personnel will need to perform such analyses and help ensure data quality,  and a description of the steps needed to address continuing data quality  problems. Unless the plan links FAA’s data requirements and staffing  needs to the analyses that will drive its proactive safety management  system and addresses the agency’s data quality problems, available data  may not be as reliable and useful as they could be to support SMS.
While NextGen technologies and procedures are intended to increase the  safety, efficiency, and capacity of the national airspace system, their  introduction could have unintended effects on system safety if not done in  a comprehensive manner. As FAA improves its ability to integrate and  analyze data from multiple sources, it plans to increase its capacity to  model the impact of NextGen changes and to identify and manage risks.  Because some NextGen changes are already taking place, it is urgent that  FAA move with all deliberate speed to advance its analytical capability.
The data that FAA obtains through voluntary reporting programs afford  insights into safety events that are not available from other sources and  are critical to improving aviation safety, but participation in these  programs has been limited by concerns about the impact of disclosure  and, especially in the case of smaller carriers, by cost considerations.  Efforts such as the training FAA provides to smaller carriers on how to  develop programs that require less capital investment have the potential to  increase participation and improve safety. However, without carrier- specific information on why air carriers are not participating in these  programs, FAA cannot determine if its efforts to increase participation are  sufficient.
section_title: Recommendations for Executive Action
paragraphs: To help improve and expand FAA’s capability to use data for aviation  safety oversight, we recommend that the Secretary of Transportation  direct the FAA Administrator to take the following four actions:    develop and implement a comprehensive plan that addresses how data fit  into FAA’s implementation of a proactive approach to safety oversight and  ensure that this plan fully describes the relevant data challenges (such as  ensuring data quality and continued access to voluntarily reported safety  data), analytical approaches, and staffing requirements and integrates  efforts to address them;    given the importance of high-quality data, extend standard quality  controls, as appropriate, to the databases that support aviation safety  oversight to ensure that the data are as reliable and valid as possible;    proceed with all deliberate speed to develop the capability to model the  impact of NextGen changes on the national airspace system and manage  any risks emerging from these changes; and    systematically identify the reasons that carriers are not participating in  voluntary reporting programs, such as through a survey, and identify and  implement further steps to encourage greater program participation,  especially by smaller carriers.
section_title: Agency Comments
paragraphs: We provided copies of a draft of this report to DOT, NASA, USDA, and  NTSB for their review and comment. DOT agreed to consider our  recommendations. DOT and NASA provided technical corrections and  clarifications, which we incorporated as appropriate. USDA had no  comments. NTSB generally agreed with our findings and  recommendations to FAA and provided several comments. First, NTSB  noted that our use of the terms “reactive” and “proactive” implied a new  approach to aviation safety data analysis that is different from past  analyses of accidents and incidents to improve safety. The agency noted  that a more efficient, effective approach to safety analysis should continue  to include FAA’s previous reactive approach as well as new, more  predictive capabilities. We agree with NTSB’s comment and note that our  report indicates that FAA plans to continue to use data to analyze past  safety events as it also works to use data more proactively. NTSB further  noted that the success of SMS will depend on the maturation of FAA’s data  analysis capabilities.
Second, NTSB agreed with our finding that the lack of a final plan for  ASIAS and for SMS implementation, which are key elements of FAA’s  planned proactive safety analysis capability, was a cause for concern. The  agency noted that it had made several recommendations to FAA to require  SMS programs for part 121, part 135, and part 91 carriers and that FAA had  not yet taken action to require these programs.
Third, regarding FAA’s access to voluntarily reported data, NTSB agreed  with our finding that the redaction of flight details from ASAP and FOQA  analyses is a serious constraint on the thoroughness and potential utility of  ASIAS and other assessments of safety. If FAA does not address these data  limitations, NTSB observed, such constraints are likely to pose serious and  continuing threats to the broader use of voluntary reporting programs to  support safety analysis. In NTSB’s view, our recommendations to FAA do  not go far enough to recommend mechanisms besides redaction, such as  statutory exemptions from disclosure, to protect these data from  enforcement and disciplinary uses or public release. We did not revise our  recommendations to FAA to include these issues because, while we found  that participation was temporarily affected, in part, by employees’ fears of  disciplinary action by their employers, we did not find evidence that  participation was inhibited by the fear of enforcement action by FAA or  public disclosure. In addition, our work indicated that the current  mechanisms to protect the data appeared to be working.
Fourth, regarding FAA’s access to data on various safety events, NTSB  noted the importance of FAA collecting the necessary data to support its  new approaches to data analysis rather than simply combining existing  data sources into an analysis program. NTSB also agreed with our finding  that independent survey efforts like NAOMS could provide a useful  complement to other data sources, including FOQA, in providing improved  data quality and analysis capabilities.
Finally, NTSB agreed with our finding that the availability of operations  data for sectors other than large commercial carriers (i.e., part 121  operators) is severely limited. The agency noted that accurate flight  activity data are not available for most of these operations and must be  estimated from FAA’s annual survey of a sample of active general aviation  aircraft. NTSB also pointed out that FAA does not require reporting for the  majority of equipment reliability or maintenance related events. To  address these shortcomings, NTSB noted its recent recommendation to  FAA to take steps to increase general aviation reporting to FAA’s service  difficulty reporting system. To correct these and other data deficiencies,  NTSB believes that FAA should explore the development of new aviation  safety data collection techniques or methods to supplement current areas  of data deficiency.
As arranged with your offices, unless you publicly announce its contents  earlier, we plan no further distribution of this report until 30 days after the  date of this letter. At that time, we will send copies of this report to  relevant congressional committees, the Secretaries of Transportation and  Agriculture, the Administrator of the Federal Aviation Administration, the  Chairman of the National Transportation Safety Board, and the  Administrator of the National Aeronautics and Space Administration. We  will also make copies available to others on request. In addition, this  report will be made available at no charge on the GAO Web site at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact me  at (202) 512-2834 or dillinghamg@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made major contributions to this report  are listed in appendix II.
Gerald L. Dillingham, Ph.D Director, Physical Infrastructure Issues  .
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: In this report, we assessed the Federal Aviation Administration’s (FAA)  capacity to use available data to oversee aviation safety. To do so, we  addressed the following questions: (1) How does FAA use data to oversee  aviation safety, and what changes, if any, has it planned? (2) To what  extent does FAA have access to data for monitoring aviation safety and the  safety of various aviation industry sectors? (3) What does FAA do to help  ensure the quality of the data it uses to oversee aviation safety?
To perform our review, we selected 10 safety events that were among  those previously identified as key by the National Aeronautics and Space  Administration’s (NASA) National Aviation Operations Monitoring Service  (NAOMS) or by the FAA-industry Commercial Aviation Safety Team  (CAST). This selection allowed us to focus our review on a manageable  subset of FAA oversight activities and data sources. (See table 4.) We then  identified 13 databases available to FAA that contained data on these  safety events and reviewed these databases. The databases are described  in the background section of this report.
To determine how FAA uses data to oversee aviation safety, we reviewed  reports by FAA, the International Civil Aviation Organization, and industry  and other published documents. In addition, we interviewed FAA, industry  associations, and other selected industry groups.
To determine the extent to which FAA has access to data for monitoring  aviation safety and the safety performance of various aviation industry  sectors, we interviewed FAA data analysts, contractors, and other officials  responsible for data management. We also reviewed previous GAO reports  on FAA’s access to data on certain aviation sectors, such as air  ambulances and air cargo operations.
To determine how FAA ensures the quality of its data, we interviewed FAA  and industry officials (see table 5). In addition, we reviewed assessments  of selected FAA data by the Department of Transportation Inspector  General and an independent review team appointed by the Secretary of  Transportation in 2008. We also derived a number of data quality  principles from our previous work on internal controls, and we assessed  the quality of 12 of the 13 selected aviation safety databases by comparing  our data quality principles with FAA’s practices for ensuring data quality.  These principles include ensuring that the data are complete and accurate,  measure intended safety concerns, and are useful for their intended  oversight purposes. To measure the extent to which FAA’s practices were  consistent with these principles, we evaluated information and all other  materials regarding the databases using a three-point scale. To validate the  results, multiple reviewers independently scored each principle. When the  initial scores differed, the reviewers collectively agreed on a final score for  each principle. Further, we used the results of GAO studies that  considered the availability, quality, and use of data in aviation safety  oversight.
In addition, to address all three research questions, we individually  interviewed 10 aviation safety experts and asked them to identify  challenges to using data for overseeing aviation safety, the reasonableness  of FAA’s current and planned efforts to use aviation safety data, and ways  that FAA could enhance its data collection and analysis processes to  improve its oversight capabilities. We selected experts who represent a  cross section of aviation stakeholders, including persons with general  knowledge of aviation safety, aircraft operations, human factors, aircraft  maintenance, and air traffic control. The experts have operational,  academic, or other professional expertise in these areas. Those experts are    Mr. Basil Barimo, Vice President, Safety and Operations, Air Transport    Mr. James Burin, Director of Technical Plans and Programs, Flight Safety    Kim Cardosi, Ph.D., U.S. Department of Transportation, Volpe Center;    Todd Curtis, Ph.D., Director, The Airsafe.com Foundation;    Mr. John Goglia, Senior Vice President for Aviation Operations and Safety  Programs, JDA Aviation Technology Solutions, former board member of  the National Transportation Safety Board;    Mr. Keith Hagy, Director, Engineering and Air Safety, Air Line Pilots    Brigadier General Leon Johnson (Air Force, retired), former Flight  Operations Manager, United Parcel Service;    Mr. Bruce Landsberg, Executive Director, Aircraft Owners and Pilots    Thomas Weitzel, Ed.D., Associate Professor, Embry Riddle Aeronautical    Mr. Dale Wright, Director, Safety and Technology, National Air Traffic  Controllers Association.
section_title: Appendix II: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the person named above, Teresa Spisak, Assistant Director;  Elizabeth Eisenstadt; N’Kenge Gibson; H. Brandon Haller; Erica Miles; and  Richard Scott made key contributions to this report.