Aviation Safety: Data Problems Threaten FAA Strides on Safety Analysis System
Published: 1995-02-08T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Pursuant to a congressional request, GAO reviewed the Federal Aviation Administration's (FAA) Safety Performance Analysis System (SPAS), focusing on: (1) whether FAA is effectively managing the SPAS acquisition; (2) the extent to which SPAS will rely on Aviation Safety Analysis System (ASAS) databases; and (3) whether FAA is effectively addressing known data quality problems with the ASAS databases.
section_title: What GAO Found
paragraphs: GAO found that: (1) FAA has generally implemented good development and acquisition procedures for SPAS; (2) FAA has maximized user involvement and system prototyping in developing and evaluating SPAS; (3) FAA has reduced SPAS development risks by using an independent verification and validation agent; (4) FAA is exploring the potential of its proposed corporate-wide area network to accommodate SPAS in order to avoid the acquisition of duplicate communication networks; (5) FAA cost estimates for SPAS software may not be reliable, since they are subjective; (6) FAA lacks a strategy for improving SPAS data sources, particularly ASAS, which jeopardizes the system's utility; (7) ASAS databases contain incomplete, inaccurate, and inconsistent data on airline inspections; (8) FAA has not yet defined its long-term data quality goals; and (9) if FAA fails to improve ASAS data, it could improperly target its limited inspection and certification resources on less important problems.

section_title: Letter
paragraphs: As requested, we are reporting to you the results of our review of the Safety Performance Analysis System, an automated decision support system that the Federal Aviation Administration (FAA) is acquiring to aid it in targeting its limited aviation inspection and certification resources. We performed this review because of our long-standing interest in helping to improve FAA’s inspection and certification programs.
We are providing copies of this report to the Secretary of Transportation, the Administrator of FAA, the Director of the Office of Management and Budget, and interested congressional committees. Copies will also be made available to others upon request.
Please contact me at (202) 512-6253 if you have any questions concerning this report or other Transportation-related information technology issues. If you have any questions concerning Transportation programs, please contact Kenneth M. Mead, Director of Transportation and Telecommunications Issues, at (202) 512-2834. Other major contributors to this report are listed in appendix II.
subsections: section_title: Introduction
paragraphs: To help ensure the safety of the more than one million people who travel on thousands of flights throughout the United States each day, FAA inspects and certifies the aviation community’s compliance with FAA regulations. To better focus its limited inspection resources, FAA needs quick access to meaningful information about airlines, aircraft, pilots, and more. However, FAA currently does not have this capability. To address this limitation, FAA is acquiring the Safety Performance Analysis System (SPAS), an automated decision support system to aid FAA in targeting its inspection and certification resources on those areas that pose the greatest aviation safety risks.
subsections: section_title: FAA Inspection and Certification Programs: A Brief Description
paragraphs: The Federal Aviation Act of 1958, as amended, requires FAA to promote the highest degree of aviation safety and establishes the safety of air passengers as a joint responsibility of airlines, aircraft manufacturers, and FAA. The airlines are responsible for operating their aircraft safely, aircraft manufacturers are responsible for designing and building aircraft that meet FAA regulations, and FAA is responsible for, among other things, (1) certifying that an airline is ready to operate and (2) conducting periodic inspections to ensure continued compliance with safety regulations. FAA is also responsible for certifying that aircraft produced in the United States or imported by domestic companies and individuals meet minimum safety standards before the aircraft can be operated.
To carry out its inspection responsibilities, FAA employs 2,300 inspectors located in 91 Flight Standards District Offices (FSDO), International Field Offices, and Certification Management Offices throughout the United States. These inspectors oversee more than 17,900 commercial aircraft, 4,800 repair stations, 401,060 aircraft mechanics, 642 pilot training schools, 193 maintenance schools, 665,000 active pilots, and 184,400 active general aviation aircraft.
FAA inspectors perform four principal functions: (1) airline operation certification, (2) routine surveillance (a process of periodic inspections of airlines and aviation-related activities), (3) accident and incident investigations, and (4) safety promotion. FAA divides its surveillance or inspection activities into three categories—operations, maintenance, and avionics. Operations inspections focus on such items as pilot performance, flight crew training, and in-flight record keeping. Maintenance inspections examine the airline’s overall maintenance program, including personnel training and established policies and procedures. Avionics inspections focus on the condition of electronic components of the aircraft.
To carry out its aircraft certification responsibilities, FAA has about 825 engineers and others to oversee the certification of new aircraft and the continued airworthiness of the existing fleet. To assist its engineers, FAA also delegates certification activities, as necessary, to designated, FAA-approved employees of manufacturers. The FAA engineers, in turn, oversee the activities of these designees.
section_title: FAA’s Need to Better Target Its Inspection Resources Is Well-Documented
paragraphs: The size of FAA’s inspection and certification workforce, while allowing it to perform its “must do” work, has prevented it from completing other important aviation oversight activities that it designates as “should do.” To assist FAA in maximizing the efficiency and effectiveness of its limited workforce, we have long encouraged it to better focus its inspection activities on those entities and areas that pose the greatest risk to aviation safety. In 1987, we recommended that FAA, in addition to having minimum standards for the type and frequency of airline inspections, target airlines displaying risk precursors (that is, characteristics that may indicate safety deficiencies). Again in 1988, we reported that by monitoring risk precursors, FAA could target for intensive inspection those airlines most likely to experience safety compliance problems, thereby improving the quality of information available on the airlines’ compliance with regulations. Similarly, we reported in 1991 that FAA needed a mechanism to make more effective use of its limited resources. We further reported that a system that systematically and uniformly determined risks could provide FAA with information vital to enhancing its inspection program. Finally, we recommended, in 1993, that FAA develop criteria for targeting inspections on high-risk conditions.
section_title: SPAS: A Brief Description
paragraphs: FAA’s response to our findings on its inspection program was to develop an automated decision support system for FAA managers, safety inspectors, and certification engineers in headquarters and field offices. This system, begun in February 1991 and designated as SPAS, is planned to be a user-friendly tool for (1) quick analysis of safety-related data,(2) generation of standard and ad hoc indicators (that is, precursors) of safety performance, and (3) identification of safety-related risk areas for investigation, either through analysis of the underlying data used to generate the risk precursor or through on-site inspection of the risk item.
subsections: section_title: SPAS History and Status
paragraphs: FAA decided in early 1991 to develop an aviation safety performance analysis system to aid it in managing its inspection program. In May 1993, FAA completed development and installation of the initial SPAS prototype at 12 field offices, FAA headquarters, and the Air Force’s Air Mobility Command. By the end of 1995, FAA plans to have developed the first operational SPAS release, which is to offer additional functions and performance capabilities above those of the prototype, such as the ability to look at the source data behind the indicators. This first operational release is to be installed at up to 30 locations. Development of the final operational SPAS release is scheduled to be completed in late 1997. This version is to be deployed to as many as 140 locations. To date, FAA has spent $6.3 million on the initial and enhanced prototypes. FAA estimates that SPAS will cost a total of $32 million to develop and install.
SPAS is to have a powerful graphical user interface that displays performance indicators in such a way that users can easily spot areas for further inquiry. FAA plans four categories of indicators or risk precursors: (1) air operator, (2) air agency (for example, flight and mechanic schools, aircraft repair stations, and so forth), (3) aircraft, and (4) air personnel. In developing the indicators, FAA is focusing first on air operators and air agencies. To date, 25 indicators have been developed and are being generated by the SPAS prototype—19 for air operators and 6 for air agencies.
section_title: SPAS Management Structure
paragraphs: FAA has established a SPAS program office, within the Office of Flight Standards, to manage SPAS. The program office is supported by the FAA Technical Center in Atlantic City, New Jersey. The Technical Center, in turn, has contracted with the Department of Transportation’s Volpe National Transportation System Center (VNTSC) for technical and analytical support, such as developing and evaluating the SPAS prototypes and defining the safety indicators. VNTSC has contracted with UNISYS Corporation to provide SPAS hardware and develop applications software in accordance with defined user requirements.
Overall SPAS program guidance and direction is provided by the SPAS Steering Committee, which is chaired by the SPAS program manager and includes representatives from four FAA regions, the FAA Office of Integrated Safety Analysis, and the Department of Defense. The Steering Committee’s responsibilities include defining systems requirements, approving SPAS indicators, and monitoring system development and implementation.
The SPAS program office is also supported by air operator, air agency, aircraft, and work program planning expert panels. These panels are responsible for defining and proposing indicators, identifying data sources for generating these indicators, and reengineering the inspection functions in light of SPAS capabilities.
section_title: Objectives, Scope, and Methodology
paragraphs: We reviewed SPAS because of our long-standing interest in helping FAA to improve its inspection and certification programs. Our objectives were to determine (1) whether FAA is effectively managing the SPAS acquisition, including its communication network, and (2) the extent to which SPAS will rely on Aviation Safety Analysis System (ASAS) databases and whether FAA is effectively addressing known data quality problems with these databases.
To accomplish our first objective, we interviewed SPAS program management about guidance governing the acquisition, and we reviewed this guidance to ensure that it provided a reasonably structured and disciplined basis for acquiring SPAS. Our review included analyzing the guidance relative to Office of Management and Budget (OMB) Circular A-109 and our 1994 report addressing how leading organizations manage information technology investments. We then interviewed program and contractor officials and reviewed system development documentation and plans to determine whether actual SPAS development processes and practices were consistent with the guidance and whether these processes and practices were exposing the program to unnecessary risks. In particular, we focused on system requirements analysis and definition, verification and validation, cost estimating, system architecture alternatives analysis, and communications planning.
We also interviewed SPAS users at four field locations that are currently operating the SPAS prototype to determine their involvement in defining SPAS requirements and their reaction to and satisfaction with the prototype. These four sites were the Flight Standards Division of the Western Pacific Region; the Van Nuys, California, FSDO; the San Jose, California, FSDO; and the Bedford, Massachusetts, FSDO. We also witnessed the operation of the prototype at these locations, and operated the prototype at the contractor’s facility in Cambridge, Massachusetts. In addition, we reviewed available SPAS program management and system development documentation, such as the SPAS Functional Description Document, SPAS working group minutes, the SPAS verification and validation contract proposal, SPAS cost estimate and budget requests, and SPAS alternative architectures analysis.
In addition, to ascertain acquisition plans and whether these plans would satisfy SPAS needs, we interviewed program and contractor officials, as well as FAA officials responsible for acquiring FAA-wide and Office of Flight Standards communication networks. In doing so, we discussed SPAS communications requirements and steps underway to satisfy them.
To accomplish our second objective, we interviewed SPAS program officials and Office of Flight Standards information resource management officials and reviewed SPAS documentation to determine what FAA and non-FAA databases will be used to generate SPAS indicators. We then discussed with these officials the accuracy, completeness, and consistency of the data residing on these databases and what plans and initiatives are underway to address any data quality shortfalls and what assurances they had that any quality problems would be addressed in time for SPAS deployment. We also reviewed published GAO and FAA reports and studies on the quality of the data in these databases, and interviewed Office of Flight Standards officials as to the status of actions to address any recommendations made.
We conducted our audit work primarily at FAA headquarters in Washington, D.C.; and VNTSC and UNISYS Corporation in Cambridge, Massachusetts. We also communicated frequently with the FAA Technical Center in Pomona, New Jersey.
Throughout our review, we discussed our preliminary results with the Director of the Office of Flight Standards. In addition, the Department of Transportation and FAA provided oral comments on a draft of this report. Their comments and our evaluation of these comments are contained in chapters 2 and 3 of this report. Additional comments provided on the contents of the draft report have been incorporated as appropriate throughout the report. We conducted our work between November 1993 and November 1994, in accordance with generally accepted government auditing standards.
section_title: FAA Management of SPAS Acquisition Is Generally Sound
paragraphs: Overall, FAA has handled key aspects of SPAS development reasonably well. In particular, its analysis and definition of SPAS requirements provided for user involvement and effectively used prototyping techniques. Moreover, recent changes to the FAA standards governing the acquisition of SPAS provide important structure and discipline that, if adhered to, should reduce SPAS development and deployment risks. Also, FAA’s decision to employ an independent verification and validation agent should further mitigate system development and acquisition risks. Last, FAA’s decision to not acquire duplicative data communication networks to support SPAS and other systems should save precious resources. However, opportunities exist to improve the SPAS cost estimates, and thus FAA management’s ability to make sound system investment decisions.
subsections: section_title: FAA’s Approach to Analyzing and Defining Requirements Appears Reasonable and Thorough
paragraphs: One of the most difficult and challenging aspects of any systems development effort is accurately and completely identifying and documenting requirements of system users. To do so successfully requires a commitment on the part of management and system developers to involve users continuously throughout the system development process. That is, the agency must recognize that user requirements cannot be accurately defined at the beginning of the development process. Instead, effective requirements definition demands a more iterative process in which requirements are continuously analyzed, validated, and refined through constant interaction with users.
FAA’s approach to analyzing and defining SPAS requirements has involved a series of steps to maximize user involvement and provide users with early “looks” at the system for evaluation and reaction, thereby better ensuring that the system will meet their needs. These steps first began in May 1991 when the SPAS Steering Committee distributed a questionnaire to 1,000 aviation safety inspectors to solicit their views on what type of automated tool would best serve their needs. The questionnaire contained a variety of questions, such as: How could an automated system help with your work? and What features would you like to see? On the basis of the 375 survey responses received and experience with the Air Force’s airline safety analysis system, FAA and VNTSC generated an initial set of SPAS requirements.
Next, FAA began validating and refining the requirements. First, FAA, in collaboration with VNTSC, held a series of group discussions and one-on-one visits with aircraft safety inspectors throughout the country. According to SPAS documentation, these discussions and visits allowed SPAS developers to see first-hand what the inspectors do on a daily basis and to listen to their ideas, thus giving the developers a keener understanding of the inspectors’ needs and helping them to refine SPAS requirements accordingly.
In October 1991 and December 1993, two expert panels consisting of dozens of experienced aviation safety inspectors and members of the SPAS management team were established to determine whether users’ needs were adequately being captured. These panels were charged with developing and recommending SPAS safety indicators and revalidating SPAS functional requirements.
To further validate SPAS requirements, FAA’s next step was to implement the SPAS Steering Committee’s recommendation to build a prototype system for users. System prototyping is an effective method of defining and refining user requirements. By quickly providing users with a system model (that is, something less than the full complement of envisioned system features and functions) with which to interact and react, prototyping allows needed adjustments to be made before making large investments in developing the final system. In our 1994 report on how leading organizations improved mission performance through strategic information management and technology, we noted that these organizations make effective use of rapid prototyping to minimize system risks and maximize benefits.
The SPAS prototype evaluation focused on the effectiveness and ease of use of the user interface, the adequacy of source systems’ data quality, and the impact of SPAS on inspectors’ daily activities. In late 1993, at the conclusion of the SPAS prototyping phase, FAA was scheduled to discontinue prototype support. However, FAA elected to continue the prototype to help in ongoing requirements refinement, early user familiarization with SPAS, and testing of new SPAS concepts.
section_title: New FAA Acquisition Policy Can Impose Valuable Structure on SPAS Development Process
paragraphs: In March 1993, FAA issued Order 1810.1F, which established its policy for initiating and managing acquisition programs like SPAS. Prior to 1810.1F, the SPAS program office was following VNTSC’s Information Systems Development guidelines for development and acquisition programs. The program office has elected to supplement the VNTSC guidelines with Order 1810.1F.
Our 1994 report on how leading organizations improve mission performance through strategic information management and technology emphasized the importance of using a disciplined process to develop and acquire information systems—one that uses explicit decision criteria, assesses benefits and costs, and involves senior program and information managers in key system decisions.
We reviewed FAA Order 1810.1F and believe it is a reasonably disciplined and organized system acquisition and development process, which if followed, could benefit programs by reducing the potential for cost growth, schedule delays, and performance deficiencies. In particular, 1810.1F imposed valuable rigor and structure on the SPAS acquisition process by applying the principles embodied in OMB Circular A-109 on major systems establishing clear lines of responsibility, authority, and accountability; requiring user and sponsoring office participation throughout the acquisition process, including at key decision points; directing that mission needs be established at the beginning of the acquisition process and then revalidated at critical decision points throughout the remainder of the process; mandating that alternative technological approaches be analyzed prior to selecting a final development strategy; and tailoring the acquisition requirements to the size, complexity, and nature of each specific program.
FAA Order 1810.1F specifies five phases and four related key decision points. Each phase produces the documentation needed to make decisions at the next decision point. SPAS is currently in phase I of the 1810.1F acquisition process, with the second decision point scheduled for September 1996. Documents required for this decision point, such as the program master plan and a cost-benefit analysis, are currently being prepared and thus were unavailable for review.
section_title: FAA’s Plans for Independent Verification and Validation Should Reduce Future SPAS Development Risks
paragraphs: Verification and validation involves analyzing and testing a system throughout its development to ensure that it meets specified requirements. The purpose of verification and validation is to better ensure the final system’s performance, integrity, reliability, and quality. Verification and validation activities are advocated by industry software standards and federal guidelines for software management, especially for systems that involve the safety and preservation of human life. When verification and validation activities are performed by an organization separate and distinct from the system developers, the additional benefit of independence accrues. This is referred to as independent verification and validation (IV&V).
During the course of our review, FAA began using an IV&V agent as a risk mitigating technique. In June 1994, it contracted with Sandia National Laboratories to examine issues relating to (1) system architecture, such as scalability, vulnerability, and robustness, (2) network and server capacity, and (3) system operation. Sandia has also subcontracted with the University of Nevada at Las Vegas for evaluation of the indicators’ appropriateness.
section_title: FAA Is Taking Steps to Avoid Duplicative SPAS Communication Networks
paragraphs: FAA has long recognized that its communications infrastructure could not satisfy the functional (for example, video conferencing) and performance (for example, response time) requirements of SPAS and other applications. To address this shortfall, FAA is acquiring a corporate wide area network (WAN), called the Administrative Data Transfer Network (ADTN) 2000. This network is intended to satisfy not only current FAA requirements for non air traffic control and administrative communications, but also to accommodate growth in communication demands through capacity expansion and technology infusion. On September 19, 1994, FAA’s Telecommunications Division awarded a 5-year contract for ADTN 2000 services. Current plans call for the network to be operational by Spring of 1995.
In addition to the agency WAN, FAA’s Office of Regulation and Certification, which includes the SPAS user community, was planning its own, independent WAN, called the Aviation Information Exchange Network (AIX). According to officials in this office, AIX was pursued because ADTN 2000 was already 18 months late and they believed that further delays would occur.
In June 1994, we raised questions about duplicative WANs and the lack of coordination between these two acquiring organizations. As a result of our inquiries, and to FAA’s credit, the two organizations began meeting. Consequently, the Office of Regulation and Certification later agreed to first evaluate whether ADTN 2000 could meet its communication needs before deciding if it would acquire its own, separate network.
section_title: Reliability of SPAS Cost Estimates Is Uncertain
paragraphs: Our 1994 review of how leading public and private organizations use information technology to improve mission performance showed that these organizations rely heavily on performance measures to, among other things, make informed system life-cycle choices, allocate resources, track progress, and learn from mistakes. One area that these organizations’ standard measurement practices focused on was resource consumption, which requires that reliable estimates of resource needs be developed and used. The estimated cost of a system’s software is one of the more critical of these resource estimates.
To develop reliable software cost estimates, industry practice is to employ one or more structured cost estimating techniques or methods, augmented by the judgement of software experts. While the estimates derived using these methods are not precise, they are more credible than relying solely on the subjective opinions of experts that are unsupported by any objective, verifiable analysis.
FAA’s current cost estimate for developing and installing SPAS is $32 million. According to SPAS program officials, the software component of this estimate was derived 3 years ago based on the subjective judgment of the technical program manager and two contractor officials. No systematic software cost estimating tool or technique, such as COCOMO (Constructive Cost Model), REVIC (Revised Intermediate COCOMO), or SLIM (Software Life Cycle Intermediate Model), was used. Further, the program has no documented analysis to support the software cost estimate, and it has not attempted to update it. Program officials told us that they relied on the judgment of contractor and program experts in estimating SPAS software costs because they have been unable to identify a reliable cost estimating tool or model appropriate for systems like SPAS, which employs a client-server based architecture.
Because of the manner in which the SPAS cost estimate was derived, the reliability of the estimate is uncertain; thus, any decisions made regarding SPAS that rely on this estimate may prove to be ill-advised. For example, to comply with FAA Order 1810.1F, the program office asked FAA’s Office of Operations Research to conduct a SPAS cost-benefit analysis. The program office plans to provide its SPAS cost estimate to serve as the basis for this analysis. Because the cost estimate is not credible, any cost-benefit analysis that relies on it will also not be credible.
FAA’s Office of Operations Research recognizes the limitations in software cost estimates that are not based, at least in part, on formal cost estimating techniques. According to a representative for the contractor performing the SPAS cost-benefit analysis for this office, the SPAS cost estimate provided by the program office will be validated using structured estimating techniques before it is used in the cost-benefit analysis.
section_title: Conclusions
paragraphs: Key aspects of FAA’s management of the SPAS acquisition are reasonably sound. The steps the program office has taken to involve users in defining requirements and evolving the prototypes are appropriate. In addition, the new FAA system acquisition requirements can bring added rigor and discipline to the SPAS development process. Further, the program office’s decision to employ an outside party to verify and validate development activities should prove beneficial. Last, FAA’s steps to avoid unnecessarily duplicative WANs for SPAS and other systems are judicious and may save scarce acquisition and operation and maintenance money. However, we believe that opportunities exist to strengthen the program office’s cost estimating techniques and thus its ability to measure performance and make informed investment decisions.
section_title: Recommendation
paragraphs: We recommend that the FAA Administrator direct the Associate Administrator for Regulation and Certification to ensure that SPAS software costs are estimated using systematic and rigorous estimating techniques and methods.
section_title: Agency Comments and Our Evaluation
paragraphs: In commenting on a draft of this report, FAA officials disagreed with our recommendation, although they agreed that the SPAS cost estimate needs to be updated. The officials stated that FAA’s approach to estimating SPAS software costs (that is, relying solely on the judgments of experts) is consistent with agency guidance. However, they added that this guidance does not specifically address software cost estimating. They also stated that the cost estimating models mentioned in our report are more appropriate for mainframe systems rather than client-server based systems, such as SPAS. Instead, the officials said that they have recently identified a software cost estimating tool that they believe is applicable to SPAS and that they are now evaluating and may acquire and use. We are encouraged by these statements. Our intent was to neither advocate a particular tool nor to ignore the value of expert judgment in deriving software cost estimates. Rather, our aim was to convince FAA to follow accepted industry cost estimating practices of augmenting expert judgment with the kind of objective, verifiable analysis that structured estimating techniques and methods offer.
section_title: Quality of SPAS Source Data Threatens System Effectiveness
paragraphs: To produce its indicators, SPAS will use data from a myriad of existing FAA databases. Because these data have been and continue to be incomplete, inconsistent, and inaccurate, the utility of SPAS is threatened. FAA initiatives underway to improve source data quality are insufficient to ensure that SPAS will receive the data it needs in 1997 to be effective.
subsections: section_title: SPAS Will Rely on Numerous Existing FAA and Non-FAA Systems for Its Source Data
paragraphs: As currently envisioned, SPAS could eventually rely on over 25 databases within FAA, other government agencies, and the aviation industry. The largest single source of data will be FAA’s Aviation Safety Analysis Subsystem (ASAS). ASAS is an umbrella term used to describe a collection of 34 largely independent FAA databases. Generally, the nature of these databases falls into one of five categories—repository of data on various components of the aviation industry, repository of data on FAA personnel, tools for managing inspector/investigator workload, reference sources for FAA regulations, and an oversight tool for FAA senior management.
The current SPAS prototype relies almost exclusively on two ASAS databases—the Program Tracking and Reporting Subsystem (PTRS) and the Vital Information Subsystem (VIS)—in generating its current complement of indicators. PTRS contains data on planned inspections of airlines and aircraft, as well as the results of these inspections. The data are entered by inspectors or support personnel and are used to inform FAA management of inspection activities. VIS contains key data on airlines, pilot and mechanic schools, repair stations, and FAA designees (that is, people and organizations that FAA empowers to act as surrogates for it in discharging specific FAA responsibilities). These data are entered by inspectors or support personnel and are used to track aviation activity.
As the number of indicators expands, FAA plans to use data from other ASAS databases. (See appendix I for description of each of the potential SPAS source data systems.)
section_title: Quality of Data Residing on SPAS Source Data Systems Is a Long-Standing Problem
paragraphs: The quality of SPAS’ outputs, and thus its utility in supporting FAA decisionmakers, depends on the quality of its inputs. FAA fully recognizes this. In fact, the Office of Flight Standards Five Year Information Strategy states that information and its quality are at the heart of SPAS’ success. Similarly, an Office of Flight Standards expert panel on data quality stated that SPAS needs a sound foundation from which to analyze, and this foundation must be in the form of reliable databases that are correct, complete, and consistent. Also, a Flight Standards Working Group stated that for advanced tools like SPAS, the data on which they operate must be correct, consistent, complete, and up-to-date, or the results will be meaningless—or even misleading.
Despite the criticality of reliable source data to SPAS’ success, the poor quality of the data on the FAA databases that SPAS will use remains a serious problem today. In our 1988 report on the feasibility of assessing safety levels of individual airlines, we concluded that none of the potential source databases could provide a satisfactory basis for developing safety indicators because the data were unreliable, incomplete, and inconsistent. At that time, one major airline described FAA’s data on aircraft accidents, incidents, and serious malfunctions as, for the most part, worthless.
In 1989 and 1991, we reported on inaccurate and incomplete inspections data in PTRS. In its response to the 1991 report, FAA agreed that PTRS was inaccurate and incomplete. In fact, an FAA-sponsored study that year concluded that PTRS cannot be used for problem diagnosis and trend analysis with any degree of reliability until data quality issues were resolved. Similarly, a 1992 Flight Standards expert panel, established to identify ways to improve the quality of PTRS data, reported that PTRS did not contain reliable, consistent data. The panel made recommendations for improvement.
While FAA has recognized its data quality problems for years and has taken some steps to address them, the problems still persist. According to 1993 SPAS documentation, many FAA databases continue to have data quality and consistency problems, critical data elements are still missing or contain erroneous data, and supporting documentation is either out-of-date or missing. A 1993 Flight Standards working group on data quality improvement also reported that the data quality problem of FAA safety-related databases remains as much an issue today as it was more than 5 years ago. Also, a 1994 Department of Transportation Inspector General report states that the database containing data on inflight “service difficulties” is neither complete nor current. For example, the report states that omissions in different data fields for each “service difficulty” occurrence in the database as of January 1993 ranged from 46 to 98 percent. In November 1994, SPAS program officials affirmed these reports by stating that the quality of data residing on the SPAS feeder databases remains a major risk item for the system.
section_title: FAA Lacks a Strategy for Ensuring the Quality of SPAS Source Data
paragraphs: Despite FAA’s recognition of both SPAS’ need for quality source data and its lack of such data, FAA has not developed a coordinated strategy for rectifying the situation. We reviewed the 1992 Five Year Flight Standards Information Strategy, and found one broad goal in this area—to “ensure quality data for decisionmaking.” We further found that “development of measurement tools to assess and improve data quality of SPAS feeder systems” and “begin data needs analysis of existing processes” were the extent of planned actions to accomplish the goal. We did not find a comprehensive strategy that (1) clearly defines measurable, interim, and long-term goals for improving data quality, (2) specifies the full extent of the problem being addressed, (3) is supported by a series of specific steps designed to meet the stated goals according to a specified schedule, and (4) designates the organizations responsible for executing the strategy and provides the associated authority and resources for doing so.
Officials with the SPAS program office, the Flight Standards Training and Automation Committee, and the Office of Flight Standards’ Information Resources Management function acknowledged that no strategy exists. Further, officials with the former two organizations stated that FAA has not yet determined what level of data quality is needed from each of the source databases. In other words, FAA has not agreed on a definition of its long-term data quality goals. Instead, these officials pointed to a few independent data improvement measures, some of which are being performed by the SPAS program office even though it is not responsible for these source databases. These measures include implementation of select recommendations made by an Office of Flight Standards working group and VNTSC on PTRS and VIS data quality improvement, development of an automated tool for measuring the quality of data residing on PTRS and VIS (the tool may eventually be applied to all SPAS source databases), and revision of the PTRS and VIS users manuals.
While we do not question the merits of these initiatives, they neither individually nor collectively represent the type of coordinated and comprehensive effort that can ensure that SPAS will receive the data it needs when the system is deployed in 1997. In commenting on a draft of this report, the Director of the Office of Flight Standards agreed that such a strategy is needed.
section_title: Conclusions
paragraphs: The axiom “garbage in, garbage out” applies to SPAS. This system will not be effective if the quality of its source data is not improved. Moreover, it could potentially misdirect FAA resources away from the higher risk aviation activities. While FAA has some initiatives underway to improve some of these data, the initiatives are isolated, incomplete, and provide little assurance that SPAS will receive the quality data it needs to be useful. Unless FAA acts quickly on this matter, SPAS will not be able to perform as intended when it is deployed in 1997. FAA must expeditiously develop a comprehensive and coordinated strategy for defining and attaining defined data quality improvement goals within specified time frames for all SPAS source databases.
section_title: Recommendation
paragraphs: We recommend that the FAA Administrator direct the Associate Administrator for Regulation and Certification to require the Office of Flight Standards to develop and implement a comprehensive and coordinated strategy, specifying how the quality of all data residing on SPAS source data systems will be brought up to the minimum level needed for SPAS to meet operational requirements. At a minimum, this strategy must include (1) clear and measurable data quality objectives for each SPAS source data system that recognize the sensitivity of SPAS’ various analyses to the respective source data inputs, (2) accurate assessments of the current quality of the data on each SPAS source data system, (3) clear statements of organizational responsibility and authority for improving the source systems’ data quality, (4) both interim and long-term milestones for attaining stated quality objectives that tie closely to SPAS development schedules, and (5) estimates of resource requirements to meet stated objectives and agency commitments to providing these resources.
section_title: Agency Comments and Our Evaluation
paragraphs: In commenting on a draft of this report, FAA officials agreed with our recommendation to develop and implement a comprehensive and coordinated strategy for improving the quality of the data residing on SPAS source databases. However, they do not believe that the data quality problem is as severe as our report describes it to be. They stated that the quality of the data has measurably improved over the last several years. To support their position, they cited various steps taken to strengthen the completeness, correctness, currency, and consistency of the databases. For example, they said that data entry edit checks have been introduced and that inspectors are now more conscious of the consistency of the data they enter. However, they could not provide any data, analysis, or otherwise verifiable evidence supporting their claims. While we acknowledge that the steps cited should produce some quality gains, without evidence of actual improvements we believe that the quality of the data on which SPAS will rely remains a significant problem. As stated in our report, FAA and Department of Transportation analyses as recent as 1993 and 1994 continue to report severe aviation safety related data limitations.