Air Traffic Control: Good Progress on Interim Replacement for Outage-Plagued System, but Risks Can Be Further Reduced
Published: 1996-10-17T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: GAO reviewed the Federal Aviation Administration's (FAA) Display Channel Complex Rehost (DCCR) project, intended as an interim replacementof the Display Channel Complex (DCC), focusing on: (1) recent outages caused by DCC; (2) whether DCC was meeting its system availability requirement; (3) FAA's projections of future DCC outages and availability; and (4) whether FAA was effectively managing the DCCR acquisition to ensure delivery of specified capabilities on schedule and within estimated cost.
section_title: What GAO Found
paragraphs: GAO found that: (1) DCC, built and deployed over 30 years ago, is critical to FAA's ability to display aircraft situational data in five of FAA's 20 air route traffic control centers; (2) DCC is also responsible for most of the major outages at the five centers from September 1994 through May 1996, accounting for about 48 percent of the total number of major outages and nearly 87 percent of unscheduled system downtime associated with these outages; (3) according to FAA, DCC was able to exceed its availability requirement from fiscal year (FY) 1990 to 1993, on average at the five centers, because of heroic maintenance efforts using "chewing gum and chicken wire"; (4) however, it fell slightly short of the requirement in FY 1994 and 1995, and FAA expects availability to decrease further because of shortages of spare parts and experienced DCC technicians; (5) decreases in DCC availability will result in costly delays for airlines and passengers; (6) FAA has made good progress in acquiring DCCR, but much remains to be accomplished; (7) thus far, the fourth and final DCCR software build is complete, and the number of reported software defects, while cumulatively slightly higher than projections, is showing a favorable trend when adjusted for defect severity; (8) also, FAA is ahead of schedule in completing informal system-level tests, formal testing is generally on schedule, and the first site is ready to begin the system acceptance process; (9) DCCR's development has benefitted from formal risk management and quality assurance programs, and FAA has plans in place to accelerate completion of formal system-level tests; (10) contractor financial reports show that DCCR is under spending estimates; (11) in light of its progress to date, FAA has an opportunity to deliver promised DCCR capabilities on time and within contract budgets; (12) the likelihood of doing so can be increased, however, by acting to mitigate two known risks associated with remaining development activities; (13) specifically, FAA's test plans call for conducting three system-level tests concurrently rather than sequentially, as is normally done; (14) by doing so, FAA expects to implement DCCR early; (15) however, FAA is not formally managing two risks associated with DCCR concurrent testing, which are: (a) staffing three test activities at the same time and thus potentially spreading test personnel too thin; and (b) not defining how it will control and synchronize changes to three system test configurations so as to prevent configuration differences among the three during testing; and (16) by formally managing these risks, FAA will greatly reduce the chances of them impeding future DCCR progress.

section_title: Letter
paragraphs: During the last year, certain air traffic control (ATC) centers have experienced a series of major outages, some of which were caused by the Display Channel Complex or DCC—a mainframe computer system that processes radar and other data into displayable images on controllers’ screens. For example, four major DCC outages occurred at the Chicago center from May through September 1995, including one that lasted roughly 5 days and another one that produced 234 flight delays. Because the permanent replacement for DCC and other aging center ATC systems has been delayed until the end of the century, the Federal Aviation Administration (FAA) recently decided to acquire an interim replacement, calling it the DCC Rehost (DCCR), and deploy it to all affected centers by early 1998.
In light of the importance of DCC—and its to be short-lived replacement (DCCR)—to FAA’s ATC mission, as well as FAA’s limited success in delivering promised ATC system capabilities on time and within budget, we reviewed the DCCR project. Our objectives were to determine (1) the portion of the recent major outages experienced at the five DCC-equipped en route centers that were attributable to DCC, (2) whether DCC was meeting its system availability requirement, (3) FAA’s projections of future DCC outages and availability, and (4) whether FAA was effectively managing the DCCR acquisition to ensure delivery of specified capabilities on schedule and within estimated cost. Appendix I provides more detailed information on our objectives, scope, and methodology.
subsections: section_title: Background
paragraphs: FAA’s air traffic management mission is to promote the safe, orderly, and expeditious flow of air traffic in the national airspace. To accomplish this mission, FAA employs a vast network of ATC and traffic flow management computer hardware, software, and communications equipment to (1) prevent collisions between aircraft and obstructions and (2) facilitate the efficient movement of aircraft through the air traffic system.
subsections: section_title: ATC Facilities and Functions
paragraphs: Automated information processing and display, communication, navigation, surveillance, and weather resources permit air traffic controllers to view key information, such as aircraft location, aircraft flight plans, and prevailing weather conditions, and to communicate with pilots. These resources reside at, or are associated with, several ATC facilities—flight service stations, air traffic control towers, terminal radar approach control (TRACON) facilities, and air route traffic control centers (en route centers). These facilities’ ATC functions are described below.
About 90 flight service stations provide pre-flight and in-flight services, primarily for general aviation aircraft, such as flight plan filing and weather report updates.
Airport towers control aircraft on the ground and before landing and after take-off when they are within about 4 nautical miles of the airport. Air traffic controllers rely on a combination of technology and visual surveillance to direct aircraft departures and approaches, maintain safe distances between aircraft, and communicate weather-related information, clearances, and other instructions to pilots and other personnel.
Approximately 180 TRACONs sequence and separate aircraft as they approach and leave busy airports, beginning about 4 nautical miles and ending about 50 nautical miles from the airport, where en route centers’ control begins.
Twenty en route centers control planes over the continental United States in transit and during approaches to some airports. Each en route center handles a different region of airspace, passing control from one to another as respective borders are reached until the aircraft reaches TRACON airspace. En route center controlled airspace usually extends above 18,000 feet for commercial aircraft.
Two en route centers—Oakland and New York—also control aircraft over the ocean. Controlling aircraft over oceans is radically different from controlling aircraft over land because radar surveillance only extends 175 to 225 miles offshore. Beyond the radars’ sight, controllers must rely on periodic radio communications through a third party—Aeronautical Radio Incorporated (ARINC), a private organization funded by the airlines and FAA to operate radio stations—to determine aircraft locations.
See figure 1 for a visual summary of the processes for controlling aircraft over the continental United States and oceans.
section_title: En Route Centers Rely on Numerous Automated Systems, Including DCC and Eventually DCCR
paragraphs: Although en route centers’ specific hardware and software configurations may differ slightly, the centers rely on over 50 systems to perform mission critical information processing and display, navigation, surveillance, communications, and weather functions. Examples include the systems that display aircraft situation data for air traffic controllers, the system that collects data from various weather sources and distributes them to weather terminals, radars for aircraft surveillance, radars for wind and precipitation detection, ground-to-ground and ground-to-air communication systems, and systems to back-up primary systems. (See appendix II for a simplified block diagram of an en route center’s systems environment.)
DCC is one of the 50-plus en route center systems. DCC runs on 1960s vintage IBM 9020E mainframe computers, and its software is written in two languages, assembly and JOVIAL. It is used at 5 of the 20 en route centers. (See figure 2 for the locations of the 20 en route centers and identification of the five that are DCC-equipped.) DCC’s purpose is to accept data from the Host Computer System (HCS) and process it to form the alphanumeric, symbolic, and map data that appear for air traffic controllers on their Plan View Displays (PVD). (See figure 3 for a simplified block diagram of DCC and the en route center systems with which it interfaces.)
section_title: When and Why Did FAA Decide to Produce and Deploy DCCR?
paragraphs: In response to expected increases in the frequency and severity of DCC problems and the possibility of delays in the system intended to permanently replace DCC as well as other en route display-related systems, FAA awarded a roughly $30 million contract in September 1994 for development of “a single, deployment-ready” interim replacement (i.e., DCCR) unit. FAA officials characterized this development effort as an “insurance policy” to protect FAA against delays in the permanent replacement, then called the Initial Sector Suite System (ISSS) and now called the Display System Replacement (DSR). (See appendix III for more information on DSR.)
In July 1995, following a flurry of DCC problems and outages and known delays with ISSS, FAA decided that there was an urgent and compelling need to replace DCC at all five DCC-equipped en route centers in the interim before DSR is ready. In making such capital investment decisions, FAA uses four criteria: sponsor (i.e., user) support; mission importance; information technology architectural conformance and maturity; and cost-effectiveness. Each criterion carries a standard weighting factor that is to be consistently applied to all proposed projects. (See figure 4 for these weighting factors.)
According to DCCR documentation and FAA officials, sponsor support and mission need (i.e., aviation safety) drove the July 1995 decision to produce and deploy DCCR. In particular, FAA’s Air Traffic Services organization, the Air Traffic Controllers Association, and the Air Transport Association strongly endorsed DCCR. Also, FAA officials told us that extensive media attention to the DCC outages, considerable congressional interest, and public safety concern were major considerations. For example, one official stated that FAA was “taking too much heat in the papers for DCC outages and wanted DCCR to solve the problem.” In FAA’s view, the need to quickly replace DCC was urgent and compelling, and DCCR was the only practical alternative to sustaining safe, orderly, and efficient air traffic services in the near-term.
FAA considered two cost estimates in analyzing DCCR’s costs versus benefits. However, the results of this analysis were inconclusive, and according to FAA officials, were not relevant to the decision to produce and deploy DCCR because of the urgent need to replace DCC. One of the cost estimates was done by the DCCR project office and the other by the program analysis and operations research office. Using the two cost estimates, the FAA analyzed three DCCR life expectancy scenarios. Under the “most likely” scenario, the project office’s cost estimate produced a DCCR net present value of negative $37 million and a benefit-to-cost ratio of 0.7 to 1. In contrast, the program analysis and operations research office’s lower cost estimate under the same scenario placed these values at $29 million and 1.4 to 1, respectively. Neither estimate considered maintenance costs. Given the expense of DCC maintenance, including it would likely have made DCCR more cost-effective under both estimates. While FAA officials agreed with our assessment of the impact of including maintenance costs, they did not quantify this impact.
The month following its July 1995 decision, FAA awarded a roughly $34 million contract to produce five DCCR systems and has publicly committed to having the first site operational in October 1997 and the fifth and last site in February 1998. (See figure 5 for the respective sites’ publicly announced delivery and operational readiness demonstration dates.)
section_title: DCCR: A Brief Description
paragraphs: DCCR’s installation and operation will not change the air traffic controllers’ current system interface and thus will be transparent to them. It consists of two components—the Display Channel Rehost Processor (DCRP) and the Display Controller and Switch (DC&S). DCRP will use a commercial, off-the-shelf IBM processor to execute about 120,000 lines of rehosted DCC code and 60,000 lines of new code. The primary contractor is developing the new code, and a subcontractor is rehosting the DCC code. DC&S uses custom-developed hardware and about 65,000 lines of new code implemented in firmware to perform keyboard, trackball, and display control functions. (See figure 6 for a simplified block diagram of DCCR and the systems with which it interfaces.)
section_title: Most Major Outages at DCC-Equipped En Route Centers Are Attributable to DCC
paragraphs: According to FAA, a major system outage is one which significantly delays air travel or produces significant media interest. Most of the recent major system outages at the five DCC-equipped centers have been DCC-related. Our analysis of FAA major outage data from September 1994 through May 1996 at the Chicago, Dallas-Ft. Worth, New York, Washington, and Cleveland en route centers showed that DCC accounted for 10 of the 21 outages, or about 48 percent. Moreover, these DCC outages were responsible for 195 of 225 hours, or about 87 percent, of unscheduled system downtime at these centers during this time. (See figures 7 and 8.)
section_title: Despite Outage Frequency, DCC Has Historically Met Its System Availability Requirement
paragraphs: System availability is defined as the time that a system is operating satisfactorily, expressed as a percentage of the time the system is required to be operational. FAA has specified a DCC system availability requirement of 99.9 percent. DCC exceeded that requirement from fiscal year 1990 through 1993, but failed to meet it in fiscal years 1994 and 1995, with availability of 99.83 and 99.81 percent, or 0.07 and 0.09 percent below the requirement, respectively. (See figure 9.) According to FAA officials, DCC’s acceptable history of availability has been attained through the extraordinarily hard work, commitment, and ingenuity of its highly skilled, but small, workforce of technicians. For example, to obtain replacement circuit boards for the 9020E, which is out of production, FAA officials told us that technicians scavenged parts from a computer used by the FAA Air Traffic Training Academy and cannibalized parts from two scrapped computers at the FAA Supply Depot.
section_title: FAA Expects Future DCC Availability to Decrease
paragraphs: Two factors determine a system’s availability—the frequency of unscheduled outages and the time to recover from each outage (i.e., mean time to restore or MTTR). According to FAA data, the number of DCC outages annually has increased by about 55 percent since calendar year 1990, from 22 to 34, and FAA predicts that this number will hold relatively steady through calendar year 2000. (See figure 10.) In contrast, the DCC MTTR grew by over 434 percent in calendar years 1994 and 1995 over previous years, and FAA predicts that DCC MTTR will grow at an average annual rate of 13 percent through the year 2000. (See figure 11.) FAA attributes increasing MTTR to depleted inventories of out-of production DCC spare parts and a shortage of experienced DCC repair technicians. Decreases in DCC availability will result in costly delays for airlines and passengers.
section_title: FAA Has Made Good Progress on DCCR but Can Reduce Risk Further
paragraphs: Thus far, FAA has made good progress on its DCCR acquisition, but much remains to be accomplished. To FAA’s credit, the fourth and final software build has completed integration testing, and some formal system-level test and demonstration activities have occurred. However, the number of software defects being found is slightly higher than projections, and despite the fact that FAA’s defect fix rate has kept pace with the higher numbers and its DCCR defect trend lines are favorable when considering defect severity, unresolved defects delayed the start of concurrent system-level testing at the Technical Center and the first site by several weeks. Notwithstanding this delay, DCCR’s operational readiness date may nevertheless be accelerated by several more months if FAA is successful in conducting system acceptance and operational tests concurrently.
Also to FAA’s credit, it has prudently made formal risk management and quality assurance integral components of the acquisition. However, two risks associated with concurrent test plans are not being formally addressed—managing contention for limited test staff among three concurrent test activities, and controlling and synchronizing changes to three DCCR system test configurations.
subsections: section_title: Software Development and Testing Activities Are Collectively Proceeding Well
paragraphs: DCCR involves both converting and migrating existing code written for DCC’s IBM 9020E platform, and writing new code. In sum, DCCR consists of about 245,000 lines of code—120,000 lines of rehosted DCC code (of which about 20,000 are modified and 100,000 are unchanged) and 125,000 lines of new code. Of this newly developed software, about 60,000 lines of code relate to the DCRP component of DCCR and 65,000 lines relate to the DC&S component.
To FAA’s credit, it has thus far completed the fourth and final DCRP software build as well as formal software integration and software testing activities. Also, the DC&S subcontractor has completed formal installation and integration testing of the DC&S firmware, and DC&S has been accepted by the DCCR prime contractor. In addition, a demonstration of DCCR was held on May 1, 1996, for the FAA Deputy Administrator, and formal system-level testing of the initial version of DCCR was completed on September 24, 1996, 2 months ahead of schedule.
One measure of software quality is the severity and density (number per one thousand lines of code) of software errors or defects. Defects are managed by (1) documenting them via program trouble reports (PTR), when they are discovered, and submitting them to a change control board, (2) determining whether they are valid, (3) assigning valid PTRs a priority on the basis of severity, and (4) resolving the valid PTRs and closing them. DCCR’s severity categories are emergency, test critical, high, medium, and low. According to the DCCR prime contractor, an emergency PTR causes test progress to stop and requires an immediate resolution in the form of a fix or an adequate workaround; a test critical PTR severely impedes test progress, and resolution is required prior to the next scheduled accumulation and reporting of valid PTRs; a high PTR must be resolved before an integration and test activity is completed; a medium PTR is a significant system or application problem, but it does not require resolution for integration and test completion; and a low PTR is a minor or insignificant system or application problem that does not require resolution for integration and test completion.
One way to gauge progress in the software maturation process is to compare the number of defects being found to projections in the number of defects expected. These projections are normally made on the basis of models that consider defect experience on like or similar software development efforts. In the case of DCCR, the actual number of cumulative PTRs discovered is slightly higher than projected. (See figure 12.) Specifically, as of July 1996, actual cumulative defects were about 17 percent over expectations. Considering the possibility of variability in model results as well as FAA’s track record during this same period in “working-off” defects at a pace consistent with defect discovery, we see no cause for alarm at this time.
Another measure of software maturation is the trend in the number of open (i.e., unresolved) PTRs over time adjusted for the PTRs’ severity mix. Using a simple weighting scale of one through five, which corresponds to the DCCR PTR severity categories, we analyzed the change in open PTRs from March 1996 through August 1996 and found a downward trend. (See figure 13.) According to software engineering guidance, a downward slope over time is ideal.
section_title: DCCR Is Ahead of Schedule and Contractor Reports Show the Contract Is Under Budget
paragraphs: According to the publicly announced DCCR schedule, the first site is to be operationally ready on October 1997. However, on the basis of our analysis of DCCR plans, contractual terms, completed activities, and discussions with project officials, DCCR could be operationally ready as early as December 1996, 10 months ahead of schedule. Currently, DCCR’s development is about 4 months ahead of the published schedule, having completed DCRP software build four integration and testing as well as formal testing of the initial version of DCCR earlier than planned. An additional 6 months may also be saved if FAA Technical Center acceptance testing, FAA Technical Center operational test and evaluation, and first site acceptance testing can be successfully accomplished concurrently, as FAA plans for DCCR, rather than sequentially, as is normally the case. Concurrent testing will be successful, however, only if the software has no significant problems.
With respect to DCCR’s financial status, FAA estimates DCCR’s project cost to be about $64 million, $48 million of which are contract costs and $16 million are other project-related activities, such as support contractors, field support, and training. Of the $48 million for contract costs, spending plans show that as of July 1996, $31.8 million was to be spent; and of the $16 million for other activities, obligation plans show that $12.4 million was to be obligated by the end of fiscal year 1996.
On the basis of the latest monthly contractor reports, cumulative contract costs through July 19, 1996, are $29.1 million, which is about $2.7 million below spending plans. However, these cost reports have not been independently verified by FAA or its support contractors, which is FAA’s normal practice on large contracts. Project officials stated that other, more costly contracts, such as DSR development and deployment, are consuming cost verification resources.
On the basis of FAA internal financial management system reports,cumulative obligations for other project-related activities through July 31, 1996, are $11.5 million, which is about $600,000 under the obligation plan with only 2 months left in the plan period. However, these obligation figures are not complete because neither the planned nor actual obligations include all project-related activities, such as FAA personnel compensation, benefits, and travel.
section_title: DCCR Has a Formal Risk Management Program
paragraphs: Acquisition of software-intensive systems, like DCCR, is inherently risky. Best practices used in government and private sector acquisition and development activities include the use of formal risk management to proactively and continually identify, assess, track, control, and report risks. Carnegie Mellon University’s Software Engineering Institute recommends a joint contractor/government risk management approach in its guide Team Risk Management: A New Model for Customer Supplier Relationships.
For DCCR, FAA and the prime contractor have a formal, collaborative risk management process that includes a risk management plan and an operational process that is consistent with this plan. They maintain a single “risk watch list” that is updated periodically on the basis of the joint FAA/contractor risk management team’s biweekly evaluation of risk information sheets submitted by FAA or contractor staff. The team assigns a severity category to each risk (high, medium, and low), develops a mitigation strategy for each risk, and tracks and reports on the strategies’ implementation. Currently, the DCCR risk watch list contains four low risks: (1) contention for FAA Technical Center laboratory resources (facilities and systems) during concurrent test activities, (2) costly updates to the DC&S firmware to correct latent errors after it is delivered to the Technical Center and the five en route centers, (3) yet-to-be-tested ability of a fully integrated DCCR to meet system-level performance parameters, and (4) increased system maintenance time, and thus system downtime, due to the lack of a remote monitoring and maintenance connection to each site’s DCRP component. The watch list also contains one medium risk, which is lack of DCCR training course materials and actual training before DCCR’s operational readiness demonstration date.
section_title: DCCR Quality Assurance Activities Are Being Performed
paragraphs: A quality assurance program exists to ensure that (1) products and processes fully satisfy established standards and procedures and (2) any deficiencies in the product, process, or their associated standards are swiftly brought to management’s attention. The quality assurance plan is the centerpiece of an effective quality assurance program. The plan defines the activities necessary to ensure that software development processes and products conform to applicable requirements and standards. To encourage and protect its objectivity and candor, the quality assurance group should be organizationally independent of project management (i.e., have an independent reporting line to senior managers).
Both FAA and the DCCR prime contractor have implemented quality assurance programs. The FAA Quality Reliability Officer, who is independent from the DCCR project office, has been actively monitoring contractor performance. Quality assurance activities performed thus far include preparing a quality assurance plan, auditing the hardware manufacturing process, monitoring project office software peer reviews, monitoring software inspections and walkthroughs, and monitoring the contractor’s configuration management activities.
section_title: FAA Plans for Conducting Concurrent System-Level Tests Add Some Risk
paragraphs: Throughout a system’s development cycle, various types of test activities occur that incrementally build on earlier tests and progressively reveal more and more about the system’s ability to meet specified functional, performance, and interface requirements. Early test activities focus on smaller system components, such as software strings and modules, and later tests address integrated software modules, eventually building toward different types of system-level test and evaluation activities. As such, each increment of tests is designed to sequentially test for and disclose different information about the system’s ability to perform as intended.
Under FAA’s normal progression of system-level testing, Technical Center acceptance testing would occur first, followed first by Technical Center operational test and evaluation, and then by first site acceptance testing.According to FAA test officials familiar with DCCR, some overlap between the conclusion of one of these tests and the beginning of another of these tests in sequence is normal. However, the degree of overlap occurring on DCCR, which is complete concurrency of all three tests, is unusual.
FAA plans to concurrently conduct Technical Center acceptance tests, Technical Center operational test and evaluation, and first site acceptance test as a way of saving time and thus implementing DCCR sooner. This approach assumes that no significant problems will arise during the test activities. According to project officials, this should be the case for DCCR because, in their opinion, (1) the system is virtually free of material defects and thus is mature, (2) FAA has experience with the DCCR commercial hardware, which is similar to that being used on another operational en route system (Peripheral Adapter Module Replacement Item), and (3) DCCR provides the same functionality as DCC.
Test concurrency, particularly the 100 percent overlap planned by FAA, carries with it additional risks that must be managed closely and carefully. For example, concurrency will increase contention for test resources, in particular Technical Center system and human resources. Also, concurrency introduces the possibility of problems being found and corrected independently during the different test activities, resulting in more than one baseline test configuration. Should this occur, the results of testing activities could be meaningless. Both DCCR project and contractor officials acknowledged both risk items. However, FAA is only formally managing contention for Technical Center system resources during testing as part of its risk management program. According to FAA officials, both (1) contention for Technical Center human resources during testing and (2) test baseline change control are being managed informally and outside the framework of the formal risk management program. By not formally managing these risks, FAA is increasing the chances that they will be overlooked and adversely affect DCCR. For example, by not formally managing the latter, FAA has not ensured that the contractor’s DCCR configuration management plan expressly defines the process for controlling changes across multiple baselines during testing, an inherently more difficult configuration management scenario than is normally encountered during single baseline system testing. Although contractor representatives described for us the process they plan to use for controlling changes over multiple baselines, the configuration management plan does not reflect this. By not having a documented configuration management process that addresses the change control complications introduced by concurrent testing, FAA is unnecessarily increasing the risk of introducing more than one test baseline configuration and thereby rendering concurrent test results meaningless.
Finally, concurrent testing will save time only if no significant system problems are found. Correcting significant problems requires stopping all tests, correcting the baseline, and then restarting testing. If all tests are not stopped and restarted using the same, corrected baseline, inconsistent configurations would be tested, producing potentially meaningless results and wasted effort.
section_title: Conclusions
paragraphs: DCC outages caused by old, out-of-production equipment have disrupted air traffic, producing costly airline delays as air traffic control centers must reduce traffic volumes to compensate for lost system capability. The outages are likely to become increasingly disruptive as the availability of DCC spare parts and repair technicians shrink.
FAA has thus far made good progress in its efforts to replace DCC with DCCR. Although key acquisition milestones, events, and risks remain, FAA is currently on track to deliver promised capabilities ahead of schedule and within budget. How successful FAA will ultimately be, however, depends on how effectively it performs key remaining tasks, such as system-level testing, and how effectively it manages known acquisition risks. While FAA has formal strategies and efforts underway to address some of these risks, two risks associated with upcoming concurrent system-level testing—contention for human test resources and test baseline configuration change control—are not being formally managed. As a result, FAA has no assurance that either risk will be carefully and effectively mitigated.
section_title: Recommendations
paragraphs: To maximize the likelihood of delivering promised DCCR capabilities on time and within contract budgets, we recommend that you direct the FAA Administrator to ensure that (1) contention for human test resources during DCCR concurrent test activities and (2) change control over system test configuration baselines during concurrent test activities are managed as formal program risks. At a minimum, this formal risk management should include definition, implementation, and tracking of risk mitigation strategies.
section_title: Agency Comments and Our Evaluation
paragraphs: On September 17, 1996, we discussed a draft of this report with Department of Transportation and FAA officials, including FAA’s DCCR Deputy Project Manager and FAA’s Program Director for Airway Facilities Requirements. These officials agreed with the report’s conclusions and recommendations, and commented that both risk areas have been added to the DCCR risk watch list. Our review of the latest risk watch list confirmed that the risks are now being formally managed.
This report contains recommendations to you. The head of a federal agency is required by 31 U.S.C. 720 to submit a written statement on actions taken on these recommendations. You should send your statement to the Senate Committee on Governmental Affairs and the House Committee on Government Reform and Oversight within 60 days after the date of this report. You must also send the written statement to the House and Senate Committees on Appropriations with the agency’s first request for appropriations made over 60 days after the date of this report.
We are sending copies of this letter to relevant congressional committees and subcommittees, the Director of the Office of Management and Budget, the Administrator of the Federal Aviation Administration, and other interested parties. We will send copies to others upon request. If you have questions or wish to discuss the issues in this report, please contact me at (202) 512-6412. Major contributors to this report are listed in appendix IV.
section_title: Objectives, Scope, and Methodology
paragraphs: Because DCCR is a critical, yet short-lived, system and because of FAA’s poor track record in acquiring ATC systems, we reviewed the DCCR acquisition. Our objectives were to determine (1) the portion of the recent major outages experienced at the five DCC-equipped en route centers that were attributable to DCC, (2) whether DCC was meeting its system availability requirement, (3) FAA’s projections of future DCC outages and availability, and (4) whether FAA was effectively managing the DCCR acquisition to ensure delivery of specified capabilities on schedule and within estimated cost.
To determine what portion of recent major outages at the five DCC-equipped en route centers were attributable to DCC, we used information from a May 21, 1996, FAA report entitled Summary of Major Outages at Centers to calculate by cause the number of major outages and the amount of down time associated with these outages. We also interviewed the FAA Airway Facilities Service official who collected the data used in the report to clarify their meaning and define the term “major outage.” We did not verify the information contained in this report concerning the number and cause of the outages or the amount of downtime resulting from the outages.
To determine whether DCC was meeting its system availability requirement, we collaborated with FAA to calculate DCC’s required availability using data from the system specification. We then compared required availability to DCC’s actual availability for fiscal years 1990 through 1995, which we obtained from FAA’s National Airspace Performance Analysis System. We did not verify the reliability of DCC’s actual availability data generated by the performance analysis system.
To assess future DCC outages and availability, we obtained FAA projections of the number of DCC outages and the associated MTTR for these outages for calendar years 1996 through 2000, reviewed FAA’s Supportability Review of Display Channel Complex (DCC) and Computer Display Channel (CDC) (Initial Report), dated May 1995, and Supportability Review Update of Display Channel Complex (DCC) Hardware, dated March 1996, and interviewed Air Traffic Services officials responsible for these reports. We also interviewed National Transportation Safety Board officials about the findings in their Special Investigation Report, Air Traffic Control Equipment Outages, dated January 1996.
To determine whether FAA is effectively managing the DCCR acquisition, we analyzed project and contractor documentation concerning (1) key acquisition and development process areas, such as test and evaluation, risk management, configuration management, and quality assurance, and (2) indicators of product quality, such as trends in reported defects. We also interviewed DCCR project officials and contractor representatives, and analyzed project office and contractor reports addressing progress against cost and schedule plans and budgets. We did not evaluate the reliability of the systems that produced these reports. On the basis of our analysis, we assessed the DCCR risk watch list to ensure that all significant risks were being formally managed.
In support of all four objectives, we visited one of the five en route centers that is DCC-equipped to observe the system in operation and discuss with controller and maintenance technician representatives DCC functions, mission importance, and performance.
We requested comments on a draft of this product from the Secretary of Transportation. On September 17, 1996, we obtained oral comments from Transportation and FAA officials. These comments have been incorporated in the report as appropriate.
We performed our work at FAA Headquarters in Washington, D.C., the FAA Technical Center in Atlantic City, New Jersey, and the Washington en route center in Leesburg, Virginia. Our work was performed from March 1996 through September 1996, in accordance with generally accepted government auditing standards.
section_title: Simplified Block Diagram of an En Route Center’s Systems Environment
paragraphs: (NARACS)
subsections: section_title: Explanatory Notes to Simplified Block Diagram of an En Route Center’s Systems Environment
paragraphs: Collects surface observations data from AWOS and ASOS and distributes these data to weather processing and display systems.
Provides capability for real-time and nonreal-time monitoring of en route center systems, remote control of equipment and facilities, communications/coordination, and system security.
Provides backup air-to-ground radio voice communications service in the event of a failure of the primary or secondary air-to-ground radio system.
Provides FDIO print capability.
Provides display capability that will be replaced by DSR.
Provides display capability that will be replaced by DSR.
Provides display capability that will be replaced by DCCR, which will in turn be replaced by DSR.
Provides display capability that will replace DCC.
Provides character and image display capability that will be replaced by DSR.
Provides an interfacility multiplexed data transmission network.
Controls local and remote air-to-ground radios.
Make legal recordings of all voice communications between air traffic controllers and pilots.
Provides a backup to HCS for radar processing, and radar track and display processing.
Provides flight data input/output capability by transferring flight data inter-/intrafacility.
Provides the processing capability to support AFSS workstations and automated pilot briefings, and maintains a national flight service database.
Processes radar surveillance data, associates flight plans with tracks, processes flight plans, performs conflict alerts, and processes weather data.
Provides capability for data entry and display and provides a standard serial data interface to connect to a RMS.
Provides capability for real-time monitoring and alarm notification, certification parameter data logging, automatic record keeping and information retrieval, and trend analysis, failure anticipation, remote control of equipment and facilities, diagnostic and fault isolation, remote adjustments, and system security.
Provides weather data processing and display.
Make legal recordings of all voice communications between air traffic controllers and pilots. (continued)
Systems Within the En Route Center and Their Functions (continued)
Provides minimum essential command, control, and communications capabilities to direct the management, operation, and reconstitution of the National Airspace System during a national or local emergency.
Provides interfacing capability to HCS.
Provides communication network for transmitting data via addressed packets.
Provides the capability to request and display NEXRAD weather data.
Provides aircraft situation display capability for the controller that is to be replaced by DSR.
Controls local and remote air-to-ground radios.
Provides FDIO remote print capability.
Provides National Radio Communications System emergency communications essential during and after earthquakes, hurricanes, and tornadoes.
Provides air-to-ground voice communication services and ground-to-ground voice communication services between controllers, other ATC personnel, and others at the same and different en route centers and other ATC facilities.
Provides track generation and traffic display as part of the Oceanic Traffic Planning System.
Oceanic system that displays aircraft position based on extrapolations from flight plans.
Provides a display showing the location of aircraft across the country that is used for strategic planning purposes.
Provides national level management and monitoring of the airspace system, including air traffic flow, aircraft operations, and en route sector and airport utilization and loading.
Systems and Facilities Outside but Interfacing With an En Route Center Automated Flight Service Station Workstation Air Route Surveillance Radar - 1 Air Route Surveillance Radar - 2 Air Route Surveillance Radar - 3 Air Route Surveillance Radar - 4 (continued)
Explanatory Notes to Simplified Block Diagram of an En Route Center’s Systems Environment Systems and Facilities Outside but Interfacing With an En Route Center (continued)
section_title: Display System Replacement: A Brief Description
paragraphs: FAA’s Display System Replacement (DSR) is precisely what its name suggests—a system to replace air traffic controllers’ existing display-related systems in each of the en route centers, including PVDs, channel complexes (i.e., DCC, DCCR, and CDC), multiplexors, display generators, and various other peripheral devices. Accordingly, DSR consists of controller workstations connected via a local area network to three interfacing systems (HCS, EDARC, and Weather and Radar Processor).
While providing controllers a modern ATC system interface (i.e., aircraft situation monitor), DSR is not intended to introduce new situation data, images, displays, or functions. Thus, FAA anticipates that DSR will minimally impact how ATC operations are performed. However, DSR is expected to provide significant improvements in display system reliability (via fault tolerant software and redundant hardware and networks), maintainability (via high level application languages and integrated monitoring and control functions), and expandability (via an open system architecture).
FAA currently plans to deploy DSR to all 20 en route centers in the continental United States, as well as ATC facilities in Anchorage and potentially in Honolulu. According to FAA’s Air Traffic Systems Development Status Report dated June 1996, DSR’s project cost estimate is about $1.06 billion, and as of May 31, 1996, $379 million has been obligated. The operational readiness date for the first site (Seattle) is October 1998 and the last site (Anchorage) is June 2000.
section_title: Major Contributors to This Report
subsections: section_title: Accounting and Information Management Division, Washington, D.C.
section_title: Office of the Chief Economist, Washington, D.C.
paragraphs: Harold J. Brumm, Economist The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.
U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000  or by using fax number (301) 258-4066, or TDD (301) 413-0006.
Each day, GAO issues a list of newly available reports and testimony.  To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone.  A recorded menu will provide information on how to obtain these lists.