Air Traffic Control: Improved Cost Information Needed to Make Billion Dollar Modernization Investment Decisions
Released: 1997-01-22T00:00:00

Letter

Introduction

The Federal Aviation Administration’s (FAA) primary mission is to ensure safe, orderly, and efficient air travel throughout the United States. FAA’s ability to fulfill this mission depends on the adequacy and reliability of the nation’s air traffic control (ATC) system, a vast network of computer hardware, software, and communications equipment. Sustained growth in air traffic and aging equipment have strained the current ATC system, limiting the efficiency of ATC operations. To combat these trends, in 1981 FAA embarked on an ambitious ATC modernization program. FAA estimates that it will spend about $34 billion on the program between 1982 and 2003.

Our work over the years has chronicled many FAA failures in meeting ATC projects’ cost, schedule, and performance goals. As a result, we designated FAA’s ATC modernization as a high-risk information technology initiative in our 1995 report series on high-risk programs.

ATC at a Glance

Automated information processing and display, communication, navigation, surveillance, and weather resources permit air traffic controllers to view key information, such as aircraft location, aircraft flight plans, and prevailing weather conditions, and to communicate with pilots. These resources reside at, or are associated with, several ATC facilities—flight service stations, air traffic control towers, terminal radar approach control (TRACON) facilities, and air route traffic control centers (en route centers). These facilities’ ATC functions are described below.

About 90 flight service stations provide pre-flight and in-flight services, such as flight plan filing and weather report updates, primarily for general aviation aircraft.

Airport towers control aircraft on the ground and before landing and after take-off when they are within about 5 nautical miles of the airport, and up to 3,000 feet above the airport. Air traffic controllers rely on a combination of technology and visual surveillance to direct aircraft departures and approaches; maintain safe distances between aircraft; and communicate weather-related information, clearances, and other instructions to pilots and other personnel.

Approximately 180 TRACONs sequence and separate aircraft as they approach and leave busy airports, beginning about 5 nautical miles and ending about 50 nautical miles from the airport, and generally up to 10,000 feet above the ground, where en route centers’ control begins.

Twenty en route centers control planes over the continental United States in transit and during approaches to some airports. Each en route center handles a different region of airspace, passing control from one to another as respective borders are reached until the aircraft reaches TRACON airspace. En route center controlled airspace usually extends above 18,000 feet for commercial aircraft. En route centers also handle lower altitudes when dealing directly with a tower, or when agreed upon with a TRACON.

Two en route centers—Oakland and New York—also control aircraft over the ocean. Controlling aircraft over oceans is radically different from controlling aircraft over land because radar surveillance only extends 175 to 225 miles offshore. Beyond the radars’ sight, controllers must rely on periodic radio communications through a third party—Aeronautical Radio Incorporated (ARINC), a private organization funded by the airlines and FAA to operate radio stations—to determine aircraft locations.

See figure 1.1 for a visual summary of the processes for controlling aircraft over the continental United States and oceans.

Overview of the ATC Modernization Program

The ATC system of the late 1970s was a blend of several generations of automated and manual equipment, much of it labor-intensive and obsolete. FAA recognized that it could increase ATC operating efficiency by increasing automation. Additionally, FAA forecasted increased future demand for air travel, brought on by airline deregulation of the late 1970s. It also anticipated that meeting the demand safely and efficiently would require improved and expanded services, additional facilities and equipment, improved workforce productivity, and the orderly replacement of aging equipment. Accordingly, in December 1981, FAA initiated its plan to modernize, automate, and consolidate the existing ATC system by the year 2000.

This ambitious modernization program includes the acquisition of new radars and automated data processing, navigation, and communication equipment in addition to new facilities and support equipment. FAA estimates that the modernization will cost over $34 billion through the year 2003 and total over 200 separate projects. ATC information systems make up a large portion of this total, accounting for 169 projects costing $20.7 billion. The Congress will have provided FAA with approximately $14.7 billion of the $20.7 billion through fiscal year 1997.

Over the past 15 years, FAA’s modernization projects have experienced substantial cost overruns, lengthy schedule delays, and significant performance shortfalls. To illustrate, the long-time centerpiece of this modernization program—the Advanced Automation System—was restructured in 1994 after estimated costs tripled from $2.5 billion to $7.6 billion and delays in putting significantly less-than-promised system capabilities into operation were expected to run 8 years or more. Similarly, increases in per-unit costs for five other major ATC projects have ranged from 50 to 511 percent, and schedule delays have averaged almost 4 years.

Our past work on the ATC modernization raised a number of concerns, including concerns about the reliability of projects’ cost estimates. For example, our review of FAA’s Oceanic Display and Planning System found that cost and schedule estimates were questionable because they were based solely on managers’ judgments and were not revised to reflect changing project demands and conditions. Also, our work on AAS highlighted that FAA underestimated the complexity of the system development and relied on its contractor’s deficient cost estimating systems. In its internal risk management guidance, FAA acknowledges that it has a history of unreliable project cost estimates, attributing some of its unenviable record in ATC projects’ cost growth to setting unrealistically low cost estimates, either because of poor cost estimating processes or inadequate system descriptions.

ATC Modernization and Maintenance Management Structure

Two major FAA organizations play key roles in the modernization and evolution of ATC systems—the Office of the Associate Administrator for Research and Acquisitions (ARA) and the Office of the Associate Administrator for Air Traffic Services (ATS). The first, ARA, manages the research, development, and acquisition of modernization projects. Within ARA, two groups are responsible for acquiring systems, while the others handle cross-cutting management functions (e.g., budget formulation, cost estimation, and program evaluation). Also, the William J. Hughes Technical Center is the ATC system test and evaluation facility and supports ATC systems’ research, engineering, and development.

ARA employs an Integrated Product Development System (IPDS) approach. A key component of IPDS is the use of Integrated Product Teams (IPT), which are cross-functional teams aligned with major business and functional areas (i.e., en route, terminal, weather and flight services, air traffic management, oceanic, communications, navigation, surveillance, infrastructure, and information systems). IPT members include systems and specialty engineers, logistics personnel, lawyers, contract specialists, and representatives from the organization responsible for the system’s operations and maintenance. IPTs are responsible for systems research, development, acquisition, and installation. Product teams within these IPTs are responsible for individual ATC system acquisitions or projects. For example, the en route IPT has product teams for the Display Channel Complex Rehost, the Display System Replacement, the Voice Switching and Control System, and several other en route systems.

Air Traffic Control: Status of FAA’s Modernization Program (GAO/RCED-94-167FS, Apr. 15, 1994).

The second major organization involved with ATC systems is ATS. ATS is responsible for directing, coordinating, controlling, and ensuring the safe and efficient utilization of the national airspace system. Organizations within ATS are responsible for planning, operating, and maintaining ATC systems. Responsibility for managing ATC systems is transferred from the IPT to ATS once the systems have been installed and are operational. See figure 1.2 for a visual summary of the ATC modernization and maintenance management structure.

During our review, we assessed six major modernization projects. These projects were the

Voice Switching and Control System (VSCS), which provides air-to-ground voice communication services and ground-to-ground voice communication services between controllers, other ATC personnel, and others at the same and different en route centers and other ATC facilities;

Standard Terminal Automation Replacement System (STARS), which is to replace critical air traffic control computers with new traffic computers, displays, and software in TRACON facilities and towers;

Display System Replacement (DSR), which is to replace air traffic controllers’ existing display-related systems in each of the en route centers;

Airport Surveillance Radar-9 (ASR-9), which monitors aircraft movement and position within a radius of 60 miles of an airport terminal;

Wide Area Augmentation System (WAAS) for the Global Positioning System (GPS), which is to provide augmentations to the Department of Defense’s GPS in order to allow improved navigation on domestic and oceanic air routes; and

Display Channel Complex Rehost (DCCR), which is an interim replacement to the mainframe computer system that processes radar and other data into displayable images on controllers’ screens.

Roles and Responsibilities for Project Cost Estimating

The FAA organization responsible for estimating costs on the projects we assessed varied depending on the project’s stage in its life cycle. According to FAA acquisition rules in place when the latest life cycle cost estimate for the projects we assessed were developed, FAA’s Investment Analysis and Operations Research organization (ASD-400), which is one of the groups within ARA that handles cross-cutting management functions, developed life cycle cost estimates early in the project life cycle—both when mission needs were evaluated and again when evaluating the costs, benefits, and feasibility of project alternatives. Once the decision was made to invest in a given alternative, the IPTs assumed responsibility for updating the cost estimates. However, IPTs could also choose to develop their own cost estimates prior to the investment decision point, rather than have ASD prepare them.

Of the six projects we reviewed, four used ASD- at some point in the project’s life cycle. The other two did not. In addition, some project managers updated the acquisition phase portion of the life cycle cost estimates periodically between the times that the full life cycle cost estimates were revised.

FAA’s organizational responsibilities for cost estimating have recently changed. In October 1995, the Congress instructed FAA to develop and implement a new acquisition management system, which would not be subject to various existing acquisition laws. On November 15, 1995, the President signed this bill into law. FAA began implementing this new acquisition management system in April 1996 with the issuance of broad policies, guiding principles, and internal procedures. While not yet fully implemented, these specify that two key decisions be made at the corporate level by the Joint Resources Council (JRC), a newly formed body comprised of the associate administrators for operations and acquisitions as well as officials responsible for acquisitions, financial services, and legal counsel. These decisions are (1) whether mission needs warrant entry into investment analysis and (2) whether to invest in the project at the conclusion of investment analysis. FAA identified the latter as the most important decision in the life cycle acquisition management process. Accordingly, FAA plans to establish a “center of excellence” for investment analysis with experts in cost estimating, risk assessment, market analysis, and affordability analysis.

Under this new scenario, investment analysis will be conducted as a joint enterprise by the FAA organization sponsoring the system (i.e., ATS) and the FAA organization responsible for acquiring it (i.e., ARA). The purpose is threefold: to ensure that (1) users buy into the solution, (2) acquisition specialists have a voice in the cost, schedule, and performance baselines they will have to live with, and (3) the investment analysis staff understands the concerns of the operations and acquisitions organizations. Under this approach, the sponsoring organization, with technical support from the investment analysis staff, develops and approves its requirements in the form of a Requirements Document. The investment analysis staff leads the effort to identify and analyze candidate solutions through market surveys, alternatives analysis, and affordability assessments, with support from the sponsoring organization and the ARA IPT responsible for acquiring it. This effort culminates in the Investment Analysis Report, which is to contain comprehensive quantitative data for each alternative, such as life cycle cost, cost-benefit ratios, and risk. The IPTs use this information to generate cost and schedule baselines for each alternative in the form of an Acquisition Program Baseline. At the investment decision point, the JRC decides on an alternative; baselines the project’s requirements, costs, schedules, performance, and benefits; and commits the agency to full funding of the program. Thereafter, any changes to these baselines must be approved by the JRC. In fact, no funding may be committed or obligated that would exceed the program cost baseline until the increase is approved by the JRC and included in agency plans and budgets.

The success of FAA’s new investment analysis and decision-making approach depends on many factors, not the least of which is the reliability of ATC project cost information discussed in this report.

Sound Investment Decisions Require Reliable Cost Estimating and Accounting Practices and Data

Reliable cost estimates and monitoring of actual costs are essential to informed investment decision-making throughout the development and maintenance of capital items, such as ATC systems. In the case of FAA, they are cornerstones to its aforementioned investment analysis and decision-making processes.

In 1994, we reported on how leading organizations improved mission performance through information technology. Among other things, we reported that successful organizations manage information system projects as investments, and continually assess the quality of projects’ estimated costs and carefully monitor projects’ actual costs against these estimates. Furthering this initiative, OMB’s 1995 guidance, Evaluating Information Technology Investments, calls for selecting information technology project investments on the basis of cost, benefit, risk, and return; controlling projects by comparing ongoing actual results being achieved with projected costs, benefits, and risks; and, finally, evaluating projects after they have been implemented to determine actual cost, benefits, risks, and returns, and modifying the selection and control processes based on lessons learned. This guidance has since been embodied in (1) the Clinger-Cohen Act of 1996, which requires the selection of information technology investments on the basis of competing projects’ estimated costs, benefits, and risks, and (2) the Chief Financial Officers (CFO) Act of 1990, which requires federal agencies to maintain integrated accounting and financial management systems that permit systematic and reliable measurement of projects’ cost and performance. Additionally, OMB Circular A-11, Part 3, requires agencies to request full up-front budget authority for all ongoing and new fixed assets (including information technology) in their fiscal year 1998 budget submission. This circular also requires a fixed asset plan and justification for major acquisitions, including, among other items, an analysis of full life cycle costs and an estimate of the risk and uncertainty in meeting project goals.

Experts Advocate Key Estimating Process Requisites

The Software Engineering Institute’s (SEI) Capability Maturity Model (CMM), the standard used by government and industry to determine the maturity of an organizations’ software development processes, also highlights the need for good estimates and good estimating processes.Three of the CMM’s key process areas for level 2 (repeatable) process maturity are project planning, project tracking, and subcontract management. These process areas must have reliable estimates for size, effort, schedule, and cost if they are to be performed successfully. The CMM further requires that the procedures that implement these key process areas be documented.

To improve the state of practice for software cost and schedule estimating, SEI developed and published (1) criteria for establishing sound estimating processes and (2) a guide for managers to use in validating an individual project’s estimate. These documents, in effect, describe “best practices” used in industry and government for estimating software costs and schedules. However, SEI found that the “best practices” are equally applicable to hardware and integrated systems projects, and therefore allows for substituting the word “system” for “software” throughout its guides and checklists. SEI also noted that while the criteria target the acquisition/development phase of a project’s life cycle, the concepts are also applicable to other phases of the life cycle.

According to SEI’s Checklists and Criteria for Evaluating the Cost and Schedule Estimating Capabilities of Software Organizations, in order to have sound estimating processes, an organization should have six attributes, or requisites, institutionally embedded in its policies and procedures. These include (1) a corporate memory, or historical database(s), for cataloging cost estimates, revisions, reasons for revisions, actuals, and other contextual information, (2) structured processes for estimating software size and the amount and complexity of existing software that can be reused, (3) cost models calibrated/tuned to reflect demonstrated accomplishments on similar past projects, (4) audit trails that record and explain values used as cost model inputs, (5) processes for dealing with externally imposed cost or schedule constraints in order to ensure the integrity of the estimating process, and (6) data collection and feedback processes that foster capturing and correctly interpreting data from work performed. SEI provides detailed checklists for assessing an organization’s satisfaction of each requisite.

These same six requisites are interwoven through seven questions SEI poses in A Manager’s Checklist for Validating Software Cost and Schedule Estimates. The seven questions are (1) Are the objectives of the estimate clear and correct? (2) Has the task been appropriately sized? (3) Are the estimated cost and schedule consistent with demonstrated accomplishments on past projects? (4) Have the factors that affect the estimate been identified and explained? (5) Have steps been taken to ensure the integrity of the estimating process? (6) Is the estimate based on reliable evidence of the organization’s past performance? and (7) Has the situation remained unchanged since the estimate was prepared? Once again, SEI provides detailed checklists for addressing these seven questions. These SEI publications are further discussed in chapter 2 and in appendixes I, II, and III.

Cost Accounting Requirements

Requirements for agency cost accounting have been evolving for decades. In a 1985 report, the Comptroller General presented a framework for strengthening agencies’ financial management structure. This report called for the integration of accounting and budgeting systems to better monitor progress against estimates and to better estimate future program costs. More specifically, it states that actual costs must be maintained and monitored in order to effectively manage programs and control costs. This approach was embodied in the CFO Act of 1990, which requires agencies to develop and maintain integrated accounting and financial management systems which provide for (1) the development and reporting of cost information and (2) the systematic measurement of performance.

Objectives, Scope, and Methodology

The objectives of our review were to determine if (1) FAA’s project cost estimates are based on good estimating policies and practices and (2) the actual costs of ATC modernization projects are being properly accumulated.

To determine if FAA’s estimates were based on good policies and practices, we researched current literature and interviewed project estimating experts to identify the key components of good cost estimating practices; obtained and analyzed FAA’s policies and practices for estimating costs to determine what criteria (directives, orders, instructions, and implementing procedures), if any, FAA has in place to guide managers in developing projects’ cost estimates; assessed FAA’s cost estimating policies, practices, tools, and techniques to determine if they incorporate the key components of good cost and schedule estimating practices advocated by SEI and other experts; and selected FAA’s five largest (based on latest life cycle cost estimates) ongoing ATC modernization projects and one project that was the subject of another GAO review, and interviewed project managers and assessed project documentation on these six projects to determine (1) how the current life cycle cost baseline was estimated and (2) how this estimating approach compared to the SEI project-level questions. To do this, we compared each projects’ documentation to SEI’s detailed checklists for each question; determined if the project satisfied, partially satisfied, or did not satisfy each checklist item and assigned points accordingly (1, .5, or 0 points, respectively); and then summed the points and presented them as a portion of the total points available (e.g., 4/10). Because the focus of this effort is to assess FAA’s cost estimating processes and not to validate the accuracy or completeness of the estimates, we did not evaluate the quality of the estimates.

To determine whether the actual costs of ATC modernization projects are being properly accumulated, we obtained and reviewed (1) selected reports and testimonies issued by GAO, the Department of Transportation’s Office of the Inspector General, and the Defense Contract Auditing Agency, (2) related policies and procedures issued by the Department of Transportation, (3) applicable accounting standards and guidance, and (4) applicable OMB directives; reviewed FAA’s policies and procedures governing ATC financial management and interviewed program managers and financial accounting staff to determine (1) their roles and responsibilities for recording and managing ATC cost information and (2) the financial processes used to accumulate and record ATC costs; and reviewed available information for the five largest (based on life cycle cost estimates) ongoing ATC projects and determined if costs are properly accumulated by (1) obtaining available financial information on the projects and identifying the cost elements included and excluded and (2) assessing reconciliation procedures among varying sources of information.

We requested comments on a draft of this product from the Secretary of Transportation. On December 10, 1996, we obtained oral comments from Transportation and FAA officials, including representatives from the Office of the Secretary of Transportation, the Executive Assistant to the FAA Chief Financial Officer, the FAA Manager of the Cost Accounting System Division, and the FAA Program Director for Investment Analysis and Operations Research. Their comments are presented and addressed in chapters 2 and 3 of this report.

We performed our work at the Federal Aviation Administration in Washington, D.C., and the Software Engineering Institute in Pittsburgh, Pennsylvania, between February and December 1996. Our work was performed in accordance with generally accepted government auditing standards.

ATC Modernization Lacks the Institutional Capacity to Produce Reliable Cost Estimates

Reliable estimates of projects’ expected costs are essential to decide among alternative investments. According to SEI, consistently producing them requires defined institutional processes for deriving estimates, archiving them, and measuring actual performance against these estimates. FAA’s cost estimating processes used on its ATC modernization projects do not meet SEI criteria. These weaknesses are exacerbated by FAA’s practice of presenting cost estimates as precise, point estimates. By doing so, FAA obscures the estimates’ inherent uncertainty and may mislead decisionmakers.

Policies and Practices for Estimating Costs Do Not Satisfy “Best Practices” Requisites

FAA’s institutional processes for estimating ATC projects’ costs do not fully satisfy any of the six SEI requisites. According to SEI, all six must be satisfied to produce credible estimates. The six requisites are described below, along with our analysis showing that FAA’s institutional policies and practices fail to meet them. (See appendixes I and II for more detail.) We shared this analysis with FAA’s cost estimating authority, who agreed that FAA’s policies and practices do not meet SEI’s process requirements.

Requisite 1: A Corporate Memory

According to SEI, estimating organizations should have a process for organizing and retaining project cost and schedule information in a historical database, and for using it as an integral part of the estimating process. This database should contain detailed information on projects’ original estimates, revised estimates, reasons for revisions, and actual performance against the estimates. It should also contain descriptive contextual information that enables people to understand and correctly interpret the data in the database.

FAA has no institutional corporate memory, or historical database, on ATC projects’ cost and schedule estimates and performance. While FAA has a number of stand-alone databases within different groups, none provide a complete picture of estimates, assumptions that make up the estimates, revisions, and actual performance on projects. For example, the Cost Benefit Analysis System (CBAS) is a database that contains some information on projects’ cost estimates and planned budget levels. However, it contains no information on how and why estimates are revised, why budget streams differ from estimates, or what projects actually cost.

Further, what limited information is available on actual cost performance is not an integral part of the project estimating process, and the information on cost estimates that is retained by the central estimating organization is not readily available to the project personnel that are responsible for updating estimates after the initial investment decision has been made.

Requisite 2: Structured Processes for Estimating Product Size and the Amount and Complexity of Existing Software That Can Be Reused

According to SEI, estimating organizations should follow well-defined, structured processes for estimating product size and the amount and complexity of existing software that can be reused. It should be clear what is included and what is excluded from size estimates, and new estimates should be checked by comparing them to measured sizes of existing software products.

FAA has no institutional process for estimating product size and the amount of existing software that can be reused. Each project manager decides to estimate software size and reuse as he or she chooses. Among ATC projects, these processes range from simple lines-of-code estimates using an individual’s personal knowledge of similar systems’ sizes to sophisticated analysis based on project-unique variables. For example, the original software size estimates for the Standard Terminal Automation Replacement System (STARS) was a rough approximation based on the number of lines of code in a predecessor system. A more recent, though not yet official, estimating effort established size estimates for each desired STARS software function and then compared the estimates to known sizes of similar functions on two other completed projects. This effort also used a checklist to ensure that no desired functions were overlooked and accounted for differences between STARS and its predecessor system. The latter, more careful size estimate is over three times the original estimate for new and modified software.

Requisite 3: Mechanisms for Extrapolating From Demonstrated Accomplishments on Past Projects

According to SEI, estimating organizations should have documented processes for extrapolating from past experiences. These processes should include the use of cost models that are calibrated and validated on the basis of actual experience. Further, differences between cost models’ outputs should be analyzed and explained.

FAA’s estimating guidance recommends 10 different cost models as acceptable estimating tools; however, there is no requirement for project estimators to use these models or to collect and use past experience to calibrate these models. As a result, projects’ use and calibration of the models is inconsistent. For example, estimators on three of the six projects we assessed did not use cost models in their estimates whereas Display System Replacement (DSR) officials not only used four cost models, but also calibrated them to past experiences on a predecessor system.

Requisite 4: Audit Trails

According to SEI, estimating organizations should prepare adequate audit trails of inputs to the estimates, including parameters used in cost models and their rationales.

FAA estimating guidance requires that cost estimates be documented and reproducible. However, the degree of documentation and the extent of any accompanying explanation is left to the discretion of each project’s estimators. As a result, the detail and quality of the audit trails on ATC projects is inconsistent. For example, only handwritten notes document how the Display Channel Complex Rehost (DCCR) estimate was derived. On the other hand, the DSR estimate and the ongoing STARS estimating effort are supported by volumes of documentation delineating the assumptions, processes, and model inputs used.

Requisite 5: Integrity in Dealing With Dictated Costs or Schedules

According to SEI, organizations should ensure that the effects of dictated, or externally imposed, costs or schedules are determined and explicitly presented to management. Estimators should document and managers should approve any changes made to model parameters to accommodate dictated costs or schedules, the rationale for making the changes, and the effect of the changes on other factors—cost, schedule, or risk.

FAA has no institutional process for ensuring integrity in dealing with dictated costs or schedules. As a result, each project manager determines his or her own response to externally imposed constraints. Only one of the projects we assessed acknowledged working under an externally imposed schedule. STARS project officials are preparing a cost estimate that shows the cost of meeting the compressed schedule. However, without an institutional policy requiring such action, there is no assurance that all dictated constraints on all ATC projects will be handled so effectively.

Requisite 6: Data Collection and Feedback Processes That Foster Capturing and Correctly Interpreting Data From Work Performed

According to SEI, organizations should have a defined process for gathering information on ongoing and completed projects (including original estimates, revised estimates, and post-mortem assessments) and entering this information into the historical database.

FAA has no institutional process for gathering information on completed projects and entering it in a historical database. Post-mortem reviews are performed rarely, and only on an ad hoc basis. Instead, each project manager determines the type and amount of information retained.

Individual ATC Projects’ Cost Estimating Approaches Are Inconsistent and Sometimes Unreliable

Because FAA does not have well-defined, institutional processes for estimating information technology projects’ costs, the approaches used and the reliability of the estimates are inconsistent.

To assist management in assessing the credibility of a given project cost estimate, SEI developed seven questions which must be answered. The seven questions are (1) Are the objectives of the estimate clear and correct? (2) Has the task been appropriately sized? (3) Are the estimated cost and schedule consistent with demonstrated accomplishments on other projects? (4) Have the factors that affect the estimate been identified and explained? (5) Have steps been taken to ensure the integrity of the estimating process? (6) Is the estimate based on reliable evidence of the organization’s past performance? and (7) Has the situation remained unchanged since the estimate was prepared? SEI developed a detailed checklist to assist in addressing each question. (See appendix III for more information on SEI’s questions and checklists.)

We applied this checklist to the most recently baselined life cycle cost estimate on six ATC projects (the five ongoing projects with the largest life cycle cost estimates plus one ongoing project that was the subject of another GAO review). These projects are the Voice Switching and Control System (VSCS), the Standard Terminal Automation Replacement System (STARS), the Display System Replacement (DSR), the Airport Surveillance Radar-9 (ASR-), the Wide Area Augmentation System for the Global Positioning System (WAAS), and the Display Channel Complex Rehost (DCCR).

Of the six systems, two (ASR-$857 million. However, there is no documentation describing how ASR-software size estimates were determined or what assumptions were used to estimate costs from these size estimates. As a result, there is no analytical way to determine the credibility of the original estimate or estimating approach. Moreover, without documentation, FAA cannot systematically use its ASR-In addition, project estimators derived official WAAS life cycle cost estimates based on experiences with the National Satellite Test Bed, a precursor to the WAAS development. However, little documentation exists on how the original WAAS size estimates were derived and what assumptions and parameters were used to estimate costs. Project officials are currently updating estimates using a published sizing methodology and the Software Life Cycle Intermediate Model (SLIM) cost model, and stated that they believe this is a much more rigorous approach than prior estimating efforts. However, this more structured approach also falls short of SEI requirements. For example, the project software estimator stated that one cost model parameter (input variable) is a ranking of the sophistication of the developer’s software environment. The estimator scored the WAAS contractor as very high on this parameter and showed us where this information was captured in the SLIM database. However, the estimator went on to explain that he chose a high score based on the contractor’s rating as a level 3 organization on SEI’s CMM and the fact that they have and use Computer-Aided Software Engineering (CASE) tools. This explanatory information was not recorded in the SLIM database or in any other documentation, and thus is not available to anyone trying to understand or validate the estimate or learn from this estimating experience.

The current official estimate for the last of the six systems, DSR, was derived using an approach that partially or fully satisfied six of the seven SEI reliability questions. Examples of DSR’s adherence to SEI guidance include (1) DSR’s software size estimate was developed by identifying needed software functions, and then determining the amount of new code needed and reusable code available for each of these software functions, (2) DSR estimators calibrated cost estimating models to past experience on DSR’s predecessor system, the Initial Sector Suite System (ISSS), and (3) estimators used templates to ensure that key cost factors would not be overlooked. However, DSR estimators did not record the rationales for the parameters they used in their cost models or explain differences among cost model results. (See appendix IV for further information on the results of our assessment.)

FAA’s Portrayal of ATC Projects’ Cost Estimates Implies an Unjustified Level of Precision

Software and systems development experts agree that early project estimates are by definition imprecise, and that this inherent imprecision decreases during the project’s life cycle as more information becomes known about the system. Some have described this phenomenon as a “cone of uncertainty” that is widest early in the life cycle and narrows over time as more becomes defined and known about the project. (See figure 2.1.) These experts emphasize that each cost estimate should include an indication of its degree of uncertainty, possibly as an estimated range or qualified by some factor of confidence. For example, a cost estimate of $1 million could be presented as a range from $750,000 to $1.25 million or as $1 million with a confidence level of 90 percent, indicating that there is a 10-percent chance that costs will exceed this estimate.

FAA does not reveal its estimates’ degree of uncertainty to managers involved in investment decisions. Instead, FAA presents its projects’ cost estimates as unqualified point estimates, thereby suggesting an element of precision that does not exist. A budget official stated that FAA presents project cost estimates as such because “this is the way OMB and Congress want to see it.” Further, he stated that in today’s environment of lean budgets, the low-end of the estimate range is all that FAA can afford and all that is salable, and therefore, this is what they present.

By presenting a point estimate instead of a range of estimates or a realistically qualified estimate, FAA is not fully disclosing all relevant information about the projects’ potential costs and inherent risk. The result is uninformed, and thus potentially unwise, investment decisions.

FAA Organizations Have Initiated Efforts to Improve Cost Estimating

During the course of our review, FAA organizations initiated several efforts to improve their processes for estimating and archiving cost information. However, these efforts are still relatively new and have not yet been institutionalized. We did not evaluate any of these efforts.

FAA’s Office of Investment Analysis and Operations Research (ASD-The same organization is also testing the Cost and Performance Management System (CPMS) to track operations and maintenance (O&M) costs by project for the first time. An airways facilities manager stated that CPMS will eventually be integrated with ACEIT to allow estimators to use actual project cost information in estimating new projects’ O&M costs.

Conclusions

Multimillion dollar, and even billion-dollar, investment decisions on air traffic control modernization projects are being made without reliable information on the projects’ estimated and actual costs. FAA does not have well-defined, structured estimating processes that are rigorously followed, and does not disclose the estimates’ inherent uncertainty. Without better estimates of cost, FAA’s new investment analysis and decision-making processes are unlikely to be effective.

Recommendations

We recommend that the Secretary of Transportation direct the FAA Administrator to institutionalize defined processes for estimating ATC projects’ costs. At a minimum, these processes should include the following SEI requisites, each of which are described in more detail in this report: a corporate memory (or historical database), which includes cost and schedule estimates, revisions, reasons for revisions, actuals, and relevant contextual information; structured approaches for estimating software size and the amount and complexity of existing software that can be reused; cost models calibrated/tuned to reflect demonstrated accomplishments on audit trails that record and explain all values used as cost model inputs; processes for dealing with externally imposed cost or schedule constraints in order to ensure the integrity of the estimating process; and data collection and feedback processes that foster capturing and correctly interpreting data from work performed.

We also recommend that the Secretary direct the Administrator to immediately begin disclosing the inherent uncertainty and range of imprecision in all ATC projects’ official cost estimates presented to executive oversight agencies or the Congress.

Additionally, we recommend that the Secretary direct the Administrator to report to the Secretary and FAA’s authorizing and appropriation committees on progress being made on these recommendations as part of the agency’s fiscal year 1999 budget submission.

Agency Comments

DOT and FAA officials provided oral comments on a draft of this report. These officials concurred with the report’s findings, conclusions, and recommendations on cost estimating. They also stated that this report will be useful as FAA strives to improve its cost estimating capabilities.

FAA Is Not Adequately Accounting for ATC Project Costs

Agencies are required to maintain adequate systems of accounting and internal controls to provide managers and other decisionmakers with reliable financial information to effectively measure performance and make sound investment decisions. In the case of the ATC modernization program, FAA is not satisfying this requirement. Specifically, ATC project information does not include all relevant project costs, including internal personnel compensation, benefits, and travel (PCB&T) costs, because FAA lacks a cost accounting system to accumulate and allocate these costs to specific projects. FAA’s internal accounting policies include a requirement for a cost accounting system; however, this policy has not been implemented. As a result, project managers are unable to measure actual costs and their ability to make informed decisions is impaired. Moreover, complete project information is not available to feed back into, and thereby improve, future project cost estimates.

Agencies Are Required to Maintain Adequate Accounting Systems

The Federal Managers’ Financial Integrity Act of 1982 (FMFIA) requires that agency systems of internal accounting and administrative control comply with internal control standards prescribed by the Comptroller General and provide reasonable assurance that among other things, obligations and costs comply with applicable law and revenues and expenditures applicable to agency operations are recorded and accounted for properly. FMFIA also requires that agency heads issue an annual report, transmitted to the President and the Congress, detailing whether their internal control systems fully comply with the act’s requirements, including the identification of material systems weaknesses and plans for corrective actions.

The Chief Financial Officers (CFO) Act of 1990 requires agencies to develop and maintain integrated agency accounting and financial management systems that comply with applicable accounting principles, standards, and requirements, including the preparation of complete, reliable, consistent, uniform, and timely information that is responsive to agency management’s financial information needs; the development and reporting of cost information; the integration of accounting and budgeting information; and the systematic measurement of performance.

Recently, the Statement of Federal Financial Accounting Standards no. (SFFAS Federal Government, was issued, effective for fiscal periods beginning after September 30, 1996. These standards require a reporting entity to accumulate and report the full cost of its activities regularly for management information purposes. The full cost of a project is described as the sum of (1) the costs of resources consumed by the project that directly or indirectly contribute to the output and (2) the costs of identifiable supporting services provided by other organizations within the reporting entity and by other reporting entities. These standards also require that the full costs of resources be assigned to outputs through costing methodologies or cost finding techniques that are most appropriate to the organization’s operating environment and that they be followed consistently. While SFFAS has only been effective for a short time, and therefore was not applicable during the period of our review, it provides cost accounting criteria which are now required to be implemented by all agencies.

Additionally, the 104th Congress passed the Federal Financial Management Improvement Act of 1996 which, among other provisions, requires agencies to comply with federal accounting standards.

ATC Project Costs Are Not Being Properly Accumulated

FAA financial systems supporting the ATC modernization program do not accumulate all project costs, and thus, managers do not receive all relevant financial information needed to effectively manage their projects. Of the five projects whose financial information we reviewed, none of the project managers could provide the total of all costs incurred from the project’s inception. Instead, they provided: contract numbers for their respective projects so that cost data for each contract could be extracted from FAA’s Departmental Accounting and Financial Information System (DAFIS) and aggregated to provide total contract costs. However, these contract costs could be understated because project officials could not verify that they provided us all applicable contract numbers. incomplete project costs. These costs were not complete because they did not include (1) Personnel Compensation, Benefits, and Travel (PCB&T) costs associated with the Facilities and Equipment (F&E) appropriations account and (2) all costs paid out of the Operations and Maintenance (O&M) appropriations account. PCB&T costs for the F&E appropriation include internal FAA costs that are related to project design, contracting, and contractor oversight. O&M costs are costs associated with the administration, operation, repair, and maintenance of operating FAA facilities and are generally the single largest component of an information systems life cycle cost. In 1995, total PCB&T costs were approximately $2 billion for all ATC projects. PCB&T and O&M costs are accounted for separately and are not allocated to individual ATC projects. Because these costs are not allocated to specific projects, full life cycle costs of projects cannot be determined and may be significantly understated.

An additional limitation is that FAA does not carry over and report the costs associated with terminated or redirected projects as part of the successor projects’ costs, even though successor projects reuse parts of the predecessors’ components. As a result, the full costs of “restructured” ATC projects are understated. Accounting for the full costs of projects requires that the costs related to usable portions of terminated or redirected projects be included in the costs of the ongoing projects. In addition, full project cost accounting information would require that costs of unused parts of terminated or redirected projects be separately identifiable within the “corporate memory.”

For example, one of the projects we reviewed, the Display System Replacement (DSR), is a follow-on to an earlier terminated project known as the Initial Sector Suite System (ISSS). According to FAA officials, DSR software and hardware salvaged from ISSS accounts for about 19 percent of DSR’s cost. However, these costs are not included in DSR’s accumulated and reported costs because, according to an FAA official, these costs are considered “sunk costs.” The term “sunk costs” is generally used to describe costs that have been incurred in the past and have no relevance to future decision-making. However, we believe these costs should be considered a part of a project’s full cost since they would be instructive in reliably estimating costs of similar systems. In addition, information about the amount of costs associated with the unused portions of terminated projects should be retained in the “corporate memory” to provide a full picture of the real cost of development projects.

FAA Lacks a Cost Accounting System for Its ATC Modernization Program

A managerial cost accounting system supports the collection, measurement, accumulation, analysis, interpretation, and communication of cost information to allow users to determine the cost of specific programs and activities and the composition of, and changes in, these costs. As mentioned above, the CFO Act requires agencies to develop and maintain a cost accounting capability that captures both budgetary and financial accounting data and generates performance measures. FAA’s internal policies require a cost accounting system and state that the cost accounting system should be integrated with the general accounting system. However, these policies have not been implemented; thus FAA project managers do not have the capability to fully account for costs being incurred for the ATC modernization program.

Instead of a cost accounting system, several financial management systems account for specific financial accounting and budgetary data, but these systems are not integrated and they do not provide the full cost information necessary for investment decisions. These systems include the following.

Departmental Accounting and Financial Information System (DAFIS): This system is the Department of Transportation’s core accounting system. However, this system is not a cost accounting system because not all cost information is captured by project. In addition, the Department’s Office of the Inspector General (OIG) reported that DAFIS data is unreliable and inaccurate. For example, the OIG reported that major balance discrepancies existed between DAFIS accounts and their supporting details. Further, in April 1996, a cost accounting systems consultant reported that DAFIS does not provide all levels of management with timely, accurate, relevant, and meaningful information.

Financial Management System (FMS): This FAA system is used by ATC project managers to establish quarterly obligation plans under the Facilities and Equipment (F&E) appropriation and to track actual obligations against these plans. However, FMS is not a cost accounting system because it (1) contains only obligation data and (2) does not contain all relevant cost data, such as those for PCB&T.

Cost of Performance System (COPS): This system is used by FAA organizations responsible for operating and maintaining ATC systems to allocate aggregated O&M obligation data to individual cost centers (that is, field maintenance organizations). COPS is not a cost accounting system because it does not contain information on actual costs. In addition, COPS’ O&M cost data is not allocated to and accumulated for individual projects.

Research, Engineering and Development, Monitoring, Analysis, and Control System (REDMACS): This system is used by FAA organizations responsible for ATC research and development projects to establish quarterly obligation plans for the Research, Engineering, and Development (RE&D) appropriation and to track actual obligations against these plans. REDMACS is not a cost accounting system because it does not contain information on actual costs.

Project managers have also developed their own unique systems to account for F&E and RE&D obligations. These “cuff” systems range from spreadsheets to more sophisticated financial management systems, but do not include O&M costs and generally do not capture actual costs. None of the systems listed above, either individually or combined, constitute a cost accounting system because they do not provide for accumulation and monitoring of total costs.

In recognition of the need for accurate and reliable cost information, FAA’s Associate Administrator for Administration established a new Cost Accounting Systems Division in August 1996 and engaged a consultant to assist the agency in defining its cost accounting requirements and in designing and implementing a system to meet these requirements. According to the Manager of the Cost Accounting System Division, this system will use data from several financial management systems, including DAFIS, and is planned to be in place by October 1, 1997.

Conclusions

Reliable project cost information that is both complete and accurate is needed to estimate future project costs, make sound investment decisions, and effectively manage projects. FAA does not have a cost accounting system capable of reliably accumulating full project cost information, and therefore cannot reliably estimate future project costs, ensure that investment decisions are sound, or manage projects effectively. Without better cost information, FAA’s new investment analysis and decision-making processes are unlikely to be effective.

Recommendations

In light of FAA’s weaknesses in accounting for and reporting ATC project costs, we recommend that the Secretary of Transportation direct the FAA Administrator to acquire or develop and implement a managerial cost accounting capability that will satisfy the requirements of SFFAS We further recommend that the Secretary report FAA’s lack of a cost accounting capability for its ATC modernization as a material internal control weakness in the Department’s fiscal year 1996 FMFIA report and in subsequent annual FMFIA reports until the problem is corrected.

Also, we recommend that the Secretary direct the Administrator to report to the Secretary and FAA’s authorizing and appropriation committees on progress being made on these recommendations as part of the agency’s fiscal year 1999 budget submission.

Agency Comments and Our Evaluation

In providing oral comments on a draft of this report, DOT and FAA officials stated that since they are in the process of acquiring a cost accounting system and plan to have an “initial operating capability” by October 1, 1997, they do not agree with our recommendations and consider them unnecessary. While we acknowledge and support FAA’s cost accounting organizational and system initiatives, it is important to note that its cost accounting system acquisition is still very early in its acquisition life cycle and much remains to be accomplished before FAA can have the cost accounting capability we recommend. In fact, FAA has yet to develop detailed functional requirements for this system, thereby precluding our analysis at this time of whether its plans will satisfy our recommendation. Additionally, until FAA implements our recommendation and improves the accuracy of underlying data in feeder systems like DAFIS, it will continue to lack adequate cost information needed to effectively manage its ATC system acquisitions. Disclosure of such a management control weakness is one of the objectives of the FMFIA, and therefore we continue to believe that FAA should report its lack of a cost accounting system as a material weakness in its FMFIA reports until the problem is corrected.