Air Traffic Control: FAA's Modernization Investment Management Approach Could Be Strengthened
Released: 1999-04-30T00:00:00

Letter

Introduction

The Federal Aviation Administration’s (FAA) mission is to promote the safe, orderly, and expeditious flow of air traffic in the United States through what is commonly referred to as the National Airspace System (NAS). FAA’s ability to fulfill its mission depends on the adequacy and reliability of the nation’s air traffic control (ATC) system—the principal component of the NAS—which comprises a vast network of radars; automated data-processing, navigation, and communications equipment; and ATC facilities. It is through the ATC system that FAA provides services such as controlling takeoffs and landings and managing the flow of traffic between airports. Sustained growth in air traffic and FAA’s aging equipment have strained the current ATC system. This growth in traffic is predicted to continue as the number of passengers traveling on U.S. airlines is expected to grow from about 674 million in 1998 to 1.055 billion by 2010, an increase of about 57 percent.

To relieve the problems of aging equipment and to accommodate the predicted growth in air traffic, FAA initiated a multibillion-dollar modernization effort in December 1981. Our work over the years has chronicled many of FAA’s failures in meeting projects’ cost, schedule, and performance goals. Because of the size, complexity, cost, and problem-plagued past of FAA’s modernization program, we have designated it a high-risk information technology investment since 1995.

The ATC System Modernization Program Is Complex, Costly, and Historically Problematic

Under its ambitious modernization program, FAA is acquiring new surveillance, data-processing, navigation, and communications equipment in addition to new facilities and support equipment. Totaling 126 active projects, the modernization is estimated to cost $26.5 billion from fiscal year 1982 through fiscal 2004. Of this total, FAA estimates that it will need $12.9 billion from fiscal year 1982 through 2004 for 59 information technology projects—the software-intensive and complex information and communications systems supporting the ATC system. These projects range from those designed to replace equipment used by controllers to communicate with aircraft and with each other to radars that provide controllers with surveillance information for separating aircraft. An example of an information technology project is the Display System Replacement project that will modernize equipment in FAA’s en route facilities by replacing 20- to 30-year-old display channels, controllers’ workstations, and network infrastructure.

Over the past 17 years, FAA’s modernization projects have experienced substantial cost overruns, lengthy delays, and significant performance shortfalls. To illustrate, the longtime centerpiece of the modernization program—the Advanced Automation System—was restructured in 1994 after estimated costs to develop the system tripled from $2.5 billion to $7.6 billion and delays in putting significantly less-than-promised system capabilities into operation were expected to run 8 years or more over the original estimates. These problems have persisted. For example, two key projects in the modernization effort—the Wide Area Augmentation System and the Standard Terminal Automation Replacement System—have encountered significant cost increases, delays, and changes in requirements. In the case of the Wide Area Augmentation System, between September 1997 and January 1998, total estimated costs increased by $600 million, or 25 percent, from $2.4 billion to $3 billion. The increased costs were attributable to FAA’s including previously overlooked costs for periodically updating the project’s equipment and to higher-than-expected operations and maintenance costs. In the case of the Standard Terminal Automation Replacement System, although FAA has not officially changed the project’s baseline approved in February 1996, the baseline is in jeopardy of being breached because of unions’ concerns surrounding human-factor and design issues, the refinement of requirements, and the interjection of a new project phase. FAA estimates that these issues have the potential to increase the project’s costs from $294 million to $410 million above the approved baseline. FAA also estimates that the project’s initial completion could be delayed by almost 2-1/2 years.

FAA Has Developed the Acquisition Management System to Manage Its Modernization Investments

Because of FAA’s contention that some of its modernization problems were caused by federal acquisition regulations, the Congress enacted legislation in November 1995 that exempted the agency from most federal procurement laws and regulations and directed FAA to develop and implement a new acquisition management system that would address the unique needs of the agency. On April 1, 1996, in response to the Congress’s action, FAA implemented a new acquisition management system. The system is intended to reduce the time and cost to field new products and services by introducing (1) a new investment management system that spans the entire life cycle of an acquisition, (2) a new procurement system that provides flexibility in selecting and managing contractors, and (3) organizational and cultural reform that supports the new investment and procurement systems.

The Acquisition Management System (AMS) provides high-level acquisition policy and guidance for selecting and controlling FAA’s investments throughout all phases of the acquisition life cycle, which is organized into a series of phases and decision points, including (1) mission analysis, (2) investment analysis, (3) solution implementation, (4) in-service management, and (5) service life extension. AMS provides guidance on the documents and decisions that result from each of these phases. For example, through the mission analysis process, FAA identifies critical needs that the agency must meet for improving the safety, security, capacity, efficiency, and effectiveness of the NAS. Approval of a mission need statement by the Joint Resources Council—FAA’s corporate decision-making body—signifies that the agency agrees that the need is critical enough to proceed to the next phase, investment analysis. During the investment analysis phase, teams of acquisition and program specialists (1) identify and analyze alternatives, (2) develop baselines and assesses affordability, (3) prepare an investment analysis report, and (4) recommend a preferred solution to the mission need. FAA then scores and ranks each proposed project based on a number of factors, including how well, relative to other projects, it meets mission needs and has a favorable cost-benefit ratio. Once a project is selected, life-cycle cost, schedule, benefits, and performance baselines are established in a formal document called the acquisition program baseline. The acquisition program baseline, which must be approved by the Joint Resources Council, is used to monitor a project’s status in achieving those baselines throughout the remaining phases of the acquisition management life cycle.

During the solution implementation phase, a multidisciplinary team develops and carries out an acquisition strategy for implementing the project. Once the project has been implemented and is in operation (the in-service management phase), the team monitors and assesses its performance, costs, and support trends; proposes fixes for any defects or other problems; incorporates product improvements; seeks new technology to enhance the capability or reduce costs; and identifies and prepares for decisions to correct capability shortfalls at the end of the project’s service-life. Finally, during the service-life extension phase, a determination is made about whether the current capability satisfies the demand for services or whether another solution offers the potential for improving safety or effectiveness or for significantly lowering costs. The team initiates a process whereby the mission need would be revalidated and the investment analysis process begun again, which could lead to a new investment decision. See figure 1.1 for a graphic depiction of FAA’s life-cycle acquisition management process.

FAA’s Acquisitions Are Funded From Two Major Budget Accounts

Funding for FAA’s investments is primarily provided through two of its budget accounts: (1) facilities and equipment and (2) operations. The facilities and equipment account covers the costs to develop, procure, and place the new equipment or facility in operation. Once the project goes into full operation, it is funded by the operations account, which covers the costs to support and maintain the new equipment or facility. Costs for planned product improvements and upgrades to the technology can be funded from either the facilities and equipment or the operations accounts.

Some investment funding is also provided through the research, engineering, and development account. This account is relatively small compared with the facilities and equipment and operations accounts. For example, in fiscal year 1999, $150 million was appropriated for research, engineering, and development; approximately $2.1 billion for facilities and equipment; and approximately $5.6 billion for operations.

Guidance Provides a Framework for Assessing Federal Agencies’ Information Technology Investment Decision-Making

Several recent management reforms—including the revision of the Paperwork Reduction Act and the passage of the Clinger-Cohen Act of 1996, the Government Performance and Results Act of 1993, and the Chief Financial Officers Act of 1990—have introduced requirements emphasizing the need for federal agencies to improve their management processes for selecting and managing information technology resources. GAO and the Office of Management and Budget have developed guidance to assist federal agencies in evaluating information technology investments. One such guide, Assessing Risks and Returns: A Guide for Evaluating Federal Agencies’ IT Investment Decision-Making, incorporates our analysis of the management practices of leading private and public sector organizations as well as of the provisions of major federal legislation (e.g., the Clinger-Cohen Act) and executive branch guidance that address investment decision-making. The guide outlines three phases of a successful investment management approach—selection, control, and evaluation. To help ensure that real, positive change is produced as agencies seek to improve their decision-making about their information technology investments, agencies need to (1) institutionalize management processes; (2) regularly validate the cost, benefit, and risk data used to support information technology decisions; and (3) focus on measuring and evaluating results. The guide provides a framework for evaluating and assessing how well a federal agency is achieving these goals and identifies specific areas where improvements can be made.

Many of the concepts in our Assessing Risks and Returns have been incorporated into the Office of Management and Budget’s Capital Programming Guide, which provides guidance on the planning, budgeting, acquisition, and management of different kinds of capital assets, including information technology. Our guide has also been endorsed by the federal Chief Information Officers Council. A third guide we prepared, Executive Guide: Leading Practices in Capital Decision-Making, summarizes 12 fundamental practices that have been successfully implemented by organizations recognized for their outstanding capital decision-making.Since information technology investments are a form of capital asset, this guide emphasizes many of the same concepts as the aforementioned guides, such as evaluating alternative approaches to achieving results; assessing investments as a portfolio; tracking projects’ costs, schedules, and performance; and conducting post-implementation reviews.

Objectives, Scope, and Methodology

Given the large expenditures required to carry out FAA’s modernization program, the past problems, and the continuing concerns about key projects funded under the program, the Chairmen and Ranking Minority Members of the Senate Commerce, Science, and Transportation Committee’s Subcommittee on Aviation and the House Transportation and Infrastructure Committee’s Subcommittee on Aviation asked us to evaluate the extent to which AMS provides a comprehensive approach for managing FAA’s modernization investments. This report addresses the extent to which FAA, through AMS, (1) has established a structured approach for selecting and controlling its investments; (2) incorporates all investments, including those in operation, in the agency’s portfolio; and (3) selects, controls, and evaluates its investments with complete and reliable information.

To address our objectives, we reviewed FAA’s overall approach for managing investments, as carried out through AMS, including the policies, procedures, and guidance for managing the life-cycle acquisition process. AMS is designed to include all of FAA’s acquisition projects, including those related to facilities, mission support, and information technology. Our review focused primarily on information technology projects, and we worked with FAA to identify the universe of such projects. Then, in concert with FAA, we selected five projects for review to obtain a more detailed understanding of how the investment management process is implemented at the project level. The five projects we reviewed were (1) the NAS Infrastructure Management System, (2) the Oceanic Automation Program, (3) the Operational and Supportability Implementation System, (4) the Standard Terminal Automation Replacement System, and (5) the Voice Switching and Control System. We selected these projects because they were in various stages of implementation when AMS was established in April 1996, were key to replacing the aging NAS infrastructure or improving its capacity and effectiveness, and represented significant expenditures. Total estimated facilities and equipment funding for each of these projects exceeds $100 million. Two of the five projects—the NAS Infrastructure Management System and the Operational and Supportability Implementation System—are subject to all AMS requirements because they were selected after AMS went into effect in April 1996. The other three projects, which were ongoing when AMS was established, are subject to AMS requirements for the remaining stages in their development and implementation.

As part of our effort, we applied the methodology prescribed in Assessing Risks and Returns. We used the questions in this guide to determine the extent to which FAA has decision-making and management processes and data in place to select information technology projects and systems, control these projects throughout their life cycles, and evaluate results and revise the processes based on lessons learned. We provided the questions in the guide to FAA officials and assessed their responses, along with supporting documentation, as a basis for determining whether FAA’s approach provided the necessary elements for managing its investments. We also incorporated the guidance from the Executive Guide: Leading Practices in Capital Decision-Making in determining whether FAA managed its investments as a portfolio; tracked projects’ costs, schedules, and performance; and conducted post-implementation reviews.

To gain an overall perspective on FAA’s investment management process, we interviewed FAA officials responsible for implementing and managing AMS. We also interviewed FAA officials responsible for preparing the facilities and equipment and operations budgets and reviewed the agency’s fiscal year 1999 budget justification documentation to understand how data on FAA’s investments are used to support FAA’s budget request to the Congress. Finally, we interviewed officials responsible for managing the five projects we reviewed to obtain their views on how AMS is applied at the project level as well as potential areas for improvement. We performed our work from April 1998 through April 1999 in accordance with generally accepted government auditing standards.

FAA’s AMS Is Designed to Provide a Disciplined, Structured Process for Selecting and Controlling Investments

Through AMS, FAA has designed and implemented processes that provide many of the key elements followed by leading organizations for selecting and controlling investments. AMS’ mission analysis and investment analysis processes are meant to provide FAA’s senior management with a basis for screening proposed projects; evaluating their relative costs, benefits, and risks; and selecting projects for funding based on their relative merits. Additionally, AMS policy requires each acquisition project to have an approved baseline that establishes the project’s life-cycle cost, schedule, benefits, and performance boundaries and that is intended to be used to monitor the project’s status in achieving those baselines.

FAA Has Established a Defined Process for Selecting Investments Funded Through the Facilities and Equipment Account

During the selection phase, leading organizations take a structured approach to determining priorities, screening and analyzing the relative merits of projects, and making decisions about which projects will be funded during the year. Such an approach builds on an organization’s assessment of where it should invest its resources for the greatest benefit over the long term. The Clinger-Cohen Act requires federal agencies to apply this sort of structured approach in deciding whether to undertake particular investments in information technology systems.

A starting point for the selection phase is the screening process in which projects being submitted for funding are compared against a uniform set of screening criteria and thresholds to determine whether the projects meet minimal requirements and to identify at what organizational level the projects should be reviewed. Next, the costs, benefits, risks, and mission focus of all the projects are assessed, and the projects are compared against each other and ranked or prioritized. In conducting their selection processes, leading organizations require all projects to have complete and accurate project proposals and justification information. Finally, a decision-making body of senior managers makes decisions about which projects to select for funding on the basis of mission needs and organizational priorities. The selection phase helps ensure that the organization selects those projects that will best support mission needs and identifies and analyzes each project’s risks and proposed benefits before a significant amount of funds is spent.

Through AMS, FAA has established two processes—mission analysis and investment analysis—that together constitute a set of policies, procedures, and guidance that are designed to enhance the agency’s ability to select investments. These two processes define what should be done and who should do it, what reports are required, who reviews and approves reports and processes, who makes corporate-level decisions, and the roles and responsibilities of those involved. AMS policy and guidance—which is on an easily accessible Internet Website—contain the procedures, process flowcharts, document templates, checklists, and other acquisition-related information needed for FAA’s project officials and senior management to understand and implement the selection processes.

Although AMS does not include an explicit screening step, screening activities are part of the agency’s mission analysis process. This process culminates in a mission need statement reflecting the Joint Resources Council’s decision that a high-priority, critical need exists and that the agency should go forward with a detailed investment analysis of proposed solutions to meet that need. During mission analysis, FAA’s operating divisions identify and quantify projected demand for and supply of services, capability shortfalls, and technological opportunities to meet those shortfalls, and summarize the major decision factors that the Joint Resources Council should evaluate in considering the need. The Joint Resources Council approves mission need statements.

Once a mission need statement is approved, AMS’ investment analysis process provides a set of detailed steps for evaluating the costs, benefits, and risks of alternative solutions and for selecting the best solution to meet the need. Under AMS, multidisciplinary investment analysis teams—comprising officials from the operating divisions and other acquisition and engineering specialists—assess each project proposed for funding to define the technical requirements; estimate the life-cycle costs, benefits, schedule, and risks; and determine the project’s affordability relative to other projects. As part of the investment analysis process, FAA scores and ranks each proposed project on the basis of defined criteria for how well—relative to other projects—it meets the agency’s mission objectives, whether it is of high priority to the organization sponsoring the project, whether it is consistent with the NAS architecture or provides critical administrative capacity for the agency, and whether it has a favorable cost-benefit ratio. AMS requires that the mission need be revalidated during the investment analysis and that the underlying analyses be documented through cost-benefit studies, risk assessments, and other documents.

After considering the results of the investment analysis process, the Joint Resources Council decides whether to proceed with a proposed project. This step is called the investment decision, and if the project is selected, the Council commits FAA to fully funding it. The project is then incorporated into the agency’s financial plan for projects funded by the facilities and equipment budget account.

AMS Requires That Projects’ Baselines Be Established and Controlled

Under AMS, FAA’s primary mechanism for controlling a project is the acquisition program baseline, which establishes a project’s life-cycle cost, schedule, benefits, and performance baselines and which is intended to be used to monitor a project’s status in achieving those baselines. The acquisition program baseline is supposed to be established when the investment decision is made for a project. The baseline, which is the agreement between the organizations within FAA that are acquiring and will use the project, sets the cost and schedule boundaries within which the project is authorized to proceed, defines the performance and benefits the project must achieve, and establishes the performance measurements for assessing the project’s success as it advances through its life cycle.

FAA’s ultimate goal is to establish an acquisition program baseline document for every acquisition project. When AMS was established in April 1996, however, the agency decided to establish two standards for baseline documentation to facilitate the preparation of baselines during FAA’s transition to AMS. One standard applies to those projects initiated after AMS and another, less detailed standard applies to those projects ongoing at the time that AMS was established. The post-AMS projects must have an acquisition program baseline that provides a full set of detailed information on each baseline element as well as a document called a parameter sheet that summarizes a subset of baseline information that is designated for the Joint Resources Council’s control and that is considered critical to assessing the project’s ability to satisfy mission need, achieve needed operational capability, achieve benefits, and meet the schedule requirements of interdependent programs. Projects begun before AMS was established are required only to have the parameter sheet. Table 2.1 summarizes the full set of baseline information contained in the acquisition program baseline and the subset of information contained in the parameter sheet.

At the agencywide level, FAA has a two-tiered process for monitoring projects’ estimated versus actual baselines. Under one process, known as the Integrated Baseline Establishment and Management (I-BEAM) process, the project officials and officials responsible for overseeing projects’ baselines monitor the projects and prepare reports. Project officials prepare a monthly status report that is supposed to analyze estimated versus actual results for the subset of baseline information that is monitored by the Joint Resources Council and summarized in the parameter sheets prepared for every project. Project officials must also inform the Joint Resources Council of any baseline violation or “breach” through a document called a baseline management notice and receive the Council’s approval for any change in the baselines. Additionally, FAA’s acquisition oversight officials prepare a monthly report that analyzes the status reports, baseline management notices, and other documents to identify baseline variances and report this information to the Joint Resources Council. Under a second process, the Acquisition Executive who heads the Council and other senior managers conduct acquisition reviews that are designed to periodically examine the status of the information in each project’s acquisition program baseline, along with assessing a project’s progress, risks, and other issues.

In addition to these two primary control processes, FAA also has a variety of other reporting systems for monitoring projects’ status, including executive-level metrics that analyze project performance, project-level reviews, and numerous contractor reports on costs, schedule, and technical performance. However, these various reporting systems do not analyze a project’s status in meeting all of the detailed baseline requirements contained in the acquisition program baselines and parameter sheets.

Conclusions

FAA has taken a positive step in establishing an investment management approach through AMS. Under AMS, FAA has developed structured processes for selecting and controlling its investments. AMS provides a defined set of policies, procedures, and reporting requirements that are designed to facilitate FAA’s efforts to analyze mission needs, identify alternative solutions to meet those needs, assess the solutions’ affordability, and establish and control a project’s performance once a solution is selected.

Lack of Oversight of the Operations Portion of Projects Prevents FAA From Managing Investments as a Complete Portfolio

FAA’s oversight of its investments is confined to new projects and those under development, limiting its ability to fully assess and make trade-offs between new and existing systems and preventing the agency from managing all of its investments as a complete portfolio. FAA’s oversight includes the agency’s processes for scoring and ranking investments prior to selection as well as its processes for establishing and monitoring financial baselines after a project has been selected. But this oversight is limited to projects that receive funding from the facilities and equipment budget account—that is, new projects and those under development—and excludes projects funded by the operations account. Moreover, the link between FAA’s investment management process and its budget process is limited to the facilities and equipment budget account, which has detailed information on investments that does not exist for the operations budget account. FAA has two ongoing initiatives that may improve its information on operations costs.

Leading organizations determine priorities and make decisions about which projects will be funded based on analyses of the relative costs, benefits, and risks of all the projects in their investment portfolios, including projects that are proposed, under development, and in operation. Such a portfolio approach allows the organizations to evaluate the relative merits of spending funds to develop new systems, enhance current systems, or continue operating and maintaining existing systems.

FAA’s Selection Process Does Not Address Operations-Funded Investment Projects

FAA’s process for scoring and ranking projects prior to selection is applied to proposed projects and those under development that will receive facilities and equipment funding, not to existing systems that are funded from the operations budget account. In contrast, leading organizations include all types of information technology projects in their selection process to create a complete strategic investment portfolio. By analyzing the entire portfolio, the senior managers of any organization can examine the costs of maintaining existing systems versus investing in new ones; comparatively rank projects based on expected costs, benefits, and risks; and reach decisions based on projects’ overall contribution to the most pressing organizational needs. FAA has not included operations-funded projects in its scoring and ranking process because, as discussed below, the agency lacks reliable data on actual operations costs for existing systems and projections of operations costs for new projects.

FAA Does Not Have a Complete and Sound Operations Cost Baseline for Its Investments

While FAA’s policy requires baseline cost estimates for the full life cycles of all projects under AMS, in practice, FAA has not yet developed a sound estimate of the operations costs for each of its projects. As of February 1999, the agency had developed operations cost projections for only 26 of the 70 projects identified by FAA as requiring an acquisition program baseline or parameter sheet under AMS. For existing systems already in the operations phase when AMS was implemented in April 1996, FAA also lacked information on life-cycle cost projections for operating the systems in the future or actual operations costs incurred for each of these systems. Moreover, FAA officials throughout the organization indicated that the estimates of operations costs that have been developed for projects are not reliable. Given the lack of data on operations costs, FAA’s financial plan for its modernization effort reports only costs funded by the facilities and equipment budget account, omitting the operations costs associated with its investments.

FAA has long recognized the need to project the operations costs of its projects over a multiyear period. In March 1993, FAA issued guidance under its predecessor acquisition system that required an annual operations plan to support long-range resource allocation planning. This plan—which was never prepared—was supposed to summarize the operations and support funding requirements of modernization projects and other acquisitions funded by the operations appropriation. The operations plan was supposed to contain a financial baseline that identified detailed operations requirements over a 5-year period and general requirements for an additional 10 years.

FAA officials told us that the agency lacks the information needed to reliably estimate operations costs over a project’s life cycle or to track actual operations costs against estimates because it does not have a cost accounting system. In January 1997, we reported that FAA lacked reliable cost estimating processes and cost accounting practices needed to effectively manage its investments in information technology. We concluded that, as a result, the Congress does not have reliable cost information to use in making decisions about FAA’s billion-dollar modernization investments.

FAA has two initiatives that it believes will improve data on operations costs. In August 1998, FAA formed an operations baseline team to address agencywide concerns about the quality of its estimates of the operations costs for modernization projects and the lack of integration between the facilities and equipment and operations budget accounts. This team is evaluating FAA’s current processes for estimating and reporting on operations costs, assessing the validity of operations cost data on a sample of projects, and exploring ways to improve the estimating process. The team, which expects to report its findings in May 1999, plans to develop a 10-year operations cost baseline for selected projects and to recommend revisions to the budget formulation process that will allow FAA to budget for the operations costs of both new and existing systems and to better address the interrelationships between the facilities and equipment and operations budget accounts.

FAA is also developing a cost accounting system that it believes will provide more reliable information on actual operations costs; however, FAA has missed its initial milestones for completing the new system’s design and generating improved operations cost data. According to officials responsible for the new cost accounting system, the agency had planned to begin accumulating data for domestic and oceanic air traffic services by October 1998. FAA officials indicated that they underestimated the complexity of developing the system and that they now expect to accumulate data for air traffic services by April 2000 and to fully implement the system by April 2001.

FAA’s Operations Budget Justifies Only a Small Portion of Its Spending for Investments

While FAA’s budget provides detailed analyses of the actual and projected costs for each of the projects funded by the facilities and equipment budget account, it provides very little project-level detail in its justification for the operations budget account. In its fiscal year 1999 President’s budget request, for example, FAA justified only 5 percent, or $295 million, of its total $5.6 billion operations budget account. This $295 million request—which represented the incremental increase over the prior year’s appropriation—contained information on all of the activities to be funded in a given year, including pay increases for the current staff, costs of hiring new staff, training, accident investigations, and other activities.

Among the details provided for the fiscal year 1999 operations budget account, FAA provided project- or system-level justifications for about 1 percent, or $62 million, of its total $5.6 billion operations request. This information, known as the “NAS Handoff” portion of the operations account, contains 1-year estimates for new equipment, systems, and facilities that were initially acquired with facilities and equipment funding but will now be funded under the operations budget account. The NAS Handoff contains details on a variety of projected operations costs, including controller overtime, logistics, systems maintenance, leased communications, and flight inspections and procedures.

One factor contributing to the different levels of detail for the two budget accounts is the lack of reliable data on operations costs for individual projects or systems. Also, FAA officials told us that they only justify incremental increases over the prior year’s operations budget account because that is traditionally all that the Congress requires of FAA in preparing its budget justifications.

Conclusions

AMS’ lack of oversight of the operations portion of FAA’s investments impedes the agency’s ability to rigorously assess and manage all of its modernization projects as a complete strategic, investment portfolio and to make sound decisions about continuing, modifying, or canceling projects. Excluding operations projects from its selection process prevents FAA from considering the relative merits of existing systems when deciding which projects to fund each year.

Recommendation

We recommend that the Secretary of Transportation direct the Administrator of FAA to establish a complete portfolio of investments—including existing systems funded by the operations budget account as well as projects funded by the facilities and equipment account—and to require the Joint Resources Council to periodically review the baseline status and merits of each of these investments throughout their entire life cycles. As part of this portfolio, cost baselines for operating and maintaining all projects should be developed, and this information should be included in the agency’s financial plan for its investments and in its annual budget request to the Congress.

Agency Comments

FAA agreed with our recommendation.

Weaknesses in the Selection, Control, and Evaluation Phases Limit FAA’s Effectiveness in Managing Its Investments

AMS has weaknesses in all three investment management phases—selection, control, and evaluation—that limit FAA’s ability to effectively manage its modernization investments. First, the information used to select projects is not validated to ensure quality control, and critical cost information used to support selection decisions is of questionable reliability. Second, FAA has not fully implemented an effective process for controlling the baselines for the cost, schedule, benefits, performance, and risks of its investments. Third, FAA lacks a post-implementation evaluation process for assessing projects’ outcomes and feeding lessons learned back into the selection and control phases for future projects. Finally, FAA has not fully implemented a standardized management information system for capturing and maintaining consistent, reliable, and easily accessible data on investments.

Weaknesses in Some Supporting Data Limit the Effectiveness of the Selection Process

AMS’ processes for selecting investments contain the key elements that leading organizations follow to ensure the selection of projects that enhance mission performance and that are cost-effective; however, weaknesses in some of the data used to support the selection processes limit AMS’ effectiveness. AMS’ mission and investment analysis processes provide FAA’s senior management with a basis for screening proposed projects; evaluating their relative costs, benefits, and risks; and selecting projects for funding based on their relative merits. But the cost information used to make selection decisions is of questionable reliability, and there is little evidence that the data or underlying analyses used in the selection process are validated to ensure accuracy, completeness, and appropriateness. As a result, FAA’s managers cannot be assured that they have all of the information needed to make sound selection decisions.

Cost Data Are of Questionable Reliability, Though Improvements Are Under Way

Consistently producing reliable cost estimates for projects requires defining institutional processes for deriving estimates and measuring actual performance against these estimates. However, the cost data used in FAA’s selection process are of questionable reliability. The five projects we reviewed prepared their cost estimates using techniques and data that we criticized in our January 1997 report. We found that FAA’s modernization program’s cost estimating processes do not satisfy recognized standards and that the agency does not have a cost accounting system capable of reliably accumulating full cost information for projects. FAA’s cost estimating techniques do not satisfy recognized standards that call for organizing and retaining cost information on projects in a historical database and using cost models that are calibrated and validated on the basis of actual experience. FAA’s processes allow each project to approach cost estimating in whatever manner its estimators choose. The result is inconsistency in the rigor and discipline with which cost estimates are derived, which in turn means estimates vary in their degrees of reliability. For example, of the six projects reviewed in our 1997 report, two were too poorly documented to permit any comparative analysis, and none of the remaining four satisfied all the recognized standards. Compounding the estimating weaknesses is FAA’s practice of presenting cost estimates as precise point estimates, thus failing to disclose the estimates’ inherent uncertainty and risks. Moreover, the effectiveness of FAA’s cost estimating processes also relies heavily on the quality of projects’ actual cost information, but FAA does not have a cost accounting system for capturing and reporting the full costs of its projects. Consequently, FAA cannot reliably use information about actual cost experiences to improve its future cost estimating efforts. We recommended that FAA institutionalize defined cost estimating processes that include, among other items, a historical database and structured approaches and tools.

In response to our 1997 report, FAA is developing a cost estimating process for its projects that is intended to satisfy recognized estimating standards; drafting guidance on reporting projects’ cost estimates as ranges rather than precise point estimates and, in fact, reporting ranges on some systems; and developing a cost accounting system. Additionally, FAA has developed a document, known as a standard work breakdown structure,which provides a good first step toward the development of a historical database on costs. FAA officials also indicated that they are completing a cost estimating handbook that contains a detailed discussion of cost estimating practices. When completed, this handbook should contribute to improving FAA’s approach to estimating projects’ costs. However, it does not require a disciplined process for estimating costs throughout the agency, and the draft handbook acknowledges that FAA still needs to develop sophisticated tools and a historical database to advance its cost estimating processes. FAA has not established firm deadlines for completing the handbook or the other tasks related to cost estimating. As for the cost accounting system, as we discuss earlier, FAA officials underestimated the complexity of developing the system and found that their implementation milestones were unrealistic. The agency plans to fully implement the cost accounting system by April 2001.

Selection Data Are Not Validated

Leading organizations validate the information and analyses submitted in a new project proposal, which helps to ensure that all the information is up-to-date, cost numbers are accurate, benefits are quantified to the extent possible, alternatives are identified, underlying assumptions are reasonable, and sensitivity analyses are conducted. Validation is also important for ensuring that a project’s risks are identified, that the impact of risks on the project’s outcomes is quantitatively or qualitatively projected, and that risk mitigation strategies are explained. Explicit verification and validation steps sensitize decisionmakers to important factors that have a bearing on projects’ actual outcomes, enhance accountability, and decrease the likelihood that project proposals will contain analyses that are based on inaccurate or incomplete data or faulty assumptions.

FAA has not completely defined the requirements for the validation process under AMS, and it is not fully carrying out validation activities. First, AMS does not require the validation of all data used in the selection process. Under AMS guidance, the FAA organization charged with carrying out the investment analysis phase is responsible for validating only the cost and schedule data, not the performance, benefits, or risk analyses used as part of the selection process. Second, AMS guidance does not specify what steps should be taken in validating selection data, nor does it require documentation of the results and resolution of the validation process. Finally, our review of the two projects for which investment decisions had been made under AMS indicated that FAA is not fully carrying out validation activities. For one project, FAA did not provide any documentary evidence of the validation efforts that were performed. For the other project, FAA provided evidence of a validation review of the cost-benefit analysis, but most of the results of that review were not incorporated into the final version of the analysis used to support the investment decision.

FAA Has Not Fully Implemented Its Process for Establishing and Tracking Key Project Baselines

To control its projects at the agencywide level, FAA relies on periodic reviews of each project’s acquisition program baseline, which, as noted earlier, is a document that establishes a project’s cost, schedule, benefits, and performance boundaries and is intended to be used to monitor a project’s status in achieving those baselines. This document is incomplete, however, because its schedule baseline does not include any milestones for project reviews during the operations phase of a project and because it does not address a project’s risks. Moreover, FAA has not completed about half of the baselines for the projects or project segments that require baselines. Additionally, the agency-level processes for tracking actual baseline performance against estimates frequently provided incomplete information on projects’ costs, schedules, benefits, and performance. Finally, the investment control group of senior managers, the Joint Resources Council, is not actively involved in monitoring all projects after the investment decisions are made. As a result, FAA’s senior managers lack key information needed to make sound decisions about the future of each project.

Leading organizations maintain a cycle of continual control and monitoring after a project has been selected. Senior executives review a project at specific milestones as the project moves through its life cycle and as the dollars spent on the project increase. At these milestones, the executives compare the expected costs, risks, and benefits of earlier phases with the actual costs incurred, risks encountered, and benefits realized to date. During the control phase, senior executives determine whether projects should be modified, continued, accelerated, delayed, or terminated. The Clinger-Cohen Act also stresses the importance of consistently monitoring the progress of federal investments in information technology in meeting cost, schedule, and performance objectives.

FAA’s Requirements for Baseline Estimates Include Most, but Not All, Key Project Parameters

Our review indicated that FAA’s acquisition program baseline is designed to capture sufficient information on most of a project’s key baseline elements, except for two limitations in the areas of schedule and risk. One limitation is that the schedule baseline does not address the operations phase of the project. At leading organizations, even after a project has been implemented, senior managers regularly review how well the acquired system meets organizational needs, including whether it needs unexpected modifications or premature replacement to meet emerging needs. These reviews are used to make decisions pertaining to the retirement or replacement of systems. In addition, because operations activities—such as hardware upgrades, system software changes, ongoing training, and maintenance costs—can consume a significant level of resources, a plan for the review of each project should be developed and periodically reevaluated. Given that AMS is designed to track projects during their entire life cycles, we expected to see one or more milestones in the schedule baseline for a project review during the operations phase of FAA’s information technology investments, which can last as long as 15 years. Such milestones would allow the Joint Resources Council to review projects periodically during the operations phase to determine whether expected performance requirements and benefits are actually being achieved in a cost-effective manner and to evaluate the need for upgrading or enhancing the technology.

The second limitation is that the acquisition program baseline does not include a project’s expected risks. The Office of Management and Budget’s guidance requires that federal agencies estimate a project’s risks and develop a strategy for mitigating those risks. While FAA performs a risk analysis as a part of the investment analysis phase, the agency does not systematically monitor each project’s risks during the control phase. Given that the acquisition program baseline is FAA’s primary mechanism for controlling projects, we expected it to include a baseline assessment of risks that could be used to monitor mitigation and resolution of actual risks that occur during a project’s life cycle. Although all five projects we reviewed had discussions of risk issues during their project reviews, none of them presented any systematic assessment of estimated versus actual risks because risk is not a required element of the acquisition program baseline.

Requiring an assessment of a project’s risks as an element of the acquisition program baseline and systematically monitoring those risks would allow FAA to identify “red flag” issues that may have an impact on a project’s cost, schedule, and performance. For example, two of the projects we reviewed—the Standard Terminal Automation Replacement System and the NAS Infrastructure Management System—have experienced difficulties with acquiring or developing software. FAA promoted both of these as projects that would use commercial off-the-shelf software that would require very little software development. For both projects, FAA underestimated the lines of software code that needed to be developed or modified, and as a result, the costs for the projects have increased and the schedules have been delayed. FAA has historically had difficulty acquiring software—the most costly and complex component of information technology systems—and the agency has initiated some efforts to improve its software acquisition processes. Improvements in these processes, coupled with an identification of the risks associated with software acquisition for new projects, would help FAA better manage such risks.

FAA Has Approved Half of Its Required Baselines, and Some Baseline Documentation Is Incomplete

FAA has identified 70 modernization projects or project segments that require baseline documentation. These 70 projects and segments account for $1.266 billion, or 55 percent, of the $2.319 billion appropriation FAA requested for fiscal year 2000 for all modernization activities funded by the facilities and equipment budget account. Most of the modernization activities that do not have baselines involve improving or sustaining ATC facilities and other buildings; providing technical support services; or funding personnel, compensation, benefits, and travel costs.

Of the 70 projects and segments, 10 involve projects that are currently in the investment analysis phase of AMS and, hence, do not yet require an approved baseline. Thus, 60 projects and segments that account for $1.205 billion in fiscal year 2000 estimated costs, currently require a baseline. Of the 60, half have approved baseline documentation (either acquisition program baselines or parameter sheets). These 30 projects and segments account for $619.4 million in estimated costs for fiscal year 2000. Table 4.1 shows the status of FAA’s efforts to develop AMS baseline documentation on the 60 ongoing projects and segments.

Our review of five projects found that some of the baseline documentation (both acquisition program baselines and parameter sheets) is incomplete. Of the five projects, two had acquisition program baselines (plus parameter sheets), and three had parameter sheets only. Only the performance element was fully documented on all of the projects, while the cost, benefits, and schedule elements were missing some of the information required by AMS. For the cost baseline, for example, two of the five projects did not estimate operations costs, and one of the two projects with acquisition program baselines did not provide information on the detailed cost elements required by AMS guidance. For the benefits baseline, one of the two projects with acquisition program baselines did not identify the measurements that would be used to determine whether a benefit had been achieved. For the schedule baseline, none of the projects estimated the in-service decision, a milestone required by AMS that specifies when the newly acquired system or equipment is certified ready for operational use.

FAA Is Not Completely Monitoring Projects’ Actual Performance Against Their Baselines

Our review and FAA’s internal evaluation of acquisition reform found that the I-BEAM and acquisition review processes frequently provide incomplete reporting on projects’ estimated versus actual performance in the areas of cost, schedule, benefits, and performance. We reviewed the monthly status reports on the five projects and found information was missing on all of the baseline elements. For example, none of the monthly status reports on the five projects analyzed operations costs; assessed estimated versus actual benefits, even though benefits were projected for fiscal years 1997 or 1998 on four of the five projects; or contained information on the performance requirements outlined in the acquisition program baseline or parameter sheet to confirm whether the original baseline requirements still applied. Our results are consistent with those of FAA’s internal evaluation, which found that the cost, schedule, and performance data in the monthly status reports were generally inconsistent with the estimates in the acquisition program baselines and parameter sheets. With regard to FAA’s tracking of baseline information through the acquisition review process, our review of the five projects showed that information was missing for all of the baseline parameters, as shown in table 4.2.

FAA officials managing the five projects we reviewed cited several factors to explain the lack of complete baseline data on their projects. First, project officials told us that frequent budget reductions—imposed either by the Congress or by FAA—make it very difficult to establish a stable baseline and to monitor a project’s performance against that baseline. Second, AMS guidance on the acquisition program baseline requirements has evolved over time, and project officials told us that detailed guidance was not available to projects that established their baselines during the first year of AMS’ implementation. Finally, the two mechanisms used by FAA to monitor projects’ status—the monthly status report and the acquisition reviews—were developed prior to AMS and thus do not always address the specific baseline elements that are supposed to be contained in the acquisition program baselines, report the current status of all baseline estimates, or report the actual deviations from those estimates.

We recognize that variances in funding levels exist and, in some cases, have had an impact on FAA’s ability to manage its projects. However, we and others have found that FAA’s problems with baseline parameters have resulted from factors other than funding. Although AMS’ detailed guidance and reporting requirements are evolving, establishing complete acquisition program baseline documentation that has been approved by the Joint Resources Council helps to ensure that all projects are being held accountable to the cost, schedule, benefits, and performance baselines established when the Council made the initial investment decision. Furthermore, the absence of complete, up-to-date data on estimated versus actual results means that FAA has little assurance that its estimates of projects’ costs, schedule, benefits, performance, or risks are sound and accurate or that projects will be managed so that they meet the agency’s expectations. This, in turn, restricts the ability of FAA’s senior management to make sound decisions about continuing the agency’s investments.

FAA’s long-standing problems with implementing projects that meet their cost, schedule, and performance objectives illustrate the need for the agency to better manage its baselines. As noted earlier, over the past 17 years, FAA’s modernization projects have experienced substantial cost overruns, lengthy delays, and significant performance shortfalls, problems that have persisted since the implementation of AMS in April 1996. Two of the projects in our review provide examples of continuing cost and schedule problems. In the case of the Standard Terminal Automation Replacement System, although FAA has not officially changed the baseline approved in February 1996, the baseline is in jeopardy of being breached because of unions’ concerns about human-factor and design issues, the refinement of the project’s requirements, and the interjection of a new project phase. FAA estimates that these issues have the potential to increase the project’s costs from $294 million to $410 million above the approved baseline. FAA also estimates that the project’s initial completion date could be delayed by almost 2-1/2 years. Similarly, in the case of the NAS Infrastructure Management System, the agency has not officially changed the baseline that was approved in March 1997. However, the project’s leader expects significant baseline breaches to occur, including a 58-percent increase in the costs from $100.8 million to $159.5 million, and a 123-percent increase in the schedule from 48 to 107 months for the project’s total duration.

Joint Resources Council Has Limited Involvement in Reviewing Projects After the Investment Decisions

Under AMS, the Joint Resources Council conducts reviews and makes decisions on each project funded from the facilities and equipment account, from the determination of mission needs to the point at which an investment decision is made. After the investment decision, the Joint Resources Council generally only becomes directly involved in monitoring a project when there is a potential or actual breach of the established baseline. As a result, the Joint Resources Council is not proactively involved in making decisions about the future of most projects when they are being developed, deployed, operated, and maintained; when decisions are being made about technology upgrades or other enhancements; and when assessments are being made about whether the system or product is achieving the intended benefits and meeting the expected performance requirements.

The Acquisition Executive, who chairs the Joint Resources Council, told us that he highlights projects whose baseline status is problematic according to data in the monthly status report and that he provides additional oversight for those projects. However, as we have stated, the data contained in the monthly status reports are incomplete and generally inconsistent with the estimated baseline elements in the acquisition program baselines and parameter sheets.

FAA’s AMS Lacks a Post-Implementation Review Process for Evaluating Investments

FAA does not have a defined, documented process for conducting post-implementation reviews of projects to assess their performance and to improve the selection and control of its other investments. FAA performs some elements of a post-implementation review in its life-cycle management review process, including such tasks as the independent operational test and evaluation of some projects prior to their deployment, operational performance monitoring, customer satisfaction surveys, and periodic reviews throughout an investment’s life cycle. However, this process is not standardized and is not required for all projects. As a result, there is no evidence that changes, especially to the selection and control phases, are being implemented based on lessons learned. Although FAA has not yet designed or implemented a post-implementation review process for individual projects, its fiscal year 1999 performance plan for the Research and Acquisitions operating division includes a new requirement for a post-deployment assessment of NAS modernization systems.

The evaluation phase “closes the loop” on the investment management process by comparing actual results against baseline estimates to assess performance and identify areas where future decision-making can be improved. Lessons that are learned during the evaluation phase can be used to improve future decisions about selecting and controlling projects. Central to this process is the post-implementation review, with its evaluation of the historical record of a project and its comparison of actual versus expected costs and benefits. Recognizing the importance of this phase of investment management, the Clinger-Cohen Act requires federal agencies to evaluate the performance of information technology projects and to use that information to decide whether to continue, modify, or terminate projects.

At leading organizations, this review generally occurs about 3 to 12 months after a project has reached its end point (i.e., the point at which the project has been fully implemented or canceled) and is generally conducted by a group other than the project team to ensure that the review is independent and objective. In conducting post-implementation reviews, an organization can survey customers to determine users’ satisfaction with the completed product and how well the project supports business processes; assess whether the investment has had its intended impacts on mission goals, cost savings, compliance with the system’s architecture, and other issues involving information accuracy, timeliness, adequacy, and appropriateness; and evaluate current and future technical issues associated with the investment.

FAA’s Efforts to Implement an Agencywide Management Information System Do Not Include Key Selection Data

Until recently, FAA lacked a centralized, standardized management information system or historical database for capturing and maintaining project information. While FAA had a number of stand-alone databases within different groups, none provided a complete picture of estimates, the assumptions that made up the estimates, revisions, and actual performance on projects. As a result, agency officials had no assurance that the project data from these stand-alone systems were complete, accurate, and up-to-date.

In March 1999, FAA officials began implementing a centralized repository and management information system under AMS—called the Simplified Program Information Reporting and Evaluation (SPIRE)—to provide access to projects’ baselines and other information. This system consolidates project information related to FAA’s processes for managing key baselines, including data from the acquisition program baselines and parameter sheets, monthly status reports, baseline management notices, meeting minutes from the acquisition reviews, Joint Resources Council decisions, and other baseline information. The system does not, however, contain key information from the selection process, such as mission need statements, cost-benefit analyses, risk assessments, or other required reports. FAA plans to implement SPIRE in three phases. Phase I, which commenced in March 1999, provides the capability to store and display the status and variance reports that have been input by project leaders. Subsequent phases, which do not yet have implementation dates, will focus on automatically generating various reports on projects’ baseline status and variances.

Informed management decisions can be made only if accurate, reliable, and up-to-date information from all phases of the investment management process is included in the decision-making process. To do this requires that agencies have a uniform mechanism—that is, a management information system with uniform data standards and entry procedures—for collecting, automating, and processing data on projects’ expected versus actual outcomes. Data in this system should include the initial cost, schedule, benefits, performance, and risk estimates that were developed during the selection process. Various analyses that were conducted to initially justify the project, along with revised estimates, reasons for revisions, and actual performance measured against the estimates, should also be included. These data need to be continually updated as each project’s implementation continues and as expenditures increase. The data also need to be easily accessible to both the project team and senior managers.

A management information system, if kept accurate and up-to-date, can make data verification and validation easier by allowing an organization to track costs, risks, and other factors over time. It is also essential from the standpoint of establishing an organizational memory throughout the selection, control, and evaluation phases of the investment management process. As such, it can be used to help assess whether projects are still aligned with mission needs and organizational objectives, determine whether projects are meeting planned performance goals, and identify possible revisions to the overall investment management process based on previous experiences and lessons learned.

FAA’s efforts to consolidate project baseline information in SPIRE are a positive step toward improving the agency’s mechanisms for tracking projects and helping to ensure the consistency of information and reporting on all the projects in its investment portfolio. However, SPIRE provides an incomplete record of project information since it does not include key information from the selection process.

Conclusions

AMS has weaknesses in its selection, control, and evaluation phases that impede FAA’s ability to manage its investments effectively and to make sound decisions about continuing, modifying, or canceling projects. First, using data that have not been validated and unreliable cost information reduces the likelihood that FAA’s managers can make informed decisions about the relative merits of competing investments. Second, requiring FAA’s senior managers to stay actively involved in the process of controlling project baselines and providing them with complete information on all projects is critical to monitoring how well projects are achieving their intended results. Third, establishing post-implementation reviews is essential for evaluating projects’ performance and identifying areas for which future decision-making could be improved during the selection and control phases. Finally, establishing a standardized management information system that includes complete information from all three investment management phases—selection, control, and evaluation—will help facilitate FAA’s efforts to track projects’ costs, risks, and other factors over time, providing senior managers with uniform, accurate, reliable, and easily accessible data on all the projects in the agency’s portfolio. Taking steps to correct these weaknesses increases the likelihood that FAA’s projects will meet established cost and schedule objectives and contribute to measurable improvements in the agency’s mission performance.

Recommendations

We recommend that the Secretary of Transportation direct the Administrator of FAA to take the following actions: Improve the selection process by (1) establishing clearly defined procedures for validating projects’ cost, schedule, benefit, performance, and risk information and (2) requiring documentation of the results of the validation procedures applied to each project.

Strengthen control over investments by (1) revising the acquisition program baseline requirements to include project risks and to add milestones for project reviews during the operations phase and (2) ensuring that project officials fully track and document estimated versus actual results on all the elements (i.e., costs, schedule, benefits, performance, and risks) contained in the baseline documentation. Initiate post-implementation evaluations for projects within 3 to 12 months of deployment or cancellation to compare the completed projects’ costs, schedule, performance, and mission improvement outcomes with the original estimates.

Incorporate key information from the selection process (e.g., mission need statements, cost-benefit analyses, and risk assessments) into FAA’s management information system for investments.

Agency Comments

FAA agreed with our recommendations.