Air Traffic Control: FAA's Acquisition Management Has Improved, but Policies and Oversight Need Strengthening to Help Ensure Results
Released: 2004-12-13T00:00:00

Letter

Background

Maintaining that federal procurement requirements contributed to some of its cost, schedule, and performance problems in the 1980s and early 1990s, FAA sought a statutory exemption from the federal acquisition system, including the FAR, and those parts of title 5 of the United States Code, parts II and III, that govern federal civilian personnel management. According to FAA, exemptions from these requirements would enable it to streamline its acquisition processes, be more responsive to the airline industry’s needs, and increase the efficiency of ATC operations while maintaining safety. Congress enacted legislation in November 1995 that exempted FAA from key federal procurement statutes and the FAR, and directed FAA to develop a new acquisition management system. In response to these legislative initiatives, FAA implemented a new, streamlined acquisition process—the Acquisition Management System (AMS)— on April 1, 1996.

We developed a knowledge-based model of commercial best practices based on our findings about how leading private firms manage costly and complex acquisitions effectively—that is, within cost, schedule, and performance targets. The use of this knowledge-based model has been found to reduce the risks associated with developing products and increase the likelihood of successful outcomes. The model divides the product development cycle into four phases and related activities. Table 1 presents these phases and activities and explains what takes place during each.

AMS provides guidance for selecting and overseeing investments over their life cycle. Like our best practices model, it is divided into phases and activities, although the divisions sometimes occur at different points. Table 2 summarizes AMS’s phases and activities.

To implement the new, performance-based organization for managing ATC modernization and operations, as the President and Congress directed in 2000, FAA appointed a chief operating officer in August 2003 and formally established the Air Traffic Organization (ATO) in February 2004. ATO, under the direction of a six-member executive council, is now responsible for further implementing acquisition reforms for major ATC systems.

AMS Is Broader and Less Prescriptive Than the FAR

AMS establishes an acquisition life-cycle management system that encompasses both contracting and program management, whereas the FAR is primarily a contracting system that focuses on contract formation and contract administration. As a result, AMS is broader in scope than the FAR. See figure 1. In addition, AMS takes the form of guidance. This guidance is expressed in documentation of FAA policy, handbooks, templates, flowcharts, forms, and standard contract language. It is not regulatory. By contrast, the FAR is a set of published regulations—a legal foundation that has the force and effect of law for the federal agencies that are required to follow it. Furthermore, the FAR is more detailed and prescriptive in establishing contracting requirements and can require more administrative involvement. This fundamental difference between AMS and the FAR may suggest to some that AMS is more flexible. FAA personnel can choose how to apply AMS’s provisions to a major acquisition. Nonetheless, procurement officials under the FAR also have flexibility because the FAR encourages innovation consistent with its direction (and other applicable legal requirements), provides a wide selection of contracting solutions, and permits contracting officials to choose the methods that they consider most suitable for a given situation.

AMS Addresses Both Procurement and Project Management, Whereas the FAR Focuses Primarily and in Far Greater Detail on Procurement

AMS comprises six policy sections and five appendixes. The procurement policy section of AMS covers a range of topics, including contract funding and administration, contracting with small and disadvantaged businesses, and compliance with labor laws. According to this section, competition is FAA’s preferred method of contracting, but single-source contracting is permitted when appropriate to fulfill the agency’s mission. This policy section also describes the procurement of commercially available or nondevelopmental items.

Other sections of AMS cover project management tools that the FAR does not address, such as investment analysis, configuration management, and integrated logistics support. AMS also addresses areas that fall outside project management and procurement, including real property management—an area that becomes important when FAA must lease or purchase real property so that it can install ATC systems such as radars or antennas on property that it does not currently own. FAA’s policy directs FAA staff to “conduct this business in a fair and equitable manner following best practices.”

Although the FAR includes requirements that address procurement planning and major systems acquisition, it does so only in the context of government procurement policy and procedure. Agencies subject to the FAR find the broader program planning requirements, which appear in AMS but not in the FAR, in documents such as the Office of Management and Budget’s Circular A-109 and in their own planning guidance. For example, DOD has issued a series of directives and instructions on this subject.

The contracting procedures set forth in section 3 of AMS do not prescribe detailed contracting procedures for various categories of procurements, as do those detailed under the FAR. Instead, AMS provides two basic contracting models for obtaining products and services through FAA’s contracting process. The first model is called “Complex and Noncommercial Source Selection” and is used for complex, large-dollar, developmental, noncommercial items and services. This is the model that typically would be used for investments approved by the Joint Resources Council. The second model is called the “Commercial and Simplified Purchase Method” and is typically used for commercial items that are less complex and less costly. Procurements of such products or services may be routine in nature and are generally purchased on a fixed-price basis. Generally, source selection under AMS follows a screening process, with the awardee being selected on a “best value” basis from among those who remain in consideration when the selection is made.

AMS Provides Broad Guidance While the FAR Establishes Detailed Requirements, but Managers Have Flexibility under Both

AMS sets out a nonregulatory FAA policy that is binding on FAA personnel as FAA employees. AMS also sets out other guidelines that FAA states should be followed unless there is a rational basis for doing otherwise. AMS is subject to such internal controls and enforcement as the Administrator decides and to general overarching legal requirements, such as the Government Performance and Results Act of 1993 (GPRA).FAA has also deemed certain acquisition laws applicable to its procurements (sometimes with modifications), such as the Service Contract Act. There is also a legal requirement, created by the 1995 legislation exempting FAA from the FAR, that small and socially or economically disadvantaged firms be given all reasonable opportunities to receive contract awards. FAA has adopted a dispute resolution process with some legal underpinnings. Otherwise, as the preface to AMS states, “nothing in this document creates or conveys any substantive rights.” In short, although FAA is subject to the general legal requirement that government decisions cannot be arbitrary or capricious, AMS does not establish regulatory requirements for the conduct of procurements and does not create or convey substantive legal rights.

In contrast to AMS, the FAR is a set of published regulatory requirements. It has the force and effect of law, and agencies that are subject to it are bound to follow it. The FAR’s requirements provide for a range of procurement strategies and approaches. In addition to negotiated procurement methods, it allows two-step sealed-bid and two-phase design-build methods, among others. It includes streamlined procedures for soliciting and evaluating offers to furnish commercial items, as well as permits the use of simplified acquisition procedures in a broad range of procurements. Furthermore, the FAR supports a diverse selection of available contract types, product- testing tools, and other tools that an agency’s contracting personnel may select when conducting an acquisition to meet the agency’s needs.

Although contracting personnel in agencies subject to the FAR are required to comply with it, they enjoy broad discretion in their management of procurements. For example, the FAR allows wide latitude in drafting requirements statements, from performance-based statements of work to design specifications as necessary. It allows broad discretion in framing solicitations and in conducting procurements, including scoring proposals, determining how negotiations will be conducted, eliminating firms whose proposals are not competitive, and selecting the awardees whose proposals afford the government the best value when evaluated against the selection criteria established in the solicitations.

Because AMS consists of broad guidance while the FAR comprises detailed and prescriptive regulatory requirements, FAA managers view AMS as giving them more flexibility than they would have under the FAR, particularly in two areas—competition and oversight. Whereas the FAR generally requires full and open competition, AMS calls for providing “reasonable access to” competition to firms interested in obtaining contracts—a less rigorous standard than full and open competition. AMS further states that the “preferred” method of selecting sources is to compete requirements among two or more sources. By contrast, full and open competition requires that all responsible sources be permitted to compete. Under AMS, there is no policy that firms that want to participate actually get a chance to do so. Rather, FAA told us that its system is beneficial because the agency can use screening requests to preselect competing firms, eliminating those firms that FAA believes are not likely to receive an award. The following example illustrates the differences between AMS and the FAR in their respective requirements on exceptions to competition. FAA may contract with a single source when this approach is determined to be in the best interest of FAA. The FAR, however, allows exceptions to full and open competition only for certain specified conditions (such as unusual and compelling urgency or the availability of only one source). The FAR describes in detail the circumstances of these conditions and the requirements for using them as justification for not providing for full and open competition. The FAR also requires the contracting officer to prepare a justification document that must generally be approved by higher-level agency procurement officials (up to the agency’s senior procurement executive) depending on the estimated dollar value of the procurement. The content of this justification is prescribed by the FAR. When not providing for full and open competition, the contracting officer is required under the FAR to solicit offers from as many potential sources as is practicable under the circumstances. The FAR prohibits contracting if the justification for less than full and open competition results from a lack of advanced planning. For a more detailed comparison of AMS and the FAR, see appendix III.

Although some of the FAA personnel we interviewed see AMS as more efficient and flexible than the FAR, other current and former FAA procurement officials we interviewed who have experience using both the FAR and AMS did not agree that AMS is more flexible than the FAR. According to these officials, the FAR may appear inflexible and cumbersome to persons who lack experience with it, but those who are familiar with it are able to navigate its complexities effectively. The FAR requires full and open competition, but as experienced procurement personnel know, the system does not break down when emergencies necessitate quick and decisive action. For example, we recently reported that agencies generally complied with applicable FAR requirements in awarding new contracts for work in Iraq using other than full and open competition.In some circumstances, the government’s legitimate need for prompt action was sufficient to justify selecting a contractor on an expedited basis from among the firms that appeared able to meet the government’s emergency need. In other cases, the agencies reasonably determined that only one source could meet their requirements.

AMS Provides Some Discipline but Does Not Ensure a Knowledge-Based Approach to Acquisition

AMS provides some discipline through its various phases, activities, and decision points for acquiring major ATC systems; however, it does not ensure the use of a knowledge-based approach found in the best practices for managing commercial product developments and DOD acquisitions that we have identified in numerous past reports.Commercial best practices call for specific knowledge to be captured and used by corporate-level decision-makers to determine whether a product has reached a level of development (product maturity) sufficient to demonstrate its readiness to move forward in the acquisition process. The capture of such knowledge and its use by executives helps to avoid cost overruns, schedule slips, and performance shortfalls that can occur if decision-makers commit to a system design before acquiring critical technology, design, or manufacturing knowledge. The absence of these key best practices under AMS puts FAA’s major ATC acquisitions at risk of cost overruns, schedule slips, and performance shortfalls.

Best Practices for Managing Acquisitions Call for a Knowledge-Based Approach, Including Criteria for Knowledge Needed and Oversight at the Corporate Executive Level

Commercial best practices call for managing acquisitions using a knowledge-based approach, including (1) using established criteria to attain specific knowledge at three critical junctures in the acquisition cycle, which we call knowledge points, and (2) requiring oversight at the corporate executive level for each of these knowledge points. For example, at each knowledge point, successful product developers apply specific indicators, or criteria, to determine whether they have attained the knowledge they need to move to the next phase or activity in the acquisition process. Such developers also conduct corporate executive- level reviews to ensure that they obtain the insights and perspectives of stakeholders throughout their organization. If the knowledge attained does not meet the criteria for advancement or if the executive reviewers determine that further development is inconsistent with their priorities, the acquisition does not move forward. Table 3 summarizes the knowledge points, criteria, oversight reviews, and timing of oversight reviews included in our model of best practices for major acquisitions.

Experience with commercial best practices has shown that to the extent that the level of knowledge called for at each knowledge point is not attained, organizations take on risks in the form of unknowns that will persist into the later stages of development, where they will take more time and money to resolve if they become problems. Such problems lead to cost increases and schedule delays.

AMS Has Some Good Features but Does Not Ensure That High Levels of Knowledge Are Attained Before Major Commitments Are Made

AMS has some good features, including phases and key decision points indicative of an acquisition process that has some elements of discipline; however, AMS does not ensure that high levels of knowledge are attained and that corporate executive-level reviews occur before major commitments of agency resources are made. For example, like the best practices model, AMS identifies critical junctures, which it terms “decision points.” Three of these decision points occur during the initial acquisition phase (mission need, initial investment, and the final investment decision).

A fourth decision point occurs before production, and a fifth decision point occurs before the start of the final acquisition phase (in-service management). AMS also calls for detailed technical and programmatic information that decision-makers can use at the first three decision points to assess whether or not FAA should initiate an acquisition program. This information includes a final requirements document, a final acquisition program baseline, a final investment analysis report, an acquisition strategy paper, and an integrated program plan. Finally, AMS, like our best practices model, calls for senior executives to review the information and determine whether the acquisition is ready to move forward. The FAA executives who make the decisions at these points include associate and assistant administrators, acquisition executives, the chief financial officer, the chief information officer, and legal counsel; they form the Joint Resources Council (JRC), FAA’s senior decision-making body for major ATC acquisitions. Table 4 summarizes this information.

AMS departs from the best practices model in two key ways—it does not call for high levels of knowledge to be attained at three critical junctures (knowledge points), and does not call for corporate executive-level oversight at one of five junctures. Specifically, AMS does not establish explicit, written criteria for (1) the information needed to determine technology maturity at solution implementation, (2) releasable drawings at critical design review and production process controls at production. Our best practices model calls for attaining specific knowledge and setting out criteria for what information should be available to help organizations minimize risks in the form of unknowns. Risks associated with such unknowns can persist into the later stages of development, where they can take more time and money to resolve if they become problems, potentially leading to cost increases and schedule delays.

In addition, AMS does not provide for corporate executive-level oversight reviews at two of the three key junctures where our best practices model calls for such reviews. Although AMS calls for three Joint Resources Council reviews during the initial acquisition phase—while our model calls for a single corporate executive-level review—AMS allows the council to delegate its oversight responsibility later in the acquisition process to the program managers within the service organization responsible for an acquisition. By contrast, our model calls for two corporate executive-level reviews later in the acquisition process.

According to FAA, its approach gives program managers flexibility, expedites decision-making, and allows the executives with the most knowledge about a major acquisition to make key decisions about its continued development. FAA’s reliance on this approach assumes that the program managers will inform higher-level managers if they are unable to meet the performance schedules and systems requirements approved by the Joint Resources Council. However, although program managers may have the most knowledge about their particular acquisition, they may not have the agencywide perspective of the Joint Resources Council members. Having an agencywide perspective, including a broad understanding of an acquisition’s potential impact on other agency projects and operations, is especially critical when an acquisition includes the production of multiple units and requires a substantial commitment of agency resources, as do FAA’s primarily multimillion-dollar acquisitions, such as controller workstations and radars.

Because decisions about moving a major acquisition forward require both a program manager’s specific knowledge of the acquisition itself and a senior executive’s understanding of the acquisition’s potential impact on other agency projects and operations, our best practices model calls for both measurable criteria at key points in the acquisition process to ensure that specific knowledge has been captured and corporate executive-level reviews to ensure that senior decision-makers have the opportunity to independently consider this knowledge. Without higher-level reviews such as our best practices model recommends and the Joint Resources Council could provide later as well as early in the acquisition process, FAA cannot ensure that it has fully considered the impact of advancing an acquisition on other agency projects and operations. This opportunity for full consideration is a central advantage of managing acquisitions as a portfolio, as we concluded in our August 2004 report on FAA’s information technology investment management process.

Figure 2 contrasts FAA’s process for reviewing an acquisition’s progress under AMS with the process that we found leads to successful commercial acquisitions. !

To facilitate the comparison of AMS with out best practices model in this report, we have done the following: (1) placed FAA's "Mission Analysis" and "Investment Analyses" activities in the "Needs and Solution Identification" phase to make it comparable with the "concept and technology development" phase in our best practices model; (2) depicted only the final investment decision point, recognizing that the investment analysis phase includes an initial investment decision; and (3) placed “system integration” and “system demonstration” in the solution implementation phase.

AMS does not explicitly call for a design review decision point, which would fall between system integration and system demonstration.

To its credit, FAA continues to improve its AMS process. For example, the agency is currently modifying its mission needs activity to make the selection of major ATC acquisitions more consistent with the overall goals of modernizing the National Airspace System. In addition, the Air Traffic Organization has established an executive council to review major acquisitions before they are sent to the Joint Resources Council. This review is designed to screen acquisitions to determine which ones are important enough to warrant higher-level review by the Council. Finally, FAA is currently revising AMS to bring it in line with the Office of Management and Budget’s guidance. Specifically, the agency is incorporating OMB Exhibit 300, which provides the investment justifications and management plans required for major ATC acquisitions.

As Implemented, AMS Has Not Resolved Long-standing Acquisition Problems, but FAA Is Beginning to Focus More on Results

According to our review of seven major ATC systems and analysis of FAA’s performance in acquiring major systems, AMS has not resolved the long- standing problems that FAA experienced before implementing AMS, but the agency is beginning to focus more on the expected results of its major acquisitions. (See table 5.) Specifically, our review found that AMS guidance did not call for requirements that were specific enough to minimize requirements growth or unplanned work for five of these systems. This lack of specificity resulted in the inadequate development or definition of requirements, growth in requirements, unplanned work, or a reduction in performance for five of these systems. In addition, for three of these systems, FAA underestimated the difficulty of modifying available software to fulfill its mission needs. Because AMS guidance was not sufficient to account for the risks associated with modifying available software, FAA encountered unexpected software development needs, higher costs, and schedule delays. The two systems we reviewed that were initiated after AMS was implemented are currently meeting cost and schedule milestones; however, both systems are showing symptoms of FAA’s past problems with developing requirements and managing software, and it is too soon to tell if these programs will remain within their cost, schedule, and performance parameters. In addition, our work on FAA’s major acquisitions, along with that of the DOTIG and others has shown that the problems FAA experienced before 1996 in acquiring major systems persist under AMS and that effective acquisition management, rather than the use of a specific contracting process (e.g., the FAR or AMS) is key to successful acquisitions. To its credit, FAA is beginning to focus more on results, largely through its new Air Traffic Organization, which has been charged with taking a more performance-based approach to managing the agency’s acquisitions.

Our Reviews of Seven Major Systems Show That Problems with Requirements and Software Management Persist under AMS

Our reviews of seven of FAA’s costliest ATC system acquisitions found that the problems FAA experienced with requirements and software management and their related impact on cost, schedule, and performance goals persist today under AMS.Figure 3 identifies these seven acquisitions and their milestones, which are expressed in terms of AMS decisions even when the acquisitions were initiated before AMS was implemented. (See app. V for a description and the status of each of these projects.) Specifically, for 6 of these 7 major ATC acquisitions, FAA did not consistently (1) clearly define system requirements at the investment decision point or (2) adequately assess software complexity. Moreover, as FAA has acknowledged, it has never managed its major acquisitions by focusing on how each would improve the efficiency of ATC operations while maintaining or improving safety. Although FAA has made progress in improving its acquisition of major ATC systems—by, for example, improving the maturity of its processes for acquiring software, using a “build a little, test a little” approach to acquisitions as it did for Free Flight Phase 1,and restructuring its organization to minimize stovepipes—long- standing problems persist in these areas. In addition, the two systems we reviewed that were initiated after AMS’s implementation are currently operating within cost and schedule goals; however, they are showing symptoms of past problems with developing requirements and managing software complexity. Moreover, our work for more than two decades— before and after AMS’s implementation—has cited these types of weaknesses as central reasons for the agency’s long history of cost, schedule, and performance shortfalls. This work has also found that the effectiveness of an agency’s acquisition management has had a greater impact on the success of its major acquisitions than the contracting process used (e.g., the FAR or AMS).

Inadequate Development or Definition of Requirements Led to Requirements Growth or Unplanned Work for Five Acquisitions

For five of the seven acquisitions we reviewed, AMS guidance did not call for requirements that were specific enough to minimize requirements growth or unplanned work. For four of these five acquisitions—STARS, LAAS, NEXCOM, and ATOP—incomplete and poorly defined requirements in the final requirements documents, used at the investment decision point to assess an acquisition’s readiness to enter the development phase, led to requirements growth, unplanned development work, or a reduction in system performance.For the fifth acquisition—ASR-11—FAA misjudged the extent to which the high-level requirements that were used to support the commercial-off-the-shelf/nondevelopmental item (COTS/NDI) procurement by the Department of Defense could result in a product capable of meeting FAA’s mission or user needs. As a result, unplanned software changes were required.

FAA’s cost estimate for STARS has grown from its original estimate of $0.94 billion in 1996 to $1.46 billion in 2004 and will deploy only 50 of the 172 STARS initially planned. Much of the cost growth has been due to FAA requirements creep. As a result, the STARS program has experienced delays of more than five years from its original plan, in part due to added requirements to the commercial-off-the-shelf Initial System Configuration (ISC). However, the STARS ISC was satisfactory for use by the Department of Defense as deployed.

A final requirements document was approved, and the development of LAAS was scheduled to begin in 1999. However, poorly established requirements resulted in the addition of 113 new requirements to the initial specification, entailing significant software and hardware changes. Furthermore, LAAS may not achieve its promised capabilities because FAA has been unable to develop technologies necessary to warn pilots of a disruption in the LAAS signal. Until this technology is developed, LAAS cannot be operated safely. As a result, FAA recently cut the fiscal year 2005 funding for LAAS, and the program will revert to a research and development effort.

FAA developed a final requirements document for the NEXCOM system, but the requirements lacked the specificity needed to assess the development risk. According to a NEXCOM contractor program official, this led to miscommunication about the program requirement relating to signal interference. This official stated that they misunderstood this requirement and had not planned on the additional development work for the NDI solution to meet such program objectives and delayed the program 21 months. Another program requirement involved the NEXCOM radios meeting or exceeding the operational coverage area of the existing voice system. The existing radios had power output levels of 50 watts but the NEXCOM contractor could only achieve 34 watts of power to meet the coverage requirement. A program official stated that the contractor and FAA had not agreed on the testing procedures to assess the power levels. This posed an “unacceptable consequence” and, as a result, FAA performed additional testing or flight checks of the reduced radio performance (50 watts versus 34 watts) and determined that the performance reduction should not affect NEXCOM’s mission or its coverage requirement.

FAA did not follow the AMS guidelines that call for completing a final requirements document before proceeding to the development phase for ATOP. The Joint Resources Council approved a delay in developing the final requirements until after contract award. This decision resulted in schedule delays and additional unplanned software development. The ATOP program office asserted that the requirements remained very stable and that the program is within cost and schedule objectives established by the Council. However, FAA’s internal documents revealed that the requirements were not adequately defined. For example, the ATOP Investment Analysis Study reported to the Joint Resources Council prior to contract award that the lack of more detailed ATOP requirements at this stage of acquisition added risk and was of concern to the investment analysis team. Under AMS, this team is responsible for, among other things, conducting risk analyses for the various acquisitions. Furthermore, an ATOP Assessment Team conducted a study in March 2003 and determined that at the ATOP contract award, “requirements were written at a high level and not mutually understood by FAA and the contractor.” However, FAA management allowed the ATOP program to proceed to solution implementation without the final requirements document and, according to the contractor, this resulted in schedule delays and growth in the amount of software needing development.

The high-level requirements for ASR-11, jointly generated by FAA and the Department of Defense, to support a COTS/NDI acquisition, resulted in a product that did not initially meet the FAA mission or user needs. The software changes that were required to meet FAA’s target detection needs, as well as significant hardware design changes, parts obsolescence, and production issues, added approximately two years to system qualification and acceptance.

FAA Underestimated Software Complexity for Three Systems

For three of the seven major ATC acquisitions we reviewed—ITWS, LAAS, and ATOP—FAA’s AMS guidance was not sufficient to address the risks associated with modifying available software to fulfill FAA’s mission needs. In all three cases, FAA officials underestimated the difficulty of modifying available software. Our work has shown that underestimates are likely to result in unexpected software development, higher costs, and schedule delays.

ITWS experienced delays from the beginning because of the complexity of its software development. Although the program appeared to be progressing according to its baseline, immediately after the critical design review in September 1998, the contractor revealed that it had exceeded the target cost by $4 million. In addition, the contractor claimed that the program did not recognize that the computer processor originally planned for the program was becoming outdated, that the manufacturer planned to discontinue its production because the market was demanding a processor with greater processing and storage capability, and that as a result, the original computer processor would not be available to the program. Consequently, ITWS experienced cost increases, schedule delays, and performance shortfalls. According to the contractor and the original acquisition plan, all systems were scheduled for delivery by December 2001, but that date has now stretched to after 2009.

LAAS’s technology maturity was not adequately assessed, and further development was needed. Specifically, the potential for radio interference through the atmosphere was not understood and could limit LAAS’s operations. FAA has now placed all LAAS activities in research and development. FAA did not adequately assess LAAS’s software development. At the time of the contract award, the contractor and FAA estimated that 80 percent of the software that LAAS required had been developed. FAA later determined that only 20 percent had been developed. FAA and the contractor attribute this discrepancy to a lack of communication on the steps necessary to satisfy the program’s requirements. FAA agrees that it should have conducted a software audit and a software capabilities assessment, but pressures to keep LAAS on schedule resulted in an inadequate assessment.

The ATOP contractor underestimated by about half the extent to which legacy nondevelopmental item software, which is the core of the ATOP system, met the program’s 1,036 requirements. As a result, a significant amount of unanticipated new software code development and other modifications were required.

ATC Systems Have Required Multiple Rebaselining Decisions to Address Delays and Cost Growth

As figure 3 illustrates, FAA initiated at least one rebaselining decision for three of the five acquisitions that were begun before AMS was implemented and were later transitioned to AMS. These rebaselining decisions responded to delays and cost growth—problems that arise when requirements are not stable, a program’s design is not fixed, or software code growth is not controlled. For example, FAA rebaselined STARS two times—first in 1999 and again in 2002. Similarly, 2 years after the investment decision for ITWS, FAA rebaselined the program twice, in 1997 and again in 2001. Given the frequency of these past rebaselining decisions for major ATC systems and the number of years that elapsed before or between the rebaselining decisions (3 to 4 years), it is too soon to tell whether the two systems that were initiated under AMS—ATOP and ERAM—will require similar rebaselinings and ultimately meet their cost, schedule, and performance goals. Although both programs are currently operating within their cost and schedule goals and have not yet been rebaselined, FAA has had problems with managing its major acquisitions in the past and is currently having difficulties developing requirements and managing software complexity. Furthermore, as we reported in May 2004, FAA’s budget increased from $9 billion in 1998 to $14 billion in 2004 but will be constrained for the foreseeable future. In such a constrained budget environment, cost growth and schedule problems can have serious negative consequences for ongoing modernization efforts—postponed benefits, costly interim systems, delays in funding other systems, or reductions in the number of units purchased.

Internal and External Reviews Have Found That FAA Has Made Some Progress but Continues to Experience Problems in Acquiring Major ATC System under AMS

Reviews of FAA’s acquisition process, conducted by FAA, GAO, the DOTIG, and others have shown that FAA has improved its management of major ATC acquisitions in recent years but continues to experience cost overruns, schedule slips, and performance shortfalls under AMS. Table 7 summarizes the results of 22 internal and external reviews of FAA’s major ATC acquisitions. According to these reviews, issued from 1997 through 2004, the same problems have persisted over many years, despite various initiatives to address them, and FAA needs to strengthen its management controls. For example, a key FAA review of eight major ATC acquisitions, published in 1999, 3 years after AMS was implemented, found that these acquisitions, though on track to meet their performance goals, were not meeting their cost and schedule baselines. FAA attributed these cost and schedule issues to new or poorly understood requirements, underestimates of the acquisitions’ technical complexity, and funding shortfalls.

In addition, our reviews of major FAA acquisitions—initiated before and after AMS was implemented—have found for more than two decades that FAA’s failure to meet schedule, cost, and performance baselines for major ATC acquisitions has been due to shortfalls in planning, weak management controls, and a lack of systematic processes for acquiring new systems, including inadequate requirements management, cost-accounting data, and estimates of technical difficulty. As we reported in August 2004, judged against the criteria of GAO’s framework for information technology (IT) investment management, which measures the maturity of an organization’s investment management processes, FAA has established about 80 percent of the basic selection and control practices that it needs to manage its mission-critical investments for the National Airspace System. For example, FAA’s business units actively monitor projects throughout their life cycles.However, the agency’s senior IT investment board does not regularly review investments that are in the “in-service management,” or operational phase, and this creates a weakness in FAA’s ability to oversee more than $1 billion of its IT investments. In addition, the agency has not yet established the practices that would enable it to effectively manage its annual IT budget of about $2.5 billion, and agency executives lack assurance that they are selecting and managing the mix of investments that best meets the agency’s needs and priorities. DOT has responded to our recommendations to FAA to strengthen its IT investment management capability.

Moreover, other reviews, such as those by Booz-Allen & Hamilton and MITRE, have identified other shortfalls, which reflect a lack of proper management controls and planning. For example, in 1997, Booz-Allen & Hamilton found, among other things, that FAA had not clearly defined organizational roles and responsibilities within the various phases of AMS and that greater guidance and training under AMS were warranted. In 1999, Booz-Allen & Hamilton reported that FAA had not demonstrated improvement in adhering to planned costs and schedules under AMS and that the agency needed to better manage its development of requirements and address persistent funding shortfalls. Moreover, in 2001, a MITRE report on selected major acquisitions found inadequate management controls and deficiencies in both contractors’ performance and in FAA’s measurement of acquisition performance. See table 7 for a chronological listing of the reviews.

FAA’s ATO Is Taking Steps to Improve Major ATC Acquisitions

FAA’s recent reorganization, which brought ATC acquisitions and operations together in the ATO, is expected to help the agency address many of the concerns we have identified for more than two decades, including those identified in this report. For example, the ATO is continuing to develop and refine specific guidance for critical areas, such as requirements management, software development, and cost estimation. In addition, as the overseer of both ATC acquisitions and operations, the ATO is in a position to facilitate more effective management of major ATC acquisitions than has occurred in the past. The ATO is attempting, for example, to link acquisition decisions directly with expected improvements in operational efficiency without compromising safety. This is important, given that FAA has spent about $2.5 billion on ATC modernization per year since 1996 while operating costs have continued to rise—from $4.6 billion to $7.5 billion over the past decade. FAA had not completed its reorganization or implemented all of its initiatives at the time of our audit.

Improvements to Requirements Development

With the establishment of the ATO, FAA consolidated requirements development from two organizations (the organization sponsoring an acquisition and the former agencywide acquisition organization) into a single new organization—the Air Traffic System Requirements Service. In addition, the ATO developed guidance to better manage requirements during the middle phase of AMS (solution implementation). According to FAA officials, some more complex development efforts may need to develop systems requirements and a more detailed requirements document than AMS currently calls for in the final requirements document. More important, in January 2003, FAA issued guidance on requirements management, Roles in Requirements Management During Solution Implementation Phase, which provides for integrated requirements teams that maintain responsibility for requirements management throughout an acquisition’s life cycle. According to this guidance, when the final requirements document is accepted by the Joint Resources Council at the investment decision point, a requirements baseline is established and any proposed changes to the requirements must be assessed for their impact on the program and shown to be operationally suitable, affordable, executable, and justifiable. An FAA official on an integrated requirements team stated that any changes that may affect an acquisition’s cost and the schedule require approval by the Executive Committee. The FAA official also stated that this guidance has already helped to stabilize NEXCOM’s requirements during the solution implementation phase. Other FAA officials representing the Joint Resources Council acknowledged that the guidance should ensure greater control over program requirements growth, but said that not all program offices have consistently applied it.

Improvements to Managing Software and System Acquisition and Development

To better manage software programs for ATC modernization acquisitions, FAA established a centralized process improvement office that reports to the Chief Information Officer (CIO). This office developed an FAA integrated capability maturity model (i-CMM), a software development and management model that is similar to a model developed by Carnegie Mellon University called the Capability Maturity Model Integration (CMMI®), which is used to appraise the maturity of an organization’s processes for acquiring software. However, FAA’s i-CMM goes beyond Carnegie Mellon’s model to reflect international standards. The CMMI® appraisal methodology calls for assessing process areas—such as project planning, requirements management, and quality assurance—by determining whether key practices are implemented and overarching goals are satisfied. Both the i-CMM model and CMMI® appraisal methodologies provide a logical framework for measuring and improving key processes needed for achieving quality software and systems.

However, as we reported in August 2004, FAA projects are not required to use the capability maturity model for process improvement, and individual projects that use the i-CMM model are allowed to choose which process areas they seek to improve and to determine when they are ready for an appraisal of their progress. To date, fewer than half of FAA’s major ATC projects have used this model. The recurring weaknesses we identified in our project-specific evaluations are due in part to the flexibility these projects were given in deciding whether and how to adopt this process improvement initiative. Furthermore, after combining its ATC organizations into a single performance-based organization (the ATO), FAA is reconsidering prior policies, and it is not yet clear whether process improvement will remain a priority. Without a strong senior-level commitment to process improvement and a consistent, institutionalized approach to implementing and evaluating it, FAA cannot ensure that key projects will continue to improve systems acquisition and development capabilities. As a result, FAA will continue to risk the project management problems—including cost overruns, schedule delays, and performance shortfalls—that have plagued past acquisitions. To address these shortcomings, we recommended that the Secretary of Transportation address specific weaknesses and institutionalize FAA’s process improvement initiatives by establishing a policy and plans for implementing and overseeing process improvement initiatives.

Improvements to Estimating Costs

FAA has taken steps to improve its cost estimation for major ATC projects by issuing guidance on how to develop and use pricing under AMS. For example, AMS policy calls for audit trails to record and explain the values that are used as inputs to cost models. In addition, it calls for agency officials, when reporting to executive oversight agencies and Congress, to disclose the level of uncertainty and imprecision that are inherent in cost estimates for major ATC systems. According to AMS policy, estimators record the procedures, ground rules and assumptions, data, environment, and events that underlie their development or update of a cost estimate. This information supports the credibility of the cost estimate, aids in the analysis of changes in program costs, enables reviewers to assess the cost estimate effectively, and contributes to the population of FAA databases that can be used for estimating the cost of future programs. Finally, despite a delay of many years, FAA officials told us that they are in the final stages of completing the agency’s cost-accounting system and plan to have it in place across the agency by the end of this calendar year, which will bring FAA into compliance with the Federal Managers' Financial Integrity Act of 1982. This measure will help reduce the likelihood of cost overruns or improper payments for unallowable costs and provide decision-makers with critical information. As we have reported in the past, a cost- accounting system is critical to managing major ATC acquisitions, because without it, FAA lacks the information it needs to reliably estimate operating costs over an acquisition’s life cycle.

Other Improvement Efforts

In May 2004, the FAA Administrator testified to Congress that, to date, in attempting to improve the efficiency of ATC operations while maintaining safety, FAA had not managed its major ATC acquisitions to be aware of their cost implications for its operations. The Administrator said, however, that the agency was taking its first steps to fundamentally change how it makes acquisition decisions by adopting a more results-oriented approach. Under this approach, the agency plans to link its decisions to fund major acquisitions directly with their expected contribution to improving operational efficiency and controlling escalating operating costs. Whereas, in the past, FAA measured results in terms of its progress in completing and deploying a major ATC system, it was now going to focus on how a given system improved operational efficiency. Such an approach holds promise for helping FAA more effectively manage its portfolio of major ATC acquisitions by providing a sound basis for choosing among competing priorities. However, because FAA has only recently begun to incorporate this type of analysis of acquisitions’ costs and operational efficiency into its decision-making and management processes, it is still too early to assess the results.

In addition, to its credit, FAA has created a training framework for its acquisition workforce, which we found mirrors human capital best practices that we have identified. In January 2003, we reported on FAA’s efforts to define and train its workforce to meet the requirements of the Clinger-Cohen Act of 1996. This act required FAA and other civilian agencies to establish education, training, and experience requirements for their acquisition workforce. Our work on public and private best practices has identified six elements of training as critical to acquisition. These elements include (1) prioritizing the acquisition initiatives most important to the agency, (2) securing top-level commitment and resources, (3) identifying those who need training on specific initiatives, (4) tailoring training to meet the needs of the workforce, (5) tracking training to ensure it reaches the right people, and (6) measuring the effectiveness of training. These six elements are crucial for successfully implementing acquisition initiatives and reforms. Agencies that do not focus their attention on these critical elements risk having an acquisition workforce that is ill equipped to implement new processes. The probability of success is higher if training is well planned rather than left to chance. In 2003, we found that FAA’s model for training its acquisition workforce largely mirrored public and private- sector best practices and that the agency had highly developed processes for four of these six elements. See figure 4.

Since 2003, FAA has taken some steps to measure the effectiveness of its training. For example, the agency collects and reviews participants’ assessments of the knowledge they have gained, the extent that learning objectives were achieved and the applicability and usefulness of the training. In addition, members of FAA’s Intellectual Capital Investment Plan Council have attempted to make qualitative judgments about the impact of the training on the effectiveness or efficiency of their organizations. However, FAA is still developing an evaluation program with metrics to measure the extent to which organizational goals are achieved when individual training objectives are met. Industry and government experts believe training and human capital investments are prerequisites for successfully introducing and implementing effective acquisition best practices. FAA’s acquisition workforce plays a critical role in addressing long-standing weaknesses that we and others have identified with FAA’s acquisition of major ATC systems. Given the importance of training for acquisition workforces, it will be important for the ATO to put mechanisms in place to comprehensively evaluate the effectiveness of the training it provides to improve the knowledge base of FAA’s acquisition workforce.

To improve its investment management decision-making and oversight of major ATC acquisitions, the ATO also initiated the following procedures: Integrate AMS and the Office of Management and Budget’s Capital Planning and Investment Control Processto develop a process for analyzing, tracking, and evaluating the risks and results of all major capital investments made by FAA.

Conduct Executive Council reviews of project breaches of 5 percent in cost, schedule, and performance to better manage cost growth; Issue monthly variance reports to upper management to keep them apprised of cost and schedule trends.

Monitor progress in meeting the goals identified in FAA’s Flight Plan, the agency’s blueprint for action through 2008. The Executive Council tracks this progress monthly and reports to the Administrator, using a color-coded system to keep her apprised of how well FAA is meeting its goals. Green denotes that a goal will be met, yellow denotes that some of the activities leading to a main goal may be in jeopardy but the overall goal can be achieved, and red denotes serious concerns about reaching a goal without major intervention. A formal progress report is issued quarterly and made publicly available on the agency’s Web site; and Increase the use of cost monitoring or earned value management systems to improve oversight of programs.

Despite FAA’s current and planned efforts to improve its acquisition of major ATC systems under the ATO, given the newness of these efforts and the agency’s poor track record in this area for more than two decades, it is critical for FAA to (1) modify AMS to more fully reflect the best practices followed by high-performing acquisition organizations, (2) follow through on planned improvement initiatives, and (3) adopt a continuous improvement approach to acquiring new ATC systems.

Conclusions

In the early 1990s, FAA contended that it needed relief from the FAR to remedy long-standing problems with cost, schedule, and performance shortfalls in its major ATC acquisitions; however, our work for more than two decades in this area has found that acquiring major ATC systems successfully depends more on managing an acquisition process well than on using a specific contracting process (e.g., the FAR or AMS). While our recent work has shown some improvement in FAA’s management of major ATC system acquisitions, some key problems that existed before 1996 persist under AMS—including difficulty with clearly defining system requirements at the investment milestone and adequately assessing complex software requirements. These problems continue to make these acquisitions vulnerable to cost, schedule, and performance shortfalls. Without further measures to improve the development and management of requirements and to better estimate the complexity of the software development needed for major ATC systems, such shortfalls are likely to persist.

Although AMS provides some discipline for acquiring major ATC systems through its various phases, activities, and decision points, it does not require that (1) specific knowledge be attained using explicit written criteria and (2) corporate executive-level oversight be provided to determine—independently from the program offices—whether a system has reached a level of development (product maturity) sufficient to move forward in the acquisition process. Commercial best practices call for such knowledge-based decision-making at the corporate executive-level to help ensure that acquisitions are not moved into the development phase prematurely, to obtain greater predictability in ATC system program costs and schedules, to improve the quality of the ATC systems that are deployed, and to deliver new capability to the National Airspace System faster. A knowledge-based approach is also important because it provides assurance that agency decision-makers have critical information about an acquisition’s ability to meet a mission need and FAA’s readiness to move forward in the acquisition process before making large commitments of agency resources. Absent such an approach, FAA lacks assurance that it has obtained the critical technological, design, or manufacturing knowledge that best practices call for to avoid cost overruns, schedule slips, and performance shortfalls. As a result, FAA is not doing all that it can to systematically address persistent shortcomings in its management of major ATC acquisitions. Moreover, although FAA has established a framework for training its acquisition workforce under the ATO, it has not yet developed comprehensive performance criteria to evaluate how effectively it has implemented this framework. As a result, the agency lacks assurance that its use of this framework is having the intended effect of improving the knowledge base of this workforce.

Recommendations for Executive Action

We are making five recommendations to the Secretary of Transportation. To reduce the risk of persistent cost and schedule shortfalls in major ATC system acquisition programs, to improve the quality of the ATC systems that are deployed, and to deliver new capability to the National Airspace System faster, we recommend that the Secretary of Transportation advise the FAA Administrator to take the four following actions: Modify AMS to specify that requirements be more clearly defined for major ATC systems, including providing more detailed guidance on setting clear, objective, and measurable requirements that reflect customers’ needs, before making large investments of agency resources.

Establish a strategy for identifying and measuring all additional development needed for complex software (e.g., commercial-off-the- shelf or nondevelopmental items) used for major ATC systems.

Develop explicit written criteria for the key decision points called for under best practices, including the capture of specific design and manufacturing knowledge.

Require corporate executive-level decisions at these key decision points (before an acquisition moves from integration to demonstration and, again, before it moves to production).

In addition, to assure FAA that the training framework it has adopted for the ATO’s acquisition workforce is improving the knowledge base of this workforce as intended, we recommend that the Secretary advise the Administrator to develop performance criteria to comprehensively evaluate the framework’s effectiveness.

Agency Comments

We provided copies of a draft of this report to DOT for review and comment and met with Department and FAA officials, including the ATO's Vice President for Acquisition and Business Services, to obtain their comments. FAA officials told us that they have made great strides in improving their acquisition of major ATC systems under AMS; however, they recognize that there is room for improvement and are firmly committed to implementing best practices for acquisitions. These officials generally agreed with the report's findings and conclusions and said that our recommendations would be useful to them as they continue to refine their acquisition management system, including training their acquisition workforce. The agency provided us with oral comments, primarily technical clarifications, which we have incorporated as appropriate.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to interested congressional committees, the Secretary of Transportation, and the Administrator, FAA. We will also make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

Please call me at (202) 512-2834 if you or your staff have any questions concerning this report. Key contributors to this report are listed in appendix VI.

FAA Has Begun Analyzing Spending Trends to Take a More Strategic Approach to Procurement

Our review of the Federal Aviation Administration’s (FAA) general procurement of goods and services focused on the Air Traffic Organization (ATO) and its predecessor offices. According to FAA officials, the ATO has recently begun to consider ways to better leverage its buying power by taking a more strategic approach to procurement. While FAA uses the Acquisition Management System (AMS) for all FAA acquisitions, including the procurement of such goods and services as office supplies, computers, telephone services, and engineering and technical support services, these procurement activities take place in a decentralized environment of independent, transaction-oriented buying processes. Each FAA unit determines its need for goods and services and procures them as necessary, leaving headquarters with limited oversight of the agency’s total procurement spending. For example, in 2003, FAA units carried out over 346,000 procurement actions for goods and services and purchase cardholders made an additional 335,000 transactions. This fragmented environment does not permit the agency to leverage its buying power through lower-cost, consolidated contracts, at the local, regional, or national level and to rationalize the number of suppliers best suited to meet the agency’s needs. At the same time, as part of a strategic procurement effort, FAA can use spend analysis to monitor trends in small and disadvantaged business participation so that it can balance the goals of lower-cost contract consolidation and promoting small business contracting opportunities.

Spend analysis, a tool used in a strategic approach to procurement, provides knowledge about how much is being spent for what goods and services, who the buyers are, who the suppliers are, and where the opportunities are to leverage buying power. Our past work shows that private companies are using spend analysis as a foundation for employing a strategic approach to procurement. The analysis identifies where numerous suppliers are providing similar goods and services—often at varying prices—and where purchasing costs can be reduced and performance improved by better leveraging buying power and reducing the number of suppliers to meet the company’s needs. Our research on commercial best practices has found that spend analysis is an important driver of strategic planning and execution. As part of an overall strategic procurement effort, companies use spend analysis to (1) define the magnitude and the characteristics of their spending, (2) understand their internal clients and supply chain, (3) create lower-cost consolidated contracts, and (4) monitor spending with small and disadvantaged businesses to achieve socioeconomic procurement goals.

We previously reported that six agencies, including DOT, did not take advantage of opportunities to obtain more favorable prices on purchase card buys with frequently used vendors—vendors where an agency spends more than $1 million annually. In these six agencies, which accounted for over 85 percent of federal government purchase card spending, frequently used vendors accounted for purchases totaling nearly $3 billion in 2002. We recommended several actions—including conducting spend analysis using available data and gathering additional information where feasible—that could ultimately help these agencies achieve $300 million annually in potential savings.

In fiscal year 2003, FAA procured nearly $4 billion in goods and services and spent an additional $132 million using purchase cards. According to senior FAA officials, the agency has just begun to implement a strategic approach to general procurements. Other federal agencies are beginning to use strategic tools such as spend analysis to improve their spending for goods and services, and some have initiatives under way to obtain more favorable prices on purchase card buys. According to a senior FAA acquisition official, FAA has to balance the need of its units to independently make purchases that pertain solely to unit requirements with the agency’s need to aggregate purchases of goods and services that are used by more than one unit. FAA has hired a consultant to help begin the use of spend analysis. This effort could reduce the agencywide costs for mobile wireless services by 40 percent—an effort expected to save the agency $8 to $10 million annually. FAA intends to expand its use of spend analysis to target other procurement category savings opportunities, including information technologies, training, facilities, and professional services, as its accounting systems improve.

FAA has taken some preliminary steps to set up a spend analysis program; however, progress has been challenging for FAA because of deficiencies in its accounting systems. For example, because the agency’s accounting system did not identify all of the mobile wireless services for which it was being billed, the contractor implementing the spend analysis had to obtain this information from the wireless providers. FAA will need to expedite its efforts in this area to fully realize potential savings. Our prior research has shown that setting up a spend analysis program can be challenging. Companies have had problems accumulating sufficient data from internal financial systems that do not capture information on all of what a company buys or is using in different, unconnected parts of the company. Despite these challenges, companies that have developed formal, centralized spend analysis programs have been able to track their costs and identify areas for strategic sourcing and savings opportunities.

In our recent report on spend analysis, we found that DOT, at the time of our review, had not yet begun to collect the data needed for a strategic approach to procurement; however, the department is engaged in ongoing efforts to improve procurements, and its top leadership is committed to using spend analysis to change the way goods and services are purchased. One obstacle to using spend analysis that the department cited during our review was a lack of comprehensive and reliable spending data. However, since we completed our review, the department reports stepping up efforts to use currently available data and evaluate business intelligence software to overcome those obstacles. In commenting on our report, Transportation’s senior procurement executive told us that the department is expanding its spend analysis efforts. For example, his office recently reviewed purchase card spending data to identify volume discount opportunities and is now using the results to negotiate new discount agreements with several office product vendors. In addition, he told us that to facilitate future agencywide purchase card spend analyses, DOT awarded a task order in June 2004 to one bank card company that will provide purchase-card audit software and enhanced data-mining capabilities. He also indicated that the department’s leadership supports fiscal year 2005 funding to enhance spend analysis capabilities and that software options for the new agencywide spend analysis system are now being evaluated as part of an ongoing financial and procurement review.

Objectives, Scope, and Methodology

To compare FAA’s Acquisition Management System (AMS) with the Federal Acquisition Regulation (FAR), we reviewed AMS and changes in it over time. We also compared FAA’s acquisition authority under the FAR and under AMS. In addition, we identified relevant recommendations from reports that we, the Department of Transportation’s Inspector General (DOTIG), and others have issued to determine which recommendations have been implemented, rejected, or left open, and to evaluate how those recommendations have modified FAA’s acquisition policies and practices. We also collected and summarized published reports and analyzed available life-cycle management data on the current status of major and nonmajor acquisitions being carried out under AMS.

To determine the ways in which FAA’s acquisition policies compare with our best practices model, we used information from several of our products that examine how commercial best practices can improve outcomes for acquisition programs. This model consists of four phases: (1) concept and technology development; (2) product development, which includes both integration and demonstration activities; (3) production; and (4) operations and support. In between these four phases are three key knowledge points at which commercial firms must have sufficient knowledge to make large investment decisions. We also reviewed and analyzed AMS, accessible at http://fast.faa.gov. Furthermore, to clarify the content of FAA’s acquisition process, we met with various FAA vice- presidents and officials from FAA's Acquisition Planning and Policy Division. Next, we compared and contrasted FAA's acquisition policies with the best practices for commercial acquisitions identified in our past reports. Our analysis focused on whether FAA's policies contained the measurable criteria and management controls necessary to achieve FAA's intent of minimizing cost, schedule, and performance risks. We also interviewed current and former FAA procurement officials that have experience using both the FAR and AMS.

To determine if FAA has effectively implemented its new acquisition authority and improved its acquisition outcomes, we reviewed seven of FAA’s most expensive major ATC acquisitions, including the Airport Surveillance Radar 11 (ASR-11), Standard Terminal Automation Replacement System (STARS), Integrated Terminal Weather System (ITWS), Local Area Augmentation System (LAAS), Next Generation Air/Ground Communications System (NEXCOM), Advanced Technologies and Oceanic Procedures (ATOP), and En Route Automation Modernization (ERAM). See table 7 for specific program costs.

We also selected these seven acquisitions because we considered them to fall into two basic categories-pre-AMS and post-AMS. Five of the acquisitions were initiated before AMS was implemented in April 1996 and were transitioned into AMS at various times before their completion. The two remaining acquisitions—ATOP and ERAM—were initiated and have remained completely under AMS. We then reviewed program documents and reports and interviewed program and agency officials responsible for developing these acquisitions, as well as other acquisitions experts in the private sector. For some acquisitions, we discussed programmatic issues with representatives of the primary contractor for the specific acquisition to obtain information on the practices and procedures used for the acquisition. In addition, we interviewed some current and former FAA procurement officials with experience using both the FAR and AMS to obtain their views on the use of each contracting process and how the two compare. Furthermore, to see how FAA has progressed in addressing problems with its acquisitions, we reviewed our work on acquisitions over the last 20 years, as well as reports by the DOTIG, FAA, Booz-Allen & Hamilton, and MITRE. Because the data in this report on cost, schedule and performance are used as background information or to otherwise provide a description of acquisitions, we did not assess their reliability.

The effect of the current budget process on FAA’s ability to successfully modernize the National Airspace System, including acquiring major ATC systems is not within the scope of this review.

Comparison of the Scope and Flexibility of FAA’s Acquisition Management System and the Federal Acquisition Regulation Process

Background

FAA’s business processes, including its acquisition of major systems, differ significantly from the business processes followed by most other federal agencies. FAA relies on its Acquisition Management System (AMS), which establishes FAA internal acquisition policy. AMS resulted from the adoption of language in the Department of Transportation and Related Agencies Appropriations Act, which directed the FAA Administrator to develop and implement an acquisition management system for FAA. The adoption of this language (section 348) followed FAA’s assertions that the requirement that it conduct procurements in accordance with the Federal Acquisition Regulation (FAR) was at least a contributing factor in its repeated failure to complete air traffic control (ATC) and other modernization programs on schedule. The Administrator was directed to put in place a system that would address the “unique needs of the agency” that FAA contended prevented its acquisitions from being timely and cost-effective.

Section 348 distinguished FAA from other federal agencies by removing FAA from the federal acquisition system. Under section 348, FAA was no longer subject to title III of the Federal Property and Administrative Services Act of 1949, which among other things requires that the government procure supplies and services competitively. It removed FAA as an agency subject to the Office of Federal Procurement Policy Act and eliminated the requirement that FAA comply with the FAR. While mandating that FAA conduct its acquisitions so that “all reasonable opportunities to be awarded contracts shall be provided to small business concerns and small business concerns owned and controlled by socially and economically disadvantaged individuals,” section 348 eliminated the requirement that FAA comply with the Small Business Act. Furthermore, it made the procurement protest system of the U.S. Government Accountability Office inapplicable to FAA, although disappointed offerors can still file protests with FAA’s Office of Dispute Resolution for Acquisition.

AMS Defines an Investment/Life-Cycle Project Management System

Much of AMS guidance concerns project, financial, and property life-cycle management issues. In fact, FAA’s policy describes AMS as applying to all investment programs regardless of cost or the appropriation funding them. It recognizes that a single investment program may span multiple procurements and projects. It applies, according to its terms, to the activities associated with needs analysis, determination of requirements, analysis of investment alternatives, establishment of investment programs, allocation and expenditure of resources, procurement and deployment of needed products and services, in-service management of fielded capability, and eventual disposal of obsolete products.

AMS focuses on the following key program milestones: Mission Analysis—encompasses those key corporate and service-level processes that define, coordinate, and integrate the work of service organizations, thereby providing strategic direction to keep FAA responsive to the service needs of its customers. Mission analysis is used to update a mission need statement, which in turn may identify capability shortfalls or technological opportunities, that is, unmet needs. Unmet needs are presented to the Joint Resources Council (JRC) for a mission need decision. To be approved, the unmet need should be supported by the updated mission need statement and the initial requirements document, including a concept of use, and the initial investment plan.

Investment Analysis—builds on the results of the mission need decision by developing detailed plans and final requirements for each proposed investment program and by defining an acquisition program baseline that establishes cost, schedule, performance, benefit, and risk- management boundaries for the program. AMS calls for planning the entire solution—an effort that may use market survey data but is based in large measure on FAA’s assumptions and data. The service organization produces a final implementation and life-cycle support strategy. A detailed program plan and an acquisition program baseline are also produced. The results are presented to the JRC for a “final investment decision.”

Solution Implementation—encompasses acquiring, accepting, deploying, installing and preparing for the operational use of an approved investment. Approval of the investment carries with it authorization for the service organization to conduct all acquisitions needed to execute the investment decision, subject to any constraints established in the final investment decision.

In-Service Decision—is an FAA system qualification milestone, which is achieved when an otherwise operational investment is satisfactorily tested to demonstrate its operational effectiveness and suitability before it is placed in service in the National Airspace System. The JRC designates the decision maker.

In-Service Management—covers activities throughout a system’s life cycle, starting at the time that an investment becomes operational. In- service product improvements may eliminate latent defects, fix systemic problems, and enhance the utility of the investment. These changes may be made within the approved acquisition program baseline without corporate-level approval. In-service management also includes planning, programming, and developing supporting budget input; monitoring and assessing performance, cost of ownership, and support trends; and planning for service-life investment decisions.

Service Life Extension—seeks a new investment decision by the JRC when a current capability is unable to satisfy demand or when another solution may be more effective. The JRC can decide to revalidate the mission need satisfied by the solution by upgrading or refurbishing fielded capability or by replacing that capability with another equivalent or new superior solution. The JRC may also decide that the capability should be retired.

Only a Portion of AMS Deals Directly with the Procurement Process

Although the FAR includes requirements addressing procurement planning and major system acquisition, AMS as just outlined differs significantly from the FAR in its focus and scope. The FAR addresses planning and major system acquisition in the context of government procurement policy and procedure. Agencies other than FAA find the broader program planning and management issues addressed in AMS outside of the FAR, in documents such as the Office of Management and Budget’s (OMB) Circular A-109, in their own planning guidance, such as the Department of Defense’s (DOD) 5000 series,and in established knowledge-based best practices. As indicated earlier, much of AMS focuses on just such issues. Only AMS section 3 addresses procurement policy and procedure.

AMS States a Nonregulatory FAA Policy

A further significant foundational difference between AMS and the FAR is that AMS sets out a nonregulatory FAA policy, whereas the FAR was adopted and is maintained as a set of published governmentwide regulatory requirements, which form a legal basis for federal agencies’ contract decision-making. AMS is binding on FAA personnel as FAA employees and establishes other guidelines that FAA states should be followed unless there is a rational basis for doing otherwise. AMS is subject to such internal controls as the Administrator chooses to enforce and general overarching legal requirements, such as the Government Performance and Results Act of 1993 (GPRA). There is a legal requirement, created by section 348, that small and socially or economically disadvantaged firms be given all reasonable opportunities to receive contract awards. FAA in its Office of Dispute Resolution for Acquisition has adopted a dispute resolution process with some legal underpinnings. Otherwise, as the preface to AMS states, “nothing in this document creates or conveys any substantive rights.” In short, FAA has assumed no legal obligation to follow AMS other than to ensure that its actions are not arbitrary and capricious or contrary to law. By contrast, the FAR has the force and effect of law, and agencies that are subject to the FAR are bound to follow it.

AMS Chapter 3 Parallels a Subset of the FAR

When FAA personnel apply the procurement methodology in AMS chapter 3, they are applying guidance that closely parallels some of the procedures set out in the FAR. The AMS Chapter 3 acquisition process parallels a subset of the varied selection of procurement methods available under the FAR, requiring that all competitive FAA contracts be negotiated with the awardee being selected on a “best value” basis. The FAR also provides a much more detailed set of information and guidance than does AMS. A comparison of high-level differences and similarities between AMS and the FAR is presented in table 8.

AMS Includes a Less Rigorous Competition Requirement Than Does the FAR

As table 8 indicates, AMS incorporates a less rigorous competition standard than the FAR imposes on the rest of the government. AMS states that it is FAA's policy to provide reasonable access to competition for firms interested in obtaining contracts. According to AMS, in selecting sources, the preferred method of procurement is to compete requirements among two or more sources. However, there is no requirement to ensure that firms that want to participate actually get a chance to do so. Instead FAA may limit competition for further consideration in its screening process to firms with known capabilities or past performance.

The FAR Gives Procurement Professionals Tighter Control over Procurement Decisions

AMS states that authority is delegated to appropriate levels. Once the final investment decision is made, and subject only to any constraints imposed by that decision, the service-level organization is responsible for conducting required acquisitions. Contracting personnel as well as other specialists are then assigned to teams that are responsible to a program manager within the service-level organization. FAA states that this approach increases the pace of doing business. By comparison, the FAR gives contracting professionals clear control over contracting decisions by requiring that procurement decisions be made by procurement professionals—typically contracting officers or their superiors.

Although FAA Project Managers View AMS as More Efficient and Flexible Than the FAR, Some Procurement Officials We Interviewed Do Not Agree

As part of our work, we interviewed project management personnel within FAA as well as current and former FAA procurement officials that have experience using both the FAR and AMS. Generally, FAA personnel see AMS as more efficient and flexible than the FAR, although 9 years after AMS’s adoption, many FAA officials have only limited knowledge of and experience with the FAR. The FAA project managers we interviewed see AMS as more efficient and flexible than the FAR, but some procurement officials with experience in applying both AMS and the FAR did not agree with the view that the FAR was unduly rigid. According to these officials, the FAR may appear inflexible and cumbersome to persons who are inexperienced with it, but those who are familiar with it are able to navigate its complexities effectively. For example, even though the FAR generally requires full and open competition—a process that can take time to give all interested firms an opportunity to participate—contracting officers may be able to expedite the procurement process by using authorized streamlined procedures or, if circumstances warrant, by justifying sole-source or limited competition.

How FAA’s Acquisition Policy Adapted Key Recommendations Made by GAO and DOT (1996-2003)

FAA Refined AMS in Response to Recommendations

Since FAA developed and implemented AMS in 1996, GAO and the DOTIG have made recommendations to improve FAA’s acquisition processes. FAA has adopted many of these recommendations and incorporated them into AMS (see table 9). These implemented recommendations address four main themes: Developing a strategy for culture change that relies on successfully integrating the various elements of acquisition, including specific responsibilities and performance measures for all stakeholders, and providing the incentives needed to promote the desired changes.

Establishing an effective management structure for developing, maintaining, and enforcing the ATC systems architecture to provide an overall plan for the National Airspace System (NAS). This management structure should assign the responsibility and accountability to develop, maintain, and enforce a complete and unified ATC system by ensuring that every project conforms to the overall plan.

Improving cost and schedule tracking to provide data for estimating the costs and schedules of programs. To estimate the costs and time needed for projects, a historical database that includes cost and schedule estimates, revisions, reasons for revisions, actual cost and schedule information, and relevant contextual information is needed.

Improving the management of modernization projects, including the use of project reviews, milestones, and baselines, and cost-accounting information to ensure that programs can be adjusted as needed.

The reports identified in table 10 provide recommendations to address problems we and the DOTIG have identified under these four themes.

Status of the Seven ATC Modernization Acquisitions That GAO Reviewed

Standard Terminal Automation Replacement System (STARS)

STARS is a joint FAA and Department of Defense (DOD) program. It will replace aging legacy terminal FAA and DOD automation systems with terminal ATC systems. Civil and military air traffic controllers across the nation are using STARS to direct aircraft near major airports.

In June 2003, FAA commissioned STARS for use at the Philadelphia International Airport in Pennsylvania. Currently, STARS is fully operational at 24 FAA terminal radar control facilities and 17 DOD facilities. Under the ATO’s new business model of breaking large and complex programs into smaller phases to control cost and schedule, STARS is a candidate for further deployment to about 120 FAA and DOD operational facilities. In May 2004, FAA changed STARS’s cost and schedule estimates for the third time and estimates that it will cost $1.46 billion to deploy STARS at 50 operational facilities.

Contractor: Raytheon.

Projected date for last deployment (F&E)

FAA’s Facilities and Equipment (F&E) account funds capital projects.

Certification issues – FAA also experienced problems in certifying STARS, in part because of aggressive scheduling. FAA’s approach to certifying STARS was oriented to rapid deployment to meet critical needs. To meet these needs, FAA compressed its original 32-month development and testing schedule into 25 months. This compressed schedule left only limited time for human factor evaluations and not enough time for computer human interface issues and involvement of controllers and maintenance technicians.

Airport Surveillance Radar Model–11 (ASR-11)

ASR-11 will provide high-quality digital data to terminal controllers in terminal environments. It will also provide a more reliable replacement for aging analog radars like ASR-7 and ASR-8; it will also provide digitized radar data for the new automation systems such as STARS. In addition, ASR-11 will provide six levels of weather information, a significant improvement over the current two levels. The ASR-11 program is a joint program with DOD—that is, DOD is managing the program to joint specifications, and FAA will provide DOD with the funds to procure 112 units. ASR-11 is a nondevelopmental item.

The in-service decision was made in 2003, and the radar is being deployed to 108 sites. The ASR-11 program is scheduled to be rebaselined for cost and schedule in fiscal year 2005.

Contractor: Raytheon.

Estimated cost (F&E)

The Capital Investment Plan does not support the service as required in the current Acquisition Program Baseline, which could put the program in jeopardy.

Integrated Terminal Weather System (ITWS)

ITWS provides automated weather information for use by air traffic controllers and supervisors in airport terminal airspace (60 miles around the airport.) It provides products that require no meteorological interpretation to air traffic controllers, air traffic managers, pilots, and airlines. ITWS provides a comprehensive current weather situation and highly accurate forecasts of expected weather conditions for the next 30 minutes.

Current FAA plans call for the installation of 34 systems that will service various airports. Six systems are operational, and feedback from users is satisfactory. In May 2004, the ATO Executive Council rebaselined the program to include a weather-forecasting capability in the production baseline, a new requirement to provide operational support for the New York prototype, and change the operations and maintenance cost baseline for the program. However, the council did not include additional funding, and therefore, in order to stay within the capital improvement program’s (CIP) funding levels, the program has proposed to defer 12 of the planned 34 systems installations.

Contractor: Raytheon.

Projected date for last deployment (F&E)

Jan 03 – Jul 03 Funding issues –The program requested and obtained approval to rebaseline. The baseline is being modified to incorporate the Terminal Convective Weather Forecasting (TCWF) capability into the production baseline. As directed by the ATO Executive Council, responsibility for funding operational support for the New York prototype system is also being added to the baseline. The ATO Executive Council also directed that the cost of the program remain at the current CIP funding levels for fiscal years 2005, 2006, and 2007. In order to stay within the CIP funding levels, the program proposed to defer 12 of the planned 34 systems installations.

Schedule issues – Because of constrained funding, 12 airports will not receive ITWS capabilities until after 2009.

Local Area Augmentation System (LAAS)

LAAS is a precision approach and landing system that will augment the Global Positioning System (GPS) to broadcast highly accurate information to aircraft on the final phases of a flight. LAAS consists of both ground and avionics components. Ground components include GPS reference receivers, which monitor and track GPS signals; very-high-frequency transmitters for broadcasting the LAAS signal to aircraft; and ground station equipment, which generates precision approach data and is housed at or near an airport. Aircraft will be equipped with avionics to receive LAAS signals.

FAA’s fiscal year 2005 budget eliminated funding for LAAS, and remaining fiscal year 2004 funds will continue to validate LAAS requirements and address radio frequency interference issues. FAA officials will reconsider national deployment when more research results are completed.

Estimated cost (F&E) $530.1 million $696.1 million Cost adequate requirements development in the early stages of the program, a lack of under of a mission software development, and an unrealistic development schedule. standing of a mission degradation issue, incomplete software development, and an Cost issues – LAAS cost estimates are not reliable, reflecting inadequate requirements development in the early stages of the program, a lack of understanding of a mission degradation issue, incomplete software development, and an unrealistic development schedule.

Schedule issues – The LAAS schedule was not realistic. Specifically, FAA lacked an understanding of the integrity requirement and software development, which were the two biggest technological maturity issues facing the LAAS program.

Performance Issues – FAA has not resolved the integrity requirement that ensures pilots are alerted in a timely manner when the LAAS signal is not reliable. FAA has not been able to prove that the system is safe during solar storms. An analysis of the effects of solar storms on the LAAS signal’s integrity is under way, but an atmospheric monitoring device that could address this issue may not be available until fiscal year 2009.

Next Generation Air/Ground Communications (NEXCOM)

The Next Generation Air/Ground Communications (NEXCOM) project is to replace the existing analog ATC communications system with a new digital system that would have greater capabilities. The initial development, of a multimodal digital radio (MDR), is to be followed by the development of aircraft avionics and ground systems. NEXCOM is expected to increase the number of available communications channels, provide simultaneous voice and data transmission between controllers and pilots, and require a digital form of authentication, designed to prevent “phantom controllers” from gaining access to the communications system. FAA plans to deploy 6,000 MDR pairs (a radio pair is one receiver and one transmitter) during the first phase, which will provide voice channels to aircraft in the en route environment.

NEXCOM completed Independent Operational Test and Evaluation assessment of the radio component at the Santa Barbara, California, Remote Center Air/Ground Communications facility, and radios were approved for in-service and national deployment in July 2004. The avionics component’s development is scheduled to be completed by 2006. However, proposed funding cuts to FAA’s fiscal year 2005 budget required the termination of the ground station development, which would enable communications in the more efficient digital mode.

Contractor: ITT for MDR.

Estimated cost (F&E) $318.40 million$318.40 million $0 Estimated cost is only for the NEXCOM MDR. The NEXCOM ground station contract was canceled in March 2004 and is being terminated.

Risks and Challenges Schedule issues—FAA planned to base the MDR on a nondevelopmental item (NDI), and the initial schedule allowed only limited development. However, FAA’s requirement that communications channels be free of signal interference (“quiet channels”) was more demanding than the NDI solution was capable of achieving. As a result, further development was necessary, delaying the initial operational capability and in-service decision by 21 months.

Performance—The NEXCOM radio meets its operational requirement for coverage. However, to achieve this requirement FAA determined that the NEXCOM radios would have to achieve the same power output level (50 watts) that the existing radios produced. The contractor is delivering radios that put out no more than 34 watts per channel. This posed an “unacceptable consequence” and FAA performed additional tests or flights checks and determined that the reduced power would not adversely affect operations and has approved the use of the lower-output radios.

Advanced Technologies and Oceanic Procedures (ATOP)

The Advanced Technologies and Oceanic Procedures (ATOP) program introduces new controller workstations, data-processing equipment, and software designed to enhance the control and flow of oceanic air traffic to and from the United States. ATOP processes aircraft position updates automatically, whereas currently, oceanic traffic control operations are performed manually and updated via paper flight strips. ATOP is designed to present flight data “electronically” in a format similar to these paper strips.

ATOP completed operational testing at its first site, the Oakland Air Route Traffic Control Center (ARTCC) and achieved initial operational capability (IOC) on June 30, 2004. Currently, ATOP is in limited use for 4 hours a day 5 days a week in one of nine sectors under Oakland’s control. Plans to fully transition ATOP to all nine sectors depend upon feedback from the initial trials and sector-by- sector capabilities. Other operational considerations still to be resolved are additional staff needs, ATOP’s training schedule, and coordination with North American Aerospace Defense Command on an interface device. FAA is currently in the early phases of installing ATOP at the New York ARTCC and is scheduled to achieve IOC in March 2005. Additional software that will incorporate radar data into ATOP is under development and scheduled to be completed by November 2004. This software is expected to be operational at the final site, the Anchorage ARTCC, in March 2006.

Contractor: Lockheed Martin Transportation and Security Solutions.

Risks and Challenges Cost issues—Although the contractor’s costs to develop ATOP have grown by approximately $20 million, FAA is not responsible for these cost increases because it has a fixed-price contract arrangement with the contractor.

Schedule issues— ATOP achieved its initial operational capability milestone of June 2004 but a more aggressive development schedule was agreed to with the ATOP contractor to achieve this milestone by April 2003 or 14 months earlier. An ATOP Assessment Team determined that the contractor could not achieve this earlier date due poor requirements development, unrealistic schedule estimates, and inadequate evaluation by the contractor of the software complexity. The development delay has exacerbated the scheduled transition from the current oceanic system to the ATOP and would cost an additional $4 million a year to operate and maintain the old system until ATOP is fully operational. Program officials told us they were not certain when the transition could be achieved because several operational issues needed to be resolved including ATOP operational trials sector by sector, training schedule, and filling new controller positions, and budgetary allocations to support these activities.

Performance – ATOP achieved initial operational capability in June 2004 and is limited basis in one of nine sectors of the Oakland Air Route Traffic Control Center.

En Route Automation Modernization (ERAM)

The En Route Automation Modernization (ERAM) program will enable air traffic controllers to provide ATC services to users of en route airspace (generally, high-altitude airspace at 10,000 feet or above). Services provided to users include separation, routing, and advisory services needed to meet FAA’s mission of providing safe, efficient, and reliable air traffic management. Specifically, ERAM is to replace the hardware and software in the current en route Host computer system, the direct-access radar channel, and associated infrastructure. This replacement will result in the installation of new system en route automation architecture at each air route traffic control center (ARTCC). In concert with other en route programs, ERAM will modernize the en route infrastructure to provide a supportable, open-standards-based system that will be the basis for future capabilities and enhancements. ERAM is to be deployed at 20 ARTCCs in the continental United States.

FAA awarded a letter contract to Lockheed Martin in December 2002. To date, ERAM has not breached any JRC cost or schedule parameters. However, the ERAM program is highly software intensive, requiring the writing of over 1 million lines of software code. In addition, Lockheed Martin is behind schedule because of software design and production control issues that Lockheed expects to resolve. Lockheed Martin officials stated that it does not expect any downstream impact from the current negative schedule variance of about $1 million.

Contractor: Lockheed Martin Transportation and Security Solutions.

Projected date for last deployment (F&E and O&M)

Software Issues – Software development is one of ERAM's major risk items. The ERAM program is a high-risk effort because of its size and the amount of software code – over 1 million lines of software code expected. Lockheed Martin is experiencing cost variances because of software engineering difficulties. According to its cost performance report, software engineering costs are being hampered by lower productivity than originally planned and by software code growth across the program. However, according to FAA officials, these additional software development costs can be easily absorbed within the contractor’s management reserve that is currently on the contract.

Key Contributors

GAO Contacts

Staff Acknowledgments

In addition to the individuals named above, Tamera Dorland, Elizabeth Eisenstadt, Brandon Haller, Bert Japikse, Carolyn Kirby, Steve Martinez, Richard Scott, Adam Vodraska, and Dale Yuge made key contributions to this report.