Air Traffic Control: FAA's Acquisition Management Has Improved, but Policies and Oversight Need Strengthening to Help Ensure Results
Published: 2004-11-12T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Federal Aviation Administration's (FAA) multibillion-dollar effort to modernize the nation's air traffic control (ATC) system has resulted in cost, schedule, and performance shortfalls for over two decades and has been on GAO's list of high-risk federal programs since 1995. According to FAA, performance shortfalls were due, in part, to restrictions imposed by federal acquisition and personnel regulations. In response, Congress granted FAA exemptions in 1995 and directed it to develop a new acquisition management system. In this report, GAO compared FAA's AMS with (1) the FAR and (2) commercial best practices for major acquisitions, and (3) examined FAA's implementation of AMS and its progress in resolving problems with major acquisitions.
section_title: What GAO Found
paragraphs: FAA's Acquisition Management System (AMS) is broader and less prescriptive than the Federal Acquisition Regulation (FAR), but both afford managers flexibility. AMS establishes an acquisition life-cycle management system, including both a contracting and program management system, whereas the FAR is primarily a contracting system. In addition, AMS takes the form of guidance--it is not regulatory, while the FAR is a set of published regulations--a legal foundation that has the force and effect of law that most federal agencies are required to follow. AMS provides some discipline for acquiring major ATC systems; however, it does not ensure a knowledge-based approach to acquisition found in the best commercial practices for managing commercial and DOD product developments that we have identified in numerous past reports. Best practices call for (1) use of explicit written criteria to attain specific knowledge at key decision points and (2) use of this knowledge by executives at the corporate level to determine whether a product is ready to move forward. Attainment and use of such knowledge by executives helps to avoid cost, schedule, and performance shortfalls that can occur if they commit to a system design prematurely. While AMS has some good features, including calling for key decision points, it falls short of best practices. GAO's review of seven major ATC systems and analysis of FAA's performance in acquiring major systems found that AMS has not resolved longstanding problems it experienced prior to its implementation of AMS--including developing requirements and managing software--and is just beginning to focus on how these acquisitions will improve the efficiency of ATC operations. While FAA has made progress by providing guidance for avoiding past weaknesses, it has not applied these improvements consistently. According to FAA officials, reorganization under and improved oversight by FAA's new performance-based Air Traffic Organization should help ensure greater consistency and an increased focus on results. Past GAO reports have demonstrated that the success of an acquisition process depends on good management, whether it be under AMS or the FAR.

section_title: Letter
paragraphs: In late 1981, the Federal Aviation Administration (FAA) began a  modernization program to replace and upgrade the National Airspace  System’s (NAS) equipment and facilities to meet the expected increase in  traffic volume, enhance the margin of safety, and increase the efficiency of  the air traffic control (ATC) system—the principal component of the NAS.  Historically, the modernization program has experienced cost overruns,  schedule delays, and performance shortfalls of large proportions and has  been on our list of high-risk programs since 1995. To date, FAA has spent  $41 billion and expects to spend an additional $7.6 billion through fiscal  year 2007 to, among other things, finalize key modernization projects  designed to replace radar, navigation, communications, and information- processing systems.
According to FAA, the performance shortfalls in its modernization program  were due, in part, to restrictions imposed by federal acquisition and  personnel requirements. In response, Congress passed legislation in 1995  that granted FAA unique acquisition and personnel exemptions, or  flexibilities, and directed FAA to develop a new acquisition management  policy. FAA issued its new acquisition management policy, called the  Acquisition Management System (AMS), in 1996 and began using the new  system instead of the Federal Acquisition Regulation (FAR). To further  address long-standing weaknesses in the ATC modernization program, the  President and Congress in 2000 directed FAA to reorganize and establish a  new organization. FAA has just begun to do so.
Now that FAA has had several years to implement the earlier procurement  flexibilities, as well as some time to reorganize, some results of its  acquisition reform should be discernable. Moreover, FAA’s experiences in  exercising its acquisition flexibilities could provide valuable information to  Congress in overseeing the use of these flexibilities.
You asked us to review the steps that FAA has taken to reform its  acquisition of major ATC systems and the impact of the reforms on FAA’s  acquisition outcomes. Specifically, you asked us to (1) compare the scope  and flexibility of AMS and the FAR, (2) compare AMS with commercial best  practices for major acquisitions, and (3) examine FAA’s implementation of  AMS and progress in addressing long-standing problems with major  acquisitions. In addition, you asked us to review FAA’s general procurement  of goods and services; we cover this topic in appendix I.
To address the first objective, we compared the topics addressed by, and  the implementation options afforded to contracting and procurement  officials under AMS and the FAR. To address the second objective, we used  a model of best practices that we derived from our body of work on how  leading private firms manage costly and complex product developments  and how the Department of Defense (DOD) manages major weapon  systems acquisitions. We used this model to assess the extent to which  FAA’s acquisition management policy mirrors the acquisition policies of  high-performing organizations in the public and private sectors. This model  consists of four phases: (1) concept and technology development; (2)  product development, which includes both integration and demonstration  activities; (3) production; and (4) operations and support. In between these  four phases are three key knowledge decision points at which commercial  firms and the government must have sufficient knowledge to make large  investment decisions. To address the third objective, we selected the seven  ATC systems with the largest budgets to explore the results of FAA’s  implementation of its acquisition management policy and procedures and  to determine how FAA has addressed issues found to have contributed to  cost, schedule, or performance problems. In selecting these seven systems,  we ensured that some were initiated before and some after April 1996,  when FAA implemented AMS. While the results of these analyses are not  generalizable to all of FAA’s major ATC acquisitions, they indicate the  extent to which the agency has made progress in addressing long-standing  problems we have identified. To further assess both the implementation  and the impact of FAA’s acquisition reforms, we reviewed our work on  FAA’s major ATC acquisition efforts since 1996 as well as the work of the  Department of Transportation’s Office of Inspector General (DOTIG), FAA,  and others. We also reviewed the actions that FAA has taken to refine AMS  in response to internal and external reviews. Finally, to review FAA’s  procurement of goods and services across the agency, we used a  commercial best-practices model for taking a more strategic approach to  procurement, along with interviews with key agency officials, to determine  whether FAA has begun to analyze spending trends to identify  opportunities to leverage its buying power. We conducted our work from  December 2003 through November 2004 in accordance with generally  accepted government auditing standards. See appendix II for additional  information on our objectives, scope, and methodology.
subsections: section_title: Background
paragraphs: Maintaining that federal procurement requirements contributed to some of  its cost, schedule, and performance problems in the 1980s and early 1990s,  FAA sought a statutory exemption from the federal acquisition system,  including the FAR, and those parts of title 5 of the United States Code, parts  II and III, that govern federal civilian personnel management. According to  FAA, exemptions from these requirements would enable it to streamline its  acquisition processes, be more responsive to the airline industry’s needs,  and increase the efficiency of ATC operations while maintaining safety.  Congress enacted legislation in November 1995 that exempted FAA from  key federal procurement statutes and the FAR, and directed FAA to  develop a new acquisition management system. In response to these  legislative initiatives, FAA implemented a new, streamlined acquisition  process—the Acquisition Management System (AMS)— on April 1, 1996.
We developed a knowledge-based model of commercial best practices  based on our findings about how leading private firms manage costly and  complex acquisitions effectively—that is, within cost, schedule, and  performance targets. The use of this knowledge-based model has been  found to reduce the risks associated with developing products and increase  the likelihood of successful outcomes. The model divides the product  development cycle into four phases and related activities. Table 1 presents  these phases and activities and explains what takes place during each.
AMS provides guidance for selecting and overseeing investments over their  life cycle. Like our best practices model, it is divided into phases and  activities, although the divisions sometimes occur at different points. Table  2 summarizes AMS’s phases and activities.
To implement the new, performance-based organization for managing ATC  modernization and operations, as the President and Congress directed in  2000, FAA appointed a chief operating officer in August 2003 and formally  established the Air Traffic Organization (ATO) in February 2004. ATO,  under the direction of a six-member executive council, is now responsible  for further implementing acquisition reforms for major ATC systems.
section_title: AMS Is Broader and Less Prescriptive Than the FAR
paragraphs: AMS establishes an acquisition life-cycle management system that  encompasses both contracting and program management, whereas the  FAR is primarily a contracting system that focuses on contract formation  and contract administration. As a result, AMS is broader in scope than the  FAR. See figure 1. In addition, AMS takes the form of guidance. This  guidance is expressed in documentation of FAA policy, handbooks,  templates, flowcharts, forms, and standard contract language. It is not  regulatory. By contrast, the FAR is a set of published regulations—a legal  foundation that has the force and effect of law for the federal agencies that are required to follow it. Furthermore, the FAR is more detailed and  prescriptive in establishing contracting requirements and can require more  administrative involvement. This fundamental difference between AMS and  the FAR may suggest to some that AMS is more flexible. FAA personnel can  choose how to apply AMS’s provisions to a major acquisition. Nonetheless,  procurement officials under the FAR also have flexibility because the FAR  encourages innovation consistent with its direction (and other applicable  legal requirements), provides a wide selection of contracting solutions, and  permits contracting officials to choose the methods that they consider  most suitable for a given situation.
subsections: section_title: AMS Addresses Both Procurement and Project Management, Whereas the FAR Focuses Primarily and in Far Greater Detail on Procurement
paragraphs: AMS comprises six policy sections and five appendixes. The procurement  policy section of AMS covers a range of topics, including contract funding  and administration, contracting with small and disadvantaged businesses,  and compliance with labor laws. According to this section, competition is  FAA’s preferred method of contracting, but single-source contracting is  permitted when appropriate to fulfill the agency’s mission. This policy  section also describes the procurement of commercially available or  nondevelopmental items.
Other sections of AMS cover project management tools that the FAR does  not address, such as investment analysis, configuration management, and  integrated logistics support. AMS also addresses areas that fall outside  project management and procurement, including real property  management—an area that becomes important when FAA must lease or  purchase real property so that it can install ATC systems such as radars or  antennas on property that it does not currently own. FAA’s policy directs  FAA staff to “conduct this business in a fair and equitable manner following  best practices.”
Although the FAR includes requirements that address procurement  planning and major systems acquisition, it does so only in the context of  government procurement policy and procedure. Agencies subject to the  FAR find the broader program planning requirements, which appear in  AMS but not in the FAR, in documents such as the Office of Management  and Budget’s Circular A-109 and in their own planning guidance. For example, DOD has issued a series of directives and instructions on this  subject.
The contracting procedures set forth in section 3 of AMS do not prescribe  detailed contracting procedures for various categories of procurements, as  do those detailed under the FAR. Instead, AMS provides two basic  contracting models for obtaining products and services through FAA’s  contracting process. The first model is called “Complex and  Noncommercial Source Selection” and is used for complex, large-dollar,  developmental, noncommercial items and services. This is the model that  typically would be used for investments approved by the Joint Resources  Council. The second model is called the “Commercial and Simplified  Purchase Method” and is typically used for commercial items that are less  complex and less costly. Procurements of such products or services may be  routine in nature and are generally purchased on a fixed-price basis.  Generally, source selection under AMS follows a screening process, with  the awardee being selected on a “best value” basis from among those who  remain in consideration when the selection is made.
section_title: AMS Provides Broad Guidance While the FAR Establishes Detailed Requirements, but Managers Have Flexibility under Both
paragraphs: AMS sets out a nonregulatory FAA policy that is binding on FAA personnel  as FAA employees. AMS also sets out other guidelines that FAA states  should be followed unless there is a rational basis for doing otherwise.  AMS is subject to such internal controls and enforcement as the  Administrator decides and to general overarching legal requirements, such  as the Government Performance and Results Act of 1993 (GPRA).FAA has  also deemed certain acquisition laws applicable to its procurements  (sometimes with modifications), such as the Service Contract Act. There  is also a legal requirement, created by the 1995 legislation exempting FAA  from the FAR, that small and socially or economically disadvantaged firms  be given all reasonable opportunities to receive contract awards. FAA has  adopted a dispute resolution process with some legal underpinnings.  Otherwise, as the preface to AMS states, “nothing in this document creates  or conveys any substantive  rights.” In short, although FAA is subject  to the general legal requirement that government decisions cannot be  arbitrary or capricious, AMS does not establish regulatory requirements for  the conduct of procurements and does not create or convey substantive  legal rights.
In contrast to AMS, the FAR is a set of published regulatory requirements. It  has the force and effect of law, and agencies that are subject to it are bound  to follow it. The FAR’s requirements provide for a range of procurement  strategies and approaches. In addition to negotiated procurement methods,  it allows two-step sealed-bid and two-phase design-build methods, among  others. It includes streamlined procedures for soliciting and evaluating  offers to furnish commercial items, as well as permits the use of simplified  acquisition procedures in a broad range of procurements. Furthermore, the  FAR supports a diverse selection of available contract types, product- testing tools, and other tools that an agency’s contracting personnel may  select when conducting an acquisition to meet the agency’s needs.
Although contracting personnel in agencies subject to the FAR are required  to comply with it, they enjoy broad discretion in their management of  procurements. For example, the FAR allows wide latitude in drafting  requirements statements, from performance-based statements of work to  design specifications as necessary. It allows broad discretion in framing  solicitations and in conducting procurements, including scoring proposals,  determining how negotiations will be conducted, eliminating firms whose  proposals are not competitive, and selecting the awardees whose proposals  afford the government the best value when evaluated against the selection  criteria established in the solicitations.
Because AMS consists of broad guidance while the FAR comprises detailed  and prescriptive regulatory requirements, FAA managers view AMS as  giving them more flexibility than they would have under the FAR,  particularly in two areas—competition and oversight. Whereas the FAR  generally requires full and open competition, AMS calls for providing  “reasonable access to” competition to firms interested in obtaining  contracts—a less rigorous standard than full and open competition. AMS  further states that the “preferred” method of selecting sources is to  compete requirements among two or more sources. By contrast, full and  open competition requires that all responsible sources be permitted to  compete. Under AMS, there is no policy that firms that want to participate  actually get a chance to do so. Rather, FAA told us that its system is  beneficial because the agency can use screening requests to preselect  competing firms, eliminating those firms that FAA believes are not likely to  receive an award. The following example illustrates the differences  between AMS and the FAR in their respective requirements on exceptions  to competition. FAA may contract with a single source when this approach  is determined to be in the best interest of FAA. The FAR, however, allows  exceptions to full and open competition only for certain specified  conditions (such as unusual and compelling urgency or the availability of  only one source). The FAR describes in detail the circumstances of these  conditions and the requirements for using them as justification for not  providing for full and open competition. The FAR also requires the  contracting officer to prepare a justification document that must generally  be approved by higher-level agency procurement officials (up to the  agency’s senior procurement executive) depending on the estimated dollar  value of the procurement. The content of this justification is prescribed by  the FAR. When not providing for full and open competition, the contracting  officer is required under the FAR to solicit offers from as many potential  sources as is practicable under the circumstances. The FAR prohibits  contracting if the justification for less than full and open competition  results from a lack of advanced planning. For a more detailed comparison  of AMS and the FAR, see appendix III.
Although some of the FAA personnel we interviewed see AMS as more  efficient and flexible than the FAR, other current and former FAA  procurement officials we interviewed who have experience using both the  FAR and AMS did not agree that AMS is more flexible than the FAR.  According to these officials, the FAR may appear inflexible and  cumbersome to persons who lack experience with it, but those who are  familiar with it are able to navigate its complexities effectively. The FAR  requires full and open competition, but as experienced procurement  personnel know, the system does not break down when emergencies  necessitate quick and decisive action. For example, we recently reported  that agencies generally complied with applicable FAR requirements in  awarding new contracts for work in Iraq using other than full and open  competition.In some circumstances, the government’s legitimate need for  prompt action was sufficient to justify selecting a contractor on an  expedited basis from among the firms that appeared able to meet the  government’s emergency need. In other cases, the agencies reasonably  determined that only one source could meet their requirements.
section_title: AMS Provides Some Discipline but Does Not Ensure a Knowledge-Based Approach to Acquisition
paragraphs: AMS provides some discipline through its various phases, activities, and  decision points for acquiring major ATC systems; however, it does not  ensure the use of a knowledge-based approach found in the best practices  for managing commercial product developments and DOD acquisitions that  we have identified in numerous past reports.Commercial best practices  call for specific knowledge to be captured and used by corporate-level  decision-makers to determine whether a product has reached a level of  development (product maturity) sufficient to demonstrate its readiness to  move forward in the acquisition process. The capture of such knowledge  and its use by executives helps to avoid cost overruns, schedule slips, and  performance shortfalls that can occur if decision-makers commit to a  system design before acquiring critical technology, design, or  manufacturing knowledge. The absence of these key best practices under  AMS puts FAA’s major ATC acquisitions at risk of cost overruns, schedule  slips, and performance shortfalls.
subsections: section_title: Best Practices for Managing Acquisitions Call for a Knowledge-Based Approach, Including Criteria for Knowledge Needed and Oversight at the Corporate Executive Level
paragraphs: Commercial best practices call for managing acquisitions using a  knowledge-based approach, including (1) using established criteria to  attain specific knowledge at three critical junctures in the acquisition cycle,  which we call knowledge points, and (2) requiring oversight at the  corporate executive level for each of these knowledge points. For example,  at each knowledge point, successful product developers apply specific  indicators, or criteria, to determine whether they have attained the  knowledge they need to move to the next phase or activity in the  acquisition process. Such developers also conduct corporate executive- level reviews to ensure that they obtain the insights and perspectives of  stakeholders throughout their organization. If the knowledge attained does  not meet the criteria for advancement or if the executive reviewers  determine that further development is inconsistent with their priorities, the  acquisition does not move forward. Table 3 summarizes the knowledge  points, criteria, oversight reviews, and timing of oversight reviews included  in our model of best practices for major acquisitions.
Experience with commercial best practices has shown that to the extent  that the level of knowledge called for at each knowledge point is not  attained, organizations take on risks in the form of unknowns that will  persist into the later stages of development, where they will take more time  and money to resolve if they become problems. Such problems lead to cost  increases and schedule delays.
section_title: AMS Has Some Good Features but Does Not Ensure That High Levels of Knowledge Are Attained Before Major Commitments Are Made
paragraphs: AMS has some good features, including phases and key decision points  indicative of an acquisition process that has some elements of discipline;  however, AMS does not ensure that high levels of knowledge are attained  and that corporate executive-level reviews occur before major  commitments of agency resources are made. For example, like the best  practices model, AMS identifies critical junctures, which it terms “decision  points.” Three of these decision points occur during the initial acquisition  phase (mission need, initial investment, and the final investment decision).
A fourth decision point occurs before production, and a fifth decision point  occurs before the start of the final acquisition phase (in-service  management). AMS also calls for detailed technical and programmatic  information that decision-makers can use at the first three decision points  to assess whether or not FAA should initiate an acquisition program. This  information includes a final requirements document, a final acquisition  program baseline, a final investment analysis report, an acquisition strategy  paper, and an integrated program plan. Finally, AMS, like our best practices  model, calls for senior executives to review the information and determine  whether the acquisition is ready to move forward. The FAA executives who  make the decisions at these points include associate and assistant  administrators, acquisition executives, the chief financial officer, the chief  information officer, and legal counsel; they form the Joint Resources  Council (JRC), FAA’s senior decision-making body for major ATC  acquisitions. Table 4 summarizes this information.
AMS departs from the best practices model in two key ways—it does not  call for high levels of knowledge to be attained at three critical junctures  (knowledge points), and does not call for corporate executive-level  oversight at one of five junctures. Specifically, AMS does not establish  explicit, written criteria for (1) the information needed to determine  technology maturity at solution implementation, (2) releasable drawings at  critical design review and production process controls at production. Our  best practices model calls for attaining specific knowledge and setting out  criteria for what information should be available to help organizations  minimize risks in the form of unknowns. Risks associated with such  unknowns can persist into the later stages of development, where they can  take more time and money to resolve if they become problems, potentially  leading to cost increases and schedule delays.
In addition, AMS does not provide for corporate executive-level oversight  reviews at two of the three key junctures where our best practices model  calls for such reviews. Although AMS calls for three Joint Resources  Council reviews during the initial acquisition phase—while our model calls  for a single corporate executive-level review—AMS allows the council to  delegate its oversight responsibility later in the acquisition process to the  program managers within the service organization responsible for an  acquisition. By contrast, our model calls for two corporate executive-level  reviews later in the acquisition process.
According to FAA, its approach gives program managers flexibility,  expedites decision-making, and allows the executives with the most  knowledge about a major acquisition to make key decisions about its  continued development. FAA’s reliance on this approach assumes that the  program managers will inform higher-level managers if they are unable to  meet the performance schedules and systems requirements approved by  the Joint Resources Council. However, although program managers may  have the most knowledge about their particular acquisition, they may not  have the agencywide perspective of the Joint Resources Council members.  Having an agencywide perspective, including a broad understanding of an  acquisition’s potential impact on other agency projects and operations, is  especially critical when an acquisition includes the production of multiple  units and requires a substantial commitment of agency resources, as do  FAA’s primarily multimillion-dollar acquisitions, such as controller  workstations and radars.
Because decisions about moving a major acquisition forward require both a  program manager’s specific knowledge of the acquisition itself and a senior  executive’s understanding of the acquisition’s potential impact on other  agency projects and operations, our best practices model calls for both  measurable criteria at key points in the acquisition process to ensure that  specific knowledge has been captured and corporate executive-level  reviews to ensure that senior decision-makers have the opportunity to  independently consider this knowledge. Without higher-level reviews such  as our best practices model recommends and the Joint Resources Council  could provide later as well as early in the acquisition process, FAA cannot  ensure that it has fully considered the impact of advancing an acquisition  on other agency projects and operations. This opportunity for full  consideration is a central advantage of managing acquisitions as a  portfolio, as we concluded in our August 2004 report on FAA’s information  technology investment management process.
Figure 2 contrasts FAA’s process for reviewing an acquisition’s progress  under AMS with the process that we found leads to successful commercial  acquisitions. !
To facilitate the comparison of AMS with out best practices model in this report, we have done the  following: (1) placed FAA's "Mission Analysis" and "Investment Analyses" activities in the "Needs and  Solution Identification" phase to make it comparable with the "concept and technology development"  phase in our best practices model; (2) depicted only the final investment decision point, recognizing  that the investment analysis phase includes an initial investment decision; and (3) placed “system  integration” and “system demonstration” in the solution implementation phase.
AMS does not explicitly call for a design review decision point, which would fall between system  integration and system demonstration.
To its credit, FAA continues to improve its AMS process. For example, the  agency is currently modifying its mission needs activity to make the  selection of major ATC acquisitions more consistent with the overall goals  of modernizing the National Airspace System. In addition, the Air Traffic  Organization has established an executive council to review major  acquisitions before they are sent to the Joint Resources Council. This  review is designed to screen acquisitions to determine which ones are  important enough to warrant higher-level review by the Council. Finally,  FAA is currently revising AMS to bring it in line with the Office of  Management and Budget’s guidance. Specifically, the agency is  incorporating OMB Exhibit 300, which provides the investment  justifications and management plans required for major ATC acquisitions.
section_title: As Implemented, AMS Has Not Resolved Long-standing Acquisition Problems, but FAA Is Beginning to Focus More on Results
paragraphs: According to our review of seven major ATC systems and analysis of FAA’s  performance in acquiring major systems, AMS has not resolved the long- standing problems that FAA experienced before implementing AMS, but  the agency is beginning to focus more on the expected results of its major  acquisitions. (See table 5.) Specifically, our review found that AMS  guidance did not call for requirements that were specific enough to  minimize requirements growth or unplanned work for five of these  systems. This lack of specificity resulted in the inadequate development or  definition of requirements, growth in requirements, unplanned work, or a  reduction in performance for five of these systems. In addition, for three of  these systems, FAA underestimated the difficulty of modifying available  software to fulfill its mission needs. Because AMS guidance was not  sufficient to account for the risks associated with modifying available  software, FAA encountered unexpected software development needs,  higher costs, and schedule delays. The two systems we reviewed that were  initiated after AMS was implemented are currently meeting cost and  schedule milestones; however, both systems are showing symptoms of  FAA’s past problems with developing requirements and managing software,  and it is too soon to tell if these programs will remain within their cost,  schedule, and performance parameters. In addition, our work on FAA’s  major acquisitions, along with that of the DOTIG and others has shown that  the problems FAA experienced before 1996 in acquiring major systems  persist under AMS and that effective acquisition management, rather than  the use of a specific contracting process (e.g., the FAR or AMS) is key to  successful acquisitions. To its credit, FAA is beginning to focus more on  results, largely through its new Air Traffic Organization, which has been  charged with taking a more performance-based approach to managing the  agency’s acquisitions.
subsections: section_title: Our Reviews of Seven Major Systems Show That Problems with Requirements and Software Management Persist under AMS
paragraphs: Our reviews of seven of FAA’s costliest ATC system acquisitions found that  the problems FAA experienced with requirements and software  management and their related impact on cost, schedule, and performance  goals persist today under AMS.Figure 3 identifies these seven acquisitions  and their milestones, which are expressed in terms of AMS decisions even  when the acquisitions were initiated before AMS was implemented. (See  app. V for a description and the status of each of these projects.)  Specifically, for 6 of these 7 major ATC acquisitions, FAA did not  consistently (1) clearly define system requirements at the investment  decision point or (2) adequately assess software complexity. Moreover, as  FAA has acknowledged, it has never managed its major acquisitions by  focusing on how each would improve the efficiency of ATC operations  while maintaining or improving safety. Although FAA has made progress in  improving its acquisition of major ATC systems—by, for example,  improving the maturity of its processes for acquiring software, using a  “build a little, test a little” approach to acquisitions as it did for Free Flight  Phase 1,and restructuring its organization to minimize stovepipes—long- standing problems persist in these areas. In addition, the two systems we  reviewed that were initiated after AMS’s implementation are currently  operating within cost and schedule goals; however, they are showing  symptoms of past problems with developing requirements and managing  software complexity. Moreover, our work for more than two decades— before and after AMS’s implementation—has cited these types of  weaknesses as central reasons for the agency’s long history of cost,  schedule, and performance shortfalls. This work has also found that the  effectiveness of an agency’s acquisition management has had a greater  impact on the success of its major acquisitions than the contracting  process used (e.g., the FAR or AMS).
section_title: Inadequate Development or Definition of Requirements Led to Requirements Growth or Unplanned Work for Five Acquisitions
paragraphs: For five of the seven acquisitions we reviewed, AMS guidance did not call  for requirements that were specific enough to minimize requirements  growth or unplanned work. For four of these five acquisitions—STARS,  LAAS, NEXCOM, and ATOP—incomplete and poorly defined requirements  in the final requirements documents, used at the investment decision point  to assess an acquisition’s readiness to enter the development phase, led to  requirements growth, unplanned development work, or a reduction in  system performance.For the fifth acquisition—ASR-11—FAA misjudged  the extent to which the high-level requirements that were used to support  the commercial-off-the-shelf/nondevelopmental item (COTS/NDI)  procurement by the Department of Defense could result in a product  capable of meeting FAA’s mission or user needs. As a result, unplanned  software changes were required.
FAA’s cost estimate for STARS has grown from its original estimate of  $0.94 billion in 1996 to $1.46 billion in 2004 and will deploy only 50 of the  172 STARS initially planned. Much of the cost growth has been due to  FAA requirements creep. As a result, the STARS program has  experienced delays of more than five years from its original plan, in part  due to added requirements to the commercial-off-the-shelf Initial  System Configuration (ISC). However, the STARS ISC was satisfactory  for use by the Department of Defense as deployed.
A final requirements document was approved, and the development of  LAAS was scheduled to begin in 1999. However, poorly established  requirements resulted in the addition of 113 new requirements to the  initial specification, entailing significant software and hardware  changes. Furthermore, LAAS may not achieve its promised capabilities  because FAA has been unable to develop technologies necessary to  warn pilots of a disruption in the LAAS signal. Until this technology is  developed, LAAS cannot be operated safely. As a result, FAA recently  cut the fiscal year 2005 funding for LAAS, and the program will revert to  a research and development effort.
FAA developed a final requirements document for the NEXCOM system,  but the requirements lacked the specificity needed to assess the  development risk. According to a NEXCOM contractor program official,  this led to miscommunication about the program requirement relating to  signal interference. This official stated that they misunderstood this  requirement and had not planned on the additional development work  for the NDI solution to meet such program objectives and delayed the  program 21 months. Another program requirement involved the  NEXCOM radios meeting or exceeding the operational coverage area of  the existing voice system. The existing radios had power output levels of  50 watts but the NEXCOM contractor could only achieve 34 watts of  power to meet the coverage requirement. A program official stated that  the contractor and FAA had not agreed on the testing procedures to  assess the power levels. This posed an “unacceptable consequence” and,  as a result, FAA performed additional testing or flight checks of the  reduced radio performance (50 watts versus 34 watts) and determined  that the performance reduction should not affect NEXCOM’s mission or  its coverage requirement.
FAA did not follow the AMS guidelines that call for completing a final  requirements document before proceeding to the development phase  for ATOP. The Joint Resources Council approved a delay in developing  the final requirements until after contract award. This decision resulted  in schedule delays and additional unplanned software development. The  ATOP program office asserted that the requirements remained very  stable and that the program is within cost and schedule objectives  established by the Council. However, FAA’s internal documents revealed  that the requirements were not adequately defined. For example, the  ATOP Investment Analysis Study reported to the Joint Resources  Council prior to contract award that the lack of more detailed ATOP  requirements at this stage of acquisition added risk and was of concern  to the investment analysis team. Under AMS, this team is responsible  for, among other things, conducting risk analyses for the various  acquisitions. Furthermore, an ATOP Assessment Team conducted a  study in March 2003 and determined that at the ATOP contract award,  “requirements were written at a high level and not mutually understood  by FAA and the contractor.” However, FAA management allowed the  ATOP program to proceed to solution implementation without the final  requirements document and, according to the contractor, this resulted  in schedule delays and growth in the amount of software needing  development.
The high-level requirements for ASR-11, jointly generated by FAA and  the Department of Defense, to support a COTS/NDI acquisition, resulted  in a product that did not initially meet the FAA mission or user needs.  The software changes that were required to meet FAA’s target detection  needs, as well as significant hardware design changes, parts  obsolescence, and production issues, added approximately two years to  system qualification and acceptance.
subsections: section_title: FAA Underestimated Software Complexity for Three Systems
paragraphs: For three of the seven major ATC acquisitions we reviewed—ITWS, LAAS,  and ATOP—FAA’s AMS guidance was not sufficient to address the risks  associated with modifying available software to fulfill FAA’s mission  needs. In all three cases, FAA officials underestimated the difficulty of  modifying available software. Our work has shown that underestimates are  likely to result in unexpected software development, higher costs, and  schedule delays.
ITWS experienced delays from the beginning because of the complexity  of its software development. Although the program appeared to be  progressing according to its baseline, immediately after the critical  design review in September 1998, the contractor revealed that it had  exceeded the target cost by $4 million. In addition, the contractor  claimed that the program did not recognize that the computer processor  originally planned for the program was becoming outdated, that the  manufacturer planned to discontinue its production because the market  was demanding a processor with greater processing and storage  capability, and that as a result, the original computer processor would  not be available to the program. Consequently, ITWS experienced cost  increases, schedule delays, and performance shortfalls. According to the  contractor and the original acquisition plan, all systems were scheduled  for delivery by December 2001, but that date has now stretched to after  2009.
LAAS’s technology maturity was not adequately assessed, and further  development was needed. Specifically, the potential for radio  interference through the atmosphere was not understood and could  limit LAAS’s operations. FAA has now placed all LAAS activities in  research and development. FAA did not adequately assess LAAS’s  software development. At the time of the contract award, the contractor  and FAA estimated that 80 percent of the software that LAAS required  had been developed. FAA later determined that only 20 percent had  been developed. FAA and the contractor attribute this discrepancy to a  lack of communication on the steps necessary to satisfy the program’s  requirements. FAA agrees that it should have conducted a software  audit and a software capabilities assessment, but pressures to keep  LAAS on schedule resulted in an inadequate assessment.
The ATOP contractor underestimated by about half the extent to which  legacy nondevelopmental item software, which is the core of the ATOP  system, met the program’s 1,036 requirements. As a result, a significant  amount of unanticipated new software code development and other  modifications were required.
section_title: ATC Systems Have Required Multiple Rebaselining Decisions to Address Delays and Cost Growth
paragraphs: As figure 3 illustrates, FAA initiated at least one rebaselining decision for  three of the five acquisitions that were begun before AMS was implemented  and were later transitioned to AMS. These rebaselining decisions  responded to delays and cost growth—problems that arise when  requirements are not stable, a program’s design is not fixed, or software  code growth is not controlled. For example, FAA rebaselined STARS two  times—first in 1999 and again in 2002. Similarly, 2 years after the  investment decision for ITWS, FAA rebaselined the program twice, in 1997  and again in 2001. Given the frequency of these past rebaselining decisions  for major ATC systems and the number of years that elapsed before or  between the rebaselining decisions (3 to 4 years), it is too soon to tell  whether the two systems that were initiated under AMS—ATOP and  ERAM—will require similar rebaselinings and ultimately meet their cost,  schedule, and performance goals. Although both programs are currently  operating within their cost and schedule goals and have not yet been  rebaselined, FAA has had problems with managing its major acquisitions in  the past and is currently having difficulties developing requirements and  managing software complexity. Furthermore, as we reported in May 2004,  FAA’s budget increased from $9 billion in 1998 to $14 billion in 2004 but will  be constrained for the foreseeable future. In such a constrained budget  environment, cost growth and schedule problems can have serious  negative consequences for ongoing modernization efforts—postponed  benefits, costly interim systems, delays in funding other systems, or  reductions in the number of units purchased.
section_title: Internal and External Reviews Have Found That FAA Has Made Some Progress but Continues to Experience Problems in Acquiring Major ATC System under AMS
paragraphs: Reviews of FAA’s acquisition process, conducted by FAA, GAO, the DOTIG,  and others have shown that FAA has improved its management of major  ATC acquisitions in recent years but continues to experience cost overruns,  schedule slips, and performance shortfalls under AMS. Table 7 summarizes  the results of 22 internal and external reviews of FAA’s major ATC  acquisitions. According to these reviews, issued from 1997 through 2004,  the same problems have persisted over many years, despite various  initiatives to address them, and FAA needs to strengthen its management  controls. For example, a key FAA review of eight major ATC acquisitions,  published in 1999, 3 years after AMS was implemented, found that these  acquisitions, though on track to meet their performance goals, were not  meeting their cost and schedule baselines. FAA attributed these cost and  schedule issues to new or poorly understood requirements, underestimates  of the acquisitions’ technical complexity, and funding shortfalls.
In addition, our reviews of major FAA acquisitions—initiated before and  after AMS was implemented—have found for more than two decades that  FAA’s failure to meet schedule, cost, and performance baselines for major  ATC acquisitions has been due to shortfalls in planning, weak management  controls, and a lack of systematic processes for acquiring new systems,  including inadequate requirements management, cost-accounting data, and  estimates of technical difficulty. As we reported in August 2004, judged  against the criteria of GAO’s framework for information technology (IT)  investment management, which measures the maturity of an organization’s  investment management processes, FAA has established about 80 percent  of the basic selection and control practices that it needs to manage its  mission-critical investments for the National Airspace System. For  example, FAA’s business units actively monitor projects throughout their life cycles.However, the agency’s senior IT investment board does not  regularly review investments that are in the “in-service management,” or  operational phase, and this creates a weakness in FAA’s ability to oversee  more than $1 billion of its IT investments. In addition, the agency has not  yet established the practices that would enable it to effectively manage its  annual IT budget of about $2.5 billion, and agency executives lack  assurance that they are selecting and managing the mix of investments that  best meets the agency’s needs and priorities. DOT has responded to our  recommendations to FAA to strengthen its IT investment management  capability.
Moreover, other reviews, such as those by Booz-Allen & Hamilton and  MITRE, have identified other shortfalls, which reflect a lack of proper  management controls and planning. For example, in 1997, Booz-Allen &  Hamilton found, among other things, that FAA had not clearly defined  organizational roles and responsibilities within the various phases of AMS  and that greater guidance and training under AMS were warranted. In 1999,  Booz-Allen & Hamilton reported that FAA had not demonstrated  improvement in adhering to planned costs and schedules under AMS and  that the agency needed to better manage its development of requirements  and address persistent funding shortfalls. Moreover, in 2001, a MITRE  report on selected major acquisitions found inadequate management  controls and deficiencies in both contractors’ performance and in FAA’s  measurement of acquisition performance. See table 7 for a chronological  listing of the reviews.
section_title: FAA’s ATO Is Taking Steps to Improve Major ATC Acquisitions
paragraphs: FAA’s recent reorganization, which brought ATC acquisitions and  operations together in the ATO, is expected to help the agency address  many of the concerns we have identified for more than two decades,  including those identified in this report. For example, the ATO is continuing  to develop and refine specific guidance for critical areas, such as  requirements management, software development, and cost estimation. In  addition, as the overseer of both ATC acquisitions and operations, the ATO  is in a position to facilitate more effective management of major ATC  acquisitions than has occurred in the past. The ATO is attempting, for  example, to link acquisition decisions directly with expected  improvements in operational efficiency without compromising safety. This  is important, given that FAA has spent about $2.5 billion on ATC  modernization per year since 1996 while operating costs have continued to  rise—from $4.6 billion to $7.5 billion over the past decade. FAA had not  completed its reorganization or implemented all of its initiatives at the time  of our audit.
subsections: section_title: Improvements to Requirements Development
paragraphs: With the establishment of the ATO, FAA consolidated requirements  development from two organizations (the organization sponsoring an  acquisition and the former agencywide acquisition organization) into a  single new organization—the Air Traffic System Requirements Service. In  addition, the ATO developed guidance to better manage requirements  during the middle phase of AMS (solution implementation). According to  FAA officials, some more complex development efforts may need to  develop systems requirements and a more detailed requirements document  than AMS currently calls for in the final requirements document. More  important, in January 2003, FAA issued guidance on requirements  management, Roles in Requirements Management During Solution  Implementation Phase, which provides for integrated requirements teams  that maintain responsibility for requirements management throughout an  acquisition’s life cycle. According to this guidance, when the final  requirements document is accepted by the Joint Resources Council at the  investment decision point, a requirements baseline is established and any  proposed changes to the requirements must be assessed for their impact on  the program and shown to be operationally suitable, affordable,  executable, and justifiable. An FAA official on an integrated requirements  team stated that any changes that may affect an acquisition’s cost and the  schedule require approval by the Executive Committee. The FAA official  also stated that this guidance has already helped to stabilize NEXCOM’s  requirements during the solution implementation phase. Other FAA  officials representing the Joint Resources Council acknowledged that the  guidance should ensure greater control over program requirements  growth, but said that not all program offices have consistently applied it.
section_title: Improvements to Managing Software and System Acquisition and Development
paragraphs: To better manage software programs for ATC modernization acquisitions,  FAA established a centralized process improvement office that reports to  the Chief Information Officer (CIO). This office developed an FAA  integrated capability maturity model (i-CMM), a software development and  management model that is similar to a model developed by Carnegie  Mellon University called the Capability Maturity Model Integration  (CMMI®), which is used to appraise the maturity of an organization’s  processes for acquiring software. However, FAA’s i-CMM goes beyond  Carnegie Mellon’s model to reflect international standards. The CMMI®  appraisal methodology calls for assessing process areas—such as project  planning, requirements management, and quality assurance—by  determining whether key practices are implemented and overarching goals  are satisfied. Both the i-CMM model and CMMI® appraisal methodologies  provide a logical framework for measuring and improving key processes  needed for achieving quality software and systems.
However, as we reported in August 2004, FAA projects are not required to  use the capability maturity model for process improvement, and individual  projects that use the i-CMM model are allowed to choose which process  areas they seek to improve and to determine when they are ready for an  appraisal of their progress. To date, fewer than half of FAA’s major ATC  projects have used this model. The recurring weaknesses we identified in  our project-specific evaluations are due in part to the flexibility these  projects were given in deciding whether and how to adopt this process  improvement initiative. Furthermore, after combining its ATC  organizations into a single performance-based organization (the ATO), FAA  is reconsidering prior policies, and it is not yet clear whether process  improvement will remain a priority. Without a strong senior-level  commitment to process improvement and a consistent, institutionalized  approach to implementing and evaluating it, FAA cannot ensure that key  projects will continue to improve systems acquisition and development  capabilities. As a result, FAA will continue to risk the project management  problems—including cost overruns, schedule delays, and performance  shortfalls—that have plagued past acquisitions. To address these  shortcomings, we recommended that the Secretary of Transportation  address specific weaknesses and institutionalize FAA’s process  improvement initiatives by establishing a policy and plans for  implementing and overseeing process improvement initiatives.
section_title: Improvements to Estimating Costs
paragraphs: FAA has taken steps to improve its cost estimation for major ATC projects  by issuing guidance on how to develop and use pricing under AMS. For  example, AMS policy calls for audit trails to record and explain the values  that are used as inputs to cost models. In addition, it calls for agency  officials, when reporting to executive oversight agencies and Congress, to  disclose the level of uncertainty and imprecision that are inherent in cost  estimates for major ATC systems. According to AMS policy, estimators  record the procedures, ground rules and assumptions, data, environment,  and events that underlie their development or update of a cost estimate.  This information supports the credibility of the cost estimate, aids in the  analysis of changes in program costs, enables reviewers to assess the cost  estimate effectively, and contributes to the population of FAA databases  that can be used for estimating the cost of future programs. Finally, despite  a delay of many years, FAA officials told us that they are in the final stages  of completing the agency’s cost-accounting system and plan to have it in  place across the agency by the end of this calendar year, which will bring  FAA into compliance with the Federal Managers' Financial Integrity Act of  1982. This measure will help reduce the likelihood of cost overruns or  improper payments for unallowable costs and provide decision-makers  with critical information. As we have reported in the past, a cost- accounting system is critical to managing major ATC acquisitions, because  without it, FAA lacks the information it needs to reliably estimate operating  costs over an acquisition’s life cycle.
section_title: Other Improvement Efforts
paragraphs: In May 2004, the FAA Administrator testified to Congress that, to date, in  attempting to improve the efficiency of ATC operations while maintaining  safety, FAA had not managed its major ATC acquisitions to be aware of  their cost implications for its operations. The Administrator said, however,  that the agency was taking its first steps to fundamentally change how it  makes acquisition decisions by adopting a more results-oriented approach.  Under this approach, the agency plans to link its decisions to fund major  acquisitions directly with their expected contribution to improving  operational efficiency and controlling escalating operating costs. Whereas,  in the past, FAA measured results in terms of its progress in completing and  deploying a major ATC system, it was now going to focus on how a given  system improved operational efficiency. Such an approach holds promise  for helping FAA more effectively manage its portfolio of major ATC  acquisitions by providing a sound basis for choosing among competing  priorities. However, because FAA has only recently begun to incorporate  this type of analysis of acquisitions’ costs and operational efficiency into its  decision-making and management processes, it is still too early to assess  the results.
In addition, to its credit, FAA has created a training framework for its  acquisition workforce, which we found mirrors human capital best  practices that we have identified. In January 2003, we reported on FAA’s  efforts to define and train its workforce to meet the requirements of the  Clinger-Cohen Act of 1996. This act required FAA and other civilian  agencies to establish education, training, and experience requirements for  their acquisition workforce. Our work on public and private best practices  has identified six elements of training as critical to acquisition. These  elements include (1) prioritizing the acquisition initiatives most important  to the agency, (2) securing top-level commitment and resources, (3)  identifying those who need training on specific initiatives, (4) tailoring  training to meet the needs of the workforce, (5) tracking training to ensure  it reaches the right people, and (6) measuring the effectiveness of training.   These six elements are crucial for successfully implementing acquisition  initiatives and reforms. Agencies that do not focus their attention on these  critical elements risk having an acquisition workforce that is ill equipped to  implement new processes.  The probability of success is higher if training is  well planned rather than left to chance. In 2003, we found that FAA’s model  for training its acquisition workforce largely mirrored public and private- sector best practices and that the agency had highly developed processes  for four of these six elements. See figure 4.
Since 2003, FAA has taken some steps to measure the effectiveness of its  training. For example, the agency collects and reviews participants’  assessments of the knowledge they have gained, the extent that learning  objectives were achieved and the applicability and usefulness of the  training. In addition, members of FAA’s Intellectual Capital Investment Plan  Council have attempted to make qualitative judgments about the impact  of the training on the effectiveness or efficiency of their organizations.  However, FAA is still developing an evaluation program with metrics to  measure the extent to which organizational goals are achieved when  individual training objectives are met. Industry and government experts  believe training and human capital investments are prerequisites for  successfully introducing and implementing effective acquisition best  practices. FAA’s acquisition workforce plays a critical role in addressing  long-standing weaknesses that we and others have identified with FAA’s  acquisition of major ATC systems. Given the importance of training for  acquisition workforces, it will be important for the ATO to put mechanisms  in place to comprehensively evaluate the effectiveness of the training it  provides to improve the knowledge base of FAA’s acquisition workforce.
To improve its investment management decision-making and oversight of  major ATC acquisitions, the ATO also initiated the following procedures:   Integrate AMS and the Office of Management and Budget’s Capital  Planning and Investment Control Processto develop a process for  analyzing, tracking, and evaluating the risks and results of all major  capital investments made by FAA.
Conduct Executive Council reviews of project breaches of 5 percent in  cost, schedule, and performance to better manage cost growth;  Issue monthly variance reports to upper management to keep them  apprised of cost and schedule trends.
Monitor progress in meeting the goals identified in FAA’s Flight Plan, the  agency’s blueprint for action through 2008. The Executive Council  tracks this progress monthly and reports to the Administrator, using a  color-coded system to keep her apprised of how well FAA is meeting its  goals. Green denotes that a goal will be met, yellow denotes that some  of the activities leading to a main goal may be in jeopardy but the overall  goal can be achieved, and red denotes serious concerns about reaching  a goal without major intervention. A formal progress report is issued  quarterly and made publicly available on the agency’s Web site; and  Increase the use of cost monitoring or earned value management  systems to improve oversight of programs.
Despite FAA’s current and planned efforts to improve its acquisition of  major ATC systems under the ATO, given the newness of these efforts and  the agency’s poor track record in this area for more than two decades, it is  critical for FAA to (1) modify AMS to more fully reflect the best practices  followed by high-performing acquisition organizations, (2) follow through  on planned improvement initiatives, and (3) adopt a continuous  improvement approach to acquiring new ATC systems.
section_title: Conclusions
paragraphs: In the early 1990s, FAA contended that it needed relief from the FAR to  remedy long-standing problems with cost, schedule, and performance  shortfalls in its major ATC acquisitions; however, our work for more than  two decades in this area has found that acquiring major ATC systems  successfully depends more on managing an acquisition process well than  on using a specific contracting process (e.g., the FAR or AMS). While our  recent work has shown some improvement in FAA’s management of major  ATC system acquisitions, some key problems that existed before 1996  persist under AMS—including difficulty with clearly defining system  requirements at the investment milestone and adequately assessing  complex software requirements. These problems continue to make these  acquisitions vulnerable to cost, schedule, and performance shortfalls.  Without further measures to improve the development and management of  requirements and to better estimate the complexity of the software  development needed for major ATC systems, such shortfalls are likely to  persist.
Although AMS provides some discipline for acquiring major ATC systems  through its various phases, activities, and decision points, it does not  require that (1) specific knowledge be attained using explicit written  criteria and (2) corporate executive-level oversight be provided to  determine—independently from the program offices—whether a system  has reached a level of development (product maturity) sufficient to move  forward in the acquisition process. Commercial best practices call for such  knowledge-based decision-making at the corporate executive-level to help  ensure that acquisitions are not moved into the development phase  prematurely, to obtain greater predictability in ATC system program costs  and schedules, to improve the quality of the ATC systems that are deployed,  and to deliver new capability to the National Airspace System faster. A  knowledge-based approach is also important because it provides assurance  that agency decision-makers have critical information about an  acquisition’s ability to meet a mission need and FAA’s readiness to move  forward in the acquisition process before making large commitments of  agency resources. Absent such an approach, FAA lacks assurance that it  has obtained the critical technological, design, or manufacturing  knowledge that best practices call for to avoid cost overruns, schedule  slips, and performance shortfalls. As a result, FAA is not doing all that it  can to systematically address persistent shortcomings in its management of  major ATC acquisitions. Moreover, although FAA has established a  framework for training its acquisition workforce under the ATO, it has not  yet developed comprehensive performance criteria to evaluate how  effectively it has implemented this framework. As a result, the agency lacks  assurance that its use of this framework is having the intended effect of  improving the knowledge base of this workforce.
section_title: Recommendations for Executive Action
paragraphs: We are making five recommendations to the Secretary of Transportation.  To reduce the risk of persistent cost and schedule shortfalls in major ATC  system acquisition programs, to improve the quality of the ATC systems  that are deployed, and to deliver new capability to the National Airspace  System faster, we recommend that the Secretary of Transportation advise  the FAA Administrator to take the four following actions:  Modify AMS to specify that requirements be more clearly defined for  major ATC systems, including providing more detailed guidance on  setting clear, objective, and measurable requirements that reflect  customers’ needs, before making large investments of agency resources.
Establish a strategy for identifying and measuring all additional  development needed for complex software (e.g., commercial-off-the- shelf or nondevelopmental items) used for major ATC systems.
Develop explicit written criteria for the key decision points called for  under best practices, including the capture of specific design and  manufacturing knowledge.
Require corporate executive-level decisions at these key decision points  (before an acquisition moves from integration to demonstration and,  again, before it moves to production).
In addition, to assure FAA that the training framework it has adopted for  the ATO’s acquisition workforce is improving the knowledge base of this  workforce as intended, we recommend that the Secretary advise the  Administrator to develop performance criteria to comprehensively  evaluate the framework’s effectiveness.
section_title: Agency Comments
paragraphs: We provided copies of a draft of this report to DOT for review and  comment and met with Department and FAA officials, including the ATO's  Vice President for Acquisition and Business Services, to obtain their  comments. FAA officials told us that they have made great strides in  improving their acquisition of major ATC systems under AMS; however,  they recognize that there is room for improvement and are firmly  committed to implementing best practices for acquisitions. These officials  generally agreed with the report's findings and conclusions and said that  our recommendations would be useful to them as they continue to refine  their acquisition management system, including training their acquisition  workforce. The agency provided us with oral comments, primarily  technical clarifications, which we have incorporated as appropriate.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to interested  congressional committees, the Secretary of Transportation, and the  Administrator, FAA. We will also make copies available to others upon  request. In addition, the report will be available at no charge on the GAO  Web site at http://www.gao.gov.
Please call me at (202) 512-2834 if you or your staff have any questions  concerning this report. Key contributors to this report are listed in  appendix VI.
section_title: FAA Has Begun Analyzing Spending Trends to Take a More Strategic Approach to Procurement
paragraphs: Our review of the Federal Aviation Administration’s (FAA) general  procurement of goods and services focused on the Air Traffic Organization  (ATO) and its predecessor offices. According to FAA officials, the ATO has  recently begun to consider ways to better leverage its buying power by  taking a more strategic approach to procurement. While FAA uses the  Acquisition Management System (AMS) for all FAA acquisitions, including  the procurement of such goods and services as office supplies, computers,  telephone services, and engineering and technical support services, these  procurement activities take place in a decentralized environment of  independent, transaction-oriented buying processes. Each FAA unit  determines its need for goods and services and procures them as necessary,  leaving headquarters with limited oversight of the agency’s total  procurement spending. For example, in 2003, FAA units carried out over  346,000 procurement actions for goods and services and purchase  cardholders made an additional 335,000 transactions. This fragmented  environment does not permit the agency to leverage its buying power  through lower-cost, consolidated contracts, at the local, regional, or  national level and to rationalize the number of suppliers best suited to meet  the agency’s needs. At the same time, as part of a strategic procurement  effort, FAA can use spend analysis to monitor trends in small and  disadvantaged business participation so that it can balance the goals of  lower-cost contract consolidation and promoting small business  contracting opportunities.
Spend analysis, a tool used in a strategic approach to procurement,  provides knowledge about how much is being spent for what goods and  services, who the buyers are, who the suppliers are, and where the  opportunities are to leverage buying power. Our past work shows that  private companies are using spend analysis as a foundation for employing a  strategic approach to procurement. The analysis identifies where  numerous suppliers are providing similar goods and services—often at  varying prices—and where purchasing costs can be reduced and  performance improved by better leveraging buying power and reducing the  number of suppliers to meet the company’s needs. Our research on  commercial best practices has found that spend analysis is an important  driver of strategic planning and execution. As part of an overall strategic  procurement effort, companies use spend analysis to (1) define the  magnitude and the characteristics of their spending, (2) understand their  internal clients and supply chain, (3) create lower-cost consolidated  contracts, and (4) monitor spending with small and disadvantaged  businesses to achieve socioeconomic procurement goals.
We previously reported that six agencies, including DOT, did not take  advantage of opportunities to obtain more favorable prices on purchase  card buys with frequently used vendors—vendors where an agency spends  more than $1 million annually. In these six agencies, which accounted for  over 85 percent of federal government purchase card spending, frequently  used vendors accounted for purchases totaling nearly $3 billion in 2002. We  recommended several actions—including conducting spend analysis using  available data and gathering additional information where feasible—that  could ultimately help these agencies achieve $300 million annually in  potential savings.
In fiscal year 2003, FAA procured nearly $4 billion in goods and services  and spent an additional $132 million using purchase cards. According to  senior FAA officials, the agency has just begun to implement a strategic  approach to general procurements. Other federal agencies are beginning to  use strategic tools such as spend analysis to improve their spending for  goods and services, and some have initiatives under way to obtain more  favorable prices on purchase card buys. According to a senior FAA  acquisition official, FAA has to balance the need of its units to  independently make purchases that pertain solely to unit requirements  with the agency’s need to aggregate purchases of goods and services that  are used by more than one unit. FAA has hired a consultant to help begin  the use of spend analysis. This effort could reduce the agencywide costs for  mobile wireless services by 40 percent—an effort expected to save the  agency $8 to $10 million annually. FAA intends to expand its use of spend  analysis to target other procurement category savings opportunities,  including information technologies, training, facilities, and professional  services, as its accounting systems improve.
FAA has taken some preliminary steps to set up a spend analysis program;  however, progress has been challenging for FAA because of deficiencies in  its accounting systems. For example, because the agency’s accounting  system did not identify all of the mobile wireless services for which it was  being billed, the contractor implementing the spend analysis had to obtain  this information from the wireless providers. FAA will need to expedite its  efforts in this area to fully realize potential savings. Our prior research has  shown that setting up a spend analysis program can be challenging.  Companies have had problems accumulating sufficient data from internal  financial systems that do not capture information on all of what a company  buys or is using in different, unconnected parts of the company. Despite  these challenges, companies that have developed formal, centralized spend  analysis programs have been able to track their costs and identify areas for  strategic sourcing and savings opportunities.
In our recent report on spend analysis, we found that DOT, at the time of  our review, had not yet begun to collect the data needed for a strategic  approach to procurement; however, the department is engaged in ongoing  efforts to improve procurements, and its top leadership is committed to  using spend analysis to change the way goods and services are purchased.  One obstacle to using spend analysis that the department cited during our  review was a lack of comprehensive and reliable spending data. However,  since we completed our review, the department reports stepping up efforts  to use currently available data and evaluate business intelligence software  to overcome those obstacles. In commenting on our report,  Transportation’s senior procurement executive told us that the department  is expanding its spend analysis efforts. For example, his office recently  reviewed purchase card spending data to identify volume discount  opportunities and is now using the results to negotiate new discount  agreements with several office product vendors. In addition, he told us that  to facilitate future agencywide purchase card spend analyses, DOT  awarded a task order in June 2004 to one bank card company that will  provide purchase-card audit software and enhanced data-mining  capabilities. He also indicated that the department’s leadership supports  fiscal year 2005 funding to enhance spend analysis capabilities and that  software options for the new agencywide spend analysis system are now  being evaluated as part of an ongoing financial and procurement review.
section_title: Objectives, Scope, and Methodology
paragraphs: To compare FAA’s Acquisition Management System (AMS) with the Federal  Acquisition Regulation (FAR), we reviewed AMS and changes in it over  time. We also compared FAA’s acquisition authority under the FAR and  under AMS. In addition, we identified relevant recommendations from  reports that we, the Department of Transportation’s Inspector General  (DOTIG), and others have issued to determine which recommendations  have been implemented, rejected, or left open, and to evaluate how those  recommendations have modified FAA’s acquisition policies and practices.  We also collected and summarized published reports and analyzed  available life-cycle management data on the current status of major and  nonmajor acquisitions being carried out under AMS.
To determine the ways in which FAA’s acquisition policies compare with  our best practices model, we used information from several of our  products that examine how commercial best practices can improve  outcomes for acquisition programs. This model consists of four phases: (1)  concept and technology development; (2) product development, which  includes both integration and demonstration activities; (3) production; and  (4) operations and support. In between these four phases are three key  knowledge points at which commercial firms must have sufficient  knowledge to make large investment decisions. We also reviewed and  analyzed AMS, accessible at http://fast.faa.gov. Furthermore, to clarify the  content of FAA’s acquisition process, we met with various FAA vice- presidents and officials from FAA's Acquisition Planning and Policy  Division. Next, we compared and contrasted FAA's acquisition policies  with the best practices for commercial acquisitions identified in our past  reports. Our analysis focused on whether FAA's policies contained the  measurable criteria and management controls necessary to achieve FAA's  intent of minimizing cost, schedule, and performance risks. We also  interviewed current and former FAA procurement officials that have  experience using both the FAR and AMS.
To determine if FAA has effectively implemented its new acquisition  authority and improved its acquisition outcomes, we reviewed seven of  FAA’s most expensive major ATC acquisitions, including the Airport  Surveillance Radar 11 (ASR-11), Standard Terminal Automation  Replacement System (STARS), Integrated Terminal Weather System  (ITWS), Local Area Augmentation System (LAAS), Next Generation  Air/Ground Communications System (NEXCOM), Advanced Technologies  and Oceanic Procedures (ATOP), and En Route Automation Modernization  (ERAM). See table 7 for specific program costs.
We also selected these seven acquisitions because we considered them to  fall into two basic categories-pre-AMS and post-AMS. Five of the  acquisitions were initiated before AMS was implemented in April 1996 and  were transitioned into AMS at various times before their completion. The  two remaining acquisitions—ATOP and ERAM—were initiated and have  remained completely under AMS. We then reviewed program documents  and reports and interviewed program and agency officials responsible for  developing these acquisitions, as well as other acquisitions experts in the  private sector. For some acquisitions, we discussed programmatic issues  with representatives of the primary contractor for the specific acquisition  to obtain information on the practices and procedures used for the  acquisition. In addition, we interviewed some current and former FAA  procurement officials with experience using both the FAR and AMS to  obtain their views on the use of each contracting process and how the two  compare. Furthermore, to see how FAA has progressed in addressing  problems with its acquisitions, we reviewed our work on acquisitions over  the last 20 years, as well as reports by the DOTIG, FAA, Booz-Allen &  Hamilton, and MITRE. Because the data in this report on cost, schedule  and performance are used as background information or to otherwise  provide a description of acquisitions, we did not assess their reliability.
The effect of the current budget process on FAA’s ability to successfully  modernize the National Airspace System, including acquiring major ATC  systems is not within the scope of this review.
section_title: Comparison of the Scope and Flexibility of FAA’s Acquisition Management System and the Federal Acquisition Regulation Process
subsections: section_title: Background
paragraphs: FAA’s business processes, including its acquisition of major systems, differ  significantly from the business processes followed by most other federal  agencies. FAA relies on its Acquisition Management System (AMS), which  establishes FAA internal acquisition policy. AMS resulted from the adoption  of language in the Department of Transportation and Related Agencies  Appropriations Act, which directed the FAA Administrator to develop and  implement an acquisition management system for FAA. The adoption of  this language (section 348) followed FAA’s assertions that the requirement  that it conduct procurements in accordance with the Federal Acquisition  Regulation (FAR) was at least a contributing factor in its repeated failure to  complete air traffic control (ATC) and other modernization programs on  schedule. The Administrator was directed to put in place a system that  would address the “unique needs of the agency” that FAA contended  prevented its acquisitions from being timely and cost-effective.
Section 348 distinguished FAA from other federal agencies by removing  FAA from the federal acquisition system. Under section 348, FAA was no  longer subject to title III of the Federal Property and Administrative  Services Act of 1949, which among other things requires that the  government procure supplies and services competitively. It removed FAA  as an agency subject to the Office of Federal Procurement Policy Act and  eliminated the requirement that FAA comply with the FAR. While  mandating that FAA conduct its acquisitions so that “all reasonable  opportunities to be awarded contracts shall be provided to small business  concerns and small business concerns owned and controlled by socially  and economically disadvantaged individuals,” section 348 eliminated the  requirement that FAA comply with the Small Business Act. Furthermore, it  made the procurement protest system of the U.S. Government  Accountability Office inapplicable to FAA, although disappointed offerors can still file protests with FAA’s Office of Dispute Resolution for  Acquisition.
section_title: AMS Defines an Investment/Life-Cycle Project Management System
paragraphs: Much of AMS guidance concerns project, financial, and property life-cycle  management issues. In fact, FAA’s policy describes AMS as applying to all  investment programs regardless of cost or the appropriation funding them.  It recognizes that a single investment program may span multiple  procurements and projects. It applies, according to its terms, to the  activities associated with needs analysis, determination of requirements,  analysis of investment alternatives, establishment of investment programs,  allocation and expenditure of resources, procurement and deployment of  needed products and services, in-service management of fielded capability,  and eventual disposal of obsolete products.
AMS focuses on the following key program milestones:  Mission Analysis—encompasses those key corporate and service-level  processes that define, coordinate, and integrate the work of service  organizations, thereby providing strategic direction to keep FAA  responsive to the service needs of its customers. Mission analysis is  used to update a mission need statement, which in turn may identify  capability shortfalls or technological opportunities, that is, unmet needs.  Unmet needs are presented to the Joint Resources Council (JRC) for a  mission need decision. To be approved, the unmet need should be  supported by the updated mission need statement and the initial  requirements document, including a concept of use, and the initial  investment plan.
Investment Analysis—builds on the results of the mission need decision  by developing detailed plans and final requirements for each proposed  investment program and by defining an acquisition program baseline  that establishes cost, schedule, performance, benefit, and risk- management boundaries for the program. AMS calls for planning the  entire solution—an effort that may use market survey data but is based  in large measure on FAA’s assumptions and data. The service  organization produces a final implementation and life-cycle support  strategy. A detailed program plan and an acquisition program baseline  are also produced. The results are presented to the JRC for a “final  investment decision.”
Solution Implementation—encompasses acquiring, accepting,  deploying, installing and preparing for the operational use of an  approved investment. Approval of the investment carries with it  authorization for the service organization to conduct all acquisitions  needed to execute the investment decision, subject to any constraints  established in the final investment decision.
In-Service Decision—is an FAA system qualification milestone, which is  achieved when an otherwise operational investment is satisfactorily  tested to demonstrate its operational effectiveness and suitability before  it is placed in service in the National Airspace System. The JRC  designates the decision maker.
In-Service Management—covers activities throughout a system’s life  cycle, starting at the time that an investment becomes operational. In- service product improvements may eliminate latent defects, fix systemic  problems, and enhance the utility of the investment. These changes may  be made within the approved acquisition program baseline without  corporate-level approval. In-service management also includes planning,  programming, and developing supporting budget input; monitoring and  assessing performance, cost of ownership, and support trends; and  planning for service-life investment decisions.
Service Life Extension—seeks a new investment decision by the JRC  when a current capability is unable to satisfy demand or when another  solution may be more effective. The JRC can decide to revalidate the  mission need satisfied by the solution by upgrading or refurbishing  fielded capability or by replacing that capability with another equivalent  or new superior solution. The JRC may also decide that the capability  should be retired.
section_title: Only a Portion of AMS Deals Directly with the Procurement Process
paragraphs: Although the FAR includes requirements addressing procurement planning  and major system acquisition, AMS as just outlined differs significantly  from the FAR in its focus and scope. The FAR addresses planning and  major system acquisition in the context of government procurement policy  and procedure. Agencies other than FAA find the broader program  planning and management issues addressed in AMS outside of the FAR, in  documents such as the Office of Management and Budget’s (OMB) Circular  A-109, in their own planning guidance, such as the Department of Defense’s  (DOD) 5000 series,and in established knowledge-based best practices. As  indicated earlier, much of AMS focuses on just such issues. Only AMS  section 3 addresses procurement policy and procedure.
section_title: AMS States a Nonregulatory FAA Policy
paragraphs: A further significant foundational difference between AMS and the FAR is  that AMS sets out a nonregulatory FAA policy, whereas the FAR was  adopted and is maintained as a set of published governmentwide regulatory  requirements, which form a legal basis for federal agencies’ contract  decision-making. AMS is binding on FAA personnel as FAA employees and  establishes other guidelines that FAA states should be followed unless  there is a rational basis for doing otherwise. AMS is subject to such internal  controls as the Administrator chooses to enforce and general overarching  legal requirements, such as the Government Performance and Results Act  of 1993 (GPRA). There is a legal requirement, created by section 348, that  small and socially or economically disadvantaged firms be given all  reasonable opportunities to receive contract awards. FAA in its Office of  Dispute Resolution for Acquisition has adopted a dispute resolution  process with some legal underpinnings. Otherwise, as the preface to AMS  states, “nothing in this document creates or conveys any substantive   rights.” In short, FAA has assumed no legal obligation to follow AMS other  than to ensure that its actions are not arbitrary and capricious or contrary  to law. By contrast, the FAR has the force and effect of law, and agencies  that are subject to the FAR are bound to follow it.
section_title: AMS Chapter 3 Parallels a Subset of the FAR
paragraphs: When FAA personnel apply the procurement methodology in AMS chapter  3, they are applying guidance that closely parallels some of the procedures  set out in the FAR. The AMS Chapter 3 acquisition process parallels a  subset of the varied selection of procurement methods available under the  FAR, requiring that all competitive FAA contracts be negotiated with the  awardee being selected on a “best value” basis. The FAR also provides a  much more detailed set of information and guidance than does AMS. A  comparison of high-level differences and similarities between AMS and the  FAR is presented in table 8.
section_title: AMS Includes a Less Rigorous Competition Requirement Than Does the FAR
paragraphs: As table 8 indicates, AMS incorporates a less rigorous competition  standard than the FAR imposes on the rest of the government. AMS states  that it is FAA's policy to provide reasonable access to competition for firms  interested in obtaining contracts. According to AMS, in selecting sources,  the preferred method of procurement is to compete requirements among  two or more sources. However, there is no requirement to ensure that firms  that want to participate actually get a chance to do so. Instead FAA may  limit competition for further consideration in its screening process to firms  with known capabilities or past performance.
section_title: The FAR Gives Procurement Professionals Tighter Control over Procurement Decisions
paragraphs: AMS states that authority is delegated to appropriate levels. Once the final  investment decision is made, and subject only to any constraints imposed  by that decision, the service-level organization is responsible for  conducting required acquisitions. Contracting personnel as well as other  specialists are then assigned to teams that are responsible to a program  manager within the service-level organization. FAA states that this  approach increases the pace of doing business. By comparison, the FAR  gives contracting professionals clear control over contracting decisions by  requiring that procurement decisions be made by procurement  professionals—typically contracting officers or their superiors.
section_title: Although FAA Project Managers View AMS as More Efficient and Flexible Than the FAR, Some Procurement Officials We Interviewed Do Not Agree
paragraphs: As part of our work, we interviewed project management personnel within  FAA as well as current and former FAA procurement officials that have  experience using both the FAR and AMS. Generally, FAA personnel see  AMS as more efficient and flexible than the FAR, although 9 years after  AMS’s adoption, many FAA officials have only limited knowledge of and  experience with the FAR. The FAA project managers we interviewed see  AMS as more efficient and flexible than the FAR, but some procurement  officials with experience in applying both AMS and the FAR did not agree  with the view that the FAR was unduly rigid. According to these officials,  the FAR may appear inflexible and cumbersome to persons who are  inexperienced with it, but those who are familiar with it are able to  navigate its complexities effectively. For example, even though the FAR  generally requires full and open competition—a process that can take time  to give all interested firms an opportunity to participate—contracting  officers may be able to expedite the procurement process by using  authorized streamlined procedures or, if circumstances warrant, by  justifying sole-source or limited competition.
section_title: How FAA’s Acquisition Policy Adapted Key Recommendations Made by GAO and DOT (1996-2003)
subsections: section_title: FAA Refined AMS in Response to Recommendations
paragraphs: Since FAA developed and implemented AMS in 1996, GAO and the DOTIG  have made recommendations to improve FAA’s acquisition processes.  FAA  has adopted many of these recommendations and incorporated them into  AMS (see table 9). These implemented recommendations address four  main themes:   Developing a strategy for culture change that relies on successfully  integrating the various elements of acquisition, including specific  responsibilities and performance measures for all stakeholders, and  providing the incentives needed to promote the desired changes.
Establishing an effective management structure for developing,  maintaining, and enforcing the ATC systems architecture to provide an  overall plan for the National Airspace System (NAS). This management  structure should assign the responsibility and accountability to develop,  maintain, and enforce a complete and unified ATC system by ensuring  that every project conforms to the overall plan.
Improving cost and schedule tracking to provide data for estimating the  costs and schedules of programs. To estimate the costs and time needed  for projects, a historical database that includes cost and schedule  estimates, revisions, reasons for revisions, actual cost and schedule  information, and relevant contextual information is needed.
Improving the management of modernization projects, including the use  of project reviews, milestones, and baselines, and cost-accounting  information to ensure that programs can be adjusted as needed.
The reports identified in table 10 provide recommendations to address  problems we and the DOTIG have identified under these four themes.
section_title: Status of the Seven ATC Modernization Acquisitions That GAO Reviewed
paragraphs: Standard Terminal Automation Replacement System (STARS)
STARS is a joint FAA and Department of Defense (DOD)  program.  It will replace aging legacy terminal FAA and  DOD automation systems with terminal ATC systems. Civil  and military air traffic controllers across the nation are using  STARS to direct aircraft near major airports.
In June 2003, FAA commissioned STARS for use at the  Philadelphia International Airport in Pennsylvania.  Currently, STARS is fully operational at 24 FAA terminal  radar control facilities and 17 DOD facilities. Under the  ATO’s new business model of breaking large and complex  programs into smaller phases to control cost and schedule,  STARS is a candidate for further deployment to about 120  FAA and DOD operational facilities.  In May 2004, FAA  changed STARS’s cost and schedule estimates for the third  time and estimates that it will cost $1.46 billion to deploy  STARS at 50 operational facilities.
Contractor: Raytheon.
Projected date for  last deployment   (F&E)
FAA’s Facilities and Equipment (F&E) account funds capital projects.
Certification issues – FAA also experienced problems in certifying STARS, in part because of  aggressive scheduling. FAA’s approach to certifying STARS was oriented to rapid deployment to meet  critical needs.  To meet these needs, FAA compressed its original 32-month development and testing  schedule into 25 months.  This compressed schedule left only limited time for human factor evaluations  and not enough time for computer human interface issues and involvement of controllers and  maintenance technicians.
Airport Surveillance Radar Model–11 (ASR-11)
ASR-11 will provide high-quality digital data to terminal controllers in terminal environments.  It will also   provide a more reliable replacement for aging analog radars like ASR-7 and ASR-8; it will also provide  digitized radar data for the new automation systems such as STARS.  In addition, ASR-11 will provide  six levels of weather information, a significant improvement over the current two levels.  The ASR-11  program is a joint program with DOD—that is, DOD is managing the program to joint specifications,  and FAA will provide DOD with the funds to procure 112 units.   ASR-11 is a nondevelopmental item.
The in-service decision was made in 2003, and the radar is being deployed to 108 sites. The ASR-11  program is scheduled to be rebaselined for cost and schedule in fiscal year 2005.
Contractor: Raytheon.
Estimated cost (F&E)
The Capital Investment Plan does not support the service as required in the current Acquisition  Program Baseline, which could put the program in jeopardy.
Integrated Terminal Weather System (ITWS)
ITWS provides automated weather information for use by air traffic controllers and supervisors in  airport terminal airspace (60 miles around the airport.) It provides products that require no  meteorological interpretation to air traffic controllers, air traffic managers, pilots, and airlines. ITWS  provides a comprehensive current weather situation and highly accurate forecasts of expected weather  conditions for the next 30 minutes.
Current FAA plans call for the installation of 34 systems that will service various airports.  Six systems  are operational, and feedback from users is satisfactory.  In May 2004, the ATO Executive Council  rebaselined the program to include a weather-forecasting capability in the production baseline, a new  requirement to provide operational support for the New York prototype, and change the operations and  maintenance cost baseline for the program.  However, the council did not include additional funding,  and therefore, in order to stay within the capital improvement program’s (CIP) funding levels, the  program has proposed to defer 12 of the planned 34 systems installations.
Contractor: Raytheon.
Projected date for  last deployment  (F&E)
Jan 03 – Jul 03  Funding issues –The program requested and obtained approval to rebaseline.  The baseline is being  modified to incorporate the Terminal Convective Weather Forecasting (TCWF) capability into the  production baseline.  As directed by the ATO Executive Council, responsibility for funding operational  support for the New York prototype system is also being added to the baseline.  The ATO Executive  Council also directed that the cost of the program remain at the current CIP funding levels for fiscal  years 2005, 2006, and 2007.  In order to stay within the CIP funding levels, the program proposed to  defer 12 of the planned 34 systems installations.
Schedule issues – Because of constrained funding, 12 airports will not receive ITWS capabilities until  after 2009.
Local Area Augmentation System (LAAS)
LAAS is a precision approach and landing system that will augment the Global Positioning System  (GPS) to broadcast highly accurate information to aircraft on the final phases of a flight. LAAS consists  of both ground and avionics components.  Ground components include GPS reference receivers, which  monitor and track GPS signals; very-high-frequency transmitters for broadcasting the LAAS signal to  aircraft; and ground station equipment, which generates precision approach data and is housed at or  near an airport.  Aircraft will be equipped with avionics to receive LAAS signals.
FAA’s fiscal year 2005 budget eliminated funding for LAAS, and remaining fiscal year 2004 funds will  continue to validate LAAS requirements and address radio frequency interference issues. FAA officials  will reconsider national deployment when more research results are completed.
Estimated cost  (F&E)  $530.1 million  $696.1 million  Cost adequate requirements development in the early stages of the program, a lack of under of a  mission software development, and an unrealistic development schedule.  standing of a mission  degradation issue, incomplete software development, and an  Cost issues – LAAS cost estimates are not reliable, reflecting inadequate requirements development in  the early stages of the program, a lack of understanding of a mission degradation issue, incomplete  software development, and an unrealistic development schedule.
Schedule issues – The LAAS schedule was not realistic.   Specifically, FAA lacked an understanding of  the integrity requirement and software development, which were the two biggest technological maturity  issues facing the LAAS program.
Performance Issues – FAA has not resolved the integrity requirement that ensures pilots are alerted in a  timely manner when the LAAS signal is not reliable.  FAA has not been able to prove that the system is  safe during solar storms.   An analysis of the effects of solar storms on the LAAS signal’s integrity is  under way, but an atmospheric monitoring device that could address this issue may not be available  until fiscal year 2009.
Next Generation Air/Ground Communications (NEXCOM)
The Next Generation Air/Ground Communications (NEXCOM) project is to replace the existing analog  ATC communications system with a new digital system that would have greater capabilities.  The initial  development, of a multimodal digital radio (MDR), is to be followed by the development of aircraft  avionics and ground systems.  NEXCOM is expected to increase the number of available  communications channels, provide simultaneous voice and data transmission between controllers and  pilots, and require a digital form of authentication, designed to prevent “phantom controllers” from  gaining access to the communications system.  FAA plans to deploy 6,000 MDR pairs (a radio pair is  one receiver and one transmitter) during the first phase, which will provide voice channels to aircraft in  the en route environment.
NEXCOM completed Independent Operational Test and Evaluation assessment of the radio  component at the Santa Barbara, California, Remote Center Air/Ground Communications facility, and  radios were approved for in-service and national deployment in July 2004.  The avionics component’s  development is scheduled to be completed by 2006.  However, proposed funding cuts to FAA’s fiscal  year 2005 budget required the termination of the ground station development, which would enable  communications in the more efficient digital mode.
Contractor: ITT for MDR.
Estimated cost  (F&E) $318.40 million$318.40 million  $0  Estimated cost is only for the NEXCOM MDR.  The NEXCOM ground station contract was canceled in  March 2004 and is being terminated.
Risks and Challenges  Schedule issues—FAA planned to base the MDR on a nondevelopmental item (NDI), and the initial  schedule allowed only limited development.  However, FAA’s requirement that communications  channels be free of signal interference (“quiet channels”) was more demanding than the NDI solution  was capable of achieving.  As a result, further development was necessary, delaying the initial  operational capability and in-service decision by 21 months.
Performance—The NEXCOM radio meets its operational requirement for coverage. However, to  achieve this requirement FAA determined that the NEXCOM radios would have to achieve the same  power output level (50 watts) that the existing radios produced.  The contractor is delivering radios that  put out no more than 34 watts per channel.  This posed an “unacceptable consequence” and FAA  performed additional tests or flights checks and determined that the reduced power would not  adversely affect operations and has approved the use of the lower-output radios.
Advanced Technologies and Oceanic Procedures (ATOP)
The Advanced Technologies and Oceanic Procedures (ATOP) program introduces new controller  workstations, data-processing equipment, and software designed to enhance the control and flow of  oceanic air traffic to and from the United States.  ATOP processes aircraft position updates  automatically, whereas currently, oceanic traffic control operations are performed manually and  updated via paper flight strips.  ATOP is designed to present flight data “electronically” in a format  similar to these paper strips.
ATOP completed operational testing at its first site, the Oakland Air Route Traffic Control Center  (ARTCC) and achieved initial operational capability (IOC) on June 30, 2004.  Currently, ATOP is in  limited use for 4 hours a day 5 days a week in one of nine sectors under Oakland’s control.  Plans to  fully transition ATOP to all nine sectors depend upon feedback from the initial trials and sector-by- sector capabilities.  Other operational considerations still to be resolved are additional staff needs,  ATOP’s training schedule, and coordination with North American Aerospace Defense Command on an  interface device.  FAA is currently in the early phases of installing ATOP at the New York ARTCC and  is scheduled to achieve IOC in March 2005.  Additional software that will incorporate radar data into  ATOP is under development and scheduled to be completed by November 2004.  This software is  expected to be operational at the final site, the Anchorage ARTCC, in March 2006.
Contractor: Lockheed Martin Transportation and Security Solutions.
Risks and Challenges  Cost issues—Although the contractor’s costs to develop ATOP have grown by approximately $20  million, FAA is not responsible for these cost increases because it has a fixed-price contract  arrangement with the contractor.
Schedule issues— ATOP achieved its initial operational capability milestone of June 2004 but a more  aggressive development schedule was agreed to with the ATOP contractor to achieve this milestone by  April 2003 or 14 months earlier.   An ATOP Assessment Team determined that the contractor could not  achieve this earlier date due poor requirements development, unrealistic schedule estimates, and  inadequate evaluation by the contractor of the software complexity.   The development delay has  exacerbated the scheduled transition from the current oceanic system to the ATOP and would cost an  additional $4 million a year to operate and maintain the old system until ATOP is fully operational.    Program officials told us they were not certain when the transition could be achieved because several  operational issues needed to be resolved including ATOP operational trials sector by sector, training  schedule, and filling new controller positions, and budgetary allocations to support these activities.
Performance – ATOP achieved initial operational capability in June 2004 and is limited basis in  one of nine sectors of the Oakland Air Route Traffic Control Center.
En Route Automation Modernization (ERAM)
The En Route Automation Modernization (ERAM) program will enable air traffic controllers to provide  ATC services to users of en route airspace (generally, high-altitude airspace at 10,000 feet or above).  Services provided to users include separation, routing, and advisory services needed to meet FAA’s  mission of providing safe, efficient, and reliable air traffic management.  Specifically, ERAM is to  replace the hardware and software in the current en route Host computer system, the direct-access  radar channel, and associated infrastructure.  This replacement will result in the installation of new  system en route automation architecture at each air route traffic control center (ARTCC).  In concert  with other en route programs, ERAM will modernize the en route infrastructure to provide a  supportable, open-standards-based system that will be the basis for future capabilities and  enhancements.  ERAM is to be deployed at 20 ARTCCs in the continental United States.
FAA awarded a letter contract to Lockheed Martin in December 2002.  To date, ERAM has not  breached any JRC cost or schedule parameters. However, the ERAM program is highly software  intensive, requiring the writing of over 1 million lines of software code. In addition, Lockheed Martin is  behind schedule because of software design and production control issues that Lockheed expects to  resolve.  Lockheed Martin officials stated that it does not expect any downstream impact from the  current negative schedule variance of about $1 million.
Contractor: Lockheed Martin Transportation and Security Solutions.
Projected date for  last deployment  (F&E and O&M)
Software Issues – Software development is one of ERAM's major risk items.  The ERAM program is a  high-risk effort because of its size and the amount of software code – over 1 million lines of software  code expected.  Lockheed Martin is experiencing cost variances because of software engineering  difficulties.  According to its cost performance report, software engineering costs are being hampered  by lower productivity than originally planned and by software code growth across the program.   However, according to FAA officials, these additional software development costs can be easily  absorbed within the contractor’s management reserve that is currently on the contract.
section_title: Key Contributors
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: In addition to the individuals named above, Tamera Dorland, Elizabeth  Eisenstadt, Brandon Haller, Bert Japikse, Carolyn Kirby, Steve Martinez,  Richard Scott, Adam Vodraska, and Dale Yuge made key contributions to  this report.