Air Traffic Control: FAA Reports Progress in System Acquisitions, but Changes in Performance Measurement Could Improve Usefulness of Information
Published: 2007-12-18T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Acquiring new systems on budget and on schedule is critically important in transitioning to the Next Generation Air Transportation System (NextGen). However, air traffic control modernization has been on GAO's high-risk list since 1995, in part due to acquisitions exceeding budget and schedule targets. The Federal Aviation Administration's (FAA) Air Traffic Organization (ATO) has responsibility for managing air traffic control acquisitions. GAO was asked to examine (1) ATO's goals, performance measures, and reporting for systems acquisitions; (2) the validity of ATO's performance measures; and (3) the implications of using ATO's performance measures to assess progress in transitioning to NextGen. To address these issues, GAO compared ATO's measures with attributes of successful performance measures, interviewed agency officials, and sought perspectives of aviation experts.
section_title: What GAO Found
paragraphs: To be consistent with federal guidance and with targets set in the Department of Transportation's strategic plan, ATO established annual acquisition goals and performance measures that call for a high percentage of its major acquisitions to be within 10 percent of budget and on schedule. ATO identifies major acquisitions and reports performance against its goals using its most recently approved budget and schedule estimates. To measure on-budget performance, ATO calculates budget increases over an 8-month period--between January and August of each year. To measure on-schedule performance, ATO selects a minimum of two annual milestones from its major acquisitions and calculates the percentage of milestones that are on schedule. Because ATO's acquisition performance measures lack objectivity, reliability, coverage of core activities, and clarity, and focus only on the preceding year, they may not provide a valid assessment of performance over time. On the positive side, the measures are aligned with FAA's strategic objectives, are measurable, have no overlap, and address governmentwide priorities. However, the performance measures lack objectivity because ATO has no objective criteria for designating which programs are "major" and should be selected for performance reporting. This makes it possible for subjective considerations to dominate the outcome and leaves the performance measures vulnerable to bias in the selection of programs for reporting. The lack of objective criteria for designating major programs also impairs the reliability of the measures (the ability of the measures to produce the same results each time they are applied under similar conditions) and undermines assurance that ATO managers include all core program activities in performance reporting each year. The performance measures also lack clarity in that they do not indicate that ATO measures the performance of many acquisitions against the most recently approved budget and schedule estimates rather than the original estimates. ATO's acquisition performance measurement and reporting could mask budget increases and schedule delays that could have a negative effect on the transition to NextGen. Although ATO reported performance that exceeded its goals for fiscal years 2004 through 2006 and showed nearly steady improvement, when measured against original baselines, acquisition performance improved but was lower than reported. Going forward, the absence of original budget and schedule information on ATO's acquisitions could give the impression that the transition to NextGen is progressing more smoothly than might actually be the case. It will be important for ATO and Congress to recognize budget increases and schedule delays so that the capacity, efficiency, and safety benefits of NextGen can be realized in a cost-efficient and timely fashion.

section_title: Letter
paragraphs: Between fiscal year 2007 and fiscal year 2011, the Federal Aviation  Administration (FAA) plans to acquire more than $14 billion worth of new  systems to continue operating the nation’s current air traffic control (ATC)  system while simultaneously transitioning to the Next Generation Air  Transportation System (NextGen). The transition to NextGen involves  acquiring numerous systems to support precision satellite navigation;  digital, networked communications; integrated weather information;  layered, adaptive security; and more. A cost-effective and timely transition  to NextGen depends in large part on FAA’s ability to keep these  acquisitions within budget and on schedule. Historically, however, FAA  has had chronic difficulties meeting budget, schedule, and performance  targets for acquisitions aimed at modernizing the National Airspace  System (NAS). In our previous reports on the ATC modernization  program, we have pointed out the potentially serious consequences of  missing budget and schedule targets, including significant budget  increases to maintain existing systems and delays in increasing the  efficiency and safety of the NAS.
In 2003, as part of its efforts to implement more businesslike operations,  FAA established annual acquisition performance goals that called for a  high percentage of critical acquisition programs to be within 10 percent of  budget and on schedule. In 2004, as part of its effort to improve its ATC  modernization program, FAA established the Air Traffic Organization  (ATO) as the FAA office responsible for operating the ATC system,  including managing the agency’s ATC acquisitions. For fiscal years 2004  through 2006, ATO reported exceeding its annual goals for having its  major acquisitions on budget and on schedule. For fiscal year 2006, ATO  reported that 100 percent of its major system acquisition investments were  within 10 percent of their annual budgets and over 97 percent were on  schedule.
Because of the criticality of on-budget and on-time acquisitions to the  efficient transition to NextGen, you asked us to determine the status of  ATO’s performance in acquiring ATC systems. To meet this objective, we  addressed the following research questions: (1) How did ATO establish  goals and performance measures for acquiring ATC systems and how are  they reported? (2) How do ATO’s acquisition performance measures  compare with key attributes of successful performance measures? (3)  What are the implications of using ATO’s existing performance measures  to assess progress in the transition to NextGen?
To determine how ATO established goals and performance measures for  acquiring ATC systems, we obtained and analyzed key acquisition  documents and interviewed FAA and ATO officials. To determine how  goals and performance are reported, we reviewed agency performance  and accountability reports and discussed internal tracking and reporting  methods with ATO officials. While ATO’s 2006 portfolio of acquisitions  contained 120 programs, we focused our analysis only on the programs  determined by ATO to be major acquisitions, and thus included in ATO’s  performance reporting. Between 2003 and 2006, the number of major  acquisitions selected for performance reporting varied from 29 to 42. To  evaluate how ATO’s acquisition performance measures compare with key  attributes of successful performance measures, we compared ATO’s  performance measures with eight key attributes of successful performance  measures that we identified in past GAO work. We also obtained the  perspectives of five aviation experts on the reasonableness of ATO’s  acquisition performance measures. We selected these experts from  government and industry to obtain a balanced perspective. To evaluate the  implications of using ATO’s existing performance measures to assess  progress in the transition to NextGen, we obtained and analyzed  acquisition budget and schedule data for fiscal years 2003 through 2006,  and drew upon our past work that determined systemic causes of budget  increases and schedule delays. Through discussions with ATO officials, we  determined that these data were sufficiently reliable for the purposes of  our report. See appendix I for a more detailed explanation of our scope  and methods. We conducted our work from January 2007 through  December 2007 in accordance with generally accepted government  auditing standards.
subsections: section_title: Background
paragraphs: Over the years, FAA has taken a number of steps to better manage its ATC  modernization program. In 1995, based on the premise that FAA would be  better able to manage ATC modernization if it were not constrained by  federal acquisition laws, FAA requested and Congress enacted legislation  that exempted the agency from most federal procurement laws and  regulations and directed FAA to develop and implement a new acquisition  management system that would address the unique needs of the agency.  In 1996, FAA implemented its Acquisition Management System, which  provides high-level acquisition policy and guidance for selecting and  controlling ATC system acquisitions through all phases of the acquisition  life cycle. In February 2004, FAA created the performance-based ATO to  control and improve FAA’s investments and operations. ATO incorporated  FAA’s former Research and Acquisitions and Air Traffic Services  organizations—essentially those that develop and acquire systems and  those that operate them—into a single organization.
ATO catalogues its acquisition programs in its Capital Investment Plan  (CIP). The fiscal year 2006 CIP contained 120 funded acquisition programs  and their anticipated total budgets. The 120 acquisition programs include  37 that have had acquisition program baselines approved by FAA’s Joint  Resources Council (JRC). These baselined programs include  communications, navigation, and surveillance systems that are key to air  traffic control operations. Acquisition program baselines show, among  other things, executive agreement on an acquisition’s estimated budget  and schedule. The JRC also approves rebaselining, through which the  agency documents and approves changes to a program’s budget and  schedule. Of the 37 baselined programs, 29 also have an exhibit 300, a  document prepared for the Office of Management and Budget (OMB) that  provides investment justification and management plans for major ATC  acquisitions. OMB requires agencies to submit exhibit 300s for major  investments, such as those that require special management attention due  to the investment’s importance to the agency, have significant program or  policy implications, or that the agency defines as major. Figure 1 provides  a breakout of the programs in the fiscal year 2006 CIP.
The 83 programs that are not baselined include facilities and infrastructure  programs; a variety of mission support programs such as training,  information technology services, contractual services, and ancillary  systems; and systems that are commercially available and ready for ATO  to use without modification.
FAA’s annual goals and measures for performance reporting, including the  two for performance in managing acquisitions, are described in the  agency’s annual strategic plan, known as the FAA Flight Plan. The Flight  Plan, which FAA began publishing in 2004, sets forth goals, objectives,  strategies, initiatives, and specific performance targets for the agency. The  Government Performance and Results Act of 1993 requires each agency to  prepare and submit to the President and Congress annual reports on  program performance for the previous fiscal year. As an administration  within the Department of Transportation, FAA is not required to prepare a  separate report, but has elected to do so following the statutory  framework and guidance for federal agencies. Each year, FAA reports its  level of success in meeting its two acquisition performance targets, as well  as its other performance targets, in its Performance and Accountability  Report. The full suite of FAA’s performance measures for fiscal year 2006  is listed in appendix III.
In our past work, we identified nine key attributes of successful  performance measures used to evaluate agencies’ performance goals and  measures. We determined that eight of these were applicable to our study  of ATO’s on-budget and on-schedule acquisition goals and performance  measures. See appendix I for more information on our methods. The  eight key attributes to which we compared ATO’s acquisition performance  measures are as follows:  1.  Linkage. Measure is aligned with division- and agencywide goals and  mission and clearly communicated throughout the organization.  2.  Measurable target. Measure has a numerical goal.  3.  Limited overlap. Measure provides new information beyond that  provided by other measures.  4.  Governmentwide priorities. Each measure covers a priority such as  quality, timeliness, and cost of service.  5.  Objectivity. Measure is reasonably free from significant bias or  manipulation.  6.  Reliability. Measure produces the same result under similar  conditions.  7.  Core program activities. Measure covers the activities that an entity is  expected to perform to support the intent of the program.  8.  Clarity. Measure is clearly stated and the name and definition are  consistent with the methodology used to calculate it.
section_title: ATO Established Annual Performance Measures to Be Consistent with Federal Guidance and Selects Major Programs for Measuring and Reporting
paragraphs: ATO developed its performance targets to be consistent with targets set in  the Department of Transportation’s strategic plan, OMB guidance, and  the Federal Acquisition Streamlining Act of 1994, which call for other  federal agencies to establish cost and schedule goals for acquisitions and  to achieve at least 90 percent of those goals. In FAA’s latest Flight Plan,  covering fiscal years 2008 through 2012, the performance targets for  acquisitions are stated as follows:  In fiscal year 2008, 90 percent of major system acquisition investments are  within 10 percent of annual budget and maintain through fiscal year 2012.
In fiscal year 2008, 90 percent of major system acquisition investments are  on schedule and maintain through fiscal year 2012.
FAA began using the basic structure of its current annual acquisition  performance measures in 2003 when FAA sought to achieve 80 percent of  designated milestones and maintain 80 percent of critical program costs  within 10 percent of the total budget as published in the CIP. In 2005, ATO  split the measure into separate targets for budget and schedule. ATO’s  acquisitions targets gradually increased to 90 percent within 10 percent of  budget and 90 percent on schedule by fiscal year 2008.
Although ATO measures performance of all acquisitions, it reports  performance only on major acquisitions in its annual Performance and  Accountability Report. At the beginning of each fiscal year, ATO managers  identify major acquisitions for performance reporting. According to ATO  officials, their selections are based on a number of program  characteristics; key among these is an acquisition’s criticality to the NAS.  ATO officials told us that judging a program as critical could be based on a  number of factors, such as the program having an OMB exhibit 300 or a  baseline document. In addition, officials stated that the selected  acquisitions were meant to represent a cross section of ATC system  acquisitions within ATO. From fiscal years 2003 through 2006, the number  of acquisition programs selected for annual performance reporting varied  between 29 and 42, or roughly a quarter of ATO’s total acquisitions  portfolio each year.
ATO measures budget and schedule performance against selected  acquisitions’ most recently approved estimates. To measure on-budget  performance, ATO compares the total amount budgeted for each selected  acquisition (i.e., the estimated budget to complete an acquisition) reported  in the January CIP with the corresponding amount reported in the August  CIP. ATO considers any program with budget growth of more than 10  percent in this 8-month time frame as not on budget. At the end of the  fiscal year, ATO divides the number of selected acquisitions that are  considered on-budget by the total number of selected acquisitions and  then reports the result as the percentage of major acquisitions within 10  percent of annual budget. To measure schedule performance, at the start  of each fiscal year ATO managers select a minimum of two schedule  milestones from each major acquisition selected for performance  reporting that year. Milestones could be dates to complete activities such  as a final installation or establishing a procurement plan. At the end of the  fiscal year, ATO divides the number of selected milestones that were met  by the total number of selected milestones. ATO reports the result as the  percentage of major programs that are on schedule.
section_title: ATO’s Acquisition Performance Measures’ Lack of Certain Successful Attributes and 1-Year Focus Impairs Validity of Performance Reporting
paragraphs: ATO’s acquisition performance measures meet four of eight key attributes  for successful performance measures that we have identified, but the lack  of objective criteria for selecting programs for performance measurement  reduces objectivity, reliability, and assurance that core programs are  included; the clarity of the performance measures could also be improved.  Taken together, the lack of successful attributes, combined with the 1-year  focus of the performance measures, may not provide a valid measure of  ATO’s acquisition performance.
subsections: section_title: ATO’s Acquisition Performance Measures Meet Four of Eight Key Attributes of Successful Performance Measures
paragraphs: We determined that eight key attributes of successful performance  measures were applicable to our study of ATO’s two acquisition  performance measures. Our analysis determined that ATO met half of  these key attributes, as detailed in table 1.
section_title: ATO’s Performance Measures Lack Objectivity, Reliability, and Assurance That Core Programs Are Included
paragraphs: Although ATO provides some general guidance for selecting acquisitions  for performance reporting, executive judgment was the primary basis for  ATO’s selections. ATO described the scope of its performance measure for  managing acquisitions in its Portfolio of Goals for fiscal year 2006 as  follows: “FAA’s Air Traffic Organization (ATO) Service Units select  specific programs that are determined to provide a capital asset to the  NAS. For FY06, 31 acquisition programs will be tracked and monitored.  Most of the programs selected are considered ‘major’ and must submit an  exhibit 300. Those that do not provide exhibit 300s are included because  they contribute an asset to the NAS with a useful life of more than two  years. The designation of ‘critical acquisition programs’ in the title of this  performance target expresses the critical value of the program to the  NAS.”
As this description illustrates, ATO’s guidance in designating programs as  major allows for a significant amount of professional judgment and does  not clearly define which programs should be included or excluded for  performance reporting. Figure 2 illustrates the types of acquisitions that  ATO selected for performance reporting in fiscal year 2006.
ATO’s guidance lacks objectivity in that it does not indicate specifically  what is to be observed and in which population or conditions. Contrary to  ATO’s statement in the Portfolio of Goals that most of the programs  selected are considered major and must have an OMB exhibit 300  prepared, only 14 of the 29 programs selected that year actually had an  OMB exhibit 300. Additionally, about half of the programs with an OMB  exhibit 300 were not selected for performance reporting. For example, the  System Approach for Safety Oversight program and the Aviation Safety  Knowledge Management Environment program each have an OMB exhibit  300, but, according to ATO officials, these programs are considered “non- NAS” and are therefore not selected for performance reporting. Likewise,  the Facilities Security Risk Management program has an approved  baseline but is not selected. ATO officials told us that facilities and  mission support programs such as this are generally not selected for  acquisition performance reporting; however, ATO has no written guidance  on this policy, and has included mission support programs in its  performance reporting in the past.
Between 2003 and 2006, the number of major acquisitions selected for  performance reporting varied from 29 to 42, and represented about a  quarter of all acquisitions each year. ATO officials told us that the  variation in the number of programs selected from year to year occurred  because of decisions to report performance on specific acquisitions, or  changes in acquisitions’ status from year to year, such as the introduction  of new acquisitions, the completion or cancellation of acquisitions, or  lapses in funding for a fiscal year. Although ATO reports performance on  about 25 percent of the acquisitions portfolio, ATO officials pointed out  that the selected acquisitions have represented between 76 and 84 percent  of the value of that portfolio. Nevertheless, because ATO has no objective  criteria for designating major programs, its performance measure is  vulnerable to bias and the possibility that important or troubled programs  could be excluded from performance reporting.
Objective performance measures should not allow subjective  considerations or judgments to dominate the outcome of the  measurement. Objectivity is important in selecting acquisitions for  performance reporting. OMB guidance allows an agency to define major  investment in its capital planning and investment control process. For  example, the Department of Defense defines a major acquisition program  as one requiring an estimated total expenditure for research, development,  testing, and evaluation of more than $365 million, or an estimated total  procurement expenditure of more than $2.2 billion in fiscal year 2000.  However, the department also allows for professional judgment as the  Secretary of Defense can designate as major any programs below the  dollar thresholds, but determined to be important.
In addition to selecting major programs for performance reporting, ATO  managers select two or more schedule milestones from each selected  program, and use these to measure schedule performance. ATO’s guidance  allows managers wide latitude to select milestones for performance  measurement and provides no guidance regarding the significance of the  milestones that managers should select. Because this provides managers  the opportunity to exclude milestones that they do not expect to meet  during the coming fiscal year, it further weakens the objectivity of the  measure. Four of our five experts commented that ATO’s lack of criteria  for selecting milestones or the ability to pick and choose milestones for  performance measurement was a shortcoming of ATO’s performance  measurement.
The lack of objective criteria for designating major programs also impairs  the key attribute of reliability and the assurance that the measures include  core program activities. Performance measures possess the key attribute  of reliability when they produce the same results each time they are  applied under the same conditions. We have reported that judgmental  evaluations can impair reliability and introduce inconsistencies, which can  affect the outcome of performance measurement. With executive  judgment serving as the primary determinant of which programs and  milestones are selected and measured, different managers could select  different programs each year, resulting in different performance results.  Likewise, the lack of objective criteria does not ensure that ATO managers  include all core program activities in performance measurement each year.  We found that ATO eliminated some acquisitions from performance  reporting for 1 or more years and then resumed reporting these  acquisitions in a subsequent year, although ATO provided reasonable  explanations for these occurrences. Nevertheless, reliability and the  inclusion of core program activities would be better assured if executive  judgment was grounded in written and objective criteria. Four of the five  experts who advised us during our review agreed that ATO needs to  improve its criteria for determining which programs are major and  consequently are included in performance reporting.
section_title: ATO’s Performance Measures Do Not Clearly Indicate the Use of Rebaselined Budget Estimates or Annual Schedule Milestones
paragraphs: ATO’s acquisition performance measures also lack a fourth key attribute  of successful performance measures: clarity. A performance measure that  is not clearly stated (i.e., contains extraneous or omits key data elements)  or that has a name or definition that is inconsistent with the way it is  calculated can confuse users and could cause managers or other  stakeholders to think that performance was better or worse than it  actually was.
subsections: section_title: ATO’s On-Budget Performance Measure Does Not Clearly Indicate the Use of Revised Budget Estimates
paragraphs: ATO’s on-budget acquisition performance measure lacks clarity in  reporting because ATO does not indicate that the acquisition performance  of baselined programs is measured using the most recently approved  budget estimates, as reflected in the January CIP. While ATO does  present Congress with valuable information about a program’s most recent  budget performance by comparing the August budget estimate against the  January budget estimate, this provides only one perspective on  performance because rebaselining resets the measurement of budget or  schedule variances to zero. Of the 31 baselined programs that ATO  selected for acquisition performance reporting in 1 or more years from  fiscal year 2003 through fiscal year 2006, the agency has rebaselined 18,  and has rebaselined some of these more than once. (See app. II.) For  example, the Standard Terminal Automation Replacement System  (STARS) was originally budgeted at $940.2 million, but has been  rebaselined twice and is now budgeted at almost $2.8 billion. However,  because ATO measures budget performance for an 8-month timeframe  against the most recently approved budget estimate, STARS was  considered on budget for fiscal years 2003 through 2006. Other  acquisitions had exceeded original budgets by between $9 million and  $159 million by March 2007, the date of the most recently rebaselined  acquisition. This information is not disclosed in ATO’s performance  reporting. (See app. II.)
ATO officials emphasized that they measure and report on annual  performance and stated they do not measure performance against original  baselines because the agency has already determined that the original  baselines cannot be met. One of the experts who advised us pointed out  that it serves no purpose to continually call a program over budget and  behind schedule if it has been successfully managed for a significant  period of time. We agree that when original baselines cannot be met,  rebaselining can be appropriate. We also agree that measuring annual  progress against the current program baseline has some value, but using  annual measurement as the sole basis for acquisitions that have been  rebaselined does not provide a complete picture of performance over time.  Four of our five experts commented that disclosure of rebaselining was  important in some form, and suggestions ranged from disclosing  rebaselining in a footnote to clearly reporting all rebaselining. The absence  of this information on rebaselining in ATO’s performance reporting could  cause managers and other stakeholders, including Congress, to think that  performance was better than it actually was.
section_title: ATO’s On-Schedule Performance Measure Does Not Clearly Indicate That It Reflects Only the Achievement of Selected Milestones within a Given Fiscal Year
paragraphs: ATO reports meeting its schedule performance goal when at least a  specified percentage “of major system acquisition investments are on  schedule….” This same wording is how the goal is reported in FAA’s 2006  Performance and Accountability Report, where the agency noted on- schedule performance of 97.44 percent. However, ATO is actually basing  its schedule performance measurement on two or more schedule  milestones within a selected program. Thus, the wording of the target and  the performance reporting gives the misleading impression that the entire  acquisition is on schedule when the reported performance is based only on  selected milestones. For example, ATO reported that the $286 million  Integrated Terminal Weather System (ITWS) acquisition was on schedule  in fiscal year 2006 because it hit its selected milestones for that year.  However, the ITWS acquisition (which began in 1997) has encountered  funding reductions, requirements growth and unplanned work, and  greater-than-expected software complexity. ITWS is now scheduled for  completion in October 2009. ATO’s annual reporting based on milestones  simply notes ITWS as on-schedule and does not make clear that the  program was originally scheduled for completion 6 years earlier, in July  2003.
Because ATO’s performance measures lack several attributes of successful  performance measures, and are focused on 1-year snapshots of  performance, they may not provide a valid assessment of acquisition  performance over time. A valid measure provides an accurate  representation of what is being measured. Many of ATO’s acquisitions  span several years and, as the next section shows, measuring performance  against original baselines provides a different perspective on acquisition  performance than that reported by ATO over the past 4 years.
section_title: ATO’s Performance Measurement Could Mask Budget Increases and Delays in the Transition to NextGen
paragraphs: When measured against original baselines, ATO shows improvement in its  managing of acquisitions, but its performance is lower than indicated in  FAA’s annual Performance and Accountability Report. The lack of original  baseline information in ATO’s performance reporting could provide  Congress and the American people with the impression that the transition  to NextGen is progressing more smoothly than might actually be the case.
subsections: section_title: Against Original Baselines, ATO’s Acquisition Performance Has Improved, but Was Lower Than Reported
paragraphs: Comparing the current status of ATO’s major ATC system acquisitions  (i.e., those that ATO selected each year for performance reporting) with  the budgets and schedules in these acquisitions’ original baselines yields  lower performance results than those reported to Congress and the  American people. According to ATO’s performance reports, the  organization showed nearly steady improvement in fiscal years 2003  through 2006 and substantially exceeded its targets for those years, twice  hitting 100 percent. (See table 2.) However, when performance is  measured against original baselines instead of annual budgets or  milestones, acquisition performance was lower than reported, but still  showed a general trend of improvement for fiscal years 2003 through 2006.  In fact, even when measured against original baselines, ATO would have  met its goals for budget in fiscal years 2004 through 2006. However, ATO  did not perform as well in meeting schedules when measured against  original baselines. ATO would have met its schedule goal only in 2005.
ATO officials told us they use a number of methods to measure the  performance of all of the acquisitions contained in the CIP. They have  implemented earned value management on all new major acquisitions as a  way to prevent, detect, report, and correct problems in acquiring major  systems and to ensure that major programs are within budget and  schedule targets. ATO officials also noted that they have monthly  meetings on the status of acquisitions and that management is constantly  apprised of program performance. In addition, officials stated that  significant changes in acquisition status, such as rebaselining, must go  through high-level agency and OMB review, and FAA reports to Congress  any program that exceeds its baseline costs by more than 50 percent.  While ATO has reported acquisitions’ variances against original baselines  to Congress on an ad hoc basis in response to questions for the record, the  only systematic reporting of ATO’s acquisitions performance to the  Congress and the American people is FAA’s annual Performance and  Accountability Report.
ATO’s establishment of annual goals and subsequent annual reporting of  performance are appropriate actions aimed at improving ATO’s  performance in managing ATC system acquisitions. Annual goals are used  throughout government to keep programs on track. We have noted that  such goals illustrate a commitment to achieving immediate, concrete, and  measurable results in the near term. However, it also is important to  provide decision makers and stakeholders with an overall understanding  of program performance in a more holistic sense. We first noted the  shortcomings of annual performance goals for acquisitions in 2005, after  ATO reported that it met its acquisition performance goals for the first  time. We cautioned that, while meeting a 1-year goal was a positive step,  annual performance targets should continue to be viewed in the broader  context of acquisitions’ original and revised baselines. The 18 rebaselined  acquisitions on which ATO reported performance from fiscal year 2003  through 2006 have collectively exceeded their original budget estimates by  approximately $4.4 billion, or 66 percent; however, over 95 percent of this  increase occurred in the Standard Terminal Automation Replacement  System and in WAAS. The 18 rebaselined acquisitions have experienced  schedule slippages between 1 and 10 years.
section_title: ATO’s Performance Measurement Could Overstate the Success of the Transition to NextGen
paragraphs: Because some of FAA’s current acquisitions form the basic building blocks  for NextGen, delays and budget increases in these acquisitions could have  significant implications for the transition to NextGen. Figure 3 illustrates  the planned transition from current systems to future systems and the  anticipated benefits. For example, STARS, discussed previously, is listed  as a current program leading to the transition to NextGen in figure 3. Our  research disclosed that the near tripling of the acquisition’s budget  resulted from insufficient involvement of stakeholders and requirements  growth—two systemic factors that we found led to acquisitions missing  their budget and schedule targets. However, the budget increases that  STARS experienced are not discussed in Performance and Accountability  Reports or in any other regular reporting to Congress.
Another example is the Airport Surveillance Radar - Model 11 (ASR-11),  which is an integrated digital system intended to replace aging analog  radars. NextGen’s plans call for the ASR-11 to provide aircraft and weather  surveillance in terminal areas of small and medium-sized airports, which  also may serve as a part of the back-up surveillance system in case the  primary satellite-based ATC system fails. However, the ASR-11 has  encountered a 59-percent increase in budget per deployed system and its  completion date has been delayed from 2005 to 2009, in part due to  requirements growth. As with STARS, the budget increases and schedule  delays experienced in the ASR-11 acquisition are not discussed in ATO’s  Performance and Accountability Reports or in any routine report to  Congress.
The absence of original budget and schedule estimates in ATO’s  performance reporting could give the impression to Congress and the  American people that ATO’s acquisitions and the transition to NextGen are  progressing more smoothly than is actually the case. Including original  budget and schedule baselines in ATO’s performance reporting could  improve the reports’ usefulness by helping Congress and other  stakeholders identify trends and take corrective action to ensure that the  capacity, efficiency, and safety benefits of NextGen are achieved in a cost- effective and timely manner.
section_title: Conclusions
paragraphs: Although ATO’s acquisition performance measures meet some of the key  attributes of successful performance measures, the attributes that the  measures lack are significant and, considered together, raise serious  questions about the measures’ validity. While a 1-year focus may be  appropriate for some performance measures, it may not provide a valid  assessment of performance over time for major ATC acquisitions that span  a number of years. Moreover, ATO’s use of subjective criteria to pick a  subset of acquisitions and milestones for performance measurement and  its lack of disclosure regarding rebaselining may not provide Congress,  aviation stakeholders, and the public with a complete picture of ATO’s  ability to deliver major ATC acquisitions on budget and on time. Such  reporting could also make budget increases and schedule delays more  difficult to identify. These issues are critical as ATO begins acquiring new  systems with a goal of completing the transition to NextGen by 2025.  Recognizing impending budget increases and schedule delays and taking  corrective action will be necessary to keep the overall NextGen effort on  track. The more quickly ATO can transition to NextGen, the more quickly  the nation will realize the increased efficiencies and safety benefits of new  systems and technologies, and avoid the costs and inefficiencies of  maintaining existing systems. By presenting the most accurate and  complete assessment possible when reporting its performance in acquiring  ATC systems, FAA will better facilitate congressional understanding and  oversight of FAA’s progress in implementing NextGen.
section_title: Recommendations for Executive Action
paragraphs: Because of the importance of ensuring that key administration and  congressional decision makers and stakeholders have complete  information on the budget and schedule performance of FAA’s critical  ATC acquisition programs—both for the most recent fiscal year and since  their inception—we are recommending that the Secretary of  Transportation direct the FAA Administrator to take the following four  actions:  1.  Improve the objectivity, reliability, and inclusion of core programs in  ATO’s acquisition performance measures by establishing written,  objective criteria and guidance for managers to use in determining  which programs are major—and thus selected for performance  reporting—and in selecting schedule milestones.  2.  Improve the clarity of ATO’s annual acquisition performance  measurement process by disclosing in its Performance and  Accountability Reports that the measurement for on-budget  performance covers 8 months and is measured against the most  recently approved budget baselines. Similarly, improve the wording of  the target and reporting for on-schedule acquisitions to disclose that  this measures 1 year of performance against selected program  milestones.  3.  Identify or establish a vehicle for regularly reporting to Congress and  the public on ATO’s overall, long-term performance in acquiring ATC  systems by providing original budget and schedule baselines for each  rebaselined program and the reasons for the rebaselining. If this  information is not added to FAA’s annual Performance and  Accountability Report, then the Performance and Accountability  Report should reference where this information can be found.  4.  Improve the usefulness of ATO’s acquisition performance reporting by  including information (in the Performance and Accountability Report  or elsewhere) on the potential effects that any budget or schedule  slippages could have on the overall transition to NextGen. This also  could include information concerning any mitigation plans ATO has  developed to lessen the effects of program slippages on the  implementation of NextGen systems.
section_title: Agency Comments
paragraphs: We provided a draft of this report to the Department of Transportation for  comment. Senior officials from ATO’s Office of Finance provided oral  comments. ATO officials generally concurred with our recommendations  and noted that they already are considering some changes to their  performance measurement and reporting process for system acquisitions.  In view of the standards discussed in the report, ATO officials agreed to  review current selection criteria of programs included for annual reporting  to address concerns over objectivity. ATO agreed to clarify wording in the  Flight Plan and future Performance and Accountability Reports to ensure  that readers understand that the report reflects agency performance for  the prior fiscal year only. In our draft report, we recommended that ATO  endeavor to report on the overall, long-term status of its acquisitions in its  annual Performance and Accountability Report. ATO officials felt strongly  that the Performance and Accountability Report is meant to reflect  performance for a single fiscal year and would not be the proper vehicle  for reporting on long-term performance. In response, we modified our  recommendation to be less prescriptive about where this information  appears, as long as it is publicly reported. ATO officials said they would  consider other reporting methods to provide Congress with longer-term  status information about the organization’s performance in acquiring ATC  systems. ATO officials also provided technical comments that were  incorporated throughout this report, as appropriate.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Transportation, and the FAA Administrator.  We will make copies available to others upon request. In addition, the  report will be available at no charge on the GAO Web site at  http://www.gao.gov.
If you or your staffs have any questions about this report, please contact  me at (202) 512-2834 or dillinghamg@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on the  last page of this report. GAO staff who made key contributions to this  report are listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: We examined (1) how the Air Traffic Organization (ATO) establishes goals  and performance measures for acquiring air traffic control (ATC) systems  and how they are reported; (2) how ATO’s acquisition performance  measures compare with key attributes of successful performance  measures; and (3) the implications of using ATO’s existing performance  measures to assess progress in the transition to the Next Generation Air  Transportation System (NextGen).
To determine how ATO established acquisition goals and performance  measures for acquiring ATC systems, we reviewed Federal Aviation  Administration’s (FAA) Flight Plans and Acquisition Management System  policy and obtained historical budget and schedule data from ATO’s  finance office on the acquisitions for which ATO reported performance  from fiscal years 2003 through 2006. To ensure that these data were  consistent with documents obtained in previous GAO work, we noted  potential discrepancies and obtained clarifying documents from ATO’s  finance office. We also discussed ATO’s acquisition goals and performance  measurement process with ATO officials. To determine how performance  is reported, we reviewed agency performance and accountability reports  and discussed ATO’s criteria for selecting acquisitions for performance  reporting with ATO officials.
To determine how ATO’s acquisition performance measures compare with  key attributes of successful performance measures, we used eight key  attributes of successful performance measures that were previously  identified by GAO as criteria for comparison against ATO’s acquisition  performance measures. The eight key attributes are:  1.  Linkage. Measure is aligned with division- and agencywide goals and  mission and clearly communicated throughout the organization.  2.  Measurable target. Measure has a numerical goal.  3.  Limited overlap. Measure provides new information beyond that  provided by other measures.  4.  Governmentwide priorities. Each measure covers a priority such as  quality, timeliness, and cost of service.  5.  Objectivity. Measure is reasonably free from significant bias or  manipulation.  6.  Reliability. Measure produces the same result under similar  conditions.  7.  Core program activities. Measure covers the activities that an entity is  expected to perform to support the intent of the program.  8.  Clarity. Measure is clearly stated and the name and definition are  consistent with the methodology used to calculate it.
There was a ninth key attribute identified by GAO that we determined was  not applicable to our study of ATO’s acquisition performance measures.  This ninth attribute is balance, which exists when a suite of measures  ensures that an organization’s various priorities are covered. Although  ATO has other performance measures that it applies to its acquisitions, in  this report we focused only on the two that FAA uses to report its  performance—the percentages of acquisitions on budget and acquisitions  on schedule. Because we did not examine ATO’s full suite of performance  measures for acquisitions, we did not consider the key attribute of  balance.
We compared attributes of ATO’s acquisition performance measures  against each of the eight key attributes to determine whether and how  ATO’s process met each attribute. We reviewed past GAO reports on  FAA’s management of the ATC modernization program, FAA’s  management of major acquisition programs, and the Department of  Defense’s acquisition management and reporting. We identified  acquisition programs whose targets and milestones were revised or  rebaselined to determine validity and consistency in program performance  reporting. We also interviewed ATO officials. Additionally, we obtained  the perspectives of five aviation experts on the reasonableness of ATO’s  acquisition performance measures. To ensure that we collectively received  a balanced and unbiased perspective, we selected experts with varying  government and industry experience. We asked each expert to address the  same set of questions relating to the reasonableness of ATO’s acquisition  performance measurement process.
To determine the implications of using ATO’s existing performance  measures to assess progress in the transition to NextGen, we analyzed the  trends for budget and schedule outcomes between the original baselines  and current budget and schedule baselines for the acquisitions that ATO  selected for performance reporting and monitoring between fiscal years  2003 and 2006. We also drew upon past work in which we undertook  detailed reviews of the status of ATC acquisition programs, and obtained  updated information as necessary from FAA by reviewing documents and  interviewing agency officials. Through discussions with ATO officials, we  determined that these data were sufficiently reliable for the purposes of  our report. We did not conduct an individual or in-depth review of the  effectiveness of the specific programs selected for performance reporting.  We also did not identify a comprehensive list of programs that were  excluded from acquisition performance reporting. This was beyond the  scope and intent of this study.
We conducted our work from January 2007 through December 2007 in  accordance with generally accepted government auditing standards.
section_title: Appendix II: Baseline History for Programs Selected for Acquisition Performance Measurement
paragraphs: Appendix II: Baseline History for Programs  Selected for Acquisition Performance  Measurement $48.1    The full name of the acquisition programs listed above are as follows:  STARS: Standard Terminal Automation Replacement System  NEXCOM: Next Generation Air-to-Ground Communication System  OASIS: Operational and Supportability Implementation System  ITWS: Integrated Terminal Weather System  WAAS: Wide Area Augmentation System  FTI: FAA Telecommunications Infrastructure  ASWON: Aviation Surface Weather Observation Network  NIMS II: National Airspace System Infrastructure Management System-Phase 2  WARP: Weather and Radar Processor  RCE: Radio Control Equipment  ATCBI: Air Traffic Control Beacon Interrogator Replacement  ASR-11: Airport Surveillance Radar - Model 11  LAAS: Local Area Augmentation System  HOCSR: HOST/Oceanic Computer System Replacement  AMASS: Airport Movement Area Safety System  LLWAS: Low Level Wind-shear Alert System  ASDE-X: Airport Surface Detection Equipment – Model X  UHF Replace: Ultra High Frequency Replacement  CPDLC: Controller-Pilot Data Link Communications  BUEC: Back-Up Emergency Communications  ATOP: Advanced Technologies and Oceanic Procedures  PRM: Precision Runway Monitor  ECG: En Route Communication Gateway  URET: User Request Evaluation Tool  TMA: Traffic Management Advisor  ERAM: En Route Automation Modernization  En Route System Mod: En Route Control Center System Modernization  TFM-I: Traffic Flow Management-Infrastructure  VRRP Next Generation: Voice Recorder Replacement Program Next Generation  WSP Tech Refresh: Weather Systems Processor Tech Refresh  VSCS Tech Refresh Phase 2: Voice Switching and Control System Tech Refresh Phase 2  0.9 million for the ASDE-X baseline approved in June 2002, which added ASDE-X  capabilities to seven ASDE- sites. The ASDE-X and ASDE-X acquisitions were combined in the  September 2005 rebaselining.
section_title: Appendix III: Performance Measures from the FAA’s Fiscal Year 2006 Performance and Accountability Report
paragraphs: 1.  Commercial air carrier fatal accident rate  2.  General aviation fatal accidents  3.  General aviation Alaska accidents  4.  Runway incursions (rate)  5.  Commercial space launch accidents  6.  Operational errors (rate)  7.  Safety risk management (number of changes)  8.  Average daily airport capacity (35 Operational Evolution Plan (OEP)  airports)  9.  Average daily airport capacity (eight metropolitan areas)  10. Annual service volume  11. Adjusted operational availability (35 OEP airports)  12. National airspace system on-time arrivals  13. Noise exposure  14. Aviation fuel efficiency  15. Aviation safety leadership  16. Bilateral safety agreements  17. External funding  18. Global positioning system-based technologies  19. Employee attitude survey (cumulative percent increase)  20. Cost control (number of activities per organization)  21. Critical acquisitions on budget  22. Critical acquisitions on schedule  23. Information security  24. Customer satisfaction (American Customer Satisfaction Index)  25. Cost-reimbursable contracts  26. Mission-critical positions  27. Reducing workplace injuries  28. Clean audit with no material weaknesses  29. Grievance processing time  30. Air traffic controller hiring plan (within 5 percent of plan)
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, key contributors to this report  were Faye Morrison (Assistant Director), David Best, Elizabeth Curda,  Elizabeth Eisenstadt, David Hooper, Edmond Menoche, Sara Ann  Moessbauer, Colleen Phillips, and Taylor Reeves.