Air Traffic Control: FAA Uses Earned Value Techniques to Help Manage Information Technology Acquisitions, but Needs to Clarify Policy and Strengthen Oversight
Published: 2008-07-18T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In fiscal year 2008, the Federal Aviation Administration (FAA) plans to spend over $2 billion on information technology (IT) investments--many of which support FAA's air traffic control modernization. To more effectively manage such investments, in 2005 the Office of Management and Budget required agencies to use earned value management (EVM). EVM is a project management approach that, if implemented appropriately, provides objective reports of project status, produces early warning signs of impending schedule delays and cost overruns, and provides unbiased estimates of a program's total costs. Among other objectives, GAO was asked to assess FAA's policies for implementing EVM on its IT investments, evaluate whether the agency is adequately using these techniques to manage key IT acquisitions, and assess the agency's efforts to oversee EVM compliance. To do so, GAO compared agency policies with best practices, performed four case studies, and interviewed key FAA officials.
section_title: What GAO Found
paragraphs: FAA has established a policy requiring the use of EVM on its major IT acquisition programs, but key components of this policy are not fully consistent with best practices of leading organizations. Specifically, FAA fully met four and partially met three components of an effective EVM policy. For example, FAA requires its program managers to obtain EVM training, but it does not enforce completion of this training or require other relevant personnel to obtain this training. Until FAA expands and enforces its policy, it will be difficult for the agency to gain the full benefits of EVM. FAA is using EVM to manage IT acquisition programs, but not all programs are ensuring that their earned value data are reliable. Case studies of four programs demonstrated that all are using or planning to use EVM systems. However, of the three programs currently collecting EVM data, only one program is adequately ensuring that its earned value data are reliable. Another program is limited in its ability to ensure data reliability because it was initiated before earned value was required. The third program did not adequately validate contractor performance data. For example, GAO found anomalies in which the contractor reported spending funds without accomplishing work and others in which the contractor reported accomplishing work while crediting funds to the government. Until programs undertake a rigorous validation of their EVM data, FAA faces an increased risk that managers may not be getting the information they need to effectively manage the programs. FAA has taken important steps to oversee program compliance with EVM policies, but its oversight process lacks sufficient rigor. Through its recurring assessments, FAA has reported that most programs have improved their earned value capabilities over time, and that 74 percent of the programs were fully compliant with national standards. However, FAA's assessments are not thorough enough to identify anomalies in contractor data, and its progress reports do not distinguish between systems that collect comprehensive data and those that do not. As a result, FAA executives do not always receive an accurate view of the quality of a program's EVM data when making investment decisions on that program.

section_title: Letter
paragraphs: In fiscal year 2008, the Federal Aviation Administration (FAA) plans to  spend approximately $2 billion on information technology (IT)  investments, many of which involve systems and technologies to  modernize the air traffic control (ATC) system or to transition to a Next  Generation Air Transportation System (NextGen). Over the past 13 years,  we have identified FAA’s ATC modernization as a high-risk initiative due  to the cost, size, and complexity of this program as well as the cost  overruns, schedule delays, and performance shortfalls that have plagued  the system acquisitions that make up this effort. To more effectively  manage such investments, in 2005 the Office of Management and Budget  (OMB) required agencies to implement earned value management (EVM).  EVM is a project management approach that, if implemented  appropriately, provides objective reports of project status, produces early  warning signs of impending schedule delays and cost overruns, and  provides unbiased estimates of anticipated costs at completion.
This report responds to your request that we review FAA’s use of EVM.  Specifically, our objectives were to (1) assess FAA’s policies for  implementing EVM on its IT investments, (2) evaluate whether the agency  is adequately using these techniques to manage key IT acquisitions,   (3) assess the agency’s efforts to oversee compliance with its EVM  policies, and (4) evaluate whether the agency is using earned value data as  part of its investment management process.
To address our objectives, we reviewed agency documentation, including  FAA-wide policies and plans governing the use of EVM on IT acquisitions,  selected programs’ documented EVM practices and performance reports,  internal EVM assessment criteria and reports, and executive management  briefings. We conducted case studies of four programs that we selected for  their large development and life-cycle costs, representation of FAA’s major  modernization initiatives, and different stages of life-cycle maturity. We  compared the agency’s policies and practices with federal standards and  best practices of leading organizations to determine the effectiveness of  FAA’s use of earned value data in managing its IT investments. We also  interviewed relevant agency officials, including key personnel on  programs selected for case study and the official responsible for  implementing EVM, and we observed working group meetings on EVM.  This report builds on a body of work we have performed on FAA’s ATC  modernization efforts.
We conducted this performance audit from November 2007 to July 2008 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives. Further details on our objectives, scope,  and methodology are provided in appendix I.
subsections: section_title: Background
paragraphs: The mission of FAA, an agency within the Department of Transportation,  is to promote the safe, orderly, and expeditious flow of air traffic in the  U.S. airspace system, commonly referred to as the National Airspace  System. To maintain its ability to effectively carry out this mission,  address an aging infrastructure, and meet an increasing demand for air  transportation, in 1981, FAA embarked on a multibillion-dollar effort to  modernize its aging ATC system. Under this modernization program, FAA  has acquired and deployed new technologies and systems—and continues  to do so today. Looking to the future, FAA is now beginning to fund  components of NextGen, a transformation to a new system that is  expected to use satellite-based technologies and state-of-the-art  procedures to handle increasing air traffic volume through 2025, while  further improving safety and security.
subsections: section_title: FAA Relies on IT to Carry out Its Mission
paragraphs: FAA relies extensively on IT to carry out its mission—both in terms of its  operational air traffic responsibilities and its administrative activities. The  agency depends on the adequacy and reliability of the nation’s ATC  system, which includes a vast network of radars, navigation and  communications equipment, and information processing systems located  at air traffic facilities across the country. Through its ATC system, FAA  provides services such as controlling takeoffs and landings, and managing  the flow of traffic between airports. For example, the Integrated Terminal  Weather System integrates local weather data to allow the maximum use  of airport runways. The Wide Area Augmentation System is used to  provide vertically guided system approaches via Global Positioning  System satellites and its own satellites to aircraft at thousands of airports  and airstrips where there is currently no vertically guided landing  capability, thereby improving safety and reducing pilot workload. FAA  also relies on IT to carry out its mission-support and administrative  operations. For example, FAA uses IT to support accident and incident  investigations, security inspections, and personnel and payroll functions.
With an IT budget of $2.1 billion for fiscal year 2008, FAA accounts for  about 83 percent of the Department of Transportation’s IT budget. For  fiscal years 2007 through 2011, FAA plans to acquire more than $14 billion  in new systems to continue operating the nation’s current ATC system,  while simultaneously transitioning to NextGen. This transition involves  acquiring numerous systems to support precision satellite navigation;  digital, networked communications; integrated weather information;  layered, adaptive security; and more. A cost-effective and timely transition  to NextGen depends in large part on FAA’s ability to keep these  acquisitions within budget and on schedule. Historically, however, FAA  has had chronic difficulties in meeting budget, schedule, and performance  targets for acquisitions aimed at modernizing the National Airspace  System. For example, in June 2005, we reported that 13 of 16 selected  major ATC system acquisitions experienced cost, schedule, or  performance shortfalls when assessed against their original milestones.  These 13 system acquisitions experienced cost increases ranging from   $1.1 million to about $1.5 billion; schedule extensions ranging from 1 to 13  years; and performance shortfalls, including safety problems.
section_title: GAO Designated FAA’s ATC Modernization as High Risk; FAA Has Taken Steps to Address Weaknesses
paragraphs: In 1995, we designated FAA’s modernization of its ATC system as a high- risk initiative because of the size, cost, and complexity of the program as  well as difficulties in meeting cost, schedule, and performance goals on  the individual projects that make up the modernization. Since then, in our  High-Risk Series updates, we have reported on FAA’s efforts to address  the underlying weaknesses that put it on the high-risk list. These include  FAA’s efforts to  institutionalize key processes for acquiring and developing software  systems,   develop and enforce its enterprise architecture,  improve its cost accounting and estimating practices, improve its ability to effectively manage IT investments, and   develop an organizational culture that supports sound acquisitions.
To the agency’s credit, FAA has taken a number of steps over the years to  better manage its ATC modernization program. Because of FAA’s  contention that its modernization efforts were hindered by federal  acquisition regulations, in November 1995 Congress enacted legislation  that exempted the agency from most federal acquisition laws and  regulations. The legislation directed FAA to develop and implement a  new acquisition management system that would address the unique needs  of the agency. In April 1996, FAA implemented an acquisition management  system that provided acquisition policy and guidance for selecting and  controlling FAA’s investments through all phases of the acquisition life  cycle. This guidance was intended to reduce the time and cost needed for  fielding new products and services by introducing (1) a new investment  management system that spans the entire life cycle of an acquisition, (2) a  new procurement system that provides flexibility in selecting and  managing contractors, and (3) organizational and human capital reforms  that support the new investment and procurement systems.
More recently, in February 2004, FAA created the performance-based Air  Traffic Organization to control and improve FAA’s investments and  operations and to better provide safe, secure, and cost-effective air traffic  services now and into the future. This change combined the groups  responsible for developing and acquiring systems with those that operate  them into a single organization. The Air Traffic Organization is led by  FAA’s Chief Operating Officer.
section_title: EVM Provides Insight on Program Cost and Schedule
paragraphs: Pulling together essential cost, schedule, and technical information in a  meaningful, coherent fashion is a challenge for most programs. Without  meaningful and coherent cost and schedule information, program  managers can have a distorted view of a program’s status and risks. To  address this issue, in the 1960s, the Department of Defense developed the  EVM technique, which goes beyond simply comparing budgeted costs with  actual costs. This technique measures the value of work accomplished in a  given period and compares it with the planned value of work scheduled  for that period and with the actual cost of work accomplished.
Differences in these values are measured in both cost and schedule  variances. Cost variances compare the earned value of the completed  work with the actual cost of the work performed. For example, if a  contractor completed $5 million worth of work and the work actually cost  $6.7 million, there would be a -$1.7 million cost variance. Schedule  variances are also measured in dollars, but they compare the earned value  of the work completed with the value of work that was expected to be  completed. For example, if a contractor completed $5 million worth of  work at the end of the month but was budgeted to complete $10 million  worth of work, there would be a -$5 million schedule variance. Positive  variances indicate that activities are costing less or are completed ahead  of schedule. Negative variances indicate activities are costing more or are  falling behind schedule. These cost and schedule variances can then be  used in estimating the cost and time needed to complete the program.
Without knowing the planned cost of completed work and work in  progress (i.e., the earned value), it is difficult to determine a program’s  true status. Earned value provides information that is necessary for  understanding the health of a program; it provides an objective view of  program status. As a result, EVM can alert program managers to potential  problems sooner than expenditures alone can, thereby reducing the  chance and magnitude of cost overruns and schedule delays. Moreover,  EVM directly supports the institutionalization of key processes for  acquiring and developing systems and the ability to effectively manage  investments—areas that are often found to be inadequate on the basis of  our assessments of major IT investments.
Because of the importance of ensuring quality earned value data, in May  1998, the American National Standards Institute (ANSI) and the Electronic  Industries Alliance (EIA) jointly established a national standard for EVM  systems. This standard, commonly called the ANSI standard, consists of  32 guidelines to instruct programs on how to establish a sound EVM  system, ensure that the data coming from the system are reliable, and use  the earned value data to manage the program. See appendix II for an  overview of this standard.
section_title: Federal Guidance Calls for Using EVM to Improve IT Management
paragraphs: In August 2005, OMB issued guidance outlining steps that agencies must  take for all major and high-risk development projects to better ensure  improved execution and performance and to promote more effective  oversight through the implementation of EVM. Specifically, this guidance  directs agencies to (1) develop comprehensive policies to ensure that  agencies are using EVM to plan and manage development activities for  major IT investments; (2) include a provision and clause in major  acquisition contracts or agency in-house project charters directing the use  of an EVM system compliant with the ANSI standard; (3) provide  documentation demonstrating that the contractor’s or agency’s in-house  EVM system complies with the national standard; (4) conduct periodic  surveillance reviews; and (5) conduct integrated baseline reviews on  individual programs to finalize the cost, schedule, and performance goals.
Building on OMB’s requirements, in July 2007, we issued a draft guide on  best practices for estimating and managing program costs. This guide  highlights the policies and practices adopted by leading organizations to  implement an effective EVM program. Specifically, in the guide, we  identify the need for organizational policies that establish clear criteria for  which programs are required to use EVM, compliance with the ANSI  standard, a standard product-oriented structure for defining work  products, integrated baseline reviews, specialized training, criteria and  conditions for rebaselining programs, and an ongoing surveillance  function. In addition, we identify key practices that individual programs  can use to ensure that they establish a sound EVM system, that the earned  value data are reliable, and that the data are used to support decision  making. OMB refers to this guide as a key reference manual for agencies in  its 2006 Capital Programming Guide.
section_title: Two FAA Executives Are Responsible for EVM Implementation
paragraphs: Two FAA executives—the Acquisition Executive and the Chief  Information Officer—are jointly responsible for implementing EVM and  ensuring its consistent application across the agency’s IT acquisitions. The  Acquisition Executive’s responsibilities include developing EVM policy  and guidance, certifying contractors’ conformance with the ANSI  standard, advising and assisting programs with integrated baseline  reviews, approving programs’ plans for continued surveillance of  contractors’ EVM systems, and managing the EVM training program and  curriculum. The Acquisition Executive established the position of EVM  Focal Point to lead these efforts.
The Chief Information Officer’s responsibilities include assisting in the  development of EVM policy and guidance, certifying programwide  conformance with the ANSI standard, performing ongoing programwide  EVM system surveillance, and managing the preparation of information  reported on programs’ annual business cases—which includes verifying  the accuracy of the program baseline, schedule and cost performance, and  corrective action plans. The Chief Information Officer established a Value  Management Office to perform these functions.
section_title: FAA Has Established an EVM Policy for Major IT Investments, but Key Components Are Not Fully Consistent with Best Practices
paragraphs: In 2005, FAA established a policy requiring the use of EVM on its major IT  investments; however, key components of this policy are not fully  consistent with best practices. We recently reported that leading  organizations establish EVM policies that    establish clear criteria for which programs are to use EVM;  require programs to comply with the ANSI standard; require programs to use a product-oriented structure for defining work  products; require programs to conduct detailed reviews of expected costs,  schedules, and deliverables (called an integrated baseline review); require and enforce EVM training;   define when programs may revise cost and schedule baselines (called  require system surveillance—routine validation checks to ensure that  major acquisitions are continuing to comply with agency policies and  standards.
Table 1 describes the key components of an effective EVM policy.
FAA began developing EVM-related policies for its IT acquisition programs  in 2005. The agency currently has a policy in place that fully addresses  four of the seven areas and partially addresses the remaining three areas  (see table 2).
Specifically, FAA has policies and guidance in its Acquisition Management  System that fully address EVM implementation on all major IT  investments, compliance with the ANSI standard, integrated baseline  reviews, and system surveillance. These policies are discussed below.
Criteria for implementing EVM on all IT major investments: FAA  requires all of its major development, modernization, and enhancement  programs to use EVM. Specifically, these are all programs with a  requirement to provide a business case to OMB. In addition, FAA requires  that all contracts and subcontracts that are expected to exceed a cost of  $10 million for development, modernization, and enhancement work must  be managed using an EVM system. Projects lasting less than 1 year are not  required to use EVM.
Compliance with the ANSI standard: FAA requires that all work activities  performed on major programs by government personnel, major  contractors, and support contractors be managed using an EVM system  that complies with industry standards. FAA’s EVM Focal Point is  responsible for certifying that contractors with contracts over $10 million  conform with the standard. FAA’s Value Management Office is responsible  for certifying that each program conforms with the standard.
Integrated baseline reviews: FAA requires each program manager to  conduct a comprehensive review of a program baseline for major  programs and contracts within 90 to 180 days of contract award or  program baseline establishment. Furthermore, an updated integrated  baseline review must be performed after a program exercises significant  contract options or executes modifications. The agency’s guidance calls  for the involvement of program management teams, prime contractor  management, and independent subject matter experts who validate the  program baselines and performance measurement processes.
System surveillance: FAA requires ongoing surveillance of all programs  and contracts that are required to use EVM systems to ensure their  continued compliance with industry standards. The Value Management  Office is responsible for providing surveillance at the program level  through annual assessments of each major program. Individual program  managers and contracting officers are responsible for conducting  surveillance on their contractors’ EVM in accordance with a surveillance  plan approved by the EVM Focal Point.
However, FAA’s policy and guidance are not consistent with best practices  in three areas: defining a product-oriented structure for defining work  products, requiring EVM training, and establishing rebaselining criteria.  These areas are discussed below.
Standard structure for defining work products: FAA requires its  programs to establish a standard work breakdown structure. However,  FAA calls for a function-oriented structure, rather than a product-oriented  one. This means that work is delineated based on functional activities,  such as design engineering, requirements analysis, and quality control. In  contrast, a product-oriented work breakdown structure reflects cost,  schedule, and technical performance on specific deliverables. Without the  level of detail provided by a product-oriented approach, program  managers may not have the information they need to make decisions on  specific program components. For example, cost overruns associated with  a specific radar component could be quickly identified and addressed  using a product-oriented structure. If a function-oriented structure were  used, these costs could be spread out over design, engineering, and quality  control.
FAA program managers can choose to use a product-oriented work  breakdown structure to manage their programs and contracts, but then  they need to transfer their data to FAA’s required function-oriented work  breakdown structure when reporting to management. EVM experts agree  that such mapping efforts are time-consuming and subject to error.  Furthermore, programs do not always map items in the same way, and, as  a result, costs may not be captured consistently across programs.
FAA officials stated that they use the functional format because it is  aligned with the agency’s cost accounting system. While this presents a  challenge, it is not insurmountable. For example, in the near-term, the  agency could develop a standard mapping function to translate product- oriented program data into the function-oriented cost accounting system.  While this approach would not resolve the time-consuming nature of  mapping (since programs would still be expected to complete this  activity), it does at least allow costs to be captured consistently across  programs. As a longer-term solution, we have repeatedly urged  government agencies to adopt cost accounting systems that provide  meaningful links among budget, accounting, and performance. Such  systems are consistent with product-oriented work breakdown  structures.
Until FAA establishes a standard product-oriented work breakdown  structure, program officials who use the function-oriented approach to  manage their contracts may not be obtaining the information they need.  Furthermore, program officials who choose to manage using a product- oriented structure will continue to spend valuable time and effort mapping  their product-oriented structures to the FAA standard, and the agency will  continue to risk that data are captured inaccurately or inconsistently  during this mapping exercise.
EVM training requirements: FAA has developed EVM training and  requires program managers to complete a minimum of 24 hours of EVM  and cost estimating training. However, the agency does not specify EVM  training requirements for program team members or senior executives  with program oversight responsibilities. In addition, the agency does not  enforce EVM training to ensure that all relevant staff have completed the  required training. Instead, individual program offices are responsible for  ensuring that their teams obtain sufficient EVM training. Some programs  ensure that all key program staff have completed the appropriate level of  training they need to understand their roles and responsibilities, while  other programs do not. Until FAA establishes EVM training requirements  for all relevant personnel (including executives with oversight  responsibilities and program staff responsible for contract management)  and verifies the completion of this training, it cannot effectively ensure  that its program staff have the appropriate skills to validate and interpret  EVM data, and that its executives fully understand the data they are given  in order to ask the right questions and make informed decisions.
Rebaselining criteria: FAA requires that programs seeking a new cost and  schedule baseline gain approval from a board of executives, called the  Joint Resources Council, which is responsible for investment decisions.  However, the agency does not define acceptable reasons for rebaselining  or require programs to identify and address the reasons for the need to  rebaseline. Until FAA addresses these elements, it will face an increased  risk that its executive managers will make decisions about programs with  incomplete information, and that these programs will continue to overrun  costs and schedules because their underlying problems have not been  identified or addressed.
section_title: Key FAA Systems Are Using EVM, but Are Not Consistently Implementing Key Practices
paragraphs: FAA is using EVM to manage system acquisition programs, but the extent  of implementation varies among programs. Case studies of four programs  demonstrated that all are using or planning to use EVM. However, the four  programs are not consistently performing EVM on the full scope of the  program (as opposed to the scope of the contract) and ensuring that the  earned value data are reliable. Until these areas are fully addressed, FAA  faces an increased risk that program managers are not adequately using  earned value to manage their programs.
Our work on best practices in EVM identified 11 key practices that are  implemented on acquisition programs of leading organizations. These  practices can be organized into three management areas: establishing a  sound EVM system, ensuring reliable data, and using earned value data to  manage. Table 3 lists these 11 key practices.
We performed case studies of four FAA system acquisitions: the Airport  Surveillance Radar (ASR-11), En Route Automation Modernization  (ERAM), Surveillance and Broadcast Services (SBS), and System Wide  Information Management (SWIM). All of the four key FAA system  programs demonstrated at least a partial level of EVM implementation.  Figure 1 summarizes our results on these selected programs. Following  the figure, we provide a summary of each key area of program  management responsibility in EVM. In addition, more details on the four  case studies are provided in appendix III.
subsections: section_title: Programs Did Not Consistently Establish Comprehensive EVM Systems, but Had Justification for These Shortfalls
paragraphs: The four programs did not consistently establish comprehensive EVM  systems, but were able to justify these shortfalls. Of the four programs,  only SBS demonstrated that it had fully implemented the six practices in  this area. For example, the program established an integrated performance  baseline that captures the full scope of work on the program and links  directly to the integrated master schedule.
Two programs—ASR-11 and ERAM—demonstrated that they partially  implemented each of the six key practices in this area. Both had a  reasonable justification for their partial EVM implementation: the systems  were initiated before FAA required projects to obtain EVM data and have  implemented work-arounds to allow them to meet FAA’s current earned  value reporting requirements. Specifically, the ASR-11 team does not  receive any EVM data, so the team established a performance  measurement baseline to estimate the work remaining on both the  contractor and government portions of the program. Alternatively, ERAM  has implemented EVM to govern the contract deliverables, but not the  government’s portion of the program. Instead, the program estimates  government costs.
The fourth program, SWIM, has initiated EVM practices, but these efforts  are still under way because the system is in an early stage in its acquisition  life cycle. At the time of our review, SWIM had fully met two of the six key  practices. For example, SWIM has a work breakdown structure and has  identified who will perform the work. In addition, the program is currently  developing its integrated master schedule and plans to complete all key  EVM process steps prior to beginning development work (which is  expected to begin in fiscal year 2009). SWIM is not currently collecting  EVM data.
section_title: Programs Did Not Consistently Ensure That EVM Data Were Reliable
paragraphs: The three programs that currently collect or estimate monthly EVM data  (ASR-11, ERAM, and SBS) did not consistently ensure that their EVM data  were reliable. Of the three programs, one fully implemented the practices  for ensuring the reliability of the prime contractor and government  performance data, one partially implemented the practices but had  justification for its shortfalls, and one partially implemented the practices.
SBS demonstrated that it fully implemented the three practices. The  program requires its technical managers to validate the earned value data  they are responsible for collecting on a monthly basis. It also established  mechanisms to alert the team if the contractor’s deliverables may not meet  system requirements. In addition, program EVM analysts are expected to  analyze and report cost and schedule performance trends and cost  estimates to complete the remaining work to the program manager and an  internal management review board.
ASR-11 partially implemented each of the three practices for ensuring that  earned value data are reliable, but had a justification for this shortfall. As  we have previously noted, ASR-11 measures government and contractor  effort; however, it is constrained in its oversight capabilities since the  prime contractor is not required to report earned value information or cost  data to FAA. As a result, the program is unable to collect or validate actual  costs expended on the contractor’s scope of work. Instead, ASR-11 relies  on schedule status to determine when planned work on a contract  deliverable has been authorized to begin—such as work to dismantle a  legacy facility site—and completed. The program depends on the receipt  of Air Force invoices to determine the actual costs for that planned effort,  and relies on its FAA teams that are on-site to get qualitative assessments  of the cost and schedule drivers impacting performance. Despite the  external constraints, ASR-11 has a skilled team in place to assess the EVM  data, perform the appropriate analyses of performance trends, and make  projections of estimated costs at program completion.
ERAM also partially implemented each of the three practices for ensuring  that earned value data are reliable. The ERAM program team analyzes the  prime contractor’s monthly EVM data and variance reports and then uses  that information to make projections of estimated costs at program  completion. However, we identified several anomalies in the contractor’s  reports over an 11-month period that suggest the contractor may not be  reliably reporting its work activities. For example:    There were multiple cases in which the contractor reported that no work  was planned or accomplished, yet funds were spent; in other cases, the  contractor reported that work was planned and accomplished, but funds  were credited to the government. There were also cases in which the  contractor reported that work was planned and dollars spent, but a  negative amount of work was performed (i.e., work that was previously  reported as completed was now reported as not completed). The  contractor did not provide an explanation for these issues in its reports to  the ERAM program office.
In September 2007, the contractor planned to complete $102 million worth  of work—a significant spike in planned work, given that the average  amount of work planned and accomplished in a single month is about   $25 million. Furthermore, the contractor reported that it accomplished  $100 million worth of that work and spent only $31 million to complete it.  The contractor did not provide a justification for this steep spike in work  planned and accomplished, or for the sizable gap between the work  accomplished and the cost of this work. The ERAM program office was  also unable to explain why this occurred.
These reporting anomalies raise questions about the reliability of the  contractor data and the quality of the program’s efforts to verify and  validate these data. Program officials were unable to explain these  anomalies. Until ERAM improves its ability to assess contract data and  resolve anomalies, it risks using inaccurate data to manage the contractor,  potentially resulting in cost overruns, schedule delays, and performance  shortfalls.
section_title: Program Management Teams Consistently Used Earned Value Data to Make Decisions
paragraphs: All three programs that currently collect monthly EVM data were able to  demonstrate that they use these data to manage their programs. The SBS  program manager conducts rigorous reviews with its internal performance  management review board to discuss the program’s earned value  performance against planned cost and schedule targets and take  appropriate actions to reverse negative trends. The ASR-11 program  manager is using the current cost and schedule variances being accrued on  site construction work to make projections on the overall cost to complete  this work and to create risk mitigation plans to address the cost and  schedule drivers. The ERAM program manager uses the earned value data  to identify areas of concern and make recommendations to the contractor  on items that should be watched, mitigated, and tracked to closure.  Currently, the program manager is monitoring the contractor’s use of  management reserve as well as fluctuating cost variances associated with  the design and engineering supporting ERAM’s initial capability.
section_title: FAA Has Taken Steps to Oversee EVM Compliance, but Its Oversight Process Lacks Sufficient Rigor
paragraphs: FAA has taken important steps to oversee compliance with EVM policies  by establishing an oversight office, assessing major systems using defined  evaluation criteria, and demonstrating improved capabilities on most  programs. However, the oversight office’s assessments are not thorough  enough to identify anomalies in contractor data, and its agencywide  progress reports can be misleading, in that the agency’s evaluation process  does not distinguish between systems that collect comprehensive data and  those that do not. As a result, FAA executives do not always receive an  accurate view of the quality of a program’s EVM data when making  investment decisions on that program.
subsections: section_title: FAA Established an Oversight Program to Ensure Compliance with EVM Requirements
paragraphs: According to best practices in program oversight, an organization should  assign responsibility for providing oversight, establish and implement a  plan for conducting oversight that is sufficiently detailed to identify  problems, and report on its progress over time. FAA established an  oversight program to ensure EVM compliance assessments on its major  programs.
In August 2005, FAA established the Value Management Office, an  organization responsible for assessing the EVM compliance of all major IT  acquisition programs. This office developed an EVM system assessment  plan to evaluate each major system program. This plan defines the  evidence needed to obtain a weak, moderate, or strong score for each of  the 32 guidelines in the ANSI standard. The group assesses each major  program’s earned value capabilities on an annual basis. In addition, this  office provides its senior executives and OMB with a summary of the EVM  compliance status of all major programs. FAA reports that its IT systems  have made major improvements in their earned value capabilities over the  last few years. For example, in August 2005, FAA reported that 6 of its 19  major IT acquisition programs (or 32 percent) had fully complied with the  standard. As of February 2008, FAA reported that 17 of its 23 major IT  programs (or 74 percent) had achieved full compliance with the ANSI  standard.
section_title: FAA’s Oversight Process Lacks Sufficient Rigor
paragraphs: While FAA’s oversight has accomplished much since it was established,  the process used to assess and report on programs lacks the rigor needed  to be a reliable gauge of agency progress. Best practices call for program  EVM oversight to include an assessment of both government and  contractor performance data to identify issues that may undermine the  validity of these data. In addition, to be transparent and reliable, reports  on the status of programs’ EVM implementation should clearly identify  situations in which programs are unable to fully comply with FAA policies.
In assessing programs’ EVM compliance, FAA’s oversight office obtains  and reviews earned value data for the program as a whole. It does not  analyze the contractor’s performance data. For example, FAA’s oversight  office did not review ERAM’s contractor data and, therefore, did not  identify anomalies in which funds were spent on no work and other work  was performed for no funds. As a result, it rated the program highly on  factors associated with data reliability.
In addition, in reporting agencywide progress in implementing EVM, the  agency’s oversight process does not distinguish between programs that  collect earned value data only on the contract level, and those that collect  integrated data on the program as a whole. For example, both ERAM and  ASR-11 use approximations to reflect their earned value data. As we have  previously noted, ASR-11 uses approximations for the entire program  because another agency administers the contract. ERAM uses  approximations only for the government portions of the program.  Nonetheless, FAA gave both of these programs their highest ratings. This  is misleading in that it portrays the performance data on these programs as  having the same level of precision as programs that have an integrated  approach to EVM. Since these programs were initiated before the EVM  requirement, it is likely that other older acquisition programs have also  implemented work-arounds. Of the 23 major programs assessed by FAA,  16 were initiated before the EVM policy was established. Until these issues  are resolved, FAA will be unable to effectively ensure that EVM  implementation is consistent across the agency, and that FAA executives  obtain an inaccurate view of the quality of an individual program’s EVM  data when making investment decisions.
section_title: FAA Has Incorporated Important EVM Performance Data into Its IT Investment Management Process
paragraphs: To obtain better insight into the progress made on its system acquisition  programs, FAA incorporated EVM performance data into its process for  reviewing IT investments. Our work in IT investment management  highlights the importance of executive decision makers having sufficient  insight into program status so that they can identify and mitigate risks, and  ensure that programs are on track against established cost and schedule  expectations. The performance data from program EVM systems are  critical for helping managers achieve sufficient insight on program status.
FAA executives are reviewing EVM data as part of their investment review  process. The level of detail in EVM data reporting is dependent on the  level of executive review. For example, executives responsible for a  portfolio of projects conduct project reviews on a quarterly basis. They  obtain project data that include cumulative cost and schedule variance  reporting over an extended period. For example, ASR-11 has reported  cumulative trends over an 11-month period. Other key reported  performance metrics include estimated costs at program completion, cost  and schedule efficiency indexes (which describe the dollar value of work  being accomplished for every dollar spent), and management reserve. At a  more senior level, FAA’s Joint Resource Council receives project data on a  monthly basis, is briefed on projects that are breaching cost and schedule  variances by more than 10 percent on a quarterly basis, and obtains  detailed briefings on projects twice a year. At this time, these briefings  contain a program dashboard matrix, which shows the earned value cost  and schedule efficiency indexes taken over a 6-month period. FAA’s Value  Management Office also has a joint initiative under way with the Joint  Resource Council to refine the dashboard matrix in order to determine the  most appropriate data, as well as level of detail, that will enable decision  makers to prevent, detect, and respond to issues in a timely manner.
section_title: Conclusions
paragraphs: FAA has taken a number of important steps to improve the management of  its IT investments through the implementation of EVM. The agency has  established policies that require the use of EVM; system acquisition  programs are using earned value data to manage their programs; an  oversight office monitors system acquisition programs’ compliance with  policy and standards; and earned value performance data are being used  by multiple levels of management as they review and manage IT  investment.
However, the agency does not fully ensure the accuracy and usefulness of  earned value data as a management tool. Specifically, FAA policies lack  sufficient guidance on the type of work structure needed to most  effectively use EVM data, training requirements do not extend to all  relevant personnel and call for this training to be monitored and enforced,  and programs are not required to identify or mitigate the root cause of any  cost and schedule overruns when they request a revised cost and schedule  baseline. In addition, FAA programs are not consistently ensuring that the  data coming from contractors are reliable. Of the three programs we  reviewed that currently collect earned value data, one program, ERAM,  had no explanation for anomalies in its contractor data wherein funds  were spent but no work was done; in another situation, work was  accomplished but funds were credited to the government. This is of  concern because both program managers and agency executives could be  making programmatic and investment decisions on the basis of inaccurate  and misleading data. Furthermore, FAA’s Value Management Office—an  internal EVM oversight group—does not evaluate the validity of contractor  data or distinguish between programs that have comprehensive earned  value systems and ones that have implemented work-arounds. As a result,  FAA executives are, in selected cases, receiving an inaccurate view of the  quality of a program’s EVM data, which could impede sound investment  decisions. Until these issues are resolved, it will be difficult for FAA to  effectively implement EVM or optimize its investment in this critical  management tool.
section_title: Recommendations for Executive Action
paragraphs: To improve FAA’s ability to effectively implement EVM on its IT  acquisition programs, we recommend that the Secretary of Transportation  direct the Acting FAA Administrator to take the following seven actions:  Modify acquisition policies governing EVM to  require the use of a product-oriented standard work breakdown structure,   enforce existing EVM training requirements and expand these  requirements to include senior executives responsible for investment  oversight and program staff responsible for program oversight, and   define acceptable reasons for rebaselining and require programs seeking  to rebaseline to (1) perform a root cause analysis to determine why  significant cost and schedule variances occurred and (2) establish  mitigation plans to address the root cause.
Direct the ERAM program office to work with FAA’s Value Management  Office to    determine the root causes for the anomalies found in the contractor’s EVM    develop a corrective action plan to resolve these problems.
Direct the Value Management Office to improve its oversight processes by  including an evaluation of contractors’ performance data as part of its  program assessment criteria, when FAA has the authority to do so, and    distinguishing between programs that collect earned value data on fully  integrated programs and those that do not in its agencywide progress  reports to provide transparency to decision makers.
section_title: Agency Comments
paragraphs: The Department of Transportation’s Director of Audit Relations provided  comments on a draft of this report via e-mail. In those comments, he said  that the department generally agreed with the findings and  recommendations contained in the draft. The department also provided  technical comments, which we have incorporated in this report as  appropriate.
We will be sending copies of this report to interested congressional  committees, the Secretary of Transportation, the Acting FAA  Administrator, and other interested parties. We will also make copies  available to others upon request. In addition, the report will be available at  no charge on our Web site at http://www.gao.gov.
If you or your staffs have any questions on the matters discussed in this  report, please contact me at (202) 512-9286 or by e-mail at  pownerd@gao.gov. Contact points for our Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.  GAO staff who made major contributions to this report are listed in  appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to (1) assess the Federal Aviation Administration’s  (FAA) policies for implementing earned value management (EVM) on its  information technology (IT) investments, (2) evaluate whether the agency  is adequately using EVM techniques to manage key system acquisitions,  (3) assess the agency’s efforts to oversee compliance with its EVM  policies, and (4) evaluate whether the agency is using EVM data as part of  its IT investment management.
To assess whether FAA has policies in place to effectively implement  EVM, we analyzed FAA’s policies and guidance that support EVM  implementation agencywide as well as on system acquisition programs.  Specifically, we compared these policies and guidance documents with  both the Office of Management and Budget’s (OMB) requirements and key  best practices recognized within the federal government and industry for  the implementation of EVM. These best practices are contained in an  exposure draft version of our cost guide. We also interviewed key agency  officials and observed FAA EVM working group meetings to obtain  information on the agency’s ongoing and future EVM plans.
To determine whether key FAA system programs are adequately using  EVM techniques, we performed case studies on 4 of FAA’s 23 system  acquisition programs currently required to use EVM: the Airport  Surveillance Radar (ASR-11), En Route Automation Modernization  (ERAM), Surveillance and Broadcast Services (SBS), and System Wide  Information Management (SWIM). In consultation with FAA officials, we  selected programs with high development and life-cycle costs, which  represented FAA’s two major modernization initiatives—the Air Traffic  Control Modernization and the Next Generation Air Transportation  System (NextGen)—and reflected different stages of life-cycle maturity.  These studies were not intended to be generalizable, but instead to  illustrate the status of a variety of programs. To determine the extent of  each program’s implementation of sound EVM, we compared program  documentation with the fundamental EVM practices implemented on  acquisition programs of leading organizations, as identified in our cost  guide. We determined whether the program implemented, partially  implemented, or did not implement each of the 11 practices. We further  analyzed the EVM data obtained from the programs to assess the program  performance against planned cost and schedule targets. Finally, we  interviewed program officials to obtain clarification on how EVM practices  are implemented and how the data are validated and used for decision- making purposes. Regarding the reliability of cost data, we did not test the  adequacy of agency or contractor cost-accounting systems. Our evaluation  of these cost data was based on what we were told by the agency and the  information they could provide.
To determine whether FAA is effectively overseeing compliance with its  EVM policies, we reviewed the quality and completeness of the agency’s  surveillance efforts on its system acquisition programs. Specifically, we  reviewed the agency’s EVM assessment reports for programs, FAA- developed EVM assessment criteria, and other relevant documents. We  further compared the results of FAA’s EVM assessment for each of the  selected case study programs with the results of our case evaluation to  ascertain the extent to which the results were in agreement. We also  interviewed key agency officials and observed FAA EVM working group  meetings to obtain information on the agency’s ongoing surveillance  efforts and issues regarding these efforts.
To evaluate whether FAA is using EVM data as part of its IT investment  management process, we analyzed senior executive management  briefings, OMB business cases (exhibit 300), and other key management  reports on program status. Specifically, we analyzed briefings and status  reports to determine the types of EVM metrics used in describing program  status for senior-level decision-making purposes. We also compared this  analysis with the key best practices recognized within the federal  government and industry for the implementation of EVM, as well as for the  execution of sound IT investment management. We also interviewed key  agency officials to obtain information on the extent of executive-level  EVM awareness and clarification on how EVM is used in FAA’s capital  planning process.
We conducted this performance audit from November 2007 to July 2008 at  FAA offices in Washington, D.C., in accordance with generally accepted  government auditing standards. Those standards require that we plan and  perform the audit to obtain sufficient, appropriate evidence to provide a  reasonable basis for our findings and conclusions based on our audit  objectives. We believe that the evidence obtained provides a reasonable  basis for our findings and conclusions based on our audit objectives.
section_title: Appendix II: Overview of Industry Guidelines That Support Sound EVM
paragraphs: Organizations must be able to evaluate the quality of an EVM system to  determine the extent to which the cost, schedule, and technical  performance data can be relied on for program management purposes. In  recognition of this, the American National Standards Institute (ANSI) and  the Electronic Industries Alliance (EIA) jointly established a national  standard for EVM systems—ANSI/EIA-748-B (commonly referred to as the  ANSI standard). This standard consists of 32 guidelines addressing  organizational structure; planning, scheduling, and budgeting; accounting  considerations; analysis and management reports; and revisions and data  maintenance. These guidelines comprise three fundamental management  functions for effectively using EVM: establishing a sound EVM system,  ensuring that the EVM data are reliable, and using earned value data for  decision-making purposes. Table 4 lists the management functions and the  ANSI guidelines.
section_title: Appendix III: Case Studies of FAA’s Implementation of EVM
paragraphs: We conducted case studies of four major system acquisition programs:  ASR-11, ERAM, SBS, and SWIM. For each of these programs, the following  sections provide a brief description of the system; an assessment of the  system’s implementation of the 11 key EVM practices; and, where  applicable, an analysis of the system’s recent earned value data and trends.  These data and trends are often described in terms of cost and schedule  variances. Cost variances compare the earned value of the completed  work with the actual cost of the work performed. Schedule variances are  also measured in dollars, but they compare the earned value of the work  completed with the value of work that was expected to be completed.  Positive variances are good—they indicate that activities are costing less  than expected or are completed ahead of schedule. Negative variances are  bad—they indicate activities are costing more than expected or are falling  behind schedule.
subsections: section_title: Airport Surveillance Radar
paragraphs: ASR-11 is a joint program sponsored by both FAA and the U.S. Air Force  to replace outdated primary radar systems at selected airports with an  integrated digital primary and secondary radar system. This investment is  also to replace the deteriorating infrastructure supporting current radar  systems with new radar facilities, including advanced grounding and  lightning protection systems, digital or fiber-optic telecommunications,  emergency backup power supplies, and enhanced physical security. The  contract was awarded in 1996 and is managed by the Air Force. The total  program cost is currently estimated at $1.15 billion, with $437.2 million  remaining to be spent (see table 5). ASR-11 is currently being deployed  across the country. As of April 2008, 44 of the total of 66 systems were  operational. FAA plans to complete deployment of these systems by March  2010.
ASR-11 fully met 2 of the 11 key practices and partially met 9 others (with  justification for not being able to fully meet these). For example, ASR-11  fully met the practices involving using earned value information to  mitigate risks and updating baselines as changes occur. ASR-11 partially  met the other practices because, while the program implemented many  key components of an effective EVM system, ASR-11 is limited in what it  can measure and validate. There are two reasons for these limitations:   (1) the contract was awarded in the mid-1990s, before FAA implemented  its EVM requirements, and (2) FAA does not have the authority to obtain  data on actual costs expended by the contractor or Air Force because Air  Force is the contracting agency. To work around these constraints, FAA’s  ASR-11 program management team developed a system that allows them  to approximate EVM reporting and tracking at the program level on the  basis of estimated (not actual) costs. Specifically, ASR-11 established a  program-level work breakdown structure, developed a work schedule, and  identified who will perform the work. ASR-11 has also implemented EVM  using estimated data and analyzes its estimated EVM results against its  performance measurement baseline. While valuable, this approximation  does not fully meet the key practices needed to establish a sound EVM  system and ensure data reliability. However, FAA is limited in what it can  measure and how it can validate the work accomplished and the dollars  spent. Table 6 shows the detailed assessment results for ASR-11.
subsections: section_title: Earned Value Data Show Cost Overruns and Schedule Delays
paragraphs: ASR-11 experienced negative cost variances between January 2007 and  December 2007 (see fig. 2). In this period, the program exceeded cost  targets by $19.2 million—which is 3.3 percent of the program budget for  that time. Similarly, the ASR-11 program was unable to complete   $20.6 million (3.4 percent) of the work planned in this period. The main  factors contributing to the cost and schedule variances were high  construction costs, due mainly to the effects of Hurricane Katrina, and an  unusually long real estate acquisition for the Green Bay, Wisconsin, ASR- 11 site. Program officials are currently working on a request to rebaseline  the program due to the current high variances.
Based on the program performance trends, we estimate that the program  will overrun its budget by between $7.6 million and $53.3 million. Our  projection of the most likely cost overrun will be about $9.8 million. In  comparison, the ASR-11 program office estimates about a $6.2 million  overrun at program completion.
section_title: En Route Automation Modernization
paragraphs: ERAM is to replace existing software and hardware in the air traffic  control automation computer system and its backup system, the Direct  Access Radar Channel, and other associated interfaces, communications,  and support infrastructure at en route centers across the country. It is a  critical effort because it is expected to upgrade hardware and software for  facilities that control high altitude air traffic. The contract was awarded in  2002. The ERAM prime contract requires EVM to be accomplished by the  contractor in accordance with the ANSI standard. The total program cost  is estimated at $2.93 billion, with $1.2 billion still to be spent (see table 7).  ERAM consists of two major components. One component has been fully  deployed and is currently in operation at facilities across the country. The  other component is scheduled for deployment through fiscal year 2009.
ERAM fully met 2 of the 11 key practices for implementing EVM and  partially met 9 others (with justification for 6 of these). ERAM fully met  the practices involving using EVM data to mitigate risks and updating  performance baselines as changes occur. ERAM partially met 6 other  practices, with justification, because of limitations in the earned value data  for the government portions of the program. Specifically, ERAM manages  its contractor using an EVM system that includes a work breakdown  structure, master schedule, and performance baseline. However, ERAM  did not implement a comprehensive EVM system that integrates  government and contractor data because this was not a requirement when  the program was initiated in 2002. Program officials reported that they  implemented a work-around to approximate the government portion of the  program. The ERAM program partially implemented the 3 remaining  practices associated with data reliability. Anomalies in the prime  contractor’s EVM reports affect the program’s ability to execute the work  plan, analyze variances, and estimate the cost of the program at  completion. Table 8 shows the detailed assessment results for ERAM.
subsections: section_title: Earned Value Data Show ERAM Is Ahead of Schedule and Under Budget, but Data May Not Be Reliable
paragraphs: Using contractor-provided data, our analysis indicates that the ERAM  program experienced positive cost and schedule performance in 2007 (see  fig. 3). Specifically, from January 2007 to December 2007, the contractor  was able to outperform its planned targets by finishing under budget by  $11.3 million (1 percent of the work for this period) and by completing  $25.5 million, or 3 percent, worth of work beyond what was planned.  Factors that contributed to the positive cost and schedule variances  include less labor needed than planned, savings in materials purchased,  and higher productivity and efficiency. For example, the program  contractor reported a positive schedule variance in 2007 due to technology  refresh activities at the William J. Hughes Technical Center being  accomplished earlier than planned.
However, as we have previously noted, our analysis of ERAM’s contractor  performance reports uncovered a number of anomalies that raise  questions regarding the reliability of these data. Furthermore, the  contractor did not provide justification for these anomalies, and the  program office was unable to explain the occurrences.
section_title: Surveillance and Broadcast Services
paragraphs: SBS is to provide new surveillance solutions that employ technology using  avionics and ground stations for improved accuracy and update rates and  provide shared situational awareness (including visual updates of traffic,  weather, and flight notices) between pilots and air traffic control. These  technologies are considered critical to achieving the FAA strategic goals of  decreasing the rate of accidents and incursions, improving the efficiency  of air traffic, and reducing congestion. The program is currently estimated  at $4.31 billion, with a total of $4.11 billion planned to be spent for the  remaining work until completion (see table 9).
The program reported that the achievement of cost, schedule, and  performance goals was expected to be tracked and monitored through  FAA best practices and established EVM processes defined by FAA.  Monthly program reviews, detailed schedule updates, and EVM reporting  are expected to be applied in accordance with the FAA EVM policy. Future  contracts are expected to include all EVM requirements since established  by FAA and to be consistent with the industry standards and OMB A-11  guidance.
SBS implemented all 11 of the key practices necessary to ensure that the  program was planned in accordance with industry standards, that the  resulting EVM data were appropriately verified and validated for  reliability, and that the SBS management team was using these data for  decision-making purposes. Table 10 shows the detailed assessment results  for SBS.
subsections: section_title: Earned Value Data Show SBS Performance Is under Cost Targets, but over Schedule Targets
paragraphs: From December 2007 to February 2008, SBS cost performance has been  mixed against its planned cost and schedule targets (see fig. 4). The  program was able to outperform its cost targets by $3.0 million. However,  the SBS program was unable to complete $4.5 million, or 6 percent of the  value of planned work. The program indicated that the positive program  cost variances were associated with key activities (including the  preliminary design review) taking less effort than expected to complete.  The negative schedule variances were primarily due to scheduling errors  and system-level testing issues. In particular, the system-level testing was  delayed due to a lack of readiness of the test environment, test  documentation, and equipment.
section_title: System Wide Information Management
paragraphs: As the key information management and data sharing system for NextGen,  SWIM is expected to provide policies and standards to support data  management, along with the core services needed to publish data to the  network, retrieve the data, secure the data’s integrity, and control access  and use of the data. SWIM is also expected to reduce the number and  types of interfaces and systems, reduce unnecessary redundancy of  information, better facilitate information-sharing, improve predictability  and operational decision making, and reduce cost of service. The FAA’s  Joint Resource Council established a baseline for the first 2 years of the  first segment of this program on June 20, 2007. The estimated life-cycle  cost for the total SWIM program is $546.1 million, with $501.3 million still  to be spent (see table 11).
SWIM is in the planning phase of its life cycle, which entails setting up the  program’s EVM system of internal controls and the resulting performance  measurement baseline. EVM data will not be available until development  work begins in fiscal year 2009.
Our assessment of SWIM’s EVM process maturity indicated that the  program is on track in its implementation of EVM. Specifically, it has fully  met two of the six key process steps for ensuring that the program is  planned in accordance with industry standards. SWIM also has work under  way to address the other four steps. We did not assess SWIM in the five  key process steps related to EVM data reliability and use in program  decision making because the program has not begun development work at  this time. Table 12 shows the detailed assessment results for SWIM.
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Colleen Phillips (Assistant  Director), Kate Agatone, Carol Cha, Neil Doherty, Nancy Glover, and  Teresa Smith made key contributions to this report.