Background

FAA catalogs its acquisition programs in its annually updated Capital  Investment Plan (CIP). The CIP identifies planned capital investment in  the National Airspace System (NAS) for the next 5 years consistent with  the amount requested in the agency’s annual budget submission.  Appendix C of the CIP, which identifies the anticipated budget line items,  is divided into five activities: (1) Engineering, Development, Test, and  Evaluation; (2) ATC Facilities and Equipment; (3) Non-ATC Facilities and  Equipment; (4) Facilities and Equipment Mission Support; and (5)  Personnel Compensation, Benefits and Travel. The CIP for fiscal years  2012 through 2016 contains 106 funded acquisition programs with  estimated total budgets (through 2016) of more than $14 billion. Of these  83 acquisition programs FAA considers ATC related, 18 involve  Engineering, Development, Test and Evaluation and 65 involve ATC  Facilities and Equipment. The 83 programs include 30 that have had  program baselines approved by FAA’s Joint Resources Council (JRC),  which is responsible for approving major programs. These 30 baselined  programs include communications, navigation, and surveillance systems  that are key to ATC operations. FAA considers 5 of the programs to be  foundational parts of NextGen, and all are key to modernizing the existing  ATC system. Figure 1 illustrates the universe of FAA acquisitions for  fiscal years 2012-2016.
FAA has developed and uses its Acquisition Management System (AMS)  to provide policies and guidance for managing ATC system programs  through all phases of the programs’ life cycles (see table 1). The Air  Traffic Organization (ATO) within FAA is responsible for operating,  maintaining, and modernizing the nation’s current ATC system.
The acquisition program baseline defines the cost, schedule and  performance baselines for the investment program. The JRC which  determines whether to approve a cost and schedule baseline also  approves rebaselining, through which the agency documents and  approves major changes to a program’s previously approved budget or  schedule. Rebaselining resets the estimated costs and schedule used to  determine how the program will be held accountable and can occur  before the program is deployed. Once a program is rebaselined, FAA  reports on the performance of the program based on the revised cost and  schedule. Although the rationale for rebaselining can be reasonable, as,  for example, when a program’s scope has been expanded, reporting a  program’s performance based on a rebaselined cost or schedule can also  skew or conceal from Congress and other stakeholders the program’s  actual total costs or overall timeline. We previously reported that the  absence of this information on rebaselining in ATO’s performance  reporting could cause managers and other stakeholders, including  Congress, to think that performance was better than it actually was. We  recommended that FAA regularly report on the overall, long-term  performance in acquiring ATC systems by providing in FAA’s annual  Performance and Accountability Report the original budget and schedule  baselines for each rebaselined program and the reasons for the  rebaselining. In response to our recommendation, FAA currently provides  this information in Appendix D of its CIP, where it details baseline cost  and schedule information for major acquisition programs.
NextGen involves changes to every aspect of air transportation (see   fig. 2). NextGen requires the acquisition of new integrated systems (both  software and hardware), flight procedures, aircraft performance  capabilities, and supporting infrastructure to transform the current air  transportation system into one that uses satellite-based surveillance and  navigation operations instead of ground-based radar. are intended to increase the efficiency and capacity of the air  transportation system while maintaining safety and accommodating  anticipated future growth. The planning for NextGen began in 2003 and  is now focused on implementing improvements in the midterm (by 2018)  and in the far term (by 2025).
GAO, Next Generation Air Transportation System: Challenges with Partner Agency and  FAA Coordination Continue, and Efforts to Integrate Near, Mid-, and Long-term Activities  Are Ongoing, GAO-10-649T (Washington, D.C.: Apr. 21, 2010).

	Most Ongoing ATC Programs Remain within Budget and on Schedule, but a Few Have Significantly Exceeded Initial Estimates

		Most Cost Estimates Remain on Target, but Cost Overruns for Three Key Programs in Total Exceeded $4 Billion

Of the 30 baselined FAA ATC programs we reviewed,19 have not  increased in cost, but 11 have experienced cost increases ranging from  $2 million to over $2 billion. And of the 19 programs whose costs have not  increased, 7 experienced a cost decrease while the remainder have not  changed significantly. However, the 11 programs that exceeded their  initial estimated costs account for over 60 percent of total program costs  for the 30 baselined programs—$11 billion of $17.7 billion. These 11  programs are among the most complex of FAA’s major acquisitions in  that each involves a large amount of software engineering. (See table 3.)
The 3 programs with the largest cost increases—totaling more than   $4 billion—are key to ATC modernization. Several factors contributed to  cost overruns for the Standard Terminal Automation Replacement  System (STARS), WAAS, and ERAM programs and required additional  congressional appropriations or reductions in program scope.
Our previous work disclosed that the near tripling of the STARS’s  budget resulted from insufficient involvement of stakeholders and  requirements growth.
The WAAS program began in 1998 with an initial cost estimate of $1  billion and a current estimate of $3 billion. We reported previously  that FAA’s lack of scientific and technical expertise resulted in  unplanned work and contributed to cost increases as well as delays in  the deployment schedule. Additionally, FAA changed how it  accounted for certain costs in the capital budget in 1999, which  further raised the cost estimate to $3.3 billion. FAA recently revised  that estimate down to the current $3 billion during the 2009  rebaselining because, according to FAA officials, they had met certain  program requirements in 2006.
As previously mentioned, ERAM is a key modernization system and  will be the backbone of the NextGen system. FAA originally submitted  to Congress an estimated cost of $2.1 billion in 2003, and the  program is now expected to cost about $2.4 billion––an increase of  about $330 million. According to FAA, various software issues (e.g.,  unsuccessful transmission messages and inaccurate data pairing of  aircraft and traffic display), as well as problems interfacing with other  facilities and systems, have contributed to the cost increases and  delays. The extent to which unanticipated requirements, unplanned  work, and underestimates of the complexity of software development,  among other factors, have contributed to other FAA ATC acquisition  program cost overruns and scheduling delays is discussed later in this  report.

	Schedules for Roughly Half of ATC Programs Are on Track, but Half of ATC Programs Are Delayed

		Half of ATC Programs Are on or Ahead of Schedule and Half Are Delayed

We found that 15 of the 30 baselined programs either have experienced  no change in schedule or were completed early or on time; however, the  other 15 programs are projected to be completed later than originally  estimated. These delayed programs range from the Integrated Display  System, which will consolidate information from several weather  subsystems into a single display, which FAA expects to be completed 2  months after its initial estimated completion date, to WAAS, which FAA  estimates will be completed in 2013—more than 14 years after its initial  estimated completion date (see table 4). Ten of the 15 programs with  schedule delays also experienced cost increases. However, even if a  schedule delay does not result in a direct cost increase to that program,  the delay can lead to increased costs for FAA because FAA staff must  continue to manage the acquisition over the longer term as it is being  implemented, as well as maintain any legacy system that the program is  replacing. Because of program interdependencies, a schedule delay can  also affect how and when other programs will be implemented.

	Several Factors Contributed to Cost Increases and Schedule Delays, and Some Could Hamper NextGen Implementation

		A Number of Factors Have Contributed to Cost Increases and Delays

Cost increases and schedule delays occurred because of several factors,  all of which have been long-standing challenges for FAA and some of  which continue to affect programs despite FAA efforts to mitigate the  factors. Specifically, these factors include (1) additional, unanticipated  system requirements work; (2) insufficient stakeholder involvement  throughout system development; (3) underestimates of the complexity of  software development; and (4) unanticipated events, including funding  decreases or work stoppages (see table 5). Of the 30 programs we  reviewed, 15 experienced cost increases, schedule delays, or both, and  we were able to determine that cost increases or schedule delays for 11  were attributable to one or more of these factors.
Following are some examples of how these contributing factors led to  cost increases or schedule delays in some of FAA’s ATC baselined  programs:
Unanticipated requirements or work: For nine of the programs in table  5, FAA has had to undertake substantially more development work than  planned because FAA program officials originally misjudged the extent  to which commercial off-the-shelf nondevelopmental solutions, such as  those procured by another agency, would meet FAA’s needs. For  example, although WAAS was being developed by an integrated  product team that included representatives from several FAA offices,  the team did not effectively resolve problems in meeting a required  performance capability—that pilots be warned in a timely manner when  a system may be providing them with potentially misleading and  possibly hazardous information. These actions resulted in unanticipated  work and contributed to the rise in WAAS’s cost from the original  estimate of $509 million in 1994 to about $2 billion in 2005.
Insufficient stakeholder involvement: As we previously reported,  ERAM was designed at a time when air traffic controllers did not  participate in efforts to design and test new systems. Because active  users of the system from different locations could not provide insight  early on, issues that could have been addressed early in the design  phase were not addressed. In response, FAA has taken steps to  improve the testing of new systems in order to reduce the likelihood of  larger-than-anticipated software issues arising during system  implementation. For example, FAA and the controllers’ union recently  entered into a memorandum of understanding to bring controllers into  the testing and evaluation phase of ERAM. Under this agreement,  the controllers’ union will have ERAM technical, evaluation, and  training representatives, as well as a team of 16 controllers (including  12 from en route facilities and 4 from terminal facilities), who will be  detailed to test and validate software fixes with contractor engineers  at the FAA Technical Center (Tech Center). In addition, our previous  work disclosed that the near tripling of the Standard Terminal  Automation Replacement System’s budget resulted from insufficient  involvement of stakeholders and requirements growth—two systemic  factors that we found led to acquisitions missing their budget and  schedule targets. This, in turn, contributed to cost growth, schedule  delays, and eventually a reduction in the number of systems to be  deployed.
Underestimates of the complexity of software development: This  factor contributed to cost increases and schedule delays for ERAM,  as well as issues with costs, scheduling, or both for two other  programs. In 2010, FAA tested ERAM at two key sites (the Seattle  and Salt Lake en route centers) on live air traffic, usually late at night  when air traffic volume was low. During this testing, FAA encountered  both anticipated and unanticipated software issues, which prompted  the test sites, at times, to revert to using FAA’s legacy en route  computer system. Specifically, software instructions to a controller in  one sector to hand off control of an aircraft to a controller in an  adjacent sector failed, and flight data were lost or reassigned to  another flight. While some testing at FAA’s Tech Center preceded  testing at the two key sites, the Tech Center could only test limited  scenarios, and none of the scenario testing identified this software  error. In addition, as discussed earlier, ERAM was designed during a  time when air traffic controllers did not participate in efforts to design  and initially test new systems. FAA anticipated the potential for  software issues at the outset of the program and initially scheduled  approximately 6 to 9 months of contingency time between the time it  achieved initial operating capability and operational readiness  demonstrations at these sites, leaving little buffer for any potential  delays. FAA worked with its contractor to correct a number of software  issues, but further testing on live air traffic at the two test sites  continued to produce critical safety errors. As a result, in March 2010,  FAA decided, with the support of the air traffic controllers’ union, to  halt all ERAM testing on live traffic and to revise the deployment  schedule. The program was rebaselined in June 2011, and the  program’s completion date was extended from December 2010 to  August 2014. As a result of the schedule delays, the rebaselined cost  estimate increased from $ 2.1 billion to $2.4 billion.
Unanticipated events: Unanticipated events at implementation sites  and unanticipated funding issues have delayed several programs’  schedules and increased costs. For example, Airport Surveillance  Radar-11 was originally scheduled to be completed in June 2009 but  was delayed to June 2010. FAA reports indicated that the delay was  due to an unusually protracted real estate acquisition at one site and  issues involving validating performance during seasonal radar  operations from other another site. Similarly, FAA’s Runway Status  Lights program—which involves installing airport lighting equipment  that visually signals to pilots when it is unsafe to enter, cross, or begin  takeoff on a runway—has experienced schedule delays because of  construction issues at five sites (Charlotte, North Carolina; Fort  Lauderdale, Florida; Las Vegas; Minneapolis; and Washington- Dulles). FAA officials attributed some of these delays to the furlough  of some FAA employees in July 2011 and a freeze on contractor  funding during the furlough, which resulted in work stoppage orders  for several projects—including Runway Status Lights. FAA program  managers will need to assess the impact of the furlough on other  programs that had experienced work stoppage orders, including ADS- B, the Standard Terminal Automation Replacement System, SWIM,  WAAS, and various weather programs.

		Interdependencies among NextGen ATC Programs Could Slow NextGen Implementation

The interdependencies of ATC acquisition programs have become more  prominent as the NextGen program shifts from planning to  implementation, so that cost increases and schedule delays in one  program could have a cascading effect on other programs. As discussed  earlier, due to the integrated nature of NextGen, the development and  delivery of many of its component programs are mutually dependent on  the development and delivery of one or more other programs. For  example, ERAM, FAA’s new en route computer system, is critical to the  delivery of ADS-B capabilities such as broadcasting flight information.  ERAM is also pivotal to the on-time implementation of two other key   which is  NextGen programs—Data Communications (DataComm),estimated to cost about $3 billion, and the NextGen information  technology architecture, SWIM, which is estimated to cost over $550  million. Due in part to ERAM’s delay, FAA was forced to delay the Data  Communications baseline date by approximately 6 months, rebaseline  SWIM segment 1, and delay the SWIM segment 2 baseline date to 2012.  The longer-term effects of these delays are unclear, but certain SWIM  capabilities could be delayed for several years, and the progress of other  programs that are dependent on SWIM’s system integration could be  hindered, as well. Thus, looking more broadly, the implementation of  NextGen—both midterm (now through 2018) and far-term (2019-2025)  schedules—will be affected by how well FAA manages program  interdependencies.
GAO-10-629. developed a full listing of how ERAM schedule slippages, or slippages in  other programs that are critical to NextGen, could either affect other  programs’ implementation schedules or delay the implementation of  capabilities and improvements. In 2008, we recommended that FAA  improve the usefulness of ATO’s acquisition performance reporting by  including information in the agency’s Performance and Accountability  report or elsewhere on the potential effect of any budget or schedule  slippages on the overall transition to NextGen. This recommendation  remains open, as FAA has not definitively indicated how it will track  slippages that will affect other dependent NextGen programs. FAA’s  acquisition management system was not designed for managing NextGen  programs in an integrated way. To assist in managing NextGen portfolios,  FAA is starting to monitor all the activities of a particular operational  improvement to ensure integration is on track. As we noted in the 2010  report, as this approach is more fully implemented, it will likely clarify the  impact of slippages in one program’s schedule on the implementation  status of other NextGen programs and operational capabilities. In  addition, as we will discuss in the next section, FAA is developing an  Integrated Master Schedule for the entire NextGen initiative that is, in  part, intended to show how changes in program schedules affect other  programs and the timelines for the NextGen initiative as a whole.  However, as we discuss later, the schedules for the four programs we  reviewed in detail are not reliable. Reliable schedules at the program level  will be needed to develop a reliable Integrated Master Schedule for  NextGen.

		Program Management Changes Are Aimed at Addressing Past Challenges

According to FAA, it is taking actions to address the factors that have  contributed to cost increases and schedule delays. In 2011, FAA  assessed the NextGen effort as part of its Foundation for Success  initiative and has implemented the “Idea to In-Service Management”  (I2I) process, which it believes will result in improvements in the way the  FAA develops, acquires, and implements new NextGen capabilities from  conception through implementation. The I2I concept is intended to  improve collaboration early in the acquisition process, resulting in better  defined capabilities and an early indication of cost and benefits. These  enhancements are intended to resolve many of the challenges associated  with overall program management and enable FAA to focus on program  management best practices. FAA believes that I2I will also result in  improvements in specific areas that have presented challenges in the  past, such as cost estimating, anticipating requirements and work,  stakeholder collaboration, software development, and systems  integration.
Also in 2011, FAA implemented a reorganization of the NextGen  Operations and Planning Office and ATO which FAA believes will support  the I2I process and improve acquisitions of NextGen programs.  Specifically, FAA created a NAS Lifecycle Planning Division within the  NextGen Operations and Planning Office to focus the integration of  NextGen programs from a cost, schedule, and systems capability  perspective. Within ATO, FAA established a new Program Management  Office, which puts the responsibility for the program management of all  NextGen and other major ATC acquisitions within a single organization.  By combining program managers into one organization, FAA hopes to  create a stronger acquisition program and improve the consistency and  implementation of best practices. According to FAA, these organizational  changes allow responsibilities for acquisitions to be better defined to more  efficiently set strategic direction, define operational requirements, ensure  system integration, oversee implementation processes, and ensure  accountability throughout the acquisition life cycle.
To improve the acquisitions management process, FAA has also divided  large acquisition programs into segments. A segmented or phased  approach is being taken with programs like SWIM and CATMT. This  approach breaks a larger program into smaller and more manageable  pieces to lower the risk. In the past, we noted that this approach can  improve management by providing for midcourse corrections and, thus,  help FAA avoid costly late-stage changes. However, this approach can  also increase the duration and possibly the cost of the program.
According to FAA officials, a segmented approach allows the agency to  more effectively manage acquisitions at both the program and enterprise  architecture level. An enterprise architecture approach provides the  structure to relate organizational mission, vision, and goals to business  processes and the technical or information technology infrastructure  required to execute them. FAA officials stated that many of the factors we  identified that contributed to cost increases and schedule delays highlight  the need for an enterprise-level perspective throughout the acquisition  process. The I2I process is intended to provide an enterprise-level focus  and improve collaboration across related programs.

	For Programs We Reviewed, FAA Did Not Consistently Meet All Characteristics of Quality Cost Estimates or Schedule Best Practices

		Selected Cost Estimates Were Generally Comprehensive and Well- Documented, but Accuracy and Credibility Need to Be Improved

Our review of the ADS-B, CATMT, SWIM, and WAAS cost estimates  showed that while each program followed at least some of the four  characteristics of high-quality and reliable cost estimates—well- documented, comprehensive, accurate, and credible—none of the  programs adhered closely enough to those characteristics to create a  reliable cost estimate. As previously noted, these characteristics  incorporate the 12 steps consistently applied by cost-estimating  organizations throughout the federal government and industry and  considered best practices for developing cost estimates. The results of  our review of the ADS-B, CATMT, SWIM, and WAAS cost estimates,  which are summarized in table 6, show that they were most aligned with  the characteristic of comprehensive cost estimates but need improvement  in the other three areas, particularly with the characteristics of accurate  and credible estimates. Imprecise estimates can result in Congress  unnecessarily authorizing and appropriating millions of dollars for  programs. As noted, in some cases, FAA, in order to stay within the  original cost estimate, modified a program’s requirements and Congress  had to appropriate more funds or reduce the scope to allow FAA to finish  the program.
FAA could better ensure that the cost estimates for these four programs,  as well as its other major acquisition programs, are reliable. Our work  shows that an assessment of these cost estimates for these programs, as  well as FAA’s other major acquisition programs, would allow FAA to  better understand if its cost estimation guidelines and our characteristics  of high-quality cost estimates are in fact being followed (See table 6).
Because the four programs were generally similar in the extent to which  they met each of the four characteristics, the following discussion  summarizes the strengths and weaknesses we found for each  characteristic across the four programs. A more detailed discussion of our  findings is contained in appendix IV.
Well-documented. Two of the four cost estimates we analyzed  substantially met the characteristic of being well-documented; the  other two partially met this characteristic. A well-documented cost  estimate is thoroughly documented, including identifying specific  source data and their significance, detailing calculations and results,  and explaining why particular cost estimating methods were chosen.  In other words, sufficient documentation exists such that an unfamiliar  analyst could recreate the cost estimate and arrive at the same  results. For example, the SWIM estimate provided detailed  documentation describing the program, in addition to the  methodology, calculations, and quantities used to develop the  estimate. However, none of the four estimates sufficiently captured  the entire source data used, addressed its reliability, or described how  various forms of data from disparate sources were normalized (i.e.,  the data were described in like terms). For example, the WAAS  estimate was based, in part, on actual labor costs from a previous  contract, but the program office provided no evidence that these data  were evaluated for reliability or accuracy. Similarly, the CATMT  estimate routinely relied on subject matter expertise as a source for  assumptions, such as the cost of labor, but the estimate did not  document the experts’ qualifications, background, underlying  assumptions, or data sources. Moreover, we noted that three of the  four estimates often substantially relied on expert opinion rather than  on data. While expert opinion can be useful in the absence of data, it  is subjective and generally should be used sparingly for cost  estimates. Since data are the foundation of every cost estimate, data  quality affects the overall quality of the estimate. In addition, because  data are gathered from a variety of sources and take many different  forms, normalization helps to improve consistency with other cost  information and enable valid comparisons and projections.
Comprehensive. All four cost estimates we analyzed substantially met  the characteristic of being comprehensive. For an estimate to be  comprehensive, it should include full life-cycle costs, completely  define the program with sufficient detail, include cost elements that  are traceable to the statement of work or objective to ensure they are  neither omitted nor double counted, and document all cost-influencing  ground rules and assumptions. We found that the ADS-B, CATMT,  and SWIM cost estimates included all life-cycle costs, regardless of  program phase or funding source, and the ADS-B and SWIM cost  estimates completely defined the program with an appropriate level of  detail. In particular, the ADS-B cost estimate included cost estimates  for both government and contractor costs, and the WAAS cost  estimate thoroughly defined the program and reflected the current  schedule. The four estimates did not fully meet the comprehensive  characteristic because they lacked evidence that all cost influencing  ground rules and assumptions were considered.
Accurate. None of the four cost estimates met or substantially met the  characteristic of being accurate. The estimates generally adjusted  costs for inflation and contained few computation or mathematical  mistakes, but they were not regularly updated to reflect schedule and  requirement changes, did not provide evidence of documenting or  reviewing differences between planned and actual costs, and were  not based on historical cost data from comparable programs. For  example, the ADS-B, CATMT, and SWIM cost estimates provided no  evidence that they were updated to reflect program changes, such as  schedule slippages or varying assumptions, and did not include the  current actual costs of the program. Although the WAAS estimate  included evidence that it was updated to reflect major changes in  technical and program requirements, such as the four rebaselinings  the program has undergone since its 1998 inception, it did not include  evidence that estimated costs were replaced with actual costs as the  program advanced. Cost estimates that are not regularly updated with  current information cannot provide decision makers with accurate  information that is necessary, for example, when new system  requirements are called for under tight budget conditions. In addition,  comparing planned and actual costs enables cost estimators to  measure the accuracy of their estimates and refine their processes. In  addition, none of the four programs more than minimally used  historical data to develop their cost estimates. Had historical data  been used, the estimators would have had additional insight into  actual costs on programs that used similar technologies, which could  be used, for example, to challenge overly optimistic assumptions and  bring more realism to the cost estimate.
Credible. None of the four cost estimates met or substantially met the  characteristic of being credible, which includes obtaining an  independent cost estimate from a group outside the acquiring  organization, and cross-checking the major cost elements in that  estimate against cost drivers identified through sensitivity and risk  analyses. The ADS-B, CATMT, SWIM, and WAAS estimates lacked  credibility largely because FAA did not obtain an independent cost  estimate for any of the programs. In addition the CATMT, SWIM, and  WAAS estimates provided little evidence that it conducted sensitivity  or risk analyses. Instead, each program received independent cost  reviews as part of the investment decision process—even though  such reviews are not required by FAA policy. FAA stated that the  Investment, Planning and Analysis (IP&A) Office in the FAA Finance  Organization does not prepare independent estimates, but it is  organizationally independent of the acquisition programs and  conducts independent reviews of all cost estimates. However, an  independent cost review is less rigorous than an independent cost  estimate. According to our cost guide, an independent cost estimate is  often more accurate because the estimating team is further removed  from the program office and less prone to accept overly optimistic  assumptions or be burdened by organizational bias. Other federal  agencies, including the Department of Defense, require independent  cost estimates. Had an independent cost estimate been completed,  the estimating team and program team could have identified the major  differences between their estimates, reconciled those differences  where possible, and provided a synopsis of the two estimates and  their differences to acquisition program management. In addition,  without sensitivity and risk analyses, cost estimators cannot measure  the effects of varying assumptions, and managers cannot determine,  for example, the rational level of contingency reserves necessary to  cover increased costs that may result from uncertainties such as  unexpected design complexity, changes in requirements, or budget  shortfalls––all of which FAA ATC programs, and in particular NextGen  programs, have experienced in recent years. We found evidence that  some level of risk analysis was conducted for ADS-B, CATMT, and  SWIM, although the analysis was not sufficiently robust. For example,  key cost drivers were not identified, and additional context about how  the estimate could be affected by software design and development  issues was not included.

		Selected Schedules Did Not Substantially Meet Most Best Practices and Are, Therefore, Unreliable

We determined that the schedules for the four programs we reviewed are  unreliable because none met or substantially met all nine of the best  practices for developing a reliable schedule. (see table 7). For example,  none of the schedules fully met best practices for capturing all activities in  an integrated master schedule, identifying critical paths and reasonable  float for all activities, or assigning resources to those activities. Moreover,  none of the schedules had documentation that provided more than  minimal evidence that they conducted a schedule risk analysis. As was  the case with our review of cost estimates for the four programs, our work  regarding the schedules for these programs shows that an assessment of  the schedules, as well as schedules for FAA’s other major acquisition  programs, would allow FAA to understand if the nine best practices for  reliable schedules are being followed.
Because the scheduling best practices are interrelated in such a way that  deficiencies in one best practice will cause deficiencies in the others, a  schedule must meet or substantially meet all nine practices to be reliable.  For example, preparing a schedule that is program-wide––including an  integrated breakdown of the work to be performed by both the  government and its contractors over the expected life of the program––is  a best practice. If the schedule does not capture all activities, then there  will be uncertainty about whether activities are sequenced correctly or if  the schedule properly reflects the resources needed to accomplish the  work, which is also a best practice. Logic and durations (that is, the time it  takes to complete a specific activity) should be used and maintained to  ensure realistic start and completion dates and to reflect the true status of  the project––a necessary condition for conducting follow-on schedule risk  analyses. Moreover, if activities are not properly sequenced with logical  links, it will not be certain if the critical path—which represents the chain  of dependent activities with the longest total duration—is valid.  Collectively, the weaknesses in not fully meeting or substantially meeting  all nine key practices increases the risk of schedule slippages and cost  overruns since a well-defined schedule helps to identify the amount of  human capital and fiscal resources that are needed to execute the  program. Therefore, by not having reliable schedules, FAA cannot  conduct meaningful oversight of an acquisition program’s progress or  determine whether the program is achieving the desired results.
The following discussion summarizes the extent to which the schedules  for the four programs we examined met best practices. More detailed  information for each program regarding scheduling best practices is  presented in appendix V.
We reviewed the schedule prepared by FAA and found it did not fully  meet any of GAO’s nine scheduling best practices, resulting in an  unreliable schedule. Evidence provided in the ADS-B schedule indicates  that it substantially met three of the nine best practices and partially,  minimally, or did not meet the other six. For example, although the ADS-B  schedule provided evidence of periodic updating, it did not capture all of  the effort currently called for in the approved baseline for the entire ADS- B program and, therefore, was not a fully integrated schedule. Without  fully integrating government activities with contractor activities, and  thereby capturing all key activities, the schedule will not reliably estimate  the program’s completion. In addition, the ADS-B schedule we reviewed  did not identify critical paths or include a schedule risk analysis, which  uses statistical techniques to predict a level of confidence in meeting a  program’s completion date; did not logically sequence all activities and  establish their durations; and had excessive float on a majority of  current and planned activities. According to program officials, a number of  the issues our analysis identified were, in part, the result of the schedule’s  limited time frame, which covered only a defined transitional period  (October 2010 through April 2011) during which responsibility for about a  third of the effort passed from the FAA to its prime contractor. Officials  also stated that although their schedule contains critical activities, it has  not had a traditional critical path since its contractor began managing the  deployment of deliverables. The FAA uses contract options to order the  scope, sequence, and requirements for key milestones. Within those  options, the contractor has the authority to implement the sequence of  more discrete activities in the order they deem most appropriate. FAA  program officials plan to rectify this problem, noting that with negotiations  now completed, they will in the near future identify a critical path to span  all program milestones.
Because the CATMT program did not prepare an FAA schedule and  instead relied on its contractor schedule, we reviewed the contractor  schedule, which we found to be unreliable. Our analysis found that the  contractor’s CATMT schedule substantially or fully met four of the nine  best practices: capturing all activities, assigning resources, establishing  the durations, and updating the schedule. For example, the CATMT  contactor schedule pertains to the current phase of the program that is  being implemented in software releases, or phases. However, there was  no overarching FAA government owned schedule that accounts for all  software releases for the entire program and would thus delineate the  relation of current software release tasks to the upper-level milestones for  the overall CATMT program. The CATMT schedule included detailed  resource information, and the program office provided evidence that  resources are tracked in detailed labor-hour spreadsheets. We also found  that 90 percent of the activities were of short duration and that the  program office regularly reviews the schedule, which is in line with best  practices. On the other hand, five of the nine best practices were either  partially, minimally, or not met. Specifically, the CATMT schedule lacked  evidence indicating that it established a critical path, accurately identified  float between activities, integrated the schedule vertically and  horizontally, sequenced all activities, or performed a schedule risk  analysis. Regarding the critical path, our analysis determined that the  CATMT schedule does not identify a critical path for the entire program.
Instead, the program is being accomplished multiple 6-month spirals;  thus, there is only a critical path for each software release, not for the  program as a whole. Without a valid program-wide critical path FAA  management cannot determine which tasks, if they slip, will have the  most detrimental effects on the project finish date. We also found that 68  percent of the remaining activities to be completed had unreasonably high  float exceeding 1,000 days, meaning that those activities could slip about  5 work years without affecting the overall project finish date, a highly  unlikely scenario. The accurate identification of critical paths and float  are inextricably linked. For example, if the schedule is missing activities or  they are not correctly linked, float estimates will be miscalculated,  resulting in an invalid critical path. Without a schedule that can produce a  true critical path, the program office will neither be positioned to provide  reliable timeline estimates nor be able to identify when problems or  changes may occur and determine the impact they may have on  subsequent work. CATMT program officials acknowledged that the  schedule did not include program-wide critical paths but noted that a  critical path exist for individual segments of the program. They also noted  that a schedule risk analysis was not performed because it was not a  contractual deliverable.
Because the SWIM schedule did not fully or substantially meet any of  GAO’s nine scheduling best practices, we found it to be unreliable. The  SWIM program differs from the others in that it is an aggregation of  NextGen acquisition programs, each developing an aspect of the SWIM  information sharing capability. Because SWIM program managers are  reliant on schedule information from a number of other programs, SWIM  schedule integration is particularly important. However, our analysis found  that the SWIM schedule was not, by any measure, fully integrated  because it provided only a synopsis of the individual system implementing  program schedules and, thus, did not fully represent the work required to  complete the overall SWIM program. This resulted in float calculations  that were unrealistic, and the resulting critical path calculations were  invalid. In addition, while the many missing activities negatively impacted  the schedule logic and the accuracy of durations, it also made the  accurate allocation of resources and comprehensive integration of  schedule activities, both horizontally and vertically, impossible. We also  noted that FAA made no effort to identify a program-wide critical path.  Program officials said that because each of the system implementing  program schedules has its own critical path, involve disparate capabilities,  and are independent of one another their individual critical paths are not  accessible through the SWIM schedule software. They therefore are not  used for overall SWIM program management. We believe that the SWIM  program itself should have its own critical path that includes, at a  minimum, acceptance of major deliverables from the system  implementing program schedules. Without a program-wide critical path,  management does not have a clear picture of the underlying project tasks  that must be performed to achieve the overall program target completion  date. Finally, although there was no risk analysis conducted on this  schedule, our analysis found that this best practice was minimally met  because a risk analysis was conducted on a separate but related  schedule, and the SWIM program office considered risk to some extent.
Like the other three programs, we found the WAAS  program schedule  prepared by FAA unreliable because it did not fully or substantially meet  any of GAO’s nine scheduling best practices; however, we also reviewed  the contractor’s schedule for the same segment and found it fully or  substantially met six best practices. For example, FAA’s WAAS program  schedule did not fully sequence activities in the order in which they are to  be carried out. More specifically, the WAAS program schedule showed  nearly half of the remaining activities were missing sequenced logic,  causing us to question the calculated dates of activities. Logic is  necessary for a schedule to show program managers when activities are  expected to start and finish; when logic is missing, activity dates cannot  adjust correctly to changes in activities. To test the ability of the schedule  to dynamically update its dates due to changes, we artificially extended  the duration of an activity to 1,500 days, which changed the activity’s  finish date. However, the duration extension had no effect on successor  activities because this activity is not tied to any successor activities.  Extending the duration to 1,500 days also pushed the project planned  finish date from September 22, 2016, to June 29, 2017; however,  because the logic links are not in place, we questioned whether the  projected finish date under this scenario is reliable.
Moreover, the WAAS program schedule had too many artificial  constraints that were driving the start and finish dates for more than 70   Constraints are usually substitutes  percent of the remaining activities.for logic and can mean that the schedule is not well planned or feasible.  Constraints also greatly reduce the ability of the program to take  advantage of possible time savings. Further, our analysis found that the  schedule did not fully capture or assign resources to all government and  contractor activities; it also did not accurately allocate resources or  consistently establish the duration of activities. In addition, while WAAS  program officials told us that the schedule was integrated vertically and  horizontally, we did not find evidence of such integration. Furthermore, we  found the WAAS program office’s schedule did not identify a critical path  for the entire program. As noted earlier, critical path and float  determinations are closely related. Our analysis of the WAAS program  office schedule found that more than half of the remaining activities had  float of more than 1,000 working days, which we believe to be  unreasonably high. Without proper determination of float, management  cannot properly reallocate resources from tasks to other tasks without  adversely affecting the overall completion date. Although program officials  said that they maintained a risk register listing the potential risks that  could impact the schedule and adjusted the schedule for these risks, we  did not find evidence that the program office had conducted a risk  analysis of its schedule.
While the schedule prepared by the contractor did not fully or  substantially meet three of the scheduling best practices, it fully or  substantially met six: capturing, sequencing, assigning resources to, and  establishing the duration of all activities; establishing the critical path; and  identifying reasonable float between activities. For example, our analysis  found that all activity durations are consistently estimated in days and  adhere to a standard 5-day workweek that accounts for holidays, and no  activities were scheduled to begin on a weekend. Officials from the  contractor said duration estimates for the schedule are based on  historical information from past performance, comparable releases,  lessons learned, similar work, and other data requirements. In addition,  our analysis traced several critical paths in the schedule. Though we  found minor interruptions in the various critical paths, the schedule’s logic,  reasonable durations, and low total float estimates allow the calculation a  valid critical path.

		WAAS Schedule Risk Analysis Indicates Risks That Managers Could Mitigate to Avoid Delays

As noted, FAA did not perform a complete schedule risk analysis for any  of the four programs we reviewed and, thus, cannot accurately estimate  these programs’ completion dates with confidence. A schedule risk  analysis, which is one of our best practices for program scheduling, uses  statistical techniques to predict a level of confidence in meeting a  program’s completion date. The objective of the analysis is to develop a  probability distribution of possible completion dates that reflect the project  and its identified risks. This analysis can help program managers both  understand the most important risks to the program and focus on  mitigating those risks. Other federal agencies, including the Department  of Defense and the National Aeronautics and Space Administration,  require schedule risk analysis for major acquisitions; the Department of  Veterans Affairs, in response to a GAO recommendation,require schedule risk analysis for major construction projects.
The other three schedules did not have the required information to conduct a schedule  risk analysis. showed four potential risks to the project. We then conducted interviews  with FAA program and contractor staff and asked them to discuss other  potential risks to the project, including how the risk would affect the  project’s timeline and the likelihood of the risk occurring. Using this  information we identified an additional 16 risks for a total of 20 risks. The  fact that our interviews identified a relatively large number of new risks  could be an indication that the contractor did not systematically analyze  the full range of risks when developing the program’s risk register. We  then consolidated the 20 risks into 14 broader risks and tested how each  would impact the duration of specific activities in the schedule. We then  ran a Monte Carlo simulation, which consisted of the computer- generated results of 3,000 estimates of the future schedule based on the  activities in the schedule, the chance that some of the activities would be  affected by some risks and the predicted effect of those risks on the  duration of each activity.
We then analyzed the potential impact of risks on the program schedule.  Since risks can effect the schedule in various ways––for example, risks  can have a large impact on the durations of activities they affect, or they  can introduce critical paths that are different from the baseline critical  path–– we analyzed the marginal impact of each of the risks we identified  to determine which would have the greatest effect on the overall  schedule. We found the following three key risks to the program, only the  first of which (limited WAAS program office resources) was originally  identified by the contractor. The three risks are  limited WAAS program office resources such as staffing; delays in software yet to be released and additional changes to  software already released and in use; and a potentially optimistic schedule completion date.
Our schedule risk analysis showed the completion of the segment of the  WAAS program covered by the schedule could slip as much as 2 months.  Specifically, the analysis showed that there is less than a 5 percent  probability that the program segment would be completed by September  6, 2012, the current baselined date for completion. However, it appears  that the segment will be completed close to the deadline since we found a  50 percent probability that the program segment will be completed by  October 23, 2012 (about 1.5 months after the current estimated date for  completion); and an 80 percent probability that the program will be  completed by November 13, 2012 (about 2.25 months after the current  estimated date for completion).
Although we did not conduct a schedule risk analysis for other FAA  programs, the result of our analysis provides examples of the types of  risks that major acquisition programs face and the impact those risks can  have on meeting acquisition program milestones, especially given the  interrelation and interdependencies among NextGen acquisitions  discussed earlier. More information on our schedule risk analysis can be  found in appendix V.

		Lack of Reliable Program Schedules Will Hinder Development of an Overall NextGen Integrated Master Schedule

FAA has begun developing an integrated master schedule for the entire  NextGen initiative that would, in part, capture related NextGen program  schedules, governance activities, and other performance and financial  data to provide real-time monitoring of the overall NextGen effort.  However, the unreliability of the four program schedules for programs that  are integral to the NextGen initiative puts this high-level master schedule  at risk. Having a reliable integrated master schedule would enable FAA to  determine how delays in one program impact other programs and the  overall NextGen implementation timeline. While it is encouraging that  FAA is beginning to develop an integrated NextGen master schedule, the  effort could be hampered by the lack of schedule integration at the  program level, as well as the failure of individual program schedules to  meet best practices. For example, since FAA does not perform schedule  risk analysis on individual programs, it cannot predict with certainty if any  of the programs will be completed on time. Therefore, the integrated  master schedule for NextGen would be built on schedules that may not  reflect accurate program completion dates. Similarly, none of the four  schedules we reviewed, which were for segments of the entire program,  had reflected how tasks for the segment affected milestones for the entire  program. Without integrated schedules at the program level, an integrated  master schedule at the NextGen initiative level would be problematic.

		FAA has Taken Steps to Improve Acquisition Program Cost Estimates and Schedules

In response to our review of the extent that the four selected acquisition  programs met best practices for cost estimates and schedules, FAA  provided information on steps it is taking to improve its processes for both  cost estimates and schedules and noted that some of the cost estimates  and schedules we reviewed were developed before the improvements  were in place. FAA stated that strengthening its cost estimation process is  part of the seven key acquisition processes it has developed, including  program management, contractor management, requirements, risk  management, measurement and analysis, verification and validation, and  quality assurance. FAA stated that it has updated its Guidelines for FAA  Cost Estimating to be consistent with the GAO Cost Guide, filling in gaps  that it had identified during a comparison of its practices to those  contained in the Cost Guide. As of November 2011, 11 of the 12 best  practices are addressed in the guidelines. According to FAA officials, the  remaining best practice—involving the creation of independent cost  estimates—is unlikely to be implemented at FAA in the foreseeable future  because FAA believes the resources required to create independent  estimates are prohibitive in current budget environments. FAA has more  than tripled the number of cost estimators in the Investment Planning and  Analysis organization, many of which work with the acquisition program  offices to provide guidance on preparing estimates. Additionally, as part  of FAA’s effort to improve acquisition certification and training, the agency  is preparing to launch a cost estimating certification program. Coupled  with a competency-based training program, FAA believes the certification  program will enhance and improve consistency of the skills of FAA cost  estimators.
In describing its efforts to improve schedules, FAA stated that it views the  development and maintenance of integrated schedules as an inherent  and critical part of its seven key acquisition functions. FAA noted that  included in its standard process for acquisition schedules are toolkits that  require programs to develop integrated program schedules that address  all nine of GAO’s best practices. FAA stated that that the current  procedures for developing best practices were not fully in place when the  four programs we reviewed began the implementation phase.

	Conclusions

FAA has made improvements in its management of air traffic control  modernization acquisitions, and most of the 30 we reviewed are currently  within the original cost estimate and half are on schedule. FAA is also  taking steps to address past issues to ensure cost estimates and  schedules are more accurate in the future, including incorporating best  practices in its acquisitions guidance and policies. Nevertheless, our  review of the FAA acquisitions found that it has yet to fully implement  several GAO-identified best practices or follow others. Following best  practices is particularly important for FAA, which must manage large,  complex, and interdependent acquisitions associated with NextGen. Cost  estimates that are imprecise can result in Congress appropriating millions  of dollars for projects based on estimates that prove to be inaccurate, and  program schedule delays can increase costs and affect the  implementation of interdependent programs. In such cases, FAA will be  forced to reduce the scope of the programs to stay within the original  estimates or Congress will need to appropriate unanticipated funds to  complete the programs. Delays and cost increases in individual programs  could have a cascading effect on other programs and ultimately affect  FAA’s timelines and goals for NextGen implementation.
Our analysis of the cost estimates and schedules for the four programs  we reviewed indicates that FAA needs to further develop requirements for  critical cost estimation and schedule procedures. Independent cost  estimates can improve the accuracy and credibility of cost estimates and  better ensure that programs will be completed within budget. A schedule  risk analysis can help FAA determine the likelihood that a program will be  completed on time. FAA stated that it has no immediate plans to conduct  independent cost estimates due to current budgetary constraints. We  recognize that conducting independent cost estimates and schedule risk  analysis takes both financial resources and some time and that it may be  appropriate to limit one or both of these analyses to instances where a  program is particularly costly, complex, or on a compressed schedule.  However, conducting independent cost estimates, schedule risk analyses,  and other analyses called for in our best practices can not only help  minimize the risk of cost overruns and schedule delays, but also provide  FAA, congressional decision makers, and other stakeholders with  important information about these critical acquisitions.
It is also important that FAA develop master schedules at the individual  acquisition program level. FAA’s lack of a fully integrated master schedule  for the programs we reviewed hampers its ability to provide accurate  information on the schedule for these programs. This information will be  needed as FAA simultaneously works to develop an integrated master  schedule for the overall NextGen initiative. The use of an integrated  master schedule can assist FAA in monitoring a program, identifying  problems that could affect later stages of the program’s implementation,  improving the accuracy of cost estimates and schedules for individual  programs, and improving the accuracy of information FAA is compiling to  monitor the costs and schedules for the NextGen initiative.
FAA has incorporated 11 of our 12 steps that are associated with the  characteristics of a high-quality and reliable cost estimate into their  acquisition guidelines. However, our analysis of the four major programs  indicates that FAA has not adequately integrated all of the steps for these  programs into its cost estimation processes, and thus the estimates are  not reliable. Similarly, although FAA addresses our nine scheduling best  practices in its acquisition guidelines, our analysis of the schedules for the  four programs indicates that the schedules are not adequately following  these best practices and are not reliable. Although the cost estimates and  schedules for some of the four programs were developed prior to FAA’s  revision of acquisition guidelines, our work shows that FAA needs to  assess its major acquisition programs to understand if its guidelines and  other best practices are, in fact, being followed. Such an assessment  would then allow FAA to better ensure that best practices for cost  estimates and schedules are being applied.

	Recommendations for Executive Action

To improve cost estimates and schedules for NextGen and other major  air traffic control acquisition programs, GAO recommends that the  Secretary of Transportation direct FAA to take the following three actions  when appropriate for major acquisition programs based on a program’s  cost, schedule, complexity, and risk:
Conduct independent cost estimates and schedule risk analysis for  major acquisition programs.
Require a fully integrated master schedule for each major acquisition  program, including those that are components of NextGen. An  integrated master schedule should horizontally and vertically link all  program activities and milestones, including government and  contractor schedules and program segments.
Conduct an assessment of major acquisition programs to ensure they  meet all of the established best practices for cost estimates and  schedules contained in GAO guidance.
Given constrained budgets, FAA should determine which programs  should be subject to these recommendations, such as those that are  particularly costly, complex, or on a compressed schedule.

	Agency Comments

We provided a draft of this report to the Department of Transportation for  review and comment. DOT and FAA responded by email and did not  comment whether or not they agreed or disagreed with our  recommendations. DOT provided comments on the results of our analysis  of the cost estimates and schedules for the four programs we reviewed in  depth. In response to our finding that the ADS-B, CATMT, SWIM, and  WAAS estimates lacked credibility largely because FAA did not obtain an  independent cost estimate for any of the cost estimates, and provided  little evidence that they conducted sensitivity or risk analyses, FAA stated  it is not convinced that an independent organization will reduce the  uncertainty of cost estimates. FAA noted that it does not have an  independent organization such as the Department of the Navy’s Center  for Cost Analysis. However, FAA stated that the Finance Organization  within ATO assessed the ADS-B program office’s Basis of Estimate as  part of the JRC Decision and that this level of independence, combined  with specific entry and exit criteria, allowed the program offices to  manage these acquisitions so that costs were controlled, risks mitigated,  and technical parameters achieved, while adhering to the planned  milestone schedule. We agree that the Finance Organization assessment  of the two cost estimates provided some degree of independence and  may have improved the accuracy of the ADS-B estimates, but it is not  clear that such an independent review will guarantee similar results for  other programs. As we stated in the report, such an independent cost  review is less rigorous than an independent cost estimate. According to  our cost guide, an independent cost estimate is often more accurate  because the estimating team is further removed from the program office  and less prone to accept overly optimistic assumptions or be burdened by  organizational bias. DOT also provided technical clarifications, which we  incorporated into the report as appropriate.
We are sending copies of this report to interested congressional  committees, the Secretary of Transportation, and the Acting Administrator  of FAA.  In addition, the report is available at no charge on the GAO  website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-2834 or dillinghamg@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix V.

Appendix I: Objectives, Scope and Methodology

In response to a congressional request, we examined the Federal  Aviation Administration’s (FAA) ability to modernize, upgrade, and replace  the National Airspace System’s (NAS) facilities and equipment to meet  projected increases in traffic volumes, enhance the system’s safety, and  increase the efficiency of the air traffic control (ATC) system—a principal  component of the NAS. FAA’s ATC acquisitions are critical to maintaining  the NAS and transitioning to the Next Generation Air Transportation  System (NextGen) over the next 10 years. Given that some key legacy  and NextGen acquisitions have experienced schedule delays and cost  overruns, which may risk the timely implementation of NextGen, we (1)  determined whether the planned costs and schedules of current FAA ATC  acquisition programs have changed since they were first submitted to  Congress; (2) examined the reasons for any changes in planned costs  and schedules; and (3) assessed the extent to which select ATC  programs adhered to best practices for determining acquisition costs and  schedules.
To describe any changes in costs and schedules of the current 30 FAA  capital ATC acquisitions, we gathered and analyzed agency data on the  estimated cost and schedules of these ATC acquisitions. We drew upon  past work in which we undertook detailed reviews of the status of ATC  and other acquisition programs and obtained updated documentation as  necessary from FAA. We interviewed FAA officials to obtain information  on FAA’s acquisition process and summarized the status of all  acquisitions, including FAA’s original and current cost estimates and  completion dates. For baselined acquisitions, we compared estimated  costs when they were submitted to Congress for approval against their  current estimates, and we analyzed planned and actual schedules.
To determine the reasons for changes in cost estimates and schedules,  we interviewed FAA officials and FAA contractors and reviewed  acquisition documentation. We analyzed information on cost increases  and delays to determine if systematic issues exist that have effects on  other FAA acquisitions.
To determine the extent to which select ATC programs adhered to best  practices for determining acquisition costs and schedules, we conducted  an in-depth review of 4 of the 30 acquisitions programs: The Automatic  Dependent Surveillance-Broadcast (ADS-B) system, the Collaborative Air  Traffic Management Technologies (CATMT) system, the System Wide  Information Management (SWIM) system, and the Wide Area  Augmentation System (WAAS). We selected these four acquisitions  based on the following criteria: (1) existence of baselining, (2) the  acquisition is at a point in the acquisition process where risks can be  identified, and (3) the acquisition is key to NextGen and legacy systems.  In addition to interviews, we collected documentation, and we analyzed  and summarized the views and information collected. We also identified  best practices that FAA could adopt or strengthen to improve its  acquisitions cost estimation and scheduling and ensured that acquisitions  follow cost and schedule best practices outlined in our Cost Estimating  and Assessment Guide. We also performed a schedule risk analysis of  the WAAS program to determine the likelihood of the project finishing on  schedule.
GAO-09-3SP. a conclusion about whether the cost estimate is reasonable. Therefore, a  good cost estimate—while taking the form of a single number—is  supported by detailed documentation that describes how it was derived  and how the expected funding will be spent in order to achieve a given  objective. For example, the documentation should capture in writing such  things as the source data used and their significance, the calculations  performed and their results, and the rationale for choosing a particular  estimating method or reference. Moreover, this information should be  captured in such a way that the data used to derive the estimate can be  traced back to and verified against their sources. Finally, the cost  estimate should be reviewed and accepted by management to ensure  there is a high level of confidence in the estimate and the estimating  process.
Comprehensive: The cost estimates should include both government  and contractor costs of the project over its full life cycle, from inception  through design, development, deployment, operation, and maintenance to  retirement of the project. The cost estimate should be structured in  sufficient detail to ensure that cost elements are neither omitted nor  double counted, and they should document all cost-influencing ground  rules and assumptions.
Accurate: The cost estimates should provide for results that are  unbiased, and they should not be overly conservative or optimistic.  Estimates are accurate when they are based on an assessment of most  likely costs, adjusted properly for inflation, and contain few, if any, minor  mistakes. In addition, the estimates should be updated regularly to reflect  material changes in the project, such as when schedules or other  assumptions change so that the estimate is always reflecting the project’s  current status. Among other things, the estimate should be grounded in  documented assumptions and a historical record of cost estimating and  actual experiences on other comparable projects.
Credible: The cost estimates should discuss any limitations of the  analysis because of uncertainty or biases surrounding data or  assumptions. Major assumptions should be varied, and other outcomes  recomputed to determine how sensitive they are to changes in the  assumptions. Risk and uncertainty analysis should be performed to  determine the level of risk associated with the estimate. Furthermore, the  estimate’s results should be cross-checked, and an independent cost  estimate conducted by a group outside the acquiring organization should  be developed to determine whether other estimating methods produce  similar results.
After reviewing documentation submitted by FAA and information  obtained during interviews, we determined the extent to which the cost  estimates met the characteristics of cost-estimating best practices for the  four projects we reviewed.
Our review of project schedules was based on research that identified a  range of best practices associated with effective schedule estimating. In  addition, we obtained the consulting services of David Hulett, Ph.D., to  assist in our risk analysis of the WAAS project schedule. We also  conducted multiple interviews with project managers, contractors, and  schedulers to determine the extent to which current project schedules met  the best practices criteria. These nine practices are:  Capturing all activities: The schedule should reflect all activities (steps,  events, outcomes, and other factors) as defined in the project’s work  breakdown structure, including activities to be performed by both the  government and its contractors.
Sequencing all activities: The schedule should be planned so that it can  meet project-critical dates. To meet this objective, activities need to be  logically sequenced in the order that they are to be carried out. In  particular, activities that must finish prior to the start of other activities  (i.e., predecessor activities) and activities that cannot begin until other  activities are completed (i.e., successor activities) should be identified.  Identifying interdependencies among activities that collectively lead to the  accomplishment of events or milestones can be used as a basis for  guiding work and measuring progress.
Assigning resources to all activities: The schedule should realistically  reflect what resources (i.e., labor, material, and overhead) are needed to  do the work, whether all required resources will be available when they  are needed, and whether any funding or time constraints exist.
Establishing the duration of all activities: The schedule should reflect  how long each activity will take to execute. In determining the duration of  each activity, the same rationale, data, and assumptions used for cost  estimating should be used for preparing the schedule. Furthermore, these  durations should be as short as possible and should have specific start  and end dates. Excessively long periods needed to execute an activity  should prompt further decomposition of the activity so that shorter  execution durations will result.
Integrating schedule activities horizontally and vertically: The  schedule should be horizontally integrated, meaning that it should link the  products and outcomes associated with already sequenced activities.  These links are commonly referred to as “hand-offs” and serve to verify  that activities are arranged in the right order to achieve aggregated  products or outcomes. The schedule should also be vertically integrated,  meaning that traceability exists among varying levels of activities and  supporting tasks and subtasks. Such mapping or alignment among levels  can enable different groups to work to the same master schedule.
Establishing the critical path for all activities: With the use of  scheduling software, the critical path—the longest-duration path through  the sequenced list of activities—should be identified. The establishment  of a project’s critical path is necessary for examining the effects of delays  in any activity along this path. Potential problems that may occur on or  near the critical path should also be identified and reflected in the  scheduling of the time for high-risk activities (see the next activity,  “Identifying float”).
Identifying reasonable float: The schedule should identify float—the  time that a predecessor activity can slip before the delay affects  successor activities—so that schedule flexibility can be determined. As a  general rule, activities along the critical path typically have the least  amount of float.
Conducting a schedule risk analysis: A schedule risk analysis uses a  good critical path method schedule and data about project schedule risks,  as well as Monte Carlo simulation techniques, to predict the level of  confidence in meeting a project’s completion date, the amount of time  contingency needed for a level of confidence, and the identification of  high-priority risks. This analysis should focus not only on critical path  activities but also on other schedule paths that may become critical. A  schedule/cost risk assessment recognizes the interrelationship between  schedule and cost and captures the risk that schedule durations and cost  estimates may vary for a variety of reasons, including limited data,  optimistic estimating, technical challenges, lack of qualified personnel,  and other external factors. As a result, the baseline schedule should  include a buffer or a reserve of extra time. A reserve of extra time for  contingencies should be calculated by performing a schedule risk  analysis. As a general rule, the reserve should be held by the project  manager and applied as needed to those activities that take longer than  scheduled because of the identified risks. Reserves of time should not be  apportioned in advance to any specific activity since the risks that will  actually occur and the magnitude of their impact are not known in  advance.
Updating the schedule using logic and durations to determine the  dates: The schedule should use logic and durations in order to reflect  realistic start and completion dates for project activities. The schedule  should be continually monitored to determine when forecasted completion  dates differ from the planned dates. This information can be used to  determine whether schedule variances will affect downstream work.  Maintaining the integrity of the schedule logic is not only necessary to  reflect the project’s true status but is also required before conducting a  schedule risk analysis. The schedule should avoid logic overrides and  artificial constraint dates that are chosen to create a certain result on  paper. Individuals trained in critical path method scheduling should be  responsible for updating the schedule.
Based on our work, we determined the extent to which estimates and  schedules for the four projects we selected met each best practices  criterion:
Not Met—project officials provided no evidence that satisfies any  portion of the criterion.
Minimally Met—project officials provided evidence that satisfies a  small portion of the criterion.
Partially Met—project officials provided evidence that satisfies about  half of the criterion.
Substantially Met—project officials provided evidence that satisfies a  large portion of the criterion.
Met—project officials provided evidence that satisfies the entire  criterion.
We conducted this performance audit from August 2010 to February 2012  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Appendix II: Detailed Purpose and Status for 30 Acquisitions GAO Reviewed

This appendix contains detailed information for 30 individual air traffic  control programs. Each overview presents information and data that was  provided by FAA. The overviews provide a description of the program and  the cost and schedule status. The overviews are based on program office  reported information as of August 2011. In most cases, we did not  validate the data provided, but reviewed the data and performed various  checks to determine they were reliable enough for our purposes.

Appendix III: Assessments of Four FAA Program Cost Estimates

This appendix provides the results of our analysis of the extent to which  the processes and methodologies used to develop and maintain the four  FAA cost estimates meet the characteristics of high-quality cost  estimates. These characteristics incorporate the 12 steps consistently  applied by cost-estimating organizations throughout the federal  government and industry and considered best practices for developing  cost estimates and that are listed in table 2 of the report. The following  tables provide the detailed results of our analysis of the program cost  estimates for Automatic Dependent Surveillance-Broadcast (ADS-B),  Collaborative Air Traffic Management Technologies (CATMT), System  Wide Information Management (SWIM), and Wide Area Augmentation  System (WAAS). “Not met” means the program provided no evidence that  satisfies any of the criteria. “Minimally met” means the program provided  evidence that satisfies a small portion of the criterion. “Partially met”  means the program provided evidence that satisfies about half of the  criterion. “Substantially met” means the program provided evidence that  satisfies a large portion of the criterion. “Fully met” means the program  provided evidence that completely satisfies the criterion.

Appendix IV: Assessments of Four FAA Program Schedules

This appendix provides the results of our analysis of the extent to which  the processes and methodologies used to develop and maintain four FAA  integrated master schedules meet nine best practices associated with  effective schedule estimating. The following tables provide the detailed  results of our analyses of the schedules for the Automatic Dependent  Surveillance-Broadcast (ADS-B), Collaborative Air Traffic Management  Technologies (CATMT), System Wide Information Management (SWIM),  and Wide Area Augmentation System (WAAS) programs compared to the  nine best practices.  “Not met” means the program provided no evidence that satisfies any  portion of the criterion. “Minimally met” means the program provided  evidence that satisfies a small portion of the criterion. “Partially met”  means the program provided evidence that satisfies about half of the  criterion. “Substantially met” means the program provided evidence that  satisfies a large portion of the criterion. “Fully met” means the program  provided evidence that satisfies the entire criterion.

	Schedule Risk Analysis

A best practice that the WAAS contractor schedule did not meet is  conducting a schedule risk analysis, which is not required by FAA’s  schedule specifications. FAA officials told us that they do not conduct  schedule risk analysis. In August and September 2011, we performed our  own schedule risk analysis on the WAAS contractor schedule, through  which we analyzed the latest version of the contractor’s schedule  available to us at the time of the analysis.
A schedule risk analysis uses statistical techniques to predict a level of  confidence in meeting a program’s completion date. This analysis focuses  on critical path activities and on near-critical and other activities since any  activity may potentially affect the program’s completion date. The  objective of the simulation is to develop a probability distribution of  possible completion dates that reflect the program and its quantified risks.  From the cumulative probability distribution, the organization can match a  date to its degree of risk tolerance. For instance, an organization might  want to adopt a program completion date that provides a 70 percent  probability that the program will finish on or before that date, leaving a 30  percent probability that it will extend beyond, or overrun that date, given  the schedule and the risks. The organization can thus adopt a plan  consistent with its desired level of confidence in the overall integrated  schedule. This analysis can give valuable insight into what-if drills and  quantify the effects of program changes.
In developing a schedule risk analysis, probability distributions for each  activity’s duration have to be established. Furthermore, risk in all activities  must be evaluated and included in the analysis. Some managers focus  only on the critical path, but because we cannot be certain how long  activities will take, we cannot know the true critical path. Consequently, it  would be a mistake to focus only on the software-calculated critical path  (those activities in which, if delayed, will negatively impact the overall  project completion date) when some off-critical-path activity might  become critical if a risk were to occur. Typically, three-point estimates— that is, best, most likely, and worst-case estimates—are used to develop  the probability distributions for the duration of workflow activities.
Once the distributions have been established, a Monte Carlo simulation  uses random numbers to select specific durations from each activity  probability distribution and calculates a new critical path and dates,  including major milestone and program completion dates. The Monte  Carlo simulation continues this random selection thousands of times,  creating a new program duration estimate and critical path each time. The  resulting frequency distribution displays the range of program completion  dates along with the probabilities that these dates will occur. Table 17  provides a range of dates and the probability of the project’s completion  on those dates or earlier, based on our 3,000 iterations that are chosen at  random during the Monte Carlo simulation. For example, according to our  schedule risk analysis, there is a less than 5 percent chance that the  project will be finished on or before September 13, 2012. Likewise, there  is an 80 percent chance that the project will be finished on or before  November 13, 2012.
Because completion on any date is uncertain, it is more realistic to show  a range of possible completion dates than to focus on a single date. In  deciding which percentile to use for prudent scheduling, there is no  international best practice standard. The chosen percentile depends on  the riskiness and maturity of the project. For some projects, we  emphasize the 80th percentile as a conservative promise-of-completion  date. While the 80th percentile may appear overly conservative, it is a  useful promise-of-completion date if a number of new but currently  unknown risks (i.e., “unknown unknowns”) are anticipated. The 50th  percentile date may expose the project to overruns.
In the case of the WAAS contractor schedule, our analysis concluded that  management should realistically expect cutover, or completion, between  October 23, 2012, and November 13, 2012, the 50th and 80th  percentiles, respectively. The artificial must-finish date constraint of  September 6, 2012, built into the schedule is unlikely. Our analysis shows  the probability of completion by September 6, 2012, is less than 5 percent  likely with the current schedule without risk mitigation. There are two  reasons why the planned end completion date is not likely to occur,  according to the results of our schedule risk analysis. First, most risks are  threats only. Only two opportunities were identified during the analysis:  (1) the estimating error of the schedule may be between -10 percent and  15 percent, and (2) there is a 65 percent chance that the 11-day formal  shadow test will not be needed. Second, there are parallel paths within  the structure of the schedule that lead to merge points. If several paths  converge to one milestone, the latest merging path determines the date.  This “merge bias” cannot accelerate schedule dates and usually adds  structural risk to the schedule.

	Identified Risks

The contractor supplied six different risks that are currently identified in  the project’s risk register. Using these risks as a basis for discussion, we  interviewed 16 experts familiar with the project, including prime contractor  officials, FAA officials, and technical FAA consultants to identify any other  risks. Each interviewee was asked four questions to address four related  points.
To estimate the probability that an identified risk will occur on the  project in such a way that some activity durations are affected. The  estimated probability is translated into the percentage of iterations that  are chosen at random during the simulation. For example, if the  expert estimates weather will have a 10 percent chance of affecting  some activities, then, on average, weather risk will occur in 10 percent  of the Monte Carlo iterations.
If the interviewee believed the identified risk was likely, the  interviewee was asked to identify which activities’ durations would be  affected. For example, activities related to steel erection or concrete  pouring may be affected if the weather risk is realized and bad  weather occurs.
After the interviewee identified affected activities, the interviewee was  asked to provide a three-point estimate of the risk’s effect on  duration—low, most likely, and high. Estimates were provided as  percentages, which were applied to the activity durations in the Monte  Carlo simulation if the risk occurred. For example, if bad weather  occurs, the duration of a 10-day steel erection activity may be affected  a minimum of 110 percent, a most likely of 150 percent, or a  maximum of 200 percent. These percentages translate into increases  in the activity’s duration of 11 days under a low-risk scenario, 15 days  under the most likely risk scenario, and 20 days under the maximum  risk scenario. If the risk is not realized, there is no change to the  activity’s original estimated duration.
Finally, the interviewee was asked to identify any unaccounted for  risks.
We began the interviews with 6 risks and, through the interview process,  identified 16 more risks. During data analysis, some risks were  consolidated with others or eliminated because of limited data. In all, 14  risks were identified and incorporated into the Monte Carlo simulation.  These include 9 risk drivers, 4 existence risks, and 1 schedule duration  risk.
The final risk drivers used in the schedule risk analysis are as follows:
Some software Release 3B problem reports require testing with a live  geostationary satellite; software simulation environment may not be  able to test all requirements.
Potential difficulty repairing Fullerton contractor’s lab safety  computers.
Software delays in software Release 3A and additional changes to  software Release 2B are likely to delay the 3B schedule.
Parallel implementation of Releases 2B, 2C, 3A, and 3B create  competition for FAA resources.
Release 3B problem report testing is more complex and subject to  uncertainty.
FAA WAAS personnel move around, and inexperienced staff must be  trained on WAAS.
Workshare strategy between FAA and its prime contractor affects the  schedule for on-the-job training learning curve.
Changes to the simulation tools may be more difficult or more time- consuming to implement than anticipated.
The final existence risks are as follows:
FAA performs formal software release 3B shadow test.
Late problem report may come from FAA.
FAA program office is resource-limited.
Inserting existence risk activities does not affect dates within the baseline  schedule because the activities initially have zero duration. The activities  have duration only if they happen to occur during an iteration of the  simulation.
The final uncertainty risk is schedule without buffer may be optimistic.
Most risks were identified by multiple respondents during the interviews.  During data analysis, data from the interviews are combined and  analyzed to create ranges and probabilities for each of the 14 risks.
Because risks are multiplicative, several risks occurring on the same  activity may overestimate the true risk. That is, by default in the Monte  Carlo simulation, risks occur in a series, one after another, so that an  activity that has several risks may be unrealistically extended if all risks  occur. In reality, an activity may recover from two or more risks  simultaneously, so that the actual risk is not multiplicative. Therefore, to  avoid overestimation of risk, risks can explicitly be defined as occurring in  parallel rather than in series. Risks that occur in a series will occur one  after the other and add (or subtract) their respective effects on duration to  the affected activity. If risks occur in parallel, on the other hand, only the  maximum effect of all risks will affect the duration. For example, if the risk  of complexity adds 3 days to the duration of software development and  the risk of staff shortages adds 4 days, then development will extend 7  days if the risks are defined in series. However, the duration will extend  only 4 days if the risks are defined as parallel. This definition of parallel  risks helps temper any risk overestimated by a multiplication of risk  factors. We defined one risk in series: software delays in Release 3A and  additional changes to Release 2B are likely to delay Release 3B  schedule. All other risks were assumed to be parallel.
Most risks were assigned directly to existing activities in the schedule.  However, some risks required adjustments to the schedule. These  adjustments involved replacing lags with activities and inserting existence  risk activities.
Lags: During our initial analysis of the contractor’s schedule, we identified  25 remaining activities with lags and 21 remaining activities with leads  (negative lags). While lags represent the passing of time between activities,  they are often misused to put activities on a specific date or to insert a  buffer for risk. That number was reduced to 20 activities with lags and 6  activities with leads with the “Rev 1” schedule that was altered by the prime  contractor for our use in the schedule risk analysis. We replaced the 33-day  lag between the R3B Start Cutover and R3B End Cutover activities with an  actual activity, ID 258 “Cutover Task.” Replacing lags with actual activities  does not affect dates within the baseline schedule because the activities  have the same duration as the lags.
Existence risk: We identified some risks that would add an indeterminate  amount of time to the overall schedule if they were realized. For example,  if the Tijuana facility is not ready because software requirements are  misunderstood, it could add 4 to 26 days to the schedule in the form of  additional facility work.
Prioritizing risks and risk mitigation: Risks can affect the schedule in  several ways: They can have a high probability of occurring; have a large- percentage impact on the durations of the activities they affect; or they  can apply to risk-critical paths, which may differ from the baseline  deterministic critical path. Beyond applying 14 risks to the schedule, we  were interested in identifying the marginal impact of each risk. That is, we  were interested in identifying which risks have the largest impact on the  schedule because these were the risks that should be targeted first for  mitigation.
To find the marginal impact of a risk on the total project risk at a certain  percentile, the Monte Carlo simulation was performed with the risk  removed. The difference between the finish dates of the simulation with  all the risks and the simulation with the missing risk yielded the marginal  impact of the risk. Table 18 gives the priority of risks at the 80th percentile  and the marginal impact of each risk.
The marginal impact directly translates to potential calendar days saved if  the risk is mitigated. Once risks are prioritized at the percentile desired by  management, a risk mitigation workshop can be implemented to deal with  the high-priority risks in order. The prioritized list of risks will form the  basis of the workshop, and risk mitigation plans can be analyzed using  the risk model to determine how much time might be saved. Project  managers cannot expect to completely mitigate any one risk, and it is not  reasonable to expect to mitigate all risks. In addition, risk mitigation will  add to the project budget. However, some opportunities may be available  to partially mitigate risks.

Appendix V: GAO Contact and Staff Acknowledgments

	GAO Contact

	Staff Acknowledgments

In addition to the contact named above, individuals making key  contributions to this report include Edward Laughlin and Karen Richey  (Assistant Directors), Lindsey Bach, David Brown, Pamela Davidson,  Tisha Derricotte, Kevin Egan, James Geibel, Bert Japikse, Delwen Jones,  Jason Lee, Dominic Nadarski, Josh Ormond, and Brian Welsh.

Summary:
Why GAO Did This Study

The Federal Aviation Administration (FAA), partnering with other federal agencies and the aviation industry, is implementing the Next Generation Air Transportation System (NextGen), a new satellite-based air traffic management system that will replace the current radar-based system and is expected to enhance the safety and capacity of the air transport system by 2025. Concurrently, FAA continues to maintain and upgrade existing air traffic control (ATC) systems that will also be needed for NextGen. This involves acquiring and implementing new software and hardware.
GAO was asked to determine (1) how, if at all, costs and schedules of FAA ATC acquisitions programs, including those related to NextGen, have changed since they were first submitted to Congress, (2) the reasons for any such changes, and (3) the extent that selected ATC programs adhere to cost and schedule best practices. To do its work, GAO reviewed 30 programs and conducted cost and schedule analysis on four programs that had an approved baseline and were NextGen related. GAO reviewed acquisition documents and interviewed FAA officials.

What GAO Found

In a review of 30 major ATC acquisition programs, all of which will contribute to the transition to NextGen, GAO found that costs for 11 of the 30 programs have increased from their initial estimates by a total of $4.2 billion and 15 programs experienced delays. The 11 acquisitions that experienced cost increases account for over 60 percent of FAAs total acquisition costs ($11 billion of $17.7 billion) for the 30 programs. The 15 acquisitions that experienced schedule delays, of which 10 also had cost increases, ranged from 2 months to more than 14 years and averaged 48 months.
Cost increases and schedule delays occurred due to several factors, many of which have been longstanding challenges for FAA. Specifically, these have involved (1) additional or unanticipated system requirements; (2) insufficient stakeholder involvement (such as controllers input) throughout system development; (3) underestimating the complexity of software development; and (4) unanticipated events including funding shortfalls or work toppages. These challenges, if they persist, will impede the implementation of NextGen, especially in light of the interdependencies among many acquisition programs, where cost increases or delays in one program can affect the costs and schedules of other programs.
For the four programs GAO selected to analyze in depth, FAA is not consistently following the characteristics of high-quality cost estimates and scheduling best practices that GAO previously identified. Regarding cost estimates, GAO found that although all four of the programs generally provided well documented and comprehensive estimates, which are two of the four characteristics, no program fully met the two other characteristics. Specifically, each program estimate was not credible because each lacked an independent cost estimate, which provides a check against FAAs estimate and three programs lacked risk or uncertainty analysis. The estimates also lacked accuracy because they were not updated regularly or based on comparable programs. Regarding scheduling practices, most programs did not substantially or fully meet the majority of the 9 best practices GAO previously identified including developing a fully integrated master schedule of all program activities and performing a schedule risk analysis. For example, without a schedule risk analysis, FAA is unable to predict, with any degree of confidence, if the estimated completion dates are realistic. FAA is implementing new processes and organizational changes to better manage acquisitions. However, by not consistently following the characteristics of highquality cost estimate and scheduling best practices, FAA cannot provide reasonable assurance to Congress and other stakeholders that NextGen and other ATC programs will avoid additional cost increases or schedule delays.

What GAO Recommends

To better estimate the cost and completion dates for major acquisitions, FAA should, among other things, require cost and schedule risk analysis, independent cost estimates and integrated master schedules. FAA did not comment on whether or not it agreed with the recommendations.