Air Traffic Control Modernization: Management Challenges Associated with Program Costs and Schedules Could Hinder NextGen Implementation
Released: 2012-02-16T00:00:00

Letter

Background

FAA catalogs its acquisition programs in its annually updated Capital Investment Plan (CIP). The CIP identifies planned capital investment in the National Airspace System (NAS) for the next 5 years consistent with the amount requested in the agency’s annual budget submission. Appendix C of the CIP, which identifies the anticipated budget line items, is divided into five activities: (1) Engineering, Development, Test, and Evaluation; (2) ATC Facilities and Equipment; (3) Non-ATC Facilities and Equipment; (4) Facilities and Equipment Mission Support; and (5) Personnel Compensation, Benefits and Travel. The CIP for fiscal years 2012 through 2016 contains 106 funded acquisition programs with estimated total budgets (through 2016) of more than $14 billion. Of these 83 acquisition programs FAA considers ATC related, 18 involve Engineering, Development, Test and Evaluation and 65 involve ATC Facilities and Equipment. The 83 programs include 30 that have had program baselines approved by FAA’s Joint Resources Council (JRC), which is responsible for approving major programs. These 30 baselined programs include communications, navigation, and surveillance systems that are key to ATC operations. FAA considers 5 of the programs to be foundational parts of NextGen, and all are key to modernizing the existing ATC system. Figure 1 illustrates the universe of FAA acquisitions for fiscal years 2012-2016.

FAA has developed and uses its Acquisition Management System (AMS) to provide policies and guidance for managing ATC system programs through all phases of the programs’ life cycles (see table 1). The Air Traffic Organization (ATO) within FAA is responsible for operating, maintaining, and modernizing the nation’s current ATC system.

The acquisition program baseline defines the cost, schedule and performance baselines for the investment program. The JRC which determines whether to approve a cost and schedule baseline also approves rebaselining, through which the agency documents and approves major changes to a program’s previously approved budget or schedule. Rebaselining resets the estimated costs and schedule used to determine how the program will be held accountable and can occur before the program is deployed. Once a program is rebaselined, FAA reports on the performance of the program based on the revised cost and schedule. Although the rationale for rebaselining can be reasonable, as, for example, when a program’s scope has been expanded, reporting a program’s performance based on a rebaselined cost or schedule can also skew or conceal from Congress and other stakeholders the program’s actual total costs or overall timeline. We previously reported that the absence of this information on rebaselining in ATO’s performance reporting could cause managers and other stakeholders, including Congress, to think that performance was better than it actually was. We recommended that FAA regularly report on the overall, long-term performance in acquiring ATC systems by providing in FAA’s annual Performance and Accountability Report the original budget and schedule baselines for each rebaselined program and the reasons for the rebaselining. In response to our recommendation, FAA currently provides this information in Appendix D of its CIP, where it details baseline cost and schedule information for major acquisition programs.

NextGen involves changes to every aspect of air transportation (see fig. 2). NextGen requires the acquisition of new integrated systems (both software and hardware), flight procedures, aircraft performance capabilities, and supporting infrastructure to transform the current air transportation system into one that uses satellite-based surveillance and navigation operations instead of ground-based radar. are intended to increase the efficiency and capacity of the air transportation system while maintaining safety and accommodating anticipated future growth. The planning for NextGen began in 2003 and is now focused on implementing improvements in the midterm (by 2018) and in the far term (by 2025).

GAO, Next Generation Air Transportation System: Challenges with Partner Agency and FAA Coordination Continue, and Efforts to Integrate Near, Mid-, and Long-term Activities Are Ongoing, GAO-10-649T (Washington, D.C.: Apr. 21, 2010).

Most Ongoing ATC Programs Remain within Budget and on Schedule, but a Few Have Significantly Exceeded Initial Estimates

Most Cost Estimates Remain on Target, but Cost Overruns for Three Key Programs in Total Exceeded $4 Billion

Of the 30 baselined FAA ATC programs we reviewed,19 have not increased in cost, but 11 have experienced cost increases ranging from $2 million to over $2 billion. And of the 19 programs whose costs have not increased, 7 experienced a cost decrease while the remainder have not changed significantly. However, the 11 programs that exceeded their initial estimated costs account for over 60 percent of total program costs for the 30 baselined programs—$11 billion of $17.7 billion. These 11 programs are among the most complex of FAA’s major acquisitions in that each involves a large amount of software engineering. (See table 3.)

The 3 programs with the largest cost increases—totaling more than $4 billion—are key to ATC modernization. Several factors contributed to cost overruns for the Standard Terminal Automation Replacement System (STARS), WAAS, and ERAM programs and required additional congressional appropriations or reductions in program scope.

Our previous work disclosed that the near tripling of the STARS’s budget resulted from insufficient involvement of stakeholders and requirements growth.

The WAAS program began in 1998 with an initial cost estimate of $1 billion and a current estimate of $3 billion. We reported previously that FAA’s lack of scientific and technical expertise resulted in unplanned work and contributed to cost increases as well as delays in the deployment schedule. Additionally, FAA changed how it accounted for certain costs in the capital budget in 1999, which further raised the cost estimate to $3.3 billion. FAA recently revised that estimate down to the current $3 billion during the 2009 rebaselining because, according to FAA officials, they had met certain program requirements in 2006.

As previously mentioned, ERAM is a key modernization system and will be the backbone of the NextGen system. FAA originally submitted to Congress an estimated cost of $2.1 billion in 2003, and the program is now expected to cost about $2.4 billion––an increase of about $330 million. According to FAA, various software issues (e.g., unsuccessful transmission messages and inaccurate data pairing of aircraft and traffic display), as well as problems interfacing with other facilities and systems, have contributed to the cost increases and delays. The extent to which unanticipated requirements, unplanned work, and underestimates of the complexity of software development, among other factors, have contributed to other FAA ATC acquisition program cost overruns and scheduling delays is discussed later in this report.

Schedules for Roughly Half of ATC Programs Are on Track, but Half of ATC Programs Are Delayed

Half of ATC Programs Are on or Ahead of Schedule and Half Are Delayed

We found that 15 of the 30 baselined programs either have experienced no change in schedule or were completed early or on time; however, the other 15 programs are projected to be completed later than originally estimated. These delayed programs range from the Integrated Display System, which will consolidate information from several weather subsystems into a single display, which FAA expects to be completed 2 months after its initial estimated completion date, to WAAS, which FAA estimates will be completed in 2013—more than 14 years after its initial estimated completion date (see table 4). Ten of the 15 programs with schedule delays also experienced cost increases. However, even if a schedule delay does not result in a direct cost increase to that program, the delay can lead to increased costs for FAA because FAA staff must continue to manage the acquisition over the longer term as it is being implemented, as well as maintain any legacy system that the program is replacing. Because of program interdependencies, a schedule delay can also affect how and when other programs will be implemented.

Several Factors Contributed to Cost Increases and Schedule Delays, and Some Could Hamper NextGen Implementation

A Number of Factors Have Contributed to Cost Increases and Delays

Cost increases and schedule delays occurred because of several factors, all of which have been long-standing challenges for FAA and some of which continue to affect programs despite FAA efforts to mitigate the factors. Specifically, these factors include (1) additional, unanticipated system requirements work; (2) insufficient stakeholder involvement throughout system development; (3) underestimates of the complexity of software development; and (4) unanticipated events, including funding decreases or work stoppages (see table 5). Of the 30 programs we reviewed, 15 experienced cost increases, schedule delays, or both, and we were able to determine that cost increases or schedule delays for 11 were attributable to one or more of these factors.

Following are some examples of how these contributing factors led to cost increases or schedule delays in some of FAA’s ATC baselined programs:

Unanticipated requirements or work: For nine of the programs in table 5, FAA has had to undertake substantially more development work than planned because FAA program officials originally misjudged the extent to which commercial off-the-shelf nondevelopmental solutions, such as those procured by another agency, would meet FAA’s needs. For example, although WAAS was being developed by an integrated product team that included representatives from several FAA offices, the team did not effectively resolve problems in meeting a required performance capability—that pilots be warned in a timely manner when a system may be providing them with potentially misleading and possibly hazardous information. These actions resulted in unanticipated work and contributed to the rise in WAAS’s cost from the original estimate of $509 million in 1994 to about $2 billion in 2005.

Insufficient stakeholder involvement: As we previously reported, ERAM was designed at a time when air traffic controllers did not participate in efforts to design and test new systems. Because active users of the system from different locations could not provide insight early on, issues that could have been addressed early in the design phase were not addressed. In response, FAA has taken steps to improve the testing of new systems in order to reduce the likelihood of larger-than-anticipated software issues arising during system implementation. For example, FAA and the controllers’ union recently entered into a memorandum of understanding to bring controllers into the testing and evaluation phase of ERAM. Under this agreement, the controllers’ union will have ERAM technical, evaluation, and training representatives, as well as a team of 16 controllers (including 12 from en route facilities and 4 from terminal facilities), who will be detailed to test and validate software fixes with contractor engineers at the FAA Technical Center (Tech Center). In addition, our previous work disclosed that the near tripling of the Standard Terminal Automation Replacement System’s budget resulted from insufficient involvement of stakeholders and requirements growth—two systemic factors that we found led to acquisitions missing their budget and schedule targets. This, in turn, contributed to cost growth, schedule delays, and eventually a reduction in the number of systems to be deployed.

Underestimates of the complexity of software development: This factor contributed to cost increases and schedule delays for ERAM, as well as issues with costs, scheduling, or both for two other programs. In 2010, FAA tested ERAM at two key sites (the Seattle and Salt Lake en route centers) on live air traffic, usually late at night when air traffic volume was low. During this testing, FAA encountered both anticipated and unanticipated software issues, which prompted the test sites, at times, to revert to using FAA’s legacy en route computer system. Specifically, software instructions to a controller in one sector to hand off control of an aircraft to a controller in an adjacent sector failed, and flight data were lost or reassigned to another flight. While some testing at FAA’s Tech Center preceded testing at the two key sites, the Tech Center could only test limited scenarios, and none of the scenario testing identified this software error. In addition, as discussed earlier, ERAM was designed during a time when air traffic controllers did not participate in efforts to design and initially test new systems. FAA anticipated the potential for software issues at the outset of the program and initially scheduled approximately 6 to 9 months of contingency time between the time it achieved initial operating capability and operational readiness demonstrations at these sites, leaving little buffer for any potential delays. FAA worked with its contractor to correct a number of software issues, but further testing on live air traffic at the two test sites continued to produce critical safety errors. As a result, in March 2010, FAA decided, with the support of the air traffic controllers’ union, to halt all ERAM testing on live traffic and to revise the deployment schedule. The program was rebaselined in June 2011, and the program’s completion date was extended from December 2010 to August 2014. As a result of the schedule delays, the rebaselined cost estimate increased from $ 2.1 billion to $2.4 billion.

Unanticipated events: Unanticipated events at implementation sites and unanticipated funding issues have delayed several programs’ schedules and increased costs. For example, Airport Surveillance Radar-11 was originally scheduled to be completed in June 2009 but was delayed to June 2010. FAA reports indicated that the delay was due to an unusually protracted real estate acquisition at one site and issues involving validating performance during seasonal radar operations from other another site. Similarly, FAA’s Runway Status Lights program—which involves installing airport lighting equipment that visually signals to pilots when it is unsafe to enter, cross, or begin takeoff on a runway—has experienced schedule delays because of construction issues at five sites (Charlotte, North Carolina; Fort Lauderdale, Florida; Las Vegas; Minneapolis; and Washington- Dulles). FAA officials attributed some of these delays to the furlough of some FAA employees in July 2011 and a freeze on contractor funding during the furlough, which resulted in work stoppage orders for several projects—including Runway Status Lights. FAA program managers will need to assess the impact of the furlough on other programs that had experienced work stoppage orders, including ADS- B, the Standard Terminal Automation Replacement System, SWIM, WAAS, and various weather programs.

Interdependencies among NextGen ATC Programs Could Slow NextGen Implementation

The interdependencies of ATC acquisition programs have become more prominent as the NextGen program shifts from planning to implementation, so that cost increases and schedule delays in one program could have a cascading effect on other programs. As discussed earlier, due to the integrated nature of NextGen, the development and delivery of many of its component programs are mutually dependent on the development and delivery of one or more other programs. For example, ERAM, FAA’s new en route computer system, is critical to the delivery of ADS-B capabilities such as broadcasting flight information. ERAM is also pivotal to the on-time implementation of two other key which is NextGen programs—Data Communications (DataComm),estimated to cost about $3 billion, and the NextGen information technology architecture, SWIM, which is estimated to cost over $550 million. Due in part to ERAM’s delay, FAA was forced to delay the Data Communications baseline date by approximately 6 months, rebaseline SWIM segment 1, and delay the SWIM segment 2 baseline date to 2012. The longer-term effects of these delays are unclear, but certain SWIM capabilities could be delayed for several years, and the progress of other programs that are dependent on SWIM’s system integration could be hindered, as well. Thus, looking more broadly, the implementation of NextGen—both midterm (now through 2018) and far-term (2019-2025) schedules—will be affected by how well FAA manages program interdependencies.

GAO-10-629. developed a full listing of how ERAM schedule slippages, or slippages in other programs that are critical to NextGen, could either affect other programs’ implementation schedules or delay the implementation of capabilities and improvements. In 2008, we recommended that FAA improve the usefulness of ATO’s acquisition performance reporting by including information in the agency’s Performance and Accountability report or elsewhere on the potential effect of any budget or schedule slippages on the overall transition to NextGen. This recommendation remains open, as FAA has not definitively indicated how it will track slippages that will affect other dependent NextGen programs. FAA’s acquisition management system was not designed for managing NextGen programs in an integrated way. To assist in managing NextGen portfolios, FAA is starting to monitor all the activities of a particular operational improvement to ensure integration is on track. As we noted in the 2010 report, as this approach is more fully implemented, it will likely clarify the impact of slippages in one program’s schedule on the implementation status of other NextGen programs and operational capabilities. In addition, as we will discuss in the next section, FAA is developing an Integrated Master Schedule for the entire NextGen initiative that is, in part, intended to show how changes in program schedules affect other programs and the timelines for the NextGen initiative as a whole. However, as we discuss later, the schedules for the four programs we reviewed in detail are not reliable. Reliable schedules at the program level will be needed to develop a reliable Integrated Master Schedule for NextGen.

Program Management Changes Are Aimed at Addressing Past Challenges

According to FAA, it is taking actions to address the factors that have contributed to cost increases and schedule delays. In 2011, FAA assessed the NextGen effort as part of its Foundation for Success initiative and has implemented the “Idea to In-Service Management” (I2I) process, which it believes will result in improvements in the way the FAA develops, acquires, and implements new NextGen capabilities from conception through implementation. The I2I concept is intended to improve collaboration early in the acquisition process, resulting in better defined capabilities and an early indication of cost and benefits. These enhancements are intended to resolve many of the challenges associated with overall program management and enable FAA to focus on program management best practices. FAA believes that I2I will also result in improvements in specific areas that have presented challenges in the past, such as cost estimating, anticipating requirements and work, stakeholder collaboration, software development, and systems integration.

Also in 2011, FAA implemented a reorganization of the NextGen Operations and Planning Office and ATO which FAA believes will support the I2I process and improve acquisitions of NextGen programs. Specifically, FAA created a NAS Lifecycle Planning Division within the NextGen Operations and Planning Office to focus the integration of NextGen programs from a cost, schedule, and systems capability perspective. Within ATO, FAA established a new Program Management Office, which puts the responsibility for the program management of all NextGen and other major ATC acquisitions within a single organization. By combining program managers into one organization, FAA hopes to create a stronger acquisition program and improve the consistency and implementation of best practices. According to FAA, these organizational changes allow responsibilities for acquisitions to be better defined to more efficiently set strategic direction, define operational requirements, ensure system integration, oversee implementation processes, and ensure accountability throughout the acquisition life cycle.

To improve the acquisitions management process, FAA has also divided large acquisition programs into segments. A segmented or phased approach is being taken with programs like SWIM and CATMT. This approach breaks a larger program into smaller and more manageable pieces to lower the risk. In the past, we noted that this approach can improve management by providing for midcourse corrections and, thus, help FAA avoid costly late-stage changes. However, this approach can also increase the duration and possibly the cost of the program.

According to FAA officials, a segmented approach allows the agency to more effectively manage acquisitions at both the program and enterprise architecture level. An enterprise architecture approach provides the structure to relate organizational mission, vision, and goals to business processes and the technical or information technology infrastructure required to execute them. FAA officials stated that many of the factors we identified that contributed to cost increases and schedule delays highlight the need for an enterprise-level perspective throughout the acquisition process. The I2I process is intended to provide an enterprise-level focus and improve collaboration across related programs.

For Programs We Reviewed, FAA Did Not Consistently Meet All Characteristics of Quality Cost Estimates or Schedule Best Practices

Selected Cost Estimates Were Generally Comprehensive and Well- Documented, but Accuracy and Credibility Need to Be Improved

Our review of the ADS-B, CATMT, SWIM, and WAAS cost estimates showed that while each program followed at least some of the four characteristics of high-quality and reliable cost estimates—well- documented, comprehensive, accurate, and credible—none of the programs adhered closely enough to those characteristics to create a reliable cost estimate. As previously noted, these characteristics incorporate the 12 steps consistently applied by cost-estimating organizations throughout the federal government and industry and considered best practices for developing cost estimates. The results of our review of the ADS-B, CATMT, SWIM, and WAAS cost estimates, which are summarized in table 6, show that they were most aligned with the characteristic of comprehensive cost estimates but need improvement in the other three areas, particularly with the characteristics of accurate and credible estimates. Imprecise estimates can result in Congress unnecessarily authorizing and appropriating millions of dollars for programs. As noted, in some cases, FAA, in order to stay within the original cost estimate, modified a program’s requirements and Congress had to appropriate more funds or reduce the scope to allow FAA to finish the program.

FAA could better ensure that the cost estimates for these four programs, as well as its other major acquisition programs, are reliable. Our work shows that an assessment of these cost estimates for these programs, as well as FAA’s other major acquisition programs, would allow FAA to better understand if its cost estimation guidelines and our characteristics of high-quality cost estimates are in fact being followed (See table 6).

Because the four programs were generally similar in the extent to which they met each of the four characteristics, the following discussion summarizes the strengths and weaknesses we found for each characteristic across the four programs. A more detailed discussion of our findings is contained in appendix IV.

Well-documented. Two of the four cost estimates we analyzed substantially met the characteristic of being well-documented; the other two partially met this characteristic. A well-documented cost estimate is thoroughly documented, including identifying specific source data and their significance, detailing calculations and results, and explaining why particular cost estimating methods were chosen. In other words, sufficient documentation exists such that an unfamiliar analyst could recreate the cost estimate and arrive at the same results. For example, the SWIM estimate provided detailed documentation describing the program, in addition to the methodology, calculations, and quantities used to develop the estimate. However, none of the four estimates sufficiently captured the entire source data used, addressed its reliability, or described how various forms of data from disparate sources were normalized (i.e., the data were described in like terms). For example, the WAAS estimate was based, in part, on actual labor costs from a previous contract, but the program office provided no evidence that these data were evaluated for reliability or accuracy. Similarly, the CATMT estimate routinely relied on subject matter expertise as a source for assumptions, such as the cost of labor, but the estimate did not document the experts’ qualifications, background, underlying assumptions, or data sources. Moreover, we noted that three of the four estimates often substantially relied on expert opinion rather than on data. While expert opinion can be useful in the absence of data, it is subjective and generally should be used sparingly for cost estimates. Since data are the foundation of every cost estimate, data quality affects the overall quality of the estimate. In addition, because data are gathered from a variety of sources and take many different forms, normalization helps to improve consistency with other cost information and enable valid comparisons and projections.

Comprehensive. All four cost estimates we analyzed substantially met the characteristic of being comprehensive. For an estimate to be comprehensive, it should include full life-cycle costs, completely define the program with sufficient detail, include cost elements that are traceable to the statement of work or objective to ensure they are neither omitted nor double counted, and document all cost-influencing ground rules and assumptions. We found that the ADS-B, CATMT, and SWIM cost estimates included all life-cycle costs, regardless of program phase or funding source, and the ADS-B and SWIM cost estimates completely defined the program with an appropriate level of detail. In particular, the ADS-B cost estimate included cost estimates for both government and contractor costs, and the WAAS cost estimate thoroughly defined the program and reflected the current schedule. The four estimates did not fully meet the comprehensive characteristic because they lacked evidence that all cost influencing ground rules and assumptions were considered.

Accurate. None of the four cost estimates met or substantially met the characteristic of being accurate. The estimates generally adjusted costs for inflation and contained few computation or mathematical mistakes, but they were not regularly updated to reflect schedule and requirement changes, did not provide evidence of documenting or reviewing differences between planned and actual costs, and were not based on historical cost data from comparable programs. For example, the ADS-B, CATMT, and SWIM cost estimates provided no evidence that they were updated to reflect program changes, such as schedule slippages or varying assumptions, and did not include the current actual costs of the program. Although the WAAS estimate included evidence that it was updated to reflect major changes in technical and program requirements, such as the four rebaselinings the program has undergone since its 1998 inception, it did not include evidence that estimated costs were replaced with actual costs as the program advanced. Cost estimates that are not regularly updated with current information cannot provide decision makers with accurate information that is necessary, for example, when new system requirements are called for under tight budget conditions. In addition, comparing planned and actual costs enables cost estimators to measure the accuracy of their estimates and refine their processes. In addition, none of the four programs more than minimally used historical data to develop their cost estimates. Had historical data been used, the estimators would have had additional insight into actual costs on programs that used similar technologies, which could be used, for example, to challenge overly optimistic assumptions and bring more realism to the cost estimate.

Credible. None of the four cost estimates met or substantially met the characteristic of being credible, which includes obtaining an independent cost estimate from a group outside the acquiring organization, and cross-checking the major cost elements in that estimate against cost drivers identified through sensitivity and risk analyses. The ADS-B, CATMT, SWIM, and WAAS estimates lacked credibility largely because FAA did not obtain an independent cost estimate for any of the programs. In addition the CATMT, SWIM, and WAAS estimates provided little evidence that it conducted sensitivity or risk analyses. Instead, each program received independent cost reviews as part of the investment decision process—even though such reviews are not required by FAA policy. FAA stated that the Investment, Planning and Analysis (IP&A) Office in the FAA Finance Organization does not prepare independent estimates, but it is organizationally independent of the acquisition programs and conducts independent reviews of all cost estimates. However, an independent cost review is less rigorous than an independent cost estimate. According to our cost guide, an independent cost estimate is often more accurate because the estimating team is further removed from the program office and less prone to accept overly optimistic assumptions or be burdened by organizational bias. Other federal agencies, including the Department of Defense, require independent cost estimates. Had an independent cost estimate been completed, the estimating team and program team could have identified the major differences between their estimates, reconciled those differences where possible, and provided a synopsis of the two estimates and their differences to acquisition program management. In addition, without sensitivity and risk analyses, cost estimators cannot measure the effects of varying assumptions, and managers cannot determine, for example, the rational level of contingency reserves necessary to cover increased costs that may result from uncertainties such as unexpected design complexity, changes in requirements, or budget shortfalls––all of which FAA ATC programs, and in particular NextGen programs, have experienced in recent years. We found evidence that some level of risk analysis was conducted for ADS-B, CATMT, and SWIM, although the analysis was not sufficiently robust. For example, key cost drivers were not identified, and additional context about how the estimate could be affected by software design and development issues was not included.

Selected Schedules Did Not Substantially Meet Most Best Practices and Are, Therefore, Unreliable

We determined that the schedules for the four programs we reviewed are unreliable because none met or substantially met all nine of the best practices for developing a reliable schedule. (see table 7). For example, none of the schedules fully met best practices for capturing all activities in an integrated master schedule, identifying critical paths and reasonable float for all activities, or assigning resources to those activities. Moreover, none of the schedules had documentation that provided more than minimal evidence that they conducted a schedule risk analysis. As was the case with our review of cost estimates for the four programs, our work regarding the schedules for these programs shows that an assessment of the schedules, as well as schedules for FAA’s other major acquisition programs, would allow FAA to understand if the nine best practices for reliable schedules are being followed.

Because the scheduling best practices are interrelated in such a way that deficiencies in one best practice will cause deficiencies in the others, a schedule must meet or substantially meet all nine practices to be reliable. For example, preparing a schedule that is program-wide––including an integrated breakdown of the work to be performed by both the government and its contractors over the expected life of the program––is a best practice. If the schedule does not capture all activities, then there will be uncertainty about whether activities are sequenced correctly or if the schedule properly reflects the resources needed to accomplish the work, which is also a best practice. Logic and durations (that is, the time it takes to complete a specific activity) should be used and maintained to ensure realistic start and completion dates and to reflect the true status of the project––a necessary condition for conducting follow-on schedule risk analyses. Moreover, if activities are not properly sequenced with logical links, it will not be certain if the critical path—which represents the chain of dependent activities with the longest total duration—is valid. Collectively, the weaknesses in not fully meeting or substantially meeting all nine key practices increases the risk of schedule slippages and cost overruns since a well-defined schedule helps to identify the amount of human capital and fiscal resources that are needed to execute the program. Therefore, by not having reliable schedules, FAA cannot conduct meaningful oversight of an acquisition program’s progress or determine whether the program is achieving the desired results.

The following discussion summarizes the extent to which the schedules for the four programs we examined met best practices. More detailed information for each program regarding scheduling best practices is presented in appendix V.

We reviewed the schedule prepared by FAA and found it did not fully meet any of GAO’s nine scheduling best practices, resulting in an unreliable schedule. Evidence provided in the ADS-B schedule indicates that it substantially met three of the nine best practices and partially, minimally, or did not meet the other six. For example, although the ADS-B schedule provided evidence of periodic updating, it did not capture all of the effort currently called for in the approved baseline for the entire ADS- B program and, therefore, was not a fully integrated schedule. Without fully integrating government activities with contractor activities, and thereby capturing all key activities, the schedule will not reliably estimate the program’s completion. In addition, the ADS-B schedule we reviewed did not identify critical paths or include a schedule risk analysis, which uses statistical techniques to predict a level of confidence in meeting a program’s completion date; did not logically sequence all activities and establish their durations; and had excessive float on a majority of current and planned activities. According to program officials, a number of the issues our analysis identified were, in part, the result of the schedule’s limited time frame, which covered only a defined transitional period (October 2010 through April 2011) during which responsibility for about a third of the effort passed from the FAA to its prime contractor. Officials also stated that although their schedule contains critical activities, it has not had a traditional critical path since its contractor began managing the deployment of deliverables. The FAA uses contract options to order the scope, sequence, and requirements for key milestones. Within those options, the contractor has the authority to implement the sequence of more discrete activities in the order they deem most appropriate. FAA program officials plan to rectify this problem, noting that with negotiations now completed, they will in the near future identify a critical path to span all program milestones.

Because the CATMT program did not prepare an FAA schedule and instead relied on its contractor schedule, we reviewed the contractor schedule, which we found to be unreliable. Our analysis found that the contractor’s CATMT schedule substantially or fully met four of the nine best practices: capturing all activities, assigning resources, establishing the durations, and updating the schedule. For example, the CATMT contactor schedule pertains to the current phase of the program that is being implemented in software releases, or phases. However, there was no overarching FAA government owned schedule that accounts for all software releases for the entire program and would thus delineate the relation of current software release tasks to the upper-level milestones for the overall CATMT program. The CATMT schedule included detailed resource information, and the program office provided evidence that resources are tracked in detailed labor-hour spreadsheets. We also found that 90 percent of the activities were of short duration and that the program office regularly reviews the schedule, which is in line with best practices. On the other hand, five of the nine best practices were either partially, minimally, or not met. Specifically, the CATMT schedule lacked evidence indicating that it established a critical path, accurately identified float between activities, integrated the schedule vertically and horizontally, sequenced all activities, or performed a schedule risk analysis. Regarding the critical path, our analysis determined that the CATMT schedule does not identify a critical path for the entire program.

Instead, the program is being accomplished multiple 6-month spirals; thus, there is only a critical path for each software release, not for the program as a whole. Without a valid program-wide critical path FAA management cannot determine which tasks, if they slip, will have the most detrimental effects on the project finish date. We also found that 68 percent of the remaining activities to be completed had unreasonably high float exceeding 1,000 days, meaning that those activities could slip about 5 work years without affecting the overall project finish date, a highly unlikely scenario. The accurate identification of critical paths and float are inextricably linked. For example, if the schedule is missing activities or they are not correctly linked, float estimates will be miscalculated, resulting in an invalid critical path. Without a schedule that can produce a true critical path, the program office will neither be positioned to provide reliable timeline estimates nor be able to identify when problems or changes may occur and determine the impact they may have on subsequent work. CATMT program officials acknowledged that the schedule did not include program-wide critical paths but noted that a critical path exist for individual segments of the program. They also noted that a schedule risk analysis was not performed because it was not a contractual deliverable.

Because the SWIM schedule did not fully or substantially meet any of GAO’s nine scheduling best practices, we found it to be unreliable. The SWIM program differs from the others in that it is an aggregation of NextGen acquisition programs, each developing an aspect of the SWIM information sharing capability. Because SWIM program managers are reliant on schedule information from a number of other programs, SWIM schedule integration is particularly important. However, our analysis found that the SWIM schedule was not, by any measure, fully integrated because it provided only a synopsis of the individual system implementing program schedules and, thus, did not fully represent the work required to complete the overall SWIM program. This resulted in float calculations that were unrealistic, and the resulting critical path calculations were invalid. In addition, while the many missing activities negatively impacted the schedule logic and the accuracy of durations, it also made the accurate allocation of resources and comprehensive integration of schedule activities, both horizontally and vertically, impossible. We also noted that FAA made no effort to identify a program-wide critical path. Program officials said that because each of the system implementing program schedules has its own critical path, involve disparate capabilities, and are independent of one another their individual critical paths are not accessible through the SWIM schedule software. They therefore are not used for overall SWIM program management. We believe that the SWIM program itself should have its own critical path that includes, at a minimum, acceptance of major deliverables from the system implementing program schedules. Without a program-wide critical path, management does not have a clear picture of the underlying project tasks that must be performed to achieve the overall program target completion date. Finally, although there was no risk analysis conducted on this schedule, our analysis found that this best practice was minimally met because a risk analysis was conducted on a separate but related schedule, and the SWIM program office considered risk to some extent.

Like the other three programs, we found the WAAS program schedule prepared by FAA unreliable because it did not fully or substantially meet any of GAO’s nine scheduling best practices; however, we also reviewed the contractor’s schedule for the same segment and found it fully or substantially met six best practices. For example, FAA’s WAAS program schedule did not fully sequence activities in the order in which they are to be carried out. More specifically, the WAAS program schedule showed nearly half of the remaining activities were missing sequenced logic, causing us to question the calculated dates of activities. Logic is necessary for a schedule to show program managers when activities are expected to start and finish; when logic is missing, activity dates cannot adjust correctly to changes in activities. To test the ability of the schedule to dynamically update its dates due to changes, we artificially extended the duration of an activity to 1,500 days, which changed the activity’s finish date. However, the duration extension had no effect on successor activities because this activity is not tied to any successor activities. Extending the duration to 1,500 days also pushed the project planned finish date from September 22, 2016, to June 29, 2017; however, because the logic links are not in place, we questioned whether the projected finish date under this scenario is reliable.

Moreover, the WAAS program schedule had too many artificial constraints that were driving the start and finish dates for more than 70 Constraints are usually substitutes percent of the remaining activities.for logic and can mean that the schedule is not well planned or feasible. Constraints also greatly reduce the ability of the program to take advantage of possible time savings. Further, our analysis found that the schedule did not fully capture or assign resources to all government and contractor activities; it also did not accurately allocate resources or consistently establish the duration of activities. In addition, while WAAS program officials told us that the schedule was integrated vertically and horizontally, we did not find evidence of such integration. Furthermore, we found the WAAS program office’s schedule did not identify a critical path for the entire program. As noted earlier, critical path and float determinations are closely related. Our analysis of the WAAS program office schedule found that more than half of the remaining activities had float of more than 1,000 working days, which we believe to be unreasonably high. Without proper determination of float, management cannot properly reallocate resources from tasks to other tasks without adversely affecting the overall completion date. Although program officials said that they maintained a risk register listing the potential risks that could impact the schedule and adjusted the schedule for these risks, we did not find evidence that the program office had conducted a risk analysis of its schedule.

While the schedule prepared by the contractor did not fully or substantially meet three of the scheduling best practices, it fully or substantially met six: capturing, sequencing, assigning resources to, and establishing the duration of all activities; establishing the critical path; and identifying reasonable float between activities. For example, our analysis found that all activity durations are consistently estimated in days and adhere to a standard 5-day workweek that accounts for holidays, and no activities were scheduled to begin on a weekend. Officials from the contractor said duration estimates for the schedule are based on historical information from past performance, comparable releases, lessons learned, similar work, and other data requirements. In addition, our analysis traced several critical paths in the schedule. Though we found minor interruptions in the various critical paths, the schedule’s logic, reasonable durations, and low total float estimates allow the calculation a valid critical path.

WAAS Schedule Risk Analysis Indicates Risks That Managers Could Mitigate to Avoid Delays

As noted, FAA did not perform a complete schedule risk analysis for any of the four programs we reviewed and, thus, cannot accurately estimate these programs’ completion dates with confidence. A schedule risk analysis, which is one of our best practices for program scheduling, uses statistical techniques to predict a level of confidence in meeting a program’s completion date. The objective of the analysis is to develop a probability distribution of possible completion dates that reflect the project and its identified risks. This analysis can help program managers both understand the most important risks to the program and focus on mitigating those risks. Other federal agencies, including the Department of Defense and the National Aeronautics and Space Administration, require schedule risk analysis for major acquisitions; the Department of Veterans Affairs, in response to a GAO recommendation,require schedule risk analysis for major construction projects.

The other three schedules did not have the required information to conduct a schedule risk analysis. showed four potential risks to the project. We then conducted interviews with FAA program and contractor staff and asked them to discuss other potential risks to the project, including how the risk would affect the project’s timeline and the likelihood of the risk occurring. Using this information we identified an additional 16 risks for a total of 20 risks. The fact that our interviews identified a relatively large number of new risks could be an indication that the contractor did not systematically analyze the full range of risks when developing the program’s risk register. We then consolidated the 20 risks into 14 broader risks and tested how each would impact the duration of specific activities in the schedule. We then ran a Monte Carlo simulation, which consisted of the computer- generated results of 3,000 estimates of the future schedule based on the activities in the schedule, the chance that some of the activities would be affected by some risks and the predicted effect of those risks on the duration of each activity.

We then analyzed the potential impact of risks on the program schedule. Since risks can effect the schedule in various ways––for example, risks can have a large impact on the durations of activities they affect, or they can introduce critical paths that are different from the baseline critical path–– we analyzed the marginal impact of each of the risks we identified to determine which would have the greatest effect on the overall schedule. We found the following three key risks to the program, only the first of which (limited WAAS program office resources) was originally identified by the contractor. The three risks are limited WAAS program office resources such as staffing; delays in software yet to be released and additional changes to software already released and in use; and a potentially optimistic schedule completion date.

Our schedule risk analysis showed the completion of the segment of the WAAS program covered by the schedule could slip as much as 2 months. Specifically, the analysis showed that there is less than a 5 percent probability that the program segment would be completed by September 6, 2012, the current baselined date for completion. However, it appears that the segment will be completed close to the deadline since we found a 50 percent probability that the program segment will be completed by October 23, 2012 (about 1.5 months after the current estimated date for completion); and an 80 percent probability that the program will be completed by November 13, 2012 (about 2.25 months after the current estimated date for completion).

Although we did not conduct a schedule risk analysis for other FAA programs, the result of our analysis provides examples of the types of risks that major acquisition programs face and the impact those risks can have on meeting acquisition program milestones, especially given the interrelation and interdependencies among NextGen acquisitions discussed earlier. More information on our schedule risk analysis can be found in appendix V.

Lack of Reliable Program Schedules Will Hinder Development of an Overall NextGen Integrated Master Schedule

FAA has begun developing an integrated master schedule for the entire NextGen initiative that would, in part, capture related NextGen program schedules, governance activities, and other performance and financial data to provide real-time monitoring of the overall NextGen effort. However, the unreliability of the four program schedules for programs that are integral to the NextGen initiative puts this high-level master schedule at risk. Having a reliable integrated master schedule would enable FAA to determine how delays in one program impact other programs and the overall NextGen implementation timeline. While it is encouraging that FAA is beginning to develop an integrated NextGen master schedule, the effort could be hampered by the lack of schedule integration at the program level, as well as the failure of individual program schedules to meet best practices. For example, since FAA does not perform schedule risk analysis on individual programs, it cannot predict with certainty if any of the programs will be completed on time. Therefore, the integrated master schedule for NextGen would be built on schedules that may not reflect accurate program completion dates. Similarly, none of the four schedules we reviewed, which were for segments of the entire program, had reflected how tasks for the segment affected milestones for the entire program. Without integrated schedules at the program level, an integrated master schedule at the NextGen initiative level would be problematic.

FAA has Taken Steps to Improve Acquisition Program Cost Estimates and Schedules

In response to our review of the extent that the four selected acquisition programs met best practices for cost estimates and schedules, FAA provided information on steps it is taking to improve its processes for both cost estimates and schedules and noted that some of the cost estimates and schedules we reviewed were developed before the improvements were in place. FAA stated that strengthening its cost estimation process is part of the seven key acquisition processes it has developed, including program management, contractor management, requirements, risk management, measurement and analysis, verification and validation, and quality assurance. FAA stated that it has updated its Guidelines for FAA Cost Estimating to be consistent with the GAO Cost Guide, filling in gaps that it had identified during a comparison of its practices to those contained in the Cost Guide. As of November 2011, 11 of the 12 best practices are addressed in the guidelines. According to FAA officials, the remaining best practice—involving the creation of independent cost estimates—is unlikely to be implemented at FAA in the foreseeable future because FAA believes the resources required to create independent estimates are prohibitive in current budget environments. FAA has more than tripled the number of cost estimators in the Investment Planning and Analysis organization, many of which work with the acquisition program offices to provide guidance on preparing estimates. Additionally, as part of FAA’s effort to improve acquisition certification and training, the agency is preparing to launch a cost estimating certification program. Coupled with a competency-based training program, FAA believes the certification program will enhance and improve consistency of the skills of FAA cost estimators.

In describing its efforts to improve schedules, FAA stated that it views the development and maintenance of integrated schedules as an inherent and critical part of its seven key acquisition functions. FAA noted that included in its standard process for acquisition schedules are toolkits that require programs to develop integrated program schedules that address all nine of GAO’s best practices. FAA stated that that the current procedures for developing best practices were not fully in place when the four programs we reviewed began the implementation phase.

Conclusions

FAA has made improvements in its management of air traffic control modernization acquisitions, and most of the 30 we reviewed are currently within the original cost estimate and half are on schedule. FAA is also taking steps to address past issues to ensure cost estimates and schedules are more accurate in the future, including incorporating best practices in its acquisitions guidance and policies. Nevertheless, our review of the FAA acquisitions found that it has yet to fully implement several GAO-identified best practices or follow others. Following best practices is particularly important for FAA, which must manage large, complex, and interdependent acquisitions associated with NextGen. Cost estimates that are imprecise can result in Congress appropriating millions of dollars for projects based on estimates that prove to be inaccurate, and program schedule delays can increase costs and affect the implementation of interdependent programs. In such cases, FAA will be forced to reduce the scope of the programs to stay within the original estimates or Congress will need to appropriate unanticipated funds to complete the programs. Delays and cost increases in individual programs could have a cascading effect on other programs and ultimately affect FAA’s timelines and goals for NextGen implementation.

Our analysis of the cost estimates and schedules for the four programs we reviewed indicates that FAA needs to further develop requirements for critical cost estimation and schedule procedures. Independent cost estimates can improve the accuracy and credibility of cost estimates and better ensure that programs will be completed within budget. A schedule risk analysis can help FAA determine the likelihood that a program will be completed on time. FAA stated that it has no immediate plans to conduct independent cost estimates due to current budgetary constraints. We recognize that conducting independent cost estimates and schedule risk analysis takes both financial resources and some time and that it may be appropriate to limit one or both of these analyses to instances where a program is particularly costly, complex, or on a compressed schedule. However, conducting independent cost estimates, schedule risk analyses, and other analyses called for in our best practices can not only help minimize the risk of cost overruns and schedule delays, but also provide FAA, congressional decision makers, and other stakeholders with important information about these critical acquisitions.

It is also important that FAA develop master schedules at the individual acquisition program level. FAA’s lack of a fully integrated master schedule for the programs we reviewed hampers its ability to provide accurate information on the schedule for these programs. This information will be needed as FAA simultaneously works to develop an integrated master schedule for the overall NextGen initiative. The use of an integrated master schedule can assist FAA in monitoring a program, identifying problems that could affect later stages of the program’s implementation, improving the accuracy of cost estimates and schedules for individual programs, and improving the accuracy of information FAA is compiling to monitor the costs and schedules for the NextGen initiative.

FAA has incorporated 11 of our 12 steps that are associated with the characteristics of a high-quality and reliable cost estimate into their acquisition guidelines. However, our analysis of the four major programs indicates that FAA has not adequately integrated all of the steps for these programs into its cost estimation processes, and thus the estimates are not reliable. Similarly, although FAA addresses our nine scheduling best practices in its acquisition guidelines, our analysis of the schedules for the four programs indicates that the schedules are not adequately following these best practices and are not reliable. Although the cost estimates and schedules for some of the four programs were developed prior to FAA’s revision of acquisition guidelines, our work shows that FAA needs to assess its major acquisition programs to understand if its guidelines and other best practices are, in fact, being followed. Such an assessment would then allow FAA to better ensure that best practices for cost estimates and schedules are being applied.

Recommendations for Executive Action

To improve cost estimates and schedules for NextGen and other major air traffic control acquisition programs, GAO recommends that the Secretary of Transportation direct FAA to take the following three actions when appropriate for major acquisition programs based on a program’s cost, schedule, complexity, and risk:

Conduct independent cost estimates and schedule risk analysis for major acquisition programs.

Require a fully integrated master schedule for each major acquisition program, including those that are components of NextGen. An integrated master schedule should horizontally and vertically link all program activities and milestones, including government and contractor schedules and program segments.

Conduct an assessment of major acquisition programs to ensure they meet all of the established best practices for cost estimates and schedules contained in GAO guidance.

Given constrained budgets, FAA should determine which programs should be subject to these recommendations, such as those that are particularly costly, complex, or on a compressed schedule.

Agency Comments

We provided a draft of this report to the Department of Transportation for review and comment. DOT and FAA responded by email and did not comment whether or not they agreed or disagreed with our recommendations. DOT provided comments on the results of our analysis of the cost estimates and schedules for the four programs we reviewed in depth. In response to our finding that the ADS-B, CATMT, SWIM, and WAAS estimates lacked credibility largely because FAA did not obtain an independent cost estimate for any of the cost estimates, and provided little evidence that they conducted sensitivity or risk analyses, FAA stated it is not convinced that an independent organization will reduce the uncertainty of cost estimates. FAA noted that it does not have an independent organization such as the Department of the Navy’s Center for Cost Analysis. However, FAA stated that the Finance Organization within ATO assessed the ADS-B program office’s Basis of Estimate as part of the JRC Decision and that this level of independence, combined with specific entry and exit criteria, allowed the program offices to manage these acquisitions so that costs were controlled, risks mitigated, and technical parameters achieved, while adhering to the planned milestone schedule. We agree that the Finance Organization assessment of the two cost estimates provided some degree of independence and may have improved the accuracy of the ADS-B estimates, but it is not clear that such an independent review will guarantee similar results for other programs. As we stated in the report, such an independent cost review is less rigorous than an independent cost estimate. According to our cost guide, an independent cost estimate is often more accurate because the estimating team is further removed from the program office and less prone to accept overly optimistic assumptions or be burdened by organizational bias. DOT also provided technical clarifications, which we incorporated into the report as appropriate.

We are sending copies of this report to interested congressional committees, the Secretary of Transportation, and the Acting Administrator of FAA. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-2834 or dillinghamg@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V.

Appendix I: Objectives, Scope and Methodology

In response to a congressional request, we examined the Federal Aviation Administration’s (FAA) ability to modernize, upgrade, and replace the National Airspace System’s (NAS) facilities and equipment to meet projected increases in traffic volumes, enhance the system’s safety, and increase the efficiency of the air traffic control (ATC) system—a principal component of the NAS. FAA’s ATC acquisitions are critical to maintaining the NAS and transitioning to the Next Generation Air Transportation System (NextGen) over the next 10 years. Given that some key legacy and NextGen acquisitions have experienced schedule delays and cost overruns, which may risk the timely implementation of NextGen, we (1) determined whether the planned costs and schedules of current FAA ATC acquisition programs have changed since they were first submitted to Congress; (2) examined the reasons for any changes in planned costs and schedules; and (3) assessed the extent to which select ATC programs adhered to best practices for determining acquisition costs and schedules.

To describe any changes in costs and schedules of the current 30 FAA capital ATC acquisitions, we gathered and analyzed agency data on the estimated cost and schedules of these ATC acquisitions. We drew upon past work in which we undertook detailed reviews of the status of ATC and other acquisition programs and obtained updated documentation as necessary from FAA. We interviewed FAA officials to obtain information on FAA’s acquisition process and summarized the status of all acquisitions, including FAA’s original and current cost estimates and completion dates. For baselined acquisitions, we compared estimated costs when they were submitted to Congress for approval against their current estimates, and we analyzed planned and actual schedules.

To determine the reasons for changes in cost estimates and schedules, we interviewed FAA officials and FAA contractors and reviewed acquisition documentation. We analyzed information on cost increases and delays to determine if systematic issues exist that have effects on other FAA acquisitions.

To determine the extent to which select ATC programs adhered to best practices for determining acquisition costs and schedules, we conducted an in-depth review of 4 of the 30 acquisitions programs: The Automatic Dependent Surveillance-Broadcast (ADS-B) system, the Collaborative Air Traffic Management Technologies (CATMT) system, the System Wide Information Management (SWIM) system, and the Wide Area Augmentation System (WAAS). We selected these four acquisitions based on the following criteria: (1) existence of baselining, (2) the acquisition is at a point in the acquisition process where risks can be identified, and (3) the acquisition is key to NextGen and legacy systems. In addition to interviews, we collected documentation, and we analyzed and summarized the views and information collected. We also identified best practices that FAA could adopt or strengthen to improve its acquisitions cost estimation and scheduling and ensured that acquisitions follow cost and schedule best practices outlined in our Cost Estimating and Assessment Guide. We also performed a schedule risk analysis of the WAAS program to determine the likelihood of the project finishing on schedule.

GAO-09-3SP. a conclusion about whether the cost estimate is reasonable. Therefore, a good cost estimate—while taking the form of a single number—is supported by detailed documentation that describes how it was derived and how the expected funding will be spent in order to achieve a given objective. For example, the documentation should capture in writing such things as the source data used and their significance, the calculations performed and their results, and the rationale for choosing a particular estimating method or reference. Moreover, this information should be captured in such a way that the data used to derive the estimate can be traced back to and verified against their sources. Finally, the cost estimate should be reviewed and accepted by management to ensure there is a high level of confidence in the estimate and the estimating process.

Comprehensive: The cost estimates should include both government and contractor costs of the project over its full life cycle, from inception through design, development, deployment, operation, and maintenance to retirement of the project. The cost estimate should be structured in sufficient detail to ensure that cost elements are neither omitted nor double counted, and they should document all cost-influencing ground rules and assumptions.

Accurate: The cost estimates should provide for results that are unbiased, and they should not be overly conservative or optimistic. Estimates are accurate when they are based on an assessment of most likely costs, adjusted properly for inflation, and contain few, if any, minor mistakes. In addition, the estimates should be updated regularly to reflect material changes in the project, such as when schedules or other assumptions change so that the estimate is always reflecting the project’s current status. Among other things, the estimate should be grounded in documented assumptions and a historical record of cost estimating and actual experiences on other comparable projects.

Credible: The cost estimates should discuss any limitations of the analysis because of uncertainty or biases surrounding data or assumptions. Major assumptions should be varied, and other outcomes recomputed to determine how sensitive they are to changes in the assumptions. Risk and uncertainty analysis should be performed to determine the level of risk associated with the estimate. Furthermore, the estimate’s results should be cross-checked, and an independent cost estimate conducted by a group outside the acquiring organization should be developed to determine whether other estimating methods produce similar results.

After reviewing documentation submitted by FAA and information obtained during interviews, we determined the extent to which the cost estimates met the characteristics of cost-estimating best practices for the four projects we reviewed.

Our review of project schedules was based on research that identified a range of best practices associated with effective schedule estimating. In addition, we obtained the consulting services of David Hulett, Ph.D., to assist in our risk analysis of the WAAS project schedule. We also conducted multiple interviews with project managers, contractors, and schedulers to determine the extent to which current project schedules met the best practices criteria. These nine practices are: Capturing all activities: The schedule should reflect all activities (steps, events, outcomes, and other factors) as defined in the project’s work breakdown structure, including activities to be performed by both the government and its contractors.

Sequencing all activities: The schedule should be planned so that it can meet project-critical dates. To meet this objective, activities need to be logically sequenced in the order that they are to be carried out. In particular, activities that must finish prior to the start of other activities (i.e., predecessor activities) and activities that cannot begin until other activities are completed (i.e., successor activities) should be identified. Identifying interdependencies among activities that collectively lead to the accomplishment of events or milestones can be used as a basis for guiding work and measuring progress.

Assigning resources to all activities: The schedule should realistically reflect what resources (i.e., labor, material, and overhead) are needed to do the work, whether all required resources will be available when they are needed, and whether any funding or time constraints exist.

Establishing the duration of all activities: The schedule should reflect how long each activity will take to execute. In determining the duration of each activity, the same rationale, data, and assumptions used for cost estimating should be used for preparing the schedule. Furthermore, these durations should be as short as possible and should have specific start and end dates. Excessively long periods needed to execute an activity should prompt further decomposition of the activity so that shorter execution durations will result.

Integrating schedule activities horizontally and vertically: The schedule should be horizontally integrated, meaning that it should link the products and outcomes associated with already sequenced activities. These links are commonly referred to as “hand-offs” and serve to verify that activities are arranged in the right order to achieve aggregated products or outcomes. The schedule should also be vertically integrated, meaning that traceability exists among varying levels of activities and supporting tasks and subtasks. Such mapping or alignment among levels can enable different groups to work to the same master schedule.

Establishing the critical path for all activities: With the use of scheduling software, the critical path—the longest-duration path through the sequenced list of activities—should be identified. The establishment of a project’s critical path is necessary for examining the effects of delays in any activity along this path. Potential problems that may occur on or near the critical path should also be identified and reflected in the scheduling of the time for high-risk activities (see the next activity, “Identifying float”).

Identifying reasonable float: The schedule should identify float—the time that a predecessor activity can slip before the delay affects successor activities—so that schedule flexibility can be determined. As a general rule, activities along the critical path typically have the least amount of float.

Conducting a schedule risk analysis: A schedule risk analysis uses a good critical path method schedule and data about project schedule risks, as well as Monte Carlo simulation techniques, to predict the level of confidence in meeting a project’s completion date, the amount of time contingency needed for a level of confidence, and the identification of high-priority risks. This analysis should focus not only on critical path activities but also on other schedule paths that may become critical. A schedule/cost risk assessment recognizes the interrelationship between schedule and cost and captures the risk that schedule durations and cost estimates may vary for a variety of reasons, including limited data, optimistic estimating, technical challenges, lack of qualified personnel, and other external factors. As a result, the baseline schedule should include a buffer or a reserve of extra time. A reserve of extra time for contingencies should be calculated by performing a schedule risk analysis. As a general rule, the reserve should be held by the project manager and applied as needed to those activities that take longer than scheduled because of the identified risks. Reserves of time should not be apportioned in advance to any specific activity since the risks that will actually occur and the magnitude of their impact are not known in advance.

Updating the schedule using logic and durations to determine the dates: The schedule should use logic and durations in order to reflect realistic start and completion dates for project activities. The schedule should be continually monitored to determine when forecasted completion dates differ from the planned dates. This information can be used to determine whether schedule variances will affect downstream work. Maintaining the integrity of the schedule logic is not only necessary to reflect the project’s true status but is also required before conducting a schedule risk analysis. The schedule should avoid logic overrides and artificial constraint dates that are chosen to create a certain result on paper. Individuals trained in critical path method scheduling should be responsible for updating the schedule.

Based on our work, we determined the extent to which estimates and schedules for the four projects we selected met each best practices criterion:

Not Met—project officials provided no evidence that satisfies any portion of the criterion.

Minimally Met—project officials provided evidence that satisfies a small portion of the criterion.

Partially Met—project officials provided evidence that satisfies about half of the criterion.

Substantially Met—project officials provided evidence that satisfies a large portion of the criterion.

Met—project officials provided evidence that satisfies the entire criterion.

We conducted this performance audit from August 2010 to February 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Detailed Purpose and Status for 30 Acquisitions GAO Reviewed

This appendix contains detailed information for 30 individual air traffic control programs. Each overview presents information and data that was provided by FAA. The overviews provide a description of the program and the cost and schedule status. The overviews are based on program office reported information as of August 2011. In most cases, we did not validate the data provided, but reviewed the data and performed various checks to determine they were reliable enough for our purposes.

Appendix III: Assessments of Four FAA Program Cost Estimates

This appendix provides the results of our analysis of the extent to which the processes and methodologies used to develop and maintain the four FAA cost estimates meet the characteristics of high-quality cost estimates. These characteristics incorporate the 12 steps consistently applied by cost-estimating organizations throughout the federal government and industry and considered best practices for developing cost estimates and that are listed in table 2 of the report. The following tables provide the detailed results of our analysis of the program cost estimates for Automatic Dependent Surveillance-Broadcast (ADS-B), Collaborative Air Traffic Management Technologies (CATMT), System Wide Information Management (SWIM), and Wide Area Augmentation System (WAAS). “Not met” means the program provided no evidence that satisfies any of the criteria. “Minimally met” means the program provided evidence that satisfies a small portion of the criterion. “Partially met” means the program provided evidence that satisfies about half of the criterion. “Substantially met” means the program provided evidence that satisfies a large portion of the criterion. “Fully met” means the program provided evidence that completely satisfies the criterion.

Appendix IV: Assessments of Four FAA Program Schedules

This appendix provides the results of our analysis of the extent to which the processes and methodologies used to develop and maintain four FAA integrated master schedules meet nine best practices associated with effective schedule estimating. The following tables provide the detailed results of our analyses of the schedules for the Automatic Dependent Surveillance-Broadcast (ADS-B), Collaborative Air Traffic Management Technologies (CATMT), System Wide Information Management (SWIM), and Wide Area Augmentation System (WAAS) programs compared to the nine best practices. “Not met” means the program provided no evidence that satisfies any portion of the criterion. “Minimally met” means the program provided evidence that satisfies a small portion of the criterion. “Partially met” means the program provided evidence that satisfies about half of the criterion. “Substantially met” means the program provided evidence that satisfies a large portion of the criterion. “Fully met” means the program provided evidence that satisfies the entire criterion.

Schedule Risk Analysis

A best practice that the WAAS contractor schedule did not meet is conducting a schedule risk analysis, which is not required by FAA’s schedule specifications. FAA officials told us that they do not conduct schedule risk analysis. In August and September 2011, we performed our own schedule risk analysis on the WAAS contractor schedule, through which we analyzed the latest version of the contractor’s schedule available to us at the time of the analysis.

A schedule risk analysis uses statistical techniques to predict a level of confidence in meeting a program’s completion date. This analysis focuses on critical path activities and on near-critical and other activities since any activity may potentially affect the program’s completion date. The objective of the simulation is to develop a probability distribution of possible completion dates that reflect the program and its quantified risks. From the cumulative probability distribution, the organization can match a date to its degree of risk tolerance. For instance, an organization might want to adopt a program completion date that provides a 70 percent probability that the program will finish on or before that date, leaving a 30 percent probability that it will extend beyond, or overrun that date, given the schedule and the risks. The organization can thus adopt a plan consistent with its desired level of confidence in the overall integrated schedule. This analysis can give valuable insight into what-if drills and quantify the effects of program changes.

In developing a schedule risk analysis, probability distributions for each activity’s duration have to be established. Furthermore, risk in all activities must be evaluated and included in the analysis. Some managers focus only on the critical path, but because we cannot be certain how long activities will take, we cannot know the true critical path. Consequently, it would be a mistake to focus only on the software-calculated critical path (those activities in which, if delayed, will negatively impact the overall project completion date) when some off-critical-path activity might become critical if a risk were to occur. Typically, three-point estimates— that is, best, most likely, and worst-case estimates—are used to develop the probability distributions for the duration of workflow activities.

Once the distributions have been established, a Monte Carlo simulation uses random numbers to select specific durations from each activity probability distribution and calculates a new critical path and dates, including major milestone and program completion dates. The Monte Carlo simulation continues this random selection thousands of times, creating a new program duration estimate and critical path each time. The resulting frequency distribution displays the range of program completion dates along with the probabilities that these dates will occur. Table 17 provides a range of dates and the probability of the project’s completion on those dates or earlier, based on our 3,000 iterations that are chosen at random during the Monte Carlo simulation. For example, according to our schedule risk analysis, there is a less than 5 percent chance that the project will be finished on or before September 13, 2012. Likewise, there is an 80 percent chance that the project will be finished on or before November 13, 2012.

Because completion on any date is uncertain, it is more realistic to show a range of possible completion dates than to focus on a single date. In deciding which percentile to use for prudent scheduling, there is no international best practice standard. The chosen percentile depends on the riskiness and maturity of the project. For some projects, we emphasize the 80th percentile as a conservative promise-of-completion date. While the 80th percentile may appear overly conservative, it is a useful promise-of-completion date if a number of new but currently unknown risks (i.e., “unknown unknowns”) are anticipated. The 50th percentile date may expose the project to overruns.

In the case of the WAAS contractor schedule, our analysis concluded that management should realistically expect cutover, or completion, between October 23, 2012, and November 13, 2012, the 50th and 80th percentiles, respectively. The artificial must-finish date constraint of September 6, 2012, built into the schedule is unlikely. Our analysis shows the probability of completion by September 6, 2012, is less than 5 percent likely with the current schedule without risk mitigation. There are two reasons why the planned end completion date is not likely to occur, according to the results of our schedule risk analysis. First, most risks are threats only. Only two opportunities were identified during the analysis: (1) the estimating error of the schedule may be between -10 percent and 15 percent, and (2) there is a 65 percent chance that the 11-day formal shadow test will not be needed. Second, there are parallel paths within the structure of the schedule that lead to merge points. If several paths converge to one milestone, the latest merging path determines the date. This “merge bias” cannot accelerate schedule dates and usually adds structural risk to the schedule.

Identified Risks

The contractor supplied six different risks that are currently identified in the project’s risk register. Using these risks as a basis for discussion, we interviewed 16 experts familiar with the project, including prime contractor officials, FAA officials, and technical FAA consultants to identify any other risks. Each interviewee was asked four questions to address four related points.

To estimate the probability that an identified risk will occur on the project in such a way that some activity durations are affected. The estimated probability is translated into the percentage of iterations that are chosen at random during the simulation. For example, if the expert estimates weather will have a 10 percent chance of affecting some activities, then, on average, weather risk will occur in 10 percent of the Monte Carlo iterations.

If the interviewee believed the identified risk was likely, the interviewee was asked to identify which activities’ durations would be affected. For example, activities related to steel erection or concrete pouring may be affected if the weather risk is realized and bad weather occurs.

After the interviewee identified affected activities, the interviewee was asked to provide a three-point estimate of the risk’s effect on duration—low, most likely, and high. Estimates were provided as percentages, which were applied to the activity durations in the Monte Carlo simulation if the risk occurred. For example, if bad weather occurs, the duration of a 10-day steel erection activity may be affected a minimum of 110 percent, a most likely of 150 percent, or a maximum of 200 percent. These percentages translate into increases in the activity’s duration of 11 days under a low-risk scenario, 15 days under the most likely risk scenario, and 20 days under the maximum risk scenario. If the risk is not realized, there is no change to the activity’s original estimated duration.

Finally, the interviewee was asked to identify any unaccounted for risks.

We began the interviews with 6 risks and, through the interview process, identified 16 more risks. During data analysis, some risks were consolidated with others or eliminated because of limited data. In all, 14 risks were identified and incorporated into the Monte Carlo simulation. These include 9 risk drivers, 4 existence risks, and 1 schedule duration risk.

The final risk drivers used in the schedule risk analysis are as follows:

Some software Release 3B problem reports require testing with a live geostationary satellite; software simulation environment may not be able to test all requirements.

Potential difficulty repairing Fullerton contractor’s lab safety computers.

Software delays in software Release 3A and additional changes to software Release 2B are likely to delay the 3B schedule.

Parallel implementation of Releases 2B, 2C, 3A, and 3B create competition for FAA resources.

Release 3B problem report testing is more complex and subject to uncertainty.

FAA WAAS personnel move around, and inexperienced staff must be trained on WAAS.

Workshare strategy between FAA and its prime contractor affects the schedule for on-the-job training learning curve.

Changes to the simulation tools may be more difficult or more time- consuming to implement than anticipated.

The final existence risks are as follows:

FAA performs formal software release 3B shadow test.

Late problem report may come from FAA.

FAA program office is resource-limited.

Inserting existence risk activities does not affect dates within the baseline schedule because the activities initially have zero duration. The activities have duration only if they happen to occur during an iteration of the simulation.

The final uncertainty risk is schedule without buffer may be optimistic.

Most risks were identified by multiple respondents during the interviews. During data analysis, data from the interviews are combined and analyzed to create ranges and probabilities for each of the 14 risks.

Because risks are multiplicative, several risks occurring on the same activity may overestimate the true risk. That is, by default in the Monte Carlo simulation, risks occur in a series, one after another, so that an activity that has several risks may be unrealistically extended if all risks occur. In reality, an activity may recover from two or more risks simultaneously, so that the actual risk is not multiplicative. Therefore, to avoid overestimation of risk, risks can explicitly be defined as occurring in parallel rather than in series. Risks that occur in a series will occur one after the other and add (or subtract) their respective effects on duration to the affected activity. If risks occur in parallel, on the other hand, only the maximum effect of all risks will affect the duration. For example, if the risk of complexity adds 3 days to the duration of software development and the risk of staff shortages adds 4 days, then development will extend 7 days if the risks are defined in series. However, the duration will extend only 4 days if the risks are defined as parallel. This definition of parallel risks helps temper any risk overestimated by a multiplication of risk factors. We defined one risk in series: software delays in Release 3A and additional changes to Release 2B are likely to delay Release 3B schedule. All other risks were assumed to be parallel.

Most risks were assigned directly to existing activities in the schedule. However, some risks required adjustments to the schedule. These adjustments involved replacing lags with activities and inserting existence risk activities.

Lags: During our initial analysis of the contractor’s schedule, we identified 25 remaining activities with lags and 21 remaining activities with leads (negative lags). While lags represent the passing of time between activities, they are often misused to put activities on a specific date or to insert a buffer for risk. That number was reduced to 20 activities with lags and 6 activities with leads with the “Rev 1” schedule that was altered by the prime contractor for our use in the schedule risk analysis. We replaced the 33-day lag between the R3B Start Cutover and R3B End Cutover activities with an actual activity, ID 258 “Cutover Task.” Replacing lags with actual activities does not affect dates within the baseline schedule because the activities have the same duration as the lags.

Existence risk: We identified some risks that would add an indeterminate amount of time to the overall schedule if they were realized. For example, if the Tijuana facility is not ready because software requirements are misunderstood, it could add 4 to 26 days to the schedule in the form of additional facility work.

Prioritizing risks and risk mitigation: Risks can affect the schedule in several ways: They can have a high probability of occurring; have a large- percentage impact on the durations of the activities they affect; or they can apply to risk-critical paths, which may differ from the baseline deterministic critical path. Beyond applying 14 risks to the schedule, we were interested in identifying the marginal impact of each risk. That is, we were interested in identifying which risks have the largest impact on the schedule because these were the risks that should be targeted first for mitigation.

To find the marginal impact of a risk on the total project risk at a certain percentile, the Monte Carlo simulation was performed with the risk removed. The difference between the finish dates of the simulation with all the risks and the simulation with the missing risk yielded the marginal impact of the risk. Table 18 gives the priority of risks at the 80th percentile and the marginal impact of each risk.

The marginal impact directly translates to potential calendar days saved if the risk is mitigated. Once risks are prioritized at the percentile desired by management, a risk mitigation workshop can be implemented to deal with the high-priority risks in order. The prioritized list of risks will form the basis of the workshop, and risk mitigation plans can be analyzed using the risk model to determine how much time might be saved. Project managers cannot expect to completely mitigate any one risk, and it is not reasonable to expect to mitigate all risks. In addition, risk mitigation will add to the project budget. However, some opportunities may be available to partially mitigate risks.

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, individuals making key contributions to this report include Edward Laughlin and Karen Richey (Assistant Directors), Lindsey Bach, David Brown, Pamela Davidson, Tisha Derricotte, Kevin Egan, James Geibel, Bert Japikse, Delwen Jones, Jason Lee, Dominic Nadarski, Josh Ormond, and Brian Welsh.