AIR TRAFFIC 

CONTROL 

MODERNIZATION

Management 

Challenges Associated 

with Program Costs 

and Schedules Could 

Hinder NextGen 

Implementation

Report to Congressional Committees

February 2012

GAO-12-223

United States Government Accountability Office

GAO
United States Government Accountability Office

Highlights of GAO-12-223, a report to 

congressional committees 

February 2012 

AIR TRAFFIC CONTROL MODERNIZATION 

Management Challenges Associated with Program 

Costs and Schedules Could Hinder NextGen 

Implementation 

Why GAO Did This Study

The Federal Aviation Administration 

(FAA), partnering with other federal 

agencies and the aviation industry, is 

implementing the Next Generation Air 

Transportation System (NextGen), a 

new satellite-based air traffic 

management system that will replace 

the current radar-based system and is 

expected to enhance the safety and 

capacity of the air transport system by 

2025. Concurrently, FAA continues to 

maintain and upgrade existing air 

traffic control (ATC) systems that will 

also be needed for NextGen. This 

involves acquiring and implementing 

new software and hardware. 

GAO was asked to determine (1) how, 

if at all, costs and schedules of FAA 

ATC acquisitions programs, including 

those related to NextGen, have 

changed since they were first 

submitted to Congress, (2) the reasons 

for any such changes, and (3) the 

extent that selected ATC programs 

adhere to cost and schedule best 

practices. To do its work, GAO 

reviewed 30 programs and conducted 

cost and schedule analysis on four 

programs that had an approved 

baseline and were NextGen related. 

GAO reviewed acquisition documents 

and interviewed FAA officials.

What GAO Recommends

To better estimate the cost and 

completion dates for major 

acquisitions, FAA should, among other 

things, require cost and schedule risk 

analysis, independent cost estimates 

and integrated master schedules. FAA 

did not comment on whether or not it 

agreed with the recommendations.

What GAO Found

In a review of 30 major ATC acquisition programs, all of which will contribute to 

the transition to NextGen, GAO found that costs for 11 of the 30 programs have 

increased from their initial estimates by a total of $4.2 billion and 15 programs 

experienced delays. The 11 acquisitions that experienced cost increases account 

for over 60 percent of FAA’s total acquisition costs ($11 billion of $17.7 billion) for 

the 30 programs. The 15 acquisitions that experienced schedule delays, of which 

10 also had cost increases, ranged from 2 months to more than 14 years and 

averaged 48 months. 

Cost increases and schedule delays occurred due to several factors, many of 

which have been longstanding challenges for FAA. Specifically, these have 

involved (1) additional or unanticipated system requirements; (2) insufficient 

stakeholder involvement (such as controllers’ input) throughout system 

development; (3) underestimating the complexity of software development; and 

(4) unanticipated events including funding shortfalls or work stoppages. These 

challenges, if they persist, will impede the implementation of NextGen, especially 

in light of the interdependencies among many acquisition programs, where cost 

increases or delays in one program can affect the costs and schedules of other 

programs. 

For the four programs GAO selected to analyze in depth, FAA is not consistently 

following the characteristics of high-quality cost estimates and scheduling best 

practices that GAO previously identified. Regarding cost estimates, GAO found 

that although all four of the programs generally provided well documented and 

comprehensive estimates, which are two of the four characteristics, no program 

fully met the two other characteristics. Specifically, each program estimate was 

not credible because each lacked an independent cost estimate, which provides 

a check against FAA’s estimate and three programs lacked risk or uncertainty 

analysis. The estimates also lacked accuracy because they were not updated 

regularly or based on comparable programs. Regarding scheduling practices, 

most programs did not substantially or fully meet the majority of the 9 best 

practices GAO previously identified including developing a fully integrated master 

schedule of all program activities and performing a schedule risk analysis. For 

example, without a schedule risk analysis, FAA is unable to predict, with any 

degree of confidence, if the estimated completion dates are realistic. FAA is 

implementing new processes and organizational changes to better manage 

acquisitions. However, by not consistently following the characteristics of highquality cost estimate and scheduling best practices, FAA cannot provide 

reasonable assurance to Congress and other stakeholders that NextGen and 

other ATC programs will avoid additional cost increases or schedule delays. 

View GAO-12-223 or key components.

 For more information, contact Gerald L. 

Dillingham, Ph.D., at 

(202) 512-2834 or dillinghamg@gao.gov . 
Page i GAO-12-223 Air Traffic Control Modernization

Letter 1 

Background 4

Most Ongoing ATC Programs Remain within Budget and on 

Schedule, but a Few Have Significantly Exceeded Initial 

Estimates 12

Schedules for Roughly Half of ATC Programs Are on Track, but 

Half of ATC Programs Are Delayed 15

Several Factors Contributed to Cost Increases and Schedule 

Delays, and Some Could Hamper NextGen Implementation 17

For Programs We Reviewed, FAA Did Not Consistently Meet All 

Characteristics of Quality Cost Estimates or Schedule Best 

Practices 24

Conclusions 39

Recommendations for Executive Action 41

Agency Comments 42

Appendix I Objectives, Scope and Methodology 45

Appendix II Detailed Purpose and Status for 30 Acquisitions GAO Reviewed 52

Appendix III Assessments of Four FAA Program Cost Estimates 82

Appendix IV Assessments of Four FAA Program Schedules 102

Appendix V GAO Contact and Staff Acknowledgments 142

Tables

Table 1: FAA Acquisition Life Cycle 6

Table 2: Characteristics of High-Quality Cost Estimates and Steps 

Related to Each Characteristic 10

Table 3: Eleven ATC Programs That Have Experienced Cost 

Increases as of August 2011 14

Contents
Page ii GAO-12-223 Air Traffic Control Modernization

Table 4: Fifteen ATC Programs That Have Experienced Delays 

Based on Original Scheduled Completion Date 16

Table 5: Key Factors Contributing to Cost Growth, Schedule 

Delays, or Both for ATC Systems 18

Table 6: GAO Analysis of the Extent FAA Acquisition Cost 

Estimates Met the Characteristics of High-Quality and 

Reliable Cost Estimates 25

Table 7: Extent Program Schedules Met Best Practices 29

Table 8: GAO’s Analysis of the FAA’s ADS-B Cost Estimates 82

Table 9: GAO’s Analysis of FAA’s CATMT Cost Estimates 87

Table 10: GAO’s Analysis of FAA’s SWIM Cost Estimates 92

Table 11: GAO’s Analysis of FAA’s WAAS Cost Estimates 97

Table 12: Extent to Which the ADS-B Schedule Met Best Practices 102

Table 13: Extent to Which the CATMT Schedule Met Best Practices 109

Table 14: Extent to Which the SWIM Schedule Met Best Practices 115

Table 15: Extent to Which the FAA-Prepared WAAS Schedule Met 

Best Practices 123

Table 16: Extent to Which the WAAS Contractor-Prepared 

Schedule Met Best Practices 131

Table 17: Probability of Project Completion 136

Table 18: Prioritized Risks at the 80th Percentile 141

Figures

Figure 1: Acquisition Programs Reported by FAA in the CIP for 

2012 through 2016 5

Figure 2: Improvements to Phases of Flight Expected under 

NextGen 8

Figure 3: Automatic Dependent Surveillance-Broadcast (ADS-B) 52

Figure 4: Advanced Technologies and Oceanic Procedures (ATOP) 53

Figure 5: Airport Surveillance Radar (ASR) Model 11 Tech Refresh 

Segment 1 54

Figure 6: Aviation Surface Weather Observation Network 

(ASWON) and Automated Surface Observing System 

(ASOS) P31 55

Figure 7: Air Traffic Control Radar Beacon Interrogator (ATCBI-6) 56

Figure 8: Collaborative Air Traffic Management Technologies 

(CATMT), Work Packages 2 and 3 (WP2 and WP3) 57

Figure 9: En Route Communication Gateway (ECG) Tech Refresh 58

Figure 10: En Route Automation Modernization (ERAM) 59

Figure 11: Integrated Display Systems (IDS) Replacement 60

Figure 12: International Flight Inspection Aircraft (IFIA) 61
Page iii GAO-12-223 Air Traffic Control Modernization

Figure 13: Instrument Flight Procedure Automation (IFPA) 62

Figure 14: Integrated Terminal Weather System (ITWS) 63

Figure 15: Next Generation Air/Ground Communication System 

(NEXCOM) Segment 1a 64

Figure16: Next Generation Weather Radar (NEXRAD) Dual 

Polarization 65

Figure 17: Power Systems Sustained Support (PS3) 66

Figure 18: Regulation and Certification Infrastructure for System 

Safety (RCISS) Segment 2 67

Figure 19: Runway Status Lights (RWSL) 68

Figure 20: Standard Terminal Automation Replacement System 

(STARS) (TAMR Phase 1) 69

Figure 21: System Wide Information Management (SWIM) 

Segment 1 70

Figure 22: Terminal Automation Modernization and Replacement 

(TAMR)—Phase 2 71

Figure 23: Tower Training Simulator Systems 72

Figure 24: Trajectory Management—Arrival Tactical Flow Time 

Based Flow Management 73

Figure 25: Terminal Doppler Weather Radar (TDWR) Service Life 

Extension Program (SLEP) 74

Figure 26: Traffic Flow Management System (TFMS) Tech Refresh 75

Figure 27: Ultra High Frequency (UHF) Radio Replacement 76

Figure 28: Next Generation Voice Recorder Replacement Program 

(VRRP) 77

Figure 29: Voice Switching and Control Switching System (VSCS) 

Tech Refresh Phase 2 78

Figure 30: Wide Area Augmentation System (WAAS) 79

Figure 31: Weather and Radar Processor (WARP) Sustain 80

Figure 32: Weather Camera Program (WCP) 81
Page iv GAO-12-223 Air Traffic Control Modernization

Abbreviations

ABRR Airborne Reroute Execution 

ADS-B Automatic Dependent Surveillance-Broadcast 

AirNav Airports and Navigations Aids database 

AMS Acquisition Management System 

APB acquisition planning baseline 

APTS Automated Procedures Tracking System 

ASDP Advanced Signal Data Processor 

ASOS Automated Surface Observing System 

ASR Airport Surveillance Radar 

ASWON Aviation Surface Weather Observation Network 

ATC air traffic control 

ATCBI-6 Air Traffic Control Radar Beacon Interrogator 

ATCSCC Air Traffic Control System Command Center 

ATO Air Traffic Organization 

ATOP Advanced Technologies and Oceanic Procedures 

AUM Arrival Uncertainty Management 

CACR Collaborative Airspace Constraint Resolution 

CATMT Collaborative Air Traffic Management Technologies

CIP Capital Investment Plan 

Cost Guide GAO Cost Estimating and Assessment Guide 

DataComm Data Communications 

DOD Department of Defense 

EBUS Enhanced Back-up Surveillance 

ECG En Route Communication Gateway

ERAM En Route Automation Modernization 

ERIDS En Route Information Display System 

FAA Federal Aviation Administration 

F&E facilities and equipment 

I2I Idea to In-Service Management

IDA Investment Decision Authority 

IDS Integrated Display Systems 

IFIA International Flight Inspection Aircraft 

IFP Instrument Flight Procedures 

IFPA Instrument Flight Procedure Automation 
Page v GAO-12-223 Air Traffic Control Modernization

IP&A Investment Planning and Analysis 

IPDS Instrument Procedure Development System 

IT information technology 

ITWS Integrated Terminal Weather System 

JRC Joint Resources Council 

NAS National Airspace System 

NextGen Next Generation Air Transportation System 

NEXCOM Next Generation Air/Ground Communication System 

NEXRAD Next Generation Weather Radar 

OE Obstacle Evaluation 

OMB Office of Management and Budget 

P3I Pre-Planned Product Improvements 

PAMRI Peripheral Adapter Module Replacement Item 

PS3 Power Systems Sustained Support 

RCISS Regulation and Certification Infrastructure for System 

Safety 

RWSL Runway Status Lights 

SIP SWIM implementation program 

SLEP Service Life Extension Program 

SRA schedule risk analysis

STARS Standard Terminal Area Replacement System 

SWIM System Wide Information Management 

TAMR Terminal Automation Modernization and Replacement 

TDWR Terminal Doppler Weather Radar 

Tech Center FAA Technical Center 

TERPS Terminal Instrument Procedures 

TFM Traffic Flow Management 

TFMS Traffic Flow Management System 

TRACON Terminal Radar Approach Control 

UHF Ultra High Frequency 

VRRP Next Generation Voice Recorder Replacement Program 

VSCS Voice Switching and Control Switching System 

WAAS Wide Area Augmentation System 

WAM Colorado Wide Area Multilateration 

WARP Weather and Radar Processor 

WCP Weather Camera Program 

WP2 Work Package 2

WP3 Work Package 3

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately.
Page 1 GAO-12-223 Air Traffic Control Modernization

United States Government Accountability Office

Washington, DC 20548

February 16, 2012

Congressional Committees

To accommodate anticipated increases in air passenger traffic over time,1

the Federal Aviation Administration (FAA) has expanded its acquisitions 

program to sustain current—or legacy—air traffic control (ATC) facilities 

and systems while simultaneously replacing or supplementing those 

systems through transition to the satellite-based Next Generation Air 

Transportation System (NextGen). This modernization effort involves 

acquiring and implementing new, advanced air traffic management 

systems, including hardware and software, to dramatically change the 

way the current aviation system is operated. As the agency transitions to 

NextGen, which has significantly increased the number, cost, and 

complexity of FAA’s acquisition programs, it is imperative that these 

programs remain on time and within budget, particularly given current 

budget constraints and the interdependencies of many NextGen-related 

ATC programs. FAA has taken several steps to improve its acquisition 

management––including implementing a cost estimating methodology, a 

cost accounting system, and a business process, and developing an 

enterprise architecture––which resulted in the removal of its acquisition 

management from the GAO High-Risk list in 2009.2

In response to your request, this report (1) describes how, if at all, the 

planned costs and schedules of current FAA ATC acquisition programs, 

including those related to NextGen, have changed since they were first 

submitted to Congress, (2) examines the reasons for any changes in 

planned costs and schedules, and (3) assesses the extent to which 

However, recent cost 

and scheduling problems among some major acquisition programs, such 

as the En Route Automation Modernization (ERAM), which is integral to 

ATC modernization, have renewed concerns about the agency’s ability to 

manage complex multibillion-dollar procurement programs.

 

1

The nation’s air traffic control system handles almost 30 million flights per year. FAA 

predicts that by 2025, the number of passengers will increase from about 700 million to 

about 1.1 billion per year, and the number of flights will increase from about 80,000 to 

more than 95,000 every 24 hours.

2

GAO, High Risk Series: An Update, GAO-09-271 (Washington, D.C.: January 2009).

 
 

Page 2 GAO-12-223 Air Traffic Control Modernization

selected ATC programs adhere to best practices for determining 

acquisition costs and schedules.

To determine the changes, if any, in the costs and schedules of FAA’s 

ATC programs, we gathered and analyzed agency data on the estimated 

costs and schedules of the 30 ATC programs3 that had baselines—that 

is, those programs whose estimated budget and schedule had received 

FAA executive approval.4 Eighteen of these programs are directly related 

to the implementation of NextGen, and all are needed to maintain and 

modernize the existing ATC system in order for it to operate in the 

NextGen environment. We also drew upon past GAO work in which we 

undertook detailed reviews of the status of ATC and other acquisition 

programs5

To examine the reasons contributing to any changes in cost estimates 

and schedules in the 30 baselined ATC programs, we interviewed FAA 

officials and contractors and reviewed program documentation. We 

analyzed information on cost increases and delays for the 30 baselined 

programs to determine the factors that contributed to cost and schedule 

changes.

and obtained updated documentation as necessary from FAA. 

We interviewed FAA officials to obtain information on their programs’ past 

and current challenges and current status and summarized the status of 

all 30 ATC programs, including their original and current cost estimates 

and completion dates. For each program, we compared its initial 

estimated cost at the time of its submission to Congress for approval 

against its current cost estimate and compared its planned and actual 

schedules.

 

3

We requested the information on the programs in August 2010.

4

According to FAA, baselined programs are acquisition programs with an agreed-to 

description of the attributes and estimated costs at a particular point in time. The baseline 

is a formal, management-approved document that serves as a starting point for tracking 

any changes in the program as it is developed and implemented for performance 

accountability. 

5

GAO, Department of Homeland Security: Assessments of Selected Complex 

Acquisitions, GAO-10-588SP, (Washington, D.C.: June 30, 2010); Defense Acquisitions: 

Assessments of Selected Weapon Programs, GAO-10-388SP (Washington, D.C.: Mar. 

30, 2010); Air Traffic Control: FAA Reports FAA Reports Progress in System Acquisitions, 

but Changes in Performance Measurement Could Improve Usefulness of Information, 

GAO-08-42 (Washington, D.C.: Dec. 18, 2007); and National Airspace System: FAA Has 

Made Progress but Continues to Face Challenges in Acquiring Major Air Traffic Control 

Systems, GAO-05-331 (Washington, D.C.: June 10, 2005).
 

Page 3 GAO-12-223 Air Traffic Control Modernization

To assess the extent to which FAA’s cost estimating and scheduling 

processes aligned with best practices, we conducted an in-depth review 

of 4 of the 30 baselined programs: the Automatic Dependent 

Surveillance-Broadcast (ADS-B) system, the Collaborative Air Traffic 

Management Technologies (CATMT) system, the System Wide 

Information Management (SWIM) system, and the Wide Area 

Augmentation System (WAAS).6 We selected these programs based on 

the following criteria: the program had reached sufficient maturity such 

that risks could be identified and the program was key to the 

modernization of the air traffic control system. We conducted in-depth 

interviews with FAA acquisition program managers for the programs 

selected. In addition to interviews, we obtained and analyzed the most 

current cost and schedule estimates for these programs. To assess the 

extent to which FAA’s acquisition practices for these programs were 

consistent with best practices, we used the 2009 GAO Cost Estimating 

and Assessment Guide (Cost Guide).7

We conducted this performance audit from August 2010 to February 2012 

in accordance with generally accepted government auditing standards. 

Those standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

In assessing each program’s cost 

estimate, we used the Cost Guide to evaluate FAA’s estimating 

methodologies, assumptions, and results to determine whether the cost 

estimate was well-documented, comprehensive, accurate, and credible. 

We also determined the extent to which the schedule was prepared in 

accordance with best practices that are fundamental to having a reliable 

schedule. In addition, we performed a schedule risk analysis on a WAAS 

schedule prepared by the contractor to determine the high-priority risks 

that the program may encounter that could affect the schedule and the 

likelihood of the program finishing on time. We selected the WAAS 

contractor schedule because we determined that it was the only schedule 

of the four programs we reviewed in detail that was sufficiently reliable for 

a risk analysis to be performed.

 

6

The four initial cost estimates we reviewed were developed at various points in the past.

7

GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and 

Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: March 2009).
 

Page 4 GAO-12-223 Air Traffic Control Modernization

conclusions based on our audit objectives. Appendix I contains more 

detailed information on our objectives, scope, and methodology.

FAA catalogs its acquisition programs in its annually updated Capital 

Investment Plan (CIP). The CIP identifies planned capital investment in 

the National Airspace System (NAS) for the next 5 years consistent with 

the amount requested in the agency’s annual budget submission. 

Appendix C of the CIP, which identifies the anticipated budget line items, 

is divided into five activities: (1) Engineering, Development, Test, and 

Evaluation; (2) ATC Facilities and Equipment; (3) Non-ATC Facilities and 

Equipment; (4) Facilities and Equipment Mission Support; and (5) 

Personnel Compensation, Benefits and Travel. The CIP for fiscal years 

2012 through 2016 contains 106 funded acquisition programs with 

estimated total budgets (through 2016) of more than $14 billion. Of these

83 acquisition programs FAA considers ATC related, 18 involve 

Engineering, Development, Test and Evaluation and 65 involve ATC 

Facilities and Equipment. The 83 programs include 30 that have had 

program baselines approved by FAA’s Joint Resources Council (JRC),8

which is responsible for approving major programs.9

 

8

The JRC is an FAA executive body consisting of associate and assistant administrators, 

acquisition program executives, the Chief Financial Officer, the Chief Information Officer, 

and legal counsel. The JRC makes executive-level decisions, including those that 

determine whether a program meets a mission need and should proceed. 

These 30 baselined 

programs include communications, navigation, and surveillance systems 

that are key to ATC operations. FAA considers 5 of the programs to be 

foundational parts of NextGen, and all are key to modernizing the existing 

ATC system. Figure 1 illustrates the universe of FAA acquisitions for 

fiscal years 2012-2016.

9

As defined in Office of Management and Budget Circular A-11, Part 7, major programs 

are assets that require special management attention because of their importance to the 

agency’s mission. These include high development, operating, or maintenance costs; high 

risk; high return; or a significant role in the administration of agency programs, finances, 

property, or other resources. 

Background 
 

Page 5 GAO-12-223 Air Traffic Control Modernization

Figure 1: Acquisition Programs Reported by FAA in the CIP for 2012 through 2016

FAA has developed and uses its Acquisition Management System (AMS) 

to provide policies and guidance for managing ATC system programs 

through all phases of the programs’ life cycles (see table 1). The Air 

Traffic Organization (ATO) within FAA is responsible for operating, 

maintaining, and modernizing the nation’s current ATC system.10

 

10FAA is implementing a reorganization of ATO that may alter the current alignment of 

program acquisition responsibilities within ATO.
 

Page 6 GAO-12-223 Air Traffic Control Modernization

Table 1: FAA Acquisition Life Cycle

Phase/activity What occurs during this phase or activity

Needs and solution identification

Mission analysis FAA identifies a capability shortfall and determines that it 

needs an investment to better carry out its mission. 

Recently, FAA began analyzing its mission needs within 

the context of its overall goals for the NAS. 

Investment analysis FAA, using an investment analysis team, evaluates 

alternatives, selects practical and affordable solutions, and 

develops a baseline of cost, schedule, and performance 

requirements. This document is called the acquisition 

program baseline.

Solution implementation

System integration Both hardware and software components and subsystems 

are integrated into a product. Also, intra- and inter-system 

compatibility are tested and analyzed.

System demonstration Tests show that the product can work as required and be 

manufactured within targets.

System production All activities are carried out to produce needed quantities. 

Each end item is tested before it leaves the factory to 

verify that it conforms to specifications and is free from 

manufacturing defects. 

In-service management All required activities are carried out, including directly 

operating, providing maintenance functions (both 

scheduled and unscheduled), and furnishing technical and 

logistics requirements.

Decommission All unneeded assets are decommissioned and removed 

from service at the end of their service lives.

Source: FAA.

The acquisition program baseline defines the cost, schedule and 

performance baselines for the investment program. The JRC which 

determines whether to approve a cost and schedule baseline also 

approves rebaselining, through which the agency documents and 

approves major changes to a program’s previously approved budget or 

schedule. Rebaselining resets the estimated costs and schedule used to 

determine how the program will be held accountable and can occur 

before the program is deployed. Once a program is rebaselined, FAA 

reports on the performance of the program based on the revised cost and 

schedule. Although the rationale for rebaselining can be reasonable, as, 

for example, when a program’s scope has been expanded, reporting a 
 

Page 7 GAO-12-223 Air Traffic Control Modernization

program’s performance based on a rebaselined cost or schedule can also 

skew or conceal from Congress and other stakeholders the program’s 

actual total costs or overall timeline. We previously reported11

NextGen involves changes to every aspect of air transportation (see 

fig. 2). NextGen requires the acquisition of new integrated systems (both 

software and hardware), flight procedures, aircraft performance 

capabilities, and supporting infrastructure to transform the current air 

transportation system into one that uses satellite-based surveillance and 

navigation operations instead of ground-based radar.

that the 

absence of this information on rebaselining in ATO’s performance 

reporting could cause managers and other stakeholders, including 

Congress, to think that performance was better than it actually was. We 

recommended that FAA regularly report on the overall, long-term 

performance in acquiring ATC systems by providing in FAA’s annual 

Performance and Accountability Report the original budget and schedule 

baselines for each rebaselined program and the reasons for the 

rebaselining. In response to our recommendation, FAA currently provides 

this information in Appendix D of its CIP, where it details baseline cost 

and schedule information for major acquisition programs.

12 These changes 

are intended to increase the efficiency and capacity of the air 

transportation system while maintaining safety and accommodating 

anticipated future growth. The planning for NextGen began in 200313 and 

is now focused on implementing improvements in the midterm (by 2018) 

and in the far term (by 2025).14

 

11GAO-08-42. 

12GAO, Next Generation Air Transportation System: Challenges with Partner Agency and 

FAA Coordination Continue, and Efforts to Integrate Near, Mid-, and Long-term Activities 

Are Ongoing, GAO-10-649T (Washington, D.C.: Apr. 21, 2010). 

13Pub. L. No. 108-176, Vision 100—Century of Aviation Reauthorization Act Pub. L. No. 

108-176, §§ 709-710, 117 Stat. 2490, 2582 (2003) authorized FAA to begin the NextGen 

initiative. 

14Midterm includes capabilities that are planned to be implemented by 2018, such as 

improved aircraft operational procedures and automated data communication between 

aircraft and controllers. Far term refers to the complete implementation of NextGen.
 

Page 8 GAO-12-223 Air Traffic Control Modernization

Figure 2: Improvements to Phases of Flight Expected under NextGen

As noted previously, we selected 4 programs for in-depth review to 

determine the extent to which their cost estimates and schedules aligned 

with best practices:

• ADS-B will enable aircraft to continually broadcast flight data such as 

position, air speed, and altitude, among other types of information, to 

air traffic controllers and other aircraft. The program was baselined in 

2007, and FAA is currently installing the hardware and software at 

approximately 800 sites across the nation. The program is scheduled 

to be completed in 2014.

• CATMT will provide new functionality and other enhancements to the 

existing Traffic Flow Management System (TFMS), such as 

automated reroutes and improved data exchanges between ATC 

facilities. CATMT was first baselined in 2008 and is scheduled to be 

completed in 2015.

• SWIM will provide an information technology infrastructure that will 

enable the multiple systems that make up the NAS to share 
 

Page 9 GAO-12-223 Air Traffic Control Modernization

information. As such, SWIM is a portfolio of capabilities that will be 

implemented by other systems. SWIM was first baselined in 2009 and 

is scheduled to be completed in 2020.

• WAAS will provide aircraft with more accurate aircraft position 

information to facilitate more direct flight paths and precision 

approaches to airports. The initial WAAS program was baselined in 

1998. The third segment of the program is scheduled to be completed 

in 2013.

Successful acquisition program management depends, in part, upon 

effective cost estimation. The cost estimate is the basis for establishing a 

program’s detailed schedule (see following discussion of schedules), as 

well as identifying the bounds for how much program costs can be 

expected to vary. We have defined cost estimates as the summation of 

individual cost elements, using established methods and valid data, to 

estimate future program costs based on what is currently known.15 As 

such, cost estimating requires both science and judgment because 

answers are seldom—if ever—entirely precise. The goal is to find a 

reasonable estimate.16

Our Cost Guide identifies 12 steps consistently applied by cost-estimating 

organizations throughout the federal government and industry and 

considered best practices for developing cost estimates. For the purposes 

of this review, we grouped these steps into four characteristics of highquality and reliable estimates: well-documented; comprehensive; 

accurate, and credible. (See table 2.)

Reliable cost estimating is a critical function 

without which agencies are at risk of experiencing cost overruns, missed 

deadlines, and performance shortfalls.

 

15GAO-09-3SP. 

16GAO-09-3SP. 
 

Page 10 GAO-12-223 Air Traffic Control Modernization

Table 2: Characteristics of High-Quality Cost Estimates and Steps Related to Each Characteristic 

Characteristic Explanation Step identified in Cost Guide

Well-documented The documentation should address the purpose of the estimate, the 

program background, a description of the system, the system’s 

schedule, the scope of the estimate (in terms of time and what is 

and is not included), the ground rules and assumptions, all data 

sources, the estimating and rationale, the results of the risk 

analysis, and a conclusion about whether the cost estimate is 

reasonable. Therefore, a good cost estimate—while taking the form 

of a single number—is supported by detailed documentation that 

describes how it was derived and how the expected funding will be 

spent in order to achieve a given objective. For example, the 

documentation should capture in writing such things as the source 

data used and their significance, the calculations performed and 

their results, and the rationale for choosing a particular estimating 

method or reference. Moreover, this information should be captured 

in such a way that the data used to derive the estimate can be 

traced back to and verified against their sources, allowing for the 

estimate to be easily replicated and updated. Finally, the cost 

estimate should be reviewed and accepted by management to 

ensure that there is a high level of confidence in the estimating 

process and in the estimate itself. 

Step 1: Define the estimate’s purpose, 

scope, and schedule

Step 3: Define the program 

characteristics

Step 5: Identify ground rules and 

assumptions

Step 6: Obtain the data

Step 10: Document the estimate

Step 11: Present the estimate to 

management for approval 

Comprehensive The cost estimates should include both government and contractor 

costs of the program over its full life cycle, from inception through 

design, development, deployment, and operation and maintenance 

to retirement. The estimates should also completely define the 

program, reflect its current schedule, and be technically reasonable. 

Comprehensive cost estimates should provide a level of detail 

appropriate to ensure that cost elements are neither omitted nor 

double-counted, and they should document all cost-influencing 

ground rules and assumptions. Establishing a product-oriented 

work breakdown structure is a best practice because it allows a 

program to track cost and schedule by defined deliverables, such 

as a hardware or software component. 

Step 2: Develop the estimating plan

Step 4: Determine the estimating 

structure 

Step 5: Identify ground rules and 

assumptions 

Accurate The cost estimates should provide for results that are unbiased, 

and they should not be overly conservative or optimistic. Estimates 

are accurate when they are based on an assessment of most likely 

costs, adjusted properly for inflation, and contain few, if any, minor 

mistakes. In addition, the estimates should be updated regularly to 

reflect material changes in the program, such as when schedules or 

other assumptions change, and actual costs so that the estimate is 

always reflecting the program’s current status. Among other things, 

the estimate should be grounded in documented assumptions and 

a historical record of cost estimating and actual experiences on 

other comparable programs. 

Step 7: Develop the point estimate

Step 12: Update the estimate to reflect 

actual costs and changes 
 

Page 11 GAO-12-223 Air Traffic Control Modernization

Characteristic Explanation Step identified in Cost Guide

Credible The cost estimates should discuss any limitations of the analysis 

because of uncertainty or biases surrounding data or assumptions. 

Major assumptions should be varied, and other outcomes 

recomputed to determine how sensitive they are to changes in the 

assumptions. Risk and uncertainty analysis should be performed to 

determine the level of risk associated with the estimate. 

Furthermore, the estimate’s results should be crosschecked, and 

an independent cost estimate conducted by a group outside the 

acquiring organization should be developed to determine whether 

other estimating methods produce similar results. For management 

to make good decisions, the program estimate must reflect the 

degree of uncertainty, so that a level of confidence can be given 

about the estimate. Having a range of costs around a point estimate 

is more useful to decision makers because it conveys the level of 

confidence in achieving the most likely cost and also informs them 

on cost, schedule, and technical risks. 

Step 7: Compare the point estimate to an 

independent cost estimate

Step 8: Conduct sensitivity analysis

Step 9: Conduct risk and uncertainty 

analysis 

Source: GAO.

The success of a program also depends in part on having an integrated 

and reliable schedule, which defines, among other things, when work 

activities will occur, how long they will take, and how they are related to 

one another. As such, the program schedule not only provides a road 

map for systematic program execution, but also provides the means by 

which to gauge progress, identify and address potential problems, and 

promote accountability. Accordingly, a schedule helps ensure that all 

stakeholders understand both the dates for major milestones and the 

activities that drive the schedule. If changes occur within a program, the 

schedule helps decision makers analyze how those changes affect the 

program. We have previously identified nine best practices that help 

ensure a reliable program schedule (these best practices are discussed 

later in this report). The reliability of the schedule will determine the 

credibility of the program’s forecasted dates, which are used for decision 

making. FAA is currently developing an integrated master schedule17 for 

the NextGen initiative that will be built in part on individual program 

schedules. The NextGen integrated master schedule is intended to be a 

comprehensive framework to support planned and actual work, providing 

decision makers with the information needed to manage the overall effort 

effectively.

 

17An integrated master schedule is required at the acquisition program level in order to 

meet our best practices, which we discuss later in this report.
 

Page 12 GAO-12-223 Air Traffic Control Modernization

Of the 30 baselined FAA ATC programs we reviewed,19 have not 

increased in cost, but 11 have experienced cost increases ranging from 

$2 million to over $2 billion. And of the 19 programs whose costs have not 

increased, 7 experienced a cost decrease while the remainder have not 

changed significantly.18

The 3 programs with the largest cost increases—totaling more than 

$4 billion—are key to ATC modernization. Several factors contributed to

cost overruns for the Standard Terminal Automation Replacement 

System (STARS), WAAS, and ERAM programs and required additional 

congressional appropriations or reductions in program scope.

However, the 11 programs that exceeded their 

initial estimated costs account for over 60 percent of total program costs 

for the 30 baselined programs—$11 billion of $17.7 billion. These 11 

programs are among the most complex of FAA’s major acquisitions in 

that each involves a large amount of software engineering. (See table 3.)

• Our previous work disclosed that the near tripling of the STARS’s 

budget resulted from insufficient involvement of stakeholders and 

requirements growth.19

 

18Some programs had neither cost increases not schedule delays, including: En route 

Communication Gateway, Integrated Display System, Regulation and Certification 

Infrastructure for System Safety, Time Based Flow Manager, and Weather and Radar 

Program. 

19GAO-08-42. 

Most Ongoing ATC 

Programs Remain 

within Budget and on 

Schedule, but a Few 

Have Significantly 

Exceeded Initial 

Estimates

Most Cost Estimates 

Remain on Target, but Cost 

Overruns for Three Key 

Programs in Total 

Exceeded $4 Billion
 

Page 13 GAO-12-223 Air Traffic Control Modernization

• The WAAS program began in 1998 with an initial cost estimate of $1 

billion20 and a current estimate of $3 billion. We reported previously 

that FAA’s lack of scientific and technical expertise resulted in 

unplanned work and contributed to cost increases as well as delays in 

the deployment schedule. Additionally, FAA changed how it 

accounted for certain costs in the capital budget in 1999,21

• As previously mentioned, ERAM is a key modernization system and 

will be the backbone of the NextGen system. FAA originally submitted 

to Congress an estimated cost of $2.1 billion in 2003, and the 

program is now expected to cost about $2.4 billion––an increase of 

about $330 million. According to FAA, various software issues (e.g., 

unsuccessful transmission messages and inaccurate data pairing of 

aircraft and traffic display), as well as problems interfacing with other 

facilities and systems, have contributed to the cost increases and 

delays. The extent to which unanticipated requirements, unplanned 

work, and underestimates of the complexity of software development, 

among other factors, have contributed to other FAA ATC acquisition 

program cost overruns and scheduling delays is discussed later in this 

report.

which 

further raised the cost estimate to $3.3 billion. FAA recently revised 

that estimate down to the current $3 billion during the 2009 

rebaselining because, according to FAA officials, they had met certain 

program requirements in 2006.

 

20In GAO-05-331, we noted that the original estimate for WAAS was $509 million for a 

terminated contract with Wilcox Electric. Subsequent estimates and baselines consider 

only the interim contract awarded to Hughes Aircraft in 1996. 

21FAA changed how it accounted for the costs of satellite leases from the operations 

account to the facilities and equipment account.
 

Page 14 GAO-12-223 Air Traffic Control Modernization

Table 3: Eleven ATC Programs That Have Experienced Cost Increases as of August 2011

Dollars in millions

Program

Original 

start date

Initial 

estimate

Current 

estimate

Cost 

increase

Percentage 

increases

Wide Area Augmentation Systema 1998 $1,001 $3,008 $2,007 199% 

Standard Terminal Automation Replacement System 1996 $940 $2,719 $1,779 189% 

En route Automation Modernization 2003 $2,154 $2,484 $330 15% 

Automated Dependant Surveillance—Broadcastc 2007 $1,682 $1,726 $44 3% 

Aviation Surface Observation Weather Networkd 2001 $351 $384 $33 10% 

Runway Status Lights 2010 $327 $352 $25 8% 

UHF Replacement 2002 $85 $93 $8 9% 

International Flight Inspection Aircraftb 2003 $27 $34 $6 23% 

Integrated Terminal Weather Systemb 1997 $276 $282 $6 2% 

Terminal Doppler Weather Radar 2003 $75 $77 $2 3% 

Tower Training Simulators 2007 $34 $36 $2 7% 

Total $6,952 $11,195 $4,243

Source: GAO analysis of FAA data.

a

According to FAA, the WAAS program is currently divided into three phases—two have been 

completed and the third is projected to be completed by 2013. However, in the CIP for fiscal years 

2012 through 2016, the WAAS program is listed as a single program costing $3 billion.

b

Program completed during GAO review.

c

According to the FAA, the ADS-B current revised budget includes congressional directed spending 

of $9.4 million in fiscal year 2008 and $6.8 million in fiscal year 2009. 

d

This program includes the ongoing pre planned product improvement segment.
 

Page 15 GAO-12-223 Air Traffic Control Modernization

We found that 15 of the 30 baselined programs either have experienced 

no change in schedule or were completed early or on time; however, the 

other 15 programs are projected to be completed later than originally 

estimated. These delayed programs range from the Integrated Display 

System, which will consolidate information from several weather 

subsystems into a single display, which FAA expects to be completed 2 

months after its initial estimated completion date, to WAAS, which FAA 

estimates will be completed in 2013—more than 14 years after its initial 

estimated completion date (see table 4). Ten of the 15 programs with 

schedule delays also experienced cost increases. However, even if a 

schedule delay does not result in a direct cost increase to that program, 

the delay can lead to increased costs for FAA because FAA staff must 

continue to manage the acquisition over the longer term as it is being 

implemented, as well as maintain any legacy system that the program is 

replacing. Because of program interdependencies, a schedule delay can 

also affect how and when other programs will be implemented.

Schedules for 

Roughly Half of ATC 

Programs Are on 

Track, but Half of ATC 

Programs Are 

Delayed 

Half of ATC Programs Are 

on or Ahead of Schedule 

and Half Are Delayed
 

Page 16 GAO-12-223 Air Traffic Control Modernization

Table 4: Fifteen ATC Programs That Have Experienced Delays Based on Original Scheduled Completion Date

Program Original start date

Original planned 

completion date

Projected 

completion date

Delay or projected 

delay (in months)

Wide Area Augmentation Systema January 1998 August 1999 September 2013 169

Air Traffic Control Beacon Interrogator-6 August 1997 September 2004 January 2012 88

Integrated Terminal Weather Systemb June 1997 July 2003 Aug 2010 85

Next Generation Air/Ground 

Communication System Segment 1

September 1998 September 2008 September 2013 60

Airport Surveillance Radar- 11d November 1997 September 2005 June 2010 57

Terminal Doppler Weather Radar February 2003 December 2013 September 2017 4 

En route Automation Modernization June 2003 December 2010 August 2014 44

Aviation Surface Weather Observation 

Networkc August 2001 September 2009 September 2012 36

UHF Replacement November 2002 September 2010 September 2013 36

International Flight Inspection Aircraft December 2003 August 2009 May 2012 33

Voice Switching and Control System 

(Tech Refresh) Phase 2

August 2006 June 2012 December 2014 30

Standard Terminal Automation 

Replacement System

February 1996 October 2005 June 2007 20

Tower Training Simulatorb December 2007 September 2009 August 2010 11

Runway Status Lights January 2010 October 2015 June 2016 8 

Integrated Display System Replacement September 2008 October 2015 December 2015 2 

Source: GAO analysis of FAA data.

a

According to FAA, the WAAS program is divided into three phases—two have been completed and 

the third is projected to be completed by 2013. However, in the CIP for fiscal years 2012 through 

2016, the WAAS program is listed as a single program costing $3 billion.

b

Program completed during GAO review.

c

Program includes the ongoing pre planned product improvement segment. 

d

The ASR-11 Program was re-baselined in September 2005 with a planned completion of September 

2009.
 

Page 17 GAO-12-223 Air Traffic Control Modernization

Cost increases and schedule delays occurred because of several factors, 

all of which have been long-standing challenges for FAA and some of 

which continue to affect programs despite FAA efforts to mitigate the 

factors. Specifically, these factors include (1) additional, unanticipated 

system requirements work; (2) insufficient stakeholder involvement 

throughout system development; (3) underestimates of the complexity of 

software development; and (4) unanticipated events, including funding 

decreases or work stoppages (see table 5). Of the 30 programs we 

reviewed, 15 experienced cost increases, schedule delays, or both,22 and 

we were able to determine that cost increases or schedule delays for 11 

were attributable to one or more of these factors.23

 

22Ten programs experienced cost delays and schedule delays, and five programs only 

experienced schedule delays.

23We did not categorize, or have available information for, the remaining four programs. 

Several Factors 

Contributed to Cost 

Increases and 

Schedule Delays, and 

Some Could Hamper 

NextGen 

Implementation 

A Number of Factors Have 

Contributed to Cost 

Increases and Delays
 

Page 18 GAO-12-223 Air Traffic Control Modernization

Table 5: Key Factors Contributing to Cost Growth, Schedule Delays, or Both for ATC Systems 

Name of ATC system

Unanticipated 

requirements or 

unplanned work

Insufficient 

stakeholder 

involvement

Underestimating 

the complexity of 

software 

development

Unanticipated 

events

Airport Surveillance Radar-11  

EnRoute Automation Modernization   

Next Generation Air-Ground Communications  

Runway Status Lights 

Voice Switching and Control System- Tech 

Refresh Phase 2  

Air Traffic Control Beacon Intergrorator-6  

Integrated Weather System   

International Flight Inspection Aircraft 

Wide Area Augmentation System    

Standard Terminal Automation Replacement 

System  

UHF Radio Replacement 

Source: GAO analysis of FAA data.

Following are some examples of how these contributing factors led to 

cost increases or schedule delays in some of FAA’s ATC baselined 

programs:

• Unanticipated requirements or work: For nine of the programs in table 

5, FAA has had to undertake substantially more development work than 

planned because FAA program officials originally misjudged the extent 

to which commercial off-the-shelf nondevelopmental solutions, such as 

those procured by another agency, would meet FAA’s needs. For 

example, although WAAS was being developed by an integrated 

product team that included representatives from several FAA offices, 

the team did not effectively resolve problems in meeting a required 

performance capability—that pilots be warned in a timely manner when 

a system may be providing them with potentially misleading and 

possibly hazardous information. These actions resulted in unanticipated 

work and contributed to the rise in WAAS’s cost from the original 

estimate of $509 million in 1994 to about $2 billion in 2005.
 

Page 19 GAO-12-223 Air Traffic Control Modernization

• Insufficient stakeholder involvement: As we previously reported,24

ERAM was designed at a time when air traffic controllers did not 

participate in efforts to design and test new systems. Because active 

users of the system from different locations could not provide insight 

early on, issues that could have been addressed early in the design 

phase were not addressed. In response, FAA has taken steps to 

improve the testing of new systems in order to reduce the likelihood of 

larger-than-anticipated software issues arising during system 

implementation. For example, FAA and the controllers’ union recently 

entered into a memorandum of understanding to bring controllers into 

the testing and evaluation phase of ERAM.25 Under this agreement, 

the controllers’ union will have ERAM technical, evaluation, and 

training representatives, as well as a team of 16 controllers (including 

12 from en route facilities and 4 from terminal facilities), who will be 

detailed to test and validate software fixes with contractor engineers 

at the FAA Technical Center (Tech Center). In addition, our previous 

work disclosed that the near tripling of the Standard Terminal 

Automation Replacement System’s budget resulted from insufficient 

involvement of stakeholders and requirements growth—two systemic 

factors that we found led to acquisitions missing their budget and 

schedule targets. This, in turn, contributed to cost growth, schedule 

delays, and eventually a reduction in the number of systems to be 

deployed.26

• Underestimates of the complexity of software development: This 

factor contributed to cost increases and schedule delays for ERAM, 

as well as issues with costs, scheduling, or both for two other 

programs. In 2010, FAA tested ERAM at two key sites (the Seattle 

and Salt Lake en route centers) on live air traffic, usually late at night 

when air traffic volume was low. During this testing, FAA encountered 

both anticipated and unanticipated software issues, which prompted 

the test sites, at times, to revert to using FAA’s legacy en route 

computer system. Specifically, software instructions to a controller in 

one sector to hand off control of an aircraft to a controller in an 

 

24GAO, Next Generation Air Transportation System: FAA’s Metrics Can Be Used to Report 

on Status of Individual Programs, but Not of Overall NextGen Implementation or 

Outcomes, GAO-10-629 (Washington, D.C.: July 27, 2010).

25Under the new contract between FAA and the controllers union, the union is to have full 

participation in the development and implementation of air traffic modernization systems. 

26GAO-08-42. 
 

Page 20 GAO-12-223 Air Traffic Control Modernization

adjacent sector failed, and flight data were lost or reassigned to 

another flight. While some testing at FAA’s Tech Center preceded 

testing at the two key sites, the Tech Center could only test limited 

scenarios, and none of the scenario testing identified this software 

error. In addition, as discussed earlier, ERAM was designed during a 

time when air traffic controllers did not participate in efforts to design 

and initially test new systems. FAA anticipated the potential for 

software issues at the outset of the program and initially scheduled 

approximately 6 to 9 months of contingency time between the time it 

achieved initial operating capability27 and operational readiness 

demonstrations at these sites, leaving little buffer for any potential 

delays. FAA worked with its contractor to correct a number of software 

issues, but further testing on live air traffic at the two test sites 

continued to produce critical safety errors. As a result, in March 2010, 

FAA decided, with the support of the air traffic controllers’ union, to 

halt all ERAM testing on live traffic and to revise the deployment 

schedule. The program was rebaselined in June 2011, and the 

program’s completion date was extended from December 2010 to 

August 2014. As a result of the schedule delays, the rebaselined cost 

estimate increased from $ 2.1 billion to $2.4 billion.28

• Unanticipated events: Unanticipated events at implementation sites 

and unanticipated funding issues have delayed several programs’ 

schedules and increased costs. For example, Airport Surveillance 

Radar-11 was originally scheduled to be completed in June 2009 but 

was delayed to June 2010. FAA reports indicated that the delay was 

due to an unusually protracted real estate acquisition at one site and 

issues involving validating performance during seasonal radar 

operations from other another site. Similarly, FAA’s Runway Status 

Lights program—which involves installing airport lighting equipment 

that visually signals to pilots when it is unsafe to enter, cross, or begin 

takeoff on a runway—has experienced schedule delays because of 

construction issues at five sites (Charlotte, North Carolina; Fort 

Lauderdale, Florida; Las Vegas; Minneapolis; and WashingtonDulles). FAA officials attributed some of these delays to the furlough 

 

27Initial operating capability is the declaration by site personnel that the system is ready for 

conditional operational use in the NAS.

28Recent reports from the Department of Transportation Inspector General and MITRE (a 

not-for profit organization chartered to work in the public interest) indicate that these cost 

and schedule estimates may be underestimated.
 

Page 21 GAO-12-223 Air Traffic Control Modernization

of some FAA employees in July 2011 and a freeze on contractor 

funding during the furlough, which resulted in work stoppage orders 

for several projects—including Runway Status Lights. FAA program 

managers will need to assess the impact of the furlough on other 

programs that had experienced work stoppage orders, including ADSB, the Standard Terminal Automation Replacement System, SWIM, 

WAAS, and various weather programs.29

The interdependencies of ATC acquisition programs have become more 

prominent as the NextGen program shifts from planning to 

implementation, so that cost increases and schedule delays in one 

program could have a cascading effect on other programs. As discussed 

earlier, due to the integrated nature of NextGen, the development and 

delivery of many of its component programs are mutually dependent on 

the development and delivery of one or more other programs. For 

example, ERAM, FAA’s new en route computer system, is critical to the 

delivery of ADS-B capabilities such as broadcasting flight information.

ERAM is also pivotal to the on-time implementation of two other key 

NextGen programs—Data Communications (DataComm),30

As we reported in 2010,

which is 

estimated to cost about $3 billion, and the NextGen information 

technology architecture, SWIM, which is estimated to cost over $550 

million. Due in part to ERAM’s delay, FAA was forced to delay the Data 

Communications baseline date by approximately 6 months, rebaseline 

SWIM segment 1, and delay the SWIM segment 2 baseline date to 2012. 

The longer-term effects of these delays are unclear, but certain SWIM 

capabilities could be delayed for several years, and the progress of other 

programs that are dependent on SWIM’s system integration could be 

hindered, as well. Thus, looking more broadly, the implementation of 

NextGen—both midterm (now through 2018) and far-term (2019-2025) 

schedules—will be affected by how well FAA manages program 

interdependencies.

31

 

29

individual FAA program offices understand their 

programs’ dependence on ERAM’s implementation, but FAA has not 

http://www.faa.gov/news/media/workstop/. 

30DataComm is planned provide capabilities for pilots and controllers to transmit digital 

messages and will eventually replace the analog voice communication system currently in 

use.

31GAO-10-629. 

Interdependencies among 

NextGen ATC Programs 

Could Slow NextGen 

Implementation 
 

Page 22 GAO-12-223 Air Traffic Control Modernization

developed a full listing of how ERAM schedule slippages, or slippages in 

other programs that are critical to NextGen, could either affect other 

programs’ implementation schedules or delay the implementation of 

capabilities and improvements.32 In 2008, we recommended that FAA 

improve the usefulness of ATO’s acquisition performance reporting by 

including information in the agency’s Performance and Accountability 

report or elsewhere on the potential effect of any budget or schedule 

slippages on the overall transition to NextGen.33 This recommendation 

remains open, as FAA has not definitively indicated how it will track 

slippages that will affect other dependent NextGen programs. FAA’s 

acquisition management system was not designed for managing NextGen 

programs in an integrated way. To assist in managing NextGen portfolios, 

FAA is starting to monitor all the activities of a particular operational 

improvement to ensure integration is on track. As we noted in the 2010 

report, as this approach is more fully implemented, it will likely clarify the 

impact of slippages in one program’s schedule on the implementation 

status of other NextGen programs and operational capabilities. In 

addition, as we will discuss in the next section, FAA is developing an 

Integrated Master Schedule for the entire NextGen initiative that is, in 

part, intended to show how changes in program schedules affect other 

programs and the timelines for the NextGen initiative as a whole.

However, as we discuss later, the schedules for the four programs we 

reviewed in detail are not reliable. Reliable schedules at the program level 

will be needed to develop a reliable Integrated Master Schedule for 

NextGen.

According to FAA, it is taking actions to address the factors that have 

contributed to cost increases and schedule delays. In 2011, FAA

assessed the NextGen effort as part of its Foundation for Success 

initiative34

 

32FAA’s Enterprise Architecture for the national airspace system shows the 

interdependencies and capabilities that may be affected by various programs, but this 

document cannot indicate specific milestones that could be affected.

and has implemented the “Idea to In-Service Management” 

(I2I) process, which it believes will result in improvements in the way the 

33GAO-08-42. 

34The Foundation for Success initiative is aimed at improving certain governance, shared 

services, human capital, and NextGen management structures to better manage FAA 

functions. 

Program Management 

Changes Are Aimed at 

Addressing Past 

Challenges
 

Page 23 GAO-12-223 Air Traffic Control Modernization

FAA develops, acquires, and implements new NextGen capabilities from 

conception through implementation. The I2I concept is intended to 

improve collaboration early in the acquisition process, resulting in better 

defined capabilities and an early indication of cost and benefits. These 

enhancements are intended to resolve many of the challenges associated 

with overall program management and enable FAA to focus on program 

management best practices. FAA believes that I2I will also result in 

improvements in specific areas that have presented challenges in the 

past, such as cost estimating, anticipating requirements and work, 

stakeholder collaboration, software development, and systems 

integration.

Also in 2011, FAA implemented a reorganization of the NextGen 

Operations and Planning Office and ATO which FAA believes will support 

the I2I process and improve acquisitions of NextGen programs.

Specifically, FAA created a NAS Lifecycle Planning Division within the 

NextGen Operations and Planning Office to focus the integration of 

NextGen programs from a cost, schedule, and systems capability 

perspective. Within ATO, FAA established a new Program Management 

Office, which puts the responsibility for the program management of all 

NextGen and other major ATC acquisitions within a single organization. 

By combining program managers into one organization, FAA hopes to 

create a stronger acquisition program and improve the consistency and 

implementation of best practices. According to FAA, these organizational 

changes allow responsibilities for acquisitions to be better defined to more 

efficiently set strategic direction, define operational requirements, ensure 

system integration, oversee implementation processes, and ensure 

accountability throughout the acquisition life cycle.

To improve the acquisitions management process, FAA has also divided 

large acquisition programs into segments. A segmented or phased 

approach is being taken with programs like SWIM and CATMT. This 

approach breaks a larger program into smaller and more manageable 

pieces to lower the risk. In the past, we noted that this approach can 

improve management by providing for midcourse corrections and, thus, 

help FAA avoid costly late-stage changes. However, this approach can 

also increase the duration and possibly the cost of the program.35

 

35GAO, Air Traffic Control: FAA’s Modernization Efforts Past, Present, and Future, 

GAO-04-227T (Washington, D.C.: Oct. 30, 2003).
 

Page 24 GAO-12-223 Air Traffic Control Modernization

According to FAA officials, a segmented approach allows the agency to 

more effectively manage acquisitions at both the program and enterprise 

architecture level. An enterprise architecture approach provides the 

structure to relate organizational mission, vision, and goals to business 

processes and the technical or information technology infrastructure 

required to execute them. FAA officials stated that many of the factors we 

identified that contributed to cost increases and schedule delays highlight 

the need for an enterprise-level perspective throughout the acquisition 

process. The I2I process is intended to provide an enterprise-level focus 

and improve collaboration across related programs.

Our review of the ADS-B, CATMT, SWIM, and WAAS cost estimates 

showed that while each program followed at least some of the four 

characteristics of high-quality and reliable cost estimates—welldocumented, comprehensive, accurate, and credible—none of the 

programs adhered closely enough to those characteristics to create a 

reliable cost estimate. As previously noted, these characteristics 

incorporate the 12 steps consistently applied by cost-estimating 

organizations throughout the federal government and industry and 

considered best practices for developing cost estimates. The results of 

our review of the ADS-B, CATMT, SWIM, and WAAS cost estimates, 

which are summarized in table 6, show that they were most aligned with 

the characteristic of comprehensive cost estimates but need improvement 

in the other three areas, particularly with the characteristics of accurate 

and credible estimates. Imprecise estimates can result in Congress 

unnecessarily authorizing and appropriating millions of dollars for 

programs. As noted, in some cases, FAA, in order to stay within the 

original cost estimate, modified a program’s requirements and Congress 

For Programs We 

Reviewed, FAA Did 

Not Consistently Meet 

All Characteristics of 

Quality Cost 

Estimates or Schedule 

Best Practices

Selected Cost Estimates 

Were Generally 

Comprehensive and WellDocumented, but Accuracy 

and Credibility Need to Be 

Improved
 

Page 25 GAO-12-223 Air Traffic Control Modernization

had to appropriate more funds or reduce the scope to allow FAA to finish 

the program.

FAA could better ensure that the cost estimates for these four programs, 

as well as its other major acquisition programs, are reliable. Our work 

shows that an assessment of these cost estimates for these programs, as 

well as FAA’s other major acquisition programs, would allow FAA to 

better understand if its cost estimation guidelines and our characteristics 

of high-quality cost estimates are in fact being followed (See table 6).

Table 6: GAO Analysis of the Extent FAA Acquisition Cost Estimates Met the Characteristics of High-Quality and Reliable 

Cost Estimates

Characteristic Characteristic description ADS-B CATMT SWIM WAAS

Well-documented The cost estimate should be supported by detailed documentation that 

describes the purpose of the estimate, the program background and system 

description, the scope of the estimate, the ground rules and assumptions, 

all data sources, estimating methodology and rationale, and the results of 

the risk analysis. Moreover, this information should be captured in such a 

way that the data used to derive the estimate can be traced back to, and 

verified against, the sources.

◑ ◑ ◕ ◕

Comprehensive The cost estimates should include costs of the program over its full life 

cycle, provide a level of detail appropriate to ensure that cost elements are 

neither omitted nor double counted, and document all cost-influencing 

ground rules and assumptions.

◕ ◕ ◕ ◕

Accurate The cost estimate should be based on an assessment of most likely costs 

(adjusted for inflation), documented assumptions, historical cost estimates, 

and actual experiences on other comparable programs. Estimates should 

be cross-checked against an independent cost estimate for accuracy, 

double counting, and omissions. In addition, the estimate should be 

updated to reflect any changes.

◑ ◑ ◑ ◑

Credible The cost estimates should discuss any limitations of the analysis because 

of uncertainty, or biases surrounding data or assumptions. Risk and 

uncertainty analysis should be performed to determine the level of risk 

associated with the estimate. Further, the estimate’s results should be 

cross-checked against an independent estimate.

◑ ◔ ◔ ◑

Source: GAO analysis of FAA documents.

 Met

◕ Substantially met

◑ Partially met

◔ Minimally met

 Not met

Note: “Not met” means the program provided no evidence that satisfies any of the best practice 

criteria. “Minimally met” means the program provided evidence that satisfies a small portion of the 

criteria. “Partially met” means the program provided evidence that satisfies about half of the criteria. 

“Substantially met” means the program provided evidence that satisfies a large portion of the criteria. 

“Met” means the program provided evidence that completely satisfies the criteria.
 

Page 26 GAO-12-223 Air Traffic Control Modernization

Because the four programs were generally similar in the extent to which 

they met each of the four characteristics, the following discussion 

summarizes the strengths and weaknesses we found for each 

characteristic across the four programs. A more detailed discussion of our 

findings is contained in appendix IV.

• Well-documented. Two of the four cost estimates we analyzed 

substantially met the characteristic of being well-documented; the 

other two partially met this characteristic. A well-documented cost 

estimate is thoroughly documented, including identifying specific 

source data and their significance, detailing calculations and results, 

and explaining why particular cost estimating methods were chosen. 

In other words, sufficient documentation exists such that an unfamiliar 

analyst could recreate the cost estimate and arrive at the same 

results. For example, the SWIM estimate provided detailed 

documentation describing the program, in addition to the 

methodology, calculations, and quantities used to develop the 

estimate. However, none of the four estimates sufficiently captured 

the entire source data used, addressed its reliability, or described how 

various forms of data from disparate sources were normalized (i.e., 

the data were described in like terms). For example, the WAAS 

estimate was based, in part, on actual labor costs from a previous 

contract, but the program office provided no evidence that these data 

were evaluated for reliability or accuracy. Similarly, the CATMT 

estimate routinely relied on subject matter expertise as a source for 

assumptions, such as the cost of labor, but the estimate did not 

document the experts’ qualifications, background, underlying 

assumptions, or data sources. Moreover, we noted that three of the 

four estimates often substantially relied on expert opinion rather than 

on data. While expert opinion can be useful in the absence of data, it 

is subjective and generally should be used sparingly for cost 

estimates. Since data are the foundation of every cost estimate, data 

quality affects the overall quality of the estimate. In addition, because 

data are gathered from a variety of sources and take many different 

forms, normalization helps to improve consistency with other cost 

information and enable valid comparisons and projections.

• Comprehensive. All four cost estimates we analyzed substantially met 

the characteristic of being comprehensive. For an estimate to be 

comprehensive, it should include full life-cycle costs, completely 

define the program with sufficient detail, include cost elements that 

are traceable to the statement of work or objective to ensure they are 

neither omitted nor double counted, and document all cost-influencing 

ground rules and assumptions. We found that the ADS-B, CATMT, 
 

Page 27 GAO-12-223 Air Traffic Control Modernization

and SWIM cost estimates included all life-cycle costs, regardless of 

program phase or funding source, and the ADS-B and SWIM cost 

estimates completely defined the program with an appropriate level of 

detail. In particular, the ADS-B cost estimate included cost estimates 

for both government and contractor costs, and the WAAS cost 

estimate thoroughly defined the program and reflected the current 

schedule. The four estimates did not fully meet the comprehensive

characteristic because they lacked evidence that all cost influencing 

ground rules and assumptions were considered.

• Accurate. None of the four cost estimates met or substantially met the 

characteristic of being accurate. The estimates generally adjusted 

costs for inflation and contained few computation or mathematical 

mistakes, but they were not regularly updated to reflect schedule and 

requirement changes, did not provide evidence of documenting or 

reviewing differences between planned and actual costs, and were 

not based on historical cost data from comparable programs. For 

example, the ADS-B, CATMT, and SWIM cost estimates provided no 

evidence that they were updated to reflect program changes, such as 

schedule slippages or varying assumptions, and did not include the 

current actual costs of the program. Although the WAAS estimate 

included evidence that it was updated to reflect major changes in 

technical and program requirements, such as the four rebaselinings 

the program has undergone since its 1998 inception, it did not include 

evidence that estimated costs were replaced with actual costs as the 

program advanced. Cost estimates that are not regularly updated with 

current information cannot provide decision makers with accurate 

information that is necessary, for example, when new system 

requirements are called for under tight budget conditions. In addition, 

comparing planned and actual costs enables cost estimators to 

measure the accuracy of their estimates and refine their processes. In 

addition, none of the four programs more than minimally used 

historical data to develop their cost estimates. Had historical data 

been used, the estimators would have had additional insight into 

actual costs on programs that used similar technologies, which could 

be used, for example, to challenge overly optimistic assumptions and 

bring more realism to the cost estimate.

• Credible. None of the four cost estimates met or substantially met the 

characteristic of being credible, which includes obtaining an 

independent cost estimate from a group outside the acquiring 

organization, and cross-checking the major cost elements in that 

estimate against cost drivers identified through sensitivity and risk 

analyses. The ADS-B, CATMT, SWIM, and WAAS estimates lacked 
 

Page 28 GAO-12-223 Air Traffic Control Modernization

credibility largely because FAA did not obtain an independent cost 

estimate for any of the programs. In addition the CATMT, SWIM, and 

WAAS estimates provided little evidence that it conducted sensitivity 

or risk analyses. Instead, each program received independent cost 

reviews as part of the investment decision process—even though 

such reviews are not required by FAA policy. FAA stated that the 

Investment, Planning and Analysis (IP&A) Office in the FAA Finance 

Organization does not prepare independent estimates, but it is 

organizationally independent of the acquisition programs and 

conducts independent reviews of all cost estimates. However, an 

independent cost review is less rigorous than an independent cost 

estimate. According to our cost guide, an independent cost estimate is 

often more accurate because the estimating team is further removed 

from the program office and less prone to accept overly optimistic 

assumptions or be burdened by organizational bias.36 Other federal 

agencies, including the Department of Defense, require independent 

cost estimates. Had an independent cost estimate been completed, 

the estimating team and program team could have identified the major 

differences between their estimates, reconciled those differences 

where possible, and provided a synopsis of the two estimates and 

their differences to acquisition program management. In addition, 

without sensitivity and risk analyses, cost estimators cannot measure 

the effects of varying assumptions, and managers cannot determine, 

for example, the rational level of contingency reserves necessary to 

cover increased costs that may result from uncertainties such as 

unexpected design complexity, changes in requirements, or budget 

shortfalls––all of which FAA ATC programs, and in particular NextGen 

programs, have experienced in recent years. We found evidence that 

some level of risk analysis was conducted for ADS-B, CATMT, and 

SWIM, although the analysis was not sufficiently robust. For example, 

key cost drivers were not identified, and additional context about how 

the estimate could be affected by software design and development 

issues was not included.

 

36An independent cost estimate can be done by an independent cost estimating office 

within an organization or by an outside contractor.
 

Page 29 GAO-12-223 Air Traffic Control Modernization

We determined that the schedules for the four programs we reviewed are 

unreliable because none met or substantially met all nine of the best 

practices for developing a reliable schedule. (see table 7). For example, 

none of the schedules fully met best practices for capturing all activities in 

an integrated master schedule, identifying critical paths and reasonable 

float for all activities, or assigning resources to those activities. Moreover, 

none of the schedules had documentation that provided more than 

minimal evidence that they conducted a schedule risk analysis. As was 

the case with our review of cost estimates for the four programs, our work 

regarding the schedules for these programs shows that an assessment of 

the schedules, as well as schedules for FAA’s other major acquisition 

programs, would allow FAA to understand if the nine best practices for 

reliable schedules are being followed.

Table 7: Extent Program Schedules Met Best Practices

Selected Schedules Did 

Not Substantially Meet 

Most Best Practices and 

Are, Therefore, Unreliable

Best practice Description

ADS-B 

(FAA 

prepared)

CATMT

(contractor 

prepared)

SWIM 

(FAA 

prepared)

WAAS

(FAA 

prepared)

WAAS

(contractor 

prepared)

1. Capturing all 

activities

A schedule should reflect all 

activities defined in the program’s 

work breakdown structure and 

include all activities to be performed 

by the government and contractor.

◑ ◕ ◔ ◔ ◕

2. Sequencing all 

activities

The schedule should be planned so 

that all activities are logically 

sequenced in the order they are to 

be carried out.

◔ ◑ ◔ ◔ ◕

3. Assigning 

resources to all 

activities

The schedule should realistically 

reflect the resources (i.e., labor 

material and overhead) needed to 

do the work, whether all required 

resources will be available when 

needed, and whether any funding or 

time constraints exist. 

◕ ◕  ◔ ◑

4. Establishing 

the duration of 

all activities

The schedule should reflect how 

long each activity will take to 

execute. 

◕  ◑ ◑ ◕

5. Integrating 

schedule 

activities 

horizontally and 

vertically

The schedule should be horizontally 

and vertically integrated—that is, it 

should link already sequenced 

activities with outcomes while also 

delineating the relation of 

supporting tasks and subtasks to 

upper-level milestones. Such 

mapping among levels enables 

different groups to work to the same 

master schedule. 

◑ ◑ ◔ ◑ ◕
 

Page 30 GAO-12-223 Air Traffic Control Modernization

Source: GAO analysis of FAA documents.

 Met

◕ Substantially met

◑ Partially met

◔ Minimally met

 Not met

Note: “Not met” means the program provided no evidence that satisfies any of the best practice 

criteria. “Minimally met” means the program provided evidence that satisfies a small portion of the 

criteria. “Partially met” means the program provided evidence that satisfies about half of the criteria. 

“Substantially met” means the program provided evidence that satisfies a large portion of the criteria. 

“Met” means the program provided evidence that completely satisfies the criteria.

Because the scheduling best practices are interrelated in such a way that 

deficiencies in one best practice will cause deficiencies in the others, a 

schedule must meet or substantially meet all nine practices to be reliable. 

For example, preparing a schedule that is program-wide––including an 

integrated breakdown of the work to be performed by both the 

government and its contractors over the expected life of the program––is 

6. Establishing 

the critical path 

The schedule should identify the 

critical path, or those activities that, 

if delayed, will negatively impact the 

overall project completion date. The 

critical path enables analysis of the 

effect delays may have on the 

overall schedule.

 ◑ ◔ ◔ ◕

7. Identifying 

reasonable 

float 

The schedule should identify float—

the amount of time an activity can 

slip in the schedule before it affects 

other activities—so that flexibility in 

the schedule can be determined. As 

a general rule, activities along the 

critical path typically have the least 

amount of float. 

 ◔ ◔ ◑ ◕

8. Conducting a 

schedule risk 

analysis 

The schedule should include a 

schedule risk analysis that uses 

statistical techniques to predict the 

probability of meeting a completion 

date. A schedule risk analysis can 

help management identify and 

understand the most important risks 

and focus on mitigating them. 

◔  ◔  ◔

9. Updating the 

schedule using 

logic and 

durations to 

determine 

dates

The schedule should use realistic 

durations for activities and be 

monitored to determine when 

forecasted completion dates differ 

from the planned dates. This 

analysis can be used to assess 

whether schedule variances will 

affect future work. 

◕ ◕ ◑ ◑ ◑
 

Page 31 GAO-12-223 Air Traffic Control Modernization

a best practice. If the schedule does not capture all activities, then there 

will be uncertainty about whether activities are sequenced correctly or if 

the schedule properly reflects the resources needed to accomplish the 

work, which is also a best practice. Logic and durations (that is, the time it 

takes to complete a specific activity) should be used and maintained to 

ensure realistic start and completion dates and to reflect the true status of 

the project––a necessary condition for conducting follow-on schedule risk 

analyses. Moreover, if activities are not properly sequenced with logical 

links, it will not be certain if the critical path—which represents the chain 

of dependent activities with the longest total duration—is valid.

Collectively, the weaknesses in not fully meeting or substantially meeting 

all nine key practices increases the risk of schedule slippages and cost 

overruns since a well-defined schedule helps to identify the amount of 

human capital and fiscal resources that are needed to execute the 

program. Therefore, by not having reliable schedules, FAA cannot 

conduct meaningful oversight of an acquisition program’s progress or 

determine whether the program is achieving the desired results.

The following discussion summarizes the extent to which the schedules 

for the four programs we examined met best practices. More detailed 

information for each program regarding scheduling best practices is 

presented in appendix V.

We reviewed the schedule prepared by FAA and found it did not fully 

meet any of GAO’s nine scheduling best practices, resulting in an 

unreliable schedule. Evidence provided in the ADS-B schedule indicates 

that it substantially met three of the nine best practices and partially, 

minimally, or did not meet the other six. For example, although the ADS-B 

schedule provided evidence of periodic updating, it did not capture all of 

the effort currently called for in the approved baseline for the entire ADSB program and, therefore, was not a fully integrated schedule. Without 

fully integrating government activities with contractor activities, and 

thereby capturing all key activities, the schedule will not reliably estimate 

the program’s completion. In addition, the ADS-B schedule we reviewed 

did not identify critical paths or include a schedule risk analysis, which 

uses statistical techniques to predict a level of confidence in meeting a 

program’s completion date; did not logically sequence all activities and 

establish their durations; and had excessive float37

 

37The amount of time by which a predecessor activity can slip before the delay affects 

successor activities.

on a majority of 

ADS-B
 

Page 32 GAO-12-223 Air Traffic Control Modernization

current and planned activities. According to program officials, a number of 

the issues our analysis identified were, in part, the result of the schedule’s 

limited time frame, which covered only a defined transitional period 

(October 2010 through April 2011) during which responsibility for about a 

third of the effort passed from the FAA to its prime contractor. Officials 

also stated that although their schedule contains critical activities, it has

not had a traditional critical path since its contractor began managing the 

deployment of deliverables. The FAA uses contract options to order the 

scope, sequence, and requirements for key milestones. Within those 

options, the contractor has the authority to implement the sequence of 

more discrete activities in the order they deem most appropriate. FAA 

program officials plan to rectify this problem, noting that with negotiations 

now completed, they will in the near future identify a critical path to span 

all program milestones.

Because the CATMT program did not prepare an FAA schedule and 

instead relied on its contractor schedule, we reviewed the contractor 

schedule, which we found to be unreliable. Our analysis found that the 

contractor’s CATMT schedule substantially or fully met four of the nine 

best practices: capturing all activities, assigning resources, establishing 

the durations, and updating the schedule. For example, the CATMT 

contactor schedule pertains to the current phase of the program that is 

being implemented in software releases, or phases. However, there was 

no overarching FAA government owned schedule that accounts for all 

software releases for the entire program and would thus delineate the 

relation of current software release tasks to the upper-level milestones for 

the overall CATMT program. The CATMT schedule included detailed 

resource information, and the program office provided evidence that 

resources are tracked in detailed labor-hour spreadsheets. We also found 

that 90 percent of the activities were of short duration and that the 

program office regularly reviews the schedule, which is in line with best 

practices. On the other hand, five of the nine best practices were either 

partially, minimally, or not met. Specifically, the CATMT schedule lacked 

evidence indicating that it established a critical path, accurately identified 

float between activities, integrated the schedule vertically and 

horizontally, sequenced all activities, or performed a schedule risk 

analysis. Regarding the critical path, our analysis determined that the 

CATMT schedule does not identify a critical path for the entire program. 

CATMT
 

Page 33 GAO-12-223 Air Traffic Control Modernization

Instead, the program is being accomplished multiple 6-month spirals38; 

thus, there is only a critical path for each software release, not for the 

program as a whole.

39 Without a valid program-wide critical path FAA 

management cannot determine which tasks, if they slip, will have the 

most detrimental effects on the project finish date. We also found that 68 

percent of the remaining activities to be completed had unreasonably high 

float exceeding 1,000 days, meaning that those activities could slip about 

5 work years40 without affecting the overall project finish date, a highly 

unlikely scenario.

41

Because the SWIM schedule did not fully or substantially meet any of 

GAO’s nine scheduling best practices, we found it to be unreliable. The 

SWIM program differs from the others in that it is an aggregation of 

NextGen acquisition programs, each developing an aspect of the SWIM 

information sharing capability. Because SWIM program managers are 

reliant on schedule information from a number of other programs, SWIM 

schedule integration is particularly important. However, our analysis found 

The accurate identification of critical paths and float 

are inextricably linked. For example, if the schedule is missing activities or 

they are not correctly linked, float estimates will be miscalculated, 

resulting in an invalid critical path. Without a schedule that can produce a 

true critical path, the program office will neither be positioned to provide 

reliable timeline estimates nor be able to identify when problems or 

changes may occur and determine the impact they may have on 

subsequent work. CATMT program officials acknowledged that the 

schedule did not include program-wide critical paths but noted that a 

critical path exist for individual segments of the program. They also noted 

that a schedule risk analysis was not performed because it was not a 

contractual deliverable.

 

38Software spiral development is an incremental approach to reduce risk, so that user 

needs and requirements are better defined.

39FAA officials said that because of the 6-month spiral development approach, the 

schedule cannot deliver a single critical path for the entire program. Instead, the critical 

paths are calculated and based by software releases. To calculate a critical path by each 

software release, the prime contractor uses an end constraint on the key deliverable 

milestone for each software release. For further explanation, see table 13 in appendix IV. 

40A work year is approximately 200 days since every organization works to a different 

calendar.

41These float values are due mostly to activities being tied to the project finish milestone, 

which is constrained to start no earlier than July 1, 2016. For further explanation, see table 

13 in appendix IV. 

SWIM
 

Page 34 GAO-12-223 Air Traffic Control Modernization

that the SWIM schedule was not, by any measure, fully integrated 

because it provided only a synopsis of the individual system implementing 

program schedules and, thus, did not fully represent the work required to 

complete the overall SWIM program. This resulted in float calculations 

that were unrealistic, and the resulting critical path calculations were 

invalid. In addition, while the many missing activities negatively impacted 

the schedule logic and the accuracy of durations, it also made the 

accurate allocation of resources and comprehensive integration of 

schedule activities, both horizontally and vertically, impossible. We also 

noted that FAA made no effort to identify a program-wide critical path. 

Program officials said that because each of the system implementing 

program schedules has its own critical path, involve disparate capabilities, 

and are independent of one another their individual critical paths are not 

accessible through the SWIM schedule software. They therefore are not 

used for overall SWIM program management. We believe that the SWIM 

program itself should have its own critical path that includes, at a 

minimum, acceptance of major deliverables from the system 

implementing program schedules. Without a program-wide critical path, 

management does not have a clear picture of the underlying project tasks

that must be performed to achieve the overall program target completion 

date. Finally, although there was no risk analysis conducted on this 

schedule, our analysis found that this best practice was minimally met 

because a risk analysis was conducted on a separate but related 

schedule, and the SWIM program office considered risk to some extent.

Like the other three programs, we found the WAAS program schedule 

prepared by FAA unreliable because it did not fully or substantially meet 

any of GAO’s nine scheduling best practices; however, we also reviewed 

the contractor’s schedule for the same segment and found it fully or 

substantially met six best practices.42

 

42We reviewed two schedules for the WAAS program: one produced by the FAA program 

office and the other produced by its prime contractor. We evaluated the contractor 

prepared schedule, which was the most current schedule available for the purposes of the 

schedule risk analysis, which is discussed more fully later in this report. 

For example, FAA’s WAAS program 

schedule did not fully sequence activities in the order in which they are to 

be carried out. More specifically, the WAAS program schedule showed 

nearly half of the remaining activities were missing sequenced logic, 

causing us to question the calculated dates of activities. Logic is 

necessary for a schedule to show program managers when activities are 

expected to start and finish; when logic is missing, activity dates cannot 

WAAS
 

Page 35 GAO-12-223 Air Traffic Control Modernization

adjust correctly to changes in activities. To test the ability of the schedule 

to dynamically update its dates due to changes, we artificially extended 

the duration of an activity to 1,500 days, which changed the activity’s 

finish date. However, the duration extension had no effect on successor 

activities because this activity is not tied to any successor activities.

Extending the duration to 1,500 days also pushed the project planned 

finish date from September 22, 2016, to June 29, 2017; however, 

because the logic links are not in place, we questioned whether the 

projected finish date under this scenario is reliable.

Moreover, the WAAS program schedule had too many artificial 

constraints that were driving the start and finish dates for more than 70 

percent of the remaining activities.43

While the schedule prepared by the contractor did not fully or 

substantially meet three of the scheduling best practices, it fully or 

substantially met six: capturing, sequencing, assigning resources to, and 

Constraints are usually substitutes 

for logic and can mean that the schedule is not well planned or feasible.

Constraints also greatly reduce the ability of the program to take 

advantage of possible time savings. Further, our analysis found that the 

schedule did not fully capture or assign resources to all government and 

contractor activities; it also did not accurately allocate resources or 

consistently establish the duration of activities. In addition, while WAAS 

program officials told us that the schedule was integrated vertically and 

horizontally, we did not find evidence of such integration. Furthermore, we 

found the WAAS program office’s schedule did not identify a critical path 

for the entire program. As noted earlier, critical path and float 

determinations are closely related. Our analysis of the WAAS program 

office schedule found that more than half of the remaining activities had 

float of more than 1,000 working days, which we believe to be 

unreasonably high. Without proper determination of float, management 

cannot properly reallocate resources from tasks to other tasks without 

adversely affecting the overall completion date. Although program officials 

said that they maintained a risk register listing the potential risks that 

could impact the schedule and adjusted the schedule for these risks, we 

did not find evidence that the program office had conducted a risk 

analysis of its schedule.

 

43A constraint predefines the start, finish, or both dates of an activity. The schedule should 

use logic and durations in order to reflect realistic start and completion dates for activities. 
 

Page 36 GAO-12-223 Air Traffic Control Modernization

establishing the duration of all activities; establishing the critical path; and 

identifying reasonable float between activities. For example, our analysis 

found that all activity durations are consistently estimated in days and 

adhere to a standard 5-day workweek that accounts for holidays, and no 

activities were scheduled to begin on a weekend. Officials from the 

contractor said duration estimates for the schedule are based on 

historical information from past performance, comparable releases, 

lessons learned, similar work, and other data requirements. In addition, 

our analysis traced several critical paths in the schedule. Though we 

found minor interruptions in the various critical paths, the schedule’s logic, 

reasonable durations, and low total float estimates allow the calculation a 

valid critical path.

As noted, FAA did not perform a complete schedule risk analysis for any 

of the four programs we reviewed and, thus, cannot accurately estimate 

these programs’ completion dates with confidence. A schedule risk 

analysis, which is one of our best practices for program scheduling, uses 

statistical techniques to predict a level of confidence in meeting a 

program’s completion date. The objective of the analysis is to develop a 

probability distribution of possible completion dates that reflect the project 

and its identified risks. This analysis can help program managers both 

understand the most important risks to the program and focus on 

mitigating those risks. Other federal agencies, including the Department 

of Defense and the National Aeronautics and Space Administration, 

require schedule risk analysis for major acquisitions; the Department of 

Veterans Affairs, in response to a GAO recommendation,44

We conducted a schedule risk analysis on the WAAS contractor prepared 

schedule, which we chose because it was relatively mature, it partially 

met or substantially met six of the nine best practices (see table 7), and it 

contained enough information to perform a schedule risk analysis.

plans to 

require schedule risk analysis for major construction projects.

45

 

44GAO, VA Construction: VA Is Working to Improve Initial Project Cost Estimates, but 

Should Analyze Cost and Schedule Risks, 

We 

reviewed the risk register that the contractor had developed, which 

GAO-10-189 (Washington, D.C.: Dec. 14, 

2009). 

45The other three schedules did not have the required information to conduct a schedule 

risk analysis.

WAAS Schedule Risk 

Analysis Indicates Risks 

That Managers Could 

Mitigate to Avoid Delays
 

Page 37 GAO-12-223 Air Traffic Control Modernization

showed four potential risks to the project. We then conducted interviews 

with FAA program and contractor staff and asked them to discuss other 

potential risks to the project, including how the risk would affect the 

project’s timeline and the likelihood of the risk occurring. Using this 

information we identified an additional 16 risks for a total of 20 risks. The 

fact that our interviews identified a relatively large number of new risks 

could be an indication that the contractor did not systematically analyze 

the full range of risks when developing the program’s risk register. We 

then consolidated the 20 risks into 14 broader risks and tested how each 

would impact the duration of specific activities in the schedule. We then 

ran a Monte Carlo simulation,46

We then analyzed the potential impact of risks on the program schedule. 

Since risks can effect the schedule in various ways––for example, risks 

can have a large impact on the durations of activities they affect, or they 

can introduce critical paths that are different from the baseline critical 

path–– we analyzed the marginal impact of each of the risks we identified 

to determine which would have the greatest effect on the overall 

schedule. We found the following three key risks to the program, only the 

first of which (limited WAAS program office resources) was originally 

identified by the contractor. The three risks are

which consisted of the computergenerated results of 3,000 estimates of the future schedule based on the 

activities in the schedule, the chance that some of the activities would be 

affected by some risks and the predicted effect of those risks on the 

duration of each activity.

• limited WAAS program office resources such as staffing;

• delays in software yet to be released and additional changes to 

software already released and in use; and

• a potentially optimistic schedule completion date. 

Our schedule risk analysis showed the completion of the segment of the 

WAAS program covered by the schedule could slip as much as 2 months.

Specifically, the analysis showed that there is less than a 5 percent 

probability that the program segment would be completed by September 

6, 2012, the current baselined date for completion. However, it appears 

 

46A Monte Carlo simulation involves the use of random numbers and probability 

distributions to examine potential outcomes.
 

Page 38 GAO-12-223 Air Traffic Control Modernization

that the segment will be completed close to the deadline since we found a 

50 percent probability that the program segment will be completed by 

October 23, 2012 (about 1.5 months after the current estimated date for 

completion); and an 80 percent probability that the program will be 

completed by November 13, 2012 (about 2.25 months after the current 

estimated date for completion).

Although we did not conduct a schedule risk analysis for other FAA 

programs, the result of our analysis provides examples of the types of 

risks that major acquisition programs face and the impact those risks can 

have on meeting acquisition program milestones, especially given the 

interrelation and interdependencies among NextGen acquisitions 

discussed earlier. More information on our schedule risk analysis can be 

found in appendix V.

FAA has begun developing an integrated master schedule for the entire 

NextGen initiative that would, in part, capture related NextGen program 

schedules, governance activities, and other performance and financial 

data to provide real-time monitoring of the overall NextGen effort. 

However, the unreliability of the four program schedules for programs that 

are integral to the NextGen initiative puts this high-level master schedule 

at risk. Having a reliable integrated master schedule would enable FAA to 

determine how delays in one program impact other programs and the 

overall NextGen implementation timeline. While it is encouraging that 

FAA is beginning to develop an integrated NextGen master schedule, the 

effort could be hampered by the lack of schedule integration at the 

program level, as well as the failure of individual program schedules to 

meet best practices. For example, since FAA does not perform schedule 

risk analysis on individual programs, it cannot predict with certainty if any 

of the programs will be completed on time. Therefore, the integrated 

master schedule for NextGen would be built on schedules that may not 

reflect accurate program completion dates. Similarly, none of the four 

schedules we reviewed, which were for segments of the entire program, 

had reflected how tasks for the segment affected milestones for the entire 

program. Without integrated schedules at the program level, an integrated 

master schedule at the NextGen initiative level would be problematic.

Lack of Reliable Program 

Schedules Will Hinder 

Development of an Overall 

NextGen Integrated Master 

Schedule 
 

Page 39 GAO-12-223 Air Traffic Control Modernization

In response to our review of the extent that the four selected acquisition 

programs met best practices for cost estimates and schedules, FAA 

provided information on steps it is taking to improve its processes for both 

cost estimates and schedules and noted that some of the cost estimates 

and schedules we reviewed were developed before the improvements 

were in place. FAA stated that strengthening its cost estimation process is 

part of the seven key acquisition processes it has developed, including 

program management, contractor management, requirements, risk 

management, measurement and analysis, verification and validation, and 

quality assurance. FAA stated that it has updated its Guidelines for FAA 

Cost Estimating to be consistent with the GAO Cost Guide, filling in gaps 

that it had identified during a comparison of its practices to those 

contained in the Cost Guide. As of November 2011, 11 of the 12 best 

practices are addressed in the guidelines. According to FAA officials, the 

remaining best practice—involving the creation of independent cost 

estimates—is unlikely to be implemented at FAA in the foreseeable future 

because FAA believes the resources required to create independent 

estimates are prohibitive in current budget environments. FAA has more 

than tripled the number of cost estimators in the Investment Planning and 

Analysis organization, many of which work with the acquisition program 

offices to provide guidance on preparing estimates. Additionally, as part 

of FAA’s effort to improve acquisition certification and training, the agency 

is preparing to launch a cost estimating certification program. Coupled 

with a competency-based training program, FAA believes the certification 

program will enhance and improve consistency of the skills of FAA cost 

estimators.

In describing its efforts to improve schedules, FAA stated that it views the 

development and maintenance of integrated schedules as an inherent 

and critical part of its seven key acquisition functions. FAA noted that 

included in its standard process for acquisition schedules are toolkits that 

require programs to develop integrated program schedules that address 

all nine of GAO’s best practices. FAA stated that that the current 

procedures for developing best practices were not fully in place when the 

four programs we reviewed began the implementation phase.

FAA has made improvements in its management of air traffic control 

modernization acquisitions, and most of the 30 we reviewed are currently 

within the original cost estimate and half are on schedule. FAA is also 

taking steps to address past issues to ensure cost estimates and 

schedules are more accurate in the future, including incorporating best 

practices in its acquisitions guidance and policies. Nevertheless, our 

FAA has Taken Steps to 

Improve Acquisition 

Program Cost Estimates 

and Schedules

Conclusions 
 

Page 40 GAO-12-223 Air Traffic Control Modernization

review of the FAA acquisitions found that it has yet to fully implement 

several GAO-identified best practices or follow others. Following best 

practices is particularly important for FAA, which must manage large, 

complex, and interdependent acquisitions associated with NextGen. Cost 

estimates that are imprecise can result in Congress appropriating millions 

of dollars for projects based on estimates that prove to be inaccurate, and 

program schedule delays can increase costs and affect the 

implementation of interdependent programs. In such cases, FAA will be 

forced to reduce the scope of the programs to stay within the original 

estimates or Congress will need to appropriate unanticipated funds to 

complete the programs. Delays and cost increases in individual programs 

could have a cascading effect on other programs and ultimately affect 

FAA’s timelines and goals for NextGen implementation.

Our analysis of the cost estimates and schedules for the four programs 

we reviewed indicates that FAA needs to further develop requirements for 

critical cost estimation and schedule procedures. Independent cost 

estimates can improve the accuracy and credibility of cost estimates and 

better ensure that programs will be completed within budget. A schedule 

risk analysis can help FAA determine the likelihood that a program will be 

completed on time. FAA stated that it has no immediate plans to conduct 

independent cost estimates due to current budgetary constraints. We 

recognize that conducting independent cost estimates and schedule risk 

analysis takes both financial resources and some time and that it may be 

appropriate to limit one or both of these analyses to instances where a 

program is particularly costly, complex, or on a compressed schedule.

However, conducting independent cost estimates, schedule risk analyses, 

and other analyses called for in our best practices can not only help 

minimize the risk of cost overruns and schedule delays, but also provide 

FAA, congressional decision makers, and other stakeholders with 

important information about these critical acquisitions.

It is also important that FAA develop master schedules at the individual 

acquisition program level. FAA’s lack of a fully integrated master schedule 

for the programs we reviewed hampers its ability to provide accurate 

information on the schedule for these programs. This information will be 

needed as FAA simultaneously works to develop an integrated master 

schedule for the overall NextGen initiative. The use of an integrated 

master schedule can assist FAA in monitoring a program, identifying 

problems that could affect later stages of the program’s implementation, 

improving the accuracy of cost estimates and schedules for individual 

programs, and improving the accuracy of information FAA is compiling to 

monitor the costs and schedules for the NextGen initiative.
 

Page 41 GAO-12-223 Air Traffic Control Modernization

FAA has incorporated 11 of our 12 steps that are associated with the 

characteristics of a high-quality and reliable cost estimate into their 

acquisition guidelines. However, our analysis of the four major programs 

indicates that FAA has not adequately integrated all of the steps for these 

programs into its cost estimation processes, and thus the estimates are 

not reliable. Similarly, although FAA addresses our nine scheduling best 

practices in its acquisition guidelines, our analysis of the schedules for the 

four programs indicates that the schedules are not adequately following 

these best practices and are not reliable. Although the cost estimates and 

schedules for some of the four programs were developed prior to FAA’s 

revision of acquisition guidelines, our work shows that FAA needs to 

assess its major acquisition programs to understand if its guidelines and 

other best practices are, in fact, being followed. Such an assessment 

would then allow FAA to better ensure that best practices for cost 

estimates and schedules are being applied.

To improve cost estimates and schedules for NextGen and other major 

air traffic control acquisition programs, GAO recommends that the 

Secretary of Transportation direct FAA to take the following three actions 

when appropriate for major acquisition programs based on a program’s 

cost, schedule, complexity, and risk:

• Conduct independent cost estimates and schedule risk analysis for 

major acquisition programs.

• Require a fully integrated master schedule for each major acquisition 

program, including those that are components of NextGen. An 

integrated master schedule should horizontally and vertically link all 

program activities and milestones, including government and 

contractor schedules and program segments.

• Conduct an assessment of major acquisition programs to ensure they 

meet all of the established best practices for cost estimates and 

schedules contained in GAO guidance.

Given constrained budgets, FAA should determine which programs 

should be subject to these recommendations, such as those that are 

particularly costly, complex, or on a compressed schedule.

Recommendations for 

Executive Action
 

Page 42 GAO-12-223 Air Traffic Control Modernization 

We provided a draft of this report to the Department of Transportation for 

review and comment. DOT and FAA responded by email and did not 

comment whether or not they agreed or disagreed with our 

recommendations. DOT provided comments on the results of our analysis 

of the cost estimates and schedules for the four programs we reviewed in 

depth. In response to our finding that the ADS-B, CATMT, SWIM, and 

WAAS estimates lacked credibility largely because FAA did not obtain an 

independent cost estimate for any of the cost estimates, and provided 

little evidence that they conducted sensitivity or risk analyses, FAA stated 

it is not convinced that an independent organization will reduce the 

uncertainty of cost estimates. FAA noted that it does not have an 

independent organization such as the Department of the Navy’s Center 

for Cost Analysis. However, FAA stated that the Finance Organization 

within ATO assessed the ADS-B program office’s Basis of Estimate as 

part of the JRC Decision and that this level of independence, combined 

with specific entry and exit criteria, allowed the program offices to 

manage these acquisitions so that costs were controlled, risks mitigated, 

and technical parameters achieved, while adhering to the planned 

milestone schedule. We agree that the Finance Organization assessment 

of the two cost estimates provided some degree of independence and 

may have improved the accuracy of the ADS-B estimates, but it is not 

clear that such an independent review will guarantee similar results for 

other programs. As we stated in the report, such an independent cost 

review is less rigorous than an independent cost estimate. According to 

our cost guide, an independent cost estimate is often more accurate 

because the estimating team is further removed from the program office 

and less prone to accept overly optimistic assumptions or be burdened by 

organizational bias. DOT also provided technical clarifications, which we 

incorporated into the report as appropriate. 

Agency Comments 
 

Page 43 GAO-12-223 Air Traffic Control Modernization

We are sending copies of this report to interested congressional 

committees, the Secretary of Transportation, and the Acting Administrator 

of FAA. In addition, the report is available at no charge on the GAO 

website at http://www.gao.gov. 

If you or your staff have any questions about this report, please contact 

me at (202) 512-2834 or dillinghamg@gao.gov. Contact points for our 

Offices of Congressional Relations and Public Affairs may be found on 

the last page of this report. GAO staff who made key contributions to this 

report are listed in appendix V.

Gerald L. Dillingham, Ph.D.

Director, Physical Infrastructure Issues
 

Page 44 GAO-12-223 Air Traffic Control Modernization

List of Committees

The Honorable Jay Rockefeller

Chairman

The Honorable Kay Bailey Hutchison

Ranking Member

Committee on Commerce, Science, and Transportation

United States Senate

The Honorable Ralph Hall

Chairman

The Honorable Eddie Bernice Johnson

Ranking Member

Committee on Science, Space, and Technology

House of Representatives

The Honorable Thomas Petri

Chairman

The Honorable Jerry Costello

Ranking Member

Subcommittee on Aviation

Committee on Transportation and Infrastructure

House of Representatives

 
Appendix I: Objectives, Scope and 

Methodology

Page 45 GAO-12-223 Air Traffic Control Modernization

In response to a congressional request, we examined the Federal 

Aviation Administration’s (FAA) ability to modernize, upgrade, and replace 

the National Airspace System’s (NAS) facilities and equipment to meet 

projected increases in traffic volumes, enhance the system’s safety, and 

increase the efficiency of the air traffic control (ATC) system—a principal 

component of the NAS. FAA’s ATC acquisitions are critical to maintaining 

the NAS and transitioning to the Next Generation Air Transportation 

System (NextGen) over the next 10 years. Given that some key legacy 

and NextGen acquisitions have experienced schedule delays and cost 

overruns, which may risk the timely implementation of NextGen, we (1) 

determined whether the planned costs and schedules of current FAA ATC 

acquisition programs have changed since they were first submitted to 

Congress; (2) examined the reasons for any changes in planned costs 

and schedules; and (3) assessed the extent to which select ATC 

programs adhered to best practices for determining acquisition costs and 

schedules.

To describe any changes in costs and schedules of the current 30 FAA 

capital ATC acquisitions, we gathered and analyzed agency data on the 

estimated cost and schedules of these ATC acquisitions.1 We drew upon 

past work in which we undertook detailed reviews of the status of ATC 

and other acquisition programs2 and obtained updated documentation as 

necessary from FAA. We interviewed FAA officials to obtain information 

on FAA’s acquisition process and summarized the status of all 

acquisitions, including FAA’s original and current cost estimates and 

completion dates. For baselined acquisitions,3

To determine the reasons for changes in cost estimates and schedules, 

we interviewed FAA officials and FAA contractors and reviewed 

we compared estimated 

costs when they were submitted to Congress for approval against their 

current estimates, and we analyzed planned and actual schedules.

 

1

We requested the information on the programs in August 2010.

2

GAO-10-588SP, GAO-10-388SP, GAO-08-42, and GAO-05-331. 

3

According to FAA, baselined acquisitions, as opposed to nonbaselined acquisitions, are 

an agreed-to description of the attributes of a product, at a point in time, that serves as a 

basis for defining change; an approved and released document, or a set of documents, 

each of a specific revision—the purpose of which is to provide a defined basis for 

managing change; the currently approved and released configuration documentation; or a 

released set of files consisting of a software version and associated configuration 

documentation.

Appendix I: Objectives, Scope and 

Methodology 
Appendix I: Objectives, Scope and 

Methodology

Page 46 GAO-12-223 Air Traffic Control Modernization

acquisition documentation. We analyzed information on cost increases 

and delays to determine if systematic issues exist that have effects on 

other FAA acquisitions.

To determine the extent to which select ATC programs adhered to best 

practices for determining acquisition costs and schedules, we conducted 

an in-depth review of 4 of the 30 acquisitions programs: The Automatic 

Dependent Surveillance-Broadcast (ADS-B) system, the Collaborative Air 

Traffic Management Technologies (CATMT) system, the System Wide 

Information Management (SWIM) system, and the Wide Area 

Augmentation System (WAAS). We selected these four acquisitions 

based on the following criteria: (1) existence of baselining, (2) the 

acquisition is at a point in the acquisition process where risks can be 

identified, and (3) the acquisition is key to NextGen and legacy systems.

In addition to interviews, we collected documentation, and we analyzed 

and summarized the views and information collected. We also identified 

best practices that FAA could adopt or strengthen to improve its 

acquisitions cost estimation and scheduling and ensured that acquisitions 

follow cost and schedule best practices outlined in our Cost Estimating 

and Assessment Guide.4

We used our Cost Estimating and Assessment Guide

We also performed a schedule risk analysis of 

the WAAS program to determine the likelihood of the project finishing on 

schedule.

5

Well-documented: The documentation should address the purpose of 

the estimate, the project background and system description, its 

schedule, the scope of the estimate (in terms of time and what is and is 

not included), the ground rules and assumptions, all data sources, the 

estimating methodology and rationale, the results of the risk analysis, and 

(Cost Guide) as a 

source of criteria for analyzing cost estimates. As noted earlier in our 

report, our Cost Guide identifies 12 steps consistently applied by costestimating organizations throughout the federal government and industry 

and considered best practices for developing cost estimates. For the 

purposes of this review, we grouped these steps into four characteristics 

of high-quality and reliable estimates—well-documented, comprehensive, 

accurate, and credible—which can be summarized as follows:

 

4

GAO-09-3SP.

5

GAO-09-3SP. 
Appendix I: Objectives, Scope and 

Methodology

Page 47 GAO-12-223 Air Traffic Control Modernization

a conclusion about whether the cost estimate is reasonable. Therefore, a 

good cost estimate—while taking the form of a single number—is 

supported by detailed documentation that describes how it was derived 

and how the expected funding will be spent in order to achieve a given 

objective. For example, the documentation should capture in writing such 

things as the source data used and their significance, the calculations 

performed and their results, and the rationale for choosing a particular 

estimating method or reference. Moreover, this information should be 

captured in such a way that the data used to derive the estimate can be 

traced back to and verified against their sources. Finally, the cost 

estimate should be reviewed and accepted by management to ensure 

there is a high level of confidence in the estimate and the estimating 

process.

Comprehensive: The cost estimates should include both government 

and contractor costs of the project over its full life cycle, from inception 

through design, development, deployment, operation, and maintenance to 

retirement of the project. The cost estimate should be structured in 

sufficient detail to ensure that cost elements are neither omitted nor 

double counted, and they should document all cost-influencing ground 

rules and assumptions.

Accurate: The cost estimates should provide for results that are 

unbiased, and they should not be overly conservative or optimistic. 

Estimates are accurate when they are based on an assessment of most 

likely costs, adjusted properly for inflation, and contain few, if any, minor 

mistakes. In addition, the estimates should be updated regularly to reflect 

material changes in the project, such as when schedules or other 

assumptions change so that the estimate is always reflecting the project’s 

current status. Among other things, the estimate should be grounded in 

documented assumptions and a historical record of cost estimating and 

actual experiences on other comparable projects.

Credible: The cost estimates should discuss any limitations of the 

analysis because of uncertainty or biases surrounding data or 

assumptions. Major assumptions should be varied, and other outcomes 

recomputed to determine how sensitive they are to changes in the 

assumptions. Risk and uncertainty analysis should be performed to 

determine the level of risk associated with the estimate. Furthermore, the 

estimate’s results should be cross-checked, and an independent cost 

estimate conducted by a group outside the acquiring organization should 

be developed to determine whether other estimating methods produce 

similar results.
Appendix I: Objectives, Scope and 

Methodology

Page 48 GAO-12-223 Air Traffic Control Modernization

After reviewing documentation submitted by FAA and information 

obtained during interviews, we determined the extent to which the cost 

estimates met the characteristics of cost-estimating best practices for the 

four projects we reviewed.

Our review of project schedules was based on research that identified a 

range of best practices associated with effective schedule estimating. In 

addition, we obtained the consulting services of David Hulett, Ph.D.,6 to 

assist in our risk analysis of the WAAS project schedule.7

Capturing all activities: The schedule should reflect all activities (steps, 

events, outcomes, and other factors) as defined in the project’s work 

breakdown structure, including activities to be performed by both the 

government and its contractors.

We also 

conducted multiple interviews with project managers, contractors, and 

schedulers to determine the extent to which current project schedules met 

the best practices criteria. These nine practices are:

Sequencing all activities: The schedule should be planned so that it can 

meet project-critical dates. To meet this objective, activities need to be 

logically sequenced in the order that they are to be carried out. In 

particular, activities that must finish prior to the start of other activities 

(i.e., predecessor activities) and activities that cannot begin until other 

activities are completed (i.e., successor activities) should be identified. 

Identifying interdependencies among activities that collectively lead to the 

accomplishment of events or milestones can be used as a basis for 

guiding work and measuring progress.

Assigning resources to all activities: The schedule should realistically 

reflect what resources (i.e., labor, material, and overhead) are needed to 

do the work, whether all required resources will be available when they 

are needed, and whether any funding or time constraints exist.

 

6

Hulett & Associates, LLC Los Angeles. Dr. Hulett is the author of “Practical Schedule 

Risk Analysis.” 

7

The WAAS contractor schedule was found reliable enough to conduct a schedule risk 

analysis. Because we found the other three schedules were unreliable, a schedule risk 

analysis could not be performed.
Appendix I: Objectives, Scope and 

Methodology

Page 49 GAO-12-223 Air Traffic Control Modernization

Establishing the duration of all activities: The schedule should reflect 

how long each activity will take to execute. In determining the duration of 

each activity, the same rationale, data, and assumptions used for cost 

estimating should be used for preparing the schedule. Furthermore, these 

durations should be as short as possible and should have specific start 

and end dates. Excessively long periods needed to execute an activity 

should prompt further decomposition of the activity so that shorter 

execution durations will result.

Integrating schedule activities horizontally and vertically: The 

schedule should be horizontally integrated, meaning that it should link the 

products and outcomes associated with already sequenced activities. 

These links are commonly referred to as “hand-offs” and serve to verify 

that activities are arranged in the right order to achieve aggregated 

products or outcomes. The schedule should also be vertically integrated, 

meaning that traceability exists among varying levels of activities and 

supporting tasks and subtasks. Such mapping or alignment among levels 

can enable different groups to work to the same master schedule.

Establishing the critical path for all activities: With the use of 

scheduling software, the critical path—the longest-duration path through 

the sequenced list of activities—should be identified. The establishment 

of a project’s critical path is necessary for examining the effects of delays 

in any activity along this path. Potential problems that may occur on or 

near the critical path should also be identified and reflected in the 

scheduling of the time for high-risk activities (see the next activity, 

“Identifying float”).

Identifying reasonable float: The schedule should identify float—the 

time that a predecessor activity can slip before the delay affects 

successor activities—so that schedule flexibility can be determined. As a 

general rule, activities along the critical path typically have the least 

amount of float.

Conducting a schedule risk analysis: A schedule risk analysis uses a 

good critical path method schedule and data about project schedule risks, 

as well as Monte Carlo simulation techniques, to predict the level of 

confidence in meeting a project’s completion date, the amount of time 

contingency needed for a level of confidence, and the identification of 

high-priority risks. This analysis should focus not only on critical path 

activities but also on other schedule paths that may become critical. A 

schedule/cost risk assessment recognizes the interrelationship between 

schedule and cost and captures the risk that schedule durations and cost 
Appendix I: Objectives, Scope and 

Methodology

Page 50 GAO-12-223 Air Traffic Control Modernization

estimates may vary for a variety of reasons, including limited data, 

optimistic estimating, technical challenges, lack of qualified personnel, 

and other external factors. As a result, the baseline schedule should 

include a buffer or a reserve of extra time. A reserve of extra time for 

contingencies should be calculated by performing a schedule risk 

analysis. As a general rule, the reserve should be held by the project 

manager and applied as needed to those activities that take longer than 

scheduled because of the identified risks. Reserves of time should not be 

apportioned in advance to any specific activity since the risks that will 

actually occur and the magnitude of their impact are not known in 

advance.

Updating the schedule using logic and durations to determine the 

dates: The schedule should use logic and durations in order to reflect 

realistic start and completion dates for project activities. The schedule 

should be continually monitored to determine when forecasted completion 

dates differ from the planned dates. This information can be used to 

determine whether schedule variances will affect downstream work. 

Maintaining the integrity of the schedule logic is not only necessary to 

reflect the project’s true status but is also required before conducting a 

schedule risk analysis. The schedule should avoid logic overrides and 

artificial constraint dates that are chosen to create a certain result on 

paper. Individuals trained in critical path method scheduling should be 

responsible for updating the schedule.

Based on our work, we determined the extent to which estimates and 

schedules for the four projects we selected met each best practices 

criterion:

• Not Met—project officials provided no evidence that satisfies any 

portion of the criterion.

• Minimally Met—project officials provided evidence that satisfies a 

small portion of the criterion.

• Partially Met—project officials provided evidence that satisfies about 

half of the criterion.

• Substantially Met—project officials provided evidence that satisfies a 

large portion of the criterion.

• Met—project officials provided evidence that satisfies the entire 

criterion.
Appendix I: Objectives, Scope and 

Methodology

Page 51 GAO-12-223 Air Traffic Control Modernization

We conducted this performance audit from August 2010 to February 2012 

in accordance with generally accepted government auditing standards. 

Those standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

conclusions based on our audit objectives.
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 52 GAO-12-223 Air Traffic Control Modernization

This appendix contains detailed information for 30 individual air traffic 

control programs. Each overview presents information and data that was 

provided by FAA. The overviews provide a description of the program and 

the cost and schedule status. The overviews are based on program office 

reported information as of August 2011. In most cases, we did not 

validate the data provided, but reviewed the data and performed various 

checks to determine they were reliable enough for our purposes.

Figure 3: Automatic Dependent Surveillance-Broadcast (ADS-B)

Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 53 GAO-12-223 Air Traffic Control Modernization

Figure 4: Advanced Technologies and Oceanic Procedures (ATOP) 
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 54 GAO-12-223 Air Traffic Control Modernization

Figure 5: Airport Surveillance Radar (ASR) Model 11 Tech Refresh Segment 1
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 55 GAO-12-223 Air Traffic Control Modernization

Figure 6: Aviation Surface Weather Observation Network (ASWON) and Automated Surface Observing System (ASOS) P31
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 56 GAO-12-223 Air Traffic Control Modernization

Figure 7: Air Traffic Control Radar Beacon Interrogator (ATCBI-6)
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 57 GAO-12-223 Air Traffic Control Modernization

Figure 8: Collaborative Air Traffic Management Technologies (CATMT), Work Packages 2 and 3 (WP2 and WP3)
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 58 GAO-12-223 Air Traffic Control Modernization

Figure 9: En Route Communication Gateway (ECG) Tech Refresh
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 59 GAO-12-223 Air Traffic Control Modernization

Figure 10: En Route Automation Modernization (ERAM)
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 60 GAO-12-223 Air Traffic Control Modernization

Figure 11: Integrated Display Systems (IDS) Replacement
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 61 GAO-12-223 Air Traffic Control Modernization

Figure 12: International Flight Inspection Aircraft (IFIA)
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 62 GAO-12-223 Air Traffic Control Modernization

Figure 13: Instrument Flight Procedure Automation (IFPA)
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed 

Page 63 GAO-12-223 Air Traffic Control Modernization 

Figure 14: Integrated Terminal Weather System (ITWS) 
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 64 GAO-12-223 Air Traffic Control Modernization

Figure 15: Next Generation Air/Ground Communication System (NEXCOM) Segment 1a
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 65 GAO-12-223 Air Traffic Control Modernization

Figure16: Next Generation Weather Radar (NEXRAD) Dual Polarization
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 66 GAO-12-223 Air Traffic Control Modernization

Figure 17: Power Systems Sustained Support (PS3)
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 67 GAO-12-223 Air Traffic Control Modernization

Figure 18: Regulation and Certification Infrastructure for System Safety (RCISS) Segment 2 
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 68 GAO-12-223 Air Traffic Control Modernization

Figure 19: Runway Status Lights (RWSL)
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 69 GAO-12-223 Air Traffic Control Modernization

Figure 20: Standard Terminal Automation Replacement System (STARS) (TAMR Phase 1)
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 70 GAO-12-223 Air Traffic Control Modernization

Figure 21: System Wide Information Management (SWIM) Segment 1
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 71 GAO-12-223 Air Traffic Control Modernization

Figure 22: Terminal Automation Modernization and Replacement (TAMR)—Phase 2
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed 

Page 72 GAO-12-223 Air Traffic Control Modernization 

Figure 23: Tower Training Simulator Systems 
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 73 GAO-12-223 Air Traffic Control Modernization

Figure 24: Trajectory Management—Arrival Tactical Flow Time Based Flow Management
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 74 GAO-12-223 Air Traffic Control Modernization

Figure 25: Terminal Doppler Weather Radar (TDWR) Service Life Extension Program (SLEP)
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 75 GAO-12-223 Air Traffic Control Modernization

Figure 26: Traffic Flow Management System (TFMS) Tech Refresh
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 76 GAO-12-223 Air Traffic Control Modernization

Figure 27: Ultra High Frequency (UHF) Radio Replacement
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 77 GAO-12-223 Air Traffic Control Modernization

Figure 28: Next Generation Voice Recorder Replacement Program (VRRP)
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 78 GAO-12-223 Air Traffic Control Modernization

Figure 29: Voice Switching and Control Switching System (VSCS) Tech Refresh Phase 2
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 79 GAO-12-223 Air Traffic Control Modernization

Figure 30: Wide Area Augmentation System (WAAS)
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 80 GAO-12-223 Air Traffic Control Modernization

Figure 31: Weather and Radar Processor (WARP) Sustain
Appendix II: Detailed Purpose and Status for 

30 Acquisitions GAO Reviewed

Page 81 GAO-12-223 Air Traffic Control Modernization

Figure 32: Weather Camera Program (WCP)
Appendix III: Assessments of Four FAA 

Program Cost Estimates

Page 82 GAO-12-223 Air Traffic Control Modernization

This appendix provides the results of our analysis of the extent to which 

the processes and methodologies used to develop and maintain the four 

FAA cost estimates meet the characteristics of high-quality cost 

estimates. These characteristics incorporate the 12 steps consistently 

applied by cost-estimating organizations throughout the federal 

government and industry and considered best practices for developing 

cost estimates and that are listed in table 2 of the report. The following 

tables provide the detailed results of our analysis of the program cost 

estimates for Automatic Dependent Surveillance-Broadcast (ADS-B), 

Collaborative Air Traffic Management Technologies (CATMT), System 

Wide Information Management (SWIM), and Wide Area Augmentation 

System (WAAS). “Not met” means the program provided no evidence that 

satisfies any of the criteria. “Minimally met” means the program provided 

evidence that satisfies a small portion of the criterion. “Partially met” 

means the program provided evidence that satisfies about half of the 

criterion. “Substantially met” means the program provided evidence that 

satisfies a large portion of the criterion. “Fully met” means the program 

provided evidence that completely satisfies the criterion.

Table 8: GAO’s Analysis of the FAA’s ADS-B Cost Estimates

Characteristic of high-quality cost 

estimates/assessment criterion/explanation

Extent to which criterion 

was met Key examples of rationale for assessment

Well-documented

• Captures the source data used, the reliability 

of the data, and how the data were made 

compatible with other data in the estimate.

Data should be collected from primary sources. 

The source, content, time, and units should be 

adequately documented. Data should also be 

analyzed to determine accuracy and reliability, 

and to identify cost drivers.

• Describes the calculations and the 

methodology used to derive each element’s 

cost.

Documentation should describe what calculation 

methods are used, as well as how they were 

applied, and explain any anomalies.

Partially met

• Lists data sources, but does not provide 

documentation of the source data, bringing into 

question the reliability of the data. (Minimally 

meets.) Data are the foundation of every cost 

estimate. Depending on data quality, an estimate 

can range anywhere from a mere guess to a 

highly defensible cost position. Data are often in 

many different forms and need to be adjusted 

before being used. The cost estimator needs 

information about the source and reliability of the 

data in order to know whether the data collected 

can be used directly or need to be modified.

• Documentation does not fully explain how FAA 

derived estimated costs. For example, FAA relied 

on expert opinion to estimate several costs but 

provided no historical data to back up the 

opinions. There was also no supporting 

information for the software cost estimates. 

(Minimally meets.) Poorly documented cost 

estimates can cause a program’s credibility to 

suffer because the documentation cannot explain 

the rationale for the methodology or the 

calculations. Estimates that lack sufficient 

documentation are not useful for updates or 

Appendix III: Assessments of Four FAA 

Program Cost Estimates
Appendix III: Assessments of Four FAA 

Program Cost Estimates

Page 83 GAO-12-223 Air Traffic Control Modernization

Characteristic of high-quality cost 

estimates/assessment criterion/explanation

Extent to which criterion 

was met Key examples of rationale for assessment

• Describes how the estimate was developed.

The data supporting the estimate should be 

available and adequately documented so that the 

estimate can be easily documented to reflect 

actual costs or program changes.

• Discusses the technical baseline description.

A technical baseline description provides a 

common definition of the program, including 

detailed technical, program, and schedule 

descriptions of the system, for a cost estimate to 

be built on. The data in the technical baseline 

should be consistent with the data used to 

develop the cost estimate.

• Provides evidnce of management review 

and acceptance.

There should be a briefing to management, 

including a clear explanation of how the cost 

estimate was derived. Management’s 

acceptance of the cost estimate should be 

documented.

information sharing and can hinder understanding 

and proper use.

• Cost calculations are described at a high level, 

but the documentation does not provide enough 

detail so that a cost analyst unfamiliar with the 

program could understand what was done and 

replicate it. Furthermore, we found an 

inconsistency between dollar values stored in the 

cost estimating tool and the Excel spreadsheets 

used to report life-cycle costs. (Partially met.)

Without good documentation, management and 

oversight officials will not be convinced that the 

estimate is credible; supporting data, lessons 

learned, and reasons why costs changed will not 

be available for future use; questions about the 

approach or data used to create the estimate 

cannot be answered; and the scope of the 

analysis cannot be thoroughly defined.

• Technical details contained within the basis-ofestimate documentation are consistent with 

corresponding details in the technical baseline. 

(Substantially meets.)

• The estimate was briefed to the Joint Resources 

Council. The briefing included a discussion of the 

scope, justification, and cost of the program. It 

also contained an overview of the program’s 

technical requirements, life-cycle costs, 

assumptions, and results of risk and sensitivity 

analysis. (Met.)

Comprehensive

• Includes all life-cycle costs.

A life-cycle cost estimate provides a complete 

and structured accounting of all resources and 

associated cost elements required to develop, 

produce, deploy, and sustain a particular 

program. It should cover the program from its 

inception through its retirement.

• Completely defines the program, reflects the 

current schedule, and is technically 

reasonable.

The cost estimate should be based on a 

documented technical baseline description, 

which provides a common definition of the 

program—including detailed technical, program, 

and schedule descriptions of the system.

Substantially met

• Includes costs from fiscal year 2007 through fiscal 

year 2035 for both government and contractor 

efforts across all phases of the program. (Met.)

• Reflects the current project schedule, but no single 

technical baseline was provided and there was no 

evidence that the technical documents had been 

updated to reflect changes. (Substantially met.)
Appendix III: Assessments of Four FAA 

Program Cost Estimates

Page 84 GAO-12-223 Air Traffic Control Modernization

Characteristic of high-quality cost 

estimates/assessment criterion/explanation

Extent to which criterion 

was met Key examples of rationale for assessment

• Has a product-oriented work breakdown 

structure, and is traceable to the program’s 

technical scope, at an appropriate level of 

detail.

A work breakdown structure provides a basic 

framework for a variety of related activities like 

estimating costs, developing schedules, 

identifying resources and potential risks, and 

providing the means for measuring program 

status using earned value management. It is 

product-oriented if it allows a program to track 

cost and schedule by defined deliverables, such 

as a hardware or software component.

• Documents all cost-influencing ground rules 

and assumptions. 

Cost estimates are typically based on limited 

information and therefore need to be bound by 

ground rules and assumptions. Ground rules are 

a set of estimating standards that provide 

guidance and common definitions, while 

assumptions are judgments about past, present, 

or future conditions that may affect the estimate.

• Reflects the FAA standard work breakdown structure, 

which is not product-oriented; instead, it breaks work 

down into functional categories. While the cost 

estimate work breakdown structure matches the work 

breakdown structure for earned value management, it 

does not match the work breakdown structure for the 

schedule. Furthermore, there was no evidence that 

the work breakdown structure had been updated as 

the program became better defined. (Partially met.)

Without a work breakdown structure, the program 

lacks a framework to develop a schedule and cost 

plan that can be used to easily track technical 

accomplishments. A standard product-oriented work 

breakdown structure facilitates the tracking of 

resource allocations and expenditures, which can 

give an agency insight to reliably estimate the cost of 

future similar programs.

• While ground rules and assumptions were 

discussed, many of the assumptions did not include 

supporting data. In addition, details supporting risk 

assumptions were incomplete because the sources 

for the risk ranges were not provided. (Partially met.)

Unless ground rules and assumptions are clearly 

documented, the cost estimate will not have a basis 

for assessing potential risks. Furthermore, the 

estimate cannot be reconstructed when the original 

estimators are no longer available.

Accurate

• Produces unbiased results.

Cost estimates should have an uncertainty 

analysis, which determines where the estimate 

falls against the range of all possible costs.

• Is properly adjusted for inflation.

Cost data should be adjusted for inflation to 

ensure that comparisons and projections are 

valid. Data should also be normalized to 

constant-year dollars to remove the effects of 

inflation.

• Contains few mistakes.

Results should be checked for accuracy, double 

counting, and omissions.

Partially met

• The cost basis of estimate contains minimum, 

most likely, and maximum values to model 

uncertainties for each cost element and to provide 

a range of costs; however, no analysis has been 

performed to determine the confidence level of 

the estimate. (Partially met.) A cost estimate is 

biased if the estimated work is overly 

conservative or too optimistic. Unless the 

estimate is based on an assessment of the most 

likely costs and reflects the degree of uncertainty 

given all of the risks considered, management will 

not be able to make good decisions.

• The cost estimate was adjusted for inflation 

correctly, but the source data for the government 

salaries inflation index were not provided.

(Substantially met.)

• We found no instances of incorrect formulas, 

double-counted costs, or omitted costs. 

Moreover, costs reported by fiscal year correctly 

summed to their life-cycle totals. (Met.)
Appendix III: Assessments of Four FAA 

Program Cost Estimates

Page 85 GAO-12-223 Air Traffic Control Modernization

Characteristic of high-quality cost 

estimates/assessment criterion/explanation

Extent to which criterion 

was met Key examples of rationale for assessment

• Is regularly updated to reflect significant 

program changes.

The cost estimate should be updated to reflect 

significant program changes, such as changes to 

schedules or other assumptions. Updates should 

also reflect actual costs so that the estimate 

always reflects the current program status.

• Documents and explains variances between 

planned and actual costs.

Variances between planned and actual costs 

should be documented, explained, and reviewed. 

For any elements whose actual costs or 

schedules differ from the estimate, the estimate 

should discuss variances and lessons learned.

• Reflects cost-estimating experiences from 

comparable programs.

The estimate should be based on historical cost 

estimation data and actual experiences from 

other comparable programs. These data should 

be reliable and relevant to the new program.

• There was no evidence that the cost estimate 

was updated to reflect changes or actual costs to 

reflect the current program status. (Not met.) A 

lack of cost estimate updates interferes with 

analysis of changes in program costs and hinders 

collection of cost and technical data to support 

future estimates. The cost estimate should be 

updated when the technical baseline changes; 

otherwise, it will lack credibility. A properly 

updated cost estimate can provide decision 

makers with accurate information for assessing 

alternative decisions.

• Variances between planned and actual costs, and 

explanations for the variance, were documented 

in earned value management data, but no 

variance for total cost at completion was included 

in the cost estimate documentation. (Substantially 

met.) Without a documented comparison between 

the current estimate (updated with actual costs) 

and the old estimate, cost estimators cannot 

determine the level of variance between the two 

estimates. That is, the estimators cannot see how 

well they are estimating and how the program is 

changing over time.

• There is some evidence that the estimate was 

based on a historical record of cost-estimating and 

actual experiences from other analogous programs. 

However, the reliability, risks, and applicability of the 

analogous data were not addressed. Also, 

comparisons were not made between ADS-B and 

analogous programs. (Minimally met.) Historical 

data provide the cost estimator with insight into 

actual costs on similar programs, including any cost 

growth that occurred after the original estimate. As a 

result, historical data can be used to challenge 

optimistic assumptions and bring more realism to a 

cost estimate.

Credible:

• Includes a sensitivity analysis that identifies 

a range of possible costs based on varying 

inputs.

A sensitivity analysis examines how changes to 

key assumptions and inputs affect the estimate. 

The estimate should identify key cost drivers, 

examine their parameters and assumptions, and 

re-estimate the total cost by varying each 

parameter between its minimum and maximum 

range.

Partially met

• While a high-level sensitivity analysis was 

provided to the Joint Resources Council for the 

equipage cost driver, other key cost drivers, 

assumptions, and data inputs were not varied. 

(Minimally met.) Because uncertainty cannot be 

avoided, it is necessary to identify the cost 

elements that represent the most risk. A 

sensitivity analysis reveals how the cost estimate 

is affected by a change in a single assumption, 

which helps the cost estimator understand the 

extent to which each variable affects the cost 

estimate. Any sources of variation should be welldocumented and traceable.
Appendix III: Assessments of Four FAA 

Program Cost Estimates

Page 86 GAO-12-223 Air Traffic Control Modernization

Characteristic of high-quality cost 

estimates/assessment criterion/explanation

Extent to which criterion 

was met Key examples of rationale for assessment

• Contains a risk and uncertainty analysis.

A risk and uncertainty analysis recognizes the 

potential for error and attempts to quantify it by 

identifying the effects of changing key cost 

drivers. 

• Includes cross-checking of major cost 

elements.

A cross-check is done by using a different 

method to see if it produces similar results.

Includes a comparison to an independent cost 

estimate conducted by another organization.

A second, independent cost estimate should be 

performed by an organization outside of the 

program office’s influence. It should be based on 

the same technical baseline, ground rules, and 

assumptions as the original estimate.

• A risk analysis for each cost element was 

performed but the risk data were essentially the 

same for each element and there was no 

documentation of correlations of risks between 

the cost elements. (Substantially met.)

• Contains no evidence that cross-checks were 

performed. (Not met.) The main purpose of crosschecking is to determine whether alternative 

methods produce similar results. If so, then 

confidence in the estimate increases, leading to 

greater credibility.

• A rough order-of-magnitude estimate was 

performed by another organization but addressed 

only a part of the total program. (Partially met.) An 

independent cost estimate is considered one of 

the best and most reliable estimate validation 

methods. It provides an independent view of 

expected program costs that tests the program 

office’s estimate for reasonableness. Without an 

independent cost estimate, decisions makers will 

lack insight into a program’s potential costs 

because independent cost estimates frequently 

use different methods and are less burdened with 

organizational bias.

Source: GAO analysis of FAA’s ADS-B cost estimate.
Appendix III: Assessments of Four FAA 

Program Cost Estimates

Page 87 GAO-12-223 Air Traffic Control Modernization

Table 9: GAO’s Analysis of FAA’s CATMT Cost Estimates

Characteristic of high-quality cost 

estimates/assessment criterion/explanation

Extent to which criterion 

was met Key examples of rationale for assessment

Well-documented:

• Captures the source data used, the reliability 

of the data, and how the data were made 

compatible with other data in the estimate;

Data should be collected from primary sources. 

The source, content, time, and units should be 

adequately documented. The data should also 

be analyzed to determine accuracy and 

reliability, and to identify cost drivers.

• Describes the calculations and the 

methodology used to derive each element’s 

cost.

Documentation should describe what calculation 

methods are used, as well as how they were 

applied, and explain any anomalies.

• Describes how the estimate was developed.

The data supporting the estimate should be 

available and adequately documented so that the 

estimate can be easily documented to reflect 

actual costs or program changes.

• Discusses the technical baseline 

description.

A technical baseline description provides a 

common definition of the program, including 

Partially met

• Lists data sources, the majority of which were 

subject matter experts. No details on the 

qualifications or background of these experts 

were provided, and there was no documentation 

about the reliability of the data. (Partially met.)

Data are the foundation of every cost estimate. 

Depending on data quality, an estimate can range 

anywhere from a mere guess to a highly 

defensible cost position. Data are often in many 

different forms and need to be adjusted before 

being used. The cost estimator needs information 

about the source and reliability of the data in 

order to know whether the data collected can be 

used directly or need to be modified.

• Documentation does not fully explain how FAA 

derived estimated costs. For example, FAA relied 

on expert opinion to estimate several costs based 

on a percentage of other cost elements but 

provided no historical data to back up the 

opinions. There was also no supporting 

information for the software cost estimates. 

(Partially met.) Poorly documented cost estimates 

can cause a program’s credibility to suffer 

because the documentation cannot explain the 

rationale for the methodology or the calculations. 

Estimates that lack sufficient documentation are 

not useful for updates or information sharing and 

can hinder understanding and proper use.

• Cost calculations are described in detail for fiscal 

year 2014, but documentation was missing for 

fiscal years 2015 through 2022. As a result, the 

documentation does not provide enough detail so 

that a cost analyst unfamiliar with the program 

could understand what was done and replicate it. 

(Partially met.) Without good documentation, 

management and oversight officials will not be 

convinced that the estimate is credible; 

supporting data, lessons learned, and reasons 

why costs changed will not be available for future 

use; questions about the approach or data used 

to create the estimate cannot be answered; and 

the scope of the analysis cannot be thoroughly 

defined.

• Many inconsistencies were found between the 

cost basis of the estimate document and the cost 

model. (Partially met.) Because the technical 

baseline is intended to serve as the basis for 

developing a cost estimate, it should be 
Appendix III: Assessments of Four FAA 

Program Cost Estimates

Page 88 GAO-12-223 Air Traffic Control Modernization

Characteristic of high-quality cost 

estimates/assessment criterion/explanation

Extent to which criterion 

was met Key examples of rationale for assessment

detailed technical, program, and schedule 

descriptions of the system, for a cost estimate to 

be built on. The data in the technical baseline 

should be consistent with the cost estimate.

• Provides evidence of management review 

and acceptance.

There should be a briefing to management, 

including a clear explanation of how the cost 

estimate was derived. Management’s 

acceptance of the cost estimate should be 

documented.

discussed in the cost estimate documentation. 

Without a technical baseline, the cost estimate 

will not be based on a comprehensive program 

description and will lack specific information about 

technical and program risks.

• The estimate was presented to the Joint 

Resources Council management during an 

investment decision briefing. The briefing 

included a program overview, life-cycle costs over 

time, risk analysis, budget justification, and 

recommendation. The briefing did not include 

detail on the estimating methodology for each 

cost element; sensitivity, risk, and uncertainty 

analysis; or an affordability analysis. 

(Substantially met.)

Comprehensive:

• Includes all life-cycle costs.

A life-cycle cost estimate provides a complete 

and structured accounting of all resources and 

associated cost elements required to develop, 

produce, deploy, and sustain a particular 

program. It should cover the program from its 

inception through its retirement.

• Completely defines the program, reflects the 

current schedule, and is technically 

reasonable.

The cost estimate should be based on a 

documented technical baseline description, 

which provides a common definition of the 

program—including detailed technical, program, 

and schedule descriptions of the system.

• Has a product-oriented work breakdown 

structure, and is traceable to the program’s 

technical scope, at an appropriate level of 

detail.

A work breakdown structure provides a basic 

framework for a variety of related activities like 

estimating costs, developing schedules, 

identifying resources and potential risks, and 

providing the means for measuring program 

status using earned value management. It is 

product-oriented if it allows a program to track 

cost and schedule by defined deliverables, such 

as a hardware or software component.

• Documents all cost-influencing ground rules 

and assumptions.

Cost estimates are typically based on limited 

information and therefore need to be bound by 

ground rules and assumptions. Ground rules are 

a set of estimating standards that provide 

Substantially met

• Includes costs from fiscal year 2009 through fiscal 

year 2022 for both government and contractor 

efforts across all phases of the program. 

However, some government labor costs for inservice management were not included. 

(Substantially met.)

• Reflects many detailed technical requirements, 

but no single technical baseline was provided. 

(Substantially met.)

• Reflects FAA’s standard work breakdown 

structure, which does not follow a product-oriented 

work breakdown structure. Instead, it breaks work 

down into functional categories. The cost estimate 

work breakdown structure matches the schedule 

work breakdown structure. However, we found no 

evidence that the work breakdown structure had 

been updated as the program became better 

defined. (Substantially met.)

• The estimate provides a limited set of costinfluencing ground rules and assumptions; 

however, many of the assumptions did not 

include supporting data. In addition, there was 

little evidence risks were identified if the 

assumption did not hold. (Partially met.) Unless 
Appendix III: Assessments of Four FAA 

Program Cost Estimates

Page 89 GAO-12-223 Air Traffic Control Modernization

Characteristic of high-quality cost 

estimates/assessment criterion/explanation

Extent to which criterion 

was met Key examples of rationale for assessment

guidance and common definitions, while 

assumptions are judgments about past, present, 

or future conditions that may affect the estimate.

ground rules and assumptions are clearly 

documented, the cost estimate will not have a 

basis for assessing potential risks. Furthermore, 

the estimate cannot be reconstructed when the 

original estimators are no longer available.

Accurate:

• Produces unbiased results.

Cost estimates should have an uncertainty 

analysis, which determines where the estimate 

falls against the range of all possible costs.

• Is properly adjusted for inflation.

Cost data should be adjusted for inflation to 

ensure that comparisons and projections are 

valid. Data should also be normalized to 

constant-year dollars to remove the effects of 

inflation.

• Contains few mistakes.

Results should be checked for accuracy, double 

counting, and omissions.

• Is regularly updated to reflect significant 

program changes.

The cost estimate should be updated to reflect 

significant program changes, such as changes to 

schedules or other assumptions. Updates should 

also reflect actual costs so that the estimate 

always reflects the current program status.

• Documents and explains variances between 

planned and actual costs.

Variances between planned and actual costs 

should be documented, explained, and reviewed. 

For any elements whose actual costs or 

Partially met

• The cost basis of estimate contains low, likely, 

and high estimates for software lines of code, but 

the estimate did not have a similar analysis for 

every assumption in the cost model. In addition, 

we found no confidence level associated with the 

estimate. (Partially met.) A cost estimate is biased 

if the estimated work is overly conservative or too 

optimistic. Unless the estimate is based on an 

assessment of the most likely costs and reflects 

the degree of uncertainty given all of the risks 

considered, management will not be able to make 

good decisions.

• The cost estimate was adjusted for inflation 

correctly and the source data for the inflation 

indexes were provided. (Met.)

• While we found no apparent mistakes in the 

calculations, as stated earlier, there were 

inconsistencies in costs between the cost model 

and the cost basis of estimate documentation. 

(Partially met.) Validating that a cost estimate is 

accurate requires thoroughly understanding and 

investigating how the cost model was 

constructed. Without access to a detailed cost 

model and estimate details, calculations may not 

be accurate or expressed consistently,

• We found no evidence that the cost estimate was 

updated to reflect changes or actual costs. (Not 

met.) A lack of cost estimate updates interferes 

with analysis of changes in program costs and 

hinders collection of cost and technical data to 

support future estimates. The cost estimate 

should be updated when the technical baseline 

changes; otherwise, it will lack credibility. A 

properly updated cost estimate can provide 

decision makers with accurate information for 

assessing alternative decisions.

• There was no documentation of variances 

between planned and actual costs. (Not met.)

Without a documented comparison between the 

current estimate (updated with actual costs) and 

the old estimate, the cost estimators cannot 
Appendix III: Assessments of Four FAA 

Program Cost Estimates

Page 90 GAO-12-223 Air Traffic Control Modernization

Characteristic of high-quality cost 

estimates/assessment criterion/explanation

Extent to which criterion 

was met Key examples of rationale for assessment

schedules differ from the estimate, the estimate 

should discuss variances and lessons learned.

• Reflects cost-estimating experiences from 

comparable programs.

The estimate should be based on historical cost 

estimation data and actual experiences from 

other comparable programs. These data should 

be reliable and relevant to the new program.

determine the level of variance between the two 

estimates. That is, the estimators cannot see how 

well they are estimating and how the program is 

changing over time.

• There is some evidence that the estimate was 

based on a historical record of cost-estimating 

and actual experiences from other analogous 

programs. For example, software costs were 

estimated using function points and the the 

Constructive Cost Model (COCOMO). However, 

no comparison was made between CATMT and 

other comparable FAA programs for the purpose 

of the validating the CATMT estimate. (Minimally 

met.) Historical data provide the cost estimator 

with insight into actual costs on similar programs, 

including any cost growth that occurred after the 

original estimate. As a result, historical data can 

be used to challenge optimistic assumptions and 

bring more realism to a cost estimate.

Credible:

• Includes a sensitivity analysis that identifies 

a range of possible costs based on varying 

inputs;

A sensitivity analysis examines how changes to 

key assumptions and inputs affect the estimate. 

The estimate should identify key cost drivers, 

examine their parameters and assumptions, and 

re-estimate the total cost by varying each 

parameter between its minimum and maximum 

range.

• Contains a risk and uncertainty analysis.

A risk and uncertainty analysis recognizes the 

potential for error and attempts to quantify it by 

identifying the effects of changing key cost 

drivers. 

• Cross-checking of major cost elements.

A cross-check is done by using a different 

method to see if it produces similar results.

Minimally met

• A sensitivity analysis was provided on the 

discount rate used in the economic analysis, but 

other key cost drivers like software design and 

development, as well as assumptions and data 

inputs, were not varied. (Minimally met.) Because 

uncertainty cannot be avoided, it is necessary to 

identify the cost elements that represent the most 

risk. A sensitivity analysis reveals how the cost 

estimate is affected by a change in a single 

assumption, which helps the cost estimator 

understand the extent to which each variable 

affects the cost estimate. Any sources of variation 

should be well-documented and traceable.

• The documentation discusses some aspects of a 

risk and uncertainty analysis, such as risk 

percents, but lacks some results, such as the 

confidence level for the cost estimate. (Partially 

met.) The program estimate should reflect the 

degree of uncertainty, so that a level of 

confidence can be provided to management 

about the estimate. An estimate without risk and 

uncertainty analysis is unrealistic because it does 

not assess the variability in the cost estimate from 

such effects as schedules slipping, missions 

changing, and proposed solutions not meeting 

users’ needs.

• The documentation states that a function point 

analysis and a historical cost estimation method 

were used to cross-check the software cost 

estimate. However, a comparison of the model 

results was not provided, and we could not 
Appendix III: Assessments of Four FAA 

Program Cost Estimates

Page 91 GAO-12-223 Air Traffic Control Modernization

Characteristic of high-quality cost 

estimates/assessment criterion/explanation

Extent to which criterion 

was met Key examples of rationale for assessment

• A comparison to an independent cost 

estimate conducted by another organization.

A second, independent cost estimate should be 

performed by an organization outside of the 

program office’s influence. It should be based on 

the same technical baseline, ground rules, and 

assumptions as the original estimate.

identify other examples of cross-checks. 

(Minimally met.) The main purpose of crosschecking is to determine whether alternative 

methods produce similar results. If so, then 

confidence in the estimate increases, leading to 

greater credibility.

• There was no evidence that an independent cost 

estimate was conducted. (Not met.) An 

independent cost estimate is considered one of 

the best and most reliable estimate validation 

methods. It provides an independent view of 

expected program costs that tests the program 

office’s estimate for reasonableness. Without an 

independent cost estimate, decision makers will 

lack insight into a program’s potential costs 

because independent cost estimates frequently 

use different methods and are less burdened with 

organizational bias.

Source: GAO analysis of FAA’s CATMT cost estimate.
Appendix III: Assessments of Four FAA 

Program Cost Estimates

Page 92 GAO-12-223 Air Traffic Control Modernization

Table 10: GAO’s Analysis of FAA’s SWIM Cost Estimates

Characteristic of high-quality cost 

estimates/assessment criterion/explanation

Extent to which criterion 

was met Key examples of rationale for assessment

Well-documented:

• Captures the source data used, the reliability 

of the data, and how the data were made 

compatible with other data in the estimate.

Data should be collected from primary sources. 

The source, content, time, and units should be 

adequately documented. Data should also be 

analyzed to determine accuracy and reliability, 

and to identify cost drivers.

• Describes the calculations and the 

methodology used to derive each element’s 

cost.

Documentation should describe what calculation 

methods are used, as well as how they were 

applied, and explain any anomalies.

• Describes how the estimate was developed.

The data supporting the estimate should be 

available and adequately documented so that the 

estimate can be easily documented to reflect 

actual costs or program changes.

• Discusses the technical baseline description.

A technical baseline description provides a 

common definition of the program, including 

detailed technical, program, and schedule 

descriptions of the system, for a cost estimate to 

be built on. The data in the technical baseline 

should be consistent with the cost estimate.

• Provides evidence of management review 

and acceptance.

There should be a briefing to management, 

including a clear explanation of how the cost 

estimate was derived. Management’s acceptance 

of the cost estimate should be documented.

Substantially met

• The documentation captures the majority of the 

data sources used. However, it does not provide 

specific backup of the data sources and does not 

capture the reliability or the accuracy of the data. 

(Partially met.) Data are the foundation of every 

cost estimate. Depending on data quality, an 

estimate can range anywhere from a mere guess 

to a highly defensible cost position. Data are 

often in many different forms and need to be 

adjusted before being used. The cost estimator 

needs information about the source and reliability 

of the data in order to know whether the data 

collected can be used directly or need to be 

modified.

• The documentation describes in detail the 

methodology, calculations, basis of data, and 

quantities used to determine most of the cost 

estimate. However, there is little detail provided 

about the cost elements that used the analogy 

and parametric estimating methodologies. 

(Substantially met.)

• The documentation provides enough detail so 

that a cost analyst unfamiliar with the program 

could understand what was done and replicate it; 

however, there was no sensitivity analysis 

performed. (Substantially met.)

• The technical details contained within the basis of 

estimate are consistent with the corresponding 

details in the technical baseline, but there is no 

mapping between the basis of estimate, the 

technical baseline, and the life-cycle cost 

estimate. (Substantially met.)

• The estimate was presented to management 

during a briefing and was accepted by the 

issuance of a Joint Resources Council record of 

decision. The briefing included a program 

overview, a comparison between the 2007 and 

2009 estimates, a breakout of cost drivers, and a 

business case analysis. In addition, a software 

estimate briefing discussed the estimating 

methodology used. However, more specific 

items, such as data sources, risk information, and 

a comparison to an independent cost estimate 

were not provided. (Partially met.) A cost 

estimate is not considered valid until 

management has approved it. It is imperative that 
Appendix III: Assessments of Four FAA 

Program Cost Estimates

Page 93 GAO-12-223 Air Traffic Control Modernization

Characteristic of high-quality cost 

estimates/assessment criterion/explanation

Extent to which criterion 

was met Key examples of rationale for assessment

management understand how the estimate was 

developed, including the risks associated with the 

underlying data and methods.

Comprehensive:

• Includes all life-cycle costs.

A life-cycle cost estimate provides a complete 

and structured accounting of all resources and 

associated cost elements required to develop, 

produce, deploy, and sustain a particular 

program. It should cover the program from its 

inception through its retirement.

• Completely defines the program, reflects the 

current schedule, and is technically 

reasonable.

The cost estimate should be based on a 

documented technical baseline description, 

which provides a common definition of the 

program—including detailed technical, program, 

and schedule descriptions of the system.

• Has a product-oriented work breakdown 

structure, and is traceable to the program’s 

technical scope, at an appropriate level of 

detail.

A work breakdown structure provides a basic 

framework for a variety of related activities like 

estimating costs, developing schedules, 

identifying resources and potential risks, and 

providing the means for measuring program 

status using earned value management. It is 

product-oriented if it allows a program to track 

cost and schedule by defined deliverables, such 

as a hardware or software component.

•

Documents all cost-influencing ground rules 

and assumptions.

Cost estimates are typically based on limited 

information and therefore need to be bound by 

ground rules and assumptions. Ground rules are 

a set of estimating standards that provide 

guidance and common definitions, while 

assumptions are judgments about past, present, 

or future conditions that may affect the estimate.

Substantially met

• Includes all life-cycle costs for each of the seven 

SWIM Implementing Programs (SIP) from fiscal year 

2011 through fiscal year 2033. However, costs for 

fiscal year 2009 and fiscal year 2010 were not 

included because these costs were already 

baselined. The cost estimate also includes both 

government and contractor efforts across all phases 

of the program. (Substantially met.)

• Reflects many detailed technical requirements, 

including scope and schedule requirements, but no 

single technical baseline was provided. Also, there 

was no evidence that the technical baseline 

requirements had been updated. (Substantially met.)

• Reflects FAA’s standard work breakdown structure, 

which does not follow a product-oriented work 

breakdown structure. Instead, it breaks work down 

into functional categories. The cost estimate work 

breakdown structure also does not match either the 

schedule or the earned value management work 

breakdown structure. In addition, we found no 

evidence that the work breakdown structure had 

been updated as the program became better defined. 

(Partially met.) Without a work breakdown structure, 

the program lacks a framework to develop a 

schedule and cost plan that can be used to easily 

track technical accomplishments. A standard, 

product-oriented work breakdown structure facilitates 

the tracking of resource allocations and expenditures, 

which can give FAA insight into reliable estimates of 

the cost of similar programs in the future.

• The estimate documents all cost-influencing ground 

rules and assumptions; however, in several 

instances, not all of the data sources supporting each 

were provided. In addition, while there was a risk 

adjustment section in the documentation, the majority 

of the risk adjustments were based on the judgment 

of the SWIM program office team. (Partially met.)

Unless ground rules and assumptions are clearly 

documented, the cost estimate will not have a basis 

for assessing potential risks. Furthermore, the 

estimate cannot be reconstructed when the original 

estimators are no longer available.
Appendix III: Assessments of Four FAA 

Program Cost Estimates

Page 94 GAO-12-223 Air Traffic Control Modernization

Characteristic of high-quality cost 

estimates/assessment criterion/explanation

Extent to which criterion 

was met Key examples of rationale for assessment

Accurate:

• Produces unbiased results.

Cost estimates should have an uncertainty 

analysis, which determines where the estimate 

falls against the range of all possible costs.

• Is properly adjusted for inflation.

Cost data should be adjusted for inflation to 

ensure that comparisons and projections are 

valid. Data should also be normalized to 

constant-year dollars to remove the effects of 

inflation.

• Contains few mistakes.

Results should be checked for accuracy, double 

counting, and omissions.

• Is regularly updated to reflect significant 

program changes.

The cost estimate should be updated to reflect 

significant program changes, such as changes to 

schedules or other assumptions. Updates should 

also reflect actual costs so that the estimate 

always reflects the current program status.

• Documents and explains variances between 

planned and actual costs.

Variances between planned and actual costs 

should be documented, explained, and reviewed. 

For any elements whose actual costs or 

schedules differ from the estimate, the estimate 

should discuss variances and lessons learned.

Partially met

• The cost basis of estimate contains minimum, 

most likely and maximum values to model 

uncertainties for each cost element. No historical 

data were provided to support any of the risk 

adjustments, and no level of confidence was 

associated with the estimate. (Minimally met.) A 

cost estimate is biased if the estimated work is 

overly conservative or too optimistic. Unless the 

estimate is based on an assessment of the most 

likely costs and reflects the degree of uncertainty 

given all of the risks considered, its usefulness to

management for making decisions is limited.

• The cost estimate was adjusted for inflation 

correctly and the source data for the inflation 

indexes were provided for all elements except for 

government salaries. (Substantially met.)

• The estimate contains no mistakes, and we found 

no evidence of double counting or omissions. 

Additionally, the documentation, equations, and 

results were consistent between the 

documentation and the cost model. (Met.)

• There was no evidence that the cost estimate 

was recently updated to reflect changes or actual 

costs to reflect current program status. The cost 

estimate was updated in 2009 from the original 

2007 estimate, which requested funding only for 

fiscal year 2009 through fiscal year 2010.

(Minimally met.) A lack of cost estimate updates 

interferes with analysis of changes in program 

costs and hinders collection of cost and technical 

data to support future estimates. The cost 

estimate should be updated when the technical 

baseline changes; otherwise, it will lack 

credibility. A properly updated cost estimate can 

provide decision makers with accurate 

information for assessing alternative decisions.

• Earned value management system reports for July 

2010 through December 2010 contained actual 

costs and variances. However, no recent earned 

value management reports captured actual costs 

and variances. In addition, because the cost 

estimate work breakdown structure did not match 

the earned value management work breakdown 

structure, it would be difficult to track variances from 

the earned value management system back to the 

cost estimate. (Minimally met.) Without a 

documented comparison between the current 

estimate (updated with actual costs) and the old 
Appendix III: Assessments of Four FAA 

Program Cost Estimates

Page 95 GAO-12-223 Air Traffic Control Modernization

Characteristic of high-quality cost 

estimates/assessment criterion/explanation

Extent to which criterion 

was met Key examples of rationale for assessment

• Reflects cost estimating experiences from 

comparable programs.

The estimate should be based on a historical 

cost estimation data and actual experiences from 

other comparable programs. These data should 

be reliable and relevant to the new program.

estimate, cost estimators cannot determine the level 

of variance between the two estimates. That is, the 

estimators cannot see how well they are estimating 

and how the program is changing over time.

• There is some evidence that the estimate was 

based on a historical record of cost estimating. 

For example, per diem and hotel rates were 

obtained from a government website; however, 

no such comparable data were provided for other 

cost elements. (Minimally met.) Historical data 

provide the cost estimator with insight into actual 

costs on similar programs, including any cost 

growth that occurred after the original estimate. 

As a result, historical data can be used to 

challenge optimistic assumptions and bring more 

realism to a cost estimate.

Credible:

• Includes a sensitivity analysis that identifies 

a range of possible costs based on varying 

inputs. 

A sensitivity analysis examines how changes to 

key assumptions and inputs affect the estimate. 

The estimate should identify key cost drivers, 

examine their parameters and assumptions, and 

re-estimate the total cost by varying each 

parameter between its minimum and maximum 

range.

• Contains a risk and uncertainty analysis.

A risk and uncertainty analysis recognizes the 

potential for error and attempts to quantify it by 

identifying the effects of changing key cost 

drivers. 

• Includes cross-checking of major cost 

elements.

A cross-check is done by using a different 

method to see if it produces similar results.

Minimally met

• While an independent evaluation includes a 

breakout of cost drivers as a percentage of total 

cost, the cost estimate documentation does not 

show evidence that a sensitivity analysis was 

performed. (Minimally met.) Because uncertainty 

cannot be avoided, it is necessary to identify the 

cost elements that represent the most risk. A 

sensitivity analysis reveals how the cost estimate 

is affected by a change in a single assumption, 

which helps the cost estimator to understand the 

extent to which each variable affects the cost 

estimate. Any sources of variation should be welldocumented and traceable.

• The documentation discusses some aspects of a 

risk and uncertainty analysis, and each element 

receives a risk adjustment. A risk briefing that 

showed cost risk ranges for all of the SIPs were 

also provided. However, the documentation did not 

discuss what level of confidence was associated 

with the cost estimate. (Partially met.).The 

program estimate must reflect the degree of 

uncertainty, so that a level of confidence can be 

provided to management about the estimate. An 

estimate without risk and uncertainty analysis is 

unrealistic because it does not assess the 

variability in the cost estimate from such effects as 

schedules slipping, missions changing, and 

proposed solutions not meeting users’ needs.

• There was no evidence that cross-checks were 

performed. (Not met.) The main purpose of crosschecking is to determine whether alternative 

methods produce similar results. If so, then 

confidence in the estimate increases, leading to 

greater credibility.
Appendix III: Assessments of Four FAA 

Program Cost Estimates

Page 96 GAO-12-223 Air Traffic Control Modernization

Characteristic of high-quality cost 

estimates/assessment criterion/explanation

Extent to which criterion 

was met Key examples of rationale for assessment

• Contains a comparison to an independent 

cost estimate conducted by another 

organization.

A second, independent cost estimate should be 

performed by an organization outside of the 

program office’s influence. It should be based on 

the same technical baseline, ground rules, and 

assumptions, as the original estimate.

• There was no evidence that an independent cost 

estimate was conducted; however, there was an 

independent assessment briefing that showed a 

review of all SIP cost estimates. (Minimally met.)

An independent cost estimate is considered one 

of the best and most reliable estimate validation 

methods. It provides an independent view of 

expected program costs that tests the program 

office’s estimate for reasonableness. Without an 

independent cost estimate, decisions makers will 

lack insight into a program’s potential costs 

because independent cost estimates frequently 

use different methods and are less burdened with 

organizational bias.

Source: GAO analysis of FAA’s SWIM cost estimate.
Appendix III: Assessments of Four FAA 

Program Cost Estimates

Page 97 GAO-12-223 Air Traffic Control Modernization

Table 11: GAO’s Analysis of FAA’s WAAS Cost Estimates

Characteristic of high-quality cost 

estimates/assessment 

criterion/explanation

Extent to which criterion was 

met Key examples of rationale for assessment

Well-documented:

• Captures the source data used, the 

reliability of the data, and how the data 

were made compatible with other data 

in the estimate.

Data should be collected from primary 

sources. The source, content, time, and 

units should be adequately documented. 

Data should also be analyzed to determine 

accuracy and reliability, and to identify cost 

drivers. 

• Describes the calculations and the 

methodology used to derive each 

element’s cost.

Documentation should describe what 

calculation methods were used, as well as 

how they were applied, and explain any 

anomalies.

• Describes how the estimate was 

developed.

The data supporting the estimate should be 

available and adequately documented so 

that the estimate can be easily documented 

to reflect actual costs or program changes.

• Discusses the technical baseline 

description.

A technical baseline description provides a 

common definition of the program, including 

detailed technical, program, and schedule 

descriptions of the system, for a cost 

estimate to be built on. The data in the 

technical baseline should be consistent with 

the cost estimate.

• Provides evidence of management 

review and acceptance.

There should be a briefing to management, 

including a clear explanation of how the cost 

estimate was derived. Management’s 

acceptance of the cost estimate should be 

Substantially met

• The documentation captures the majority of the 

data sources used; however, it does not capture 

the reliability or the accuracy of the data. (Partially 

met.) Data are the foundation of every cost 

estimate. Depending on data quality, an estimate 

can range anywhere from a mere guess to a highly 

defensible cost position. Data are often in many 

different forms and need to be adjusted before 

being used. The cost estimator needs information 

about the source and reliability of the data in order 

to know whether the data collected can be used 

directly or need to be modified.

• The documentation describes in great detail the 

methodology, calculations, basis of data, and 

quantities used in the cost estimate. However, 

estimates for some elements were based on the 

judgment of subject matter experts. (Substantially 

met.)

• The documentation provides enough detail so that 

a cost analyst unfamiliar with the program could 

understand what was done and replicate it. There 

was no sensitivity analysis performed, and some 

cost elements did not have a risk adjustment 

because they were considered ongoing efforts. 

(Substantially met.)

• The technical details contained within the basis of 

the estimate are consistent with the corresponding 

details in the technical baseline. However, the cost 

estimate does not reference the technical baseline 

documents, and there is no mapping between the 

basis of estimate, the technical baseline, and the 

cost estimate. (Substantially met.)

• The estimate was presented to management 

during a briefing and was accepted by the 

issuance of a Joint Resources Council record of 

decision. The briefing included a program overview 

and specific details about the service area, 

previous baselines, and a cost benefit analysis. In 

addition, information about equipment 

procurement, the schedule, and the spending plan 

were also presented along with a risk analysis, an 

affordability analysis, and an independent 

evaluation. However, more specific items such as 

data sources for each cost element, a sensitivity 

analysis, an uncertainty analysis, and a 

comparison against an independent cost estimate 

were not provided. (Substantially met.) 
Appendix III: Assessments of Four FAA 

Program Cost Estimates

Page 98 GAO-12-223 Air Traffic Control Modernization

Characteristic of high-quality cost 

estimates/assessment 

criterion/explanation

Extent to which criterion was 

met Key examples of rationale for assessment

documented.

Comprehensive:

• Includes all life-cycle costs.

A life-cycle cost estimate provides a 

complete and structured accounting of all 

resources and associated cost elements 

required to develop, produce, deploy, and 

sustain a particular program. It should cover 

the program from its inception through its 

retirement.

• Completely defines the program, 

reflects the current schedule, and is 

technically reasonable.

The cost estimate should be based on a 

documented technical baseline description, 

which provides a common definition of the 

program, including detailed technical, 

program, and schedule descriptions of the 

system.

• Has a product-oriented work breakdown 

structure, and is traceable to the 

program’s technical scope, at an 

appropriate level of detail.

A work breakdown structure provides a 

basic framework for a variety of related 

activities like estimating costs, developing 

schedules, identifying resources and 

potential risks, and providing the means for 

measuring program status using earned 

value management. It is product-oriented if it 

allows a program to track cost and schedule 

by defined deliverables, such as a hardware 

or software component.

• Documents all cost-influencing ground 

rules and assumptions.

Cost estimates are typically based on limited 

information and therefore need to be bound 

Substantially met

• Includes all life-cycle costs from fiscal year 2009 

through fiscal year 2028 except sunk costs and 

disposition costs. The cost estimate covers both 

government and contractor efforts for the solution 

development, implementation, and in-service 

management phases of the program. (Partially 

met.) A life-cycle cost estimate should 

encompass all past, present, and future costs for 

every aspect of the program, regardless of 

funding source—including all government and 

contractor costs. Life-cycle cost estimates can 

enhance decision making by allowing for design 

trade-off studies to be evaluated on a total cost 

basis as well as on a technical and performance 

basis.

• Reflects many detailed technical requirements, 

including scope and schedule requirements, but 

no single technical baseline was provided. 

Consistent with best practices, we found 

evidence that the technical baseline requirements 

had been updated to reflect technical, program, 

and schedule changes to the program. 

(Substantially met.)

• Reflects FAA’s standard work breakdown 

structure, which does not follow a productoriented work breakdown structure. Instead, it 

breaks work down into functional categories. The 

cost estimate work breakdown structure also 

does not match either the schedule or the earned 

value management work breakdown structure; 

however, there was a mapping from the schedule 

work breakdown structure to the cost work 

breakdown structure. In addition, we found 

evidence that the work breakdown structure had 

been updated as the program became better 

defined. (Partially met.) Without a work 

breakdown structure, the program lacks a 

framework to develop a schedule and cost plan 

that can be used to easily track technical 

accomplishments. A standard, product-oriented 

work breakdown structure facilitates the tracking 

of resource allocations and expenditures, which 

can give the agency a basis to reliably estimate 

the cost of future similar programs.

• The estimate documents all cost-influencing 

ground rules and assumptions. In addition, there 

were additional ground rules and assumptions 

found in the technical baseline’s authoritative 
Appendix III: Assessments of Four FAA 

Program Cost Estimates

Page 99 GAO-12-223 Air Traffic Control Modernization

Characteristic of high-quality cost 

estimates/assessment 

criterion/explanation

Extent to which criterion was 

met Key examples of rationale for assessment

by ground rules and assumptions. Ground 

rules are a set of estimating standards that 

provide guidance and common definitions, 

while assumptions are judgments about 

past, present, or future conditions that may 

affect the estimate.

references. However, in several instances, not all 

of the data sources supporting the assumptions 

were provided. In addition, while there was a risk 

adjustment section in the documentation, the 

majority of the risk adjustments were blank, 

based on the assumption that the costs were well 

defined by this point. (Substantially met.). 

Accurate:

• Produces unbiased results.

Cost estimates should have an uncertainty 

analysis, which determines where the 

estimate falls against the range of all 

possible costs.

Is properly adjusted for inflation.

Cost data should be adjusted for inflation to 

ensure that comparisons and projections are 

valid. Data should also be normalized to 

constant-year dollars to remove the effects 

of inflation.

Contains few mistakes.

Results should be checked for accuracy, 

double counting, and omissions.

• Is regularly updated to reflect significant 

program changes.

• The cost estimate should be updated to 

reflect significant program changes, 

such as changes to schedules or other 

assumptions. Updates should also 

reflect actual costs so that the estimate 

always reflects the current program 

status.

• Documents and explains variances 

between planned and actual costs.

• Variances between planned and actual 

costs should be documented, explained, 

and reviewed. For any elements whose 

actual costs or schedules differ from the 

estimate, the estimate should discuss 

variances and lessons learned.

• Reflects cost estimating experiences 

from comparable programs.

The estimate should be based on a historical 

cost estimation data and actual experiences 

from other comparable programs. These 

data should be reliable and relevant to the 

new program.

Partially met 

• A minimal uncertainty analysis was performed for 

select cost elements with an 80 percent 

confidence level reported. No analysis was done 

to show the accuracy of the data sources used 

for this analysis. (Minimally met.) A cost estimate 

is biased if the work estimate is overly 

conservative or too optimistic. Unless the 

estimate is based on an assessment of the most 

likely costs and reflects the degree of uncertainty 

given all of the risks considered, management will 

not be able to make good decisions.

• The cost estimate was adjusted for inflation 

correctly, and the source data for the inflation 

indexes were provided. (Met.)

• The estimate contains no mistakes, and we found 

no evidence of double counting or omissions. 

Additionally, the documentation, equations, and 

results were consistent between the 

documentation and the cost model. (Met.)

• There was evidence that the cost estimate was 

updated to reflect changes in technical and 

program requirements. For example, the WAAS 

program has undergone four rebaselines from 

fiscal year 1994 through fiscal year 2009. During 

each rebaseline, the cost estimate was updated. 

However, the cost model is not updated on a 

frequent basis. It is only updated to reflect major 

changes in technical and program requirements. 

There was no evidence that the cost estimate 

was updated with actual costs. (Partially met.) A 

lack of cost estimate updates interferes with 

analysis of changes in program costs and hinders 

collection of cost and technical data to support 

future estimates. The cost estimate should be 

updated when the technical baseline changes; 

otherwise, it will lack credibility. A properly 

updated cost estimate can provide decision 

makers with accurate information for assessing 

alternative decisions.

• Earned value management system reports for 

November 2008 through February 2009 

contained actual costs and variances. However, 
Appendix III: Assessments of Four FAA 

Program Cost Estimates

Page 100 GAO-12-223 Air Traffic Control Modernization

Characteristic of high-quality cost 

estimates/assessment 

criterion/explanation

Extent to which criterion was 

met Key examples of rationale for assessment

no recent earned value management reports 

captured actual costs and variances. In addition, 

because the cost estimate work breakdown 

structure did not match the earned value 

management work breakdown structure, updating 

the cost estimate would not be a straightforward 

effort. (Partially met.) Without a documented 

comparison between the current estimate 

(updated with actual costs) and the old estimate, 

cost estimators cannot determine the level of 

variance between the two estimates. That is, the 

estimators cannot see how well they are 

estimating and how the program is changing over 

time.

• There is some evidence that the estimate was 

based on a historical record of cost estimating. 

For example, software source lines of code were 

estimated based on analogous historical data, but 

there was no documentation to support these 

costs. For the other cost elements, we did not 

find sufficient documentation of historical costs 

from comparable programs. (Minimally met.)

Historical data provide the cost estimator with 

insight into actual costs on similar programs, 

including any cost growth that occurred after the 

original estimate. As a result, historical data can 

be used to challenge optimistic assumptions and 

bring more realism to a cost estimate.

Credible:

• Includes a sensitivity analysis that 

identifies a range of possible costs 

based on varying inputs;

A sensitivity analysis examines how 

changes to key assumptions and inputs 

affect the estimate. The estimate should 

identify key cost drivers, examine their 

parameters and assumptions, and reestimate the total cost by varying each 

parameter between its minimum and 

maximum range.

• A risk and uncertainty analysis.

A risk and uncertainty analysis recognizes 

the potential for error and attempts to 

quantify it by identifying the effects of 

changing key cost drivers.

Partially met

• Minimal evidence was provided to show that a 

sensitivity analysis was provided for some cost 

elements, such as labor. However, we could not 

determine if key cost drivers were identified 

across the entire program or if input ranges were 

varied. (Minimally met.) Because uncertainty 

cannot be avoided, it is necessary to identify the 

cost elements that represent the most risk. A 

sensitivity analysis reveals how the cost estimate 

is affected by a change in a single assumption, 

which helps the cost estimator to understand the 

extent to which each variable affects the cost 

estimate. Any sources of variation should be welldocumented and traceable.

• The documentation discusses some aspects of a 

risk and uncertainty analysis, such as a risk 

adjustment; however, only a few (4 out of 40) of 

the cost elements contained detailed 

documentation to support the risk adjustments. 

The majority of the cost elements (36 out of 40) 

did not have any risk adjustments applied. In 

addition, the documentation did not discuss what 
Appendix III: Assessments of Four FAA 

Program Cost Estimates

Page 101 GAO-12-223 Air Traffic Control Modernization

Characteristic of high-quality cost 

estimates/assessment 

criterion/explanation

Extent to which criterion was 

met Key examples of rationale for assessment

• Cross-checking of major cost elements.

A cross-check is done by using a different 

method to see if it produces similar results.

• A comparison to an independent cost 

estimate conducted by another 

organization.

A second, independent cost estimate should 

be performed by an organization outside of 

the program office’s influence. It should be 

based on the same technical baseline, 

ground rules, and assumptions, as the 

original estimate.

level of confidence was associated with the cost 

estimate. (Partially met.) The program estimate 

must reflect the degree of uncertainty, so that a 

level of confidence can be provided to 

management about the estimate. An estimate 

without risk and uncertainty analysis is unrealistic 

because it does not assess the variability in the 

cost estimate from such effects as schedules 

slipping, missions changing, and proposed 

solutions not meeting users’ needs.

• There was minimal evidence that cross-checks 

were identified. During the fiscal year 2004 

rebaseline, the basis of estimate was reviewed by 

several independent staff, but there was no 

evidence to supporting specific elements being 

cross-checked. (Minimally met.) The main 

purpose of cross-checking is to determine 

whether alternative methods produce similar 

results. If so, then confidence in the estimate 

increases, leading to greater credibility.

• There was no evidence that an independent cost 

estimate was conducted. However, as stated 

earlier, there was an independent review during 

the fiscal year 2004 rebaseline; since that time, 

the cost model has been reviewed several times. 

For example, in 2007 an independent evaluation 

analyzed eight cost issues. (Partially met.) An 

independent cost estimate is considered one of 

the best and most reliable estimate validation 

methods. It provides an independent view of 

expected program costs that tests the program 

office’s estimate for reasonableness. Without an 

independent cost estimate, decisions makers will 

lack insight into a program’s potential costs 

because independent cost estimates frequently 

use different methods and are less burdened with 

organizational bias.

Source: GAO analysis of FAA’s WAAS cost estimate.
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 102 GAO-12-223 Air Traffic Control Modernization

This appendix provides the results of our analysis of the extent to which 

the processes and methodologies used to develop and maintain four FAA 

integrated master schedules meet nine best practices associated with 

effective schedule estimating. The following tables provide the detailed 

results of our analyses of the schedules for the Automatic Dependent 

Surveillance-Broadcast (ADS-B), Collaborative Air Traffic Management 

Technologies (CATMT), System Wide Information Management (SWIM), 

and Wide Area Augmentation System (WAAS) programs compared to the 

nine best practices.

“Not met” means the program provided no evidence that satisfies any 

portion of the criterion. “Minimally met” means the program provided 

evidence that satisfies a small portion of the criterion. “Partially met” 

means the program provided evidence that satisfies about half of the 

criterion. “Substantially met” means the program provided evidence that 

satisfies a large portion of the criterion. “Fully met” means the program 

provided evidence that satisfies the entire criterion.

Table 12: Extent to Which the ADS-B Schedule Met Best Practices 

Best practice Explanation

Extent to which 

criterion was met GAO analysis

1. Capturing all 

activities

The schedule should reflect all 

activities as defined in the project’s 

work breakdown structure, which 

defines in detail the work 

necessary to accomplish a 

project’s objectives—including 

activities to be performed by both 

the owner and the contractors.

Partially met Our analysis found that detailed activities 

within the schedule are mapped to a fourlevel work breakdown structure, an 

organization breakdown structure, and 

unique control account numbers. Activities 

are further mapped to service areas, control 

account managers, and project leaders. 

Level-of-effort activities are included in the 

schedule and are clearly marked as such. In 

addition, several risk mitigation activities are 

included in the schedule. For example, 

activities associated with a contingency plan 

for the 2011 Mississippi River flood are 

included.

The government owns and maintains the 

program schedule for the ADS-B program, 

which covers all government and nonprime 

contract work through fiscal year 2013. The 

schedule is planned in 6-month rolling wave 

periods. Officials stated that they have not 

requested a detailed schedule deliverable 

from the prime contractor because the 

contractor portion represents work 

performed in the fixed-price subscription 

phase. The contractor owns a separate 

schedule that is not a required deliverable 

Appendix IV: Assessments of Four FAA 

Program Schedules
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 103 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

to the government. The government 

integrated master schedule includes prime 

contractor touch points that are updated 

monthly according to the contractor’s 

progress. However, without fully integrating

all contractor and government activities, we 

cannot guarantee that the schedule has 

adequately captured all key activities 

necessary for the program’s completion.

Officials stated that work needs to be 

accomplished in and beyond fiscal year 

2014, but the program schedule does not 

contain long-term planning activities beyond 

the current FAA-approved baseline. Effort 

from fiscal year 2015 to fiscal year 2020 is 

not included in the schedule because the 

program has no budget until Joint 

Resources Council approval of the next 

baseline. However, officials noted that the 

prime contract extends to 2025. Relevant 

guidance states that a comprehensive 

schedule should reflect all activities for a 

project and recognizes that there can be 

uncertainties and unknown factors in 

schedule estimates due to, among other 

things, limited data.

2. Sequencing all 

activities

The schedule should be planned 

so that critical project dates can be 

met. To meet this objective, 

activities need to be logically 

sequenced—that is, listed in the 

order in which they are to be 

carried out. In particular, activities 

that must be completed before 

other activities can begin 

(predecessor activities), as well as 

activities that cannot begin until 

other activities are completed 

(successor activities), should be 

identified. This helps ensure that 

interdependencies among activities 

that collectively lead to the 

accomplishment of events or 

milestones can be established and 

used as a basis for guiding work 

and measuring progress.

Minimally met Our analysis found 207 remaining activities, 

or work that remains to be accomplished 

(19 percent), are missing predecessor logic 

and 289 remaining activities (27 percent) 

are missing successor logic. These counts 

include 50 remaining activities (5 percent) 

that are missing both predecessor and 

successor logic.

In addition, our analysis found a number of 

activities with dangling logic. Each 

activity—other than the start and finish 

milestones—must have its start date 

driven by a previous activity, and each 

activity must drive the start date of another 

activity. Tasks with dangling logic can 

either carry on indefinitely without affecting 

downstream activities (finish date has no 

successor) or must start earlier in order to 

finish on time (start date has no 

predecessor). The schedule includes 372 

remaining activities, or 34 percent, with 

dangling successors.

The reliability of the schedule is also 

hindered by the presence of lags and 

leads (negative lags).
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 104 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

Lags represent the passing of time 

between activities but are often misused to 

put activities on a specific date or to insert 

a buffer for risk. In particular, 22 percent of 

the remaining activities are affected by at 

least one lag or lead. Lags should be 

justified because they cannot have risk or 

uncertainty. Furthermore, lags often 

represent work or a delay whose duration 

is unknown, yet are hard-coded into the 

schedule. Using lags rather than activities 

to account for work or to insert a buffer for 

risks causes problems because lags 

persist even when activities are delayed. 

As a result, when activities take longer 

than planned, the lags may actually distort 

the true logic of the schedule. Negative 

lags are generally not valid because 

negative time is not demonstrable; 

additionally, a lead implies the ability to 

predict the finish date of a predecessor 

activity with certainty. In general, leads can 

be replaced with straightforward finish-tostart logic once tasks are broken down 

further.

FAA officials said that there is broken and 

incomplete logic for the detail planning 

period (Wave “G”) that we assessed. They 

stated that the schedule we evaluated was 

in transition because of ongoing 

negotiations with the prime contractor over 

ADS-B service volume.

Having all interdependencies between 

tasks identified is necessary for the 

schedule to properly calculate dates and 

predict changes in the future. Without the 

correct linkages, tasks that slip early in the 

schedule do not transmit delays to tasks 

that should depend on them. When this 

happens, the schedule will not provide a 

sufficient basis for understanding the 

program as a whole, and users of the 

schedule will lack confidence in the dates 

and the critical path. Unless complete 

network logic is established, the schedule 

cannot predict impacts on the project’s 

planned finish date of, among other things, 

misallocated resources, delayed tasks, 

external events, and unrealistic deadlines.
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 105 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

3. Assigning resources 

to all activities

The schedule should reflect what 

resources (e.g., labor, materials, 

and overhead) are needed to do 

the work, whether all required 

resources will be available when 

needed, and whether any funding 

or time constraints exist.

Substantially met The ADS-B integrated master schedule is 

resource-loaded at the work package level 

by resource name, and contains 

approximately 250 resources, including 

both government and nongovernment fulltime-equivalents. Each has responsibility 

for managing the resources at that level. 

The program office leadership teams 

define what work will be done in each 

control account at a high level and which 

resources are available. The control 

account manager is then responsible for 

assigning resources at the work package 

level. Once the rolling wave planning is 

approved, resources are assigned 

accordingly with resource leveling 

occurring outside of the schedule. Officials 

use the integrated master schedule to 

track all resources that are not related to 

the prime contract (approximately 70 

percent of the baseline work).

Allocation reports are run during the initial 

rolling wave planning process to ensure 

resources are neither underallocated nor 

overallocated, but are not assessed 

monthly to track resource allocations along 

with schedule progress updates. However, 

allocations are assessed whenever a 

change request is initiated to ensure 

resources are available for these additional 

activities.

4. Establishing the 

duration of all 

activities

The schedule should realistically 

reflect how long each activity will 

take to execute. In determining the 

duration of each activity, the same 

rationale, historical data, and 

assumptions used for cost 

estimating should be used. 

Durations should be as short as 

possible and have specific start 

and end dates. The schedule 

should be continually monitored to 

determine when forecasted 

completion dates differ from 

planned dates; this information can 

be used to determine whether 

schedule variances will affect 

subsequent work.

Substantially met Our analysis found that approximately 75 

percent of the activities within the detail 

planning period of the schedule had 

original durations of 44 days or less, which 

generally meets best practices. Our 

analysis also found that all durations are 

consistently estimated in days and adhere 

to a standard 5-day workweek that 

accounts for holidays. Our analysis found 

no activities scheduled to begin on a 

weekend. Program officials stated that 

long-duration planning packages exist in 

the schedule for the period fiscal year 

2012 through fiscal year 2014 and that 

they hold high-level resources and budget 

meetings until officials are able to plan 

future rolling wave periods.

Officials stated that duration estimates are 

based on estimates from responsible 

control account managers who follow 

earned value management guidance. 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 106 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

However, these estimates are not formally 

justified or documented in a basis of 

estimate. The schedule aligns to an 

earned value management system, and 

the earned value technique used for each 

activity is stored in the schedule.

5. Integrating the 

schedule 

horizontally and 

vertically

The schedule should be 

horizontally integrated, meaning 

that it should link products and 

outcomes associated with other 

sequenced activities. These links 

are commonly referred to as 

“hand-offs” and serve to verify that 

activities are arranged in the right 

order to achieve aggregated 

products or outcomes. The 

schedule should also be vertically 

integrated, meaning that the dates 

for starting and completing 

activities in the integrated master 

schedule should be aligned with 

the dates for supporting tasks and 

subtasks. Such mapping or 

alignment among levels enables 

different groups to work to the 

same master schedule.

Partially met The schedule is vertically integrated, with 

low-level activities being traceable to 

higher-level summary activities in the work 

breakdown structure hierarchy. However, 

our analysis determined that the schedule 

is not horizontally integrated because of 

the issues described in Best Practice 2. 

Officials stated that all logic is not in place 

for the current detail planning period. 

Instead, the scheduler has used a 

combination of soft constraints, lags, and 

out-of-sequence updates in an attempt to 

match as closely as possible actual 

execution of activities while the 

deployment phase is under negotiation. 

Horizontal integration demonstrates that 

the overall schedule is rational, planned in 

a logical sequence, accounts for the 

interdependencies of activities, and 

provides a way to evaluate current status. 

Therefore, if the schedule lacks horizontal 

integration, the effects of slipped tasks on 

downstream work cannot be determined 

and there will be little confidence in the 

calculated dates or critical path.

6. Establishing the 

critical path for all 

activities

Scheduling software should be 

used to identify the critical path, 

which represents the chain of 

dependent activities with the 

longest total duration. Establishing 

a project’s critical path is 

necessary to examine the effects 

of any activity slipping along this 

path. Potential problems along or 

near the critical path should also 

be identified and reflected in 

scheduling the duration of high-risk 

activities.

Not met Officials stated that there is no critical path 

in the current schedule because of the 

incomplete logic noted in Best Practice 2. 

Officials stated that once the program 

began deploying service volumes, a critical 

path was difficult to derive because the 

contractor has the authority to implement 

service volumes in any order it chooses. 

However, now that negotiations for service 

volumes are complete, program officials 

plan to begin identifying a critical path to all 

Office of Management and Budget (OMB) 

Exhibit 300 (annual performance report) 

milestones.

Without a valid program critical path, 

management cannot determine which 

slipped tasks will have detrimental effects 

on the project finish date and it cannot 

determine if float within the schedule can 

be used to mitigate critical tasks by 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 107 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

reallocating resources from tasks that can 

safely slip to tasks that must be completed 

on time. Until the schedule can produce a 

true critical path, the program office will not 

be positioned to provide reliable timeline 

estimates and it will not be able to identify 

when problems or changes may occur and 

what effect they may have on downstream 

work.

7. Identifying 

reasonable float

The schedule should identify the 

float—the amount of time by which 

a predecessor activity can slip 

before the delay affects successor 

activities—so that a schedule’s 

flexibility can be determined. As a 

general rule, activities along the 

critical path have the least float. 

Total float is the total amount of 

time by which an activity can be 

delayed without delaying the 

project’s completion, if everything 

else goes according to plan.

Not met Officials stated that because of the missing 

logic in the current detail planning period, 

the float represented in the schedule is not 

realistic. Our analysis confirmed this, as 

we found many activities within the 

schedule with extremely high float; that is, 

unrealistically high float that is greater than 

199 days. Specifically, there are 12 (1 

percent) in-progress activities and 745 

planned activities (69 percent) that are 

able to slip more than 199 days without 

affecting the end date of the program.

Program officials stated they do not yet 

monitor float in the schedule because this 

particular version of the ADS-B schedule is 

in transition during deployment 

negotiations with the prime contractor. 

8. Conducting a 

schedule risk 

analysis

A schedule risk analysis (SRA) 

should be performed using 

statistical techniques to predict the 

level of confidence in meeting a 

project’s completion date. This 

analysis focuses not only on critical 

path activities but also on activities 

near the critical path, since they 

can affect the project’s status.

Minimally met Officials stated that a schedule risk 

analysis is not part of the rolling wave 

planning process. They perform 

uncertainty analysis on the cost estimate, 

but do not implement it on the schedule. 

Our analysis found this practice to be 

minimally met because, as noted in Best 

Practice 1, risk mitigation activities are 

entered into the schedule as baseline 

change requests. The program scheduler 

does not mark these as risk mitigation 

activities per se, but the documentation 

associated with a baseline change request 

will list the additional activities. An example 

of a risk mitigation activity is “Boxer 

Contingency Plan,” which was added for 

the 2011 flooding of the Mississippi River.
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 108 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

9. Updating the 

schedule using logic 

and durations to 

determine dates

The schedule should be 

continuously updated using logic 

and durations to determine realistic 

start and completion dates for 

program activities. The schedule 

should be analyzed continuously 

for variances to determine when 

forecasted completion dates differ 

from planned dates. This analysis 

is especially important for those 

variations that affect activities

identified as being in a project’s 

critical path and can influence a 

scheduled completion date.

Substantially met Our analysis found that the schedule has a 

valid and current status date. Our analysis 

also found there are no activities with start 

or finish dates in the past that are missing 

actual start and actual finish dates, and 

there are no activities with actual start and 

actual finish dates in the future.

Contractor updates are received by each 

control account manager in the form of 

expected finish dates, so the program 

scheduler is aware when a task is running 

late. Because the contractor has the 

authority to deploy service volumes in any 

order, officials are not overly concerned 

about the effect of a late task on the 

program as a whole. They stated that the 

control account managers know each 

service volume negotiated deadline and 

are aware of the effects of the slipped 

tasks. However, too many activities 

slipping into the future may create a bowwave of effort that the contractor may not 

have the resources to complete near the 

expected deployment completion date in 

fiscal year 2013. In addition, activities 

should be updated using actual duration or 

effort and estimates of remaining duration 

or effort. The scheduling software will 

calculate the percentage of completed 

activities and a forecasted finish date 

based on actual effort and planned effort 

remaining.

Our analysis found 60 activities (6 percent) 

as occurring out of sequence. That is, 

activities have started before their 

predecessor was completed, contrary to 

the network logic. Out-of-sequence 

progress should be corrected, either by 

changing the logical relationships to reflect 

what has actually happened or stopping 

work on the successor activity until the 

predecessor activity is complete (called 

“retained logic”). FAA officials stated they 

expect many activities to be done out of 

sequence as they adjust to the negotiated 

deployment dates with the contractor. In 

future rolling waves, as logic is put in place 

and negotiated dates are firm, they expect 

less out-of-sequence logic.

Source: GAO analysis of ADS-B schedule.
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 109 GAO-12-223 Air Traffic Control Modernization

Table 13: Extent to Which the CATMT Schedule Met Best Practices 

Best practice Explanation

Extent to which 

criterion was met GAO analysis

1. Capturing all 

activities

The schedule should reflect all 

activities as defined in the project’s 

work breakdown structure, which 

defines in detail the work necessary 

to accomplish a project’s objectives—

including activities to be performed by 

both the owner and the contractors.

Substantially met The scope of work for the CATMT project, which 

has no defined end state, is defined in 6-month 

increments. The schedule is owned and 

maintained by the prime contractor, who is 

responsible for capturing all activities authorized in 

the schedule. The schedule tracks governmentfurnished information and equipment deliverables 

and it covers all work in the statement of work. The 

work breakdown structure maps to a standardized 

FAA work breakdown structure but is functionally 

oriented rather than product-oriented in 

accordance with FAA standards.

Our analysis was able to successfully track detail 

work activities in the schedule to statement of work 

requirements, contract line item numbers, and task 

orders.

2. Sequencing all 

activities

The schedule should be planned so 

that critical project dates can be met. 

To meet this objective, activities need 

to be logically sequenced—that is, 

listed in the order in which they are to 

be carried out. In particular, activities 

that must be completed before other 

activities can begin (predecessor 

activities), as well as activities that 

cannot begin until other activities are 

completed (successor activities), 

should be identified. This helps 

ensure that interdependencies 

among activities that collectively lead 

to the accomplishment of events or 

milestones can be established and 

used as a basis for guiding work and 

measuring progress.

Partially met For the 481 remaining activities, or work that 

remains to be completed, we found few issues 

affecting the sequencing logic of the schedule. For 

example, all activities in the schedule had at least 

one predecessor and one successor activity.

However, our analysis found several issues that 

would affect the ability of the schedule to 

dynamically forecast future dates. For example, we 

found three Must Finish On constraints (1 percent 

of the remaining activities). A Must Finish On 

constraint means that the task, whether linked or 

not, must finish on a certain date. That is, it 

prevents the activity from finishing any earlier or 

later than a certain date, thereby overwriting 

network logic. The use of hard constraints is 

essentially the same as marking a date on a 

calendar and, therefore, defeats the purpose of 

using a dynamic scheduling tool. Officials stated 

that one Must Finish On constraint, the “Ready for 

Operations” milestone for Release 5, is used to 

create the critical path through Release 5, a viable 

reason that was not justified in documentation. 

Officials provided no justification for the remaining 

two Must Finish On constraints. We also found 24 

Start No Earlier Than constraints (5 percent of the 

remaining activities). Start No Earlier Than 

constraints are considered “soft” in that they allow 

the activity to slip into the future if their 

predecessor activity is delayed. However, the 

activity cannot begin earlier than its constraint 

date. Officials stated that the constraints were 

justified but only provided justification for 3 of the 

24 Start No Earlier Than constraints. For example, 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 110 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

officials stated “Tech Refresh Development Tech 

Ops” is an obsolete task that will be deleted from 

the schedule and removed from the contract. In 

addition, officials said a Start No Earlier Than 

constraint was in place to prevent multiple level-ofeffort tasks (management and other oversight 

related activities that continue until the detail 

activities they support have been completed) from 

being identified as critical activities. Activities with 

constraints typically are substitutes for logic and 

can mean that the schedule is not well planned 

and may not be feasible.

Our analysis also found 33 tasks with lags (7 

percent of the remaining activities), including 28 

activities with positive lags and 5 activities with 

leads (negative lags). According to officials, the 

lags are justified, but it is not their policy to 

document justifications. Lags should be justified 

because they cannot have risk or uncertainty. Lags 

often represent work or a delay whose duration is 

unknown, yet are hard-coded into the schedule. 

Activities represented by lags are not, in fact, risk 

free. Lags are often used to put activities on a 

specific date or to insert a buffer for risk. These 

lags persist even when activities are delayed and 

use up the buffer. When that happens, the lag may 

no longer be necessary. Leads are discouraged, 

as negative time is not demonstrable. Leads can 

often be replaced by additional tasks and linked 

using the appropriate scheduling logic. If used 

improperly, lags can distort float calculations in the 

schedule and corrupt the calculation of the critical 

path. 

3. Assigning 

resources to all 

activities

The schedule should reflect what 

resources (e.g., labor, materials, and 

overhead) are needed to do the work, 

whether all required resources will be 

available when needed, and whether 

any funding or time constraints exist.

Substantially met Our analysis found 48 resources in the schedule; 

however, resources are actually tracked outside of 

the schedule in detailed labor hour spreadsheets. 

Officials said this way of managing resources 

meets agency criteria. Moreover, they stated that 

because of the time frame represented by the 

schedule, the program may have three software 

releases in progress at any one time, which results 

in an overlapping of resources. Staffing levels are 

reviewed during the monthly management review 

meetings to ensure resources identified by the 

contractor map to the associated activities in the 

schedule.

Because resources are tracked outside the 

schedule rather than loaded within the schedule 

itself, management’s ability to fully determine 

whether all required resources will be available 

when needed, and whether any funding or time 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 111 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

constraints exist, is hampered. A resource-loaded 

schedule implies that all required labor and 

significant materials, equipment, and other costs 

are assigned to the appropriate activities within the 

schedule. Assigning resources to activities ensures 

that resources are used to determine activity 

durations because resource requirements directly 

relate to the duration of an activity.

4. Establishing the 

duration of all 

activities

The schedule should realistically 

reflect how long each activity will take 

to execute. In determining the 

duration of each activity, the same 

rationale, historical data, and 

assumptions used for cost estimating 

should be used. Durations should be 

as short as possible and have 

specific start and end dates. The 

schedule should be continually 

monitored to determine when 

forecasted completion dates differ 

from planned dates; this information 

can be used to determine whether 

schedule variances will affect 

subsequent work.

Met Excluding level-of-effort activities, more than 90 

percent of the remaining detailed activities meet 

best practices for having durations of 44 days or 

less. In addition, all activities are assigned to 

standard calendars that account for holidays.

In estimating durations, program officials stated 

that initially they used production rates for software 

lines of code to help determine durations. Now that 

the program is in the execution phase, officials 

have several years of history pertaining to 

production, industry standards, and defect rates on 

which to base duration estimates. Officials also 

said they try to keep the durations as short as 

possible, but activities such as coding are naturally 

3-month-long tasks whose duration cannot 

artificially be shortened to 44 days.

5. Integrating 

schedule 

activities 

horizontally and 

vertically

The schedule should be horizontally 

integrated, meaning that it should link 

products and outcomes associated 

with other sequenced activities. 

These links are commonly referred to 

as “hand-offs” and serve to verify that 

activities are arranged in the right 

order to achieve aggregated products 

or outcomes. The schedule should 

also be vertically integrated, meaning 

that the dates for starting and 

completing activities in the integrated 

master schedule should be aligned 

with the dates for supporting tasks 

and subtasks. Such mapping or 

alignment among levels enables 

different groups to work to the same 

master schedule.

Partially met Vertical integration—that is, the ability to 

consistently trace work breakdown structure 

elements between detailed and summary master 

schedules—is demonstrated in the schedule 

because lower-level activities and milestones roll 

up into higher-level summary activities. In addition, 

the schedule shows the dependencies between 

CATMT and other NextGen programs, such as 

EnRoute Automation Modernization (ERAM) and 

SWIM. Officials said in order to help manage these 

dependencies on ERAM and SWIM, CATMT 

development leads attend ERAM and SWIM 

weekly meetings and discuss risk dependencies. 

In addition, officials stated they also track the 

ERAM and SWIM schedules and any related 

schedule slips.

However, horizontal traceability is hampered by 

the reliance on date constraints and lags as 

discussed in Best Practice 2. For example, 

extending the durations of selected activities in the 

schedule more than 400 working days had no 

effect on the project finish date. If the schedule 

lacks horizontal integration, then the effects of 

slipped tasks on downstream work cannot be 

determined and there will be little confidence in the 

calculated dates or critical path. Any logic errors 

between detail, intermediate, and summary 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 112 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

schedules will cause inconsistent dates between 

schedules and will cause different expectations 

between senior management and activity owners.

6. Establishing the 

critical path for 

all activities

Scheduling software should be used 

to identify the critical path, which 

represents the chain of dependent 

activities with the longest total 

duration. Establishing a project’s 

critical path is necessary to examine 

the effects of any activity slipping 

along this path. Potential problems 

along or near the critical path should 

also be identified and reflected in 

scheduling the duration of high-risk 

activities.

Partially met Officials said because of the 6-month spiral 

development approach, the schedule cannot 

deliver a single critical path for the entire program. 

Instead, the critical paths are calculated and based 

on by software releases. To calculate a critical 

path by each software release, the prime 

contractor uses an end constraint on the key 

deliverable milestone for each software release.

At the time of our analysis, officials stated that only 

work related to Release 5 was subject to a critical 

path analysis because that was the focus of 

management. We were able to trace a contiguous 

critical path for Release 5, beginning with the 

project status date and ending with the Release 5 

finish milestone. However, the validity of the 

Release 5 critical path is hampered by the 

existence of seven in-progress and remaining 

detail activities within Release 5 that have over 

1,300 working days of total float. We also 

determined that a small number of activities 

outside of Release 5 were currently under way (for 

example, in other task orders). Because these 

activities fall outside of Release 5, and therefore 

outside the purview of a Release 5 critical path 

analysis, management may not be fully aware of 

the effect of any delay on these activities.

The critical path is directly related to the logical 

sequencing of events and float calculations. If the 

schedule is missing dependencies or if activities 

are not linked correctly, float estimates will be 

miscalculated. Incorrect float estimates will result 

in an invalid critical path and will hinder the ability 

of management to allocate resources from 

noncritical activities to those that must be 

completed on time. Without a valid critical path, 

there is no true program control because 

management cannot determine which slipped 

tasks will have detrimental effects on the project 

finish date. Until the schedule can produce a true 

critical path, the program office will not be 

positioned to provide reliable timeline estimates 

and it will not be able to identify when problems or 

changes may occur and what effect they may have 

on downstream work. 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 113 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

7. Identifying 

reasonable float

The schedule should identify the 

float—the amount of time by which a 

predecessor activity can slip before 

the delay affects successor 

activities—so that a schedule’s 

flexibility can be determined. As a 

general rule, activities along the 

critical path have the least float. Total 

float is the total amount of time by 

which an activity can be delayed 

without delaying the project’s 

completion, if everything else goes 

according to plan.

Minimally Met While officials said the schedule does not contain 

excessive float values and float values are 

monitored as part of the schedule’s health 

assessment, our analysis found unreasonable 

amounts of total float throughout the schedule. For 

example, 325 (68 percent) of the 481 remaining 

activities have float values greater than 1,000 

days. These unreasonable float values are due 

mostly to activities being tied to the project finish 

milestone, which is constrained to start no earlier 

than July 1, 2016. Interim milestones that are 

scheduled to finish earlier, such as those marking 

the end of task orders, are tied to the project finish 

milestone as predecessors and are, therefore, 

showing large amounts of float that do not reflect 

actual flexibility in the schedule. The majority of 

high-float activities are level-of-effort activities, 

many of which are extended to the end of these 

interim milestones and are thus associated with 

unreasonable amounts of float as well. As noted in 

our analysis of the schedule’s adherence to Best 

Practice 6, a hard constraint is in place to calculate 

a critical path through Release 5 activities; 

however, our analysis also found unreasonable 

amounts of total float in some of these activities. 

For example, several activities have more than 

1,000 days of float, including “Test and Deploy”—

which shows a total float value of 1,280 days. 

Because of this excessive float, delays in the 

activities above will have no effect on the finish 

date of Release 5.

Float estimates are directly related to the logical 

sequencing of activities and, therefore, if the 

schedule is not properly sequenced, float 

calculations will be miscalculated. Without reliable 

float estimates, management may be unable to 

allocate resources from noncritical activities to 

activities that cannot slip without affecting the 

project finish date.
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 114 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

8. Conducting a 

schedule risk 

analysis

A schedule risk analysis should be 

performed using statistical techniques 

to predict the level of confidence in 

meeting a project’s completion date. 

This analysis focuses not only on 

critical path activities but also on 

activities near the critical path, since 

they can affect the project’s status.

Not met A risk analysis of the schedule’s vulnerability to 

slippages in the completion of activities has not 

been performed. Officials stated that a schedule 

risk analysis was not conducted because it is not a 

contractual deliverable. However, to address 

schedule risk with software development, the 

contractor performed cross-check and risk 

analyses on estimated lines of code using industry 

standard software development estimating tools. In 

addition, the officials stated they perform what-if 

analyses and discuss risks during their monthly 

earned value management analysis reviews.

However, a comprehensive schedule risk analysis 

is an essential tool for decision makers. A 

schedule risk analysis can be used to determine a 

level of confidence in meeting the completion date 

or whether proper reserves have been 

incorporated into the schedule. A schedule risk 

analysis will calculate schedule reserve, which can 

be set aside for those activities identified as high 

risk. Without this reserve, the program faces the 

risk of delays to the scheduled completion date if 

any delays occur on critical path activities.

9. Updating the 

schedule using 

logic and 

durations to 

determine the 

dates

The schedule should be continuously 

updated using logic and durations to 

determine realistic start and 

completion dates for program 

activities. The schedule should be 

analyzed continuously for variances 

to determine when forecasted 

completion dates differ from planned 

dates. This analysis is especially 

important for those variations that 

affect activities identified as being in 

a project’s critical path and can 

influence a scheduled completion 

date.

Substantially met The schedule is updated by the contractor weekly, 

and the contractor also provides a formal update of 

the schedule to the program office on a monthly or 

bimonthly basis as necessary. Officials stated that 

during the monthly schedule reviews, FAA is an 

active participant in discussions involving network 

logic and forecasted activity dates.

We found 47 activities (10 percent of remaining 

activities) with start dates in the past or with no 

actual start dates and 49 activities (10 percent of 

remaining activities) with finish dates in the past 

with no actual finish dates. The program office said 

the majority of these data anomalies are 

associated with “on-hold” activities. After reviewing 

the schedule, we did find that majority of these 

activities were labeled as “on-hold.” We found no 

activities in the schedule with actual start dates in 

the future or actual finish dates in the future.

Source: GAO analysis of CATMT schedule.
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 115 GAO-12-223 Air Traffic Control Modernization

Table 14: Extent to Which the SWIM Schedule Met Best Practices 

Best practice Explanation

Extent to which 

criterion was met GAO analysis

1. Capturing all 

activities

The schedule should reflect all 

activities as defined in the project’s 

work breakdown structure, which 

defines in detail the work 

necessary to accomplish a 

project’s objectives—including 

activities to be performed by both 

the owner and the contractors.

Minimally met Our analysis found that, despite a substantial 

amount of detail and clearly marked level-ofeffort activities, the program schedule does not 

effectively represent the work required to 

complete SWIM, primarily because of the way 

in which the schedule was created and the 

limited authority of the SWIM program office. 

The Integrated Schedule is a synopsis of the 

individual SWIM implementation program (SIP) 

schedules and includes only activities whose 

timeliness is assumed to be crucial to meeting 

Joint Resources Council milestone dates, 

resulting in a schedule that likely excludes a 

portion of the program’s scope. Because of 

missing activities, it is unlikely that this 

schedule will highlight opportunities for process 

improvement (e.g., identifying redundant 

activities), what-if analysis, and risk mitigation.

The SWIM program office does not have 

purview over the content that is included in the 

individual SIP schedules. Instead, this 

responsibility is held by FAA’s Joint Resources 

Council secretariat. Thus, the SWIM program 

office cannot enforce the use of a standard 

work breakdown structure for all SIP schedules 

or dictate the way in which the SIPs name their 

activities. Our analysis found repetitive naming 

of activities across SIP schedules that are not 

associated with specific products or phases. 

This makes communication difficult between 

teams, particularly between team members 

that are responsible for updating and 

integrating multiple schedules.

If the schedule does not fully and accurately 

reflect the project’s activities, it will not serve as 

an appropriate basis for analysis and may 

result in unreliable completion dates, time 

extension requests, and delays. Since the 

schedule is used for coordination, missing 

elements will hinder coordination efforts, 

increasing the likelihood of disruption and 

delays. In addition, if activities are missing from 

the schedule, then other best practices will not 

be met. Without accounting for all necessary 

activities, it is uncertain whether: all activities 

are scheduled in the correct order, resources 

are properly allocated, missing activities would 

appear on the critical path, or a schedule risk 

analysis accounts for all risk.
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 116 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

2. Sequencing all 

activities

The schedule should be planned 

so that critical project dates can be 

met. To meet this objective, 

activities need to be logically 

sequenced—that is, listed in the 

order in which they are to be 

carried out. In particular, activities 

that must be completed before 

other activities can begin 

(predecessor activities), as well as 

activities that cannot begin until 

other activities are completed 

(successor activities), should be 

identified. This helps ensure that 

interdependencies among activities 

that collectively lead to the 

accomplishment of events or 

milestones can be established and 

used as a basis for guiding work 

and measuring progress.

Minimally met Our analysis found that the activities in the 

SWIM Integrated Schedule were not properly 

sequenced because of missing dependencies 

and the presence of dangling activities, date 

constraints, and lags. Dangling logic reduces the 

credibility of the calculated activity start and 

finish dates and the identity of the critical paths 

because the slip or elongation of an activity that 

has no logical successor will not affect the 

scheduled dates of successor activities.

In addition, the schedule contains 23 constraints 

(6 percent of the remaining activities), or work 

that remains to be completed, including 13 soft 

constraints and 10 hard constraints. Program 

officials stated that hard constraints were added 

to enable the creation of the critical path for the 

subprogram. However, officials stated that float 

is not monitored. If float is not monitored, the 

critical path may be invalid, and there is thus no 

credible reason to include these constraints in 

the schedule.

Constraints jeopardize the ability of the schedule 

to remain dynamic—that is, to compute the 

correct delivery dates and critical path if 

durations change. Constraints artificially set 

dates in the software and may mislead users of 

the schedule, since they do not force the actual 

project to do anything. A large number of 

activities with constraints typically are substitutes 

for logic and can mean that the schedule is not 

well planned and may not be feasible.

The quality of the schedule is significantly 

degraded by the presence of lags and leads 

(negative lags). In particular, 61 percent of the 

remaining activities are affected by at least one 

lag or lead. Program officials explained that lags 

are used to represent collections of activities that 

were present in the original SIP schedules but 

deemed noncrucial during the Integrated 

Schedule creation process. The purpose of 

these lags is then to preserve the underlying 

network logic and timeline. However, in this 

case, a summary activity would be a more 

appropriate modeling choice, since it would 

allow for logic preservation, but also maintain the 

dynamic nature of the schedule.

Using lags rather than activities to account for 

work or to insert a buffer for risks causes 

problems because lags persist even when 

activities are delayed. For this reason, lags 

should be eliminated and replaced with activities 

so they can be tracked. Furthermore, lags 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 117 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

lessen the ability of the project schedule to 

dynamically respond to changes in the status of 

predecessor activities, and the lags’ need to be 

constantly manually updated defeats the 

purpose of a dynamic schedule and can cause 

errors in the schedule when there are too many 

lags to maintain.

3. Assigning resources 

to all activities

The schedule should reflect what 

resources (e.g., labor, materials, 

and overhead) are needed to do 

the work, whether all required 

resources will be available when 

needed, and whether any funding 

or time constraints exist.

Not met The schedule is not resource-loaded. 

According to program officials, the SWIM 

program is somewhat atypical with respect to 

resource loading. In particular, the substantive 

effort for the program is handled by SIP 

support contractors and the SWIM program 

office serves to monitor the SIPs’ progress and 

ensure that deliverables are completed on 

time. Moreover, in accordance with the 

program’s federated approach, the SWIM 

program office does not dictate the content that 

is included in the individual SIP schedules. As 

a result, the SIPs are not required to resourceload their schedules and, thus, the Integrated 

Schedule itself is not resource loaded.

Because this schedule does not have 

resource assignments, it cannot be used to 

monitor productivity or resource-constrained 

activities, allocate idle resources, or level 

resources across activities.

4. Establishing the 

duration of all 

activities

The schedule should realistically 

reflect how long each activity will 

take to execute. In determining the 

duration of each activity, the same 

rationale, historical data, and 

assumptions used for cost 

estimating should be used. 

Durations should be as short as 

possible and have specific start 

and end dates. The schedule 

should be continually monitored to 

determine when forecasted 

completion dates differ from 

planned dates; this information can 

be used to determine whether 

schedule variances will affect 

subsequent work.

Partially met The requirements for compliance with this best 

practice have been partially met, primarily 

because of the nature of the schedule creation 

and update processes. Durations for many of 

the activities in the schedule are significantly 

longer than 2 working months (44 working 

days). Specifically, 190 (55 percent) of the 

remaining activities have a duration longer than 

the threshold of 44 days, and half of the 

activities are scheduled for longer than 66 

days. Furthermore, the average duration of all 

remaining activities is 158 days, or roughly 7 

working months. Such a schedule is likely to be 

very challenging to manage and is indicative of 

insufficient detail. Long-duration activities also 

make it difficult for management to gauge 

progress and can skew schedule risk analysis 

results.

Officials stated that many of these lengthy 

durations are the result of the way in which 

the schedule was created. Those activities 

that are not level-of-effort were originally 

summary activities in the SIP schedule from 

which they were extracted and, thus, are 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 118 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

composed of many detailed activities with

shorter durations.

In addition, because the method by which 

activity durations are estimated is not within 

the purview of the SWIM program office, there 

is little insight into how the durations are 

derived. However, as noted in the discussion 

of Best Practice 3, there is no requirement for 

the SIP contractor schedules to be resourceloaded. Because activity durations are based 

on many underlying assumptions, including 

the number of hours per workweek, number of 

full-time-equivalent personnel, productivity 

rates, and other factors, they should be 

estimated under normal conditions, not 

optimal conditions. If durations are not 

realistic, program management does not have 

the information necessary to properly allocate 

resources to tasks and understand how those 

tasks affect downstream work.

5. Integrating the 

schedule horizontally 

and vertically

The schedule should be 

horizontally integrated, meaning 

that it should link products and 

outcomes associated with other 

sequenced activities. These links 

are commonly referred to as 

“hand-offs” and serve to verify that 

activities are arranged in the right 

order to achieve aggregated 

products or outcomes. The 

schedule should also be vertically 

integrated, meaning that the dates 

for starting and completing 

activities in the integrated master 

schedule should be aligned with 

the dates for supporting tasks and 

subtasks. Such mapping or 

alignment among levels enables 

different groups to work to the 

same master schedule.

Minimally met This schedule suffers from a lack of horizontal 

integration for reasons identified in other Best 

Practice responses. The schedule failed 

several tests for horizontal integration, primarily 

because of the hard constraints on the SIP 

completion milestones. For example, the 

duration of a critical activity was increased by 

50 days, changing its float (and that of its 

predecessor) from 0 to 50 days and its finish 

date from February 28, 2011, to May 9, 2011, 

but without any effect on the project completion 

date. Further increasing this activity’s duration 

by 1,500 days had the expected effect on the 

finish dates of the activity and its successor, 

but left the project’s completion date 

unchanged and several years before the new 

finish date of the critical activity. Horizontal 

integration demonstrates that the overall 

schedule is rational, planned in a logical 

sequence, accounts for interdependent 

activities, and provides a way to evaluate 

current status. Therefore, if the schedule lacks 

horizontal integration, the effects of slipped 

tasks on subsequent work cannot be 

determined and there will be little confidence in 

the calculated dates or critical path.

Schedules that are not horizontally integrated 

may not depict relationships between different 

program elements and product hand-offs. 

When this happens, product hand-offs of 

project subcomponents cannot be fully traced 

to the end product, leading to less effective 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 119 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

management of the project. According to 

officials, all parties involved in the creation 

and updating of the SWIM program office 

schedules meet routinely for status reviews 

and have signed off on all “touch points” and 

hand-offs. However, because of the structure 

of the schedule, it is difficult to determine 

where key hand-offs are made and how they 

are related to top-level milestone dates.

As noted in the discussion of Best Practice 1, 

there are several inconsistencies between 

lower- and upper-level milestone dates. In 

addition, this schedule does not contain all 

lower-level activities but, rather, only a 

selection of activities from the lower-level SIP 

schedules. Any logic errors between detail, 

intermediate, and summary schedules will 

cause inconsistent dates between schedules, 

as well as different expectations between 

senior management and activity owners.

6. Establishing the 

critical path for all 

activities 

Scheduling software should be 

used to identify the critical path, 

which represents the chain of 

dependent activities with the 

longest total duration. Establishing 

a project’s critical path is 

necessary to examine the effects 

of any activity slipping along this 

path. Potential problems along or 

near the critical path should also 

be identified and reflected in 

scheduling the duration of high-risk 

activities.

Minimally met Based on the description of critical path 

management practices provided by the SWIM 

program office, we analyzed this best practice 

from the traditional, top-level perspective, as 

well as from that of the individual SIPs. In 

both cases, we found that the requirements 

for compliance were only minimally met. For 

each analysis that we performed, we traced 

the paths of activities with the least amount of 

total float since they are considered “drivers” 

of a critical finish milestone.

Program officials stated that, although the SIP 

schedules are integrated in the schedule, the 

SIPs are unrelated and, thus, each has its own 

critical path. However, the full individual critical 

paths are not accessible through the Integrated 

Schedule, and the SIP schedules are not used 

by the SWIM program office for management 

purposes, so their existence offers little

advantage from a SWIM management 

perspective. Furthermore, the SWIM program 

itself should have its own critical path, which 

includes, at the very least, acceptance of all 

major deliverables from the SIPs.

We first examined the SWIM-wide critical path 

identified by the scheduling software. This 

critical path was found to be invalid for a variety 

of reasons, including the inclusion of line-ofeffort activities, date constraints, activities with 

excessive durations, and lags and leads. 

Additionally, issues identified during the 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 120 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

analysis of other best practices—particularly 

Best Practices 1 and 2—prevent this critical 

path from being valid. Without a valid program 

critical path, management cannot determine 

which slipped tasks will have detrimental 

effects on the project finish date.

We also analyzed the schedule from the 

perspective of each SIP to determine the 

validity of individual critical paths. In each 

case, the critical path failed to meet the 

requirements for this best practice because of 

the presence of one or more of the following: 

line-of-effort activities, lags, leads, long 

durations, program management activities, 

and excessive float.

7. Identifying

reasonable float

The schedule should identify the 

float—the amount of time by which 

a predecessor activity can slip 

before the delay affects successor 

activities—so that a schedule’s 

flexibility can be determined. As a 

general rule, activities along the 

critical path have the least float. 

Total float is the total amount of 

time by which an activity can be 

delayed without delaying the 

project’s completion, if everything 

else goes according to plan.

Minimally met Program officials noted that total float is within 

acceptable limits for the project but that float 

is not monitored by the SWIM program office. 

However, our analysis found unreasonable 

float calculations in the schedule. In particular, 

107 (28 percent) of the 393 remaining detail 

and milestone activities have float values 

greater than or equal to 100 working days; 

that is, 27 percent of all remaining activities 

can slip more than 4 working months without 

affecting successor activities. Additionally, two 

activities with excessive negative float values 

(95 working days) were discovered.

The abnormal float values are primarily 

caused by the schedule integration process. 

In particular, these high values arise because 

of large differences between the finish dates 

of individual SIP capabilities and the SWIM 

program completion date. Program officials 

noted that adding links between the 

completion dates of the individual SIP 

capabilities and the project completion 

milestone (in order to lower the float values) is 

unnecessary since float management is not 

used by the SWIM program office. Officials 

stated that total float is not monitored because 

the schedule is not resource-loaded and the 

work is performed at the SIP level. In addition 

to the integration process, other possible 

causes of excessive float are the missing 

dependencies and constraints noted in the 

discussion of Best Practice 2.

Float is directly related to the logical 

sequencing of events and the critical path. If 

the schedule is missing activities or 

dependencies, or links activities incorrectly, 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 121 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

float estimates will not be accurate. Incorrect 

float estimates may result in an invalid critical 

path and an inaccurate assessment of project 

completion dates. In addition, inaccurate 

values of total float result in a false depiction 

of true project status, which could lead to 

decisions that may jeopardize the project. 

8. Conducting a 

schedule risk 

analysis

A schedule risk analysis should be 

performed using statistical 

techniques to predict the level of 

confidence in meeting a project’s 

completion date. This analysis 

focuses not only on critical path 

activities but also on activities near 

the critical path, since they can 

affect the project’s status.

Minimally met No schedule risk analysis was conducted for 

this schedule. However, our analysis found 

that this best practice was minimally met 

because a risk analysis was conducted on a 

separate higher-level SWIM schedule and risk 

management has been considered, to some 

extent, by the SWIM program office.

The schedule risk analysis was performed on 

a high-level schedule, consisting of standard 

work packages designed to represent a 

generic software development program, in 

order to satisfy Joint Resources Council 

requirements. According to the SWIM 

program office, the SWIM Joint Resources 

Council met with each of the SIP program 

offices to develop high-level schedules that 

were risk adjusted and priced for the original 

cost estimate. These individual schedules 

were combined, resulting in a schedule with 

182 activities. This risk analysis does not 

comply with this best practice because the 

underlying schedule was created using 

standard work packages, not specific to 

SWIM, and does not accurately capture the 

relationships between the SIP capabilities and 

the success of the SWIM program. 

Furthermore, even if a risk analysis had been 

performed on the Integrated Schedule, 

several issues have been identified that would 

call its validity into question. If the schedule 

risk analysis is to be credible, the program 

must have a quality schedule that reflects 

reliable logic and clearly identifies the critical 

path—conditions that the SWIM program’s 

Integrated Schedule does not meet.

Program officials have stated that it has not 

been practice to perform risk analysis on the 

Integrated Schedule because it is a derivative 

of several lower-level schedules and does not 

capture the work of the original program. 

However, a schedule risk analysis should be 

conducted on each schedule used for 

planning purposes, in order to provide a more 

realistic view of the expected project timeline. 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 122 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

9. Updating the 

schedule using logic 

and durations to 

determine dates

The schedule should be 

continuously updated using logic 

and durations to determine realistic 

start and completion dates for 

program activities. The schedule 

should be analyzed continuously 

for variances to determine when 

forecasted completion dates differ 

from planned dates. This analysis 

is especially important for those 

variations that affect activities 

identified as being in a project’s 

critical path and can influence a 

scheduled completion date.

Partially met Our analysis found that the requirements for 

compliance with this best practice have been 

partially met. According to program officials, 

the schedule is updated regularly during 

monthly status meetings. This has been 

partially confirmed by the latest schedule 

status on January 31, 2011, shortly before the 

program office transmitted it to us. Despite the 

regularity of these meetings, however, the 

update process described by program officials 

is quite labor-intensive and likely prone to 

error. 

We discovered evidence of the inherent error 

in this process through our analysis. Although 

the schedule was recently updated, there 

were 32 activities with start dates in the past 

but without a recorded actual start date, and 

46 activities with finish dates in the past but 

without an actual finish date entered in the 

schedule. If these activities have started 

(finished), their actual start (finish) date 

should be updated to reflect this 

accomplishment. If not, the start (finish) date 

should be updated to reflect the current plan.

Additionally, our analysis identified 28 

activities in the schedule with finish dates that 

occurred before the status date, 58 activities 

with remaining durations exceeding their 

planned finish date, and 21 activities updated 

out of sequence. The existence of these date 

anomalies indicates error in the durationestimating methodology or neglect in the 

schedule-updating process, and a schedule 

with remaining out-of-sequence progress may 

have the wrong logic in place and hence have 

inaccurate critical paths and completion 

dates. 

Source: GAO analysis of SWIM schedule.
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 123 GAO-12-223 Air Traffic Control Modernization

Table 15: Extent to Which the FAA-Prepared WAAS Schedule Met Best Practices 

Best practice Explanation

Extent to which 

criterion was met GAO analysis

1. Capturing all 

activities

The schedule should reflect all 

activities as defined in the project’s 

work breakdown structure, which 

defines in detail the work necessary to 

accomplish a project’s objectives—

including activities to be performed by 

both the owner and the contractors.

Minimally met Our analysis found that the program schedule 

is not fully integrated. Program officials said 

the government’s integrated master schedule, 

which is maintained by FAA, is made up of 15 

subschedules; however, these subschedules 

are not linked to the government’s integrated 

master schedule. Furthermore, identification of 

these subschedules in the government’s 

integrated master schedule is difficult because 

that schedule does not contain any custom 

text fields, statements of work, or contract line 

Item mappings. Without these mappings, it is 

impossible to determine which activities in the 

government’s integrated master schedule 

belong to which subschedules and whether all 

the work has been captured.

In addition, we found that the work breakdown 

structure for the government’s integrated 

master schedule does not match the work 

breakdown structure used for the cost 

estimate or the work breakdown structure 

used in the program’s Earned Value 

Management System Control Plan. Because 

the work breakdown structure is not 

standardized, we cannot be sure that the 

schedule contains all of the work necessary to 

accomplish the program’s objectives, and it is 

difficult, if not impossible, to identify activities 

across cost, schedule, and earned value 

management reporting.

Without an integrated master schedule that 

accounts for all planned government and 

contractor effort, management is not able to 

reliably estimate planned dates beyond the 

current schedule’s end date of September 

2016. Moreover, unless all activities are 

accounted for, it is uncertain whether all 

activities are properly sequenced, resources 

are properly assigned, the critical path reflects 

all activities, or a schedule risk analysis 

accounts for all risk. Finally, if the schedule 

does not fully and accurately reflect the 

project’s activities, it will not serve as an 

appropriate basis for analysis and may result 

in unreliable completion dates, time extension 

requests, and delays.
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 124 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

2. Sequencing all 

activities

The schedule should be planned so 

that critical project dates can be met. 

To meet this objective, activities need 

to be logically sequenced—that is, 

listed in the order in which they are to 

be carried out. In particular, activities 

that must be completed before other 

activities can begin (predecessor 

activities), as well as activities that 

cannot begin until other activities are 

completed (successor activities), 

should be identified. This helps ensure 

that interdependencies among 

activities that collectively lead to the 

accomplishment of events or 

milestones can be established and 

used as a basis for guiding work and 

measuring progress.

Minimally met Our analysis found 116 remaining activities—

or remaining work to be completed. (36 

percent) missing predecessor logic and 162 

activities (48 percent) missing successor logic.

In addition, our analysis found a number of 

activities with constraints. The schedule 

includes 198 activities (49 percent) with Start 

No Earlier Than constraints. These constraints 

are considered “soft” in that they allow the 

activity to slip into the future if their 

predecessor activity is delayed. However, the 

activity cannot begin earlier than its constraint 

date. We also found 92 activities (23 percent) 

with Finish No Earlier Than constraints. These 

constraints are also considered “soft” date 

constraints because they prevent activities 

from finishing earlier than their constraint date.

The reliability of the schedule is also hindered 

by the presence of lags and leads (negative 

lags). In particular, 38 activities (9 percent) are 

affected by at least one lag or lead. Lags 

should be justified because they cannot have 

risk or uncertainty. Furthermore, lags often 

represent work or a delay whose duration is 

unknown, yet are hard-coded into the schedule. 

Using lags rather than activities to account for 

work or to inset a buffer for risks causes 

problems because lags persist even when 

activities are delayed. As a result, when 

activities take longer, the lags may actually 

distort the true logic of the schedule. Negative 

lags are generally not valid because time is not 

demonstrable; moreover, a lead implies the 

ability to predict the finish date of a predecessor 

activity with certainty. In general, leads can be 

replaced with straightforward finish-to-start logic 

once tasks are broken down further.

Having all interdependencies between tasks 

identified is necessary for the schedule to 

properly calculate dates and predict changes. 

Without the right linkages, tasks that slip early 

in the schedule do not transmit delays to tasks 

that should depend on them. When this 

happens, the schedule will not provide a 

sufficient basis for understanding the program 

as a whole, and users of the schedule will lack 

confidence in the dates and the critical path. 

Unless complete network logic is established, 

the schedule cannot predict effects on the 

project’s planned finish date of, among other 

things, misallocated resources, delayed tasks, 

external events, and unrealistic deadlines.
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 125 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

3. Assigning 

resources to all 

activities

The schedule should reflect what 

resources (e.g., labor, materials, and 

overhead) are needed to do the work, 

whether all required resources will be 

available when needed, and whether 

any funding or time constraints exist.

Minimally met Program officials stated that the schedule is 

not resource-loaded, and our analysis 

confirmed that resources are not assigned to 

activities. Program officials said the general 

resource allocation process begins with the 5-

year Strategic Plan that lays out the program’s 

long-term goals.

Program officials said given the current 

political and budget environment, it is very 

difficult to hire government employees, and, as 

a result, it is easier to fill resource gaps with 

contractor personnel than with government 

personnel.

Assigning resources to activities ensures that 

resources are used to determine activity 

durations because resource requirements 

directly relate to the duration of an activity. 

Labor, material, equipment, and funding 

requirements are examined to determine the 

feasibility of the schedule, so that resources 

provide a benchmark of the total and perperiod cost of the project. If the current 

schedule does not allow for insight into current 

or projected allocation of resources, then the 

risk of the program slipping is significantly 

increased.

4. Establishing the 

duration of all 

activities

The schedule should realistically 

reflect how long each activity will take 

to execute. In determining the duration 

of each activity, the same rationale, 

historical data, and assumptions used 

for cost estimating should be used. 

Durations should be as short as 

possible and have specific start and 

end dates. The schedule should be 

continually monitored to determine 

when forecasted completion dates 

differ from planned dates; this 

information can be used to determine 

whether schedule variances will affect 

subsequent work.

Partially met Our analysis found that more than 70 percent 

of the remaining activities within the detail 

planning period of the schedule had original 

durations of 44 days or less, which generally 

meets best practices. Forty-one activities had 

long durations, of between 200 and 630 days 

that appear to be based on a fiscal year’s 

period of work, without regard to deliverables 

either at the end or at an intermediate point of 

shorter duration. Usually activities in the 

schedule are organized around producing 

deliverables rather than on the end of the 

calendar or fiscal year, or of a 12-month 

period.

Program officials said the team uses past 

experiences to estimate activity durations. 

Officials also said durations are determined in 

meetings between program engineers, control 

account managers, and upper management. 

We found no deadlines in the schedule, and 

program officials noted that there is no 

mandated date for the schedule.

Although the activity durations in the 

program’s schedule appear to be reasonable, 

not assigning resources in the schedule 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 126 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

directly affects the estimated work and 

duration of activities. Assigning resources to 

activities inside the schedule ensures that 

resources are used to determine activity 

durations because resource requirements 

directly relate to the duration of an activity. 

Therefore, because the schedule does not 

include adequate resources, it is questionable 

that the activity durations realistically reflect 

how long each activity will take to execute. 

Furthermore, only one activity had a calendar 

assigned to it. All of the other activities had no 

calendar assignment, which means that 

holidays were not accounted for in this 

schedule, immediately calling into question the 

validity of the calculated dates.

5. Integrating 

schedule 

activities 

horizontally and 

vertically

The schedule should be horizontally 

integrated, meaning that it should link 

products and outcomes associated 

with other sequenced activities. These 

links are commonly referred to as 

“hand-offs” and serve to verify that 

activities are arranged in the right 

order to achieve aggregated products 

or outcomes. The schedule should 

also be vertically integrated, meaning 

that the dates for starting and 

completing activities in the integrated 

master schedule should be aligned 

with the dates for supporting tasks 

and subtasks. Such mapping or 

alignment among levels enables 

different groups to work to the same 

master schedule.

Partially met The results for Best Practice 2, Sequencing of 

All Activities, and Best Practice 7: Identifying 

Reasonable Float, on activities and paths 

indicate that the logic is not in place to 

demonstrate 

vertical integration in the integrated master 

schedule.

The WAAS program office said vertical 

integration is demonstrated in the contractor 

schedules, because the lower-level schedules 

map to milestones in the integrated master 

schedule. However, we found that the dates 

for two key milestones in the detailed 

contractor schedule did not match the dates in 

the government’s integrated master schedule, 

indicating a problem with vertical integration 

among the detailed and higher-level 

schedules.

Program officials said horizontal integration is 

identified in the schedule when activities are 

handed off from givers to receivers, but our 

analysis could not identify these hand-offs in 

the schedule. In addition, the schedule does 

not fully demonstrate horizontal traceability 

because it does not include complete logic 

from program start to program finish and does 

not fully integrate the entire scope of work for 

all parties involved in the program.

Program officials said the WAAS program 

uses both the acquisition program baseline 

and the Office of Management and Budget 

(OMB) Exhibit 300 to track the program.a

However, our analysis found that the WAAS 

integrated master schedule only reflects work 

approved through 2016, even though the 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 127 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

acquisition program baseline and OMB Exhibit 

300 have estimated work planned beyond that 

date. Without horizontal traceability, there is 

no assurance that the forecasted dates within 

the schedule will be determined by network 

logic and progress to date rather than artificial 

constraints.

We also tested horizontal integrity by 

extending the durations of key activities for 

Release 3B, the current release during the 

GAO audit, to observe the effects on 

successor activity dates and the project finish 

date. Extending the durations of selected 

activities in the schedule by more than 1,000 

working days had no effect on the successor 

activities or the project finish date.

Unless a schedule is fully horizontally 

integrated, the effects of slipped tasks on 

succeeding work cannot be determined. 

Horizontal integration demonstrates that an 

overall schedule is rational, planned in a 

logical sequence, accounts for 

interdependencies between work and planning 

packages, and provides a way to evaluate a 

program’s current status. When schedules are 

not horizontally integrated, relationships 

between different program teams cannot be 

seen and product hand-offs cannot be 

identified. Vertical integration traces the 

consistency of data between work breakdown 

structure elements within the layers of the 

schedule—master, intermediate, detailed. 

Without schedules that are vertically 

integrated, lower-level schedules cannot be 

clearly traced to upper-tiered milestones, and, 

as a result, total schedule integrity is 

compromised. This can hamper the ability of 

different teams to use the schedule to manage 

expectations.

6. Establishing the 

critical path for 

all activities

Scheduling software should be used 

to identify the critical path, which 

represents the chain of dependent 

activities with the longest total 

duration. Establishing a project’s 

critical path is necessary to examine 

the effects of any activity slipping 

along this path. Potential problems 

along or near the critical path should 

also be identified and reflected in 

scheduling the duration of high-risk 

activities.

Minimally met Officials stated that there is no critical path in 

the current schedule, but they know what the 

critical activities are in the subschedules that 

make up the integrated master schedule. As 

evidence, they pointed out hard-copy versions 

of five detail schedule critical paths for the 

subschedules, including those for the safety 

computer. However, we were unable to find a

valid “critical path” in the schedule itself. Both 

critical paths in the schedule start with a Start 

No Earlier Than constraint, which is not in line 

with best practices. Moreover, one of the 

paths includes several activities with lags that 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 128 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

equate to 39 months (approximately 854 

working days) of unexplained duration.

Without a valid program critical path, 

management cannot determine which slipped 

tasks will have detrimental effects on the 

project finish date and whether float within the 

schedule can be used to mitigate delays in 

critical tasks by reallocating resources from 

tasks that can safely slip to tasks that must be 

completed on time. Until the schedule can 

produce a true critical path, the program office 

will not be positioned to provide reliable 

timeline estimates and it will not be able to 

identify when problems or changes may occur 

and how they may affect downstream work

7. Identifying 

reasonable float

The schedule should identify the 

float—the amount of time by which a 

predecessor activity can slip before 

the delay affects successor 

activities—so that a schedule’s 

flexibility can be determined. As a 

general rule, activities along the 

critical path have the least float. Total 

float is the total amount of time by 

which an activity can be delayed 

without delaying the project’s 

completion if everything else goes 

according to plan.

Partially met Our analysis found excessive float values. We 

found 238 activities (58 percent) with float 

values greater than 1,000 working days, or 46 

months. Our analysis also found 160 activities 

(39 percent) with float values greater than 148 

working days, or 7 months.

Float estimates are directly related to the 

logical sequencing of activities. Therefore, if 

the schedule is not properly sequenced, float 

estimates will be miscalculated. Without 

reliable float estimates, management may be 

unable to allocate resources from noncritical 

activities to activities that cannot slip without 

affecting the project finish date. Furthermore, 

because the critical path is directly related to 

the logical sequencing of events and float 

calculations, if the schedule is missing 

dependencies or if activities are incorrectly 

linked, float estimates will be miscalculated, 

resulting in an invalid critical path.

8. Conducting a 

schedule risk 

analysis

A schedule risk analysis should be 

performed using statistical techniques 

to predict the level of confidence in 

meeting a project’s completion date. 

This analysis focuses not only on 

critical path activities but also on 

activities near the critical path, since 

they can affect the project’s status.

Not met Officials stated that a schedule risk analysis 

had not been performed on the integrated 

master schedule. Program officials said that 

although they do not have a WAAS 

contingency schedule, they do manage risk 

with a risk register with mitigation plans and 

that the integrated master schedule is also 

risk-adjusted. Our analysis found no evidence 

of risk mitigation activities in the schedule.

Performing a schedule risk analysis on the 

integrated master schedule, using statistical 

analysis, would allow the program office to 

model the effects of work not yet on contract. 

One way to estimate the effects of the scope 

of work not on contract is through the use of a 

Monte Carlo simulation (used to approximate 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 129 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

the probability outcomes of multiple trials by

generating random numbers) to estimate the 

durations and start dates. This would give the 

program office some visibility into risk impacts 

(understanding of the effects of risk). 

However, for a schedule risk analysis to be 

credible, the program must have a good 

schedule network that clearly identifies the 

critical path and represents all work in the 

schedule, neither of which is demonstrated in 

the program’s schedule

A schedule risk analysis will calculate 

schedule reserve, which can be set aside for 

those activities identified as high risk. Without 

this reserve, the program’s scheduled 

completion date faces the risk of delays, if any 

delays were to occur on critical path activities. 

However, if the schedule risk analysis is to be 

credible, the program must have a quality 

schedule that reflects reliable logic and clearly 

identifies the critical path—conditions that the 

schedule does not meet.

9. Updating the 

schedule using 

logic and 

durations to 

determine the 

dates

The schedule should be continuously 

updated using logic and durations to 

determine realistic start and 

completion dates for program 

activities. The schedule should be 

analyzed continuously for variances to 

determine when forecasted 

completion dates differ from planned 

dates. This analysis is especially 

important for those variations that 

affect activities identified as being in a 

project’s critical path and can 

influence a scheduled completion 

date.

Partially Met Program officials said the status of the 

schedule is updated monthly by incorporating 

actual effort achieved by the contractors and 

the support team members. Whether the 

schedule is properly updated is questionable. 

During a January 2011 meeting with the 

program office, officials said they did not 

believe the schedule status date was correct 

and, for the purposes of the audit, GAO should 

use the date in the filename. The status date 

is an input into the calculations used to update 

and schedule remaining work. Neither the date 

on which someone is viewing the schedule nor 

the latest save date of the file should be used 

as a substitute for a valid status date. Program 

officials said they have hired a dedicated 

scheduler who is responsible for updating the 

schedule.

Our analysis also found several data 

anomalies. For example, the schedule 

includes 19 remaining activities (5 percent) 

with start dates in the past with no recorded 

actual start dates; 17 remaining activities (4 

percent) with finish dates in the past with no 

recorded actual finish; 15 remaining activities 

(4 percent) with actual start dates in the future; 

and 12 remaining activities (3 percent) with 

actual finish dates in the future.

Despite the program office’s efforts to keep 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 130 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

the schedule updated, the integrated master 

schedule does not capture all activities and 

the schedule does not contain the complete 

network logic between all activities necessary 

for the schedule to correctly forecast the start 

and end dates of activities within the plan. As 

such, it is questionable that the schedule 

reflects the program’s true status.

As a best practice, the schedule should be 

continually monitored to determine when 

forecasted completion dates differ from their 

planned dates. This information can be used 

to determine whether schedule variances will 

affect downstream work. Maintaining the 

integrity of the schedule logic is not only 

necessary to reflect the program’s true status, 

but is also required before conducting a 

schedule risk analysis. Without a documented, 

consistently applied schedule change control 

process, program managers can continually 

revise the schedule to match performance. 

Such adjustments hinder the project 

manager’s understanding of the project’s true 

performance. Good documentation helps with 

analyzing changes in the program schedule 

and identifying the reasons for variances 

between estimates and actual results, thereby 

contributing to the collection of cost, schedule, 

and technical data that can be used to support 

future estimates.

Source: GAO analysis of WAAS schedule.

a

An Exhibit 300—also called a Capital Asset Plan and Business Case—is used to justify resource 

requests for major investments and is intended to enable an agency to demonstrate to its own 

management, as well as to OMB, that a major project is well planned.
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 131 GAO-12-223 Air Traffic Control Modernization

Table 16: Extent to Which the WAAS Contractor-Prepared Schedule Met Best Practices 

Best practice Explanation

Extent to which 

criterion was met GAO analysis

1. Capturing all 

activities

The schedule should reflect all 

activities as defined in the project’s 

work breakdown structure, which 

defines in detail the work necessary 

to accomplish a project’s 

objectives—including activities to be 

performed by both the owner and the 

contractors.

Substantially met Our analysis found that the contractor’s schedule is 

governed by a statement of work and the contract 

with FAA. Detailed activities are cross-referenced 

to elements in the project’s work breakdown 

structure that map to the task order structure in the 

schedule using custom text fields. Because the 

contractor did not provide a work breakdown 

structure, although we found custom text fields in 

the schedule, we cannot tell without a work 

breakdown structure whether all the work is 

included in the schedule. 

2. Sequencing all 

activities

The schedule should be planned so 

that critical project dates can be met. 

To meet this objective, activities need 

to be logically sequenced—that is, 

listed in the order in which they are to 

be carried out. In particular, activities 

that must be completed before other 

activities can begin (predecessor 

activities), as well as activities that 

cannot begin until other activities are 

completed (successor activities), 

should be identified. This helps 

ensure that interdependencies 

among activities that collectively lead 

to the accomplishment of events or 

milestones can be established and 

used as a basis for guiding work and 

measuring progress.

Substantially met Our analysis found 6 remaining activities (1 

percent) missing predecessor or successor logic. 

In addition, our analysis found 6 remaining 

activities (1 percent) with dangling logic where the 

start of the activity had no predecessor (only F-F 

predecessors). Each activity—other than the start 

and finish milestones—must have its start date 

driven by a previous activity, and each activity 

must drive the start date of another activity. Tasks 

with dangling logic can either carry on indefinitely 

without affecting downstream activities (finish date 

has no successor) or must start earlier in order to 

finish on time (start date has no predecessor). 

Dangling activities are those activities with logic 

that do not give the right activity start and finish 

dates when durations change. 

Though we found just a few missing dependencies 

and dangling activities, we identified 46 activities (7 

percent) that are affected by at least one lag or 

lead. Lags should be justified because they cannot 

have risk or uncertainty. They often represent work 

or a delay whose duration is unknown, yet are 

hard-coded into the schedule. Negative lags—also 

known as leads—are discouraged because 

negative time is not demonstrable. Leads can often 

be replaced by additional tasks and linked using 

the appropriate F-S logic. 

3. Assigning 

resources to all 

activities

The schedule should reflect what 

resources (e.g., labor, materials, and 

overhead) are needed to do the 

work, whether all required resources 

will be available when needed, and 

whether any funding or time 

constraints exist.

Partially met Program officials stated that the schedule is not 

resource-loaded and our analysis confirmed that 

resources are not assigned to activities. Program 

officials said resources are managed via their 

earned value management system in which 

resources are specified by resource group and skill 

types and are tied to the schedule by the work 

breakdown structure.

Although the contractor may be conducting this 

analysis of resources via its earned value 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 132 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

management system or its critical chain process, it 

did not provide any supporting documentation to 

validate this assertion.

Assigning resources to activities ensures that 

resources are used to determine activity durations 

because resource requirements directly relate to 

the duration of an activity. Labor, material, 

equipment, and funding requirements are 

examined to determine the feasibility of the 

schedule, so that resources provide a benchmark 

of the total and per-period cost of the project. If the 

current schedule does not allow for insight into 

current or projected overallocation of resources, 

then the risk of the program slipping is significantly 

increased.

4. Establishing the 

duration of all 

activities

The schedule should realistically 

reflect how long each activity will take 

to execute. In determining the 

duration of each activity, the same 

rationale, historical data, and 

assumptions used for cost estimating 

should be used. Durations should be 

as short as possible and have 

specific start and end dates. The 

schedule should be continually 

monitored to determine when 

forecasted completion dates differ 

from planned dates; this information 

can be used to determine whether 

schedule variances will affect 

subsequent work.

Substantially Met Our analysis found that more than half of the 154 

remaining detailed activities (107 or 69 percent) 

meet best practices for durations being less than 

44 days (or 2 working months). Our analysis found 

that all durations are consistently estimated in days 

and adhere to a standard 5-day workweek that 

accounts for holidays or uses calendars that 

account for FAA moratoriums. Our analysis found 

no activities scheduled to begin on a weekend. 

Program officials said duration estimates for the 

schedule are based on historical information from 

past performance, comparable releases, lessons 

learned, similar work, and other data requirements. 

5. Integrating 

schedule 

activities 

horizontally and 

vertically

The schedule should be horizontally 

integrated, meaning that it should link 

products and outcomes associated 

with other sequenced activities. 

These links are commonly referred to 

as “hand-offs” and serve to verify that 

activities are arranged in the right 

order to achieve aggregated products 

or outcomes. The schedule should 

also be vertically integrated, meaning 

that the dates for starting and 

completing activities in the integrated 

master schedule should be aligned 

with the dates for supporting tasks 

and subtasks. Such mapping or 

alignment among levels enables 

different groups to work to the same 

master schedule.

Substantially met FAA program officials said vertical integration is 

demonstrated in the contractor’s schedule because 

the lower-level schedules map to milestones in the 

FAA integrated master schedule. However, we 

found that dates for the same milestone activities 

in the contractor and FAA schedule did not match. 

Horizontal integration is also hampered in the 

schedule. For example, extending the durations of 

selected activities in the schedule more than 500 

working days had no effect on the project finish 

date.

Unless the schedule is fully horizontally integrated, 

the effects of slipped tasks on succeeding work 

cannot be determined. Horizontal integration 

demonstrates that the overall schedule is rational, 

planned in a logical sequence, accounts for 

interdependencies between work and planning 

packages, and provides a way to evaluate current 

status. When schedules are not horizontally 

integrated, relationships between different program 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 133 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

teams cannot be seen and product hand-offs 

cannot be identified. Vertical integration traces the 

consistency of data between work breakdown 

structure elements within the layers of the 

schedule—master, intermediate, detailed. Without 

schedules that are vertically integrated, lower-level 

schedules cannot be clearly traced to upper-tiered 

milestones, and, as a result, total schedule integrity 

is compromised. This can hamper the ability of 

different teams to use the schedule to manage 

expectations.

6. Establishing the 

critical path for 

all activities

Scheduling software should be used 

to identify the critical path, which 

represents the chain of dependent 

activities with the longest total 

duration. Establishing a project’s 

critical path is necessary to examine 

the effects of any activity slipping 

along this path. Potential problems 

along or near the critical path should 

also be identified and reflected in 

scheduling the duration of high-risk 

activities.

Substantially met Program officials said the schedule contained a 

critical path. Our analysis traced several critical 

paths in the schedule. Though we found lags and 

deadlines, which create an interruption in the 

various critical paths, the schedule’s logic, 

reasonable durations, and low total float estimates 

allow the scheduling software to calculate a valid 

critical path.

7. Identifying 

reasonable float

The schedule should identify the 

float—the amount of time by which a 

predecessor activity can slip before 

the delay affects successor 

activities—so that a schedule’s 

flexibility can be determined. As a 

general rule, activities along the 

critical path have the least float. Total 

float is the total amount of time by 

which an activity can be delayed 

without delaying the project’s 

completion, if everything else goes 

according to plan.

Substantially met Of the 154 remaining detailed tasks, 148 or 96 

percent, have float values of less than 100 days. 

The other 6 activities, or 4 percent, have float 

values of between 106 and 299 days. Though we 

found some negative float values in the schedule 

and a few activities with float values greater than 

100 days, the logical sequencing of activities, 

realistic durations, and evidence of accurate 

updating demonstrate that the total float values in 

this schedule are reasonable.

8. Conducting a 

schedule risk 

analysis

A schedule risk analysis should be 

performed using statistical 

techniques to predict the level of 

confidence in meeting a project’s 

completion date. This analysis 

focuses not only on critical path 

activities but also on activities near 

the critical path, since they can affect 

the project’s status.

Minimally met Program officials said there is no contractual 

requirement to conduct a schedule risk analysis 

and said that FAA has never expressed an interest 

in conducting such an analysis. Regarding 

managing and assessing risks in the schedule, 

program officials said they use a monthly risk 

management board process through which they 

address schedule risk with FAA. Program officials 

said that, going forward, they plan to conduct a 

Monte Carlo analysis of their schedules.

In our analysis, we found that the contractor’s 

schedule includes a 22-day “buffer” activity; this is 

a placeholder activity with a duration of 22 days 

that is intended to be used as contingency should 

delays occur within the release 3B effort. However, 
Appendix IV: Assessments of Four FAA

Program Schedules

Page 134 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

our analysis found that the buffer does not produce 

22 days of savings when removed from the 

schedule. Removing the buffer from the schedule 

saves 3 days because the buffer is not on the 

current critical path.

A comprehensive schedule risk analysis is an 

essential tool for decision makers. A schedule risk 

analysis can be used to determine a level of 

confidence in meeting the completion date or 

whether proper reserves have been incorporated 

into the schedule. A schedule risk analysis will 

calculate schedule reserve, which can be set aside 

for those activities identified as high-risk. Without 

this reserve, the program faces the risk of delays to 

the scheduled completion date if any delays occur 

in critical path activities.

9. Updating the 

schedule using 

logic and 

durations to 

determine the 

dates

The schedule should be continuously 

updated using logic and durations to 

determine realistic start and 

completion dates for program 

activities. The schedule should be 

analyzed continuously for variances 

to determine when forecasted 

completion dates differ from planned 

dates. This analysis is especially 

important for those variations that 

affect activities identified as being in 

a project’s critical path and can 

influence a scheduled completion 

date.

Partially met Our analysis found that the schedule has a valid 

and current status date. Our analysis also found 

there are no activities with start or finish dates in 

the past that are missing actual start and actual 

finish dates, and there are no activities with actual 

start and actual finish dates in the future. Program 

officials said that schedule updates are provided by 

the control account managers. The scheduler, who 

is certified in earned value management and has 

been doing scheduling for 2 years, provides the 

control account manager with a status update 

template that provides the scheduler with inprogress and percentage-complete activities. The 

scheduler then reviews this information with the 

control account manager for clarity and 

completeness.

In addition to schedule updates, program officials 

said they maintain a list of logic changes, which 

include changes to the sequencing of activities or 

delays in starting planned work. Program officials 

said that they track both start and finish variances 

in the Microsoft Project file as well as those tasks 

whose status is unreported or that are late. 

Program officials also said the schedules are 

archived daily, weekly, and monthly. Though 

program officials provided testimonial evidence 

that they adhere to best practices when updating 

their schedule, they did not provide any 

documentation to validate these assertions.

As a best practice, the schedule should be 

continually monitored to determine when 

forecasted completion dates differ from the 

planned dates. This information can be used to 

determine whether schedule variances will affect 

downstream work. Maintaining the integrity of the 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 135 GAO-12-223 Air Traffic Control Modernization

Best practice Explanation

Extent to which 

criterion was met GAO analysis

schedule logic is not only necessary to reflect true 

status, but is also required before conducting a 

schedule risk analysis. Without a documented, 

consistently applied schedule change control 

process, project performers can continually revise 

the schedule to match performance, thereby 

hindering the project manager’s understanding of 

the project’s true performance. Good 

documentation helps with analyzing changes in the 

program schedule and identifying the reasons for 

variances between estimates and actual results. It 

thus contributes to the collection of cost, schedule, 

and technical data that can be used to support 

future estimates.

Source: GAO analysis of WAAS contractor-prepared schedule.

A best practice that the WAAS contractor schedule did not meet is 

conducting a schedule risk analysis, which is not required by FAA’s 

schedule specifications. FAA officials told us that they do not conduct 

schedule risk analysis. In August and September 2011, we performed our 

own schedule risk analysis on the WAAS contractor schedule, through 

which we analyzed the latest version of the contractor’s schedule 

available to us at the time of the analysis.

A schedule risk analysis uses statistical techniques to predict a level of 

confidence in meeting a program’s completion date. This analysis focuses 

on critical path activities and on near-critical and other activities since any 

activity may potentially affect the program’s completion date. The 

objective of the simulation is to develop a probability distribution of 

possible completion dates that reflect the program and its quantified risks. 

From the cumulative probability distribution, the organization can match a 

date to its degree of risk tolerance. For instance, an organization might 

want to adopt a program completion date that provides a 70 percent 

probability that the program will finish on or before that date, leaving a 30 

percent probability that it will extend beyond, or overrun that date, given 

the schedule and the risks. The organization can thus adopt a plan 

consistent with its desired level of confidence in the overall integrated 

schedule. This analysis can give valuable insight into what-if drills and 

quantify the effects of program changes.

In developing a schedule risk analysis, probability distributions for each 

activity’s duration have to be established. Furthermore, risk in all activities 

must be evaluated and included in the analysis. Some managers focus 

Schedule Risk 

Analysis 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 136 GAO-12-223 Air Traffic Control Modernization

only on the critical path, but because we cannot be certain how long 

activities will take, we cannot know the true critical path. Consequently, it 

would be a mistake to focus only on the software-calculated critical path 

(those activities in which, if delayed, will negatively impact the overall 

project completion date) when some off-critical-path activity might 

become critical if a risk were to occur. Typically, three-point estimates—

that is, best, most likely, and worst-case estimates—are used to develop 

the probability distributions for the duration of workflow activities.

Once the distributions have been established, a Monte Carlo simulation 

uses random numbers to select specific durations from each activity 

probability distribution and calculates a new critical path and dates, 

including major milestone and program completion dates. The Monte 

Carlo simulation continues this random selection thousands of times, 

creating a new program duration estimate and critical path each time. The 

resulting frequency distribution displays the range of program completion 

dates along with the probabilities that these dates will occur. Table 17 

provides a range of dates and the probability of the project’s completion 

on those dates or earlier, based on our 3,000 iterations that are chosen at 

random during the Monte Carlo simulation. For example, according to our 

schedule risk analysis, there is a less than 5 percent chance that the 

project will be finished on or before September 13, 2012. Likewise, there 

is an 80 percent chance that the project will be finished on or before 

November 13, 2012.

Because completion on any date is uncertain, it is more realistic to show 

a range of possible completion dates than to focus on a single date. In 

deciding which percentile to use for prudent scheduling, there is no 

international best practice standard. The chosen percentile depends on 

the riskiness and maturity of the project. For some projects, we 

emphasize the 80th percentile as a conservative promise-of-completion 

date. While the 80th percentile may appear overly conservative, it is a 

useful promise-of-completion date if a number of new but currently 

unknown risks (i.e., “unknown unknowns”) are anticipated. The 50th 

percentile date may expose the project to overruns.

Table 17: Probability of Project Completion 

Baseline

5th

percentile

50th

percentile

80th

percentile

95th

percentile

Finish date Sept. 6, 2012 Sept. 18, 2012 Oct. 23, 2012 Nov. 13, 2012 Dec. 6, 2012

Calendar days beyond scheduled finish date - 12 47 68 91

Source: GAO analysis of contractor data
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 137 GAO-12-223 Air Traffic Control Modernization

In the case of the WAAS contractor schedule, our analysis concluded that 

management should realistically expect cutover, or completion, between 

October 23, 2012, and November 13, 2012, the 50th and 80th 

percentiles, respectively. The artificial must-finish date constraint of

September 6, 2012, built into the schedule is unlikely. Our analysis shows 

the probability of completion by September 6, 2012, is less than 5 percent 

likely with the current schedule without risk mitigation. There are two 

reasons why the planned end completion date is not likely to occur, 

according to the results of our schedule risk analysis. First, most risks are 

threats only. Only two opportunities were identified during the analysis: 

(1) the estimating error of the schedule may be between -10 percent and 

15 percent, and (2) there is a 65 percent chance that the 11-day formal 

shadow test will not be needed. Second, there are parallel paths within 

the structure of the schedule that lead to merge points. If several paths 

converge to one milestone, the latest merging path determines the date.

This “merge bias” cannot accelerate schedule dates and usually adds 

structural risk to the schedule.

The contractor supplied six different risks that are currently identified in 

the project’s risk register. Using these risks as a basis for discussion, we 

interviewed 16 experts familiar with the project, including prime contractor 

officials, FAA officials, and technical FAA consultants to identify any other 

risks. Each interviewee was asked four questions to address four related 

points.

• To estimate the probability that an identified risk will occur on the 

project in such a way that some activity durations are affected. The 

estimated probability is translated into the percentage of iterations that 

are chosen at random during the simulation. For example, if the 

expert estimates weather will have a 10 percent chance of affecting 

some activities, then, on average, weather risk will occur in 10 percent 

of the Monte Carlo iterations.

• If the interviewee believed the identified risk was likely, the 

interviewee was asked to identify which activities’ durations would be 

affected. For example, activities related to steel erection or concrete 

pouring may be affected if the weather risk is realized and bad 

weather occurs.

• After the interviewee identified affected activities, the interviewee was 

asked to provide a three-point estimate of the risk’s effect on 

duration—low, most likely, and high. Estimates were provided as 

Identified Risks
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 138 GAO-12-223 Air Traffic Control Modernization

percentages, which were applied to the activity durations in the Monte 

Carlo simulation if the risk occurred. For example, if bad weather 

occurs, the duration of a 10-day steel erection activity may be affected 

a minimum of 110 percent, a most likely of 150 percent, or a 

maximum of 200 percent. These percentages translate into increases 

in the activity’s duration of 11 days under a low-risk scenario, 15 days 

under the most likely risk scenario, and 20 days under the maximum 

risk scenario. If the risk is not realized, there is no change to the 

activity’s original estimated duration.

• Finally, the interviewee was asked to identify any unaccounted for 

risks.

We began the interviews with 6 risks and, through the interview process, 

identified 16 more risks. During data analysis, some risks were 

consolidated with others or eliminated because of limited data. In all, 14 

risks were identified and incorporated into the Monte Carlo simulation.

These include 9 risk drivers, 4 existence risks, and 1 schedule duration 

risk.1

The final risk drivers used in the schedule risk analysis are as follows:

• Some software Release 3B problem reports require testing with a live 

geostationary satellite; software simulation environment may not be 

able to test all requirements.

• Potential difficulty repairing Fullerton contractor’s lab safety 

computers.

• Software delays in software Release 3A and additional changes to 

software Release 2B are likely to delay the 3B schedule.

• Parallel implementation of Releases 2B, 2C, 3A, and 3B create 

competition for FAA resources.

• Release 3B problem report testing is more complex and subject to 

uncertainty.

 

1

The schedule duration risk is applied to each activity’s duration to represent the inherent 

inaccuracy of scheduling. 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 139 GAO-12-223 Air Traffic Control Modernization

• FAA WAAS personnel move around, and inexperienced staff must be 

trained on WAAS.

• Workshare strategy between FAA and its prime contractor affects the 

schedule for on-the-job training learning curve.

• Changes to the simulation tools may be more difficult or more timeconsuming to implement than anticipated.

The final existence risks are as follows:

• FAA performs formal software release 3B shadow test.

• Late problem report may come from FAA.

• FAA program office is resource-limited.

Inserting existence risk activities does not affect dates within the baseline 

schedule because the activities initially have zero duration. The activities 

have duration only if they happen to occur during an iteration of the 

simulation.

The final uncertainty risk is schedule without buffer may be optimistic.

Most risks were identified by multiple respondents during the interviews.

During data analysis, data from the interviews are combined and 

analyzed to create ranges and probabilities for each of the 14 risks.

Because risks are multiplicative, several risks occurring on the same 

activity may overestimate the true risk. That is, by default in the Monte 

Carlo simulation, risks occur in a series, one after another, so that an 

activity that has several risks may be unrealistically extended if all risks 

occur. In reality, an activity may recover from two or more risks 

simultaneously, so that the actual risk is not multiplicative. Therefore, to 

avoid overestimation of risk, risks can explicitly be defined as occurring in 

parallel rather than in series. Risks that occur in a series will occur one 

after the other and add (or subtract) their respective effects on duration to 

the affected activity. If risks occur in parallel, on the other hand, only the 

maximum effect of all risks will affect the duration. For example, if the risk 

of complexity adds 3 days to the duration of software development and 

the risk of staff shortages adds 4 days, then development will extend 7 

days if the risks are defined in series. However, the duration will extend 

only 4 days if the risks are defined as parallel. This definition of parallel 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 140 GAO-12-223 Air Traffic Control Modernization

risks helps temper any risk overestimated by a multiplication of risk 

factors. We defined one risk in series: software delays in Release 3A and 

additional changes to Release 2B are likely to delay Release 3B 

schedule. All other risks were assumed to be parallel.

Most risks were assigned directly to existing activities in the schedule.

However, some risks required adjustments to the schedule. These 

adjustments involved replacing lags with activities and inserting existence 

risk activities.

Lags: During our initial analysis of the contractor’s schedule, we identified 

25 remaining activities with lags and 21 remaining activities with leads

(negative lags). While lags represent the passing of time between activities, 

they are often misused to put activities on a specific date or to insert a 

buffer for risk. That number was reduced to 20 activities with lags and 6 

activities with leads with the “Rev 1” schedule that was altered by the prime 

contractor for our use in the schedule risk analysis. We replaced the 33-day 

lag between the R3B Start Cutover and R3B End Cutover activities with an 

actual activity, ID 258 “Cutover Task.” Replacing lags with actual activities 

does not affect dates within the baseline schedule because the activities 

have the same duration as the lags.

Existence risk: We identified some risks that would add an indeterminate 

amount of time to the overall schedule if they were realized. For example, 

if the Tijuana facility is not ready because software requirements are 

misunderstood, it could add 4 to 26 days to the schedule in the form of 

additional facility work.

Prioritizing risks and risk mitigation: Risks can affect the schedule in 

several ways: They can have a high probability of occurring; have a largepercentage impact on the durations of the activities they affect; or they 

can apply to risk-critical paths, which may differ from the baseline 

deterministic critical path. Beyond applying 14 risks to the schedule, we 

were interested in identifying the marginal impact of each risk. That is, we 

were interested in identifying which risks have the largest impact on the 

schedule because these were the risks that should be targeted first for 

mitigation.

To find the marginal impact of a risk on the total project risk at a certain 

percentile, the Monte Carlo simulation was performed with the risk 

removed. The difference between the finish dates of the simulation with 

all the risks and the simulation with the missing risk yielded the marginal 
Appendix IV: Assessments of Four FAA 

Program Schedules

Page 141 GAO-12-223 Air Traffic Control Modernization

impact of the risk. Table 18 gives the priority of risks at the 80th percentile 

and the marginal impact of each risk.

Table 18: Prioritized Risks at the 80th Percentile

Priority 

level Risk

Marginal impact 

in calendar days

1 FAA program office is resource-limited 13

2 Software delays in Release 3A and additional changes 

to Release 2B are likely to delay Release 3B schedule

7 

3 Release 3B problem report development and test are 

more complex than planned and subject to uncertaintya 7 

4 FAA personnel move around and often get 

inexperienced people to train

8 

5 Workshare strategy between FAA and prime contractor 

affects schedule for on-the-job training learning curve

15

6 Late problem report may come from FAA 7 

7 Some Release 3B problem reports require testing with 

a live geostationary satellite; software simulation 

environment may not be able to test all requirements

6 

8 Schedule without buffer may be optimistic 5 

Total 68

Source: GAO analysis of WAAS schedule

a

These are two separate risks but are assigned the same priority because they are correlated 100 

percent in our analysis. That is, we assumed if complexity affected one, it would necessarily affect the 

other.

The marginal impact directly translates to potential calendar days saved if 

the risk is mitigated. Once risks are prioritized at the percentile desired by 

management, a risk mitigation workshop can be implemented to deal with 

the high-priority risks in order. The prioritized list of risks will form the 

basis of the workshop, and risk mitigation plans can be analyzed using 

the risk model to determine how much time might be saved. Project 

managers cannot expect to completely mitigate any one risk, and it is not 

reasonable to expect to mitigate all risks. In addition, risk mitigation will 

add to the project budget. However, some opportunities may be available 

to partially mitigate risks.
Appendix V: GAO Contact and Staff 

Acknowledgments

Page 142 GAO-12-223 Air Traffic Control Modernization

Gerald Dillingham, Ph.D., (202) 512-2834 or dillinghamg@gao.gov

In addition to the contact named above, individuals making key 

contributions to this report include Edward Laughlin and Karen Richey 

(Assistant Directors), Lindsey Bach, David Brown, Pamela Davidson, 

Tisha Derricotte, Kevin Egan, James Geibel, Bert Japikse, Delwen Jones, 

Jason Lee, Dominic Nadarski, Josh Ormond, and Brian Welsh.

Appendix V: GAO Contact and Staff 

Acknowledgments 

GAO Contact

Staff 

Acknowledgments 

(540219)
The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and 

policies; and provides analyses, recommendations, and other assistance 

to help Congress make informed oversight, policy, and funding decisions. 

GAO’s commitment to good government is reflected in its core values of

accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no 

cost is through GAO’s website (www.gao.gov). Each weekday afternoon, 

GAO posts on its website newly released reports, testimony, and 

correspondence. To have GAO e-mail you a list of newly posted products, 

go to www.gao.gov and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of 

production and distribution and depends on the number of pages in the 

publication and whether the publication is printed in color or black and 

white. Pricing and ordering information is posted on GAO’s website, 

http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 

MasterCard, Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube. 

Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. 

Visit GAO on the web at www.gao.gov. 

Contact:

Website: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-

4400, U.S. Government Accountability Office, 441 G Street NW, Room 

7125, Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548

GAO’s Mission

Obtaining Copies of 

GAO Reports and 

Testimony 

Order by Phone

Connect with GAO

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Congressional 

Relations 

Public Affairs

Please Print on Recycled Paper.