Page 1 GAO-14-547R C-130 Avionics Modernization Program

441 G St. N.W.

Washington, DC 20548

May 29, 2014

Congressional Committees

Defense Acquisitions: Assessment of Institute for Defense Analyses’ C-130 Avionics 

Modernization Program Analysis

The Air Force’s C-130 Avionics Modernization Program (AMP), which entered development in 

2001, was to standardize and upgrade the cockpit and avionics for various configurations of the 

C-130 fleet. The C-130, a four-engine turboprop aircraft also known as “Hercules” that is used 

primarily for military transport, was originally designed in the 1950s. The AMP upgrades were 

intended to ensure that the C-130 can satisfy the navigation and safety requirements it needs to 

operate globally while at the same time reducing crew size by one. The program would also 

have replaced many systems for which manufacturers no longer exist—a situation referred to as 

diminishing manufacturing sources.

In 2007, the program experienced a critical breach of a statutory cost threshold,

1 and in 

February 2012, the Department of Defense (DOD) proposed cancelling the C-130 AMP. The 

National Defense Authorization Act for Fiscal Year 2013 (NDAA) stated that the Secretary of the 

Air Force may not cancel the program until 90 days after the Secretary provided the 

Congressional defense committees a cost benefit analysis conducted by the Institute for 

Defense Analyses (IDA). The NDAA required the Secretary to seek an agreement with IDA to 

conduct an independent study of the costs and benefits of upgrading and modernizing the 

legacy C-130H airlift fleet via AMP compared to the costs and benefits of a reduced-scope 

program.

2 IDA began its work in March 2013 and produced the required study in September 

2013, followed by a detailed annex in December 2013.3 The National Defense Authorization Act 

for Fiscal Year 2014 mandated that we review IDA’s analysis.4

 1 DOD is required to notify Congress whenever a major defense acquisition program’s unit cost experiences cost 

growth that exceeds certain thresholds. This is commonly referred to as a Nunn-McCurdy breach. Critical breaches 

occur when the program acquisition unit cost or procurement unit cost increases by at least 25 percent over the 

current baseline estimate or at least 50 percent over the original. When a program experiences a Nunn-McCurdy 

breach of the critical cost growth threshold, DOD is required to take a number of steps including reassessing the 

program and submitting a certification to Congress in order to continue the program. 10 U.S.C. §§ 2433 and 2433a.

This report describes IDA’s study 

and provides our assessment.

2 Pub. L. No. 112-239, § 143.

3 Institute for Defense Analyses, C-130 Avionics Modernization Analysis, (Alexandria, VA: Sept. 2013); Institute for 

Defense Analyses, C-130 Avionics Modernization Analysis: Annex, (Alexandria, VA: Dec. 2013).

4 Pub. L. No. 113-66, § 133 (2013). The NDAA for Fiscal Year 2014 also prohibited the Air Force from using any 

fiscal year 2014 appropriations to cancel or modify the C-130 aircraft.
Page 2 GAO-14-547R C-130 Avionics Modernization Program

To conduct this work, we obtained IDA’s September 2013 study of the C-130 modernization 

alternatives and associated December 2013 Annex and reviewed the data and assumptions that 

IDA used to reach its conclusions and recommendations. GAO focused its assessment on IDA’s 

cost estimates and did not assess the operational effectiveness portion of IDA’s study as 

estimated costs were the major discriminating factor in IDA’s conclusions. We compared how 

IDA’s estimates were developed to the cost-estimating best practices outlined in the GAO Cost 

Estimating and Assessment Guide.5

We conducted this performance audit from February 2014 through May 2014 in accordance with 

generally accepted government auditing standards. Those standards require that we plan and 

perform our audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that the evidence obtained 

provides a reasonable basis for our findings and conclusions based on our audit objectives. 

Specifically, we analyzed the extent to which IDA’s cost 

estimates met the characteristics of high-quality cost estimates. We also discussed the study 

results with relevant DOD officials. In addition, we reviewed IDA’s C-130 AMP annex report to 

determine the methodologies used to support its findings. More detail on our methodology in 

applying the cost estimating best practices identified in the GAO Cost Estimating Guide can be 

found in enclosures II and III.

Results in Brief

IDA’s study is comprised of two key components: (1) its estimate of the costs of AMP and two 

reduced scope alternatives referred to as Options A and B, which offer fewer avionics upgrades 

than AMP and, unlike AMP, would both require a navigator in the cockpit: and (2) its 

assessment of the operational effectiveness—that is, benefits—of AMP and each alternative. 

IDA estimated that that AMP’s costs for research, development, test and evaluation (RDT&E), 

procurement, and operation and support (O&S) over a 25-year period would total $2.137 

billion—significantly higher than the estimated $1.531 billion for Option A and $530 million for 

Option B. Regarding benefits, IDA found that the lower-cost options offered nearly as much 

capability as AMP, and the study concluded that the Air Force should not pursue the C-130 

AMP program as currently defined.

We found that IDA’s cost analysis generally followed best practices for high-quality cost 

estimates, which are critical to the success of any program. We analyzed the cost estimating 

methodology IDA used in its study compared to best practices and found that IDA’s study 

substantially met two of the characteristics expected of high-quality cost estimates—that they be 

comprehensive and credible—and partially met the remaining two characteristics—being welldocumented and accurate. 

Background

When the C-130 AMP entered development in 2001, it was intended for 20 variants of the 

C-130 for a total of 519 aircraft. In June 2007, following the program’s critical Nunn-McCurdy 

breach, DOD conducted a five-month review. Subsequently, in accordance with 10 U.S.C. 

 5 GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program 

Costs, GAO-09-3SP, (Washington, D.C.: March 2009).
Page 3 GAO-14-547R C-130 Avionics Modernization Program

§ 2433, the Under Secretary of Defense for Acquisition, Technology and Logistics certified the

C-130 AMP program to Congress, but for a reduced number of 222 aircraft.

The C-130 AMP entered production in 2010, and five aircraft were produced before production 

was halted. Figure 1 shows the cockpit of a C-130 with AMP. 

Figure 1: Cockpit of a C-130 with AMP.

In the President’s Budget for Fiscal Year 2013, DOD did not propose any funding for the C-130 

AMP, which would have effectively cancelled the program in favor of a minimum 

Communication, Navigation, Surveillance/Air Traffic Management (CNS/ATM) solution for the 

legacy C-130 combat delivery fleet. According to DOD, CNS/ATM would address diminishing 

manufacturing source issues and complete fleet modifications to meet required navigation 

performance mandated by 2020. Congress did not authorize any modifications to the program, 

but rather required IDA’s cost-benefit study of the issue. 

IDA Study Examined Lower-Cost Options to the C-130 AMP 

IDA’s cost-benefit analysis of C-130 AMP and alternative options is comprised of two key 

components: its estimate of the costs of AMP and the alternatives, and its assessment of the 

operational effectiveness—that is, benefits—of AMP and each of the alternatives. In addition to 

AMP, IDA examined reduced-scope alternatives it referred to as Options A and B. It reported 

that AMP was the most sophisticated of the three alternatives and Option B the least. IDA’s 

study noted that they differ in many details. For example, the alternatives differ with respect to 

level of area navigation and required navigation performance, potentially imposing future 
Page 4 GAO-14-547R C-130 Avionics Modernization Program

restrictions on airfield access for Options A and B and aircraft routing for Option B. IDA also 

noted that the most significant difference is the removal of the navigator from the cockpit 

allowed by the degree of automation, a head-up display and advanced flight assistance 

equipment by the AMP alternative.6

To provide the cost portion of the cost-benefit analysis, IDA developed 25-year cost estimates 

for the alternatives and assessed their operational implications. IDA developed estimates for 

RDT&E, procurement, and O&S for the alternatives and combined those estimates to determine 

overall 25-year costs for each alternative. IDA assumed a fleet of 192 C-130Hs for all options, 

the number of aircraft defined in the Air Force Capability Development Document; appropriately 

excluded the approximately $1.7 billion previously spent on the AMP program; and estimated 

the costs in constant fiscal year 2013 dollars. The results of IDA’s analysis of the costs of AMP 

and Options A and B are presented in table 1.

Both Options A and B would have fewer avionics upgrades 

than AMP and would require the navigator. Unlike Option B, Option A would replace older 

technology gauges with multi-function displays, as would AMP. In addition, unlike AMP and 

IDA’s Option A, Option B would not replace the current self-contained navigation system 

(SCNS) with a more modern SCNS capable of higher location and planning precision. IDA 

concluded that the lower-cost options to C-130 AMP had nearly as much capability and that the 

Air Force should not pursue the C-130 AMP program as currently defined. 

Table 1: IDA Cost Estimates for Three C-130 Modernization Alternatives 

C-130 AMP Option A Option B

Cost Estimates (constant fiscal year 2013 dollars in millions)

RDT&E $104 $437 $140

Procurement 3,060 1,320 480

O&S Savingsa -1,027 -226 -90

 Totals 2,137 1,531 530

Source: GAO presentation of IDA results

a

These estimates represent solely the avionics-related O&S costs and savings to be incurred as a result of 

implementing each of the alternatives. A negative amount depicts a net savings for that option. 

IDA’s estimates showed that that the AMP’s costs for RDT&E, procurement, and O&S over a 

25-year period were significantly higher than the alternative options it considered. Specifically, 

the analysis shows while Options A and B have higher development costs and lower O&S 

savings (due primarily to the fact that they do not enable the reduction of one crew member), 

these costs are more than offset by lower procurement costs.

Regarding overall effectiveness, IDA found the AMP’s operational effectiveness advantage over 

Options A and B to be marginal. Specifically, the study found Option A to be at least as 

operationally effective as AMP since both AMP and Option A address the diminishing 

manufacturing sources problem. (See enclosure I for further details on IDA’s analyses and 

results.)

IDA’s Cost Estimating Generally Followed Best Practices 

Given that IDA reported finding Option A at least as operationally effective as AMP, and that 

estimated costs were the apparent discriminating factor in IDA’s recommendation that the Air 

Force not pursue AMP, we limited our assessment of IDA’s cost estimating quality to AMP and 

 6 These capabilities are similar to those found in the C-130J, which also operates without a navigator.
Page 5 GAO-14-547R C-130 Avionics Modernization Program

Option A. We found that IDA’s cost analysis either substantially or partially met each of the four 

characteristics expected of high quality cost estimates. Such estimates are critical to the 

success of any program as they provide the basis for informed investment decision making, 

realistic budget formulation and program resourcing, meaningful progress measurement, 

proactive course correction when warranted, and accountability for results. These 

characteristics are described briefly below in table 2. A more detailed description of each 

characteristic of high-quality cost estimates and our complete assessment can be found in 

enclosure II.

Specifically, we analyzed the cost estimating practices used by IDA in its analysis of the C-130 

AMP and alternative options against best practices for high-quality cost estimates. As table 2 

shows, we found that the cost estimates in IDA’s study substantially met two of the 

characteristics expected of high-quality cost estimates—comprehensiveness and credibility—

and partially meet the remaining two characteristics—being well-documented and accurate. 

Table 2: Evaluation of IDA’s Cost Estimates

Characteristics of high-quality cost estimates Our evaluationa

1. Comprehensive (estimate is complete and accounts for all possible costs) Substantially Met

2. Well documented (detail and quality of the information used) Partially Met

3. Accurate (should be unbiased and neither overly conservative or optimistic) Partially Met

4. Credible (to include sensitivity analysis, risk and uncertainty analysis) Substantially Met

Source: GAO assessment. a For the purposes of our analysis, we determined that a characteristic was “met” if IDA provided complete data that 

satisfies the entire criterion; “substantially met” if IDA provided data that satisfies a large portion of the criterion; 

“partially met” if IDA provided evidence that satisfies about half of the criterion; “minimally met” if IDA provided 

evidence that satisfies a small portion of the criterion; and “not met” If IDA provided no data that satisfies any of the 

criterion. 

More specifically we found the following:

• Comprehensiveness: We found the estimates to be substantially comprehensive 

because the estimates appeared to include the pertinent life cycle costs and the 

estimates’ cost element structure was at an sufficient level of detail considering the 

purpose of the study was a cost benefit analysis comparison of alternatives. However, 

we also found that few of the typical technical baseline elements were described in the 

study and that ground rules and assumptions were incomplete. Specifically, the study 

itself did not identify inflation indices or their source, although IDA subsequently 

identified that they used the OSD Comptroller’s Green Book for indices.

• Documentation: We found the estimates partially met best practices for this 

characteristic. The study documentation described the estimating methodology used to 

derive each cost element, but did not describe in detail the calculations performed to 

arrive at the estimate. Additionally, the documentation described step-by-step how the 

estimate was developed, but it did not contain sufficiently detailed data to allow a cost 

analyst unfamiliar with the program to replicate the estimate. Furthermore, the 

documentation does not contain the source data used to develop the estimates.

• Accuracy: We found the estimates partially met best practices for this characteristic. 

The estimates were based on actual costs from the C-130 AMP and other programs, 

program office estimates for C-130 AMP and other similar programs, cost estimating 

relationships, and factors, but because the historical source data were not included, we 
Page 6 GAO-14-547R C-130 Avionics Modernization Program

could not assess the reliability and relevance of the data. Additionally, while the study 

team performed uncertainty analysis and displayed the point estimates and low and high 

ranges of the estimate, they did not show the confidence levels of the point estimates to 

allow us to determine if the estimates were unbiased.

• Credibility: We found the estimates substantially met best practices for this 

characteristic. The study included a risk and uncertainty analysis and displayed the 

results in the difference between the net present value of AMP and Option A. However, 

the documentation did not contain cumulative probability distributions, known as “S 

curves,” of the alternative estimates. Further, the study sensitivity analysis was limited to 

examination of the most significant contributors to AMP cost variance.

Life cycle cost analyses may be sensitive to underlying assumptions, such as, the time period 

covered by an analysis. In this case, IDA was tasked with and developed costs for a 25-year 

period. Over that period, it estimated that the estimate for Option A was nearly $600 million less 

than for AMP. Comparatively, in the last year of its study, IDA estimated AMP annual savings 

that were $47 million greater than for Option A. Hence, had the time period been less than 25 

years, the difference between the AMP and Option A estimates could have been greater; had 

the time period been longer than 25 years, the difference could have been less. Other 

assumptions can similarly influence results. For example, IDA noted in its conclusions that 

factors outside of the scope of its study —including retirement schedules for C-130s and the 

acquisition of new C-130Js—should inform decisions determining the specific upgrade program 

chosen. 

We are not making recommendations in this report.

Agency Comments and Our Evaluation

DOD and IDA provided comments on a draft of this report, which are reprinted in enclosures IV 

and V, respectively. In its comments, DOD referred to our work as validating IDA’s analysis. We 

note that we analyzed the cost estimating methodology IDA used in its study compared to best 

practices for producing reliable and valid cost estimates. However, we did not produce separate 

estimates nor did we evaluate IDA’s recommendations. DOD also commented that it took into 

consideration the data sources that supported IDA’s conclusions when reviewing IDA’s 

recommendations. 

In addition, IDA commented that it agreed with our assessment that its cost analysis generally 

followed best practices for high quality estimates. Specifically, IDA agreed with our assessment 

that its cost analysis was comprehensive and credible, but believed they warranted a higher 

rating of “met”, rather than “substantially met” for these two characteristics. IDA also noted that it 

strongly believes our assessments of its study against the remaining two characteristics—being 

well-documented and accurate—should have been higher. As noted in our report, we assessed 

IDA’s cost estimating as having partially met the criteria for being well documented because it 

did not, in our opinion, contain sufficiently detailed data to allow a cost analyst unfamiliar with 

the program to replicate the estimate. Similarly, we assessed IDA’s estimate as having only 

partially met the best practice of accuracy because the historical source data were not included, 

and we could not, therefore, assess the reliability and relevance of the data. Our assessment 

was governed by the information contained in the report itself, which was limited in these two 

areas. That does not necessarily mean that the information did not exist elsewhere. IDA also 

commented that we did not find any evidence that its cost estimates were not accurate and that 

nothing in our assessment undermines its findings and recommendations. 
Page 7 GAO-14-547R C-130 Avionics Modernization Program

- - - - -

We are sending copies of this report to interested congressional committees; the Secretary of 

Defense; the Under Secretary of Defense for Acquisition, Technology, and Logistics; and the 

Secretary of the Air Force. This report also is available at no charge on GAO’s website at 

http://www.gao.gov.

Should you or your staff have any questions on the matters covered in this report, please 

contact me at (202) 512-4841 or sullivanm@gao.gov. Contact points for our Offices of 

Congressional Relations and Public Affairs may be found on the last page of this report. Key 

contributors to this report were Bruce H. Thomas, Assistant Director; Marvin Bonner, Analyst-inCharge; John Beauchamp; Susan Ditto; Karen Richey; Brian Bothwell; Marie Ahearn; and 

Kenneth E. Patton.

Michael J. Sullivan 

Director, Acquisition and Sourcing Management

Enclosures – 5
Page 8 GAO-14-547R C-130 Avionics Modernization Program

List of Committees

The Honorable Carl Levin

Chairman

The Honorable James Inhofe

Ranking Member

Committee on Armed Services

United States Senate

The Honorable Richard J. Durbin

Chairman

The Honorable Thad Cochran

Ranking Member

Subcommittee on Defense

Committee on Appropriations

United States Senate

The Honorable Howard P. “Buck” McKeon

Chairman

The Honorable Adam Smith

Ranking Member

Committee on Armed Services

House of Representatives

The Honorable Rodney Frelinghuysen

Chairman

The Honorable Pete Visclosky

Ranking Member

Subcommittee on Defense

Committee on Appropriations

House of Representatives 
Page 9 GAO-14-547R C-130 Avionics Modernization Program

Enclosure I: IDA’s Cost Estimating and Operational Effectiveness Methodology and 

Results

RDT&E Cost Estimates

In developing estimated RDT&E costs associated with AMP and each of the options, IDA 

reported that it included nonrecurring design costs as well as the cost of building the test article 

and conducting associated testing. Utilizing data from the last C-130 AMP Selected Acquisition 

Report, the C-130J Avionics upgrade, and the KC-10 AMP, IDA estimated that the remaining 

development costs for AMP to be about $104 million. To estimate the development costs for 

Options A and B, IDA utilized data from the C-130 AMP program, removing costs associated 

with reduced content (eliminated requirements and fewer variants and conversions) and sunk 

costs. IDA then accounted for the reduced content of Options A and B, determining the subset 

of equipment and associated cost necessary for the functionality of Options A and B, including 

government costs, as it did with AMP. Table 3 summarizes IDA’s RDT&E cost estimates for 

C-130 AMP and the alternatives considered.

Table 3: IDA’s C-130 AMP and Options RDT&E Cost Estimates

C-130 AMP Option A Option B

Cost Estimates (constant fiscal year 2013 dollars in millions)

IDA’s estimates $104 $437 $140

Source: GAO presentation of IDA results 

As a cross-check on its RDT&E cost estimates for Options A and B, IDA examined additional 

data. They collected data from the C-130J avionics upgrade and reduced its content to derive a 

comparison cost estimate. IDA also developed comparison estimates based on the C-5 AMP 

development program (deemed similar to Option A) and the KC-10 AMP development program 

(deemed similar to Option B). In addition, IDA reported that it examined Air Force Cost Analysis 

Agency (AFCAA) estimates for the two successor programs the Air Force had proposed for 

C-130 AMP—sometimes called the “Optimize” and the “Minimize” CNS/ATM programs. Table 4 

provides IDA’s Option A and Option B RDT&E cost estimate and the associated cross-check 

estimates.

Table 4: IDA’s Cost and Cross-check Estimates for RDT&E for Options A and B

Option A Option B

Cost Estimates (constant fiscal year 2013 dollars in millions)

IDA cost estimate $437 $140

IDA Adjusted C-130J avionics upgrade analogy cross-check 418 220

IDA C-5 AMP analogy cross-check 501

IDA KC-10 AMP analogy cross-check 57

AFCAA “Maximize” and “Minimize” estimates cross-check 368 185

Source: GAO presentation of IDA results 

As the table above shows, the IDA estimate for RDT&E costs fall between the upper and lower 

bounds of the other estimates.
Page 10 GAO-14-547R C-130 Avionics Modernization Program

Procurement Cost Estimates

In discussing its estimation of procurement costs, IDA noted that programs that modify existing 

platforms have distinctive features that result in cost elements that are unique relative to a newstart acquisition. In developing its procurement cost estimates for the C-130 AMP and the 

alternatives, IDA examined those elements: kits (i.e. hardware), installation, and other 

acquisition costs (e.g. training equipment, peculiar support equipment, depot standup, initial 

spares, and interim contractor support), and program support (e.g. system engineering and 

program management). IDA reported that it utilized acquisition schedules based on plans and 

estimates previously put forth by the Air Force and assumed a sole-source procurement 

strategy for each of the three alternatives.

IDA noted that estimating procurement costs for AMP was unique as cost data for completed 

production-configuration kits and installations for five aircraft were available. It estimated 

procurement costs for the AMP by defining price-improvement curves for the kits and installation 

and extrapolating previous experience forward to define fly-away cost (the sum of kits and 

installation costs). IDA reported that it estimated other acquisition costs using the relationships 

of other costs to flyaway costs suggested by historical data for C-130 and other transport aircraft 

modification programs. For Options A and B, IDA’s estimates used “catalog” prices presented 

by AFCAA for off-the-shelf components that would meet Option A and B defined capabilities. 

IDA reported that it estimated the cost of associated hardware needed for the installation of 

those off-the-shelf components (e.g. brackets, mounts, cables, etc.) and the cost of installation 

itself based on a combination of historical factors and other analyses. IDA also calculated 

average procurement unit costs for each of the alternatives. Table 5 summarizes IDA’s 

procurement total and average unit (per aircraft) cost estimates for C-130 AMP and the 

alternatives considered.

Table 5: IDA’s C-130 AMP and Options Procurement Cost Estimates

C-130 AMP Option A Option B

Cost Estimates (constant fiscal year 2013 dollars in millions)

Total estimated procurement costs $3,060 $1,320 $480

Average estimated unit procurement cost 16.4 7.0 2.5

Source: GAO presentation of IDA results 

IDA calculated the average procurement unit costs to enable comparison with estimates it 

developed based on information from other sources, thereby providing a cross-check against its 

estimates for AMP and Option A.7 For example, IDA developed two cross-check estimates for 

AMP: one based on the 2010 Selected Acquisition Report produced for the C-130 AMP 

program; the other based on a C-130 AMP System Program Office “should cost” estimate.8 To 

make these estimates comparable, IDA adjusted the unit costs from them to account for 

differences in their associated quantities with the number of AMP aircraft used in calculating its 

estimate and adjusted them to constant fiscal year 2013 dollars. Table 6 provides IDA’s AMP 

procurement cost and associated cross-check estimates.

 7 The IDA study did not provide procurement cross-check estimate calculations for Option B or explain why it did not.

8 In accordance with direction provided in Interim DOD Instruction 5000.2. “Operation of the Defense Acquisition 

System,” and memorandums issued by the Under Secretary of Defense for Acquisition, Technology, and Logistics, 

each program is to conduct a “should-cost” analysis which includes justifying each cost under the program’s control 

with the aim of reducing negotiated prices for contracts and obtaining other efficiencies in program execution.
Page 11 GAO-14-547R C-130 Avionics Modernization Program

Table 6: IDA’s Cost and Cross-check Estimates for Procurement of AMP

C-130 AMP

Cost Estimates (constant fiscal year 2013 dollars in millions)

IDA estimated average unit procurement cost $16.4

IDA cross-check estimate based on C-130 AMP Final SAR 19.5

IDA cross-check estimate based on adjusted “should cost” 14.3

Source: GAO presentation of IDA results 

As can be seen, the IDA estimate for AMP procurement costs falls between the upper and lower 

bounds of the other estimates.

O&S Cost Estimates

IDA estimated O&S costs for the three alternatives—O&S for 192 aircraft (including aircraft 

previously completed) over a 25 year period—in constant fiscal year 2013 dollars and also 

calculated a net present value for each. IDA reported that it included only the incremental O&S 

costs associated with each alternative, which for all three modernization alternatives amounted 

to savings from the current C-130 fleet. Elements of O&S costs included personnel, indirect 

costs, training, fuel, depot-level reparables, diminishing manufacturing sources, consumables, 

sustaining engineering, program management, support equipment, and software.

IDA’s study indicates that the largest driver of cost savings in its O&S estimate for AMP relates 

to the elimination of the navigator position from the C-130 cockpit—savings from personnel 

costs as well as training. To calculate these savings, IDA reported that it adjusted information 

from a 2010 AMP manpower estimate report as the basis for manpower reductions and applied 

current salary, training costs and turnover rate information from the Air Force. It assumed no 

navigator-related savings for Options A and B. Another large driver of AMP cost savings relative 

to the other options was related to fuel consumption. IDA estimated the incremental O&S costs 

for fuel by factoring weight considerations for the avionic systems added or removed by each 

alternative along with costs relating to potential noncompliance with relevant future 

requirements. IDA noted that while AMP allows unrestricted global operations by meeting 

required navigational performance and area navigation, aircraft that do not meet those 

requirements—such as Options A and B—can have restrictions on maximum altitude and 

landings, which can reduce flying efficiency. To account for this, IDA studied historical C-130H 

flying and estimated the potential cost associated with not meeting future CNS/ATM 

requirements. In other O&S cost aspects, IDA estimated Options A and B to be less costly than 

the AMP. Table 7 provides IDA’s total calculated 25-year life cycle O&S costs (actually savings) 

for each of the alternatives.

Table 7: IDA’s C-130 AMP and Options O&S Savings Estimates

C-130 AMP Option A Option B 

Cost Estimates (constant fiscal year 2013 dollars in millions)

IDA’s total 25-year O&S estimates $-1,027 $-226 $-90

Source: GAO presentation of IDA results 
Page 12 GAO-14-547R C-130 Avionics Modernization Program

Total Estimated Costs

Combining its RDT&E, procurement, and O&S cost estimates from the options, IDA calculated 

total costs for each of the options in fiscal year 2013 constant dollars. In addition, it discounted 

those amounts to compute net present values for each of the options. Table 8 provides IDA’s 

calculated total costs and net present values for the options.

Table 8: IDA’s 25-year Total Cost (RDT&E, Procurement, and O&S) Estimates for C-130 AMP and Options 

C-130 AMP Option A Option B 

IDA’s 25-year total estimates (constant fiscal year 2013 dollars in billions) $2.12 $1.53 $.53

IDA’s 25-year net present value estimates (in billions of discounted dollars) 2.11 1.46 .51

Source: GAO presentation of IDA results 

IDA Took An Additional Step to Address Estimation Uncertainty

To deal with the cost estimating uncertainty, IDA performed cost-risk analysis developing a costrisk model and utilizing Monte Carlo simulation. For all but two of the variables in its model, IDA 

utilized triangular distributions defined by a most likely value—its point estimates—and lower 

and upper limits. For example, the RDT&E upper and lower limits were based on the range of 

RDT&E costs experienced on similar programs. For cost risk relating to the elimination of a 

navigator position from AMP, based on discussions with Air Force and U.S. Coast Guard 

Personnel, IDA considered the case where some tactical missions (e.g. airdrop) require a 

navigator or sensor operator even with AMP aircraft. To account for this risk, using analytical 

judgment, IDA defined a discrete distribution. Lastly, IDA accounted for variations in flying hours 

by utilizing a normal distribution centered on the C-130H planned flying program. IDA’s analysis 

accounted for correlation among inputs. For example, IDA’s model included positive correlation 

between procurement and depot-level costs as both depend on the cost of materials.

IDA ran one hundred thousand simulations, finding that in 98 percent of the simulations AMP’s 

net present value cost was greater than Option A. IDA also performed sensitivity analysis to 

identify facts that have the most effect on the AMP estimates, finding that AMP procurement 

cost was the most significant contributor to cost estimating variance, accounting of 72 percent of 

the observed variance. The second largest contributor was AMP depot-level reparable cost, 

which accounted for 13 percent.

IDA’s Analysis of Operational Effectiveness 

IDA defined effectiveness as the ability of a modernized C-130 fleet to enable C-130 

operations—consisting of combat delivery, non-combat cargo and passenger movement, and 

the necessary functions (e.g. training) to support the first two. IDA noted that given that aircraft 

availability is integrally tied to all aspects of operations, it should be included as a component of 

effectiveness. Overall, IDA found the AMP’s operational effectiveness advantage over Options 

A and B to be marginal. It stated that it found Option A to be at least as operationally effective 

as AMP since both AMP and Option A address diminishing manufacturing sources. Additionally, 

it noted that training and scheduling inefficiencies have largely been solved outside of avionics 

modernization and crew work loads are not an issue with Option A as it retains the navigator 

position.

In assessing the aircraft availability of the AMP and alternatives, IDA estimated the upper bound 

on expected improvements to the mission capable rate as a measure of availability as the 

mission capable rate is an indicator of whether the C-130 fleet is ready to perform its mission 
Page 13 GAO-14-547R C-130 Avionics Modernization Program

when needed.9

Another area included in IDA’s operational effectiveness assessment was combat delivery and 

passenger and cargo movement. IDA chose to include these areas as part of the effectiveness 

assessment because the primary mission of the C-130 is to deliver troops and cargo to the 

forward operating locations and drop-zones in a combat environment. Although the AMP and 

the reduced-scope alternatives were not intended to change combat delivery capabilities, 

elimination of the navigator in the AMP redistributes the workload and responsibilities of the 

remaining crew members during combat delivery. In evaluating the effectiveness of combat 

delivery of AMP and the alternatives, IDA examined whether AMP’s combat delivery would be 

affected by replacing the navigator position. IDA found that some subject matter experts believe 

that removal of the navigator position would increase crew workload, especially in high-threat 

environments. IDA reported that Operations Iraqi Freedom and Enduring Freedom, with the C130J and C-17 aircraft, which also eliminate the navigator position, indicates that some 

percentage of airlift missions require augmentation with a third pilot to mitigate crew workload. 

In summary, IDA noted that if additional crew members are needed for AMP, then AMP will cost 

more than it estimated in calculating AMP O&S costs as it gave AMP full credit for removal of 

the navigator in determining its base AMP O&S costs.

IDA examined historical mission readiness data for the C-130H fleet in 

determining the upper bound. It found that the best-case possible mission capable rate 

improvement that AMP could achieve was 1.4 percent. IDA then looked at the other extreme 

Option B. IDA found a best-case improvement for Option B of only 0.5 percent. IDA concluded 

that mission capable rate improvement is not a discriminating factor with regard to operational 

effectiveness. 

In evaluating non-combat cargo and passenger movement, IDA sought to determine if 

differences in required area navigation and navigation performance for AMP and the alternative 

options might impose restrictions on aircraft routing and airfield access, thereby affecting 

operational effectiveness. To determine the effectiveness of the modernization alternatives in 

this respect, IDA used two methodologies—assessing the effect of limited required area 

navigation and navigation performance capabilities on airfield approaches and assessing the 

impact to flying high-altitude routes. IDA found that AMP provides the most operational flexibility 

if the aviation community is converted to a predominately precision navigation-based system 

that completely replaces the current ground-based system. There are currently no definitive 

FAA plans to do so. IDA also found that the AMP alternatives could experience delays at large 

airfields because of their limited navigation approach capability—potentially affecting up to 33 

percent of C-130H sorties if certain larger airfields are required to adopt the newer technology. 

In assessing the effects to flying high altitude routes, IDA concluded that access to higher 

altitudes afforded by the AMP and Option A navigation capability result in modest fuel savings 

and increased payload capacity at extreme flying ranges and payloads, but may not be 

operationally significant for the most common routes and cargo weights. 

IDA also assessed operations support—involving primarily improvements to training and 

scheduling—for AMP and the alternatives. IDA’s objective was to determine if the different 

modernization options sufficiently standardized the C-130H fleet to eliminate or reduce 

inefficiencies associated with tailoring training to each variant of the aircraft. Standardization 

had been cited as a key reason for the C-130 AMP. To address this question, IDA obtained 

equipment lists for the C130H and sought to determine which avionics equipment varied across 

the C-130 fleet. IDA determined that there are eight avionics components that are the primary 

 9 Mission capable rate is a measure of an aircraft’s readiness to perform its missions.
Page 14 GAO-14-547R C-130 Avionics Modernization Program

drivers for variations in C-130H training. Option A would standardize four of those differences 

and Option B one, while AMP would eliminate all of the avionics component differences. IDA 

noted that while differences in training would exist under Options A and B, the present day 

version of this training is no longer the concern it was a decade ago as a result of the selective 

retirement of aircraft that has minimized the number of C-130 variants. IDA also noted that the 

fewer number of C-130H variants and sub-variants facilitates mission planning and scheduling. 

In summary, IDA found that while neither Options A nor B completely eliminates the need for 

differences in training, the significantly reduced number of C-130 variants has largely solved the 

training problems to the point that they are no longer of operational concern.
Page 15 GAO-14-547R C-130 Avionics Modernization Program

Enclosure II: Characteristics of High-Quality and Reliable Cost Estimates and Our 

Detailed Assessment of IDA’s Cost-Benefit Analysis

Our research has identified a number of best practices that are the basis of effective program 

cost estimating and should result in reliable and valid cost estimates that management can use 

for making informed decisions. These four characteristics of a high-quality and reliable cost 

estimate are that it is comprehensive, credible: well-documented, and accurate. 

• Comprehensive: The cost estimate should include both government and contractor 

costs of the program over its full life cycle, from inception of the program through design, 

development, deployment, and operation and maintenance to retirement of the program. 

It should also completely define the program, reflect the current schedule, and be 

technically reasonable. Comprehensive cost estimates should be structured in sufficient 

detail to ensure that cost elements are neither omitted nor double counted. Finally, 

where information is limited and judgments must be made, the cost estimate should 

document all cost-influencing ground rules and assumptions. 

• Credible: The cost estimate should discuss any limitations of the analysis because of 

uncertainty or biases surrounding data or assumptions. Major assumptions should be 

varied, and other outcomes recomputed to determine how sensitive they are to changes 

in the assumptions. Risk and uncertainty analysis should be performed to determine the 

level of risk associated with the estimate. Further, the estimate’s cost drivers should be 

crosschecked, and an independent cost estimate conducted by a group outside the 

acquiring organization should be developed to determine whether other estimating 

methods produce similar results.

• Well-documented: A good cost estimate—while taking the form of a single number—is 

supported by detailed documentation that describes how it was derived and how the 

expected funding will be spent in order to achieve a given objective. Therefore, the 

documentation should capture in writing such things as the source data used, the 

calculations performed and their results, and the estimating methodology used to derive 

each element’s cost. Moreover, this information should be captured in such a way that 

the data used to derive the estimate can be traced back to, and verified against their 

sources so that the estimate can be easily replicated and updated. The documentation 

should also discuss the technical baseline description and how the data were 

normalized. Finally, the documentation should include evidence that the cost estimate 

was reviewed and accepted by management.

• Accurate: The cost estimate should provide for results that are unbiased, and it should 

not be overly conservative or optimistic. An estimate is accurate when it is based on an 

assessment of most likely costs, adjusted properly for inflation, and contains few, if any, 

minor mistakes. Among other things, the estimate should be grounded in a historical 

record of cost estimating and actual experiences on other comparable programs.

Table 9 provides our detailed assessment of IDA’s cost estimating compared to best practices.
Page 16 GAO-14-547R C-130 Avionics Modernization Program

Table 9: Detailed Assessment of IDA’s Cost Estimating Compared to Best Practices

Characteristic Overall 

Assessment*

Best Practice Individual 

Assessment

Comprehensive Substantially 

Met

The cost estimate includes all life cycle costs Substantially 

Met

The cost estimate completely defines the program, reflects the 

current schedule, and is technically reasonable

Partially Met

The cost estimate is product-oriented, traceable to the 

statement of work/objective, and at an appropriate level of detail 

to ensure that cost elements are neither omitted nor doublecounted.

Substantially 

Met

The estimate documents all cost-influencing ground rules and 

assumptions. 

Partially Met

Well documented Partially Met The documentation should capture the source data used, the 

reliability of the data, and how the data were normalized.

Partially Met

The documentation describes in sufficient detail the calculations 

performed and the estimating methodology used to derive each 

element’s cost.

Partially Met

The documentation describes step by step how the estimate 

was developed so that a cost analyst unfamiliar with the 

program could understand what was done and replicate it

Partially Met

The documentation discusses the technical baseline description 

and the data in the baseline is consistent with the estimate

Partially Met

The documentation provides evidence that the cost estimate 

was reviewed and accepted by management.

Substantially 

Met

Accurate Partially Met The cost estimate results are unbiased, not overly conservative 

or optimistic and based on an assessment of most likely costs.

Partially Met

The estimate has been adjusted properly for inflation. Substantially 

Met

The estimate contains few, if any, minor mistakes. Substantially 

Met

The cost estimate is regularly updated to reflect significant 

changes in the program so that it is always reflecting current 

status.

Not Applicable

Variances between planned and actual costs are documented, 

explained, and reviewed. 

Not Applicable

The estimate is based on a historical record of cost estimating 

and actual experiences from other comparable programs. 

Partially Met

The estimating technique for each cost element was used 

appropriately.

Partially Met

Credible Substantially 

Met

The cost estimate includes a sensitivity analysis that identifies a 

range of possible costs based on varying major assumptions, 

parameters, and data inputs.

Partially Met

A risk and uncertainty analysis was conducted that quantified 

the imperfectly understood risks and identified the effects of 

changing key cost driver assumptions and factors.

Substantially 

Met

Major cost elements were cross-checked to see whether results 

were similar.

Substantially 

Met

An independent cost estimate was conducted by a group 

outside the acquiring organization to determine whether other 

estimating methods produce similar results.

Not Applicable

Source: GAO analysis of IDA’s data 

* Not Met - IDA provided no evidence that satisfies any of the criterion, Minimally Met – IDA provided evidence that 

satisfies a small portion of the criterion, Partially Met – IDA provided evidence that satisfies about half of the criterion, 

Substantially Met – IDA provided evidence that satisfies a large portion of the criterion, and Met – IDA provided 

complete evidence that satisfies the entire criterion.
Page 17 GAO-14-547R C-130 Avionics Modernization Program

Enclosure III: Objectives, Scope, and Methodology 

To conduct this work, we obtained IDA’s September 2013 cost benefit analysis study of the 

C-130 AMP program and reviewed it to determine the data and assumptions that IDA used to 

reach its conclusions and recommendations. To analyze IDA’s cost estimating, we relied on 

GAO’s Cost Estimating and Assessment Guide. In developing this guide, our cost experts 

assessed measures consistently applied by cost-estimating organizations throughout the federal 

government and industry and considered best-practices for the development of reliable costestimates. We analyzed the cost estimating practices used by the IDA in its analysis of the 

C-130 AMP and alternative options against these best practices. For reporting, we collapsed 

these best practices into four general characteristics for sound cost estimating: comprehensive, 

well documented, accurate, and credible.

We assigned each characteristic a rating of either met, substantially met, partially met, minimally 

met, or not met. We determined the overall assessment rating by assigning each individual 

rating a number: not met = 1, minimally met = 2, partially met =3, substantially met = 4, and met 

= 5. Then, we took the average of the individual assessment ratings to determine the overall 

rating for each of the four characteristics. The resulting average becomes the Overall 

Assessment as follows: not met = 1.0 to 1.4, minimally met = 1.5 to 2.4, partially met = 2.5 to 

3.4, substantially met = 3.5 to 4.4, and met = 4.5 to 5.0. A cost estimate is considered reliable if 

the overall assessment ratings for each of the four characteristics are substantially or fully met. 

If any of the characteristics are not met, minimally met, or partially met, then the cost estimate 

does not fully reflect the characteristics of a high-quality estimate and cannot be considered 

reliable. Because the cost estimates were developed for a cost-benefit analysis and not for a 

program office estimate, we did not assess two areas within the “accurate” characteristic—

regular updating of cost estimates and tracking cost variances—and one area within the 

credible characteristic—independent cost estimates.

Further, we interviewed agency officials from IDA, the Air Force Cost Analysis Agency, the 

Assistant Secretary of the Air Force Acquisition Office, and the Director of the Office of the 

Secretary of Defense’s Cost Assessment and Program Evaluation office to discuss the IDA 

study results. In addition, we reviewed IDA’s C-130 AMP annex report to determine the 

methodologies used to support findings in the September 2013 report. 

Given that IDA reported finding Option A nearly (if not as) operationally effective as AMP, and 

that costs were the apparent driving factor in IDA’s recommendation that the Air Force not 

pursue AMP, we limited our assessment of IDA’s cost estimating to AMP and Option A.
Page 18 GAO-14-547R C-130 Avionics Modernization Program

Enclosure IV: Comments from the Department of Defense
Page 19 GAO-14-547R C-130 Avionics Modernization Program

Enclosure V: Comments from the Institute for Defense Analyses

(121191)
This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately.
The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and 

policies; and provides analyses, recommendations, and other assistance 

to help Congress make informed oversight, policy, and funding decisions. 

GAO’s commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no 

cost is through GAO’s website (www.gao.gov). Each weekday afternoon, 

GAO posts on its website newly released reports, testimony, and 

correspondence. To have GAO e-mail you a list of newly posted products, 

go to www.gao.gov and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of 

production and distribution and depends on the number of pages in the 

publication and whether the publication is printed in color or black and 

white. Pricing and ordering information is posted on GAO’s website, 

http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537.

Orders may be paid for using American Express, Discover Card, 

MasterCard, Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube.

Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.

Visit GAO on the web at www.gao.gov.

Contact:

Website: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-

4400, U.S. Government Accountability Office, 441 G Street NW, Room 

7125, Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548

GAO’s Mission

Obtaining Copies of 

GAO Reports and 

Testimony

Order by Phone

Connect with GAO

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Congressional 

Relations

Public Affairs

Please Print on Recycled Paper.