Defense Acquisitions: DOD's Practices and Processes for Multiyear Procurement Should Be Improved
Released: 2008-02-07T00:00:00

Letter

Background

Multiyear contracting is a special authority for acquiring more than one year’s requirements —including weapon systems— under a single contract award without having to exercise an option for each program year after the first. It is an exception to the full-funding policy that requires the entire procurement cost of a weapon or piece of equipment to be funded in the year in which the item is procured. Under a multiyear procurement, DOD can contract for up to 5 years of quantities, although funding is still appropriated on an annual basis.

Multiyear procurement can potentially save money and improve the defense industrial base by permitting the more efficient use of a contractor’s resources. Multiyear contracts are expected to achieve lower unit costs compared to annual contracts through one or more of the following sources: (1) purchase of parts and materials in economic order quantities (EOQ), (2) improved production processes and efficiencies, (3) better utilized industrial facilities, (4) limited engineering changes due to design stability during the multiyear period, and (5) cost avoidance by reducing the burden of placing and administering annual contracts. Multiyear procurement also offers opportunities to enhance the industrial base by providing defense contractors a longer and more stable time horizon for planning and investing in production and by attracting subcontractors, vendors, and suppliers. However, multiyear procurement also entails certain risks that must be balanced against potential benefits, such as the increased costs to the government should the multiyear contract be changed or canceled and decreased annual budget flexibility for the program and across DOD’s portfolio of weapon systems. Additionally, multiyear contracts often require greater budgetary authority in the earlier years of the procurement to economically buy parts and materials for multiple years of production than under a series of annual buys.

Although DOD had been entering into multiyear contracts on a limited basis prior to the 1980s, the Department of Defense Authorization Act, 1982, codified the authority for DOD to procure on a multiyear basis major weapon systems that meet certain criteria. Since that time, DOD has annually submitted various weapon systems as multiyear procurement candidates for congressional authorization. Over the past 25 years, Congress has authorized the use of multiyear procurement for approximately 140 acquisition programs, including some systems approved more than once. Section 2306b of title 10, United States Code, governs the use of multiyear contracting authority for the procurement of property by DOD. It specifies six statutory requirements, or criteria, that an acquisition program is expected to meet in order to be considered for multiyear contracting. These criteria are listed in table 1 below.

Expected costs to be avoided should be sufficient to offset the added risk the government assumes with a multiyear contract in the form of a cancellation liability, decreased flexibility in future funding decisions and any erroneous assumptions in the estimates. Immature, volatile programs and those at risk of future changes should not be proposed as MYP candidates because such instability puts the savings attributed to efficiencies of production and EOQ buying at risk. The multiyear approach should be reserved for established production operations and low risk technology.

In submitting candidates for multiyear authorization by the Congress, the heads of the respective military departments vouch that each program complies with the criteria in table 1. Additionally, the Secretary of Defense is required to certify to Congress that the current future years defense program fully funds the support costs associated with the multiyear contract and that planned production will not be less than the minimum economic rates given the existing tooling and facilities. Multiyear contracts historically account for a substantial share of the defense procurement dollar. Figure 1 shows that DOD has budgeted about $10 million annually for multiyear contracts since fiscal year 2000, accounting for more than 13 percent of DOD’s total budget for procurement over this time frame. Over the period, the general trend shows an increase in total defense procurement, but multiyear obligations holding fairly steady, resulting in a downward trend of the percentage obligated on multiyear contracts. For 2007, the large increase in total defense procurement caused a drop below 12 percent obligated on multiyear.

DOD’s Practices and Processes for Justifying and Approving Multiyear Programs Could Be Improved

The statutory criteria for a multiyear procurement require that a candidate program make realistic cost estimates, expect to achieve substantial savings, and provide adequate evidence that the program is stable in terms of funding, requirements, and design. Some recent programs of questionable stability and savings submitted to the Congress for multiyear authorization raise concerns about DOD’s management and controls for justifying multiyear candidates. We found that DOD does not provide sufficient guidance and direction to ensure a rigorous, disciplined process supported by adequate empirical data for preparing and reviewing multiyear candidates. This increases the risk of poor outcomes and inappropriate, unstable programs approved for multiyear procurement.

Questions about the Appropriateness of Recently Approved Programs

We reviewed DOD’s multiyear justification data submitted in recent defense budgets and, in particular, examined two newly approved programs—the Air Force’s F-22A fighter and the joint V-22 tilt rotor aircraft. The F-22A acquisition has had a turbulent history with a lengthy development period, major cost increases and quantity decreases, changes in mission, and disagreements within DOD about the total number required. The Air Force’s submission of the F-22A for multiyear procurement generated considerable debate over its merits and whether it met the legal and business conditions conducive to success. We also examined the V-22 tilt rotor aircraft multiyear proposal, another acquisition program with a turbulent history and continuing challenges.

Funding and Requirements Issues on F-22A Raptor Program

The latest restructure of the F-22A acquisition occurred in December 2005. DOD extended production 2 years, added four aircraft and $1 billion in procurement funds, and proposed to buy the final 60 aircraft under two separate 3-year multiyear contracts for the airframes and engines. Multiyear costs and savings estimates were not completed in time for submission with the fiscal year 2007 defense budget. The Air Force later submitted the completed MYP justification package with estimated total multiyear costs of $8.7 billion and projected savings of $235 million, or 2.6 percent, compared to estimated annual contracts. Multiyear proponents cited the projected total dollar savings as substantial and believed there was little risk that the remaining 60 aircraft would not be procured. On the other hand, multiyear critics argued that the low percent of savings predicted, the short time frame for accruing savings, and the program’s relatively unstable past made it an inappropriate multiyear candidate.

In prior work, we determined that the restructured F-22A program was underfunded and questioned whether the proposed multiyear strategy met statutory criteria. We identified concerns about savings, funding, requirements, and design stability that we believed needed to be addressed before the multiyear plan could be justified. For example, the Air Force did not fully fund the multiyear proposal and asked for incremental funding. Also, a major development program to add new capabilities and improve reliability of the F-22A has begun; these efforts could result in future design modifications which may require retrofit onto aircraft purchased under the multiyear contract. We also noted that having only a 3-year period of performance at the end of production limited the ability to achieve savings normally expected under multiyear authority such as EOQ buys and cost reduction initiatives to improve manufacturing efficiency.

To provide for EOQ buys, Congress subsequently added $210 million to the F-22A advance procurement budget. In authorizing a multiyear contract for the F-22A, Congress specified certain conditions to be met and prohibited the use of incremental funding. The Secretary of Defense was required to certify that all statutory requirements have been met, including the determination that the contract will result in substantial savings of the total anticipated costs of carrying out the program through annual contracts. DOD submitted its certifications to the Congress in June 2007 and subsequently awarded the F-22A multiyear contract in August 2007. For the fiscal year 2009 budget cycle, the Air Force continues to propose buying more aircraft than the 183 in the current defense plan.

Stability and Production Concerns about the V-22 Osprey Program

DOD submitted the V-22 Osprey for multiyear procurement authorization in the fiscal year 2007 President’s Budget. Officials proposed a 5-year multiyear contract to acquire 185 aircraft for about $10.1 billion. Multiyear savings were projected at $435 million, or 4.2 percent, compared to the estimated costs for annual contracts.

Ongoing changes in quantity, funding, design, and concerns about production raise questions about the stability of this program and its appropriateness for multiyear contracting. Subsequent to congressional authorization for multiyear contracting, DOD reduced its planned procurements quantity from 185 to 167 due to service funding limitations with DOD cutting the proposed procurement request for fiscal year 2009 by $234 million.

Development and test efforts continue with a number of design changes under review to address serious safety, reliability, and performance problems. The program office has aggressively prioritized these issues and is making improvements to the V-22 platform by funding engineering design changes for the correction of deficiencies. One such deficiency is leaking hydraulic fluid causing engine compartment fires. Design changes to fix this deficiency are being studied and implemented. In comments on a draft of this report the program office stated it is confident that these engineering design changes will address the hydraulic leak problems.

To date, DOD has procured 111 aircraft in 11 years. Production aircraft continue to be conditionally accepted with deviations and waivers. Engineering investigations to fix these issues are not complete as the program continues to work to minimize these deviations and waivers. Even so, the planned production rate for the multiyear period is twice the current fiscal year 2007 production rate of 17 V-22s. This increase, coupled with design and production problems, raises concerns over the contractor’s ability to meet such a demand. DOD reviews and assessments of the V-22 production ramp up have endorsed the increase with known risks that require continued management. Officials told us that the supplier base should be able to meet the elevated production rate, but expressed concerns about the availability of spare parts and the challenges in managing manufacturing and installation at three different and dispersed facilities.

Insufficient Guidance and Management Direction by DOD to Ensure High Quality Decisions

The statutory criteria in section 2306b of title 10, United States Code, establish a framework for limiting multiyear contracts to very stable programs that appropriately balance risks with anticipated savings. DOD’s process and practices for justifying multiyear candidates, however, provide little specific guidance on the meaning and application of the criteria, a process that allows for subjective interpretations about how well a particular program meets the criteria. Cost and savings estimating techniques and data also vary considerably and inadequate records are kept to document decisions and supporting reasons. As a result, costs, savings, and evidence supporting stability is not consistently prepared, reviewed, and documented.

Concerns about DOD’s Processes and Practices for Justifying Multiyear Candidates

The statutory criteria are general in tone and qualitative by nature to provide application over a wide range of programs. The regulations DOD uses to implement the criteria provide contract policies and establish a process for developing and justifying multiyear candidates. Each candidate program prepares a budget justification package, normally for inclusion with the annual defense budget submission that provides funding requirements and estimated cost savings expected under a multiyear contract compared to the estimated costs for a series of annual contracts. The justification package also includes short statements about how programs are believed to meet each of the six statutory criteria. Weapon system acquisition program officials prepare the package for subsequent reviews by military service acquisition commands, service headquarters offices, and OSD offices responsible for program policy and budget oversight. Approved candidates are submitted to the Congress for authorization.

We discussed DOD’s justification process, multiyear contracting policies, and management practices with OSD and military service officials at each organizational level. We collected historical and budget data on approved multiyear programs since fiscal year 1982 and tracked more recent multiyear candidates through the budget process. We specifically examined nine major programs representing each military department: three had completed multiyear contracts, two were beginning multiyear contracts, and four programs were just recently authorized by the Congress. Several programs have been approved for more than one multiyear contract and, collectively, these programs accounted for 17 of the 59 approvals granted since 1996.

We found that DOD provides little in the way of supplemental guidance to operationalize the statutory criteria by amplifying terms such as “reasonable,” “substantial,” and “stable” and quantifying where possible to provide more objectivity and rigor to the multiyear review process. Guidance for the most part restates the statutory criteria and establishes formats for submitting budget justification materials, but does not provide much elucidation for interpreting and applying the criteria and establishing internal evidence standards for demonstrating criteria are met. From our review of justification packages and our discussions with DOD officials responsible for generating and reviewing multiyear justification packages, we determined that reviewers interpret and apply criteria differently and that the methods and data used to compute contract costs and savings and providing evidence to document program stability vary in quality and sophistication.

For example, officials we talked to at all levels of the review process had different ideas and perspectives on what constituted substantial savings when applying the criterion. An official in the Navy, for example, expected programs to project at least 10 percent savings, but would consider candidates under that level. Some Army officials wanted to see 10 percent savings or hundreds of millions of dollars. An OSD official applied a “rule of thumb” of 4 to 5 percent. He said that programs under that amount would typically be more closely evaluated to ensure they were viable, but that programs over that amount would generally receive a less detailed check of reasonableness and to ensure paperwork requirements were met. An Air Force official told us that a 5 percent savings level should be considered the floor for a genuinely viable candidate for multiyear and a 10 percent savings level achievable, although he cautioned that recent statutory changes to eligible cancellation ceiling costs will likely have a negative impact on future multiyear savings.

Further, review of the justification packages for the F-22A and the V-22, both submitted in the same fiscal year for approval, indicated differences in how the design stability criterion was applied. Initial operational capability, an important milestone for stability, had been declared 2 years prior to requesting multiyear authority on the F-22A, while initial operational capabilities for both variants of the V-22 are not scheduled to be achieved until after multiyear approval was granted, at least 2 years later for the Navy’s variant.

Comparing multiyear savings and judging reasonableness of the estimates is complicated because the techniques and data used to estimate cost savings can vary substantially in form, sophistication, and detail. For example, for its first multiyear contract proposal on the C-17A, the Air Force simply used the prime contractor’s offer to save 5.5 percent, a figure that was considered a “management challenge” the contractor believed it could meet. We found other instances where it was unclear what data was used in formulating the savings estimates. For example, the Army’s budget justifications for the Bradley Fighting Vehicle and Abrams Tank estimated savings at 5 percent and 10 percent, respectively, and provided very sketchy details to document how the savings estimates were derived and compliance with the stability criteria.

On the other hand, some analyses can be very involved, provide a range of estimated savings, and use independent third parties to validate data. For example, the Navy’s F-18E/F cost estimates and methodologies were independently verified by OSD’s Cost Analysis Improvement Group. For the F-22A multiyear proposal, two defense research centers developed contract costs and savings estimates. Several very different methodologies were employed, including cost improvement curves based on historical actual production costs, production cost estimate models for single year contracts less reductions for expected savings, and summation of savings initiatives from the prime and subcontractors. These studies produced a wide range of potential savings, from $235 million to $643 million.

Also, cost and savings estimates in general may be subject to biases and other factors that impact their fidelity and reliability. Our extensive past body of work on DOD’s major acquisitions suggests that to gain approval and continued funding for a weapon system, the acquisition environment encourages programs to submit overoptimistic estimates about a weapon system’s readiness for production and to underestimate its costs. Systems therefore appear more affordable from an investment perspective and can fit within forecasts of available funds. These circumstances invariably lead to acquisition programs costing substantially more than originally estimated. Furthermore, prior reports have discussed the importance of using present value analysis to account for the time value of money when evaluating and comparing costs of alternative annual and multiyear contracts. The timing of government expenditures is expected to differ with a multiyear contract expected to have relatively more up-front costs (to fund EOQ for example) and lower costs in the outyears compared to a series of annual contracts. Although the justification packages are required to have a present value analysis of the savings estimates, according to an OSD official and a defense research study, the cost savings estimates in then-year dollars are the primary estimates used in making cost decisions. Our review of the justification packages appear to confirm this because the section of the multiyear exhibit highlighting the benefit to the government contained only the then-year dollar estimate rather than the present value estimate.

Documentation and Record- Keeping Deficiencies

Through our discussions with officials and inspection of records, we determined that DOD’s review process for the multiyear justification budget packages does not adequately capture important information and events to document decisions and help ensure that consistent and reliable determinations are made regarding multiyear criteria. Once approved, OSD officials stated few records are kept on multiyear programs regarding how they determined whether multiyear candidates met the six statutory criteria. According to OSD and service officials, much of the discussion on a program proposed for multiyear should have already taken place during regular executive-level reviews of major weapon systems and been agreed upon before the multiyear justification package is reviewed for submission in the budget request. Review of the justification package then essentially becomes a paperwork formality rarely involving any surprises. Also, we found programs can be proposed “out of cycle” with the President’s Budget submission—as in the case of the F-22A, and may not be included in the budget details that could affect the review path the multiyear candidate takes to obtain approval. Without maintaining records that document decisions and the data supporting them, it is difficult to ensure the quality and comprehensiveness of stakeholder reviews based on the criteria, fidelity of the data used, and supporting rationales for decisions.

Finally, officials at every level of the multiyear justification process—from program offices, through higher headquarters, and on to primary OSD action offices—indicated that they recently were appointed to their current positions or the person responsible during the multiyear justification process was no longer in that position. We believe this contributed to “knowledge gaps,” historical record-keeping deficiencies, and differences in interpretation and application of multiyear decision criteria. Turnover implies loss of experience and corporate knowledge; in this environment, improved and more definitive guidance and retention of comprehensive historical records is even more important.

Actual Results on Some Completed Contracts Were Much Different Than Predicted in Budget Justifications

Implementing the statutory criteria requires realistic estimates of multiyear and annual contract costs. This requirement provides fidelity to savings projections and allows for accurate estimates of funding requirements over the life of the multiyear contract. We reviewed cost performance and results on completed multiyear contracts for the Air Force’s C-17A Globemaster transport, the Navy’s F/A-18E/F Super Hornet fighter, and the Army’s Apache Longbow helicopter. Although the precise impact of multiyear contracting is difficult to determine, our analysis shows that these programs—presumably approved based on their demonstrated stability—experienced substantial changes during contract execution. These changes significantly increased unit costs and drove up total funding requirements much beyond the estimates submitted to Congress in the budget justification materials. Each was also impacted by contract provisions and changes in business conditions.

Cost Increases Lead to Questions about Realism of Budget Estimates Used to Justify Multiyear

We found that unit cost growth on these programs ranged from 10 to 30 percent more than projected by the budget justification data. Table 2 shows the growth in unit and total contract costs. We also found that, for two of the three programs, actual multiyear contract costs exceeded the original estimates for annual contract costs. The third program, the F/A- 18E/F, came in below annual estimates, but also bought fewer systems than planned. This reduction in quantity would have also likely decreased annual costs had that alternative been selected.

Substantial cost increases for completed multiyear contracts on the three programs meant that Congress had to eventually provide considerably more funding than originally budgeted. We do not know how cost growth affected the level of savings achieved, if any, because we do not know how an alternative series of annual contracts would have fared. In comments on a draft of this report, DOD officials stressed that cost growth due to labor and material price escalation under a multiyear contract would likely have also occurred under an alternative series of annual contracts. The final MYP contract values in table 3 also include price increases resulting from engineering design changes made to the baseline weapon system. Although these factors may limit the ability to make inferences about the level of savings achieved, a case could be made that multiyear savings and costs did not materialize as presented in the multiyear justification materials.

As discussed earlier, our past body work suggests that defense acquisition programs are prone to underestimating costs and overstating readiness. While this tendency would apply to annual as well as multiyear contracts, it is arguably more problematic for multiyear contracts because of the government’s increased exposure to risk over multiple years. DOD officials agreed with us that multiyear contracting should be held to a high standard because of its special requirements, funding commitments, and risks.

Impacts on Cost and Performance from Contract Provisions and Other Factors

We also collected information on multiyear contract provisions for the three programs with completed multiyear contracts and for the recently awarded F-22A contract. Each of these programs awarded a fixed-price contract for the multiyear procurement, but they were not always firm- fixed-price contracts, which typically entail the least risk to the government. The multiyear contracts contained standard provisions that provided flexibility to increase or decrease costs based on inflation, labor rate changes, and/or material cost fluctuations. The multiyear contracts also included provisions for early cancellation, quantity variations, and/or design changes. In some cases, the government waived provisions for cost and pricing data, which according to officials decreased the government’s insight. Figure 2 below shows the frequency of the various contract provisions in our case studies.

Case Studies Provide Details on Meeting Cost and Program Expectations

The three case studies provide insight into multiyear contracting expectations and realities and the internal and external factors that affected actual execution. Each case raises some questions about the accuracy of cost and savings estimates used to justify multiyear procurement and the degree to which requirements, design, and funding were stable.

C-17A Globemaster: Risky Strategy Affected Multiyear Funding Stability

The estimated savings for the first C-17A Globemaster multiyear contract was simply a percentage amount submitted by the contractor. For the second multiyear contract, Air Force officials assumed the same percent of savings and added additional savings based on the use of a controversial funding strategy. This strategy relied on incremental funding, advanced buys of parts, and large potential cancellation liability to maintain a production schedule of 15 aircraft per year even though all of these aircraft had not been fully funded. The unfunded liability to the Air Force had the contract been canceled eventually grew to $1.5 billion. Concerned that this incremental funding strategy violated DOD’s full- funding policy and could potentially violate the Anti-Deficiency Act, Congress increased C-17A procurement funding a total of $745 million in fiscal years 2003 and 2005 to fully fund all aircraft. In annual DOD appropriations acts, Congress also has prohibited incremental funding of multiyear contracts.

In addition, the costs for both C-17A multiyear contracts were affected by economic price adjustments. On the first multiyear, overhead costs were significantly increased as a result of a merger between two major defense contractors. The Air Force subsequently paid $150 million to cover cost increases resulting from the merger and another $50 million to remove the clause from the contract. By the end of the multiyear contract, unit costs had increased 15 percent. The multiyear justification materials submitted to Congress supported a plan to buy 80 aircraft at an average cost of $179 million; instead the Air Force eventually paid about $207 million per aircraft. During the second multiyear contract, the contractor made large contributions to its pension fund, which triggered the price adjustment clause and resulted in a potential cost increase of over $530 million. The Air Force is in the process of restructuring the contract to reduce this amount.

The Air Force also awarded two multiyear contracts for the C-17A engine, the F117. These procurements appear to have been successful with demonstrated stability during the multiyear period and price breaks based on the multiyear contract. The F117 engine is a commercially available engine with a stable design and manufacturing process. There were no engineering or design changes; no advanced procurement or EOQ requirements; and no cancellation ceilings associated with either contract. The only potential cost risk was the economic price adjustment clause, but officials stated that the actual adjustments were not exclusively in the contractor’s favor. According to the program office, the first multiyear contract resulted in a savings of 10 percent, more than it had originally expected. Savings from the second multiyear contract are consistent with the original estimate.

F/A-18E/F Super Hornet: Design and Requirement Changes Impact Costs

For its first F/A-18E/F multiyear contract, the Navy did not award a firm- fixed-price contract because the program was early in the production phase and there were still ongoing design development efforts on the airframe. During the contract period, the economic price adjustment clause resulted in the Navy paying an additional $378 million because of labor rate and material cost increases. The first multiyear contract also included a variation-in-quantity clause that permitted an upward or downward adjustment of six aircraft per year. Annual quantities and the specific mix of buys between the two models changed more than once during the multiyear period. By the end of the multiyear contract, the number of aircraft procured had dropped from 222 to 210 aircraft and the average unit costs had increased by 10 percent, compared to the budget estimates.

In a May 2000 report, we had questioned whether the Navy was ready to enter into its first multiyear contract for full-rate production. Deficiencies identified during operational testing had not been corrected, and to avoid costly retrofitting and redesign, we believed that corrections should be made and tested before entering into the contract.

The Navy proposed buying another 210 Super Hornets on a second multiyear contract, but later changed the requirement to procure 154 Super Hornets and 56 of the new E/A-18G Growler, an electronic attack variant still in product development. The Navy’s total requirement for the Super Hornet had been reduced, and the new Growler was needed to replace aging EA-6B aircraft in the electronic attack mission. The follow- on multiyear also included a variation-in-quantity clause, but this time it only covered upward adjustments. Multiyear costs and funding were further impacted by the economic price adjustment clauses. As in the C-17A’s case, the F/A-18E/F multiyear contract was affected by the contractor’s large pension fund contribution. The Navy estimated that it could have been obligated to pay over $1 billion, which is nearly the same as the amount of cost savings originally estimated to justify the multiyear contract award. However, the Navy renegotiated the terms of the clause and restructured the contract to bring the price adjustment down to about $152 million.

Longbow Apache: Modifications and Unplanned Work Increased Unit Costs

The Army’s Longbow Apache helicopter experienced significant cost increases during both its multiyear procurements. Army officials stated that increases were largely because of aircraft modifications and unplanned work. These modifications included a voice data recorder and an improved rotor blade assembly that would enhance operational safety. Contract costs were also increased by additional unplanned work. Program officials explained that it is very difficult to predict the condition of fielded aircraft when they return to be upgraded or remanufactured. Along with normal wear and tear, many operational aircraft were returned with extensive corrosion and battle damage; others had been cannibalized for parts. Remanufacturing these aircraft required significantly more effort and funding than originally planned. By the end of the first multiyear contract, the Apache’s average unit cost had increased by 30 percent; at the end of the second multiyear, these costs had increased by 25 percent.

While it may be difficult to predict unusual wear and tear on a system and it is common to incorporate new modifications over time, it is especially problematic to roll these costs into the multiyear contract that had been assumed stable and that had been justified based on initial cost estimates without these new add-ons. OSD cost analysts are studying this issue to determine the proper accounting for modifications under multiyear contracting.

Estimated Savings for Candidate Programs Are Trending Downward

The amount and percentage of savings expected from a multiyear contract compared to a series of estimated annual contracts is the most visible and perhaps the most critical criteria in the eyes of many stakeholders. The savings requirement in definition and application has evolved over the years. A threshold level of 10 percent savings emphasized during the 1980s was eliminated and replaced with a nonquantifiable requirement for “substantial savings” since fiscal year 1991, allowing wide flexibility in its interpretation. Although a direct causal link is not demonstrated, our analysis of multiyear programs approved by Congress shows that estimated savings were on average higher before the substantial savings requirement than after. Other factors—lower quantities of modern systems being procured, stricter termination liability allowances, and contraction in the defense industrial base—may also contribute to decreased estimated savings for current and future systems by lessening the benefits from large quantity buys and efficient production rates, key attributes of multiyear contracts.

Multiyear Savings Percentages Were Generally Lower after Changes in the Statutory Savings Criteria

When Congress codified the authority for multiyear procurement contracting in December 1981, there was no specific savings criterion in the law that candidate programs had to meet. However, the impetus behind multiyear was provided by DOD studies at the time predicting savings averaging 10 to 20 percent. The 10 percent figure became a savings benchmark for decision-makers in the early 1980s when judging the merits of candidate programs. In the late 1980s, this benchmark became a threshold requirement for many candidate programs as Congress began stipulating a 10 to 12 percent savings amount in annual defense authorization acts for selected programs.

In November 1989, Congress decided to codify the 10 percent savings requirement and other conditions and limitations previously imposed on an annual basis. However, a year later, that threshold was struck from the U.S. Code, and the requirement for “substantial savings” was substituted after DOD had argued that a rigid threshold limited the potential for savings on stable, low risk programs projecting lesser savings amounts. This substantial savings requirement has remained unchanged since November 1990, with specific savings requirements stipulated in annual legislation for two candidate programs approved by Congress during this time. Figure 3 illustrates the evolution of the savings requirement.

Our analysis of estimated savings for approved multiyear programs determined that the average savings level trended lower after the substantial savings criterion was adopted. Although programs have been approved during both eras over a wide range of savings below and above 10 percent, the change in the law provided decision makers the flexibility to propose and approve candidate programs since fiscal year 1991 with lower savings estimates on average compared to the 1980s. As previously discussed, an unofficial rule of thumb for savings normally expected is now down to 4 to 5 percent.

Figure 4 illustrates the range of savings and general trend as the savings requirement evolved. It suggests that a shift did occur since the change to substantial savings in 1991. A larger proportion of multiyear programs with estimated savings of less than 10 percent were approved after 1991. On the other hand, candidate programs approved prior to 1991 show a larger proportion of savings of 10 percent or more. This finding is supported by a defense consultant study that calculated estimated savings averaged 13 percent for candidate programs from 1982 to 1989.

Other Factors Affecting Estimated Savings

In addition to the revised savings requirement, other factors may also be impacting the level of savings for current and future multiyear programs: Smaller quantities procured. The higher cost for modern weapon systems and changes in required force structure has resulted in generally smaller procurement quantities for new systems compared to predecessor systems. Smaller quantities of systems bought under multiyear contracts may not provide the same opportunity to achieve savings through such mechanisms as EOQ buys of parts and materials. Past multiyears, on the F-16 Falcon aircraft and Black Hawk helicopters, for example, procured hundreds of systems. In contrast, the more recent multiyear procurement of the F/A-18E/F Super Hornet procured less than half the F-16 quantity. Analysis of the data on MYP candidate programs that we collected and summarized in table 3 below showed that the median multiyear procurement quantity for aircraft MYP candidate programs declined over 40 percent from the 1980s to the present era with a concurrent decline in average savings. With fewer aircraft procured during the multiyear period, savings from economic buys and optimized production is typically smaller.

Cancellation liability changes. If a multiyear contract is cancelled, the government is liable for reimbursing the contractor for its incurred costs up to a negotiated cancellation ceiling typically in the contract. Until recently, DOD has been able to include both recurring and nonrecurring costs. Multiyear contracts awarded during the past 25 years have included a cancellation liability, but the cancellation ceiling for the C-17A multiyear contract awarded in 2002 was considered very large with a potential liability of more than $1.5 billion. Some members of Congress were concerned by its size and concluded that this large liability inappropriately committed the government to a production schedule for which funding had not been appropriated. As a result, for the past several years Congress has limited the cancellation liability for multiyear contracts to nonrecurring costs only. Some DOD officials expect this change to limit savings on current and future multiyear contracts as contractors choose not to bear financial risks previously borne by the government. A major source of multiyear savings is the EOQ buys, and without including these kinds of recurring costs in the cancellation ceiling, fewer contractors may buy supplies and materials up front in bulk to limit their risks should the multiyear contract be cancelled early.

Declining competition. Some DOD and defense research center officials believe that the consolidation and contraction of the defense industry and resulting decline in competition and contraction among vendors and suppliers, make it harder to wring savings from EOQ buys. For example, an OSD official stated that because the F-22A was originally designed well over a decade ago, it is now experiencing diminishing manufacturing sources on many components as well as parts and equipment obsolescence. Similarly, a defense research center official believes that diminishing manufacturing sources negatively impacted the multiyear savings potential for the F-22A.

Multiyear Contracting Results Are Uncertain

DOD does not perform post contract assessments of completed multiyear contracts to validate actual results and determine cost effectiveness of its investments. Some prior studies, including GAO work, provide some limited, but inconclusive insights into multiyear results and benefits. OSD and the military services do not maintain adequate, comprehensive records on historical and current multiyear contracts that could facilitate better tracking and provide perspective for future multiyear efforts.

DOD Does Not Have a Formal Process for Assessing and Tracking Multiyear Contract Results

DOD does not have a formal process for assessing the cost and performance of completed multiyear contracts. According to DOD officials, once a program is approved for multiyear, they do not track subsequent performance nor validate actual results against expectations established in the budget justification submissions. Also, they do not have an official method to capture and share lessons learned with other programs considering multiyear contracts that could improve prospects for successful outcomes.

However, we did find that individual program offices may make efforts to ascertain benefits and learn lessons that can be applied to future multiyear submissions, but it appears that these efforts are isolated. Based on prior multiyear experiences, F/A-18E/F Super Hornet program officials decided to require cost and pricing data to better inform cost estimates for their next multiyear application. Also, they based expected savings on cost reduction initiatives rather than EOQ buys because they believed these initiatives had a better return on investment. Similarly, the results of our case studies discussed earlier demonstrate that assessing actual results can glean valuable information about contract costs and performance that can be used to improve future multiyear outcomes.

Prior Studies of Multiyear Cost Effectiveness Show Limited and Inconclusive Results

Some attempts to assess historical multiyear performance have been made, but validating actual savings is elusive. According to DOD and defense research center officials as well as the studies they conducted, calculating the actual cost savings from the use of a multiyear contract and comparing results to original expectations is very problematic for several reasons: (1) multiyear cost and other program data is unavailable; (2) lack of comparable data on costs of annual contracts; and (3) original assumptions change from the justification package, such as design modifications and variations in buy quantities, labor, and material rates.

Recent studies by two defense research centers attempted to gain a historical perspective on actual multiyear savings achieved for past contracts in order to provide context as to the relative level of expected savings for the F-22A multiyear proposal. In reports, both centers noted that the government does not collect or save data needed to do a detailed analysis and that, once programs are approved and implemented, important assumptions on which original savings estimates were based often changed.

In particular, since a series of annual contracts are not executed for the same procurement once a multiyear strategy has been adopted, comparisons of actual multiyear costs can only be compared to hypothetical estimates of annual contracts. Further, changes that occur in a multiyear contract environment can also occur in an annual contract environment, and the exact effect on the actual costs for the annual contracts is unknown. In reviewing the literature, one center noted that it found “very few examples of serious and methodologically credible attempts to validate claimed savings and savings percentages after the fact once programs had been completed,” and none that produced, in the center’s opinion, definitive findings.

In the case of the F/A-18E/F program, researchers from one center felt they had more data available on this multiyear contract than for others, and they attempted to validate actual savings. They concluded that while the available data supported savings in the neighborhood of original estimates, a definitive answer would require a more detailed analysis of EOQ data and cost reduction initiatives. Similarly, in our 1988 report on the first F-16 multiyear procurement, we concluded that savings were likely achieved from EOQ buys in the order of magnitude expected, but we were unable to make definitive conclusions about total savings achieved.

Inadequate Records Maintained on Current and Historical Multiyear Programs

Undergirding attempts to track and assess multiyear performance is the quality and sufficiency of data. As discussed earlier in this report, we found that DOD at all organizational levels does not keep adequate records on multiyear programs to document stakeholder reviews and the empirical evidence used to justify multiyear contracts. OSD Comptroller officials told us that, while they are the final authority on approving multiyear candidates and the ultimate owner of the review process, they do not track multiyear packages through the approval process and after the final decision has been made to submit it to the Congress. DOD has no comprehensive, central information system that records the status of multiyear candidates in-process, candidates that have been approved or disapproved, detailed information on how multiyear criteria were applied, or information on specific criteria contained in the final justification package.

DOD is not required to, nor does it maintain a central database of historical or ongoing multiyear contracts. In response to our inquiries, the OSD office ultimately responsible for the multiyear procurement review process was unable to provide us the justification packages for over half of the programs approved by Congress since 1992, including very recent submissions. To obtain more complete and accurate data that could be used to track performance and conduct trend analyses, information must be compiled from many different sources, including budgets, program office files, contractor studies, and contracting databases. For example, our efforts to identify and track multiyear contract information contained in two major federal databases were only partially successful. Some valuable multiyear contract information was readily available through these sources. However, the type of data stored and storage format—as well as issues pertaining to reliability, consistency, and comparability— limited their current usefulness in tracking and evaluating multiyear contracts. However, with some improvements particularly with reliability, these databases could support future studies.

Conclusions

The statutory criteria for approving the multiyear procurement of major DOD weapon systems clearly establish requirements to limit multiyear authorization to stable, low risk programs with realistic cost estimates and reasonable expectation for savings. To move forward otherwise is to accept significant risks with little chance of reward. However, DOD does not have an adequate process with controls in place to ensure multiyear candidates meet all the criteria and are supported by sufficient empirical evidence. Inconsistent application of criteria, questionable cost and savings estimates, and inadequate documentation increase potential for approving inappropriate, unstable multiyear programs and incurring costly, poor outcomes when plans go awry and conditions change. Improving guidance, ensuring decisions are informed by knowledge, and maintaining better records are critical needs, as well as important tools for retaining corporate memory given frequent turnover of personnel at all levels of the justification review process. It is not possible to calculate accurately the cost to taxpayers that has resulted from these conditions, but the lack of a disciplined and rigorous process that demands knowledge about stability and costs provides potential for significant waste of taxpayer dollars.

Furthermore, DOD does not track and evaluate actual performance on multiyear contracts for major DOD weapon systems. Once a contract is awarded for a multiyear program, little effort is made to collect data and assess actual results to compare performance against original expectations and to validate savings and other benefits achieved. Assessing results could provide valuable insights and lessons learned on prior experience and identify opportunities to improve future multiyear procurements. Not having a clear picture of actual performance further emphasizes the criticality of getting it right up front by ensuring only appropriate programs go to Congress for approval.

Therefore, despite a long history and substantial funding for major DOD weapons system multiyear contracts, DOD does not know whether it has gotten a reasonable return on its investments for the extra risks incurred. Some concerns noted in this report, such as the practice of understating costs and overselling benefits, apply also to annual contracts, but the standards should be higher for multiyear contracts because of the larger up-front investments required and the government’s exposure to risk should the program fail or be substantially changed. Strengthening both the front end of the process—identifying and justifying good candidates— and the back end—assessing results and gleaning lessons learned from completed contracts—can help ensure costs and risks are adequately balanced for new multiyear programs and improve future outcomes. Underpinning both ends, it is important to capture and make available essential data on multiyear decisions and subsequent performance that can be readily accessed by stakeholders and prospective users of multiyear procurement authority.

Recommendations

To improve the outcomes of the multiyear justification reviews of major DOD weapon systems, the Secretary of Defense should direct appropriate offices within the Under Secretary of Defense (USD) (Comptroller) and USD (Acquisition, Technology & Logistics) to: (1) improve and expand guidance provided to military services to better define multiyear decision criteria for major DOD weapon systems and to facilitate more consistent, objective, and knowledge-based evaluations of these multiyear candidates within DOD; (2) establish a process for third party validation of the costs and savings data submitted for such candidate programs; and (3) implement a central database for maintaining historical records and for effectively monitoring and tracking major DOD weapon system multiyear procurements, to include documenting the specific decisions made by stakeholders and their rationales for decisions.

To provide lessons learned for informing and improving future major DOD weapon system multiyear candidate programs and to ensure DOD is earning a sufficient return on its investments in multiyear contracts for major DOD weapon systems, the Secretary of Defense should direct that the responsible military service, in conjunction with appropriate elements within OSD, conduct after-action assessments at the conclusion of all multiyear contracts used to procure major DOD weapon systems to determine their effectiveness in achieving predicted benefits while managing associated risks. These assessments should identify major deviations, if any, between the unit costs predicted in the multiyear justification package and the unit costs actually incurred. The assessments should also substantiate—to the extent practicable—savings achieved and identify reasons and causes contributing to overall performance results and attemp to isolate those issues peculiar to the multiyear contract from those that would likely have also affected annual contracts if a multiyear strategy had not been employed. Internally, DOD should use the results of these assessments to provide lessons learned to both industry and the government that can help inform and lead to better and more supportable decisions on future multiyear candidate programs.

Agency Comments and Our Evaluation

DOD provided us with written comments on a draft of this report. The comments appear in appendix II. DOD also separately provided technical comments which we reviewed and incorporated as appropriate. In written comments, DOD concurred with our two recommendations to improve guidance and to implement a central database for maintaining records and tracking multiyear programs. DOD partially concurred with the other two recommendations.

DOD partially concurred with our recommendation to establish a process for third party validation of the costs and savings data submitted for candidate programs. In its comments, DOD stated that independent third party validations of cost and savings are done on selected programs and, in developing new guidance in response to our first recommendation, would consider whether the benefits of requiring validation on all programs warrant the delays and costs of validation. Our review of five new proposals and six approved multiyear contracts found only one such instance, and our discussions with service and OSD officials show that third party validations are rare. We believe that independent third party validations would result in more accurate and comprehensive cost and savings information critical to congressional and DOD decision making on multiyear candidates. Our review identified inconsistent practices in preparing and reviewing multiyear proposals and varying degrees of quality and completeness in the initial cost and savings estimates made by the weapon system program offices. An independent third party check would help ensure that appropriate multiyear candidates are submitted to the Congress for approval.

DOD also partially concurred with our recommendation that after-action assessments be conducted to provide lessons learned for informing and improving future multiyear candidate programs. In its comments, DOD agreed that after-action reports may be of value for certain multiyear programs, but questioned the value for all programs because (1) the extensive time before the assessment results are known and can be applied; and (2) the difficulty in determining actual savings. The intent of the recommendation is to learn lessons that can be applied to strengthen future multiyear proposals and improve their prospects for success. This in formulation is not time-bounded as DOD has contracted for studies that drew useful lessons from programs many years earlier. Collecting and distributing data on lessons learned would provide a continuing database of knowledge for future programs. Furthermore, while we recognize difficulties and constraints in calculating actual savings, this does not preclude the department from making good faith efforts that can provide valuable, albeit imperfect information. Also, after-action assessments include more than savings calculations. As pointed out in the report, one program assessed itself and identified important contractual features and other factors that it used to improve subsequent multiyear proposals. We note that the practice of doing after-action reports is widespread in the department and used for many different kinds of activities, including military contingency operations and logistics functions, and that these efforts provide planners and decision makers with critical lessons learned for applying to and improving future actions.

We are sending copies of this report to the Secretary of Defense; and the Secretaries of the Air Force, Army, and Navy; and the Director of the Office of Management and Budget. We will provide copies to others on request. This report will also be available at no charge on GAO’s Web site at http://www.gao.gov. Key contributors to this report were Bruce Fairbairn, Assistant Director; Noah Bleicher; Matthew Drerup; Mary Jo Lewnard; Rae Ann Sapp; and Bob Swierczek.

If you have any questions about this report or need additional information, please call me at (202) 512-4841 (sullivanm@gao.gov). Contact points for the offices of Congressional Relations and Public Affairs are located on the last page of this report.

Appendix I: Scope and Methodology

Our review was limited to major DOD weapon systems that have received congressional approval in annual defense authorization and/or appropriations acts to award a multiyear contract and meet the statutory requirements identified in 10 U.S.C. § 2306b. Work was performed at the Offices of the Secretary of Defense and the three military service headquarters (Navy, Army, and Air Force), in Washington, D.C.; Naval Air Systems Command, in Patuxent River, Maryland; Aviation Missile Command, in Huntsville, Alabama; Aeronautical Systems Center, at Wright-Patterson Air Force Base, Ohio; Institute of Defense Analysis (IDA), Alexandria, Virginia; and RAND National Defense Research Institute, Arlington, Virginia.

To evaluate DOD’s multiyear review process, we compiled a list of the all the candidate programs approved by Congress to use a multiyear contract by examining DOD authorization and appropriations acts going back to fiscal year 1982 when the statutory language in 10 U.S.C § 2306b was first enacted. We then reviewed DOD multiyear justification packages submitted to Congress in recent defense budgets, identified statutory criteria authorizing the use of multiyear procurement, and considered regulatory policies and procedures used within the Services and at OSD to prepare and evaluate multiyear justification packages. We discussed with officials at acquisition program offices and at higher command review levels how they interpreted and applied the statutory criteria and guidance to evaluate the appropriateness and feasibility of multiyear candidates. We reviewed DOD and congressional actions on recent multiyear candidates and examined specifically two major programs recently approved for multiyear contracting—the F-22 Raptor and V-22 Osprey—to illustrate how this process works and address some questions raised about the appropriateness of these candidates and data used in the justification packages. For these systems, we extensively drew upon GAO’s work in prior and ongoing engagements.

We conducted limited case studies for selected multiyear aircraft contracts to assess outcomes and the internal and external events affecting performance. Because our sample of DOD aircraft multiyear contracts was not randomly selected, we cannot project our findings to other programs. These case studies included three major DOD weapon systems—C-17A Globemaster, F/A-18E/F Super Hornet, and the Apache Longbow Helicopter. These aircraft programs have fully executed at least one multiyear contract in the recent past and also have ongoing follow-on multiyear contracts. We reviewed the multiyear proposal packages submitted to Congress, annual budget information, Selected Acquisition Reports, contract file documentation, and information in DOD contract databases for multiyear contracts awarded by these programs. We calculated unit cost changes and identified key programmatic and environmental changes impacting the execution of the multiyear programs and compared these to the original projections in their multiyear justification packages. To identify the characteristics of multiyear contracts and how they affect the costs, risks, and savings for selected multiyear aircraft contracts, we reviewed contract file documentation and information on the DOD DD 350 Individual Contract Action Report (ICAR) database and the Federal Procurement Data System-Next Generation (FPDS-NG). As part of the case study approach, we reviewed the types of contracts, contract clauses, and other contract modifications to determine how they affected the unit costs under a multiyear contract. Further, we discussed the multiyear contracts included in our limited case study approach and the F-22A program’s August 2007 multiyear contract award with program officials to help us assess the effects that specific contract provisions have on unit costs during contract execution.

To research the legislative history underlying the adoption and subsequent repeal of the 10-percent savings requirement and the current requirement that defense multiyear procurement contracts achieve “substantial savings,” we reviewed the evolution of multiyear savings criteria in early defense initiatives, the fiscal year 1982 codification, subsequent amendments, and the savings criteria contained in annual authorization and appropriations acts subsequent to the granting of multiyear authority. To make comparisons in average and median levels of estimated savings on multiyear candidate programs as the criteria for awarding a multiyear contract changed, we reviewed savings estimates in the budget justification packages for multiyear candidates submitted to Congress since 1982 and past GAO reviews of multiyear candidates conducted in the 1980s. We were able to obtain savings estimates for approximately 94 of the 141 approvals granted by Congress to award a multiyear contract.

To determine the extent to which DOD tracks performance and validates savings and other benefits actually achieved by multiyear contracts, we evaluated the kind and extent of cost data and program information maintained at the services and OSD, and how they use these data to determine whether predicted savings and other benefits were actually achieved by multiyear contracting. We also reviewed DOD policies and guidance for estimating and validating multiyear savings, and discussed with DOD officials practices used to estimate and monitor multiyear savings. We discussed with DOD officials record-keeping and management oversight requirements and reasons why they do not have a formal process for assessing multiyear results. We reviewed two recent major studies done by defense research firms that summarized estimated savings on historical programs and performed case studies on selected programs to identify key events and practices that affected ultimate performance. We discussed with IDA and RAND officials their cost estimating techniques and assumptions used in their F-22A multiyear studies that supported the planned multiyear contract. We reviewed and summarized their attempts to validate savings and other benefits from prior multiyear programs. We also reviewed prior GAO work on selected weapon systems and the results of our work during the 1980s when Congress regularly asked us to review DOD’s multiyear candidates.

We conducted this performance audit from June 2007 to February 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contacts and Staff Acknowledgments

Staff Acknowledgments