Defense Acquisitions: DOD's Practices and Processes for Multiyear Procurement Should Be Improved
Published: 2008-02-07T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: DOD spends $10 billion annually on multiyear procurement (MYP) contracts for weapons systems. MYPs may save money through more efficient relationships with suppliers and producers, but may also suffer losses if cancelled and can limit future budget flexibility. Recently, Congress has been concerned about DOD's management of the process and savings realized by MYPs. GAO was asked to evaluate DOD's review process for MYP candidates; examine MYP program outcomes; identify the impact of changes to MYP savings threshold guidance, and determine how much DOD validates MYP performance. To do this, GAO reviewed statutes and other guidance, held discussions with relevant officials, examined DOD budget justifications and contracts, and conducted limited case studies.
section_title: What GAO Found
paragraphs: DOD's process for justifying multiyear programs leaves questions about the appropriateness of some approved MYPs and the cost effectiveness of investments made for the risks assumed, as indicated by recent submissions for the F-22A and V-22. Although the law has clear requirements for stable, low risk programs with realistic cost and savings estimates, lack of guidance and a rigorous process is not achieving this. It is difficult to precisely determine the impact of multiyear contracting on procurement costs. GAO studies of three recent MYPs identified unit cost growth ranging from 10 to 30 percent compared to original estimates, due to changes in labor and material costs, requirements and funding, and other factors. In some cases, actual MYP costs were higher than estimates for annual contracts. Although annual contracts also have unit cost growth, it is arguably more problematic for MYP's because of the up-front investments and the government's exposure to risk over multiple years. MYP savings were on average higher before changes in law called for "substantial savings" rather than a specific quantitative standard. Other factors--lower quantities of modern systems procured, stricter cancellation liability allowances, and contraction in the defense industrial base--may have also impacted savings by lessening opportunities for more efficient purchases, a key attribute of MYPs. DOD does not track multiyear results against original expectations and makes little effort to validate if actual savings were achieved. GAO's case studies indicate that evaluating actual MYP results provides valuable information on the veracity of original estimates in the justification packages, the impacts on costs and risks from internal and external events, and lessons learned that can be used to improve future multiyear candidates and savings opportunities.

section_title: Letter
paragraphs: For the past 25 years, the Department of Defense (DOD) has used  multiyear contracts to procure thousands of major weapon systems,  investing about $10 billion annually in recent years. Multiyear  procurement is a special authority to contract for up to 5 program years of  requirements in one year. When used appropriately, multiyear contracting  can save money compared to a series of annual contracts by allowing  more economic procurement from suppliers and more efficient  production. Multiyear contracting can also entail some risks of substantial  losses if a program is reduced or a contract is cancelled early and can limit  DOD’s and the Congress’ future budget flexibility. To identify multiyear  candidates that demonstrate sufficient benefits and manageable risks, the  law requires certain criteria be met before an agency can enter into a  multiyear contract.
During discussions on the President’s Budget for fiscal year 2007,  committee members were concerned about the amount of projected  savings realized on multiyear contracts and had questions about the DOD’s  management and oversight of programs with multiyear contracts.  Accordingly, you asked us to (1) evaluate DOD’s review process for  multiyear procurement candidates submitted to Congress for approval; (2)  examine cost and program outcomes on selected multiyear programs; (3)  identify impacts from changes in guidance and interpretation of the  savings requirement; and (4) determine the extent to which DOD tracks  and validates multiyear performance.
In conducting our work, we identified statutory criteria and implementing  policies and procedures used by DOD to prepare and evaluate multiyear  justification proposal packages. We discussed management oversight,  practices, and results of the justification process with cognizant officials  from each of the military departments and the Office of the Secretary of  Defense (OSD). We examined DOD historical budgetary and contractual  records to compile and summarize data on prior and current multiyear  programs. We looked in more detail at nine major programs representing  each military department: three had completed multiyear contracts, two  were beginning multiyear contracts, and four programs were just recently  authorized by the Congress. Several programs have been approved for  more than one multiyear contract and, collectively, these programs  comprised almost one-third of approvals granted since 1996. We  researched the legislative history on the required savings criterion and  DOD’s efforts to interpret and apply it. We reviewed and discussed two  recent multiyear studies conducted by the defense research organizations,  RAND National Defense Research Institute and the Institute for Defense  Analyses (IDA). Appendix I further explains the report’s scope and  methodology. We conducted this performance audit from June 2007 to  February 2008 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Multiyear contracting is a special authority for acquiring more than one  year’s requirements —including weapon systems— under a single contract  award without having to exercise an option for each program year after  the first. It is an exception to the full-funding policy that requires the entire  procurement cost of a weapon or piece of equipment to be funded in the  year in which the item is procured. Under a multiyear procurement, DOD  can contract for up to 5 years of quantities, although funding is still  appropriated on an annual basis.
Multiyear procurement can potentially save money and improve the  defense industrial base by permitting the more efficient use of a  contractor’s resources. Multiyear contracts are expected to achieve lower  unit costs compared to annual contracts through one or more of the  following sources: (1) purchase of parts and materials in economic order  quantities (EOQ), (2) improved production processes and efficiencies, (3)  better utilized industrial facilities, (4) limited engineering changes due to  design stability during the multiyear period, and (5) cost avoidance by  reducing the burden of placing and administering annual contracts.  Multiyear procurement also offers opportunities to enhance the industrial  base by providing defense contractors a longer and more stable time  horizon for planning and investing in production and by attracting  subcontractors, vendors, and suppliers. However, multiyear procurement  also entails certain risks that must be balanced against potential benefits,  such as the increased costs to the government should the multiyear  contract be changed or canceled and decreased annual budget flexibility  for the program and across DOD’s portfolio of weapon systems.  Additionally, multiyear contracts often require greater budgetary authority  in the earlier years of the procurement to economically buy parts and  materials for multiple years of production than under a series of annual  buys.
Although DOD had been entering into multiyear contracts on a limited  basis prior to the 1980s, the Department of Defense Authorization Act,  1982, codified the authority for DOD to procure on a multiyear basis  major weapon systems that meet certain criteria. Since that time, DOD has  annually submitted various weapon systems as multiyear procurement  candidates for congressional authorization. Over the past 25 years,  Congress has authorized the use of multiyear procurement for  approximately 140 acquisition programs, including some systems  approved more than once. Section 2306b of title 10, United States Code,  governs the use of multiyear contracting authority for the procurement of  property by DOD. It specifies six statutory requirements, or criteria, that  an acquisition program is expected to meet in order to be considered for  multiyear contracting. These criteria are listed in table 1 below.
Expected costs to be avoided should be sufficient to offset the added risk  the government assumes with a multiyear contract in the form of a  cancellation liability, decreased flexibility in future funding decisions and  any erroneous assumptions in the estimates. Immature, volatile programs  and those at risk of future changes should not be proposed as MYP  candidates because such instability puts the savings attributed to  efficiencies of production and EOQ buying at risk. The multiyear approach  should be reserved for established production operations and low risk  technology.
In submitting candidates for multiyear authorization by the Congress, the  heads of the respective military departments vouch that each program  complies with the criteria in table 1. Additionally, the Secretary of Defense  is required to certify to Congress that the current future years defense  program fully funds the support costs associated with the multiyear  contract and that planned production will not be less than the minimum  economic rates given the existing tooling and facilities. Multiyear  contracts historically account for a substantial share of the defense  procurement dollar. Figure 1 shows that DOD has budgeted about $10  million annually for multiyear contracts since fiscal year 2000, accounting  for more than 13 percent of DOD’s total budget for procurement over this  time frame. Over the period, the general trend shows an increase in total  defense procurement, but multiyear obligations holding fairly steady,  resulting in a downward trend of the percentage obligated on multiyear  contracts. For 2007, the large increase in total defense procurement  caused a drop below 12 percent obligated on multiyear.
section_title: DOD’s Practices and Processes for Justifying and Approving Multiyear Programs Could Be Improved
paragraphs: The statutory criteria for a multiyear procurement require that a candidate  program make realistic cost estimates, expect to achieve substantial  savings, and provide adequate evidence that the program is stable in terms  of funding, requirements, and design. Some recent programs of  questionable stability and savings submitted to the Congress for multiyear  authorization raise concerns about DOD’s management and controls for  justifying multiyear candidates. We found that DOD does not provide  sufficient guidance and direction to ensure a rigorous, disciplined process  supported by adequate empirical data for preparing and reviewing  multiyear candidates. This increases the risk of poor outcomes and  inappropriate, unstable programs approved for multiyear procurement.
subsections: section_title: Questions about the Appropriateness of Recently Approved Programs
paragraphs: We reviewed DOD’s multiyear justification data submitted in recent  defense budgets and, in particular, examined two newly approved  programs—the Air Force’s F-22A fighter and the joint V-22 tilt rotor  aircraft. The F-22A acquisition has had a turbulent history with a lengthy  development period, major cost increases and quantity decreases, changes  in mission, and disagreements within DOD about the total number  required. The Air Force’s submission of the F-22A for multiyear  procurement generated considerable debate over its merits and whether it  met the legal and business conditions conducive to success. We also  examined the V-22 tilt rotor aircraft multiyear proposal, another  acquisition program with a turbulent history and continuing challenges.
subsections: section_title: Funding and Requirements Issues on F-22A Raptor Program
paragraphs: The latest restructure of the F-22A acquisition occurred in December 2005.  DOD extended production 2 years, added four aircraft and $1 billion in  procurement funds, and proposed to buy the final 60 aircraft under two  separate 3-year multiyear contracts for the airframes and engines.  Multiyear costs and savings estimates were not completed in time for  submission with the fiscal year 2007 defense budget. The Air Force later  submitted the completed MYP justification package with estimated total  multiyear costs of $8.7 billion and projected savings of $235 million, or 2.6  percent, compared to estimated annual contracts. Multiyear proponents  cited the projected total dollar savings as substantial and believed there  was little risk that the remaining 60 aircraft would not be procured. On the  other hand, multiyear critics argued that the low percent of savings  predicted, the short time frame for accruing savings, and the program’s  relatively unstable past made it an inappropriate multiyear candidate.
In prior work, we determined that the restructured F-22A program was  underfunded and questioned whether the proposed multiyear strategy met  statutory criteria. We identified concerns about savings, funding,  requirements, and design stability that we believed needed to be addressed  before the multiyear plan could be justified. For example, the Air Force  did not fully fund the multiyear proposal and asked for incremental  funding. Also, a major development program to add new capabilities and  improve reliability of the F-22A has begun; these efforts could result in  future design modifications which may require retrofit onto aircraft  purchased under the multiyear contract. We also noted that having only a  3-year period of performance at the end of production limited the ability to  achieve savings normally expected under multiyear authority such as EOQ  buys and cost reduction initiatives to improve manufacturing efficiency.
To provide for EOQ buys, Congress subsequently added $210 million to  the F-22A advance procurement budget. In authorizing a multiyear  contract for the F-22A, Congress specified certain conditions to be met  and prohibited the use of incremental funding. The Secretary of Defense  was required to certify that all statutory requirements have been met,  including the determination that the contract will result in substantial  savings of the total anticipated costs of carrying out the program through  annual contracts. DOD submitted its certifications to the Congress in June  2007 and subsequently awarded the F-22A multiyear contract in August  2007. For the fiscal year 2009 budget cycle, the Air Force continues to  propose buying more aircraft than the 183 in the current defense plan.
section_title: Stability and Production Concerns about the V-22 Osprey Program
paragraphs: DOD submitted the V-22 Osprey for multiyear procurement authorization  in the fiscal year 2007 President’s Budget. Officials proposed a 5-year  multiyear contract to acquire 185 aircraft for about $10.1 billion. Multiyear  savings were projected at $435 million, or 4.2 percent, compared to the  estimated costs for annual contracts.
Ongoing changes in quantity, funding, design, and concerns about  production raise questions about the stability of this program and its  appropriateness for multiyear contracting. Subsequent to congressional  authorization for multiyear contracting, DOD reduced its planned  procurements quantity from 185 to 167 due to service funding limitations  with DOD cutting the proposed procurement request for fiscal year 2009  by $234 million.
Development and test efforts continue with a number of design changes  under review to address serious safety, reliability, and performance  problems.  The program office has aggressively prioritized these issues  and is making improvements to the V-22 platform by funding engineering  design changes for the correction of deficiencies. One such deficiency is  leaking hydraulic fluid causing engine compartment fires. Design changes  to fix this deficiency are being studied and implemented.  In comments on  a draft of this report the program office stated it is confident that these  engineering design changes will address the hydraulic leak problems.
To date, DOD has procured 111 aircraft in 11 years. Production aircraft  continue to be conditionally accepted with deviations and waivers.   Engineering investigations to fix these issues are not complete as the  program continues to work to minimize these deviations and waivers.  Even so, the planned production rate for the multiyear period is twice the  current fiscal year 2007 production rate of 17 V-22s. This increase, coupled  with design and production problems, raises concerns over the  contractor’s ability to meet such a demand. DOD reviews and assessments  of the V-22 production ramp up have endorsed the increase with known  risks that require continued management. Officials told us that the  supplier base should be able to meet the elevated production rate, but  expressed concerns about the availability of spare parts and the  challenges in managing manufacturing and installation at three different  and dispersed facilities.
section_title: Insufficient Guidance and Management Direction by DOD to Ensure High Quality Decisions
paragraphs: The statutory criteria in section 2306b of title 10, United States Code,  establish a framework for limiting multiyear contracts to very stable  programs that appropriately balance risks with anticipated savings. DOD’s  process and practices for justifying multiyear candidates, however,  provide little specific guidance on the meaning and application of the  criteria, a process that allows for subjective interpretations about how  well a particular program meets the criteria. Cost and savings estimating  techniques and data also vary considerably and inadequate records are  kept to document decisions and supporting reasons. As a result, costs,  savings, and evidence supporting stability is not consistently prepared,  reviewed, and documented.
subsections: section_title: Concerns about DOD’s Processes and Practices for Justifying Multiyear Candidates
paragraphs: The statutory criteria are general in tone and qualitative by nature to  provide application over a wide range of programs. The regulations DOD  uses to implement the criteria provide contract policies and establish a  process for developing and justifying multiyear candidates. Each  candidate program prepares a budget justification package, normally for  inclusion with the annual defense budget submission that provides funding  requirements and estimated cost savings expected under a multiyear  contract compared to the estimated costs for a series of annual contracts.  The justification package also includes short statements about how  programs are believed to meet each of the six statutory criteria. Weapon  system acquisition program officials prepare the package for subsequent  reviews by military service acquisition commands, service headquarters  offices, and OSD offices responsible for program policy and budget  oversight. Approved candidates are submitted to the Congress for  authorization.
We discussed DOD’s justification process, multiyear contracting policies,  and management practices with OSD and military service officials at each  organizational level. We collected historical and budget data on approved  multiyear programs since fiscal year 1982 and tracked more recent  multiyear candidates through the budget process. We specifically  examined nine major programs representing each military department:  three had completed multiyear contracts, two were beginning multiyear  contracts, and four programs were just recently authorized by the  Congress. Several programs have been approved for more than one  multiyear contract and, collectively, these programs accounted for 17 of  the 59 approvals granted since 1996.
We found that DOD provides little in the way of supplemental guidance to  operationalize the statutory criteria by amplifying terms such as  “reasonable,” “substantial,” and “stable” and quantifying where possible to  provide more objectivity and rigor to the multiyear review process.  Guidance for the most part restates the statutory criteria and establishes  formats for submitting budget justification materials, but does not provide  much elucidation for interpreting and applying the criteria and  establishing internal evidence standards for demonstrating criteria are  met. From our review of justification packages and our discussions with  DOD officials responsible for generating and reviewing multiyear  justification packages, we determined that reviewers interpret and apply  criteria differently and that the methods and data used to compute  contract costs and savings and providing evidence to document program  stability vary in quality and sophistication.
For example, officials we talked to at all levels of the review process had  different ideas and perspectives on what constituted substantial savings  when applying the criterion. An official in the Navy, for example, expected  programs to project at least 10 percent savings, but would consider  candidates under that level. Some Army officials wanted to see 10 percent  savings or hundreds of millions of dollars. An OSD official applied a “rule  of thumb” of 4 to 5 percent. He said that programs under that amount  would typically be more closely evaluated to ensure they were viable, but  that programs over that amount would generally receive a less detailed  check of reasonableness and to ensure paperwork requirements were met.  An Air Force official told us that a 5 percent savings level should be  considered the floor for a genuinely viable candidate for multiyear and a  10 percent savings level achievable, although he cautioned that recent  statutory changes to eligible cancellation ceiling costs will likely have a  negative impact on future multiyear savings.
Further, review of the justification packages for the F-22A and the V-22,  both submitted in the same fiscal year for approval, indicated differences  in how the design stability criterion was applied. Initial operational  capability, an important milestone for stability, had been declared 2 years  prior to requesting multiyear authority on the F-22A, while initial  operational capabilities for both variants of the V-22 are not scheduled to  be achieved until after multiyear approval was granted, at least 2 years  later for the Navy’s variant.
Comparing multiyear savings and judging reasonableness of the estimates  is complicated because the techniques and data used to estimate cost  savings can vary substantially in form, sophistication, and detail. For  example, for its first multiyear contract proposal on the C-17A, the Air  Force simply used the prime contractor’s offer to save 5.5 percent, a figure  that was considered a “management challenge” the contractor believed it  could meet. We found other instances where it was unclear what data was  used in formulating the savings estimates. For example, the Army’s budget  justifications for the Bradley Fighting Vehicle and Abrams Tank estimated  savings at 5 percent and 10 percent, respectively, and provided very  sketchy details to document how the savings estimates were derived and  compliance with the stability criteria.
On the other hand, some analyses can be very involved, provide a range of  estimated savings, and use independent third parties to validate data. For  example, the Navy’s F-18E/F cost estimates and methodologies were  independently verified by OSD’s Cost Analysis Improvement Group. For  the F-22A multiyear proposal, two defense research centers developed  contract costs and savings estimates. Several very different methodologies  were employed, including cost improvement curves based on historical  actual production costs, production cost estimate models for single year  contracts less reductions for expected savings, and summation of savings  initiatives from the prime and subcontractors. These studies produced a  wide range of potential savings, from $235 million to $643 million.
Also, cost and savings estimates in general may be subject to biases and  other factors that impact their fidelity and reliability. Our extensive past  body of work on DOD’s major acquisitions suggests that to gain approval  and continued funding for a weapon system, the acquisition environment  encourages programs to submit overoptimistic estimates about a weapon  system’s readiness for production and to underestimate its costs. Systems  therefore appear more affordable from an investment perspective and can  fit within forecasts of available funds. These circumstances invariably lead  to acquisition programs costing substantially more than originally  estimated. Furthermore, prior reports have discussed the importance of  using present value analysis to account for the time value of money when  evaluating and comparing costs of alternative annual and multiyear  contracts. The timing of government expenditures is expected to differ  with a multiyear contract expected to have relatively more up-front costs  (to fund EOQ for example) and lower costs in the outyears compared to a  series of annual contracts. Although the justification packages are  required to have a present value analysis of the savings estimates,  according to an OSD official and a defense research study, the cost savings  estimates in then-year dollars are the primary estimates used in making  cost decisions. Our review of the justification packages appear to confirm  this because the section of the multiyear exhibit highlighting the benefit to  the government contained only the then-year dollar estimate rather than  the present value estimate.
section_title: Documentation and Record- Keeping Deficiencies
paragraphs: Through our discussions with officials and inspection of records, we  determined that DOD’s review process for the multiyear justification  budget packages does not adequately capture important information and  events to document decisions and help ensure that consistent and reliable  determinations are made regarding multiyear criteria. Once approved,  OSD officials stated few records are kept on multiyear programs regarding  how they determined whether multiyear candidates met the six statutory  criteria. According to OSD and service officials, much of the discussion on  a program proposed for multiyear should have already taken place during  regular executive-level reviews of major weapon systems and been agreed  upon before the multiyear justification package is reviewed for submission  in the budget request. Review of the justification package then essentially  becomes a paperwork formality rarely involving any surprises. Also, we  found programs can be proposed “out of cycle” with the President’s  Budget submission—as in the case of the F-22A, and may not be included  in the budget details that could affect the review path the multiyear  candidate takes to obtain approval. Without maintaining records that  document decisions and the data supporting them, it is difficult to ensure  the quality and comprehensiveness of stakeholder reviews based on the  criteria, fidelity of the data used, and supporting rationales for decisions.
Finally, officials at every level of the multiyear justification process—from  program offices, through higher headquarters, and on to primary OSD  action offices—indicated that they recently were appointed to their  current positions or the person responsible during the multiyear  justification process was no longer in that position. We believe this  contributed to “knowledge gaps,” historical record-keeping deficiencies,  and differences in interpretation and application of multiyear decision  criteria. Turnover implies loss of experience and corporate knowledge; in  this environment, improved and more definitive guidance and retention of  comprehensive historical records is even more important.
section_title: Actual Results on Some Completed Contracts Were Much Different Than Predicted in Budget Justifications
paragraphs: Implementing the statutory criteria requires realistic estimates of  multiyear and annual contract costs. This requirement provides fidelity to  savings projections and allows for accurate estimates of funding  requirements over the life of the multiyear contract. We reviewed cost  performance and results on completed multiyear contracts for the Air  Force’s C-17A Globemaster transport, the Navy’s F/A-18E/F Super Hornet  fighter, and the Army’s Apache Longbow helicopter. Although the precise  impact of multiyear contracting is difficult to determine, our analysis  shows that these programs—presumably approved based on their  demonstrated stability—experienced substantial changes during contract  execution. These changes significantly increased unit costs and drove up  total funding requirements much beyond the estimates submitted to  Congress in the budget justification materials. Each was also impacted by  contract provisions and changes in business conditions.
subsections: section_title: Cost Increases Lead to Questions about Realism of Budget Estimates Used to Justify Multiyear
paragraphs: We found that unit cost growth on these programs ranged from 10 to 30  percent more than projected by the budget justification data. Table 2  shows the growth in unit and total contract costs. We also found that, for  two of the three programs, actual multiyear contract costs exceeded the  original estimates for annual contract costs. The third program, the F/A- 18E/F, came in below annual estimates, but also bought fewer systems  than planned. This reduction in quantity would have also likely decreased  annual costs had that alternative been selected.
Substantial cost increases for completed multiyear contracts on the three  programs meant that Congress had to eventually provide considerably  more funding than originally budgeted. We do not know how cost growth  affected the level of savings achieved, if any, because we do not know how  an alternative series of annual contracts would have fared. In comments  on a draft of this report, DOD officials stressed that cost growth due to  labor and material price escalation under a multiyear contract would likely  have also occurred under an alternative series of annual contracts.  The  final MYP contract values in table 3 also include price increases resulting  from engineering design changes made to the baseline weapon system.  Although these factors may limit the ability to make inferences about the  level of savings achieved, a case could be made that multiyear savings and  costs did not materialize as presented in the multiyear justification  materials.
As discussed earlier, our past body work suggests that defense acquisition  programs are prone to underestimating costs and overstating readiness.  While this tendency would apply to annual as well as multiyear contracts,  it is arguably more problematic for multiyear contracts because of the  government’s increased exposure to risk over multiple years. DOD  officials agreed with us that multiyear contracting should be held to a high  standard because of its special requirements, funding commitments, and  risks.
section_title: Impacts on Cost and Performance from Contract Provisions and Other Factors
paragraphs: We also collected information on multiyear contract provisions for the  three programs with completed multiyear contracts and for the recently  awarded F-22A contract. Each of these programs awarded a fixed-price  contract for the multiyear procurement, but they were not always firm- fixed-price contracts, which typically entail the least risk to the  government. The multiyear contracts contained standard provisions that  provided flexibility to increase or decrease costs based on inflation, labor  rate changes, and/or material cost fluctuations. The multiyear contracts  also included provisions for early cancellation, quantity variations, and/or  design changes. In some cases, the government waived provisions for cost  and pricing data, which according to officials decreased the government’s  insight. Figure 2 below shows the frequency of the various contract  provisions in our case studies.
section_title: Case Studies Provide Details on Meeting Cost and Program Expectations
paragraphs: The three case studies provide insight into multiyear contracting  expectations and realities and the internal and external factors that  affected actual execution. Each case raises some questions about the  accuracy of cost and savings estimates used to justify multiyear  procurement and the degree to which requirements, design, and funding  were stable.
subsections: section_title: C-17A Globemaster: Risky Strategy Affected Multiyear Funding Stability
paragraphs: The estimated savings for the first C-17A Globemaster multiyear contract  was simply a percentage amount submitted by the contractor. For the  second multiyear contract, Air Force officials assumed the same percent  of savings and added additional savings based on the use of a  controversial funding strategy. This strategy relied on incremental funding,  advanced buys of parts, and large potential cancellation liability to  maintain a production schedule of 15 aircraft per year even though all of  these aircraft had not been fully funded. The unfunded liability to the Air  Force had the contract been canceled eventually grew to $1.5 billion.  Concerned that this incremental funding strategy violated DOD’s full- funding policy and could potentially violate the Anti-Deficiency Act,  Congress increased C-17A procurement funding a total of $745 million in  fiscal years 2003 and 2005 to fully fund all aircraft. In annual DOD  appropriations acts, Congress also has prohibited incremental funding of  multiyear contracts.
In addition, the costs for both C-17A multiyear contracts were affected by  economic price adjustments. On the first multiyear, overhead costs were  significantly increased as a result of a merger between two major defense  contractors. The Air Force subsequently paid $150 million to cover cost  increases resulting from the merger and another $50 million to remove the  clause from the contract. By the end of the multiyear contract, unit costs  had increased 15 percent. The multiyear justification materials submitted  to Congress supported a plan to buy 80 aircraft at an average cost of $179  million; instead the Air Force eventually paid about $207 million per  aircraft. During the second multiyear contract, the contractor made large  contributions to its pension fund, which triggered the price adjustment  clause and resulted in a potential cost increase of over $530 million. The  Air Force is in the process of restructuring the contract to reduce this  amount.
The Air Force also awarded two multiyear contracts for the C-17A engine,  the F117. These procurements appear to have been successful with  demonstrated stability during the multiyear period and price breaks based  on the multiyear contract. The F117 engine is a commercially available  engine with a stable design and manufacturing process. There were no  engineering or design changes; no advanced procurement or EOQ  requirements; and no cancellation ceilings associated with either contract.  The only potential cost risk was the economic price adjustment clause, but  officials stated that the actual adjustments were not exclusively in the  contractor’s favor. According to the program office, the first multiyear  contract resulted in a savings of 10 percent, more than it had originally  expected. Savings from the second multiyear contract are consistent with  the original estimate.
section_title: F/A-18E/F Super Hornet: Design and Requirement Changes Impact Costs
paragraphs: For its first F/A-18E/F multiyear contract, the Navy did not award a firm- fixed-price contract because the program was early in the production  phase and there were still ongoing design development efforts on the  airframe. During the contract period, the economic price adjustment  clause resulted in the Navy paying an additional $378 million because of  labor rate and material cost increases. The first multiyear contract also  included a variation-in-quantity clause that permitted an upward or  downward adjustment of six aircraft per year. Annual quantities and the  specific mix of buys between the two models changed more than once  during the multiyear period. By the end of the multiyear contract, the  number of aircraft procured had dropped from 222 to 210 aircraft and the  average unit costs had increased by 10 percent, compared to the budget  estimates.
In a May 2000 report, we had questioned whether the Navy was ready to  enter into its first multiyear contract for full-rate production. Deficiencies  identified during operational testing had not been corrected, and to avoid  costly retrofitting and redesign, we believed that corrections should be  made and tested before entering into the contract.
The Navy proposed buying another 210 Super Hornets on a second  multiyear contract, but later changed the requirement to procure 154  Super Hornets and 56 of the new E/A-18G Growler, an electronic attack  variant still in product development. The Navy’s total requirement for the  Super Hornet had been reduced, and the new Growler was needed to  replace aging EA-6B aircraft in the electronic attack mission. The follow- on multiyear also included a variation-in-quantity clause, but this time it  only covered upward adjustments. Multiyear costs and funding were  further impacted by the economic price adjustment clauses. As in the        C-17A’s case, the F/A-18E/F multiyear contract was affected by the  contractor’s large pension fund contribution. The Navy estimated that it  could have been obligated to pay over $1 billion, which is nearly the same  as the amount of cost savings originally estimated to justify the multiyear  contract award. However, the Navy renegotiated the terms of the clause  and restructured the contract to bring the price adjustment down to about  $152 million.
section_title: Longbow Apache: Modifications and Unplanned Work Increased Unit Costs
paragraphs: The Army’s Longbow Apache helicopter experienced significant cost  increases during both its multiyear procurements. Army officials stated  that increases were largely because of aircraft modifications and  unplanned work. These modifications included a voice data recorder and  an improved rotor blade assembly that would enhance operational safety.  Contract costs were also increased by additional unplanned work.  Program officials explained that it is very difficult to predict the condition  of fielded aircraft when they return to be upgraded or remanufactured.  Along with normal wear and tear, many operational aircraft were returned  with extensive corrosion and battle damage; others had been cannibalized  for parts. Remanufacturing these aircraft required significantly more effort  and funding than originally planned. By the end of the first multiyear  contract, the Apache’s average unit cost had increased by 30 percent; at  the end of the second multiyear, these costs had increased by 25 percent.
While it may be difficult to predict unusual wear and tear on a system and  it is common to incorporate new modifications over time, it is especially  problematic to roll these costs into the multiyear contract that had been  assumed stable and that had been justified based on initial cost estimates  without these new add-ons. OSD cost analysts are studying this issue to  determine the proper accounting for modifications under multiyear  contracting.
section_title: Estimated Savings for Candidate Programs Are Trending Downward
paragraphs: The amount and percentage of savings expected from a multiyear contract  compared to a series of estimated annual contracts is the most visible and  perhaps the most critical criteria in the eyes of many stakeholders. The  savings requirement in definition and application has evolved over the  years. A threshold level of 10 percent savings emphasized during the 1980s  was eliminated and replaced with a nonquantifiable requirement for  “substantial savings” since fiscal year 1991, allowing wide flexibility in its  interpretation. Although a direct causal link is not demonstrated, our  analysis of multiyear programs approved by Congress shows that  estimated savings were on average higher before the substantial savings  requirement than after. Other factors—lower quantities of modern systems  being procured, stricter termination liability allowances, and contraction  in the defense industrial base—may also contribute to decreased  estimated savings for current and future systems by lessening the benefits  from large quantity buys and efficient production rates, key attributes of  multiyear contracts.
subsections: section_title: Multiyear Savings Percentages Were Generally Lower after Changes in the Statutory Savings Criteria
paragraphs: When Congress codified the authority for multiyear procurement  contracting in December 1981, there was no specific savings criterion in  the law that candidate programs had to meet. However, the impetus  behind multiyear was provided by DOD studies at the time predicting  savings averaging 10 to 20 percent. The 10 percent figure became a savings  benchmark for decision-makers in the early 1980s when judging the merits  of candidate programs. In the late 1980s, this benchmark became a  threshold requirement for many candidate programs as Congress began  stipulating a 10 to 12 percent savings amount in annual defense  authorization acts for selected programs.
In November 1989, Congress decided to codify the 10 percent savings  requirement and other conditions and limitations previously imposed on  an annual basis. However, a year later, that threshold was struck from the  U.S. Code, and the requirement for “substantial savings” was substituted  after DOD had argued that a rigid threshold limited the potential for  savings on stable, low risk programs projecting lesser savings amounts.  This substantial savings requirement has remained unchanged since  November 1990, with specific savings requirements stipulated in annual  legislation for two candidate programs approved by Congress during this  time. Figure 3 illustrates the evolution of the savings requirement.
Our analysis of estimated savings for approved multiyear programs  determined that the average savings level trended lower after the  substantial savings criterion was adopted. Although programs have been  approved during both eras over a wide range of savings below and above  10 percent, the change in the law provided decision makers the flexibility  to propose and approve candidate programs since fiscal year 1991 with  lower savings estimates on average compared to the 1980s. As previously  discussed, an unofficial rule of thumb for savings normally expected is  now down to 4 to 5 percent.
Figure 4 illustrates the range of savings and general trend as the savings  requirement evolved. It suggests that a shift did occur since the change to  substantial savings in 1991. A larger proportion of multiyear programs with  estimated savings of less than 10 percent were approved after 1991. On the  other hand, candidate programs approved prior to 1991 show a larger  proportion of savings of 10 percent or more. This finding is supported by a  defense consultant study that calculated estimated savings averaged 13  percent for candidate programs from 1982 to 1989.
section_title: Other Factors Affecting Estimated Savings
paragraphs: In addition to the revised savings requirement, other factors may also be  impacting the level of savings for current and future multiyear programs:    Smaller quantities procured. The higher cost for modern weapon  systems and changes in required force structure has resulted in  generally smaller procurement quantities for new systems compared to  predecessor systems. Smaller quantities of systems bought under  multiyear contracts may not provide the same opportunity to achieve  savings through such mechanisms as EOQ buys of parts and materials.  Past multiyears, on the F-16 Falcon aircraft and Black Hawk  helicopters, for example, procured hundreds of systems. In contrast,  the more recent multiyear procurement of the F/A-18E/F Super Hornet  procured less than half the F-16 quantity. Analysis of the data on MYP  candidate programs that we collected and summarized in table 3 below  showed that the median multiyear procurement quantity for aircraft  MYP candidate programs declined over 40 percent from the 1980s to  the present era with a concurrent decline in average savings. With  fewer aircraft procured during the multiyear period, savings from  economic buys and optimized production is typically smaller.
Cancellation liability changes. If a multiyear contract is cancelled, the  government is liable for reimbursing the contractor for its incurred  costs up to a negotiated cancellation ceiling typically in the contract.  Until recently, DOD has been able to include both recurring and  nonrecurring costs. Multiyear contracts awarded during the past 25  years have included a cancellation liability, but the cancellation ceiling  for the C-17A multiyear contract awarded in 2002 was considered very  large with a potential liability of more than $1.5 billion. Some members  of Congress were concerned by its size and concluded that this large  liability inappropriately committed the government to a production  schedule for which funding had not been appropriated. As a result, for  the past several years Congress has limited the cancellation liability for  multiyear contracts to nonrecurring costs only. Some DOD officials  expect this change to limit savings on current and future multiyear  contracts as contractors choose not to bear financial risks previously  borne by the government. A major source of multiyear savings is the  EOQ buys, and without including these kinds of recurring costs in the  cancellation ceiling, fewer contractors may buy supplies and materials  up front in bulk to limit their risks should the multiyear contract be  cancelled early.
Declining competition. Some DOD and defense research center  officials believe that the consolidation and contraction of the defense  industry and resulting decline in competition and contraction among  vendors and suppliers, make it harder to wring savings from EOQ buys.  For example, an OSD official stated that because the F-22A was  originally designed well over a decade ago, it is now experiencing  diminishing manufacturing sources on many components as well as  parts and equipment obsolescence. Similarly, a defense research center  official believes that diminishing manufacturing sources negatively  impacted the multiyear savings potential for the F-22A.
section_title: Multiyear Contracting Results Are Uncertain
paragraphs: DOD does not perform post contract assessments of completed multiyear  contracts to validate actual results and determine cost effectiveness of its  investments. Some prior studies, including GAO work, provide some  limited, but inconclusive insights into multiyear results and benefits. OSD  and the military services do not maintain adequate, comprehensive  records on historical and current multiyear contracts that could facilitate  better tracking and provide perspective for future multiyear efforts.
subsections: section_title: DOD Does Not Have a Formal Process for Assessing and Tracking Multiyear Contract Results
paragraphs: DOD does not have a formal process for assessing the cost and  performance of completed multiyear contracts. According to DOD  officials, once a program is approved for multiyear, they do not track  subsequent performance nor validate actual results against expectations  established in the budget justification submissions. Also, they do not have  an official method to capture and share lessons learned with other  programs considering multiyear contracts that could improve prospects  for successful outcomes.
However, we did find that individual program offices may make efforts to  ascertain benefits and learn lessons that can be applied to future multiyear  submissions, but it appears that these efforts are isolated. Based on prior  multiyear experiences, F/A-18E/F Super Hornet program officials decided  to require cost and pricing data to better inform cost estimates for their  next multiyear application. Also, they based expected savings on cost  reduction initiatives rather than EOQ buys because they believed these  initiatives had a better return on investment. Similarly, the results of our  case studies discussed earlier demonstrate that assessing actual results  can glean valuable information about contract costs and performance that  can be used to improve future multiyear outcomes.
section_title: Prior Studies of Multiyear Cost Effectiveness Show Limited and Inconclusive Results
paragraphs: Some attempts to assess historical multiyear performance have been  made, but validating actual savings is elusive. According to DOD and  defense research center officials as well as the studies they conducted,  calculating the actual cost savings from the use of a multiyear contract and  comparing results to original expectations is very problematic for several  reasons: (1) multiyear cost and other program data is unavailable; (2) lack  of comparable data on costs of annual contracts; and (3) original  assumptions change from the justification package, such as design  modifications and variations in buy quantities, labor, and material rates.
Recent studies by two defense research centers attempted to gain a  historical perspective on actual multiyear savings achieved for past  contracts in order to provide context as to the relative level of expected  savings for the F-22A multiyear proposal. In reports, both centers noted  that the government does not collect or save data needed to do a detailed  analysis and that, once programs are approved and implemented,  important assumptions on which original savings estimates were based  often changed.
In particular, since a series of annual contracts are not executed for the  same procurement once a multiyear strategy has been adopted,  comparisons of actual multiyear costs can only be compared to  hypothetical estimates of annual contracts. Further, changes that occur in  a multiyear contract environment can also occur in an annual contract  environment, and the exact effect on the actual costs for the annual  contracts is unknown. In reviewing the literature, one center noted that it  found “very few examples of serious and methodologically credible  attempts to validate claimed savings and savings percentages after the fact  once programs had been completed,” and none that produced, in the  center’s opinion, definitive findings.
In the case of the F/A-18E/F program, researchers from one center felt  they had more data available on this multiyear contract than for others,  and they attempted to validate actual savings. They concluded that while  the available data supported savings in the neighborhood of original  estimates, a definitive answer would require a more detailed analysis of  EOQ data and cost reduction initiatives. Similarly, in our 1988 report on  the first F-16 multiyear procurement, we concluded that savings were  likely achieved from EOQ buys in the order of magnitude expected, but we  were unable to make definitive conclusions about total savings achieved.
section_title: Inadequate Records Maintained on Current and Historical Multiyear Programs
paragraphs: Undergirding attempts to track and assess multiyear performance is the  quality and sufficiency of data. As discussed earlier in this report, we  found that DOD at all organizational levels does not keep adequate records  on multiyear programs to document stakeholder reviews and the empirical  evidence used to justify multiyear contracts. OSD Comptroller officials  told us that, while they are the final authority on approving multiyear  candidates and the ultimate owner of the review process, they do not track  multiyear packages through the approval process and after the final  decision has been made to submit it to the Congress. DOD has no  comprehensive, central information system that records the status of  multiyear candidates in-process, candidates that have been approved or  disapproved, detailed information on how multiyear criteria were applied,  or information on specific criteria contained in the final justification  package.
DOD is not required to, nor does it maintain a central database of  historical or ongoing multiyear contracts. In response to our inquiries, the  OSD office ultimately responsible for the multiyear procurement review  process was unable to provide us the justification packages for over half  of the programs approved by Congress since 1992, including very recent  submissions. To obtain more complete and accurate data that could be  used to track performance and conduct trend analyses, information must  be compiled from many different sources, including budgets, program  office files, contractor studies, and contracting databases. For example,  our efforts to identify and track multiyear contract information contained  in two major federal databases were only partially successful. Some  valuable multiyear contract information was readily available through  these sources. However, the type of data stored and storage format—as  well as issues pertaining to reliability, consistency, and comparability— limited their current usefulness in tracking and evaluating multiyear  contracts. However, with some improvements particularly with reliability,  these databases could support future studies.
section_title: Conclusions
paragraphs: The statutory criteria for approving the multiyear procurement of major  DOD weapon systems clearly establish requirements to limit multiyear  authorization to stable, low risk programs with realistic cost estimates and  reasonable expectation for savings. To move forward otherwise is to  accept significant risks with little chance of reward. However, DOD does  not have an adequate process with controls in place to ensure multiyear  candidates meet all the criteria and are supported by sufficient empirical  evidence. Inconsistent application of criteria, questionable cost and  savings estimates, and inadequate documentation increase potential for  approving inappropriate, unstable multiyear programs and incurring  costly, poor outcomes when plans go awry and conditions change.  Improving guidance, ensuring decisions are informed by knowledge, and  maintaining better records are critical needs, as well as important tools for  retaining corporate memory given frequent turnover of personnel at all  levels of the justification review process. It is not possible to calculate  accurately the cost to taxpayers that has resulted from these conditions,  but the lack of a disciplined and rigorous process that demands knowledge  about stability and costs provides potential for significant waste of  taxpayer dollars.
Furthermore, DOD does not track and evaluate actual performance on  multiyear contracts for major DOD weapon systems. Once a contract is  awarded for a multiyear program, little effort is made to collect data and  assess actual results to compare performance against original  expectations and to validate savings and other benefits achieved.  Assessing results could provide valuable insights and lessons learned on  prior experience and identify opportunities to improve future multiyear  procurements. Not having a clear picture of actual performance further  emphasizes the criticality of getting it right up front by ensuring only  appropriate programs go to Congress for approval.
Therefore, despite a long history and substantial funding for major DOD  weapons system multiyear contracts, DOD does not know whether it has  gotten a reasonable return on its investments for the extra risks incurred.  Some concerns noted in this report, such as the practice of understating  costs and overselling benefits, apply also to annual contracts, but the  standards should be higher for multiyear contracts because of the larger  up-front investments required and the government’s exposure to risk  should the program fail or be substantially changed. Strengthening both  the front end of the process—identifying and justifying good candidates— and the back end—assessing results and gleaning lessons learned from  completed contracts—can help ensure costs and risks are adequately  balanced for new multiyear programs and improve future outcomes.  Underpinning both ends, it is important to capture and make available  essential data on multiyear decisions and subsequent performance that  can be readily accessed by stakeholders and prospective users of  multiyear procurement authority.
section_title: Recommendations
paragraphs: To improve the outcomes of the multiyear justification reviews of major  DOD weapon systems, the Secretary of Defense should direct appropriate  offices within the Under Secretary of Defense (USD) (Comptroller) and  USD (Acquisition, Technology & Logistics) to: (1) improve and expand  guidance provided to military services to better define multiyear decision  criteria for major DOD weapon systems and to facilitate more consistent,  objective, and knowledge-based evaluations of these multiyear candidates  within DOD; (2) establish a process for third party validation of the costs  and savings data submitted for such candidate programs; and (3)  implement a central database for maintaining historical records and for  effectively monitoring and tracking major DOD weapon system multiyear  procurements, to include documenting the specific decisions made by  stakeholders and their rationales for decisions.
To provide lessons learned for informing and improving future major DOD  weapon system multiyear candidate programs and to ensure DOD is  earning a sufficient return on its investments in multiyear contracts for  major DOD weapon systems, the Secretary of Defense should direct that  the responsible military service, in conjunction with appropriate elements  within OSD, conduct after-action assessments at the conclusion of all  multiyear contracts used to procure major DOD weapon systems to  determine their effectiveness in achieving predicted benefits while  managing associated risks. These assessments should identify major  deviations, if any, between the unit costs predicted in the multiyear  justification package and the unit costs actually incurred. The assessments  should also substantiate—to the extent practicable—savings achieved and  identify reasons and causes contributing to overall performance results  and attemp to isolate those issues peculiar to the multiyear contract from  those that would likely have also affected annual contracts if a multiyear  strategy had not been employed. Internally, DOD should use the results of  these assessments to provide lessons learned to both industry and the  government that can help inform and lead to better and more supportable  decisions on future multiyear candidate programs.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD provided us with written comments on a draft of this report.  The  comments appear in appendix II.  DOD also separately provided technical  comments which we reviewed and incorporated as appropriate.  In written  comments, DOD concurred with our two recommendations to improve  guidance and to implement a central database for maintaining records and  tracking multiyear programs. DOD partially concurred with the other two  recommendations.
DOD partially concurred with our recommendation to establish a process  for third party validation of the costs and savings data submitted for  candidate programs. In its comments, DOD stated that independent third  party validations of cost and savings are done on selected programs and,  in developing new guidance in response to our first recommendation,  would consider whether the benefits of requiring validation on all  programs warrant the delays and costs of validation. Our review of five  new proposals and six approved multiyear contracts found only one such  instance, and our discussions with service and OSD officials show that  third party validations are rare.  We believe that independent third party  validations would result in more accurate and comprehensive cost and  savings information critical to congressional and DOD decision making on  multiyear candidates.  Our review identified inconsistent practices in  preparing and reviewing multiyear proposals and varying degrees of  quality and completeness in the initial cost and savings estimates made by  the weapon system program offices.  An independent third party check  would help ensure that appropriate multiyear candidates are submitted to  the Congress for approval.
DOD also partially concurred with our recommendation that after-action  assessments be conducted to provide lessons learned for informing and  improving future multiyear candidate programs.  In its comments, DOD  agreed that after-action reports may be of value for certain multiyear  programs, but questioned the value for all programs because (1) the  extensive time before the assessment results are known and can be  applied; and (2) the difficulty in determining actual savings.  The intent of  the recommendation is to learn lessons that can be applied to strengthen  future multiyear proposals and improve their prospects for success.  This  in formulation is not time-bounded as DOD has contracted for studies that  drew useful lessons from programs many years earlier.  Collecting and  distributing data on lessons learned would provide a continuing database  of knowledge for future programs.  Furthermore, while we recognize  difficulties and constraints in calculating actual savings, this does not  preclude the department from making good faith efforts that can provide  valuable, albeit imperfect information. Also, after-action assessments  include more than savings calculations.  As pointed out in the report, one  program assessed itself and identified important contractual features and  other factors that it used to improve subsequent multiyear proposals. We  note that the practice of doing after-action reports is widespread in the  department and used for many different kinds of activities, including  military contingency operations and logistics functions, and that these  efforts provide planners and decision makers with critical lessons learned  for applying to and improving future actions.
We are sending copies of this report to the Secretary of Defense; and the  Secretaries of the Air Force, Army, and Navy; and the Director of the  Office of Management and Budget. We will provide copies to others on  request. This report will also be available at no charge on GAO’s Web site  at http://www.gao.gov. Key contributors to this report were Bruce  Fairbairn, Assistant Director; Noah Bleicher; Matthew Drerup; Mary Jo  Lewnard; Rae Ann Sapp; and Bob Swierczek.
If you have any questions about this report or need additional information,  please call me at (202) 512-4841 (sullivanm@gao.gov). Contact points for  the offices of Congressional Relations and Public Affairs are located on  the last page of this report.
section_title: Appendix I: Scope and Methodology
paragraphs: Our review was limited to major DOD weapon systems that have received  congressional approval in annual defense authorization and/or  appropriations acts to award a multiyear contract and meet the statutory  requirements identified in 10 U.S.C. § 2306b. Work was performed at the  Offices of the Secretary of Defense and the three military service  headquarters (Navy, Army, and Air Force), in Washington, D.C.; Naval Air  Systems Command, in Patuxent River, Maryland; Aviation Missile  Command, in Huntsville, Alabama; Aeronautical Systems Center, at  Wright-Patterson Air Force Base, Ohio; Institute of Defense Analysis  (IDA), Alexandria, Virginia; and RAND National Defense Research  Institute, Arlington, Virginia.
To evaluate DOD’s multiyear review process, we compiled a list of the all  the candidate programs approved by Congress to use a multiyear contract  by examining DOD authorization and appropriations acts going back to  fiscal year 1982 when the statutory language in 10 U.S.C § 2306b was first  enacted. We then reviewed DOD multiyear justification packages  submitted to Congress in recent defense budgets, identified statutory  criteria authorizing the use of multiyear procurement, and considered  regulatory policies and procedures used within the Services and at OSD to  prepare and evaluate multiyear justification packages. We discussed with  officials at acquisition program offices and at higher command review  levels how they interpreted and applied the statutory criteria and guidance  to evaluate the appropriateness and feasibility of multiyear candidates. We  reviewed DOD and congressional actions on recent multiyear candidates  and examined specifically two major programs recently approved for  multiyear contracting—the F-22 Raptor and V-22 Osprey—to illustrate how  this process works and address some questions raised about the  appropriateness of these candidates and data used in the justification  packages. For these systems, we extensively drew upon GAO’s work in  prior and ongoing engagements.
We conducted limited case studies for selected multiyear aircraft  contracts to assess outcomes and the internal and external events  affecting performance. Because our sample of DOD aircraft multiyear  contracts was not randomly selected, we cannot project our findings to  other programs. These case studies included three major DOD weapon  systems—C-17A Globemaster, F/A-18E/F Super Hornet, and the Apache  Longbow Helicopter. These aircraft programs have fully executed at least  one multiyear contract in the recent past and also have ongoing follow-on  multiyear contracts. We reviewed the multiyear proposal packages  submitted to Congress, annual budget information, Selected Acquisition  Reports, contract file documentation, and information in DOD contract  databases for multiyear contracts awarded by these programs. We  calculated unit cost changes and identified key programmatic and  environmental changes impacting the execution of the multiyear programs  and compared these to the original projections in their multiyear  justification packages. To identify the characteristics of multiyear  contracts and how they affect the costs, risks, and savings for selected  multiyear aircraft contracts, we reviewed contract file documentation and  information on the DOD DD 350 Individual Contract Action Report (ICAR)  database and the Federal Procurement Data System-Next Generation  (FPDS-NG). As part of the case study approach, we reviewed the types of  contracts, contract clauses, and other contract modifications to determine  how they affected the unit costs under a multiyear contract. Further, we  discussed the multiyear contracts included in our limited case study  approach and the F-22A program’s August 2007 multiyear contract award  with program officials to help us assess the effects that specific contract  provisions have on unit costs during contract execution.
To research the legislative history underlying the adoption and subsequent  repeal of the 10-percent savings requirement and the current requirement  that defense multiyear procurement contracts achieve “substantial  savings,” we reviewed the evolution of multiyear savings criteria in early  defense initiatives, the fiscal year 1982 codification, subsequent  amendments, and the savings criteria contained in annual authorization  and appropriations acts subsequent to the granting of multiyear authority.  To make comparisons in average and median levels of estimated savings  on multiyear candidate programs as the criteria for awarding a multiyear  contract changed, we reviewed savings estimates in the budget  justification packages for multiyear candidates submitted to Congress  since 1982 and past GAO reviews of multiyear candidates conducted in the  1980s. We were able to obtain savings estimates for approximately 94 of  the 141 approvals granted by Congress to award a multiyear contract.
To determine the extent to which DOD tracks performance and validates  savings and other benefits actually achieved by multiyear contracts, we  evaluated the kind and extent of cost data and program information  maintained at the services and OSD, and how they use these data to  determine whether predicted savings and other benefits were actually  achieved by multiyear contracting. We also reviewed DOD policies and  guidance for estimating and validating multiyear savings, and discussed  with DOD officials practices used to estimate and monitor multiyear  savings. We discussed with DOD officials record-keeping and management  oversight requirements and reasons why they do not have a formal process  for assessing multiyear results. We reviewed two recent major studies  done by defense research firms that summarized estimated savings on  historical programs and performed case studies on selected programs to  identify key events and practices that affected ultimate performance. We  discussed with IDA and RAND officials their cost estimating techniques  and assumptions used in their F-22A multiyear studies that supported the  planned multiyear contract. We reviewed and summarized their attempts  to validate savings and other benefits from prior multiyear programs. We  also reviewed prior GAO work on selected weapon systems and the results  of our work during the 1980s when Congress regularly asked us to review  DOD’s multiyear candidates.
We conducted this performance audit from June 2007 to February 2008 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contacts and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments