Defense Acquisitions: Information for Congress on Performance of Major Programs Can Be More Complete, Timely, and Accessible
Published: 2005-03-28T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: DOD has more than $1 trillion worth of major defense acquisition programs, on which it must report to Congress, including a comparison of a current program's costs to a baseline containing its cost, quantity, schedule, and performance goals. When these goals are changed, the program is "rebaselined" to reflect current status. However, measuring current estimates against the most recent baseline without additional perspectives may obscure for Congress how programs are performing over time. Concerned over this, you asked GAO to examine how DOD's use of rebaselining has affected the adequacy of data provided to Congress on major defense acquisition programs.
section_title: What GAO Found
paragraphs: DOD could be reporting more complete information beyond what is required by law on the unit cost performance of major defense acquisition programs to Congress for its authorization and appropriations deliberations. DOD does present Congress with valuable information about a program's performance by comparing the latest unit cost estimate against the most recent approved baseline. However, this provides only one perspective on performance because rebaselining shortens the period of performance reported and resets the measurement of cost growth to zero. Other meaningful perspectives are not reported. First, DOD does not report the cumulative unit cost growth, in constant dollars, that a program has experienced since the first full baseline was established. For example, DOD reported in the 2003 Selected Acquisition Report (SAR), the most recent available, that the F/A-22 Raptor program's unit cost decreased by 0.33 percent in the previous 4 months--since the latest rebaselining. DOD did not report that the program's unit cost had cumulatively increased by 72 percent in the last 143 months. Second, the change in unit cost between one budget request to Congress and the next is not measured or reported. For example, DOD reported in the 2003 SAR that unit cost for the Stryker program increased by 1.34 percent in the 2 months since the latest rebaselining; it did not report that unit cost had grown by 21 percent in the previous 12 months. DOD could be more timely in reporting to Congress that it has rebaselined individual programs. A key factor is that DOD is not required by statute or its own policies to report a program's rebaseline to Congress. Although DOD includes the latest rebaselining actions in the April SARs, a rebaseline approved after early April may not be reported to Congress before it enacts the authorization and appropriations legislation. For example, the DD(X) Destroyer program established a new baseline on April 23, 2002, but did not report this new baseline to Congress in a SAR until April 2003. As a result, between April 2002 and the passage of the fiscal year 2003 defense budget, the SAR provided Congress did not reflect the approved baseline for the DD(X) program. Congressional oversight of DOD's adherence to established cost and schedule baselines is unnecessarily constrained because DOD classifies about 50 percent of the SARs it submits to Congress, despite the fact that only a small amount of data in each of these SARs is actually classified. This reporting practice restricts access to the unclassified cost, quantity, and schedule data for congressional staff without security clearances and requires special handling procedures of that unclassified data by those with clearances.

section_title: Letter
paragraphs: The Department of Defense (DOD) has more than $1 trillion worth of  major defense acquisition programs in development and production. For  the purposes of congressional oversight and decision making, DOD is  required by law to report to Congress the unit costs (costs divided by  quantities) for each program.  DOD reports this unit cost for both the  current budget estimate and the most recent acquisition program baseline,  which describes the cost, quantity, schedule, and performance goals of a  program. DOD reports these comparisons in a Selected Acquisition  Report (SAR).  These reports, typically provided in early April, aid the  Congress during its deliberations over the defense authorization and  appropriations legislation.
As an acquisition program proceeds, DOD may determine that the original  baselined goals approved by decision makers are unachievable. Under  such circumstances, DOD will “rebaseline” the program so the goals more  realistically reflect the program’s current status. However, to the extent  that current estimates are only measured against the most recent baseline,  Congress may not be getting in each SAR a clear picture on just how well  programs are performing over time.
Concerns over these issues led you to request that we evaluate how DOD’s  use of rebaselining has affected the adequacy of data provided to Congress  on the performance of major defense acquisition programs. To address  this objective, we assessed SAR cost, schedule, and quantity data for  individual acquisition programs using federal control criteria such as  accuracy, completeness, timeliness, and accessibility.  This report focuses  on (1) the completeness of unit cost reporting and (2) the timeliness of  reporting on rebaselining and the accessibility of unclassified data.
In conducting our review, we reviewed pertinent statutes and DOD  acquisition system guidance; analyzed more than 650 SARs, both quarterly  and annual, of all major defense acquisition programs for the period 1996  to 2003; and interviewed cognizant DOD officials. We conducted our work  from February 2004 to January 2005 in accordance with generally  accepted government auditing standards.
subsections: section_title: Background
paragraphs: In 1967, to obtain consistent, reliable data on major defense acquisition  programs, DOD instituted a reporting system to summarize program cost,  schedule, and performance information called the SAR. The purpose was  to focus management attention on a program’s performance and changes  to its acquisition plan. In 1975, Congress established the SAR as a statutory  reporting requirement to Congress.
Consistent with statute, DOD requires an acquisition program baseline at  program initiation that establishes objective and threshold values for cost,  quantity, schedule, and performance parameters. Objective values  represent what the user desires and expects. Thresholds represent the  acceptable limits to those values that, in the user’s judgment, still provide  the needed capability. The acquisition program baseline is derived from  the user’s performance and schedule needs and the best estimates of total  program cost consistent with projected funding.
A baseline, according to DOD acquisition guidance,  consists of the  following:    Performance parameters—-The total number of performance  parameters should be the minimum number needed to characterize the  major drivers of operational performance. The number and specificity  of performance parameters may change over time.
Schedule parameters—-Schedule parameters should include, at a  minimum, the projected dates for program initiation, other major  decision points, and initial operating capability. The program manager  may propose, and the milestone decision authority may approve, other  critical system events.
Cost parameters—-Cost parameters should reflect realistic cost  estimates of the total program and/or increment, and should be  identified as life-cycle cost. The elements of program life-cycle cost  include   research, development, test, and evaluations;    procurement costs;    military construction costs;    acquisition-related operations and maintenance costs, if any;  total system quantity (to include both fully configured development  and production units);    average procurement unit cost (defined as total procurement cost  divided by total procurement quantity);    program acquisition unit cost (defined as the total of all acquisition- related appropriations divided by the total quantity of fully configured  end items); and    any other cost objectives established by the milestone decision  authority.
The defense acquisition management framework defines the stages  through which typical programs proceed. As each stage concludes, a  decision must be made to initiate, continue, advance, adjust, or terminate  a project or program work effort or phase. The review associated with  each of these decision points typically addresses program progress and  risk, affordability, program trade-offs, acquisition strategy updates, and the  development of exit criteria for the next phase or effort. The milestone  decision authority is responsible for approving the program structure as  part of the acquisition strategy. Milestone decision points are Milestone A,  for entry into the technology development phase; Milestone B, for entry  into the system development and demonstration phase (acquisition  program initiation); and Milestone C, for entry into the production and  deployment phase.
There are three fixed acquisition program baselines (APB) tied to these  milestone decisions. The concept baseline approved at Milestone A; the  development baseline approved at Milestone B; and the production  baseline approved at Milestone C. This first full estimate, generally  established at Milestone B, is sometimes referred to as the original  business case for the program. All estimates include a mix of both sunk  and projected remaining costs.
Programs are rebaselined based on a change in requirements, a change in  acquisition phase, or a program restructuring. They are also rebaselined  when there is a realized cost overrun or schedule slip beyond certain  thresholds. Rebaselining can occur at any time and cover any phase of the  defense acquisition program. All rebaselines must be approved by the  milestone decision authority. A program may therefore have several  approved program baselines during development, but only one currently  approved program baseline. Programs also monitor a current estimate of  the elements of a baseline and report differences as they develop.
Recognizing the need to establish a monitoring mechanism to provide  oversight of cost growth in DOD major defense acquisition programs,  Congress has required DOD to (1) notify Congress whenever unit cost  growth is at least 15 percent, and (2) “certify” the program to Congress  when the unit cost growth is at least 25 percent above the latest approved  acquisition program baseline cost estimate. However, Congress did not  require DOD to report when a program has been rebaselined.
The reporting requirement is commonly referred to as Nunn-McCurdy,  after the congressional leaders responsible for the requirement.  Exceeding either the 15 or 25 percent unit cost threshold is referred to as a  Nunn-McCurdy unit cost breach. The unit cost reporting includes two  elements—program acquisition unit cost (PAUC) and procurement unit  cost (PUC). DOD refers to the PUC as the average procurement unit cost  (APUC). Unit cost is considered a key measure because it describes  buying power—the average cost to buy each unit.
Unit cost information is submitted to Congress in DOD’s annual  comprehensive SARs. In addition to the comprehensive annual report to  Congress for the period ending December 31, DOD also prepares a  quarterly report for the second, third, and fourth quarters of the fiscal year  when there has been    an increase of 15 percent or greater in the current estimate of the  program acquisition unit cost or average procurement unit cost in base- year dollars;    a 6-month or greater delay in the current estimate of any schedule  milestone since the current estimate of the previous report; or    a milestone B or C and associated acquisition program baseline  approval within 90 days prior to the quarterly report as of that date.
section_title: Information for Congress on Unit Cost Performance Could Be More Complete
paragraphs: DOD could be reporting more complete information on unit cost  performance to Congress for its authorization and appropriations  deliberations.  A new baseline serves an important management control  purpose when program goals are no longer achievable, because it presents  an important perspective on the program’s current status and acquisition  strategy. However, by comparing the latest unit cost estimate with the  most recent approved baseline, DOD provides an incomplete perspective  on a program’s performance because a rebaseline shortens the period of  performance reported and resets the measurement of cost growth to zero.  In providing information to Congress, DOD does not measure and report  two meaningful perspectives—cumulative unit cost growth in constant  dollars and the change in unit cost between one budget request and the  next. In its SARs, DOD also does not present how unit cost growth is  adjusted for Nunn-McCurdy determinations when a rebaselining reduces  the number of units to be procured or increases system capabilities.  Current reporting policies and practices could be improved to provide  more useful information to decision makers and strengthen accountability  for performance.
subsections: section_title: Cumulative Unit Cost Changes Are Not Measured and Reported in Constant Dollars
paragraphs: DOD does not measure and report cumulative unit cost growth in constant  dollars.  Cumulative cost data provides an important perspective because  it reflects changes and performance over the life of a program—-from the  first full estimate, when the business case was first made to Congress, to  the present. Measuring change in constant dollars removes the effects of  inflation, which are beyond the control of individual programs, and is used  to measure real program cost growth.
DOD reports to Congress on the unit cost growth of programs in two  different sections of the SAR.  One section reports the current estimate  against the latest approved baseline in both then-year and constant dollars.  However, because of rebaselining, the latest approved baseline may be in  place for only a short period of time and the measurement of unit cost  growth is reset to zero.   The other section reports the cumulative  historical change only in then-year dollars, which includes the effects of  inflation.
Rebaselining can occur during each phase of acquisition, sometimes  frequently on individual programs. For example, in 2003, DOD reported on  81 programs; of this number, 49 or (60 percent) had multiple rebaselinings  over the life of the program. Table 1 provides examples of the highest  number of rebaselinings in individual DOD acquisition programs.
As illustrated in table 2, DOD reports unit cost growth in constant dollars  only for the most recent period of performance.  It does not present a full  history of a program’s unit cost growth in a single document. For example,  DOD reported in the December 2003 SAR that the Marine Corps’ H-1  helicopter upgrade program’s unit cost has shrunk by 1 percent in the last  20 months; however, DOD did not report that the program’s unit cost had,  in constant dollars, doubled in the last 87 months.
DOD reports in another section of the SAR the cumulative unit cost  change, but only in then-year dollars which includes the effects of  inflation.  The statute that requires submission of SARs to Congress calls  for reporting the history of the Program Acquisition Unit Cost and the  Procurement Unit Cost from the date the program was first included in the  SAR to the end of the quarter for which the current SAR is submitted.  Determining the change in constant dollars, which removes the effects of  inflation, provides more meaningful data.  For example, as shown in table  3, the unit cost growth reported to Congress for the F/A-18E/F was almost  zero percent because it was presented in then-year dollars. DOD did not  report that the costs increased by more than one-third, once the effects of  inflation are removed.
section_title: Year-to-Year Unit Cost Changes Are Not Measured and Reported
paragraphs: DOD does not measure and report the changes in unit cost between the  latest budget request and the prior budget. Between the 2004 and 2005  budget requests, DOD rebaselined 7 of the 81 major defense acquisition  programs. When DOD rebaselined these programs, it reset the cost growth  to zero.  For example, the Stryker armored vehicle program reported a  little over 1 percent unit cost growth in the 2 months since it rebaselined,  but DOD did not report the meaningful perspective that the program  experienced a 20 percent growth between annual budget requests. (Table  4 illustrates the differences in program acquisition unit cost when annual  changes are not reported.)
section_title: In Some Cases, DOD Reduces the Magnitude of Unit Cost Growth for Nunn-McCurdy Determinations
paragraphs: Congress established unit cost as a key measure of buying power—the  average cost to buy each unit. Congress also established the Nunn- McCurdy cost increase thresholds of 15 percent and 25 percent for unit  cost growth that require detailed reporting. As defined by law, unit costs  are determined by dividing total cost by total quantities. If quantities  decrease but costs stay the same or do not decrease proportionately, unit  costs would necessarily increase.
For determining Nunn-McCurdy breaches, DOD policy allows the  exclusion of the unit cost increases associated with reductions in quantity  or increases in capabilities on individual programs. DOD refers to these  actions as programmatic adjustments, which require the approval of a new  acquisition program baseline. DOD believes that these adjustments clarify  legitimate cost growth and applies its exclusion policy to determine all  Nunn-McCurdy breaches.
By making these adjustments, DOD reduced the number and the  magnitude of unit cost increases reported to Congress that would have  otherwise exceeded the Nunn-McCurdy thresholds. The effects of these  adjustments are generally not visible in the reports to Congress. As a  result, Congress may not be getting the information it sought on program  buying power in requiring unit cost reporting.
For example, as seen in table 5, in December 2002, the Comanche  Helicopter program experienced a PAUC growth of about 62 percent over  27 months. However, after excluding the unit cost increases associated  with a program restructure and a reduction in quantity of 561 aircraft,  DOD reported a Nunn-McCurdy breach of 18 percent, less than the 25  percent threshold that would have triggered a certification. DOD  explained that the 18 percent cost growth was driven by weight growth,  functionality changes, overhead rates, and estimating differences. The  quantity reduction accounted for the other 44 percent of the cost growth.
DOD would have reported or would have had to certify many more Nunn- McCurdy breaches if not for the adjustments made to unit cost growth to  account for reduced quantities or increased capabilities. The number of  programs that would have had Nunn-McCurdy breaches reported to  Congress between fiscal years 2001 and 2003 would have increased by  about 50 percent—from 17 to 25 programs. The number of programs that  reported a breach of the 25 percent threshold, which requires a formal  certification by the Secretary of Defense to Congress, would have  increased by 90 percent, going from 10 to 19 programs during the same  period. Three of those programs—the Joint Strike Fighter, Comanche, and  the MH-60R—reported breaching the 15 percent level. These three  programs notified Congress of an APUC breach and therefore also  reported the PAUC changes.  Other than those three programs, the  adjustments made to unit costs in table 6 were not disclosed in the  relevant SAR. For example, the Bradley Upgrade program experienced a  PAUC growth of 16 percent, but reported to Congress a 0.4 percent PAUC  reduction.  (Table 6 below shows the full unit cost growth and the  adjusted unit cost growth used to determine breaches of Nunn-McCurdy  thresholds.)
section_title: Reporting of Rebaselinings Could Be More Timely and Unclassified Data Is Unnecessarily Restricted
paragraphs: DOD could be more timely in reporting to Congress that it has rebaselined  individual programs. Although DOD’s time lines for reporting program  acquisition breach data to Congress are consistent with statute, in most  instances the establishment of a new baseline is not reported to Congress  until months after the rebaseline has occurred. A key factor here is DOD is  not required by statute or its own policies to report a program’s rebaseline  to Congress.
DOD typically reports the latest program rebaseline information in its  December SAR, which is actually submitted to Congress the following  April. Sometimes data can arrive for congressional consideration as much  as 12 months after a program has rebaselined. Between the April 2003 and  the April 2004 SARs to Congress, DOD rebaselined 9 programs and on  average these rebaselines were not reported for 6 months.  Between 1996  and 2003 about two-thirds of the rebaselinings occurred between April and  December. A new baseline approved after early April may not be reported  to Congress before the enactment of authorization and appropriations  legislation.
For example, the DD(X) Destroyer program established a new baseline on  April 23, 2002, but did not report this new baseline to Congress in a SAR  until April 2003.  As result, between the April 2002 and the passage of the  fiscal year 2003 defense budget, the SAR provided to Congress did not  reflect the approved baseline for the DD(X) program.  (Table 7 shows  instances of the time lag in reporting rebaselines to Congress.)
In addition, the accessibility of unclassified SAR data is unnecessarily  restricted. DOD classified about 50 percent of the SARs it submitted to  Congress in 2003, involving a total acquisition investment of $454 billion.   However, only a small amount of data contained in each classified SAR is  actually classified. The classified data is generally only one of the eighteen  sections in a report—performance characteristics.  Performance  characteristics include such items as speed, range, and reliability. Because  these SARs are classified, special handling procedures must be used by  those congressional staff with the appropriate clearances even to access  the unclassified cost and schedule data.  This practice also completely  blocks access for those staff without clearances to the unclassified cost  and schedule data.  As a result, congressional oversight of DOD’s  adherence to established cost and schedule baselines is unnecessarily  constrained.
section_title: Conclusions
paragraphs: Congress faces hard choices in trying to balance competing demands for  funds for major defense acquisition programs and for ongoing defense  operations as well as for increasing levels of funding needed for other  federal programs. A number of opportunities exist for DOD to give  Congress more complete information on the performance of major  defense acquisition programs for its oversight and decision-making  responsibilities. Although DOD has provided unit cost data required by  law, DOD would better capture the true nature of program performance by  also measuring and reporting unit cost changes in constant dollars using  cumulative and year-to-year perspectives.  Basic data to measure and  report such changes is already collected.  To provide these additional  perspectives, DOD would have to make the necessary comparisons and  present them in the SARs.
Because of DOD’s policy to adjust unit cost growth when determining  Nunn-McCurdy breaches by excluding the costs associated with  reductions in quantities or increases in capabilities, Congress is not  receiving information on the full extent of unit cost growth for these  programs. Therefore, DOD should fully disclose the nature and extent of  these adjustments.  Congress may also wish to determine the  appropriateness of these adjustments.
Other changes would result in more timely and accessible reporting of unit  cost data. One would be to inform Congress when a rebaselining action  occurs, without issuing a new SAR and the other would be to separate the  reporting of classified and unclassified information.
section_title: Recommendations for Executive Action
paragraphs: To provide Congress with more complete, timely, and accessible  information, we recommend that the Secretary of Defense implement the  following four recommendations in the department’s reporting of data on  major defense acquisition program baselines (APB) and performance    measure and report a full history of unit cost performance in constant  dollars by comparing the latest cost and quantity estimates with:  the first full estimate (typically the original acquisition program  baseline established at Milestone B);   the current approved program baseline or, if the program  rebaselines, the prior approved program baseline; and  the estimate established with the previous year’s budget request.  fully disclose to Congress the nature and extent of programmatic  adjustments affecting Nunn-McCurdy threshold determinations,  pending any congressional direction on this issue;    notify Congress when rebaselining actions are approved; and   separately report classified and unclassified SAR information.
section_title: Matters for Congressional Consideration
paragraphs: DOD's policy of excluding the effects of quantity reductions or capability  increases in determining unit cost breaches of Nunn-McCurdy thresholds  may not provide Congress the information it sought on program buying  power in requiring unit cost reporting.  Therefore, Congress may wish to  consider whether DOD’s use of these programmatic adjustments is  appropriate.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, DOD concurred with our  recommendations. DOD also provided us with technical comments, which  we have incorporated as appropriate. DOD expressed concern that the  report could be misinterpreted to imply that DOD is not fully complying  with statutory reporting requirements. As a result, with the exception of  the issue of programmatic adjustments, we have clarified the language of  the report to focus on the benefits to Congress of providing more  complete information, beyond what the law currently requires.  In the case  of the programmatic adjustments, we believe that Congress may wish to  determine whether DOD is providing it with the information it sought on  buying power when it required unit cost reports. Their written comments  to our recommendations appear in appendix II.
On our first recommendation, DOD stated that a unit cost comparison in  constant dollars from the current estimate back to multiple reference  points (i.e., Milestone B, approved APB, and previous year’s President  budget) would be more useful than what is currently being reported to  Congress.  DOD agreed to make these comparisons and also stated that it  will inform the Congress of this change in the SAR content.
On our second recommendation, DOD acknowledged that it has been  excluding programmatic changes from Nunn-McCurdy unit cost breach  determinations since February 1995, just after the Federal Acquisition  Streamlining Act established the APB as the Nunn-McCurdy unit cost  baseline. DOD agreed with GAO that the programmatic impacts to unit  cost are not entirely clear in the SAR as currently reported and agreed to  provide full details of programmatic adjustments in future reports to the  Congress.  DOD stated that it will inform the Congress of this change in  the SAR content.
On our third recommendation, DOD stated that notification of revised  APBs could be accomplished if Congress so requests.  While we are not  advocating that a SAR be generated to notify the Congress when a major  defense acquisition program is rebaselined, we remain convinced that  DOD should initiate notification via an appropriate medium to the  Congress whenever these programs are rebaselined.
We are sending copies of this report to the Secretary of Defense and other  interested parties. We will also provide copies to others on request. In  addition, the report will be available at no charge on the GAO Web site at  http://www.gao.gov.
Should you or your staff have any questions on matters discussed in this  report, please contact me on (202) 512-4841 or David Best, Assistant  Director, at (202) 512-8078. Principal contributors to this report were Leon  S. Gill, Danny Owens, Adam Vodraska, Robert Swierczek, Wendy Smythe,  Tana Davis, and Judy Lasley.
section_title: Appendix I: Scope and Methodology
paragraphs: To evaluate to what extent DOD’s use of rebaselining affected the  adequacy of data provided to Congress on the performance of major  defense acquisition programs, we relied on the Federal Internal Control  Standards to assess the comprehensiveness, timeliness, accessibility, and  appropriateness of the contents of data provide by DOD to Congress. Due  to the classified nature of most defense system’s performance parameters,  our analysis of rebaselining did not include the baseline system’s  operational performance parameters. We reviewed pertinent statutes, such  as 10 U.S.C. § 2432 (SAF), § 2433 (Unit Cost Reports), and § 2435 (Baseline  Description) and DOD acquisition system guidance, both the current  version and proposed revisions. We analyzed more than 650 SARs, both  quarterly and annual reports covering an 8 year period from 1996 to 2003,  to determine which program had growth that exceeded DOD’s internal 10- percent or the Nunn-McCurdy 15- and 25-percent thresholds. We arrayed  the data extracted from these reports to show all program breaches and  rebaselines by service, programs, breach types, and year. To assess the  reliability of the data contained in the SARs, we used the standardized  GAO data reliability interview template to interview cognizant DOD  officials from the office of the Under Secretary of Defense for Acquisition,  Technology, and Logistics (Acquisition Resources and Analysis). After  reviewing DOD’s management controls in place for assessing data  reliability, we concluded that management controls were adequate and the  data was sufficiently reliable for our engagement purposes. Further, we  also interviewed cognizant DOD officials in Washington, D.C., from the  offices of the Under Secretary of Defense for Acquisition, Technology, and  Logistics (Acquisition Resources and Analysis); Program Analysis and  Evaluation; Cost Analysis Improvement Group (Weapons System Cost  Analysis Division); and Office of General Counsel. We conducted our  review between February 2004 and January 2005 in accordance with  generally accepted government auditing standards.
section_title: Appendix II: Comments from the Department of Defense