Defense Acquisitions: Information for Congress on Performance of Major Programs Can Be More Complete, Timely, and Accessible
Released: 2005-03-28T00:00:00

Letter

Background

In 1967, to obtain consistent, reliable data on major defense acquisition programs, DOD instituted a reporting system to summarize program cost, schedule, and performance information called the SAR. The purpose was to focus management attention on a program’s performance and changes to its acquisition plan. In 1975, Congress established the SAR as a statutory reporting requirement to Congress.

Consistent with statute, DOD requires an acquisition program baseline at program initiation that establishes objective and threshold values for cost, quantity, schedule, and performance parameters. Objective values represent what the user desires and expects. Thresholds represent the acceptable limits to those values that, in the user’s judgment, still provide the needed capability. The acquisition program baseline is derived from the user’s performance and schedule needs and the best estimates of total program cost consistent with projected funding.

A baseline, according to DOD acquisition guidance, consists of the following: Performance parameters—-The total number of performance parameters should be the minimum number needed to characterize the major drivers of operational performance. The number and specificity of performance parameters may change over time.

Schedule parameters—-Schedule parameters should include, at a minimum, the projected dates for program initiation, other major decision points, and initial operating capability. The program manager may propose, and the milestone decision authority may approve, other critical system events.

Cost parameters—-Cost parameters should reflect realistic cost estimates of the total program and/or increment, and should be identified as life-cycle cost. The elements of program life-cycle cost include research, development, test, and evaluations; procurement costs; military construction costs; acquisition-related operations and maintenance costs, if any; total system quantity (to include both fully configured development and production units); average procurement unit cost (defined as total procurement cost divided by total procurement quantity); program acquisition unit cost (defined as the total of all acquisition- related appropriations divided by the total quantity of fully configured end items); and any other cost objectives established by the milestone decision authority.

The defense acquisition management framework defines the stages through which typical programs proceed. As each stage concludes, a decision must be made to initiate, continue, advance, adjust, or terminate a project or program work effort or phase. The review associated with each of these decision points typically addresses program progress and risk, affordability, program trade-offs, acquisition strategy updates, and the development of exit criteria for the next phase or effort. The milestone decision authority is responsible for approving the program structure as part of the acquisition strategy. Milestone decision points are Milestone A, for entry into the technology development phase; Milestone B, for entry into the system development and demonstration phase (acquisition program initiation); and Milestone C, for entry into the production and deployment phase.

There are three fixed acquisition program baselines (APB) tied to these milestone decisions. The concept baseline approved at Milestone A; the development baseline approved at Milestone B; and the production baseline approved at Milestone C. This first full estimate, generally established at Milestone B, is sometimes referred to as the original business case for the program. All estimates include a mix of both sunk and projected remaining costs.

Programs are rebaselined based on a change in requirements, a change in acquisition phase, or a program restructuring. They are also rebaselined when there is a realized cost overrun or schedule slip beyond certain thresholds. Rebaselining can occur at any time and cover any phase of the defense acquisition program. All rebaselines must be approved by the milestone decision authority. A program may therefore have several approved program baselines during development, but only one currently approved program baseline. Programs also monitor a current estimate of the elements of a baseline and report differences as they develop.

Recognizing the need to establish a monitoring mechanism to provide oversight of cost growth in DOD major defense acquisition programs, Congress has required DOD to (1) notify Congress whenever unit cost growth is at least 15 percent, and (2) “certify” the program to Congress when the unit cost growth is at least 25 percent above the latest approved acquisition program baseline cost estimate. However, Congress did not require DOD to report when a program has been rebaselined.

The reporting requirement is commonly referred to as Nunn-McCurdy, after the congressional leaders responsible for the requirement. Exceeding either the 15 or 25 percent unit cost threshold is referred to as a Nunn-McCurdy unit cost breach. The unit cost reporting includes two elements—program acquisition unit cost (PAUC) and procurement unit cost (PUC). DOD refers to the PUC as the average procurement unit cost (APUC). Unit cost is considered a key measure because it describes buying power—the average cost to buy each unit.

Unit cost information is submitted to Congress in DOD’s annual comprehensive SARs. In addition to the comprehensive annual report to Congress for the period ending December 31, DOD also prepares a quarterly report for the second, third, and fourth quarters of the fiscal year when there has been an increase of 15 percent or greater in the current estimate of the program acquisition unit cost or average procurement unit cost in base- year dollars; a 6-month or greater delay in the current estimate of any schedule milestone since the current estimate of the previous report; or a milestone B or C and associated acquisition program baseline approval within 90 days prior to the quarterly report as of that date.

Information for Congress on Unit Cost Performance Could Be More Complete

DOD could be reporting more complete information on unit cost performance to Congress for its authorization and appropriations deliberations. A new baseline serves an important management control purpose when program goals are no longer achievable, because it presents an important perspective on the program’s current status and acquisition strategy. However, by comparing the latest unit cost estimate with the most recent approved baseline, DOD provides an incomplete perspective on a program’s performance because a rebaseline shortens the period of performance reported and resets the measurement of cost growth to zero. In providing information to Congress, DOD does not measure and report two meaningful perspectives—cumulative unit cost growth in constant dollars and the change in unit cost between one budget request and the next. In its SARs, DOD also does not present how unit cost growth is adjusted for Nunn-McCurdy determinations when a rebaselining reduces the number of units to be procured or increases system capabilities. Current reporting policies and practices could be improved to provide more useful information to decision makers and strengthen accountability for performance.

Cumulative Unit Cost Changes Are Not Measured and Reported in Constant Dollars

DOD does not measure and report cumulative unit cost growth in constant dollars. Cumulative cost data provides an important perspective because it reflects changes and performance over the life of a program—-from the first full estimate, when the business case was first made to Congress, to the present. Measuring change in constant dollars removes the effects of inflation, which are beyond the control of individual programs, and is used to measure real program cost growth.

DOD reports to Congress on the unit cost growth of programs in two different sections of the SAR. One section reports the current estimate against the latest approved baseline in both then-year and constant dollars. However, because of rebaselining, the latest approved baseline may be in place for only a short period of time and the measurement of unit cost growth is reset to zero. The other section reports the cumulative historical change only in then-year dollars, which includes the effects of inflation.

Rebaselining can occur during each phase of acquisition, sometimes frequently on individual programs. For example, in 2003, DOD reported on 81 programs; of this number, 49 or (60 percent) had multiple rebaselinings over the life of the program. Table 1 provides examples of the highest number of rebaselinings in individual DOD acquisition programs.

As illustrated in table 2, DOD reports unit cost growth in constant dollars only for the most recent period of performance. It does not present a full history of a program’s unit cost growth in a single document. For example, DOD reported in the December 2003 SAR that the Marine Corps’ H-1 helicopter upgrade program’s unit cost has shrunk by 1 percent in the last 20 months; however, DOD did not report that the program’s unit cost had, in constant dollars, doubled in the last 87 months.

DOD reports in another section of the SAR the cumulative unit cost change, but only in then-year dollars which includes the effects of inflation. The statute that requires submission of SARs to Congress calls for reporting the history of the Program Acquisition Unit Cost and the Procurement Unit Cost from the date the program was first included in the SAR to the end of the quarter for which the current SAR is submitted. Determining the change in constant dollars, which removes the effects of inflation, provides more meaningful data. For example, as shown in table 3, the unit cost growth reported to Congress for the F/A-18E/F was almost zero percent because it was presented in then-year dollars. DOD did not report that the costs increased by more than one-third, once the effects of inflation are removed.

Year-to-Year Unit Cost Changes Are Not Measured and Reported

DOD does not measure and report the changes in unit cost between the latest budget request and the prior budget. Between the 2004 and 2005 budget requests, DOD rebaselined 7 of the 81 major defense acquisition programs. When DOD rebaselined these programs, it reset the cost growth to zero. For example, the Stryker armored vehicle program reported a little over 1 percent unit cost growth in the 2 months since it rebaselined, but DOD did not report the meaningful perspective that the program experienced a 20 percent growth between annual budget requests. (Table 4 illustrates the differences in program acquisition unit cost when annual changes are not reported.)

In Some Cases, DOD Reduces the Magnitude of Unit Cost Growth for Nunn-McCurdy Determinations

Congress established unit cost as a key measure of buying power—the average cost to buy each unit. Congress also established the Nunn- McCurdy cost increase thresholds of 15 percent and 25 percent for unit cost growth that require detailed reporting. As defined by law, unit costs are determined by dividing total cost by total quantities. If quantities decrease but costs stay the same or do not decrease proportionately, unit costs would necessarily increase.

For determining Nunn-McCurdy breaches, DOD policy allows the exclusion of the unit cost increases associated with reductions in quantity or increases in capabilities on individual programs. DOD refers to these actions as programmatic adjustments, which require the approval of a new acquisition program baseline. DOD believes that these adjustments clarify legitimate cost growth and applies its exclusion policy to determine all Nunn-McCurdy breaches.

By making these adjustments, DOD reduced the number and the magnitude of unit cost increases reported to Congress that would have otherwise exceeded the Nunn-McCurdy thresholds. The effects of these adjustments are generally not visible in the reports to Congress. As a result, Congress may not be getting the information it sought on program buying power in requiring unit cost reporting.

For example, as seen in table 5, in December 2002, the Comanche Helicopter program experienced a PAUC growth of about 62 percent over 27 months. However, after excluding the unit cost increases associated with a program restructure and a reduction in quantity of 561 aircraft, DOD reported a Nunn-McCurdy breach of 18 percent, less than the 25 percent threshold that would have triggered a certification. DOD explained that the 18 percent cost growth was driven by weight growth, functionality changes, overhead rates, and estimating differences. The quantity reduction accounted for the other 44 percent of the cost growth.

DOD would have reported or would have had to certify many more Nunn- McCurdy breaches if not for the adjustments made to unit cost growth to account for reduced quantities or increased capabilities. The number of programs that would have had Nunn-McCurdy breaches reported to Congress between fiscal years 2001 and 2003 would have increased by about 50 percent—from 17 to 25 programs. The number of programs that reported a breach of the 25 percent threshold, which requires a formal certification by the Secretary of Defense to Congress, would have increased by 90 percent, going from 10 to 19 programs during the same period. Three of those programs—the Joint Strike Fighter, Comanche, and the MH-60R—reported breaching the 15 percent level. These three programs notified Congress of an APUC breach and therefore also reported the PAUC changes. Other than those three programs, the adjustments made to unit costs in table 6 were not disclosed in the relevant SAR. For example, the Bradley Upgrade program experienced a PAUC growth of 16 percent, but reported to Congress a 0.4 percent PAUC reduction. (Table 6 below shows the full unit cost growth and the adjusted unit cost growth used to determine breaches of Nunn-McCurdy thresholds.)

Reporting of Rebaselinings Could Be More Timely and Unclassified Data Is Unnecessarily Restricted

DOD could be more timely in reporting to Congress that it has rebaselined individual programs. Although DOD’s time lines for reporting program acquisition breach data to Congress are consistent with statute, in most instances the establishment of a new baseline is not reported to Congress until months after the rebaseline has occurred. A key factor here is DOD is not required by statute or its own policies to report a program’s rebaseline to Congress.

DOD typically reports the latest program rebaseline information in its December SAR, which is actually submitted to Congress the following April. Sometimes data can arrive for congressional consideration as much as 12 months after a program has rebaselined. Between the April 2003 and the April 2004 SARs to Congress, DOD rebaselined 9 programs and on average these rebaselines were not reported for 6 months. Between 1996 and 2003 about two-thirds of the rebaselinings occurred between April and December. A new baseline approved after early April may not be reported to Congress before the enactment of authorization and appropriations legislation.

For example, the DD(X) Destroyer program established a new baseline on April 23, 2002, but did not report this new baseline to Congress in a SAR until April 2003. As result, between the April 2002 and the passage of the fiscal year 2003 defense budget, the SAR provided to Congress did not reflect the approved baseline for the DD(X) program. (Table 7 shows instances of the time lag in reporting rebaselines to Congress.)

In addition, the accessibility of unclassified SAR data is unnecessarily restricted. DOD classified about 50 percent of the SARs it submitted to Congress in 2003, involving a total acquisition investment of $454 billion. However, only a small amount of data contained in each classified SAR is actually classified. The classified data is generally only one of the eighteen sections in a report—performance characteristics. Performance characteristics include such items as speed, range, and reliability. Because these SARs are classified, special handling procedures must be used by those congressional staff with the appropriate clearances even to access the unclassified cost and schedule data. This practice also completely blocks access for those staff without clearances to the unclassified cost and schedule data. As a result, congressional oversight of DOD’s adherence to established cost and schedule baselines is unnecessarily constrained.

Conclusions

Congress faces hard choices in trying to balance competing demands for funds for major defense acquisition programs and for ongoing defense operations as well as for increasing levels of funding needed for other federal programs. A number of opportunities exist for DOD to give Congress more complete information on the performance of major defense acquisition programs for its oversight and decision-making responsibilities. Although DOD has provided unit cost data required by law, DOD would better capture the true nature of program performance by also measuring and reporting unit cost changes in constant dollars using cumulative and year-to-year perspectives. Basic data to measure and report such changes is already collected. To provide these additional perspectives, DOD would have to make the necessary comparisons and present them in the SARs.

Because of DOD’s policy to adjust unit cost growth when determining Nunn-McCurdy breaches by excluding the costs associated with reductions in quantities or increases in capabilities, Congress is not receiving information on the full extent of unit cost growth for these programs. Therefore, DOD should fully disclose the nature and extent of these adjustments. Congress may also wish to determine the appropriateness of these adjustments.

Other changes would result in more timely and accessible reporting of unit cost data. One would be to inform Congress when a rebaselining action occurs, without issuing a new SAR and the other would be to separate the reporting of classified and unclassified information.

Recommendations for Executive Action

To provide Congress with more complete, timely, and accessible information, we recommend that the Secretary of Defense implement the following four recommendations in the department’s reporting of data on major defense acquisition program baselines (APB) and performance measure and report a full history of unit cost performance in constant dollars by comparing the latest cost and quantity estimates with: the first full estimate (typically the original acquisition program baseline established at Milestone B); the current approved program baseline or, if the program rebaselines, the prior approved program baseline; and the estimate established with the previous year’s budget request. fully disclose to Congress the nature and extent of programmatic adjustments affecting Nunn-McCurdy threshold determinations, pending any congressional direction on this issue; notify Congress when rebaselining actions are approved; and separately report classified and unclassified SAR information.

Matters for Congressional Consideration

DOD's policy of excluding the effects of quantity reductions or capability increases in determining unit cost breaches of Nunn-McCurdy thresholds may not provide Congress the information it sought on program buying power in requiring unit cost reporting. Therefore, Congress may wish to consider whether DOD’s use of these programmatic adjustments is appropriate.

Agency Comments and Our Evaluation

In written comments on a draft of this report, DOD concurred with our recommendations. DOD also provided us with technical comments, which we have incorporated as appropriate. DOD expressed concern that the report could be misinterpreted to imply that DOD is not fully complying with statutory reporting requirements. As a result, with the exception of the issue of programmatic adjustments, we have clarified the language of the report to focus on the benefits to Congress of providing more complete information, beyond what the law currently requires. In the case of the programmatic adjustments, we believe that Congress may wish to determine whether DOD is providing it with the information it sought on buying power when it required unit cost reports. Their written comments to our recommendations appear in appendix II.

On our first recommendation, DOD stated that a unit cost comparison in constant dollars from the current estimate back to multiple reference points (i.e., Milestone B, approved APB, and previous year’s President budget) would be more useful than what is currently being reported to Congress. DOD agreed to make these comparisons and also stated that it will inform the Congress of this change in the SAR content.

On our second recommendation, DOD acknowledged that it has been excluding programmatic changes from Nunn-McCurdy unit cost breach determinations since February 1995, just after the Federal Acquisition Streamlining Act established the APB as the Nunn-McCurdy unit cost baseline. DOD agreed with GAO that the programmatic impacts to unit cost are not entirely clear in the SAR as currently reported and agreed to provide full details of programmatic adjustments in future reports to the Congress. DOD stated that it will inform the Congress of this change in the SAR content.

On our third recommendation, DOD stated that notification of revised APBs could be accomplished if Congress so requests. While we are not advocating that a SAR be generated to notify the Congress when a major defense acquisition program is rebaselined, we remain convinced that DOD should initiate notification via an appropriate medium to the Congress whenever these programs are rebaselined.

We are sending copies of this report to the Secretary of Defense and other interested parties. We will also provide copies to others on request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

Should you or your staff have any questions on matters discussed in this report, please contact me on (202) 512-4841 or David Best, Assistant Director, at (202) 512-8078. Principal contributors to this report were Leon S. Gill, Danny Owens, Adam Vodraska, Robert Swierczek, Wendy Smythe, Tana Davis, and Judy Lasley.

Appendix I: Scope and Methodology

To evaluate to what extent DOD’s use of rebaselining affected the adequacy of data provided to Congress on the performance of major defense acquisition programs, we relied on the Federal Internal Control Standards to assess the comprehensiveness, timeliness, accessibility, and appropriateness of the contents of data provide by DOD to Congress. Due to the classified nature of most defense system’s performance parameters, our analysis of rebaselining did not include the baseline system’s operational performance parameters. We reviewed pertinent statutes, such as 10 U.S.C. § 2432 (SAF), § 2433 (Unit Cost Reports), and § 2435 (Baseline Description) and DOD acquisition system guidance, both the current version and proposed revisions. We analyzed more than 650 SARs, both quarterly and annual reports covering an 8 year period from 1996 to 2003, to determine which program had growth that exceeded DOD’s internal 10- percent or the Nunn-McCurdy 15- and 25-percent thresholds. We arrayed the data extracted from these reports to show all program breaches and rebaselines by service, programs, breach types, and year. To assess the reliability of the data contained in the SARs, we used the standardized GAO data reliability interview template to interview cognizant DOD officials from the office of the Under Secretary of Defense for Acquisition, Technology, and Logistics (Acquisition Resources and Analysis). After reviewing DOD’s management controls in place for assessing data reliability, we concluded that management controls were adequate and the data was sufficiently reliable for our engagement purposes. Further, we also interviewed cognizant DOD officials in Washington, D.C., from the offices of the Under Secretary of Defense for Acquisition, Technology, and Logistics (Acquisition Resources and Analysis); Program Analysis and Evaluation; Cost Analysis Improvement Group (Weapons System Cost Analysis Division); and Office of General Counsel. We conducted our review between February 2004 and January 2005 in accordance with generally accepted government auditing standards.

Appendix II: Comments from the Department of Defense