Defense Logistics: Improvements Needed to Enhance Oversight of Estimated Long-term Costs for Operating and Supporting Major Weapon Systems
Released: 2012-02-02T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: With the nation facing fiscal challenges and the potential for tighter defense budgets, Congress and the Department of Defense (DOD) have placed more attention on controlling the billions of dollars spent annually on weapon system operating and support (O&S) costs. These costs include, costs for repair parts, maintenance, and personnel, and account for about 70 percent of the total costs of a weapon system over its life cycle. The selected acquisition report (SAR) is DOD’s key recurring status report on the cost, schedule, and performance of major defense acquisition programs and is intended to provide authoritative information for congressional oversight of these programs. Oversight of O&S costs is important because many of the key decisions affecting these life-cycle costs are made during the acquisition process. GAO reviewed weapon system O&S cost estimates that DOD submits in the SAR. Specifically, GAO determined the extent to which the SARs provide consistent and reliable O&S cost estimate information that enables effective oversight of these weapon system costs. To conduct its review, GAO analyzed SAR data for 84 major systems that submitted O&S cost estimates in the 2010 SAR and selected a nonprobability sample of 15 systems for more in-depth review.

section_title: What GAO Found
paragraphs: DOD’s reports to Congress on estimated weapon system O&S costs are often inconsistent and sometimes unreliable, limiting visibility needed for effective oversight of these costs. The SAR statute requires that life-cycle cost reporting for major weapon systems be uniform, to the extent practicable, across the department, but GAO found a number of inconsistent practices in how program offices were reporting life-cycle O&S cost estimates in the SAR. Program offices were inconsistent in (1) the explanatory information they included with the cost estimates; (2) the source of the cost estimate they cited as the basis for the reported costs; (3) the unit of measure they used to portray average costs; (4) the frequency with which they updated reported costs; and (5) the reporting of costs for an antecedent system being replaced by the new weapon system. For example, 35 (42 percent) of the 84 programs that reported O&S costs in the 2010 SAR did not cite a source of these data, contrary to DOD’s guidance, and 57 (68 percent) of the programs did not report O&S costs for an antecedent system. Also, O&S cost submissions in the SAR did not always incorporate best practices for presenting cost estimates, such as tracking cost changes over time and identifying cost drivers. In addition, 11 systems did not provide O&S cost estimates in the 2010 SAR.

Although SARs are intended to provide Congress with authoritative program information on major weapon systems, 7 of the 15 sample programs GAO reviewed submitted unreliable O&S cost estimate data in the 2007, 2009, or 2010 SARs. For example, an Air Force program underreported O&S costs by $2.1 billion (fiscal year 2002 dollars), or 18 percent. While some of the program offices did not provide an explanation for the errors in the submitted data, others cited specific reasons. For example, one Navy program office underreported O&S costs in the SAR and explained that it excluded certain costs that were not under its control, such as externally funded spare parts and military personnel. However, excluding such costs is contrary to the SAR statute. An Air Force program reported current and projected funding for the program rather than estimated life-cycle O&S costs. This practice also had the effect of underreporting these costs.

DOD’s reports to Congress on estimated weapon system O&S costs were often inconsistent and sometimes unreliable due to a lack of (1) detailed implementation guidance for reporting these costs and (2) an effective process for reviewing the O&S cost sections of the SAR before final submission to Congress. DOD’s guidance collectively provides minimal instructions for O&S cost reporting. The guidance also does not incorporate some of the best practices GAO has identified for presenting cost estimates. Further, although the SAR data submitted by program offices are subject to multiple reviews within the military services and by the Office of the Secretary of Defense, this review process has not provided assurance that O&S costs are reported consistently and reliably. In the absence of improvements to the SAR guidance and to the review process, deficiencies in reporting O&S costs are likely to continue. Improved reporting of O&S costs in the SAR could help to place more emphasis on assessing, managing, and controlling long-term weapon system O&S costs.

section_title: What GAO Recommends
paragraphs: To enhance visibility of weapon system O&S costs during acquisition, GAO recommends that DOD improve its guidance to program offices on cost reporting and also improve its process for reviewing these costs prior to final submission of the SAR to Congress. DOD concurred with GAO’s recommendations.

section_title: Letter
paragraphs: With the nation facing fiscal challenges and the potential for tighter  defense budgets, both Congress and the Department of Defense (DOD)  in recent years have placed more attention on controlling the billions of  dollars spent annually on weapon system operating and support (O&S)  costs. These costs include, among other things, costs for repair parts,  maintenance, and personnel, and historically have accounted for about  70 percent of a weapon system’s total costs. In short, the acquisition of a  weapon system today involves a significant financial commitment to that  system over its entire life cycle, a period that may last several decades  from the system’s development to the time it is removed from DOD’s  inventory. For example, the program office for the F-35 Joint Strike  Fighter, the newest aircraft being acquired for the Air Force, Navy, and  Marines, estimated in 2010 that life-cycle O&S costs were about $1.0  trillion, in addition to an estimated $379.4 billion in total acquisition costs.

Since 1969, the selected acquisition report (SAR) has been the key  recurring summary status report to Congress on the cost, schedule, and  performance of DOD’s major defense acquisition programs. (We also  use the term major weapon systems to refer to these programs, although  some programs involve other types of defense systems.) The SAR is  intended to provide authoritative information needed for congressional  oversight of these programs. Although much of the data reported within  the SAR is acquisition related, in 1985 Congress amended the SAR  statute to require that a full life-cycle cost analysis also be included in the  SAR, and subsequently specified that this life-cycle cost analysis include  estimated O&S costs. This information is important for effective oversight  because many of the key decisions affecting a weapon system’s O&S  costs over its life cycle are made while the system is still in the acquisition  process. Acquisition-related decisions about the design, materials, and  technology for the system affect the logistics support that is eventually  needed to keep the system available and ready after it enters into DOD’s  inventory. Further, investments to improve reliability, availability, and  maintainability during acquisition can reduce the future O&S costs of  DOD’s weapon systems. For example, higher reliability can be designed  into the weapon system during its development by reducing technical  requirements, using highly reliable and proven components, or investing  more in early testing, thus improving prospects for readiness and  reducing O&S costs across the life cycle. As the visibility and  management of O&S costs have become more of a focus in the  department, the Office of the Secretary of Defense (OSD) has recognized  the SAR as a source of O&S cost information for oversight of weapon  system acquisitions. Therefore, the O&S costs reported in the SARs are  not only reported to Congress, but they are also included in internal OSD  management reports as well.

In response to the Weapon Systems Acquisition Reform Act of 2009, we  previously reviewed the growth in O&S costs of major weapon systems  and reported that DOD lacked key information needed to effectively  manage and reduce O&S costs—including life-cycle O&S cost estimates  and complete historical data on actual O&S costs. DOD generally  concurred with the recommendations in that report and stated that it was  taking corrective actions. More recently, the National Defense  Authorization Act for Fiscal Year 2012 directs DOD to take a number of  actions consistent with our recommendations, with the aim of better  assessing, managing, and controlling weapon system O&S costs. In light  of the findings from our prior work, you requested that we review the O&S  cost estimate information for major weapon systems that DOD submits to  Congress within the annual SARs. Specifically, our objective was to  determine the extent to which the SARs provide consistent and reliable  O&S cost estimate information to enable effective oversight of these  weapon system costs.

To meet our objective, we reviewed statutory requirements and DOD  guidance for reporting weapon system O&S cost estimates in the SARs.  We also reviewed DOD cost-estimating guidance, which identifies the  specific categories of O&S costs for weapon systems, and GAO-identified  cost-estimating best practices to identify the scope and nature of cost  estimate information needed for effective program management and  oversight. We interviewed DOD and military service officials responsible  for weapon system acquisition, logistics, and cost analysis to understand  DOD’s approach and process for reporting O&S cost estimates in the  SARs.

We obtained SARs from the Defense Acquisition Management  Information Retrieval system, which is a web-based system used within  DOD to collect and maintain SAR information submitted by the program  offices. We determined that the data in this system accurately reflected  information submitted by weapon system program offices and therefore  were sufficiently reliable for the purposes of our analysis. We obtained  and analyzed annual SAR data for all 84 major weapon systems that  reported O&S costs in their annual SAR for 2010. This analysis covered  the time period from 2005 through 2010 with the exception of 2008. In  addition, from these 84 weapon systems, we selected a sample of 15  programs for further analysis. Through a data collection instrument,  interviews, and other contacts, we obtained information from the program  offices on the O&S cost reports they had submitted in the SAR, including  cost estimates, where available, that formed the basis for their cost  reports. We designed the sample to ensure that a range of weapon  systems were represented based on commodity type and the military  service under which the system acquisition was being managed. The  results from this nonprobability sample cannot be used to make  inferences about all major weapon systems because the sample may not  reflect all characteristics of the population. We also obtained SARs for 11  weapon systems that reported an annual SAR for 2010 but did not  identify O&S costs in the SAR. We obtained information from the program  offices about the reasons these costs were not reported. Our scope and  methodology is discussed further in appendix I.

We conducted this performance audit from February 2011 to February  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: Statutory Requirements for Submitting SARs to Congress
paragraphs: Requirements for submitting SARs to Congress, including the timing of  these reports and the types of information to be included, are established  in statute. Under 10 U.S.C. § 2432, the Secretary of Defense shall submit  to Congress at the end of each fiscal-year quarter a report on current  major defense acquisition programs. Each SAR for the first quarter of a  fiscal year (also known as the comprehensive annual SAR) shall be  designed to provide to the Committee on Armed Services of the Senate  and the Committee on Armed Services of the House of Representatives  the information these committees need to perform their oversight  functions. The comprehensive annual SAR shall be submitted within 60  days after the date on which the President’s Budget is sent to Congress  for the following fiscal year.

The statute also requires that the annual SAR include a full life-cycle cost  analysis for each major defense acquisition program and each designated  major subprogram included in the report that is in the system  development and demonstration stage or has completed that stage. Further, the Secretary of Defense must ensure that this requirement is  implemented in a uniform manner, to the extent practicable, throughout  DOD. The term full life-cycle cost, with respect to a major weapon  system, means all costs of development, procurement, military  construction, and operations and support, without regard to funding  source or management control. If the major weapon system has an  antecedent system, a full life-cycle cost analysis for that system must also  be reported. The SAR reporting requirement ceases after 90 percent of  the items are delivered or 90 percent of planned expenditures under the  program are made.

This stage of acquisition is now called engineering and manufacturing development.

section_title: DOD’s Guidance for Implementing the Statutory SAR Requirements
paragraphs: DOD has issued various guidance documents that implement the  statutory SAR requirements. This guidance is contained in an acquisition  instruction, a guidebook on defense acquisition best practices, a draft  SAR policy, and an annual memorandum on preparing SARs. DOD  also has developed instructions and training for entering SAR data into  the Defense Acquisition Management Information Retrieval system.  According to officials, program offices rely on DOD’s implementation  guidance because the services do not have their own formal SAR  reporting guidance. DOD’s guidance is summarized below and discussed  more fully in appendix II.

DOD’s acquisition process includes a series of decision milestones as the systems  enter different stages of development and production. As part of the process, the DOD  component or joint program office responsible for the acquisition program is required to  prepare life-cycle cost estimates, which include O&S costs, to support these decision  milestones and other reviews. Key decision milestones include milestone B, which  approves entry into the engineering and manufacturing development phase, and  milestone C, which approves entry into the production and deployment phase, including  low-rate initial production. Continuation into full-rate production occurs after the full-rate  production decision review is held. In conjunction with a milestone decision, a program  may be rebaselined, which means that the cost, quantity, schedule, and performance  goals are changed to reflect the current status. that program offices should provide explanatory information such as the  source and date of the cost estimate, assumptions underlying the  estimate, the antecedent system used for comparison purposes, and an  explanation of how average annual costs were calculated. DOD officials  stated that programs should report the cost estimate that was developed  for the latest acquisition milestone decision. According to the guidance,  programs should report total estimated O&S costs and should also report  average O&S costs by a unit of measure (e.g., average annual cost per  squadron, average annual cost per system). DOD’s guidance states that  if a program has an antecedent system,assumptions should be submitted for the antecedent system.   then O&S costs and  In addition to its SAR implementation guidance, DOD has issued  guidance for developing weapon system O&S cost estimates, which  provide the basis for the O&S cost section of each SAR. Specifically, the  OSD Cost Analysis Improvement Group, now known as the Cost  Assessment and Program Evaluation (CAPE) office, has established  guidance for preparing and presenting life-cycle O&S cost estimates at  acquisition milestone reviews. O&S cost elements, for example, are to be grouped into six major areas—unit-level personnel, unit operations,  maintenance, sustaining support, continuing system improvements, and  indirect support—which are further broken down into 23 subelements. In  addition, we have identified federal government best practices for  preparing and presenting cost estimates. These practices include tracking  cost estimates over time; identifying the major cost drivers; identifying the  method and process for estimating each cost element; and comparing the  program-developed cost estimate to an independent cost estimate.

section_title: DOD’s Process and Schedule for Preparing SARs
paragraphs: When required, a comprehensive annual SAR is prepared for each major  weapon system by the program office that is managing its acquisition.  Program offices are responsible for weapon systems throughout the life  cycle, to include implementing, managing, and/or overseeing their  development, production, fielding, sustainment, and disposal. The  reporting time frame for the annual SAR is linked to the issuance of the  President’s Budget, which occurs early in the calendar year, and the cost,  schedule, and performance data reported in the annual SAR should  reflect this budget request.

The Office of the Under Secretary of Defense for Acquisition, Technology  and Logistics begins the process by sending out its annual memorandum  guidance in mid-January. Program offices then enter data into the  Defense Acquisition Management Information Retrieval system and  submit the SARs to OSD acquisition officials, generally after some level of  internal review by the program office, the Program Executive Officer,  and the military service under which the program is organized. OSD  officials review the SAR submissions, and officials within the Office of the  Assistant Secretary of Defense (Logistics and Materiel Readiness) focus  on the O&S section of the reports. OSD officials then hold a series of  meetings with the services and program office representatives to discuss  the SAR submissions and any recommended changes. Consistent with  the statutory requirement, the final annual SAR is typically submitted to  Congress in April, 60 days after the President’s Budget has been  submitted in February.

section_title: DOD’s Reports to Congress on Estimated Weapon System O&S Costs Were Inconsistent and Sometimes Unreliable
paragraphs: Program offices reporting life-cycle O&S cost estimates in the SAR were  often inconsistent in their cost reporting and also did not follow best  practices for presenting cost estimates. In addition, some programs did  not provide any O&S cost estimates in the 2010 SAR. Further, several of  the programs we reviewed in more depth reported unreliable O&S cost  data. The main cause for these deficiencies was a lack of detailed SAR  implementation guidance for reporting O&S costs. In addition, DOD’s  process for reviewing the O&S cost sections of the SAR prior to their final  submission did not provide assurance that the program offices reported  costs uniformly, to the extent practicable, and that these reported costs  were reliable. In the absence of improvements to the SAR guidance and  to DOD’s review process, deficiencies in reporting estimated life-cycle  O&S costs are likely to continue. Such deficiencies may limit visibility  needed for effective oversight of long-term weapon system O&S costs  during the acquisition process.
subsections: section_title: Program Offices Were Inconsistent in Reporting Estimated Life-Cycle O&S Costs and Did Not Incorporate Best Practices for Presenting These Cost Estimates
paragraphs: The SAR statute requires that life-cycle cost reporting for major weapon  systems be uniform, to the extent practicable, across the department, but  we found a number of inconsistent practices in how program offices were  reporting life-cycle O&S cost estimates in the SAR. Based on the SAR  submissions we reviewed, program offices were inconsistent in (1) the  explanatory information they included with the cost estimates, (2) the  source of the cost estimate they cited as the basis for the reported costs,  (3) the unit of measure they used to portray average costs, (4) the  frequency with which they updated reported costs, and (5) the reporting of  antecedent system costs. In addition to these inconsistencies, we found  that SAR submissions also did not incorporate best practices for  presenting cost estimates, such as tracking cost changes over time and  identifying cost drivers. In addition, 11 systems did not provide O&S cost  estimates in the 2010 SAR.

Submitting more consistent cost reports and incorporating best practices  for presenting cost estimates would improve visibility of estimated life- cycle O&S costs in the SAR, as decision makers will have more  information with which to evaluate the reported data. For example, the  inclusion of the date and the source of the reported estimate provides  context about the currency of the reported costs and the level of review  (that is, whether the cost estimate was prepared by the program office, by  the military service, or by CAPE). Likewise, the inclusion of significant  assumptions underlying the cost estimate, an explanation of changes in  the cost estimate from the prior year, and information on major cost  drivers provides insight into the cost challenges facing the program. In  addition, showing average costs using a common unit of measure allows  for easier comparison of program costs to the costs of similar  commodities (such as other aircraft programs).

DOD’s implementation guidance for the SAR directs programs to include  explanatory information in the narrative accompanying the O&S cost  estimates, such as the source and date of the cost estimate, assumptions  underlying the estimate (such as operating tempo, expected reliability and  maintainability of the system, maintenance concept, and manning and  logistics policies), the antecedent program used for comparison purposes,  and an explanation of how average costs were calculated. Although  explanatory information can provide context and background for  understanding reported costs, we found that the explanatory information  included in the O&S narrative was often minimal.

Of the 84 programs that reported O&S costs in the 2010 SAR, we found  that 35 (42 percent) did not include the source of the estimate and 12 (14  percent) did not include the date of the estimate in the O&S narrative.  Additionally, for the 15 programs in our sample, we found that beyond  providing a few basic details such as the number of units that were to be  acquired, their expected service life, and operating tempo, where  applicable, the O&S narrative contained minimal explanation of reported  cost estimates and the assumptions underlying these estimates, as the  following examples illustrate:

The program office for the Army’s High Mobility Artillery Rocket  System included several assumptions, such as the number of  launchers and the service life. However, instead of reporting  additional O&S cost estimate assumptions (such as operating tempo  and expected reliability/maintainability) in the SAR narrative, the  program stated that this information was available in the service cost  estimate.

The Joint Mine Resistant Ambush Protected (MRAP) vehicle program  office noted a few specific assumptions, such as the expected service  life of the fleet and the cost per mile for replenishing spare parts.  However, the remaining O&S narrative for the program generically  explained that the estimate included personnel, training, facilities,  vehicle and component repair, and sustainment overhauls, but  provided no other specifics on these areas.

Only three programs—the Army’s High Mobility Artillery Rocket  System, the Air Force Joint Primary Aircraft Training System, and the  Army’s Force XXI Battle Command Brigade and Below (FBCB2)— included the maintenance concept planned for that system in their  O&S narratives. However, even in these cases, the explanatory  information for O&S costs was very limited.

None of the 15 programs included assumptions on the reliability and  maintainability of the weapon system in their O&S narrative.

While not required by DOD’s implementation guidance, 1 of the 15  programs in our sample included explanatory information on cost drivers  in the SAR O&S narrative. The V-22’s SAR submission for 2009 provided  an explanation of the significant O&S cost increase from the prior SAR in  2007. In the 2007 SAR, the total O&S costs reported were $48.8 billion  (fiscal year 2005 dollars). In the 2009 SAR, the program reported that this  amount had grown to $75 billion (fiscal year 2005 dollars), and that the  O&S cost category showing the greatest increase was unit-level  consumption. In the O&S narrative, the program office attributed the  majority of the cost increase to changes in the methodology used to  estimate unit-level consumption costs. Specifically, the estimate was  updated with the actual costs of parts from fiscal year 2009 and with  projected future cost growth for parts higher than OSD’s inflation  indices. The program office also noted actions being taken to reduce  unit-level consumption costs, such as changes to contracting strategy and  accelerated timelines for repair capabilities.

GAO-identified best practices for presenting cost estimates include  identifying the largest cost elements and cost drivers, and providing  enough information for informed decision making. In addition, we have  previously reported that leading companies identify major drivers of O&S  costs and work with manufacturers to reduce these costs. During our current review, we found that programs typically use CAPE’s O&S cost  element structure in reporting O&S costs, but their presentation is limited  to the six major elements (e.g., unit-level personnel, maintenance, indirect  support). Since each major O&S cost element includes various costs, this  information is not sufficient to identify specific cost drivers. Using lower- level cost elements, as provided for in CAPE’s cost element structure,  could provide greater visibility of O&S costs for oversight by decision  makers. For example, as noted in the case of the V-22 discussed above,  the unit-level consumption cost element consists of a number of  subelements that can provide additional insight into the discrete factors  driving a change in the estimated life-cycle O&S costs for that system.

Various cost estimates may be developed over the life cycle of a weapon  system, and DOD officials stated that programs should report the cost  estimate developed for the latest acquisition milestone decision. We  found that program offices—when a source was cited—cited several  different sources as the basis for their reported O&S cost information in  the 2010 SAR, and they did not provide an explanation for selecting the  source that was used rather than another source that may have been  available. As shown in table 1, for the 84 programs that included O&S  costs in the 2010 SAR, 42 (50 percent) of the programs cited a specific  cost estimate as the source of reported O&S costs. These sources were  either a program office cost estimate, service cost estimate, or CAPE  independent cost estimate. Another 35 programs (42 percent) did not cite  a source, as previously noted. The remaining 7 programs (8 percent)  cited a source other than a specific cost estimate.

Five programs in the “other” category in table 1 referred to cost estimates  but did not provide enough detail to determine what type of cost estimate  was used. For example, one program cited a “validated cost estimate”  without additional specificity about this estimate. Similarly, four programs  stated only that the source of their SAR O&S costs was a cost estimate  prepared for an acquisition decision, but they did not provide additional  information to identify a specific cost estimate. The remaining two  programs in the “other” category in table 1, both of which were included in  our sample, cited a source other than a cost estimate. One of these  programs, the Navy Multiband Terminal, reported total costs from the  milestone C acquisition program baseline, despite the existence of a  service cost estimate prepared for the acquisition decision in July 2010.  The other program, the Air Force’s Navstar Global Positioning System  (GPS), reported using current and future funding data instead of a cost  estimate.

Overall, six programs in our sample were among those that did not cite  the source of the estimate used to report O&S costs. When we asked  these six programs what source was used, five stated that the O&S cost  estimate data in the 2010 SAR were derived from program office cost  estimates, and the remaining program office stated that the source was a  CAPE independent cost estimate. The other nine programs in our sample  had cited a source, with five citing a program office cost estimate; two  citing a service cost estimate; and two, as noted above, citing either an  acquisition program baseline or funding data as the source of their O&S  costs.

As shown in table 1, some programs cited a CAPE independent cost  estimate as the source of the O&S costs reported in the 2010 SAR.  However, we found that one program in our sample, the LHA 6 America  Class, cited a program office cost estimate even though CAPE had  developed an independent cost estimate. Further, while not required, the  program did not mention in the SAR that an independent cost estimate  had been developed. Since 2005 the LHA 6 America Class program has  reported total O&S costs of $4.45 billion (fiscal year 2006 dollars) in its  SAR submissions, reflecting a 2005 program office cost estimate.  However, CAPE’s 2006 independent cost estimate of the program’s O&S  costs was about $300 million (7 percent) higher. According to a CAPE  memorandum, this higher estimate was also not adjusted for cost growth  above inflation. CAPE noted that O&S costs for the LHA 1, an antecedent  system, had increased 4 percent annually since 1990 due to increased  mission personnel and overhaul costs. According to CAPE, adjusting for  this same rate of cost growth above inflation in its LHA 6 estimate would  result in an additional $530 million throughout the system’s life cycle, or  total O&S costs of $5.29 billion. Additionally, F-35 officials told us that  they plan to continue using the program office’s cost estimate to report  O&S costs in the SAR although CAPE is preparing an independent cost  estimate for the program’s next acquisition milestone.

GAO-identified best practices for presenting cost estimates include  providing a comparison of the program estimate to an independent cost  estimate, with an explanation of results and differences. Such a  comparison is beneficial because an independent cost estimate should  provide an objective and unbiased assessment of expected program  costs that tests the program’s estimate for reasonableness. History has  shown a pattern of higher, more accurate cost estimates the further away  from the program office the independent cost estimate is prepared. In the  2009 Weapon Systems Acquisition Reform Act, Congress placed greater  emphasis on independent review of program cost estimates by requiring  that CAPE review cost estimates prepared in connection with all major  weapon systems, and conduct independent cost estimates for certain  systems prior to the milestone A, milestone B, low-rate initial production,  and full-rate production acquisition decisions. Prior to the Act, CAPE was  required to conduct independent cost estimates for some programs, but  was not required to review cost estimates prepared for all major weapon  systems.

DOD’s implementation guidance for the SAR states that programs should  report average O&S costs in a unit of measure determined by the military  service under which the system’s acquisition is being managed.  Programs are to report these average costs using CAPE’s major cost  elements. We found that several program offices had changed the unit of  measure they reported in the SAR from that used in previous SARs. In  addition, we found that the units of measure that were being reported  varied, particularly among aircraft programs. These inconsistencies make  it difficult to compare a program’s current and prior-year costs, or to  compare costs of similar programs.

Of the 84 programs that reported O&S costs in the 2010 SAR, 5 (6  percent) changed the average unit of measure reported from that used  the prior year. Specifically, two aircraft programs went from reporting  costs per squadron in the 2009 SAR to reporting costs per aircraft in their  2010 SAR, a missile program went from reporting costs per unit in the  2009 SAR to reporting total program costs in the 2010 SAR, and two  programs for communications systems went from reporting total program  costs in the 2009 SAR to reporting costs per radio in the 2010 SAR.  These last two programs— Joint Tactical Radio System (JTRS) Ground  Mobile Radios and JTRS Handheld, Manpack, and Small Form Fit—were  included in our sample. When we asked why they changed the unit of  measure, program officials responded that the decision was made based  on feedback they received from OSD when their 2010 SAR submissions  were undergoing review. Of the 5 programs, only the two aircraft  programs disclosed in the SAR that the unit of measure for that system  had changed from the prior year. These two programs reported that they  changed the unit of measure in order to standardize the calculation and  increase the comparability of programs within the same major command.

Also, based on analysis of the 84 systems, we found the most variation in  the unit of measure among aircraft systems. Different programs reported  the average cost per flying hour or the average annual cost per aircraft,  per squadron of aircraft, or per the entire fleet. This issue was also  evident among the programs in our sample that we analyzed in more  depth. For example, the F-35 program reported average cost per flying  hour, the V-22 program reported average cost per aircraft, the F-22  program reported average cost per squadron, and the Joint Primary  Aircraft Training System (a training aircraft) reported average cost for the  whole fleet. Ship costs, in contrast, were generally reported as average  cost per ship or hull, although one ship program reported average annual  cost per fleet. O&S costs for ground and other types of weapon systems  were usually reported as either cost per weapon system unit or total cost  for all weapon system units. However, a few other metrics were reported  by these programs, such as average annual cost per battalion or per  brigade combat team.

Although portraying average costs with a unit of measure could be useful  for tracking cost changes over time, we found that it was generally not  possible to identify changes in estimated O&S costs based on the  information reported in a single, annual SAR, since programs do not  report costs from the prior SAR. Although major weapon system  programs are required to identify and reconcile changes to estimated  acquisition costs from the prior SAR, and to provide an explanation for  each change, this is not required for O&S costs. Even though two of our  sample programs, the V-22 and the Navstar GPS, included a statement in  the SAR narrative that their O&S costs had changed, it was not possible  to tell how much without the prior year’s cost data. Our year-to-year  comparisons of reported costs in the SARs showed that cost changes  were occurring. For example, we found that the total estimated O&S costs  for the JTRS Handheld, Manpack, and Small Form Fit program  decreased from $25.5 billion (fiscal year 2004 dollars) in 2009 to $10.2  billion in 2010 (fiscal year 2004 dollars). This $15.3 billion decrease  occurred despite an increase in acquisition quantity of about 5,000 radios,  from around 216,000 to around 221,000. This change, as well as the  reasons for the change, was not identified in the SAR narrative. Similarly,  we found that the total estimated O&S costs for the F-35 program  increased $50 billion (fiscal year 2002 dollars) from 2009 to 2010. The  reason for this increase was not explained in the O&S narrative in the  SAR.

According to GAO-identified best practices for presenting cost estimates,  cost estimates should be tracked over time. Specifically, after an estimate  is updated, a comparison of the current and prior estimate should be  routinely performed and documented, and the results reported to decision  makers. A documented comparison allows cost estimators to see how  well they are estimating and how the program is changing over time. It  also allows others to track the estimates and to identify when, by how  much, and why the program cost more or less than planned.

Updated cost estimates can help to ensure that decision makers have the  most current data available on a program. The SAR statute requires  major defense acquisition programs to begin reporting when the program  is approved to begin the development phase of the acquisition process at  milestone B, and DOD’s implementation guidance similarly states that a  SAR should first be submitted when a program is initiated, normally at  milestone B, or designated as a major defense acquisition program, and  also when the program is rebaselined after a major milestone decision.  DOD officials stated that programs should report the cost estimate  developed for the latest acquisition milestone decision.  Our analysis for  the 84 major weapon system programs that included O&S costs in the  2010 SAR showed that program offices were inconsistent in the  frequency of their O&S cost updates between 2005 and 2010. In many  cases, programs provided more frequent updates than required by DOD’s  guidance, sometimes annually. However, 8 (13 percent) of the 61  programs that were included in the SAR every year during the 2005 to  2010 period did not update their O&S costs at any time during that period.  In contrast, 47 programs (56 percent) of the 84 programs in the 2010  SAR reported using a cost estimate that was prepared in 2010 or 2011 as  the source of their O&S costs.reporting SARs in 2009 or 2010.

These included 7 programs that began  Of the 15 programs in our sample, 3 did not update their SAR O&S costs  during the period between 2005 and 2010, 5 updated their costs once, 5  updated their costs 2 or 3 times, and 2 updated their O&S costs 4 times  during the period. For example, the Navy’s LHA 6 America Class  program office has consistently reported the O&S costs estimated for  milestone B, the program’s only acquisition milestone while under SAR  reporting requirements, in the annual SARs since 2005. Program officials  told us that they were in the process of developing a new cost estimate  for the LHA 7, the next ship in the America Class, and planned to use the  new estimate as the source to report O&S costs in the program’s 2011  SAR submission, if complete. Also, the Army’s FBCB2 program has not  updated its O&S SAR costs and is reporting costs estimated in 2004,  even though the program’s production quantity has quadrupled since  then. FBCB2 program officials told us that since its full-rate production  decision in 2004, the program has experienced nearly continuous  changes to its production quantity requirement, resulting in a significant  effort to maintain and update the acquisition portion of the cost estimates  and little time to research and update the O&S portion of the cost  estimates.

In contrast, several of our sample programs updated their O&S costs  annually. The F-35 program has updated the reported SAR O&S costs  annually since 2006, the beginning of the period we reviewed. According  to F-35 program officials, they chose to do this because the F-35 is a  high-visibility, high-interest program. Further, estimating O&S costs  annually helps inform DOD leadership and keeps partner countries  updated, program officials noted. Additionally, the Joint MRAP program  office has updated its SAR O&S costs annually since the program began  reporting these costs in 2009 and plans to do so until the services  assume responsibility for the system around 2013. Program officials said  they are incorporating actual cost data from the field as it becomes  available and updating O&S costs annually in order to give the services  the best data once the transfer takes place. Finally, the Army’s MQ-1C  program has updated its SAR O&S costs annually since 2009. Although  these costs were updated in 2010 for several reasons, including an  increase in the number of systems to be acquired, program officials said  they do not plan to update the program’s O&S costs annually.

Officials for the remaining programs in our sample, which updated their  O&S costs intermittently, gave various reasons for updating their  program’s SAR O&S costs when they did. While one program updated  the SAR as required to reflect the O&S costs estimated for an acquisition  decision, other programs in our sample chose to update the costs after  they developed estimates to reflect changes to the acquisition program  (e.g., changes in production quantity or schedule), to incorporate actual  O&S costs that are considerably different than previously estimated, or to  comply with guidance not related to the SAR. For example, the Navy’s   V-22 program office updated the O&S costs in the 2009 SAR because  actual O&S costs incurred after the program’s initial operational capability  in 2007 for the Marine Corps and 2009 for the Air Force were significantly  higher than had been anticipated in the program’s most recent cost  estimate. Prior to the 2009 update, the V-22 was reporting costs based on  the estimate completed for an acquisition decision in 2005. The V-22  program office, in conjunction with U.S. Naval Air Systems Command,  plans to review the program’s O&S costs annually and update the SAR as  necessary until the program stops reporting SARs. According to officials,  the final deliveries of the V-22 are scheduled for 2020. As another  example, the Joint Primary Aircraft Training System program updated  O&S costs in the 2010 SAR after reporting the same costs since 2001.  According to officials, an updated program office cost estimate was  developed to comply with a policy from the program’s major command  that cost estimates be updated annually.

DOD acquisition best practices and GAO-identified cost-estimating best  practices call for maintaining updated estimates of program costs.  According to the Defense Acquisition Guidebook, although a DOD or  service cost estimate is required at milestone reviews, it is a good  practice for this estimate, or at least its underlying program office cost  estimate, to be updated more frequently, usually annually. Updated  estimates should be useful in program management and financial  management throughout the life of the program. GAO-identified best  practices call for continual updates of cost estimates to keep them  relevant and current, as most programs do not remain static, especially  those in development. Routine updates that incorporate actual data result  in higher-quality estimates as the program matures. Further, updating the  cost estimate provides an accuracy check, defense of the estimate over  time, shorter estimate preparation times, and archived cost and technical  data for use in future estimates.

In accordance with the SAR statute, DOD’s implementation guidance  states that if a program has an antecedent system, then O&S costs and  assumptions should be submitted for the antecedent system. We found  that program offices, however, were inconsistent in reporting on  antecedent system costs, with many not reporting any O&S cost data.  Specifically, 57 (68 percent) of the 84 programs reporting O&S costs in  the 2010 SAR did not report O&S costs for an antecedent system. It was  unclear from the SARs how program offices had identified an antecedent  system or whether, in cases where no antecedent system costs were  included, the program offices had determined that an antecedent system  did not exist.

Nine of the 15 programs in our sample did not report O&S costs for an  antecedent system in the 2010 SAR. Officials from these program offices  provided various reasons for not reporting antecedent system costs,  including that the system was the first of its type or not intended to  replace any other system, that the system had advanced capabilities or  no other system was similar enough for comparison, and that the system  was replacing several legacy systems. As an example, Joint MRAP  program officials said other systems, such as the High Mobility  Multipurpose Wheeled Vehicle, were too different for cost comparisons.  As another example, the Navy Multiband Terminal program began  reporting in the 2006 SAR and has never reported antecedent O&S costs.  According to program officials, an antecedent system was not identified  because the system was replacing several legacy weapon systems.  However, during a joint OSD/Navy SAR review meeting in March 2011,  the program office was instructed to list two systems as antecedent  systems in the 2010 SAR. While the program identified the Super High  Frequency and Navy Extremely High Frequency Satellite programs as  antecedent systems in the O&S section of its SAR, it also stated that  program costs for these systems were not readily available.

The SAR statute requires that all program costs be reported, regardless  of funding source or management control. However, we found that of the  95 major weapon systems that had passed milestone B and reported  costs in the 2010 SAR, 11 (12 percent) did not identify any O&S costs in  their SARs. The 11 programs, as of December 2010, accounted for a total  estimated investment of $56.7 billion (fiscal year 2011 dollars) for  research and development, procurement, military construction, and  acquisition-related operation and maintenance (see table 2). Most of the  programs that did not report O&S costs were modifications to other  weapon systems but qualify as major defense acquisition programs  based on their procurement or research and development costs.

Eight of the programs that did not report O&S costs are major  modifications to, or subsystems of, Air Force weapon systems. When  we asked why O&S costs were not reported, officials from six Air Force  programs said they did not report O&S costs in the 2010 SAR because  they do not fund or track these costs. For example, officials for two  programs associated with the C-5 aircraft explained that all O&S fleet  costs are the responsibility of another entity, the System Program  Manager at Warner Robins Air Logistics Center in Georgia. Program  officials for the other two Air Force programs, the B-2 Radar  Modernization Program and B-2 Extremely High Frequency Satellite  Communications program, told us that these modification programs were  expected to reduce O&S costs and they could not input cost reductions  into DOD’s Defense Acquisition Management Information Retrieval  system, the database that maintains SAR data. In contrast to these  modification programs, the Air Force’s C-130 Avionics Modernization  Program did report total estimated O&S costs in the 2010 SAR.

According to officials, one of the remaining three programs—the Army’s  Apache Block IIIB—was not required to report O&S costs in the SAR, as  approved by the Defense Acquisition Executive. The other two  programs are the Chemical Demilitarization-Assembled Chemical  Weapons Alternatives, and the Chemical Demilitarization-U.S. Army  Chemical Materials Agency. According to the SAR for each program,  O&S costs are reported in other sections of the reports. For example,  program officials told us that O&S costs for the Assembled Chemical  Weapons Alternatives program are captured in research, development,  test, and evaluation costs. According to program officials, the Chemical  Demilitarization program is a one-of-a-kind national environmental and  safety program that is unlike weapon systems that report SARs. Further,  officials said that the two programs have not separately reported any O&S  costs since they were designated major defense acquisition programs in  1994.

section_title: Program Offices Sometimes Provided Unreliable Life-Cycle O&S Cost Estimates in Their SARs
paragraphs: SARs are intended to provide Congress with authoritative program  information on the cost, schedule, and performance of major weapon  systems, but we found that some programs submitted unreliable O&S  cost data. More specifically, our review of SAR reports for the 15  programs in our sample identified inaccurate cost estimates and other  errors in SARs submitted in 2007, 2009, and 2010. (As noted earlier,  DOD did not submit SARs in 2008.) While some of the program offices  told us specific reasons for the errors, others did not provide an  explanation.

Based on our analysis of O&S cost data reported in the SAR compared  with the underlying cost estimates and other information provided by the  program offices, we found that 7 of the 15 programs reported inaccurate  O&S costs in one or more of the three annual SARs.

The F-35 Joint Strike Fighter program office underreported the  average cost per flying hour for the aircraft in the 2010 SAR. The  average, steady-state O&S cost per flying hour was reported as  $16,425 (fiscal year 2002 dollars). Program officials told us that the  number of aircraft used in the estimate for the Air Force’s inventory  was not accurate and the estimate also did not project for future cost  growth above inflation. The estimate included approximately 528 extra  aircraft that when calculating the average cost per flying hour, resulted  in higher flight hours and lower average costs per hour. Further,  according to the SAR, some of the F-35’s O&S costs were  intentionally excluded from the estimate to enable comparison with  the antecedent system, the F-16 C/D. Costs for support equipment  replacement, modifications, and indirect costs were removed from the  F-35’s cost per flying hour since they were not available for the F-16  C/D. Officials calculated that the revised cost per flying hour for the F- 35 was $23,557 (fiscal year 2002 dollars), or 43 percent higher, after  including the excluded costs, projecting for future cost growth above  inflation, and correcting the number of aircraft. However, they noted  that the total O&S life-cycle cost reported in the SAR for the F-35 was  accurate because it was calculated separately from the average cost  per flying hour.

The Navy Multiband Terminal program office underreported estimated  life-cycle O&S costs in the 2010 SAR. The program reported $219.1  million in total O&S costs but excluded an additional $591.3 million for  externally funded depot-level repairables ($148.4 million) and military  personnel ($442.9 million), which were included in a 2010 service cost  estimate. Therefore, only 27 percent of the program’s estimated  total O&S costs were reported in the 2010 SAR. Program officials  stated that these costs are not under the control of the program office  and should not be reported in the SARs. However, the SAR statute  states that full life-cycle costs, including O&S costs, should be  reported without regard to funding source or management control.

The Air Force Joint Primary Aircraft Training System program office  underreported O&S costs in the 2007 and 2009 SARs, both of which  were based on a 2001 service cost estimate. The program, which  includes the T-6 aircraft and a ground-based training system, reported  total O&S costs of $9.4 billion (fiscal year 2002 dollars) in both SARs  but excluded $2.1 billion (fiscal year 2002 dollars)—or about 18  percent—of O&S costs for the program’s ground-based training  system. Program officials have reported the same O&S costs since  the annual 2002 SAR. The program, which updated its O&S estimate  in 2011, included these costs in the total O&S costs reported in the  2010 SAR.

The Army’s High Mobility Artillery Rocket System program office  overstated O&S costs in the program’s 2007, 2009, and 2010 SARs.  Although program office estimates were provided to us for the 3  years, the estimates did not match the costs reported in the SARs.  The O&S costs reported in 2007 were higher than the estimate by  $11.1 million (fiscal year 2003 dollars), and the $988 million (fiscal  year 2003 dollars), reported in both 2009 and 2010, was higher than  the estimates by about $300 million (fiscal year 2003 dollars), or about  43 percent. Program officials told us that the costs had been reported  incorrectly in each year.

The JTRS Handheld, Manpack, and Small Form Fit program  underreported total O&S costs in the annual 2007 SAR. The SAR  stated that the O&S costs had been updated, but the O&S costs were  unchanged from prior annual SARs. Program officials also provided  us with an estimate that matched the numbers reported in the 2007  SAR. When asked why the costs had not changed, program officials  stated that while the costs for procurement and research,  development, test, and evaluation were correctly updated in 2007, the  O&S costs were not. They explained that the reported costs of $4.9  billion (fiscal year 2004 dollars) should have been higher by $120  million (fiscal year 2004 dollars), but they did not provide us the  estimate on which that higher amount was based.

The Air Force’s Navstar GPS program, as noted earlier, did not report  a life-cycle cost estimate in the annual SARs from 2007 through 2010.  For example, according to the 2010 SAR, the O&S costs reported  were based on funding for fiscal years 2008 through 2016. Program  officials confirmed that the O&S amounts reported included actual  funding for the current year and funding from the Air Force’s budget  system for the remaining years. However, even this amount—about  $469 million (fiscal year 2000 dollars) in 2010, for example—was  significantly understated. According to program officials, the amount  reported in the SAR is only 60 percent of the program’s actual  requirements of approximately $782 million—a difference of $313  million—and the program has historically been funded to 90 percent of  requirements with supplemental funds. However, this was not noted in  the SARs.

The FBCB2 program underreported total O&S costs in the annual  2007 through 2010 SARs. As explained earlier, reported O&S costs  were estimated for the program’s final acquisition milestone, full-rate  production, in 2004. In subsequent years, however, the program’s  procurement quantities increased and were about 305 percent higher  in the 2010 SAR than the amount used to develop the estimate.  Further, total O&S costs of $468 million (fiscal year 2005 dollars)  reported in the SARs were $129 million less than the $596.2 million  estimated in 2004. Officials initially indicated that some of the  estimated O&S costs were likely included with the program’s  acquisition costs in the SAR, but they were unable to reconcile the  costs in the two documents.

We also found examples of inaccuracies in other data reported in the  O&S cost section of the SARs. For example, the 2010 SAR for the Joint  MRAP states that the program’s O&S costs were reviewed by CAPE in  2010, but program officials and prior-year SARs stated that the review  actually occurred in 2008. Further, neither CAPE nor the program office  was able to provide any record of the 2008 review. As another example,  the 2010 SAR for the F-22 indicates that the reported O&S costs were  based on a 2004 acquisition decision estimate that was updated with  analyses from 2010 to bring the estimate in line with the current approved  F-22 production program and operational concepts. However, the O&S  costs reported are identical to those reported in the 2009 SAR, which  states it was updated based on analyses from 2009.

Implementation of the GAO-identified best practices already discussed  could improve the reliability of O&S costs reported in the SARs. Together,  the best practices work to provide more assurance that the correct  information is reported. For example, routinely updating O&S cost  estimates—and related SAR data—will likely require more frequent  changes to the reported cost data. Therefore, it is less probable that an  error or omission will be regularly reported. In addition, as noted earlier,  comparing a program’s cost estimate with an independent cost estimate,  and explaining any significant differences, could help decision makers  monitor the reasonableness of the reported data. Finally, tracking O&S  costs over time, by presenting the current year and prior-year program  cost estimates and explaining significant differences, would also help to  test the reasonableness of reported costs.

section_title: DOD Could Improve O&S Cost Reporting with Detailed Implementation Guidance and a More Effective SAR Review Process
paragraphs: DOD’s reports to Congress on estimated weapon system O&S costs were  often inconsistent and sometimes unreliable due to a lack of detailed  implementation guidance for reporting these costs. In addition, DOD’s  process for reviewing the O&S cost sections of the SAR prior to final  submission did not provide assurance that the program offices reported  costs uniformly, to the extent practicable, and that these reported costs  were reliable. In the absence of improvements to the SAR guidance and  to DOD’s review process, deficiencies in reporting estimated life-cycle  O&S costs are likely to continue.

DOD’s existing implementation guidance collectively provides minimal,  and in some areas conflicting, instructions for O&S cost reporting. For  example, the guidance does not identify which cost estimate or estimates  should be used to report O&S costs when more than one estimate is  available. Often multiple cost estimates are prepared by the program  office, the service, and CAPE to support acquisition decisions. Further,  DOD officials stated that O&S costs reported in the SAR should be  updated only at acquisition milestones. Because many years may pass  between these milestones, however, reported O&S costs may become  outdated, no longer reflecting the status of the current acquisition  program.

DOD’s guidance also provides very little detail on how program offices  should discuss assumptions underlying the cost estimate. DOD’s draft  SAR policy, for example, only mentions several assumptions for  consideration, such as operating tempo, expected reliability and  maintainability of the system, the maintenance concept, and manning and  logistics policies, and does not provide specific examples. In addition, the  statutory SAR requirement to report all program costs, regardless of  funding source or management control, is reflected in none of DOD’s  SAR implementation guidance; it is reflected in training course materials  on using the Defense Acquisition Management Information Retrieval  system. Finally, DOD’s draft SAR policy provides conflicting instructions  on cost reporting for antecedent systems. The draft policy states that  antecedent costs should be reported “whenever those costs have  previously been developed.” However, in the appendix, the draft guidance  states that O&S costs will be reported for antecedent systems “when the  replacement system is required to report O&S costs.” DOD officials could  not explain the reason for this variance in the guidance.

While some program offices we contacted indicated that DOD’s  implementation guidance on reporting O&S costs in the SAR was  sufficient, officials from several program offices in our sample indicated  that more detailed guidance would be helpful when they prepare their  annual SAR submissions. These officials stated that there was minimal  guidance provided on what should be included in the O&S narrative and  that there needed to be more consistency in SAR O&S reporting.  Additionally, they explained that the current guidance is vague, unclear,  open to interpretation, and does not provide useful information or  examples for how programs should be reporting these costs. Officials  from one program also stated that there is no direction on the comparison  of program costs to the antecedent system’s costs, so the approach to  making this comparison is open to interpretation. They noted that the  guidance does not specify whether the program office should alter the  weapon system’s O&S costs to enable a true comparison with the costs  for the antecedent system, or whether the weapon system’s O&S costs  should be reported without modification. Finally, while several program  offices told us that the Defense Acquisition University provides useful  training on acquisition reporting in general, they said that the materials  should be more readily available as program representatives could not  always attend the class and that the O&S section of the SAR was not  covered sufficiently.

The SAR data submitted by program offices are subject to multiple  reviews within the military services and by OSD, but this review process  has not provided assurance that O&S costs are reported consistently and  reliably. Although our review did not include a full evaluation of DOD’s  SAR review process, OSD officials explained that once they receive the  SAR submissions, there is a relatively short amount of time to review the  SAR O&S data. For example, according to the SAR review schedule, the  Office of the Assistant Secretary of Defense (Logistics and Materiel  Readiness) usually has about a week to review the O&S cost  submissions. We also noted that “SAR review guidance” that is included  with the annual memorandum on preparing SARs does not provide  additional direction to the program offices on what to include in their O&S  cost submissions. In some cases, the annual memorandum is less  specific than the draft SAR policy. The deficiencies in DOD’s  implementation guidance likely hinder the effective review of SAR O&S  cost information at all levels.

The department’s emphasis on weapon system O&S costs has been  increasing in recent years, but the primary focus continues to be on  acquisition costs. According to OSD acquisition officials, the SAR started  as—and is still often viewed as—primarily an acquisition report. This  perspective was reflected in comments from some program officials. For  example, officials at one program office told us that, due to a constantly  changing acquisition program, their time was largely spent on estimating  acquisition costs. Another program office noted that the focus of the SAR  statute was acquisition costs and that O&S costs will vary based on  emerging needs. Several other programs indicated that O&S cost  estimating was not particularly useful, as their systems had not yet  entered into production or sustainment, and actual cost data were either  not yet available or could not be obtained by the program office. Finally,  other program offices stated that since they do not fund the support of the  weapon system, the O&S cost estimates should be done by the  organizations responsible for providing this funding.

section_title: Conclusions
paragraphs: Without more consistent and reliable reporting of estimated weapon  system O&S costs, Congress and senior DOD officials may have limited  visibility of information needed to effectively oversee the full life-cycle  costs associated with weapon system acquisitions. Improvements in the  reporting of these data could provide a more complete picture of the  potential total financial commitment being made to these systems over a  period lasting many decades. SAR cost estimates are reported early  during acquisition, when there is the greatest chance for managing or  reducing future O&S costs. By facilitating inquiries on changes from prior  cost estimates and cost drivers, such information could affect acquisition  investment decisions and result in tradeoffs that otherwise might not be  considered. Furthermore, improvements to SAR reporting would be  consistent with a provision in the National Defense Authorization Act for  Fiscal Year 2012 directing DOD to take actions aimed at better  assessing, managing, and controlling weapon system O&S costs.

section_title: Recommendations for Executive Action
paragraphs: To improve visibility over estimated life-cycle O&S costs during weapon  system acquisition, we recommend that the Secretary of Defense take the  following two actions.

First, we recommend that the Secretary of Defense direct the Under  Secretary of Defense for Acquisition, Technology and Logistics to revise  DOD’s guidance for implementing statutory SAR requirements. The  revisions, at a minimum, should provide additional detail on the following  areas:  the explanatory information that should be included in the O&S  narrative, including the specific assumptions underlying the cost  estimate; the source to be used as the basis for reported O&S cost estimate  information, especially when more than one source is available (such  as a program office cost estimate, service cost estimate, and CAPE  independent cost estimate); a consistent unit of measure for reporting average costs over time by  commodity type—or other designated weapon system group—as  agreed to by OSD and the services;  criteria for identifying an antecedent system and reporting on the  results of the cost comparison in the SAR; and reporting O&S costs for major modifications to existing weapon  systems.

In revising the guidance, the Under Secretary of Defense should  incorporate best practices for preparing and presenting cost estimates,  including: a comparison of current-year to prior-year O&S cost estimates; the  identification of cost drivers that resulted in changes in these  estimates, if significant; and the level of detail that should be reported; a comparison of the reported cost estimate with the most recent  independent cost estimate, along with an explanation of any  significant differences between the two estimates; and

The frequency with which O&S costs reported in the SAR should be  updated, including guidance on what changes in the program’s status  should trigger an update.

Second, we also recommend that the Secretary of Defense direct that the  Under Secretary of Defense for Acquisition, Technology and Logistics, in  conjunction with the Secretaries of the Army, the Air Force, and the Navy,  evaluate the current review process, identify any weaknesses, and  institute corrective actions as needed to provide greater assurance that  estimated life-cycle O&S costs included in the SAR reports submitted by  program offices consistently follow the implementation guidance,  including any revisions to the guidance as described above, and report  reliable cost data. As part of this evaluation, DOD should consider  whether additional steps are necessary for the department to enhance the  emphasis placed on reporting estimated life-cycle O&S costs in the SAR.

section_title: Agency Comments and Our Evaluation
paragraphs: DOD provided comments on a draft of this report. In its comments, DOD  agreed with both of our recommendations. The department’s written  comments are reprinted in appendix III. DOD also provided technical  comments that we have incorporated into this report where appropriate.

In concurring with our first recommendation to revise DOD’s guidance for  implementing statutory SAR requirements, DOD noted that the focus of  the SAR has always been primarily on acquisition rather than  sustainment. DOD further stated that Congress, in requiring DOD to add  O&S costs to the SAR report, did not intend for DOD to develop O&S  costs for each submission but to report the latest available estimate for  the program. Our report recognizes that the development of new O&S  cost estimates is not required for each annual SAR submission. However,  these costs represent a significant proportion of a system’s total costs  over its life cycle. Moreover, we found that the timing of updates to the  O&S costs reported in the SAR varied widely, as DOD has not identified  what changes in a program's status—other than established acquisition  milestones, which can be many years apart—should trigger such  updates. We also continue to believe that DOD needs to clearly identify  the source and date of the O&S cost estimate data reported in the SAR.  Our recommendations reflect these and other weaknesses in the current  reporting of O&S costs.

DOD’s comments identified actions it plans to take to implement our  recommendations. DOD stated that it will expand and update its current  guidance for the O&S cost section of the SAR, as contained in the  Defense Acquisition Guidebook. DOD plans to make revisions specifically  with regard to assumptions and ground rules (e.g., the source and date of  the estimate reported); a consistent unit of measure for reporting O&S  costs for each type of commodity; identifying, and reporting on,  antecedent systems; and reporting O&S costs for major modifications.  These planned revisions to the guidance are positive steps. We plan to  monitor DOD’s actions as part of our recommendation follow-up process.

Regarding other revisions to the guidance that we recommend to  incorporate best practices for O&S cost reporting, DOD stated that the  department is not yet in a position to add a credible O&S cost variance  analysis. Although DOD does not define what it means by "cost variance  analysis," it is reasonable to expect that such analysis would involve  comparing changes from a previous cost estimate and identifying any  significant cost drivers. DOD noted that it is implementing new O&S- related requirements from the National Defense Authorization Act for  Fiscal Year 2012, as well as previous requirements from the Weapon  Systems Acquisition Reform Act of 2009, including requirements that deal  with cost variance analysis. DOD stated that it is premature to determine  to what extent DOD's implementation of these requirements will affect the  reporting of O&S costs in the SAR. With these and other ongoing  activities related to the management and control of O&S costs, DOD  would prefer to defer these additional reporting requirements for the SAR  for now.

We are aware that DOD has a number of ongoing activities to improve  the management and control of O&S costs and must respond to several  new requirements, as stated in DOD's comments. For example, the O&S- related guidance required by the National Defense Authorization Act for  Fiscal Year 2012 must be issued within 180 days from the date the Act  was enacted, which was December 31, 2011. If such activities result in  improved visibility of O&S costs within the department, and DOD  coordinates these activities with efforts to improve O&S cost reporting in  the SAR, then we agree that it may be preferable to delay implementation  of the best practices we recommend in our report. However, we continue  to believe that these best practices, when implemented, will provide better  information on the current status and direction of long-term O&S costs  and thus can improve congressional oversight of weapon system costs.  Therefore, these elements of our recommendation remain valid.

DOD also concurred with our second recommendation to evaluate and  make any changes needed to strengthen its current process for reviewing  O&S cost reporting prior to submission of SARs to Congress. In its  comments, DOD cited actions it would take in the short term to improve  the review of O&S costs prior to submission of SAR reports at the end of  March 2012. DOD stated that the O&S cost section will be given  additional emphasis during this reporting period. Subsequently, DOD will  convene a joint OSD/DOD component working group that will evaluate  the current SAR review process, identify any weaknesses, and institute  corrective actions as needed to improve the data quality for the estimated  life-cycle O&S costs reported in the SAR. We believe these actions, when  implemented, will meet the intent of our recommendation.

We are sending copies of this report to interested congressional  committees; the Secretary of Defense; the Secretaries of the Army, the  Navy, and the Air Force; the Acting Under Secretary of Defense for  Acquisition, Technology and Logistics; and the Director, Office of  Management and Budget. In addition, the report will be available at no  charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions concerning this report, please  contact me at (404) 679-1808 or russellc@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. Key contributors are listed in appendix IV.

section_title: Appendix I: Scope and Methodology
paragraphs: To determine the extent to which the selected acquisition reports (SAR)  provide consistent and reliable operating and support (O&S) cost  estimate information that enables effective oversight of major weapon  system costs, we reviewed statutory requirements in 10 U.S.C. § 2432 for  reporting weapon system life-cycle costs in the SARs, as well as  Department of Defense (DOD) implementation guidance for the SAR. We  also reviewed DOD guidance for preparing weapon system O&S costs  and GAO-identified cost-estimating best practices to identify the scope  and nature of cost estimate information needed for effective program  management and oversight.

We interviewed and obtained documentation from DOD and military  service officials responsible for weapon system acquisition, logistics, and  cost analysis to understand DOD’s approach and process for reporting  O&S cost estimates in the SARs. Offices we contacted included the  following:

Office of the Under Secretary of Defense for Acquisition, Technology  and Logistics

Office of the Director, Acquisition Resources and Analysis

Office of the Deputy Assistant Secretary of Defense for Materiel

Office of the Director, Cost Assessment and Program Evaluation

Office of the Deputy Assistant Secretary of the Army, Cost and

Naval Center for Cost Analysis

Air Force Cost Analysis Agency  We obtained SARs for all 95 weapon systems that reported a December  2010 SAR. These reports were contained in the Defense Acquisition  Management Information Retrieval system, which is a web-based system  used within DOD to collect and maintain SAR information submitted by  program offices. We determined that the data in this system accurately  reflected information submitted by weapon system program offices and  therefore were sufficiently reliable for the purposes of our analysis. After  determining that a total of 84 of the 95 weapon systems included O&S  costs in their December 2010 SARs, we analyzed the annual SARs that  were submitted for these systems between 2005 through 2010.Specifically, we analyzed the SARs to determine the types and scope of  explanatory information included in the O&S narrative accompanying the  cost estimate data; the source of the O&S cost estimate cited as the basis  for the reported costs; the units of measure used to present O&S costs;  the frequency that O&S costs were updated from year to year; and the  extent to which O&S costs for antecedent systems were reported. We  compared the SARs across each of these categories to determine the  extent to which information was reported consistently across all 84  weapon systems.

From the population of 84 weapon systems that included O&S cost  estimates in the 2010 SARs, we selected a sample of 15 weapon   We designed the sample to ensure that a  systems for further analysis. range of weapon systems were represented based on commodity type  and service responsible for managing the program. We selected three or  four weapon systems per service and at least one commodity type within  each service for a total sample size of 15.distribution of weapon systems’ total costs across our sample selection in  terms of both dollars and the upper and lower 50 percent of weapon  systems that reported O&S costs in the 2010 SAR. We determined that  the sample contained an adequate mix of high- and low-dollar weapon  systems for our purposes. The results from this nonprobability sample  cannot be used to make inferences about all major weapon systems  because the sample may not reflect all characteristics of the population.  The 15 programs in our sample are shown in table 3.

section_title: Appendix II: DOD’s Guidance Implementing the Statutory SAR Requirements
paragraphs: This appendix provides additional information on DOD’s guidance that  implements the statutory SAR requirements in 10 U.S.C. § 2432.

DOD has issued various guidance documents that implement the  statutory SAR requirements. DOD Instruction 5000.02, which addresses  the operation of the defense acquisition system, includes guidance on  SARs that is similar to the basic statutory requirements in 10 U.S.C. §  2432. The guidance, for example, states that SARs should be submitted  at program initiation (normally milestone B except for some ship  programs) or at the time that the program is designated as a major  defense acquisition program. It reiterates that programs shall report  annually, with the exception of quarterly reports that are required when  acquisition costs increase or schedules slip. Further, the instruction  requires the submission of quarterly SARs after the program rebaselining  that occurs after a major milestone decision (i.e., milestone C or  milestones B and C for some ship programs).

Another source of guidance on SAR reporting is the Defense Acquisition   which describes discretionary best practices for acquisition Guidebook, professionals to consider while meeting various reporting requirements  throughout the acquisition process. The guidebook contains a section  summarizing the statutory requirements for SAR content and submission  and reiterates that a full life-cycle analysis of costs should be reported for  programs, including each evolutionary increment, as available, and for  antecedent programs, if applicable. states that assumptions underlying the estimate should be included.  Operating tempo, expected reliability and maintainability of the system,  maintenance concept, and manning and logistics policies are provided as  examples of the estimate assumptions that should be included in the  SAR. Finally, the draft policy states that programs should report the total  estimated O&S costs, and estimate assumptions, for an antecedent  system if one has been identified and these costs were previously  developed for that system.

Each year the Under Secretary of Defense for Acquisition, Technology  and Logistics issues a memorandum to the military services that provides  guidance for preparing the annual SARs, including instructions for  programs that have reached milestone B and are required to report O&S  costs. For fiscal years 2007, 2009, and 2010, this annual guidance  states that programs should report total estimated O&S costs in both  constant and then-year dollars, and that the assumptions that formed the  basis of the estimate and the date of the estimate should be included.  Further, programs should report an average unit of measure (e.g.,  average annual cost per squadron, average annual cost per system) for  the O&S costs of both the current weapon system and the antecedent  system in constant dollars. If there is no antecedent system, this should  be stated in the narrative of the O&S cost section. If there is an  antecedent system but the data are not currently available, the  antecedent system should be identified in the narrative, along with a  statement that the required data are not available (e.g., “the O&S costs  for the antecedent system are not currently available, but will be provided  in the next SAR”). Finally, programs should explain in the narrative how  the average annual costs were calculated using the estimated O&S cost  total.

Under Secretary of Defense for Acquisition, Technology and Logistics, SAR Data Entry  Instructions (draft) (Nov. 5, 2010).

SAR training on using this system. According to Office of the Secretary of  Defense (OSD) officials, the primary class, Acquisition Reporting for  Major Defense Acquisition Programs and Major Automated Information  Systems, is usually offered in January and October. During the 4-day  class, participants receive step-by-step instruction on report preparation  using the system’s web application. The training materials include basic  SAR O&S cost reporting information. For example, estimate assumptions  should be reported, calculation of average costs from total O&S costs  should be provided, and costs should always be updated at major  acquisition milestones. The training materials reiterate that costs should  include both direct and indirect costs, regardless of funding source or  management control.

section_title: Appendix III: Comments from the Department of Defense

section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact name above, the following staff members made  key contributions to this report: Tom Gosling, Assistant Director; Kristine  Hassinger; Susannah Hawthorne; Charles Perdue; Janine Prybyla;  William M. Solis; and Erik Wilkins-McKee.


Question-summary annotations: questions: question: What is the quality of the DOD's reports to Congress on estimated weapon system costs?
answer: DOD’s reports to Congress on estimated weapon system O&S costs are often inconsistent and sometimes unreliable, limiting visibility needed for effective oversight of these costs.

question: In what ways did GAO find inconsistencies with the reporting?
answer: The SAR statute requires that life-cycle cost reporting for major weapon systems be uniform, to the extent practicable, across the department, but GAO found a number of inconsistent practices in how program offices were reporting life-cycle O&S cost estimates in the SAR. Program offices were inconsistent in (1) the explanatory information they included with the cost estimates; (2) the source of the cost estimate they cited as the basis for the reported costs; (3) the unit of measure they used to portray average costs; (4) the frequency with which they updated reported costs; and (5) the reporting of costs for an antecedent system being replaced by the new weapon system.
child_questions: question: To what extent was not reporting O&S costs for an antecedent system a problem?
answer: For example, 35 (42 percent) of the 84 programs that reported O&S costs in the 2010 SAR did not cite a source of these data, contrary to DOD’s guidance, and 57 (68 percent) of the programs did not report O&S costs for an antecedent system.
child_questions: question: In what other ways did program offices not incorporate best practices in their O&S cost reporting?
answer: Also, O&S cost submissions in the SAR did not always incorporate best practices for presenting cost estimates, such as tracking cost changes over time and identifying cost drivers. In addition, 11 systems did not provide O&S cost estimates in the 2010 SAR.

questions: question: What is the purpose of SARS?
answer: Although SARs are intended to provide Congress with authoritative program information on major weapon systems, 7 of the 15 sample programs GAO reviewed submitted unreliable O&S cost estimate data in the 2007, 2009, or 2010 SARs.
child_questions: question: How was the information different Air Force programs submitted unreliable?
answer: For example, an Air Force program underreported O&S costs by $2.1 billion (fiscal year 2002 dollars), or 18 percent. An Air Force program reported current and projected funding for the program rather than estimated life-cycle O&S costs.

question: How was the information a Navy program office submitted unreliable and problematic?
answer: For example, one Navy program office underreported O&S costs in the SAR and explained that it excluded certain costs that were not under its control, such as externally funded spare parts and military personnel. However, excluding such costs is contrary to the SAR statute.

questions: question: Why are the DOD O&S reports unreliable?
answer: DOD’s reports to Congress on estimated weapon system O&S costs were often inconsistent and sometimes unreliable due to a lack of (1) detailed implementation guidance for reporting these costs and (2) an effective process for reviewing the O&S cost sections of the SAR before final submission to Congress. The guidance also does not incorporate some of the best practices GAO has identified for presenting cost estimates.
child_questions: question: What aspects of cost reporting has DOD provided minimal guidance on?
answer: DOD’s guidance collectively provides minimal instructions for O&S cost reporting.

question: What potential benefits will arise from improving the reporting of O&S costs?
answer: Improved reporting of O&S costs in the SAR could help to place more emphasis on assessing, managing, and controlling long-term weapon system O&S costs.

question: What review processes is SAR data subject to?
answer: Further, although the SAR data submitted by program offices are subject to multiple reviews within the military services and by the Office of the Secretary of Defense, this review process has not provided assurance that O&S costs are reported consistently and reliably.

question: Without improvements to the SAR guidance, what will continue?
answer: In the absence of improvements to the SAR guidance and to the review process, deficiencies in reporting O&S costs are likely to continue.
child_questions: question: Improved reporting of O&S costs in the SAR could help with what?
answer: Improved reporting of O&S costs in the SAR could help to place more emphasis on assessing, managing, and controlling long-term weapon system O&S costs.

questions: question: Why have Congress and DOD placed their attention on O&S costs?
answer: With the nation facing fiscal challenges and the potential for tighter defense budgets, Congress and the Department of Defense (DOD) have placed more attention on controlling the billions of dollars spent annually on weapon system operating and support (O&S) costs.
child_questions: question: What is included in these costs?
answer: These costs include, costs for repair parts, maintenance, and personnel, and account for about 70 percent of the total costs of a weapon system over its life cycle.

question: What is SAR?
answer: The selected acquisition report (SAR) is DOD’s key recurring status report on the cost, schedule, and performance of major defense acquisition programs and is intended to provide authoritative information for congressional oversight of these programs.
child_questions: question: Why is oversight of O&S costs important?
answer: Oversight of O&S costs is important because many of the key decisions affecting these life-cycle costs are made during the acquisition process.

question: How did GAO conduct its review of O&S cost estimates?
answer: GAO reviewed weapon system O&S cost estimates that DOD submits in the SAR. Specifically, GAO determined the extent to which the SARs provide consistent and reliable O&S cost estimate information that enables effective oversight of these weapon system costs. To conduct its review, GAO analyzed SAR data for 84 major systems that submitted O&S cost estimates in the 2010 SAR and selected a nonprobability sample of 15 systems for more in-depth review.