Missile Defense: Actions Needed to Improve Information for Supporting Future Key Decisions for Boost and Ascent Phase Elements
Published: 2007-04-17T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Defense (DOD) has spent about $107 billion since the mid-1980s to develop a capability to destroy incoming ballistic missiles. DOD has set key decision points for deciding whether to further invest in capabilities to destroy missiles during the initial phases after launch. In March 2006, DOD issued a report on these capabilities in response to two mandates. To satisfy a direction from the House Appropriations Committee, GAO agreed to review the report. To assist Congress in evaluating DOD's report and preparing for future decisions, GAO studied the extent to which DOD (1) analyzed technical and operational issues and (2) presented complete cost information. To do so, GAO assessed the report's methodology, explanation of assumptions and their effects on results, and whether DOD followed key principles for developing life-cycle costs.
section_title: What GAO Found
paragraphs: The report DOD's Missile Defense Agency (MDA) submitted to Congress in March 2006 included some useful technical and operational information on boost and ascent phase capabilities by describing these elements, listing upcoming decision points, and discussing geographic areas where boost and ascent elements could intercept missiles shortly after launch. However, the information in the report has several limitations because the analysis did not involve key DOD stakeholders such as the services and combatant commands in preparing the report and did not clearly explain modeling assumptions and their effects on results as required by relevant research standards. MDA's report states that, at this time, some data is limited, and operational concepts that discuss operations from forward locations have not been fully vetted with the services and combatant commands. However, the report did not explain how each element's performance may change if developing technologies do not perform as expected. Also, it did not address the challenges in establishing bases at the locations cited or provide information on the quantity of each element required for various deployment periods. Moving forward, DOD has an opportunity to involve stakeholders in analyzing operational and technical issues so that senior DOD and congressional leaders will have more complete information on which to base upcoming program decisions following key tests in 2008 and 2009 for the Kinetic Energy Interceptor and Airborne Laser boost and ascent phase programs. MDA's report provided some cost estimates for developing and fielding boost and ascent phase capabilities, but these estimates have several limitations and will require refinement before they can serve as a basis for DOD and congressional decision makers to compare life-cycle costs for the elements. MDA's report states that there is uncertainty in estimating life-cycle costs because the elements are early in development. However, based on a comparison of the estimates in the report with key principles for developing life-cycle cost estimates, GAO found that MDA's estimates did not include all cost categories, including costs to establish and sustain operations at U.S. bases and at forward overseas operating locations. Also, MDA's estimates did not calculate costs based on realistic quantities of each element the combatant commanders or services would need to conduct the mission. Finally, MDA did not conduct a sensitivity analysis to assess the effect of key cost drivers on total costs. MDA officials stated that further analysis of the costs for each element along with measures to assess their confidence would help to better inform DOD and congressional decision makers in making investment decisions following key tests in 2008 and 2009.

section_title: Letter
paragraphs: The new security environment includes some states, such as North Korea,  that are aggressively pursuing development of long-range missiles and  weapons of mass destruction. In 2002, President Bush directed that the  Department of Defense (DOD) proceed with plans to develop and put in  place an initial set of ballistic missile defense capabilities beginning in  2004. DOD has spent about $107 billion since the mid-1980s to develop a  capability to destroy incoming ballistic missiles in all phases of their flight,  including the initial phases after launch, called the boost and ascent  phases. The purpose of boost and ascent phase elements of a ballistic  missile defense system (BMDS) is to engage incoming missiles before  release of warheads or countermeasures. DOD is developing three BMDS  boost and ascent phase capabilities: Airborne Laser (boost phase only),  Kinetic Energy Interceptor (boost and ascent phase), and the Aegis  Ballistic Missile Defense element that will fire the Standard Missile-3 block  2A (ascent phase) from BMDS-capable Aegis surface combatants.  According to the Missile Defense Agency (MDA), the agency responsible  for developing an integrated ballistic missile defense system, DOD plans to  spend $1.6 billion for the Airborne Laser; $800 million for the Kinetic  Energy Interceptor; and $200 million for the Aegis Standard Missile-3  Block 2A during fiscal years 2006–08. The long-term costs for these  elements will be much greater if DOD continues their development and  fielding. DOD projects the initial boost and ascent phase capabilities will  not be available until about 2014 through 2020. Congress has raised  questions about the affordability of pursuing both the Airborne Laser and  Kinetic Energy Interceptor programs.
Over the next few years, DOD and Congress will be asked to make  important program and investment decisions on boost and ascent phase  capabilities—at a time when the federal government is facing a large and  growing fiscal imbalance. Further, we have identified the competing  demands of investing in transformational capabilities while continuing to  invest in legacy systems as one of the most significant national defense  challenges for the 21st century. MDA has established key decision points  at which it will decide whether to continue to invest in certain elements of  the integrated system. For example, one key decision point is the booster  flight test for the Kinetic Energy Interceptor element, scheduled for 2008,  at which time a decision will be made on the program’s future. A key  decision point for the Airborne Laser is the lethal shootdown  demonstration scheduled for 2009. Finally, in 2009, MDA plans a motor  test for the Standard Missile-3 block 2A.
In March 2006, MDA issued a report on boost and ascent phase capabilities  in response to two mandates—one in the House Appropriations  Committee Report on the Defense Appropriations Bill for Fiscal Year 2006  and one in the National Defense Authorization Act for Fiscal Year 2006.  The House Appropriations Committee directed the Secretary of Defense to  conduct a study on boost and ascent phase intercepts including an  assessment of operational capabilities, quantities of operational assets  required for various deployment periods, basing options, and an  assessment of life-cycle costs. Life-cycle costs are the total cost to the  government for a program over its full life, consisting of research and  development, production, operations, maintenance, and disposal costs and  are helpful in assessing whether a system’s cost is affordable. The  Department of Defense Authorization Act for Fiscal Year 2006 required the  Secretary of Defense to conduct an assessment of U.S. missile defense  programs that are designed to provide capability against threat ballistic  missiles in the boost/ascent phase. The purpose of the assessment was to  compare and contrast capabilities, asset requirements, and the costs for  making the boost and ascent phase programs operational. MDA, on behalf  of DOD, prepared one report to satisfy both requirements and sent the  report to all four defense committees on March 30, 2006. According to  MDA officials, that report was prepared specifically to satisfy the  congressional requirements, not for use at any key decision point.
To satisfy a direction from the House Appropriations Committee report,  we agreed to review the DOD report including assessing the report’s  methodologies, assumptions, completeness, and results. This report is our  assessment of the March 2006 MDA report and how DOD can build on this  information to support future key decision points. Accordingly, we  assessed the extent to which (1) information reported by DOD includes  analysis of technical and operational issues and whether any additional  information is needed to support future decision making and (2) DOD  presented cost information to Congress that is complete and transparent.  To assess the technical and operational information, we analyzed the  extent to which the MDA report clearly explained technical maturity,  modeling assumptions and their effect on results, and involved  stakeholders in analyses of operational issues—steps which contribute to  a sound and complete quality study and are embodied in relevant generally  accepted research standards. To assess the cost information in the March  2006 report, we compared MDA’s cost data with key principles compiled  from various DOD and GAO sources that describe how to develop  accurate and reliable life-cycle cost estimates which are to include  development, production, and all direct and indirect operating and support  costs. We discussed the data and results of this comparison with DOD  officials and identified some limitations, which we discuss in this report.  We conducted our work between June 2006 and February 2007 in  accordance with generally accepted government auditing standards. See  appendix I for a more complete description of our scope and  methodology.
This report is one in a series we have issued on ballistic missile defense  (see the list of related GAO products at the end of this report). Some of  these reports have focused on assessments of program goals and progress  in developing each element. For example, our March 2006 report found  that MDA made progress in the development and fielding of some  elements but that time pressures caused MDA to stray from a knowledge- based acquisition strategy. This is significant because straying from a  knowledge-based approach opens the door to greater cost and  performance risks by not having the right information available for  decision makers at the right time. Another report focused on DOD’s  planning to operate BMDS. We reported in May 2006 that DOD has not  established criteria that would have to be met before declaring BMDS  operational and that the Future Years Defense Program (FYDP), a major  source of budget information, does not provide complete and transparent  data on ballistic missile defense operational costs.
subsections: section_title: Background
paragraphs: In 2002, the Secretary of Defense created MDA to develop an integrated  system that would have the ability to intercept incoming missiles in all  phases of their flight. In developing BMDS, MDA is using an incremental  approach to field militarily useful capabilities as they become available.  MDA plans to field capabilities in 2-year blocks. The configuration of a  given block is intended to build on the work completed in previous blocks.  For example, Block 2006 is intended to build on capabilities developed in  Block 2004, and is scheduled to field capabilities during calendar years  2006–07.
The integrated BMDS is comprised of various elements, three of which are  intended to intercept threat missiles in their boost or ascent phase. Table 1  below describes each of these elements and shows the MDA projected  dates for key decision points, initial capability, and tested operational  capability.
During the past year, Congress requested additional information and  analyses on the boost and ascent phase elements from DOD. Specifically,  House Report 109-119 on the Department of Defense Appropriations Bill  for Fiscal Year 2006 directed the Secretary of Defense to conduct a study  to review the early engagement of ballistic missiles to include boost and  ascent phase intercepts and submit the report to the congressional defense  committees. The report was to include, but not be limited to    an assessment of the operational capabilities of systems against  ballistic missiles launched from North Korea or a location in the Middle  East against the continental United States, Alaska, or Hawaii;    an assessment of the quantity of operational assets required for  deployment periods of 7 days, 30 days, 90 days, and 1 year;    basing options; and    an assessment of life-cycle costs to include research and development  efforts, procurement, deployment, operating, and infrastructure costs.
In addition, the National Defense Authorization Act for Fiscal Year 2006  required the Secretary of Defense to assess missile defense programs  designed to provide capability against threat ballistic missiles in the  boost/ascent phase of flight. The purpose of this assessment was to  compare and contrast    capabilities of those programs (if operational) to defeat ballistic  missiles from North Korea or a location in the Middle East against the  continental United States, Alaska, or Hawaii; and    asset requirements and costs for those programs to become  operational with the capabilities referred to above.
MDA, on behalf of DOD, prepared one report to satisfy both of the above  requirements and sent the report to all four defense committees on March  30, 2006. The report included technical, operational, and cost information  for each of the three boost and ascent phase BMDS elements. The  remainder of this report discusses our assessment of the MDA report and  how DOD can build on this information to support future key decision  points.
section_title: MDA’s March 2006 Report Includes Some Useful Information but Has Several Limitations
paragraphs: MDA’s March 2006 report to Congress included some useful technical and  operational information on boost and ascent phase capabilities. However,  the information in the report has several limitations—such as not  including stakeholders in the analysis or explaining how assumptions  affect results. Moving forward, DOD can enhance its ability to make  informed decisions at future key decision points by including stakeholders  DOD-wide in conducting analyses to provide complete technical and  operational information. Otherwise, senior DOD and congressional  decision makers may be limited in their ability to effectively assess the  technical progress and operational effects of proceeding with one or more  boost and ascent phase element.
subsections: section_title: Report Provided Some Basic Information about Boost and Ascent Phase Elements
paragraphs: The March 2006 report to Congress contained some useful technical and  operational information for Congress. For example, the report included a  detailed description of the three boost and ascent phase elements, which  could be useful for those unfamiliar with these elements. Additionally, the  report listed upcoming knowledge points where DOD will review the  progress MDA has made toward developing each of the boost and ascent  phase elements. Further, the report discussed geographic areas where  boost and ascent phase elements could intercept missiles shortly after  launch based on desired technical capabilities. Also, MDA used a model  to assess the desired capabilities of each BMDS element for the March  2006 report to Congress. Further, the modeling environment was used for  several past BMDS analyses and the results were benchmarked against  other models. Finally, MDA performed a sensitivity analysis that compared  how the results in the modeling changed when different assumptions for  targets’ propellants, ascent times, hardness levels, and burn times were  used. To provide context, the report explained that the boost and ascent  phase elements are in the early stages of development and that the  operational concepts are not yet mature.
section_title: MDA’s Analyses Have Limitations but DOD Can Improve Information to Support Future Decisions
paragraphs: The information in the March 2006 report has several limitations because  the analyses did not involve stakeholders and did not clearly explain  modeling assumptions and their effects on results as identified by relevant  research standards. The relevant research standards and our prior work  have shown that coordination with stakeholders from study design  through reporting, and clearly explained assumptions and their effects on  results, can enable DOD officials to make fully informed program  decisions. As a result, the March 2006 report presents an incomplete  picture of technical capabilities, such as development challenges to be  overcome in order to achieve desired performance, and it does not clearly  explain the effects of operational assumptions, such as basing locations,  asset quantities, and base support requirements. As a step in the right  direction, MDA stated that it plans to develop criteria to assess the  boost/ascent phase elements at major decision points in a process  involving the combatant commands. Although MDA officials told us that  they consult stakeholders in a variety of forums other than the March 2006  report, they did not clearly state whether or how the services or other  DOD stakeholders would be involved in developing criteria for key  decision points or the extent to which their analyses would include  information on technical and operational issues.
subsections: section_title: Data Collection and Analysis for the March 2006 Report Did Not Involve Key DOD Stakeholders
paragraphs: MDA’s analyses did not involve soliciting or using information from key  DOD stakeholders such as the services, combatant commands, and joint  staff from study design through reporting. For example, officials from the  Office of the Secretary of Defense for Program Analysis and Evaluation  and the Defense Intelligence Agency stated there were areas where  additional information would have improved the fidelity of the results.  First, the officials stated that there is uncertainty that the boost and ascent  phase elements would achieve their desired capabilities within the  timeframe stated in the report. Second, officials from both organizations  stated that the report could have been enhanced by presenting different  views of the type and capability of threats the United States could face and  when these threats could realistically be expected to be used by  adversaries. Third, officials from the Office of the Secretary of Defense for  Program Analysis and Evaluation said that the MDA report did not  distinguish between countermeasures that could be used in the near term  and countermeasures that may be more difficult to implement. MDA  officials said that they worked with the Office of the Secretary of Defense  for Program Analysis and Evaluation in conducting analyses before they  began work on the March 2006 report. MDA also stated that it discussed  the draft March 2006 report with Office of the Secretary of Defense for  Program Analysis and Evaluation officials and included some of their  comments in the report’s final version. However, without communication  with stakeholders from study design through reporting, MDA may not have  had all potential inputs that could have affected how the type, capability,  and likelihood of countermeasures to the boost and ascent phase elements  were presented in its report.
Additionally, MDA did not solicit information from the services, combatant  commands, or Joint Staff regarding operational issues that could have  affected information about basing and the quantities of elements that  could be required to support operations. Although the elements have to be  located in close proximity to their intended targets, and the report  discusses placing the elements at specific forward overseas locations, the  report does not include a basing analysis explaining what would need to  be done to support operations at these locations. Specifically, the report  did not include any discussion of the infrastructure or security/force  protection that will be needed for the BMDS elements. Although the report  mentions some support requirements—such as the Airborne Laser’s need  for unique maintenance and support equipment and skilled personnel to  maintain the laser—the report did not fully explain how these support  requirements would be determined, who would provide or fund them, or  explain the operational effect if this support is not provided. For instance,  without an adequate forward operating location, the boost and ascent  phase elements would have to operate from much further away which  would significantly limit the time an element is in close proximity to  potential targets. Developing such information with the services, Joint  Staff, and combatant commands could provide a much more complete  explanation of operational issues and challenges. The services typically  perform site analyses to ascertain what support is needed for a new  weapon system at either a U.S. or overseas location. This comprehensive  analysis examines a range of issues from fire protection to security, to  infrastructure, to roads and airfields. In addition, U.S. Strategic Command  and service officials told us that this type of support must be planned for  in advance when adding a new system to any base, either in the United  States or a forward location.
MDA also did not involve stakeholders in assessing the quantities of each  element for deployment periods of 7 days, 30 days, 90 days, and 1 year.  The report stated that limited data exist at this time for a full assessment  of this issue, and service, Joint Staff, and MDA officials acknowledged that  the quantities of each element used in the report are MDA assumed  quantities. Service, Joint Staff, and U.S. Strategic Command officials stated  that they have not completed analyses to assess quantities the warfighters  may require. We understand that operational concepts will continue to  evolve and could affect required quantities. However, stakeholders such as  the services, Joint Staff, or combatant commands could have assisted  MDA in assessing potential quantities required for various deployment  periods. In addition, MDA did not solicit information from the services,  Joint Staff, or combatant commands to determine if those organizations  were conducting force structure analyses for the boost and ascent phase  elements. We learned that the Navy had done a preliminary analysis in July  2005 and that the Joint Staff has begun a capabilities mix study and both  include, in part, an analysis of quantities. Thus, in preparing for future  decision points, MDA’s analysis could be strengthened by including  stakeholders to leverage other analyses. For example, MDA could have  presented a range of scenarios to show how the quantities required to  intercept adversary missiles could vary depending upon the number of  sites covered and whether continuous, near-continuous, or sporadic  coverage is provided.
section_title: Assumptions and Their Effects on Data Presented in the MDA Report Were Not Clearly Explained
paragraphs: The March 2006 report to Congress did not clearly explain the  assumptions used in the modeling of the BMDS elements’ capabilities and  did not explain the effects those assumptions may have had on the results.  First, the model inputs for the technical analysis assumed desired rather  than demonstrated performance, and the report does not fully explain  challenges in maturing technologies or how these performance predictions  could change if the technologies are not developed as desired or assumed.  For example, although the model MDA used is capable of showing  different results based on different performance assumptions, the report  did not explain how the number of successful intercepts may change if  less than 100 percent of the desired technical capabilities are developed as  envisioned. Thus the results represent the best expected outcome. Second,  the report does not explain the current status of technical development or  the challenges in maturing each element’s critical technologies as desired  or assumed in the report. DOD best practices define Technology  Readiness Levels on a scale of 1–9, and state which level should be  reached to progress past specific program decision points. However, the  March 2006 report does not explain the current Technology Readiness  Level for any of the boost and ascent phase elements’ critical technologies  or the extent to which the technology has to mature to attain the  performance assumed in the report. For example, the report does not  explain that some of the technologies for the Airborne Laser have to  improve between 60 percent and 80 percent and the report does not  discuss any of the challenges MDA faces in doing so.
section_title: DOD Can Significantly Improve Life-Cycle Cost Estimates to Support Future Investment and Budget Decisions for Boost and Ascent Phase Capabilities
paragraphs: The March 2006 report to Congress provides cost estimates for each of the  boost and ascent phase capabilities; however, the cost estimates in the  report have several limitations that raise questions about their usefulness.  We compared the report’s cost estimates with various DOD and GAO  sources that describe key principles for developing accurate and reliable  life-cycle cost estimates. Based on our analysis, we found that MDA did  not include all cost categories, calculate costs based on warfighter  quantities, and did not conduct a sensitivity analysis to assess the effects  of cost drivers. Moreover, although MDA’s report acknowledges  uncertainty in the cost estimates, the report does not fully disclose the  limitations of the cost estimates. DOD can significantly improve the  completeness of and confidence in cost estimates for boost and ascent  phase capabilities as it prepares for future investment and budget  decisions. For example, although DOD did not have its cost estimate for its  March 2006 report independently verified because doing so would have  taken several months, MDA officials agreed that independently verified  cost estimates will be critical to support major decision points for boost  and ascent phase capabilities. In addition, as these capabilities mature,  MDA officials agreed that showing cost estimates over time and  conducting uncertainty analyses will be needed to support key program  and investment decisions.
subsections: section_title: Cost Estimates Did Not Include All Cost Categories
paragraphs: The cost estimates provided in the MDA report included some  development, production, and operations/support costs for each boost and  ascent phase element but were not fully developed or verified according to  key principles for developing life-cycle cost estimates. Life-cycle costs are  the total cost to the government for a program over its full life, including  the costs of research and development, investment, operating and support,  and disposal.
Based on our comparison of the life-cycle cost estimates in the report with  key principles for developing life-cycle cost estimates, we found that the  estimates were incomplete in several ways. First, the cost estimates did  not include all cost categories, such as costs to establish and sustain  operations at U.S. bases. Instead, MDA assumed that the elements would  be placed at existing bases with sufficient base support, infrastructure and  security; however, some of these costs such as infrastructure could be  significant. For example, an MDA planning document cited about $87  million for infrastructure costs to support a ground-based BMDS element  (Terminal High Altitude Area Defense). Army officials confirmed that  training facilities, missile storage buildings, and a motor pool were built at  a U.S. base specifically to support this element and it is likely that similar  infrastructure would be needed to support the land-based Kinetic Energy  Interceptor. Additionally, MDA’s cost estimates did not include costs to  establish and sustain operations at forward overseas locations, even  though the report states that the elements will have to be located in close  proximity to their targets, and the operational concepts for Kinetic Energy  Interceptor and Airborne Laser, although in early development, state that  these elements will be operated from forward locations. Again, these are  important factors to consider—the Airborne Laser operational concept  and the MDA report acknowledge that unique support will be required to  support operations at any forward location for the Airborne Laser such as  chemical facilities, unique ground support equipment, and maintenance.  Service, Joint Staff, and U.S. Strategic Command officials also said that  these elements would have to be located forward and could be used as a  strategic deterrent in peacetime.
Second, the production and operating cost estimates were not based on  warfighter quantities, that is, quantities of each element that the services  and combatant commands may require to provide needed coverage of  potential targets. MDA assumed a certain quantity of each element. For  example, MDA officials told us that they assumed 96 Standard Missile-3  block 2A missiles because, at the time MDA prepared the report, they  planned to buy 96 block 1A missiles developed to intercept short-range  ballistic missiles. However, MDA did not solicit input from the services,  Joint Staff, or combatant commands on whether they had done or begun  analyses to determine element quantities.
Third, MDA did not conduct a sensitivity analysis to identify the effects of  cost drivers. A sensitivity analysis is a way to identify risk by  demonstrating how the cost estimates would change in response to  different values for specific cost drivers. Therefore, a sensitivity analysis  should be performed when developing cost estimates, and the results  should be documented and reported to decision makers. This means, for  example, that MDA could have computed costs with and without  significant categories of costs such as forward bases to identify the effect  that adding forward bases would have on operating costs. The House  Armed Services Committee report on the National Defense Authorization  Bill for Fiscal Year 2006 recognized that operational capabilities and costs  must be taken into account when making decisions on future funding  support. Finally, the cost estimates did not estimate costs over time—a  process known as time phasing—which can assist decision makers with  budgetary decisions. The MDA report showed an annual cost estimate but  did not state for how many years the development, production, and  operating costs may be incurred. Although MDA officials stated they did  not prepare time-phased cost estimates in order to prepare the report to  Congress in a timely manner, they agreed that showing cost estimates over  time would be important information to support investment decisions at  key decision points.
section_title: Cost Estimates Were Not Assessed or Independently Verified for MDA’s 2006 Report but MDA Officials See Need to Do So for Key Decision Points
paragraphs: Key principles for developing life-cycle cost estimates also include two  steps for assessing the confidence of cost estimates. However, MDA did  not take these steps to assess the confidence of the estimates reported in  March 2006. First, the Missile Defense Agency did not conduct a risk  analysis to assess the level of uncertainty for most of the cost estimates in  the MDA report. Risk and uncertainty refer to the fact that, because a cost  estimate is a prediction of the future, it is likely that the estimated cost will  differ from the actual cost. It is useful to perform a risk analysis to  quantify the degree of uncertainty in the estimates. By using standard  computer simulation techniques, an overall level of uncertainty can be  developed for cost estimates. In contrast, MDA officials told us that they  could only provide a judgmental confidence level for the most of the cost  estimates. Second, MDA did not have the cost estimates in the report  verified by an independent organization such as DOD’s Cost Analysis  Improvement Group because doing so would have taken several months.  However, MDA officials agreed that independent verification of cost  estimates would be important information to support investment decisions  at key decision points. According to the key principles that we have  identified, all life-cycle cost estimates should be independently verified to  assure accuracy, completeness, and reliability. MDA has recognized the  value in independently developed cost estimates. In 2003, MDA and the  Cost Analysis Improvement Group developed a memorandum of  understanding that said, in part, the Cost Analysis Improvement Group  would develop independent cost estimates for the approved BMDS and its  elements as appropriate during development in anticipation of transition  to production, but MDA officials said that little work was completed under  this agreement, which has expired.
Developing complete cost estimates in which decision makers can have  confidence is important since life-cycle cost estimates usually form the  basis for investment decisions and annual budget requests. Specifically,  life-cycle cost estimates that include all cost categories, show costs over  time, include warfighter quantities, include an assessment of cost drivers,  and are independently verified are important because accurate life-cycle  cost estimates can be used in formulating funding requests contained in  the President’s Budget and DOD’s future funding plan, the Future Years  Defense Program (FYDP) submitted to Congress. Therefore, there is a  need for DOD to provide transparent budget and cost planning  information to Congress. In May 2006, GAO reported that the FYDP, a  major source of budget and future funding plans, does not provide  complete and transparent data on ballistic missile defense operational  costs because the FYDP’s structure does not provide a way to identify and  aggregate these costs. It is important that Congress has confidence in  boost and ascent phase estimates because Congress has indicated that it is  concerned with the affordability of pursuing both the Airborne Laser and  Kinetic Energy Interceptor programs in parallel through 2008.
As we reported in 2003, DOD assumes increased investment risk by not  having information available for decision makers at the right time, and the  level of anticipated spending magnifies this risk. Otherwise, senior DOD  and congressional decision makers may be limited in their ability to assess  the relative cost of the elements if all cost categories are not included and  cost drivers are not identified. Considering competing demands, this could  also limit Congress’s ability to consider investment decisions or evaluate  whether continued expenditures are warranted. MDA officials stated that,  in developing the cost estimates for the March 2006 report, they decided  not to follow some of the key principles for developing life-cycle cost  estimates such as time phasing and independent verification of the cost  estimates in order to complete the report in a timely manner. However, the  officials also agreed that these key principles are important in developing  complete, accurate, and reliable life-cycle cost estimates for supporting  investment decisions at key decision points. Therefore, in the future, when  preparing cost estimates to be used in support of key decision points,  MDA could provide decision makers with more complete, accurate, and  reliable cost estimates by better adhering to key principles for developing  life-cycle cost estimates.
section_title: Conclusions
paragraphs: Our review of MDA’s March 2006 report on boost and ascent phase  elements identified a number limitations but helps to illuminate the kind of  information that DOD and congressional decision makers will need  following upcoming tests for boost and ascent phase elements. We  recognize that the March 2006 report was prepared in response to  congressional direction rather than to support program decisions. We also  recognize that, at the time of MDA’s report, these elements were early in  their development and information was incomplete and changing. Thus,  the focus of our analysis was to identify additional information that could  enhance future program and investment decisions.
In particular, the House Armed Services Committee has raised questions  about the affordability of pursuing both the Kinetic Energy Interceptor and  the Airborne Laser in parallel through the projected knowledge point  demonstrations, which are now scheduled for 2008 and 2009 respectively.  It is important that these decisions be both well-informed and transparent  because of the long-term funding consequences. DOD and congressional  decision makers’ ability to assess which elements can be fully developed,  integrated, and operated relative to the others will be enhanced if they  have the benefit of information based on more rigorous analysis than that  contained in MDA’s March 2006 report. Looking forward, as DOD  strengthens its analyses to support future key decisions, DOD and  congressional decision makers will be able to use more complete  information to assess force structure, basing, support, and infrastructure  requirements, as well as technical maturity, budget requests, and FYDP  spending plans, in deciding whether or not to continue developing one,  two, or all three boost and ascent phase elements and in what quantities.
section_title: Recommendations for Executive Action
paragraphs: To provide decision makers with information that enables them to clearly  understand the technical progress and operational implications of each  boost and ascent phase element and make fully informed, fact-based,  program decisions at future key decision points, we recommend that the  Secretary of Defense direct the Under Secretary of Defense for  Acquisition, Technology, and Logistics to take the following actions to  support key decision points for the BMDS boost and ascent phase  elements:  Include all DOD stakeholders (including services, combatant  commands, Joint Staff) in developing and analyzing operational issues  regarding what is needed to support operations at U.S. bases and  potential forward locations, including basing assessments, force  structure and quantity requirements, infrastructure, security/force  protection, maintenance, and personnel.
Provide specific information on the technical progress of each element.
Specifically, the analysis should explain current technical maturity  versus desired technical maturity and capabilities of all major  components and subsystems, reasonable model inputs on element  performance, and provide a clear explanation of assumptions and their  effect on results.
Use the results of these analyses at each key decision point.
To provide decision makers with complete and reliable data on the costs  of each boost/ascent phase BMDS element to enhance investment and  budget decisions, we recommend that the Secretary of Defense take the  following actions:    Direct the Under Secretary of Defense for Acquisition, Technology, and  Logistics to require MDA to prepare and—to support key decision  points—periodically update a full life-cycle cost estimate for each  boost/ascent phase element, in accordance with key principles for  developing accurate and reliable life-cycle cost estimates, that includes  all operational costs, including costs to establish and sustain  operations at U.S. bases and forward locations, and that is based on  warfighter quantities, includes sensitivity analyses, and reflects time  phasing.
Direct an independent group, such as the Cost Analysis Improvement  Group, to prepare an independent life-cycle cost estimate for each  capability at each key decision point.
Direct MDA and services to report independently verified life-cycle  cost estimates along with budget requests and FYDP funding plans for  each boost/ascent phase element.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, DOD agreed with our  recommendations regarding the need for analysis of technical progress  and operational issues to support key boost and ascent phase element  decision points. DOD also agreed that an independent life-cycle cost  estimate may be needed to inform some key decision points while they  may not be needed at other decision points. However, DOD did not agree  to prepare and periodically update full life-cycle cost estimates for each  boost and ascent phase element to support key decision points, and report  independently verified life cycle cost estimates with budget requests and  FYDP funding plans. As discussed below, we continue to believe our  recommendations have merit and that DOD should take the additional  actions we have recommended to provide a rigorous analytical basis for  future decisions, enhance the transparency of its analyses, and increase  accountability for key decisions that could involve billions of dollars. The  department’s comments are reprinted in their entirety in appendix II.
DOD agreed with our recommendations that all DOD stakeholders be  included in developing and analyzing operational issues, that specific  information on technical progress be provided to explain current versus  desired capabilities, and that the results of both analyses be used at key  decision points. DOD stated in its comments that officials from MDA, the  military departments, the combatant commanders, and other organizations  are collaborating to develop an operational BMDS. Moreover, the annual  BMDS Transition and Transfer Plan is coordinated with the service  secretaries and other stakeholders and serves as a repository for plans,  agreements, responsibilities, authorities, and issues. DOD also stated that  key program decisions are and will continue to be informed by detailed  technical analysis, including assessment of element technical maturity.  However, DOD did not clearly explain how future decision making will be  enhanced or how analyses of operational issues will be conducted if, as in  the case of the Kinetic Energy Interceptor, DOD has not assigned a service  responsibility for operating the element once it is developed. We continue  to believe that DOD and congressional decision makers will need more  complete information on support requirements at upcoming decision  points as well as a clear comparison of current versus desired technical  capabilities in deciding whether or not to continue developing one, two, or  all three boost and ascent phase elements.
Regarding our recommendations to improve cost estimates used to  support key investment decisions, DOD partially concurred that  independent life-cycle cost estimates may be required to inform some key  decision points but stated that other key decision points may not.  However, DOD did not agree that it should routinely prepare and  periodically update a full life-cycle cost estimate for each boost and ascent  phase element. DOD said that it continuously assesses all aspects of its  development efforts and will direct an independent evaluation of life-cycle  costs for boost and ascent phase elements if circumstances warrant or if  MDA’s Director declares an element mature enough to provide a militarily  useful capability. However, if, as DOD’s comments suggest, such costs are  not assessed until circumstances warrant or MDA’s Director declares an  element mature enough to provide a militarily useful capability, these  costs may not be available early enough to help shape important program  and investment decisions and consider trade-offs among elements.  Moreover, DOD’s Operating and Support Cost Estimating Guide,  published by the Cost Analysis Improvement Group, states that when the  Cost Analysis Improvement Group assists the Office of the Secretary of  Defense components in their review of program costs, one purpose is to  determine whether a new system will be affordable to operate and  support. Therefore, such analysis must be done early enough to provide  cost data that will be considered in making a decision to field, produce, or  transition an element. We continue to believe our recommendation has  merit because the development of life-cycle cost estimates that include  potential operations and support costs would improve the information  available to decision makers and increase accountability for key decisions  that could involve billions of dollars at a time when DOD will likely face  competing demands for resources.
Finally, DOD did not agree to report independently verified life-cycle cost  estimates along with budget requests and FYDP funding plans for each  boost and ascent phase element. DOD stated that operations and support  segments of the budget are organized by functional area rather than by  weapon system and are dependent on operations and support concepts of  the employing military department. DOD further stated that development  of total life-cycle cost estimates for operational BMDS capabilities requires  agreement between MDA and the lead military department on roles and  responsibilities for fielded BMDS capabilities that transcend the annual  transition planning cycle but serve as a basis for budget submittals. We  recently reported that MDA enjoys flexibility in developing BMDS but this  flexibility comes at the cost of transparency and accountability. One  purpose of cost estimates is to support the budget process by providing  estimates of the funding required to efficiently execute a program.  Also,  independent verification of cost estimates allows decision makers to  gauge whether the program is executable. Thus, cost estimating is the  basis for establishing and defending budgets and is at the heart of the  affordability issue. This principle is stated in DOD procedures which  specify that when cost results are presented to the Office of the Secretary  of Defense Cost Analysis Improvement Group, the program office- developed life-cycle cost estimate should be compared with the FYDP and  differences explained. Therefore, we continue to believe that our  recommendation has merit because, without an independent cost estimate  that can be compared to budget requests and FYDP funding plans,  congressional decision makers may not have all the necessary information  to assess the full extent of future resource requirements if the boost and  ascent phase capabilities go forward, or assess the completeness of the  cost estimates that are in the budget request and FYDP funding plans.
We are sending copies of this report to the Secretary of Defense; the  Commander, U.S. Strategic Command; the Director, Missile Defense  Agency; Chairman, the Joint Chiefs of Staff; and the Chiefs of Staff of the  Army, Navy, and Air Force. We will make copies available to others upon  request. In addition, the report will be available at no charge on the GAO  Web site at http://www.gao.gov.
If you or your staff have any questions, please call either Janet St. Laurent  on (202) 512-4402 or Paul Francis on (202) 512-2811. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on the  last page of this report. Staff members who made key contributions to this  report are listed in appendix III.
section_title: Appendix I: Scope and Methodology
paragraphs: During this review, we focused on assessing the analytical approach the  Missile Defense Agency (MDA) used to develop its March 2006 report to  Congress, as well as the methodology for developing the cost estimates for  each of the three Ballistic Missile Defense System (BMDS) boost and  ascent phase elements.
To assess the extent to which the Department of Defense (DOD) is  developing technical and operational information useful for oversight and  that will support decision making at key points, we compared the  analytical approach DOD used to develop its March 2006 report with  generally accepted research standards that are relevant for defense studies  such as this, that define a sound and complete study, and that cover all  phases of a study—design, execution, and presentation of results. The  following were our sources for these standards:    GAO, Government Auditing Standards: 2003 Revision, GAO-03-673G  (Washington, D.C.: June 2003);    GAO, Designing Evaluations, GAO/PEMD-10.1.4 (Washington, D.C.:    GAO, Dimensions of Quality, GAO/QTM-94-1 (Washington, D.C.:    RAND Corporation, RAND Standards for High-Quality Research and  Analysis (Santa Monica, Calif.: June 2004);    Air Force, Office of Aerospace Studies, Analysts Handbook: On  Understanding the Nature of Analysis (January 2000);    Air Force, Office of Aerospace Studies, Air Force Analysis Handbook,  A Guide for Performing Analysis Studies: For Analysis of Alternatives  or Functional Solution Analysis (July 2004);    Department of Defense, DOD Modeling and Simulation (M&S)
Verification, Validation, Accreditation (VV&A), Instruction 5000.61  (Washington, D.C.: May 2003);    Department of Defense, Data Collection, Development, and  Management in Support of Strategic Analysis, Directive 8260.1  (Washington, D.C.: Dec. 2, 2003); and    Department of Defense, Implementation of Data Collection,  Development, and Management for Strategic Analyses, Instruction  8260.2 (Washington, D.C.: Jan. 21, 2003).
For a more complete description of these standards and how we identified  them, see GAO-06-938, appendix I. In applying these standards, we  focused on the extent to which stakeholders were involved in study design  and analysis as well as the extent to which assumptions were reasonable  and their effects on results were clearly explained. We assessed MDA  briefings that explained the modeling used for the technical analysis  projecting the elements’ capabilities. To assess the basis for the assumed  performance parameters used to model each element’s performance, we  traced and verified a nonprobability sample of these parameters to their  source documentation and concluded that they were generally supported.  To evaluate the DOD report’s characterization of threats, we reviewed  Defense Intelligence Agency documents and discussed the type and  capability of threats and expected BMDS capabilities with officials from  the Office of the Secretary of Defense for Program Analysis and  Evaluation and the Defense Intelligence Agency. In addition, to gain an  understanding of the extent to which DOD has assessed warfighter  quantities for the boost and ascent phase elements, the development of  operational concepts, and operational implications of employing the boost  and ascent phase elements at forward locations, we evaluated DOD and  service guidance on assessing sites and support for new weapon systems  and discussed these issues with officials from the Joint Staff; U.S. Army  Headquarters and Space and Missile Defense Command; U.S. Strategic  Command; the office of the Chief of Naval Operations Surface Warfare  Directorate, Ballistic Missile Defense Division; Air Combat Command; and  the office of the Secretary of the Air Force for Acquisition, Global Power  Directorate. Finally, we discussed the results of all our analyses with  officials in the Joint Staff; U.S. Strategic Command; the Army’s Space and  Missile Defense Command; Office of the Secretary of Defense for  Acquisition, Technology, and Logistics; Missile Defense Agency; the office  of the Chief of Naval Operations Surface Warfare Directorate, Ballistic  Missile Defense Division; the office of the Secretary of the Air Force for  Acquisition Global Power Directorate; and Air Combat Command.
To assess the extent to which DOD presented cost information to  Congress that is complete and transparent, we first assessed how MDA  developed its estimates and then compared the method by which those  estimates were prepared to key principles compiled from various DOD and  GAO sources that describe how to develop accurate and reliable life-cycle  cost estimates to determine their completeness and the extent to which  DOD took steps to assess confidence in the estimates. The following were  our sources for compiling the cost criteria:    Department of Defense, Assistant Secretary of Defense (Program  Analysis and Evaluation), Cost Analysis Guidance and Procedures,  DOD Manual 5000.4-M (December 1992);    Department of Defense, Office of the Secretary of Defense Cost  Analysis Improvement Group, Operating and Support Cost Estimating  Guide (May 1992);    Department of Defense, Defense Acquisition University, Defense  Acquisition Guidebook (online at http://akss.dau.mil/dag);    Department of Defense, Defense Acquisition University, Introduction  to Cost Analysis (April 2006);    Air Force, Office of Aerospace Studies, Air Force Analysis Handbook:  A Guide for Performing Analysis Studies for Analysis of Alternatives or  Functional Solution Analysis (July 2004);    Air Force, Base Support and Expeditionary Site Planning, Air Force  Instruction 10-404 (March 2004); and    GAO, GAO Cost Assessment Guide (currently under development).
In addition, we met with DOD officials from MDA, U.S. Strategic  Command, the Joint Staff, Army, Navy and Air Force to determine the  extent to which they were involved in developing the cost estimates for  the DOD report. Finally, we corroborated our methodology and results  with officials from the Office of the Under Secretary of Defense, Program,  Analysis and Evaluation (Cost Analysis Improvement Group) and the  Office of the Under Secretary of Defense (Comptroller) and they agreed  that our methodology for examining the report’s cost estimates was  reasonable and consistent with key principles for developing accurate and  reliable life-cycle cost estimates. We identified some data limitations with  the cost estimates which we discuss in this report.
We provided a draft of this report to DOD for its review and incorporated  its comments where appropriate. Our review was conducted between June  2006 and February 2007 in accordance with generally accepted  government auditing standards.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Acknowledgments
paragraphs: In addition to the individuals named above, Barbara H. Haynes and  Gwendolyn R. Jaffe, Assistant Directors; Brenda M. Waterfield; Todd Dice;  Jeffrey R. Hubbard; Nabajyoti Barkakati; Hai V. Tran; Ron La Due Lake;  and Susan C. Ditto made key contributions to this report.
section_title: Related GAO Products
paragraphs: Defense Transportation: Study Limitations Raise Questions about the  Adequacy and Completeness of the Mobility Capabilities Study and  Report. GAO-06-938. Washington, D.C.: September 20, 2006.
Defense Management: Actions Needed to Improve Operational Planning  and Visibility of Costs for Ballistic Missile Defense. GAO-06-473.  Washington, D.C.: May 31, 2006.
Defense Acquisitions: Missile Defense Agency Fields Initial Capability  but Falls Short of Original Goal. GAO-06-327. Washington, D.C.: March 15,  2006.
Defense Acquisitions: Actions Needed to Ensure Adequate Funding for  Operation and Sustainment of the Ballistic Missile Defense System.  GAO-05-817. Washington, D.C.: September 6, 2005.
Military Transformation: Actions Needed by DOD to More Clearly  Identify New Triad Spending and Develop a Long-term Investment  Approach. GAO-05-962R. Washington, D.C.: August 4, 2005.
Military Transformation: Actions Needed by DOD to More Clearly  Identify New Triad Spending and Develop a Long-term Investment  Approach. GAO-05-540. Washington, D.C.: June 30, 2005.
Defense Acquisitions: Status of Ballistic Missile Defense Program in  2004. GAO-05-243. Washington, D.C.: March 31, 2005.
Future Years Defense Program: Actions Needed to Improve  Transparency of DOD’s Projected Resource Needs. GAO-04-514.  Washington, D.C.: May 7, 2004.
Missile Defense: Actions Are Needed to Enhance Testing and  Accountability. GAO-04-409. Washington, D.C.: April 23, 2004.
Missile Defense: Actions Being Taken to Address Testing  Recommendations, but Updated Assessment Needed. GAO-04-254.  Washington, D.C.: February 26, 2004.
Missile Defense: Additional Knowledge Needed in Developing System for  Intercepting Long-Range Missiles. GAO-03-600. Washington, D.C.: August  21, 2003.
Missile Defense: Alternate Approaches to Space Tracking and  Surveillance System Need to Be Considered. GAO-03-597. Washington,  D.C.: May 23, 2003.
Missile Defense: Knowledge-Based Practices Are Being Adopted, but  Risks Remain. GAO-03-441. Washington, D.C.: April 30, 2003.
Missile Defense: Knowledge-Based Decision Making Needed to Reduce  Risks in Developing Airborne Laser. GAO-02-631. Washington, D.C.: July  12, 2002.
Missile Defense: Review of Results and Limitations of an Early National  Missile Defense Flight Test. GAO-02-124. Washington, D.C.: February 28,  2002.
Missile Defense: Cost Increases Call for Analysis of How Many New  Patriot Missiles to Buy. GAO/NSIAD-00-153. Washington, D.C.: June 29,  2000.
Missile Defense: Schedule for Navy Theater Wide Program Should Be  Revised to Reduce Risk. GAO/NSIAD-00-121. Washington, D.C.: May 31,  2000.