Defense Acquisitions: Charting a Course for Improved Missile Defense Testing
Published: 2009-02-25T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Missile Defense Agency (MDA) has spent about $56 billion and will spend about $50 billion more through 2013 to develop a Ballistic Missile Defense System (BMDS). This testimony is based on two reviews GAO was directed to conduct in 2008. In addition to our annual review assessing the annual cost, testing, schedule, and performance progress MDA made in developing BMDS, we have also reported on MDA's targets program. In this testimony we discuss (1) the productivity of MDA's recent test program, (2) the consequences of the testing shortfalls, and (3) key factors that should be considered as MDA revises its approach to testing. GAO assessed contractor cost, schedule, and performance; tests completed; and the assets fielded during 2008. GAO also reviewed pertinent sections of the U.S. Code, acquisition policy, and the activities of a new missile defense board.
section_title: What GAO Found
paragraphs: The scale, complexity, cost and safety associated with testing the missile defense system constitute a unique challenge for MDA, test agencies and other oversight organizations. This challenge is heightened by the fact that missile defense assets are developed, produced, and fielded concurrently. Overall, during fiscal year 2008, testing has been less productive than planned. While MDA completed several key tests that demonstrated enhanced performance of BMDS, all elements of the system had test delays and shortfalls, in part due to problems with the availability and performance of target missiles. GMD in particular was unable to conduct either of its two planned intercept attempts in fiscal year 2008. While it did subsequently conduct one in December 2008, it was not able to achieve all primary objectives because the target failed to release its countermeasures. As a result, aspects of the fielded ground-launched kill vehicles may not be demonstrated since no more flight tests have been approved. Target missiles continue as a persistent problem in fiscal year 2008 as poor target performance caused several tests to either fail in part or in whole. Testing shortfalls have had several consequences. First, they have delayed the validation of models and simulations, which are needed to assess the system's overall performance. As a result, the performance of the fielded BMDS as a whole cannot yet be determined. Second, the production and fielding of assets has continued and in some cases has gotten ahead of testing. For example, enhanced Exoatmospheric Kill Vehicles will now be produced and delivered before they are flight tested. Third, MDA has relied on a reduced basis--fewer test, model, and simulation results--to declare capabilities as operational in the field. MDA has undertaken a three-phase review of the entire BMDS test program that involves identifying critical variables that have not been proven to date, determining what test scenarios are needed to collect the data, and developing an affordable, prioritized schedule of flight and ground tests. This review, as long as it continues to involve test and evaluation organizations, appears to offer a sound approach for closing the gaps that exist between testing, modeling, and simulation. Critical to being able to implement the approach will be addressing the factors that have limited the productivity of the current test approach, such as the availability and performance of targets. An additional consideration in a new testing approach must be to ensure that assets are sufficiently tested before they are produced and fielded. An important consideration in this regard is for modeling, simulation, and testing events to be re-synchronized so that they properly inform decisions on producing, fielding, and declaring assets operational. Contingency plans could then be formed for adjusting the pace of these decisions should shortfalls occur in modeling, simulation, or testing. Because MDA has indicated implementation will take time, managing the transition may need to include reassessing the ambitious fiscal year 2009 test plan. In the mean time, MDA will have to be prudent in making decisions to produce and field assets.

section_title: Letter
paragraphs: I am pleased to be here today to discuss the future of the Missile Defense  Agency’s (MDA’s) testing program.
MDA has been charged with developing and fielding the Ballistic Missile  Defense System (BMDS), a system expected to be capable of defending  the United States, deployed troops, friends, and allies against ballistic  missiles of all ranges in all phases of flight. In fulfilling this charge, MDA  placed an initial set of missile defense components in the field in  December 2005.
The National Defense Authorization Acts for fiscal years 2002, 2007 and  2008 mandated that we prepare annual assessments of MDA’s ongoing  cost, schedule, testing, and performance progress. In March 2009, we plan  to issue our report covering MDA’s progress toward achieving its goals  during fiscal year 2008 as well as its efforts to improve transparency,  accountability, and oversight. Additionally, in September 2008, we issued a  report on MDA’s Target Program. My statement today will focus on the  testing-related issues covered in both reports. We conducted these  performance audits from February 2008 to February 2009 in accordance  with generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on our  audit objectives.
subsections: section_title: Background
paragraphs: The Missile Defense Agency’s mission is to develop an integrated and  layered BMDS to defend the United States, its deployed forces, allies, and  friends. In order to meet this mission, MDA is developing a highly complex  system of systems—land, sea and space based sensors, interceptors and  battle management. Since its initiation in 2002, MDA has been given a  significant amount of flexibility in executing the development and fielding  of the BMDS. To enable MDA to field and enhance a missile defense  system quickly, the Secretary of Defense in 2002 delayed the entry of the  BMDS program into the Department of Defense’s traditional acquisition  process until a mature capability was ready to be handed over to a military  service for production and operation. Therefore, the program concurrently  develops, tests and fields assets. This approach helped MDA rapidly  deploy an initial capability. On the other hand, because MDA can field  assets before all testing is completed, it has fielded some assets whose  capability is uncertain.
Because MDA develops and fields assets continuously, it combines  developmental testing with operational testing.  In general, developmental  testing is aimed at determining whether the system design will satisfy the  desired capabilities; operational testing determines whether the system is  effective, survivable, and suitable in the hands of the user.  MDA conducts  testing both on the ground and in flight.  The most complex of these is an  end-to-end flight test that involves a test of all phases of an engagement  including detecting, tracking and destroying a target with an interceptor  missile.  An end-to-end intercept involves more than one MDA element.   For example, a recent intercept test involved a target flown out of Kodiak,  Alaska, tracked by the AN/TPY-2 radar located in Alaska, and the Beale  upgraded early warning radar located in California, the Sea-based X-band  radar and an Aegis radar located at different points in the Pacific. All of  the radars communicated with fire control centers in Alaska to guide an  interceptor launched from California to hit the target over the Pacific  Ocean.
Due to the complexity, scale, safety constraints, and cost involved, MDA is  unable to conduct a sufficient number of flight tests to fully understand  the performance of the system. Therefore, MDA utilizes models and  simulations, anchored by flight tests, to understand both the  developmental and operational performance of the system.  To ensure  confidence in the accuracy of modeling and simulation the program goes  through a process called accreditation. The models are validated  individually using flight and other test data and accredited for their  intended use. Models and simulations are used prior to a flight test to  predict performance, the flight test is then run to gather data and verify the  models, and then data is analyzed after the flight and reconstructed using  the models and simulations to confirm their accuracy.
MDA intends to group these models into system-level representations  according to user needs. One such grouping is the annual performance  assessment, a system-level end-to-end simulation that assesses the  performance of the BMDS configuration as it exists in the field. The  performance assessment integrates element-specific models into a  coherent representation of the BMDS. Fundamentally, performance  assessments anchored by flight tests are a comprehensive means to fully  understand the performance capabilities and limitations of the BMDS.
In addition to testing, modeling and simulation, and performance  assessments, MDA also has a formal process for determining when a  newly fielded asset or group of assets can be declared operational—that is,  cleared for use by the warfighter in operational situations.  MDA uses a  variety of information as a basis to assess a new capability for declaration.   For example, MDA will define in advance tests, models, and simulations it  will use to base a specific decision on whether an asset or capability can  be declared ready for fielding. Each capability designation so designated  represents upgraded capacity to support the overall function of BMDS in  its mission as well as the level of MDA confidence in the system’s  performance.
To assess testing related progress in fiscal year 2008, we examined the  accomplishments of ten BMDS elements that MDA is developing and  fielding. Our work included examining documents such as Program  Execution Reviews, test plans and reports, and production plans. We also  interviewed officials within each element program office and within MDA  functional directorates. In addition, we discussed each element’s test  program and its results with DOD’s Office of the Director, Operational  Test and Evaluation. We also interviewed officials from the Office of the  Under Secretary of Defense for Acquisition, Technology, and Logistics.
section_title: Test, Targets and Performance Challenges Continue During Fiscal Year 2008
paragraphs: MDA continues to experience difficulties achieving its goals for testing.   During fiscal year 2008, while several tests showed progress in individual  elements and some system level capabilities, all BMDS elements  experienced test delays or shortfalls.  Most were unable to accomplish all  objectives and performance challenges continued for many. Table 1  summarizes test results and target performance for the BMDS elements  during the year.
Because of delays in flight test and a key ground test, MDA was unable to  achieve any of the six knowledge points the MDA Director had scheduled  for fiscal year 2008.  In May 2007, the MDA Director established key  system-level and element-level knowledge points, each based on an event  that was to provide critical information—or knowledge—for a decision  requiring his approval. For example, two knowledge points that MDA had  to defer because of testing problems were confirmation of a new target’s  performance and assessment of the SM-3 Block 1A missile’s ability to  engage and intercept a long range target.
GMD in particular continues to experience testing problems and delays.  Based on its September 2006 plan, MDA had expected to conduct 7 GMD  interceptor flight tests from the start of fiscal year 2007 through the first  quarter of fiscal year 2009. MDA however was only able to conduct two, as  shown in figure 1.
GMD was unable to conduct either of its planned intercept attempts  during fiscal year 2008 – FTG-04 and FTG-05. MDA first delayed and then  later cancelled the FTG-04 test in May 2008 due to a problem with a  telemetry component in the interceptor’s Exoatmospheric Kill Vehicle.   The cancellation of FTG-04 removed an important opportunity to obtain  end-game performance data needed to develop GMD models and to verify  the capability of the fielded Capability Enhancement I (CE-I) EKV.   Moreover, MDA planned to test the CE-I EKV against a dynamic target  scene with countermeasures in both the FTG-04 and FTG-05 flight tests.   However, since FTG-04 was cancelled and the target failed to release the  countermeasure in FTG-05, the fielded CE-I’s ability against  countermeasures still has not been verified.  According to MDA no more  CE-I EKV flight tests have been approved.
The test delays led MDA to restructure its flight test plan for fiscal year  2009, increasing the number of tests, compressing the amount of time to  analyze and prepare for subsequent tests, and increasing the scope of  individual tests. For example, MDA plans to conduct 14 of 18 flight tests in  the third and fourth quarter of fiscal year 2009. Past testing performance  raises questions about whether this is realistic. In fiscal year 2008, MDA  had planned to conduct 18 flight tests, but it only accomplished 10, and  delayed several flight tests into 2009.  In the next GMD end-to-end flight  test—FTG-06 in fourth quarter fiscal year 2009 to first quarter fiscal year  2010 —MDA is accepting a higher level of risk than it previously expected  in conducting this first test of an enhanced configuration of the Kill  Vehicle called the Capability Enhancement II (CE-II) because it will  include several objectives that had planned to be previously tested, but  have not been. For example, the FTG-06 flight test will be the first GMD  test assessing both a CE-II EKV and a complex target scene. Adding to the  risk, it will be only the second test using a newly developed FTF LV-2  target.  Moreover, MDA in January 2008 had merged FTG-06 and FTG-07,  thereby eliminating an additional opportunity to gather important  information from an intercept. FTG-07 will instead be an intercept test of  the two-stage interceptor intended for the European site.
subsections: section_title: Poor Target Missile Performance Continues to Hamper BMDS Testing
paragraphs: Problems with the reliability and availability of targets (which are  themselves ballistic missiles) have increasingly affected BMDS  development and testing since 2006. As MDA recently acknowledged,  target availability became, in some cases, a pacing item for the overall test  program. As was noted in Table 1, problems with targets have reduced  testing of GMD, Sensors, and THAAD during 2008.
Repeated target problems and test cancellations have particularly reduced  opportunities to demonstrate the ability of sensors to discriminate the real  target from countermeasures. In the mid-course of flight, a more  sophisticated threat missile could use countermeasures in an attempt to  deceive BMDS radars and interceptor sensors as to which is the actual  reentry vehicle. In order to improve the effectiveness of the BMDS against  evolving threats, MDA elements are developing advanced discrimination  software in their component’s sensors to distinguish the threat reentry  vehicle from countermeasures and debris. The cancellation of FTG-04 and  subsequent target problems during FTX-03 and FTG-05 prevented  opportunities to gather data to test how well discrimination software  performs in an operational environment. The current fielded configuration  of the GMD kill vehicle has not been tested against countermeasures.
To address the growing need for more sophisticated and reliable targets  for the future BMDS test program, MDA has been developing a new set of  targets called the Flexible Target Family (FTF), which was intended to  provide new short, medium, and long-range targets with ground, air, and  sea launch capabilities. It was viewed as a family in the sense that the  different target sizes and the variants within those sizes would use  common components.  MDA embarked on this major development without  estimating the cost to develop the family of target missiles. MDA  proceeded to develop and even to produce some FTF targets without a  sound business case and, consequently, their acquisition has not gone as  planned. The funds required for the FTF were spent sooner than expected  and were insufficient for the development.
Development of all FTF sizes and variants has been discontinued except  for the 72-inch diameter ground-launched target, referred to as the LV-2.   With guidance from the Missile Defense Executive Board, MDA is  currently conducting a comprehensive review of the targets program to  determine the best acquisition strategy for future BMDS targets. It is  expected to be completed in mid-2009. Whether or not MDA decides to  restart the acquisition of the 52-inch diameter targets, or other FTF  variants, depends on the results of this review.
The process of qualifying FTF target components for the LV-2 was more  difficult than expected. While many of the LV-2’s components are found on  existing systems, their form, fit, function, and the environment they must  fly in are different. Consequently, many critical components initially failed  shock and vibration testing and other qualification tests and had to be  redesigned. MDA has acknowledged that the component qualification  effort ran in parallel with design completion and initial manufacturing.  So  far, the resultant delays in the LV-2 target have had two consequences.   First, a planned test flight of the LV-2 itself for the Space Tracking and  Space Surveillance program was delayed and instead its first flight will be  as an actual target for an Aegis BMD intercept. Second, because the LV-2  was not ready, that Aegis intercept test was deferred from fiscal year 2008  to third quarter fiscal year 2009.
section_title: Other Consequences of Less Productive Testing
paragraphs: In addition to delaying progress on individual elements, testing problems  have had other consequences for BMDS.  Specifically, the reduced  productivity of testing has delayed understanding the overall performance  of BMDS, production and fielding have in some cases gotten ahead of  testing, and declarations of capabilities ready for fielding have been made  based on fewer tests and less modeling and simulation than planned.
subsections: section_title: Overall Performance of BMDS Can Not Yet Be Assessed
paragraphs: The overall performance of the BMDS cannot yet be assessed because  MDA lacks a fully accredited end-to-end model and simulation capability  and, according to the BMDS Operational Test Agency, it will not have that  capability until 2011 at the earliest. The lack of sufficient flight test data  has inhibited the validation of the models and simulations needed for the  ground tests and the simulation. MDA’s modeling and simulation program  enables it to assess the capabilities and limitations of how BMDS performs  under a wider variety of conditions than can be accomplished through the  limited number of flight tests conducted. Flight tests alone are insufficient  because they only demonstrate a single collection data point of element  and system performance. Flight tests are, however, an essential tool used  to both validate performance of the BMDS and to anchor the models and  simulations to ensure they accurately reflect real performance. Computer  models of individual elements replicate how those elements function.  These models are then aggregated into various combinations that simulate  the BMDS engagement of enemy ballistic missiles.
Developing an end-to-end system-level model and simulation has been  difficult. MDA’s first effort to bring together different element models and  simulations to produce a fully accredited, end-to-end model and  simulation was for the first annual performance assessment of the fielded  BMDS configuration in 2007. Performance Assessment 2007 was  unsuccessful primarily because of inadequate data, particularly flight test  data, for verification and validation to support accreditation. Instead,  Performance Assessment 2007 used several models and simulations that  represented different aspects of the BMD system and were not fully  integrated. Consequently, acting on a joint recommendation between MDA  and the Operational Test Agency, MDA officials cancelled the 2008  performance assessment in April 2008 because of developmental risks  associated with modeling and simulations, focusing instead on testing and  models for Performance Assessment 2009.
According to the BMDS Operational Test Agency’s January 2009 Modeling  and Simulation accreditation report, confidence in MDA’s Modeling and  Simulation efforts remains low although progress was made during the  year. Out of 40 models, the BMDS Operational Test Agency recommended  in January 2009 full accreditation for only 6 models, partial accreditation  for 9 models, and no accreditation for 25 models. MDA is now exercising  stronger central leadership to provide guidance and resources as they  coordinate the development of verified and validated models and  simulations.
MDA intends to verify and validate models and simulations by December  2009 for Performance Assessment 2009. However, BMDS Operational Test  Agency officials stated that there is a high risk that the performance  assessment 2009 analysis will be delayed because of remaining challenges  and MDA’s delayed progress in accreditation.  MDA does not expect to  have a single end-to-end simulation for use in performance assessments  until 2010.
section_title: Production and Fielding Proceed Despite Delays in Testing and Assessments
paragraphs: Testing problems have contributed to a concurrent development,  manufacturing and fielding strategy in which assets are produced and  fielded before they are fully demonstrated through testing and modeling.  For example, although a test of the ability of the SM-3 Block 1A missile to  engage and intercept a long range ballistic target was delayed until the  third quarter of fiscal year 2009, MDA purchased 20 of the missiles in fiscal  year 2008 ahead of schedule.
While the GMD program has only been able to conduct two intercepts  since 2006 for assessing the fielded configuration, the production of  interceptors has continued. From the beginning of fiscal year 2007 through  the first quarter of fiscal year 2009, MDA planned to conduct 7 flight tests  and field 16 new ground-based interceptors. The plan included a test that  would utilize two ground-based interceptors against a single target, known  as a salvo test.  By January 2009, GMD had conducted only 2 flight tests  and dropped the salvo test; yet it fielded 13 ground-based interceptors.
Moreover, the GMD program had planned to conduct an intercept test to  assess the enhanced version of the EKV called the Capability  Enhancement II (CE-II) in the first quarter of fiscal year 2008, months  before emplacing any interceptors with this configuration. However,  developmental problems with the new configuration’s inertial  measurement unit and the target delayed the first flight test with the CE-II  configuration—FTG-06—until at least fourth quarter fiscal year 2009.  Despite these delays, emplacements will proceed; MDA expects to have  emplaced five CE-II interceptors before this flight test. More importantly,  GMD projects that the contractor will have manufactured and delivered 10  CE-II EKVs before that first flight test demonstrates the CE-II capability.
This amounts to over half of the CE-II EKV deliveries that are currently  under contract.
section_title: Declaration of Capabilities Proceed with Reduced Levels of Information
paragraphs: When MDA determines that a capability can be considered for operational  use it does so through a formal declaration. MDA bases its declarations on,  among other things, a combination of models and simulations—such as  end-to-end performance assessments (from missile launch to attempted  intercept)—and ground tests all anchored to flight test data.
In fiscal year 2008, MDA declared it had fielded 7 of 17 BMDS capabilities  planned for 2008 (postponing 10). In doing so MDA largely reduced the  basis for the declarations due in part to test problems and delays.   Specifically, MDA had intended to use a GMD flight test that was  cancelled, a key ground test that was delayed and a performance  assessment that was cancelled. MDA had to shift the basis of the 7  declarations to previous flight and ground tests.
section_title: Review of BMDS Modeling and Testing Holds Promise, but Must Anticipate Contingences
paragraphs: MDA has undertaken a three-phase review of the entire BMDS modeling,  simulation, and test program.  According to MDA, the three phases involve  identifying critical variables that have not been proven to date,  determining what test scenarios are needed to collect the data, and  developing an affordable and prioritized schedule of flight and ground  tests. MDA intends to complete all three phases of the review by May 2009.  At this point, our knowledge of the review is limited, as we have only had  an introductory briefing on it.  Nonetheless, the review appears to offer a  sound approach for closing the gaps that exist between testing, modeling,  and simulation.  Further, the involvement of test and evaluation  organizations is encouraging.
While sound, the success of this approach hinges on providing sufficient  resources, ensuring robustness, and anticipating contingencies.  In  addition to linking the critical modeling and simulation variables with test  events, the review will have to address the factors that have limited the  productivity of the current test approach, such as the availability and  performance of targets. MDA’s current approach to testing could be  characterized as a just-in-time approach to having the test assets, such as  targets, ready. This left little margin to solve issues that arise leading up to  the tests. Accordingly, the third phase of MDA’s new approach—properly  resourcing the tests with sufficient time, funding and reliable targets—will  be key. MDA has indicated that its revision will result in a more robust test  plan, providing more margin to conduct the tests through, for example,  having spare interceptors and targets available.
Other contingencies that a new approach to modeling, simulation, and  testing should anticipate include unexpected or incomplete test results,  and problems in accrediting the models that are needed for aggregated  simulations, such as performance assessments.  An important  consideration in this regard is for modeling, simulation, and testing events  to be re-synchronized so that they properly inform decisions on producing,  fielding, and declaring assets operational.  Contingency plans could then  be formed for adjusting the pace of these decisions should shortfalls occur  in modeling, simulation, or testing.
MDA has indicated that this new approach to testing will take time to  implement, with partial implementation in fiscal year 2010 and full  implementation not occurring until fiscal year 2011. Therefore, MDA must  manage the transition to the new testing approach.  In particular, the  ambitious fiscal year 2009 flight test plan may need to be reassessed with  the goal of establishing a robust series of tests that can withstand some  delays without causing wholesale changes to the test plan during the  transition. In the mean time, MDA will have to be prudent in making  decisions to produce and field additional assets.
Our annual report on missile defense is in draft and with DOD for  comment.  It will be issued in final by March 13, 2009.  In that report, we  are recommending additional steps to further improve the transparency,  accountability, and oversight of the missile defense program.  Our  recommendations include actions to improve cost reporting as well as  testing and evaluation. DOD is in the process of preparing a formal  response to the report and its recommendations.
Madame Chairman, this concludes my statement. I would be pleased to  respond to any questions you or members of the subcommittee may have.
section_title: Contacts and Staff Acknowledgments
paragraphs: For questions about this statement, please contact me at (202) 512-4841 or  Francisp@gao.gov. Individuals making key contributions to this statement  include David B. Best, Assistant Director; Steven B. Stern; LaTonya D.  Miller; Thomas Mahalek; Ivy Hübler; Meredith Allen Kimmett; Kenneth E.  Patton; and Alyssa Weir.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.