Defense Acquisitions: Charting a Course for Improved Missile Defense Testing
Released: 2009-02-25T00:00:00

Letter

Background

The Missile Defense Agency’s mission is to develop an integrated and layered BMDS to defend the United States, its deployed forces, allies, and friends. In order to meet this mission, MDA is developing a highly complex system of systems—land, sea and space based sensors, interceptors and battle management. Since its initiation in 2002, MDA has been given a significant amount of flexibility in executing the development and fielding of the BMDS. To enable MDA to field and enhance a missile defense system quickly, the Secretary of Defense in 2002 delayed the entry of the BMDS program into the Department of Defense’s traditional acquisition process until a mature capability was ready to be handed over to a military service for production and operation. Therefore, the program concurrently develops, tests and fields assets. This approach helped MDA rapidly deploy an initial capability. On the other hand, because MDA can field assets before all testing is completed, it has fielded some assets whose capability is uncertain.

Because MDA develops and fields assets continuously, it combines developmental testing with operational testing. In general, developmental testing is aimed at determining whether the system design will satisfy the desired capabilities; operational testing determines whether the system is effective, survivable, and suitable in the hands of the user. MDA conducts testing both on the ground and in flight. The most complex of these is an end-to-end flight test that involves a test of all phases of an engagement including detecting, tracking and destroying a target with an interceptor missile. An end-to-end intercept involves more than one MDA element. For example, a recent intercept test involved a target flown out of Kodiak, Alaska, tracked by the AN/TPY-2 radar located in Alaska, and the Beale upgraded early warning radar located in California, the Sea-based X-band radar and an Aegis radar located at different points in the Pacific. All of the radars communicated with fire control centers in Alaska to guide an interceptor launched from California to hit the target over the Pacific Ocean.

Due to the complexity, scale, safety constraints, and cost involved, MDA is unable to conduct a sufficient number of flight tests to fully understand the performance of the system. Therefore, MDA utilizes models and simulations, anchored by flight tests, to understand both the developmental and operational performance of the system. To ensure confidence in the accuracy of modeling and simulation the program goes through a process called accreditation. The models are validated individually using flight and other test data and accredited for their intended use. Models and simulations are used prior to a flight test to predict performance, the flight test is then run to gather data and verify the models, and then data is analyzed after the flight and reconstructed using the models and simulations to confirm their accuracy.

MDA intends to group these models into system-level representations according to user needs. One such grouping is the annual performance assessment, a system-level end-to-end simulation that assesses the performance of the BMDS configuration as it exists in the field. The performance assessment integrates element-specific models into a coherent representation of the BMDS. Fundamentally, performance assessments anchored by flight tests are a comprehensive means to fully understand the performance capabilities and limitations of the BMDS.

In addition to testing, modeling and simulation, and performance assessments, MDA also has a formal process for determining when a newly fielded asset or group of assets can be declared operational—that is, cleared for use by the warfighter in operational situations. MDA uses a variety of information as a basis to assess a new capability for declaration. For example, MDA will define in advance tests, models, and simulations it will use to base a specific decision on whether an asset or capability can be declared ready for fielding. Each capability designation so designated represents upgraded capacity to support the overall function of BMDS in its mission as well as the level of MDA confidence in the system’s performance.

To assess testing related progress in fiscal year 2008, we examined the accomplishments of ten BMDS elements that MDA is developing and fielding. Our work included examining documents such as Program Execution Reviews, test plans and reports, and production plans. We also interviewed officials within each element program office and within MDA functional directorates. In addition, we discussed each element’s test program and its results with DOD’s Office of the Director, Operational Test and Evaluation. We also interviewed officials from the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics.

Test, Targets and Performance Challenges Continue During Fiscal Year 2008

MDA continues to experience difficulties achieving its goals for testing. During fiscal year 2008, while several tests showed progress in individual elements and some system level capabilities, all BMDS elements experienced test delays or shortfalls. Most were unable to accomplish all objectives and performance challenges continued for many. Table 1 summarizes test results and target performance for the BMDS elements during the year.

Because of delays in flight test and a key ground test, MDA was unable to achieve any of the six knowledge points the MDA Director had scheduled for fiscal year 2008. In May 2007, the MDA Director established key system-level and element-level knowledge points, each based on an event that was to provide critical information—or knowledge—for a decision requiring his approval. For example, two knowledge points that MDA had to defer because of testing problems were confirmation of a new target’s performance and assessment of the SM-3 Block 1A missile’s ability to engage and intercept a long range target.

GMD in particular continues to experience testing problems and delays. Based on its September 2006 plan, MDA had expected to conduct 7 GMD interceptor flight tests from the start of fiscal year 2007 through the first quarter of fiscal year 2009. MDA however was only able to conduct two, as shown in figure 1.

GMD was unable to conduct either of its planned intercept attempts during fiscal year 2008 – FTG-04 and FTG-05. MDA first delayed and then later cancelled the FTG-04 test in May 2008 due to a problem with a telemetry component in the interceptor’s Exoatmospheric Kill Vehicle. The cancellation of FTG-04 removed an important opportunity to obtain end-game performance data needed to develop GMD models and to verify the capability of the fielded Capability Enhancement I (CE-I) EKV. Moreover, MDA planned to test the CE-I EKV against a dynamic target scene with countermeasures in both the FTG-04 and FTG-05 flight tests. However, since FTG-04 was cancelled and the target failed to release the countermeasure in FTG-05, the fielded CE-I’s ability against countermeasures still has not been verified. According to MDA no more CE-I EKV flight tests have been approved.

The test delays led MDA to restructure its flight test plan for fiscal year 2009, increasing the number of tests, compressing the amount of time to analyze and prepare for subsequent tests, and increasing the scope of individual tests. For example, MDA plans to conduct 14 of 18 flight tests in the third and fourth quarter of fiscal year 2009. Past testing performance raises questions about whether this is realistic. In fiscal year 2008, MDA had planned to conduct 18 flight tests, but it only accomplished 10, and delayed several flight tests into 2009. In the next GMD end-to-end flight test—FTG-06 in fourth quarter fiscal year 2009 to first quarter fiscal year 2010 —MDA is accepting a higher level of risk than it previously expected in conducting this first test of an enhanced configuration of the Kill Vehicle called the Capability Enhancement II (CE-II) because it will include several objectives that had planned to be previously tested, but have not been. For example, the FTG-06 flight test will be the first GMD test assessing both a CE-II EKV and a complex target scene. Adding to the risk, it will be only the second test using a newly developed FTF LV-2 target. Moreover, MDA in January 2008 had merged FTG-06 and FTG-07, thereby eliminating an additional opportunity to gather important information from an intercept. FTG-07 will instead be an intercept test of the two-stage interceptor intended for the European site.

Poor Target Missile Performance Continues to Hamper BMDS Testing

Problems with the reliability and availability of targets (which are themselves ballistic missiles) have increasingly affected BMDS development and testing since 2006. As MDA recently acknowledged, target availability became, in some cases, a pacing item for the overall test program. As was noted in Table 1, problems with targets have reduced testing of GMD, Sensors, and THAAD during 2008.

Repeated target problems and test cancellations have particularly reduced opportunities to demonstrate the ability of sensors to discriminate the real target from countermeasures. In the mid-course of flight, a more sophisticated threat missile could use countermeasures in an attempt to deceive BMDS radars and interceptor sensors as to which is the actual reentry vehicle. In order to improve the effectiveness of the BMDS against evolving threats, MDA elements are developing advanced discrimination software in their component’s sensors to distinguish the threat reentry vehicle from countermeasures and debris. The cancellation of FTG-04 and subsequent target problems during FTX-03 and FTG-05 prevented opportunities to gather data to test how well discrimination software performs in an operational environment. The current fielded configuration of the GMD kill vehicle has not been tested against countermeasures.

To address the growing need for more sophisticated and reliable targets for the future BMDS test program, MDA has been developing a new set of targets called the Flexible Target Family (FTF), which was intended to provide new short, medium, and long-range targets with ground, air, and sea launch capabilities. It was viewed as a family in the sense that the different target sizes and the variants within those sizes would use common components. MDA embarked on this major development without estimating the cost to develop the family of target missiles. MDA proceeded to develop and even to produce some FTF targets without a sound business case and, consequently, their acquisition has not gone as planned. The funds required for the FTF were spent sooner than expected and were insufficient for the development.

Development of all FTF sizes and variants has been discontinued except for the 72-inch diameter ground-launched target, referred to as the LV-2. With guidance from the Missile Defense Executive Board, MDA is currently conducting a comprehensive review of the targets program to determine the best acquisition strategy for future BMDS targets. It is expected to be completed in mid-2009. Whether or not MDA decides to restart the acquisition of the 52-inch diameter targets, or other FTF variants, depends on the results of this review.

The process of qualifying FTF target components for the LV-2 was more difficult than expected. While many of the LV-2’s components are found on existing systems, their form, fit, function, and the environment they must fly in are different. Consequently, many critical components initially failed shock and vibration testing and other qualification tests and had to be redesigned. MDA has acknowledged that the component qualification effort ran in parallel with design completion and initial manufacturing. So far, the resultant delays in the LV-2 target have had two consequences. First, a planned test flight of the LV-2 itself for the Space Tracking and Space Surveillance program was delayed and instead its first flight will be as an actual target for an Aegis BMD intercept. Second, because the LV-2 was not ready, that Aegis intercept test was deferred from fiscal year 2008 to third quarter fiscal year 2009.

Other Consequences of Less Productive Testing

In addition to delaying progress on individual elements, testing problems have had other consequences for BMDS. Specifically, the reduced productivity of testing has delayed understanding the overall performance of BMDS, production and fielding have in some cases gotten ahead of testing, and declarations of capabilities ready for fielding have been made based on fewer tests and less modeling and simulation than planned.

Overall Performance of BMDS Can Not Yet Be Assessed

The overall performance of the BMDS cannot yet be assessed because MDA lacks a fully accredited end-to-end model and simulation capability and, according to the BMDS Operational Test Agency, it will not have that capability until 2011 at the earliest. The lack of sufficient flight test data has inhibited the validation of the models and simulations needed for the ground tests and the simulation. MDA’s modeling and simulation program enables it to assess the capabilities and limitations of how BMDS performs under a wider variety of conditions than can be accomplished through the limited number of flight tests conducted. Flight tests alone are insufficient because they only demonstrate a single collection data point of element and system performance. Flight tests are, however, an essential tool used to both validate performance of the BMDS and to anchor the models and simulations to ensure they accurately reflect real performance. Computer models of individual elements replicate how those elements function. These models are then aggregated into various combinations that simulate the BMDS engagement of enemy ballistic missiles.

Developing an end-to-end system-level model and simulation has been difficult. MDA’s first effort to bring together different element models and simulations to produce a fully accredited, end-to-end model and simulation was for the first annual performance assessment of the fielded BMDS configuration in 2007. Performance Assessment 2007 was unsuccessful primarily because of inadequate data, particularly flight test data, for verification and validation to support accreditation. Instead, Performance Assessment 2007 used several models and simulations that represented different aspects of the BMD system and were not fully integrated. Consequently, acting on a joint recommendation between MDA and the Operational Test Agency, MDA officials cancelled the 2008 performance assessment in April 2008 because of developmental risks associated with modeling and simulations, focusing instead on testing and models for Performance Assessment 2009.

According to the BMDS Operational Test Agency’s January 2009 Modeling and Simulation accreditation report, confidence in MDA’s Modeling and Simulation efforts remains low although progress was made during the year. Out of 40 models, the BMDS Operational Test Agency recommended in January 2009 full accreditation for only 6 models, partial accreditation for 9 models, and no accreditation for 25 models. MDA is now exercising stronger central leadership to provide guidance and resources as they coordinate the development of verified and validated models and simulations.

MDA intends to verify and validate models and simulations by December 2009 for Performance Assessment 2009. However, BMDS Operational Test Agency officials stated that there is a high risk that the performance assessment 2009 analysis will be delayed because of remaining challenges and MDA’s delayed progress in accreditation. MDA does not expect to have a single end-to-end simulation for use in performance assessments until 2010.

Production and Fielding Proceed Despite Delays in Testing and Assessments

Testing problems have contributed to a concurrent development, manufacturing and fielding strategy in which assets are produced and fielded before they are fully demonstrated through testing and modeling. For example, although a test of the ability of the SM-3 Block 1A missile to engage and intercept a long range ballistic target was delayed until the third quarter of fiscal year 2009, MDA purchased 20 of the missiles in fiscal year 2008 ahead of schedule.

While the GMD program has only been able to conduct two intercepts since 2006 for assessing the fielded configuration, the production of interceptors has continued. From the beginning of fiscal year 2007 through the first quarter of fiscal year 2009, MDA planned to conduct 7 flight tests and field 16 new ground-based interceptors. The plan included a test that would utilize two ground-based interceptors against a single target, known as a salvo test. By January 2009, GMD had conducted only 2 flight tests and dropped the salvo test; yet it fielded 13 ground-based interceptors.

Moreover, the GMD program had planned to conduct an intercept test to assess the enhanced version of the EKV called the Capability Enhancement II (CE-II) in the first quarter of fiscal year 2008, months before emplacing any interceptors with this configuration. However, developmental problems with the new configuration’s inertial measurement unit and the target delayed the first flight test with the CE-II configuration—FTG-06—until at least fourth quarter fiscal year 2009. Despite these delays, emplacements will proceed; MDA expects to have emplaced five CE-II interceptors before this flight test. More importantly, GMD projects that the contractor will have manufactured and delivered 10 CE-II EKVs before that first flight test demonstrates the CE-II capability.

This amounts to over half of the CE-II EKV deliveries that are currently under contract.

Declaration of Capabilities Proceed with Reduced Levels of Information

When MDA determines that a capability can be considered for operational use it does so through a formal declaration. MDA bases its declarations on, among other things, a combination of models and simulations—such as end-to-end performance assessments (from missile launch to attempted intercept)—and ground tests all anchored to flight test data.

In fiscal year 2008, MDA declared it had fielded 7 of 17 BMDS capabilities planned for 2008 (postponing 10). In doing so MDA largely reduced the basis for the declarations due in part to test problems and delays. Specifically, MDA had intended to use a GMD flight test that was cancelled, a key ground test that was delayed and a performance assessment that was cancelled. MDA had to shift the basis of the 7 declarations to previous flight and ground tests.

Review of BMDS Modeling and Testing Holds Promise, but Must Anticipate Contingences

MDA has undertaken a three-phase review of the entire BMDS modeling, simulation, and test program. According to MDA, the three phases involve identifying critical variables that have not been proven to date, determining what test scenarios are needed to collect the data, and developing an affordable and prioritized schedule of flight and ground tests. MDA intends to complete all three phases of the review by May 2009. At this point, our knowledge of the review is limited, as we have only had an introductory briefing on it. Nonetheless, the review appears to offer a sound approach for closing the gaps that exist between testing, modeling, and simulation. Further, the involvement of test and evaluation organizations is encouraging.

While sound, the success of this approach hinges on providing sufficient resources, ensuring robustness, and anticipating contingencies. In addition to linking the critical modeling and simulation variables with test events, the review will have to address the factors that have limited the productivity of the current test approach, such as the availability and performance of targets. MDA’s current approach to testing could be characterized as a just-in-time approach to having the test assets, such as targets, ready. This left little margin to solve issues that arise leading up to the tests. Accordingly, the third phase of MDA’s new approach—properly resourcing the tests with sufficient time, funding and reliable targets—will be key. MDA has indicated that its revision will result in a more robust test plan, providing more margin to conduct the tests through, for example, having spare interceptors and targets available.

Other contingencies that a new approach to modeling, simulation, and testing should anticipate include unexpected or incomplete test results, and problems in accrediting the models that are needed for aggregated simulations, such as performance assessments. An important consideration in this regard is for modeling, simulation, and testing events to be re-synchronized so that they properly inform decisions on producing, fielding, and declaring assets operational. Contingency plans could then be formed for adjusting the pace of these decisions should shortfalls occur in modeling, simulation, or testing.

MDA has indicated that this new approach to testing will take time to implement, with partial implementation in fiscal year 2010 and full implementation not occurring until fiscal year 2011. Therefore, MDA must manage the transition to the new testing approach. In particular, the ambitious fiscal year 2009 flight test plan may need to be reassessed with the goal of establishing a robust series of tests that can withstand some delays without causing wholesale changes to the test plan during the transition. In the mean time, MDA will have to be prudent in making decisions to produce and field additional assets.

Our annual report on missile defense is in draft and with DOD for comment. It will be issued in final by March 13, 2009. In that report, we are recommending additional steps to further improve the transparency, accountability, and oversight of the missile defense program. Our recommendations include actions to improve cost reporting as well as testing and evaluation. DOD is in the process of preparing a formal response to the report and its recommendations.

Madame Chairman, this concludes my statement. I would be pleased to respond to any questions you or members of the subcommittee may have.

Contacts and Staff Acknowledgments

For questions about this statement, please contact me at (202) 512-4841 or Francisp@gao.gov. Individuals making key contributions to this statement include David B. Best, Assistant Director; Steven B. Stern; LaTonya D. Miller; Thomas Mahalek; Ivy Hübler; Meredith Allen Kimmett; Kenneth E. Patton; and Alyssa Weir.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.