Defense Acquisitions: Production and Fielding of Missile Defense Components Continue with Less Testing and Validation Than Planned
Released: 2009-03-13T00:00:00

Letter

Background

MDA’s mission is to develop an integrated and layered system to defend the United States and its deployed forces, friends, and allies against ballistic missile attacks. The BMDS aims to engage all ranges of ballistic missiles during all phases of flight. This challenging expectation requires complex coordination within an integrated system of defensive components—space-based sensors, surveillance and tracking radars, advanced interceptors, and a battle management, command, control, and communication component.

A typical engagement scenario to defend against an intercontinental ballistic missile would occur as follows: Infrared sensors aboard early-warning satellites detect the hot plume of a missile launch and alert the command authority of a possible attack.

Upon receiving the alert, land- or sea-based radars are directed to track the various objects released from the missile and, if so designed, to identify the warhead from among spent rocket motors, decoys, and debris.

When the trajectory of the missile’s warhead has been adequately established, an interceptor—consisting of a kill vehicle mounted atop a booster—is launched to engage the threat. The interceptor boosts itself toward a predicted intercept point and releases the kill vehicle.

The kill vehicle uses its onboard sensors and divert thrusters to detect, identify, and steer itself into the warhead. With a combined closing speed of up to 10 kilometers per second (22,000 miles per hour), the warhead is destroyed above the atmosphere through a “hit to kill” collision with the kill vehicle.

Some interceptors use sensors to steer themselves into the inbound ballistic missile. Inside the atmosphere, these systems kill the ballistic missile using a range of mechanisms such as direct collision between the missile and the inbound ballistic missile or killing it with the combined effects of a blast fragmentation warhead (heat, pressure, and grains/shrapnel) in cases where a direct hit does not occur.

Table 1 provides a brief description of 10 BMDS elements currently under development by MDA.

To manage BMDS development, MDA uses an acquisition strategy defined by a block structure. From its inception in 2002 through 2007, MDA developed BMDS capability in biennial increments, ultimately delivering two blocks—Block 2004 and Block 2006. These 2-year blocks each built on preceding blocks and enhanced the development and capability of the BMDS. However, in response to recommendations from GAO, in December 2007 MDA announced a new block structure that was intended to improve the program’s transparency, accountability, and oversight. The new blocks are not based on biennial time periods, but instead focus on fielding capabilities that address particular threats. Because the new block structure is not aligned to regular time periods, multiple blocks are underway concurrently. Table 2 details the current blocks and categories included in the BMDS block structure.

MDA uses a Statement of Baselines and Goals to report modifications to established block baselines. For those blocks that are currently or will soon be underway, block baselines are created to make a firm commitment to Congress. The Statement of Goals also includes the following: BMDS Baseline Capabilities – Assets and engagement sequence groups that will be made available for fielding for a particular block. During 2008, cost baselines were under development for Block 2.0 and Block increments 3.1/3.2. MDA established schedule and performance baselines for Blocks 1.0, 2.0, 3.1, and 3.2 in 2008.

BMDS Capability Goals – Assets and engagement sequence groups expected to be made available for future blocks.

Adversary Benchmarks – Adversary missile systems used for block performance estimates.

BMDS Budget Breakdowns – Detailed fielding, development, and integration budgets for each block and BMDS Capability Development activity.

MDA also uses an incremental declaration process to designate BMDS capability for its blocks. Three capability designations are applied to all BMDS elements, their hardware and software components, and engagement sequence groups. This allows these BMDS features to play a limited role in system operations before they have attained their expected level of capability. Each capability designation in the delivery schedule represents upgraded capacity to support the overall function of BMDS in its mission as well as the level of MDA confidence in the system’s performance. The designations are defined as follows: Early Capability Delivery signifies readiness for contingency use. At this point, MDA has determined that the capability can be utilized by the BMDS. When integrated, the capability must be adequately demonstrated to build sufficient confidence that it will safely perform as intended without degrading the existing capabilities of the BMDS.

Partial Capability Delivery is an intermediate state of maturity indicating that a capability has been shown through testing to perform as intended in certain scenarios. At this point, MDA is sufficiently confident that the capability can support the warfighter’s partially mission-capable objectives and logistics support is adequate to achieve defensive operations.

Full Capability Delivery is the point at which a capability satisfies the BMDS block objectives and is considered to be completely mature and ready for full operational use.

MDA’s capability goals for fiscal year 2008 for Blocks 1.0, 2.0, 3.1, and 3.2 are shown in table 3.

Cost Tracking Deficiencies Hinder Assessment of Cost Performance

MDA has not yet established baselines for total costs or unit costs, both fundamental markers that most programs use to measure progress. MDA had planned to establish total cost baselines at the element and block levels in 2008, but the initial set of total cost baselines will not be available until the spring of 2009. Similarly, MDA has not established unit costs for selected assets, such as GBIs. Consequently, for the sixth year, we have been unable to assess MDA’s overall progress on total or unit cost. While MDA plans to establish some total cost baselines in 2009, most efforts will not be captured in a baseline. MDA also plans to establish unit costs, another improvement, but is considering a narrower definition of unit cost than is used by other weapon system programs. MDA’s definition will report a subset of procurement costs called flyaway costs, which only includes the major piece of equipment and excludes all research and development as well as some procurement costs—those for support equipment and spares. Moreover, these unit costs will only be tracked within those blocks that are baselined, which will represent a minority of those assets being produced and fielded. Without total cost baselines in place, the BMDS concept continually evolves, as indicated by the number of realignments to the program of work at the individual contract level. While the changing nature of the BMDS and the lack of total cost baselines precludes analysis of total cost progress, we were able to analyze contractor fiscal year performance on the current work under the contract. We were also able to project overruns or underruns at completion for BMDS contracts using the contracts’ current budgeted costs at completion as a basis for our projections. However, in some cases, the current budgeted cost at completion changed significantly over time. In one case, the budgeted cost at completion increased by approximately five times its original value. Our analysis of fiscal year 2008 progress shows that several prime contractors exceeded budgeted costs.

Absence of Cost Baselines Prevents Assessment of System-Level Costs

To provide accountability, major defense acquisition programs are required by statute to document program goals in an acquisition program baseline. MDA is not yet required to establish an acquisition program baseline because of the acquisition flexibilities it has been granted. However, Congress has enacted legislation requiring MDA to establish some baselines. Baselines serve an important discipline both by ensuring that the full cost commitment is considered before embarking on major development efforts and by identifying cost growth as a program proceeds. Since we began annual reporting on missile defense in 2004, we have been unable to assess overall progress on cost—that is, comparing BMDS baselined costs with actual costs. For example, under the prior block structure, we reported that BMDS costs grew by at least $1 billion, but the total cost growth could not be determined because MDA did not account for all costs for a given block.

In response to recommendations we made in March 2008, MDA agreed to develop cost estimates and to provide independent verification for blocks outlined under its new approach. Upon conclusion of those estimates, MDA will develop cost baselines. In addition, on April 1, 2008, the Director, MDA, testified before a Senate Armed Services Subcommittee that cost baselines for the new block structure would be available by the end of 2008. As of January 2009, the agency had not yet developed full cost baselines for any blocks. MDA plans to have these cost baselines for Blocks 2.0, 3.1 and 3.2 completed, independently reviewed by DOD’s Cost Analysis Improvement Group, and released by the spring of 2009. The only information that was available for this report was limited to budget projections for BMDS blocks and capability development for fiscal years 2008 through 2013, totaling approximately $42 billion.

Even with the release of some block cost estimates in 2009, all block costs will not be baselined and no date has been established for when the remaining block costs will be baselined. MDA does not plan to baseline Block 1.0 costs, but will provide the actual costs of the block since it is near completion. Full cost baselines for Block 2.0 are anticipated to be available in the spring of 2009, but only portions of Block 3.0 and none of Block 4.0 and 5.0 costs will be baselined at this time. As figure 1 shows, if MDA does complete baselines as planned, they will only cover about 26 percent of its block and capability development costs.

At this point, MDA plans to baseline between 2 and 26 percent of BMDS block and capability development costs from fiscal years 2010 to 2013 as depicted above. MDA has not determined when other blocks will be baselined. If other blocks were to be baselined before 2013, the percentage of funding baselined would be increased. The rapid decline in percentage of baselined funds also shows that initial baselines are being set late in a block’s duration. For example, Block 2.0 will be completed within 2 years of its baseline being set. If baselines are to facilitate management and oversight, they will have to be set sooner for Block 3.3 and beyond. Additionally, agency officials stated that although cost estimates will be developed for individual capability development efforts—such as ABL, KEI, and MKV—the agency does not plan to baseline their costs until these elements are matured and moved into a defined block. MDA may eventually baseline these elements as part of a block once a firm commitment can be made to Congress. The budgets for capability development elements account for approximately $22 billon—or more than half of MDA’s fiscal year 2008 6-year Future Years Defense Plan for BMDS blocks and capability development.

Planned Unit Cost Reporting Will Not Be Comprehensive

Major defense acquisition programs are required by statute to report certain unit costs to Congress, track unit cost growth against their original and current baseline estimates, and perform an additional assessment of the program if certain cost growth thresholds are reach This cost monitoring mechanism helps ensure that programs are being held accountable. MDA is not yet required to report these unit costs because of the acquisition flexibilities it has been granted by DOD, but Congress has enacted legislation requiring MDA to provide unit cost reporting data for certain BMDS elements, and MDA does plan to develop and report unit costs for some of its assets in the spring of 2009. The agency has also established thresholds for reporting cost growth. However, the approach MDA is taking, while an improvement, provides a much less comprehensive assessment of unit cost compared to the traditional acquisition costs that are typically reported for major defense acquisition programs. Normally, unit costs are reported in two ways: (1) program acquisition unit cost, which is the total cost for the development and procurement of, and system-specific military construction for, the acquisition program divided by the number of fully configured end items to be produced, or (2) average procurement unit cost, which is the total of all funds programmed to be available for obligation for procurement divided by the number of end items to be procured. ed.

MDA’s development of the BMDS outside of DOD’s normal acquisition process makes it difficult to compare the actual unit co st of a delivered asset with its planned unit cost. For example, MDA plans to only report recurring unit flyaway costs for the blocks that are baselined. Figure 2 reveals the significant reduction in standard areas of costs covered by MDA’s approach compared to that normally reported for major defense acquisition programs.

MDA’s decision to report only flyaway unit costs will not capture research and development costs associated with BMDS assets—which account for more than 97 percent of the nearly $56 billion MDA costs to date. In addition, the procurement costs for initial spares and support equipment are not included. Thus, while the flyaway cost baseline will provide visibility into changes in recurring manufacturing costs, it will not provide a basis for comparison with the costs of other DOD programs. If a cost increase occurs in research and development or nonrecurring procurement, it will not be reported in MDA’s unit cost.

Agency officials told us that the reason for using flyaway unit c the new MDA block structure. They further explained that within the block structure, there are many cases where MDA procures and delivers a weapon system for more than one block and, in some cases, the same configuration in more than one block. For example, THAAD deliveries are included in Blocks 2.0 and 5.0. Agency officials cite this as a key difference between MDA and other defense programs. For MDA, most of the development costs are assigned to the first block where a capability is delivered and very little of the development cost is assigned to subsequent blocks. MDA officials further stated that if the agency were to use the standard unit cost methodology, it would show very dissimilar unit costs between the first and subsequent block deliveries and the difference could not be explained by learning curves and manufacturing efficiencies. MDA officials also told us that they chose unit flyaway cost for unit cost reporting because flyaway cost provides a better measure of what the individual system components cost to procure. However, MDA is not precluded from also determining and reporting unit costs by taking the entire cost of the asset being developed without regard to the capability or block for which it is originally developed.

Frequent Realignments Indicate That Full Scope Is Not Yet Determined

Without a firm cost commitment or baseline in place for the full scope of the BMDS, in some cases the work currently under contract changes frequently. These changes manifest themselves in realignments that often add scope, cost and time to the value of the work under the contract. Since contracts began on the BMDS, MDA has performed 31 realignm where, in some cases, the period of stability between these major realignments average only between 1 to 2 years. These frequent chang indicate that the total BMDS effort has not been fully determined and is likely to grow. While we have been able to make an assessment of contractor costs, that assessment is limited to the current approved program. es Until total and unit cost baselines are established, the only tool for us to use in assessing BMDS costs is the costs reported on individual contracts under BMDS’s Earned Value Management System. All BMDS contracts that we assessed have a cost and schedule baseline against which progress, measured by cost and schedule performance, can be measured. It is appropriate for a program to realign its current status with the remaining contractual effort when officials conclude that the baseline no longer provides valid performance assessment information. A program can realign its current status with the remaining contractual effort through rebaselines, replans, and restructures.

A rebaseline is a more general term to describe a major realignment of the performance measurement baseline used to better correlate the work plan with the baseline budget, scope, and schedule and can refer to replans and restructures as well. A replan is a reallocation of schedule or budget for the remaining effort within the existing constraints of the contract.

A restructure includes adding funds to the performance manageme budget that exceed the value of the negotiated contra contract modification.

MDA Contractors Overran Fiscal Year Cost and Schedule

Our analysis of contractor costs indicates that during fiscal year 2008, MDA contractors collectively overran budgeted costs by $152.4 million. These overruns occurred in 11 of 14 MDA contracts we reviewed, with the STSS contract accounting for more than 50 percent of the total. Based on cost performance during the fiscal year and using formulas accepted within the cost community, we estimate that at completion the cumulative overrun in the contractors’ budgeted costs could be from about $2.0 billion to $3.0 billion. Our projections are based on the current budgeted costs at completion for each contract we assessed, which represents the total current planned value of the contract. However, the budgeted costs at completion, in some cases, have grown significantly over time. For example, the ABL contractor reported budgeted costs at completion totaling about $724 million in 1997, but as depicted in table 5, that cost has since grown to about $3.6 billion. Our assessment only reveals the overrun or underrun since the latest adjustment to the budget at completion. It does not capture, as cost growth, the difference between the original and current budgeted costs at completion. As a result, comparing the underruns or overruns for MDA programs in table 5 with cost growth on major defense acquisition programs is not appropriate because those major defense acquisition programs have established their full scope of work as well as developed total cost baselines, while these have not been developed for MDA programs. Our analysis is presented in table 5. Appendix II provides further details on the cost and schedule performance outlined in the table.

Technical difficulties caused most elements to overrun their fiscal year 2008 budgeted costs. For example, STSS attributed most of its overrun of approximately $87.9 million to hardware problems on the program’s second space vehicle including the flight communication box and the main spacecraft computer. The box overheated during testing which required a thorough test of the unit. Upon successful completion of this testing, it was determined it did not require a replacement. In addition, the program had a failure in the main spacecraft computer for which the program office initially recommended the removal of the entire computer from the spacecraft. However, after extensive research and testing, the program manager determined that the event with the spacecraft was an unverifiable failure with a low probability of occurrence and low mission impact and decided not to remove the computer from the spacecraft to resolve the issue. However, as a result of these issues, the launch was delayed from April 2008 to at least the third quarter of fiscal year 2009.

The MKV Task Order 6 and ABL contractors also experienced technical difficulties. The MKV contractor for Task Order 6 reported cost overruns during the fiscal year of $1.4 million due mostly to software development issues and late delivery of government-furnished components. ABL’s fiscal year cost overruns of $10.6 million were mainly related to late deliveries of key laser system components and the acquisition or refurbishment of the Beam Control/Fire Control system components. For example, a major component of the laser system required redesign and fabrication, delaying planned delivery and installation onto the aircraft. Also, multiple Beam Control/Fire Control hardware components either were not refurbished to specification or failed initial testing, delaying delivery and integration testing. The overall effect was an approximate 1-month slip that the contractor believes will be made up in time to make the current lethality demonstration planned for the end of fiscal year 2009.

Three elements’ contracts—Aegis BMD’s contract for 27 SM-3 Block 1A missiles, MKV Task Order 7, and GMD—performed below their fiscal year budgeted costs by nearly $58.3 million with the GMD element accounting for approximately $53.9 million of that. The GMD element’s underruns occurred partially because the contractors delayed or eliminated some planned work. For example, the GMD program did not emplace the three GBIs it expected to in fiscal year 2008 or conduct either of its two planned flight tests as scheduled during the fiscal year. As a result, it employed less labor than originally intended. Drivers for the MKV Task Order 7 contract’s fiscal year cost underruns include a restructuring of the effort and decisions to use one rather than several approaches for coordinated attack, and using less manpower than originally planned in its procurement and software efforts. Lastly, the Aegis BMD contract for 27 SM-3 Block 1A missiles underran its fiscal year 2008 budget by approximately $3 million due in part to spending less than planned for engineering efforts with the missile’s third stage component as well as adjustments made in program management, labor efficiencies, and material transfers in the missile’s fourth stage component.

While Some Tests Succeeded, Others Were Deferred; Overall System Performance Cannot Yet Be Assessed

Although several tests showed progress in individual elements and some system-level capabilities, all BMDS elements experienced test delays and shortfalls, in part due to problems with the availability and performance of target missiles. Most significantly, GMD was unable to conduct either of its planned intercept attempts during fiscal year 2008, however it was able to conduct one delayed intercept test in December 2008. As a result, key performance capabilities of the current configuration of the GMD kill vehicles may not be demonstrated and the new configuration is being fielded prior to flight testing. As a consequence of testing problems, none of the six MDA Director’s test knowledge points for 2008 were achieved. Poor performance of targets continues to be a problem that caused several tests to either fail in part or in whole. Shortfalls in testing have delayed validating the models and simulations that are used to assess the overall performance of the BMDS as a whole. Consequently, comprehensive assessments of the capabilities and limitations of the BMDS are not currently possible and therefore MDA still does not have the capability to model or simulate BMDS capability from enemy missile launch to its engagement.

Test, Targets, and Performance Challenges Continued during Fiscal Year 2008 for Several Elements

During fiscal year 2008, all BMDS elements experienced delays in conducting tests, most were unable to accomplish all objectives, and performance challenges continued for many. Moreover, the inability of MDA to conduct its full fiscal year 2008 flight test campaign as planned precluded the agency from collecting key information specified by the Director, MDA—known as Director’s test knowledge points—to make certain decisions at critical points in some BMDS programs. Table 6 below summarizes test results and target performance for BMDS elements during the fiscal year.

As a result of test delays, MDA restructured its flight test plan for fiscal year 2009, increasing the number of tests and compressing the amount of time to analyze and prepare for subsequent tests. For example, MDA plans to conduct 14 of 18 flight tests in the third and fourth quarter of fiscal year 2009. MDA’s past performance raises questions about whether this is realistic. In fiscal year 2008, MDA had planned to conduct 18 flight tests, but it only accomplished 10, plus it had several flight tests delayed into 2009 from 2008. An MDA official acknowledged that the 2009 plan is aggressive, but stated that it can be achieved. Specifics of each element’s testing experience during fiscal year 2008 follow.

According to Aegis BMD officials, budgetary constraints prompted the Aegis BMD element to delay some tests, reducing the number of tests planned for 2008. However, the program was able to successfully complete its first test involving two non-separating targets, conduct a short-range ballistic missile intercept, and participate in a THAAD intercept during the fiscal year. The program also planned to participate in a BMDS-level ground test during the year, but the test was delayed until at least the second quarter of fiscal year 2009 because of real-world events. Finally, Aegis BMD standard missile flight tests showed that interoperability issues persist between THAAD and Aegis BMDS with respect to correlation and object reporting.

ABL experienced delays during fiscal year 2008, but achieved all of its primary test objectives. The program planned to complete the installation of its high energy laser on the aircraft by June 2008 in preparation for testing. However, it was not completed until August 2008 because of problems with activating some of the laser’s subsystems. The program delayed the final testing of the laser until the problems could be resolved. Once the problems were resolved, the ABL program was able to complete functionality testing of the laser in September 2008.

C2BMC experienced delays in conducting tests, but achieved several test objectives. For example, software upgrade verification testing slipped from fiscal year 2008 to 2009 but the program was able to participate in many other system-level ground and flight tests during the year that enabled the program to demonstrate multiple capabilities, including situational awareness and sensor management. The C2BMC element extended development for its next software release, 6.4, by more than a year because of delays in system-level BMDS testing and challenges in developing the network server for version 6.2, as well as unplanned work to incorporate effects from earth rotation in the 6.4 C2BMC planning architecture. C2BMC added earth rotation effects to address a requirement that Spiral 6.2 and later releases have the ability to model the true extent of ranges for long-range threats. Finally, C2BMC is still developing its capability to generate a single track from multiple sensors through a new resource management function, the Global Engagement Manager. For example, the development team for this function had to modify the design of this function’s new track processing that experienced an unacceptable level of delays when processing data.

In fiscal year 2008, the GMD program was unable to conduct either of its two planned intercept attempts—FTG-04 and FTG-05. MDA first delayed and then later canceled the FTG-04 test in May 2008 due to a problem with a telemetry component in the interceptor’s Exoatmospheric Kill Vehicle (EKV) needed to gather test data. MDA also delayed FTG-05 from fiscal year 2008 and conducted it in December 2008. Over the past two years MDA had expected to conduct seven GMD interceptor flight tests by the end of the first quarter of fiscal year 2009. However, MDA was only able to conduct two, as shown in figure 3.

The cancellation of FTG-04 raised concerns within the test community and members of Congress. FTG-04 was at first delayed and then canceled. MDA replaced it with a test to assess sensor capability—FTX-03. The sensor test allowed GMD to verify fire control software and integration with multiple sensors. The DOT&E was not consulted on the decision to cancel FTG-04 and expressed concern that the elimination of any intercept test reduced the opportunity to gather data that might have increased confidence in the models and simulations. In the conference report accompanying the National Defense Authorization Act for Fiscal Year 2008, conferees expressed concern about the loss of the FTG-04 flight test and requested that we review the circumstances and the effects on the BMDS. Details of our review of the FTG-04 flight test cancellation appear in appendix III.

Because GMD conducted FTG-05 in December 2008, there are only two full sets of GMD intercept data to date available for analysis which limits the ability to verify and validate the models and simulations. Additionally, FTG-04 and the subsequent test—FTG-05—were planned to present different stresses to the kill vehicle which would provide critical data needed to further verify the fielded configuration of the kill vehicle. The cancellation and subsequent restructuring of the first test caused a delay in FTG-05 from the third quarter of fiscal year 2008 until December 2008. In the FTG-05 test, the interceptor hit its intended target. However, MDA judged the target as a failure because it failed to release its countermeasures as planned. Consequently, all primary test objectives were not achieved.

Looking forward to the next GMD intercept flight test—FTG-06 in at least the fourth quarter of fiscal year 2009—MDA is accepting a higher level of risk than it previously expected in conducting this first test of the CE-II EKV because it will contain several objectives that had planned to be previously tested, but have not been. MDA had set up an approach to test one new major component change at a time. For example, MDA had planned to test the CE-I EKV first against simple targets, then the CE-I against a complex target, and once that had been proven MDA planned to test the CE-II EKV against a complex target. However, MDA was not able to test the CE-I EKV against a complex target due to a target failure. Due to testing problems, GMD has only been able to assess the CE-I EKV with a target without countermeasures. As a result, the FTG-06 flight test will be the first GMD test assessing both a CE-II EKV and a complex target scene. Adding to the risk, this will be only the second test using a newly developed FTF LV-2 target.

During fiscal year 2008, the KEI program experienced problems during testing that required it to rework components which, in turn, caused a delay to subsequent testing. More importantly, due to technical issues experienced by the program over the past two years, the first booster flight test—a key decision point for the program—has been delayed by nearly a year and is not scheduled to occur until at least the fourth quarter of fiscal year 2009. Technical difficulties delayed the MKV program’s fiscal year 2008 hover test until fiscal year 2009. This hover test will allow the program to integrate and test key components of the system in a repeatable ground-based free flight environment as their technologies reach maturity. Although originally planned for the fourth quarter of fiscal year 2008, the test was successfully conducted in December 2008. The STSS program encountered problems during testing that forced the program to delay the launch of its demonstration satellites from April 2008 to at least the third quarter of fiscal year 2009. The program continued to experience technical difficulties with its space vehicles. For example, during testing, the program experienced problems with its main spacecraft computer as well as an overheating flight communications box. After extensive testing, the program determined that these components were acceptable for flight.

Similarly, in fiscal year 2008, the Sensors element also experienced flight test delays as well as difficulties in achieving planned objectives due to target performance, but met some primary objectives. The element successfully participated in other tests during the fiscal year which demonstrated the ability for the sensors to acquire and track a target. One test event, FTX-03, provided the first opportunity for four key sensors— Sea-based X-band radar, AN/TPY-2, Upgraded Early Warning Radar, and an Aegis BMD radar—to operate in a more operationally realistic test scenario. This test demonstrated the capability for the sensors to correlate target information in order to conduct an intercept test. However, the target failed to release its countermeasures as planned. This failure precluded sensors from assessing capability against a dynamic lethal target scene with countermeasures. As a result, the sensors could not collect all of the expected data, which delayed the element’s ability to develop algorithms needed for the discrimination capability. These objectives will need to be addressed in future testing. The BMDS Operational Test Agency has had ongoing concerns regarding the formatting, tracking, and accounting of messages from GMD sensors. The timely reception of messages from sensors to weapon systems is key to support decisions and achieve effective intercepts. Since 2000 the BMDS Operational Test Agency has reported these concerns to MDA about poor data collection and management practices involving sensors affecting its assessment of tests. These data management problems prevented the analysis of message data, according to BMDS Operational Test Agency officials. In response, the contractor proposed a message monitoring system among communications nodes. Consequently, MDA recommended that this issue be closed out, but the BMDS Operational Test Agency still considers the matter to be open because GMD has not funded the monitoring system.

THAAD planned to conduct three intercept attempts, but due to a target failure, it was only able to conduct two. The program could not complete its final flight test of the fiscal year because the target experienced an anomaly during flight. The test was planned to be a BMDS-level event and was designated as a developmental test/operational test mission utilizing multiple BMDS elements and operationally realistic criteria. The program also expected to demonstrate that it could launch more than one THAAD interceptor during the engagement. Program officials rescheduled this test for the second quarter of fiscal year 2009. In addition, THAAD’s radar data collection test, RDC-2, was planned for 2008 but was deleted due to target availability and funding. As a result, program officials told us that these test objectives will be covered in the future with hardware-in-the-loop simulations and other radar events. The program successfully completed its first two planned tests for the fiscal year. In October 2007, THAAD successfully demonstrated an intercept of a target outside the earth’s atmosphere. This was the first time THAAD had successfully conducted an intercept outside of the atmosphere since 1999. Additionally, in June 2008, THAAD completed a successful intercept of a separating target. This intercept utilized warfighter procedures developed by the U.S. Army Air Defense School.

Key MDA Test Knowledge Points Not Achieved

As a consequence of flight test delays as well as a delay in a key ground test, MDA was unable to achieve any of the Director’s test knowledge points scheduled for fiscal year 2008 as shown in table 7.

In May 2007, the Director, MDA, established key system-level and element- level knowledge points to provide critical information for making key decisions regarding the BMDS. According to MDA, these knowledge points are unique management approaches chosen to manage MDA’s critical program risks. Each knowledge point is based on an event that provides critical information—or knowledge—for a key MDA decision requiring the Director’s approval.

In fiscal year 2008, among the Director’s test knowledge points delayed, MDA had to defer the confirmation of the 72” target performance due to delays in qualifying components. Additionally, MDA had to delay the confirmation of the booster for the KEI program as problems were encountered during testing of the nozzle.

Poor Target Missile Performance Continues to Hamper BMDS Testing

While targets have caused problems in fiscal year 2008 testing, poor performance of targets is not new. Targets’ reliability and availability problems have significantly affected BMDS development and testing since 2006, and issues have grown even more problematic in recent years. Although target anomalies and failures have affected many of the missile defense elements, THAAD and GMD have been most affected. In 2006, the THAAD program was unable to achieve its first intercept attempt (FTT-04) because the target did not function properly. In 2007, two THAAD radar characterization tests (RDC-1c and RDC-1d) were unsuccessful due to target anomalies. These tests flew targets with characteristics needed for radar observation in support of advanced discrimination algorithm development. However, target problems prevented an opportunity for the radar to exercise all of the planned algorithms, causing a loss of expected data. In addition to target failure issues, the THAAD program deferred some flight tests because targets were not available, which cost the program about $201 million. GMD also experienced similar long-term effects on its flight test schedule when it was unable to achieve primary test objectives in a 2007 intercept attempt (FTG-03) due to a target failure.

MDA’s existing targets are becoming less capable of meeting requirements for near-term flight tests. These targets are aging and likely to grow even less reliable with time; some components, such as the rocket motors, are more than 40 years old. Among other things, MDA’s Targets and Countermeasures program office has also had problems incorporating requirements into contracts and has experienced problems obtaining supplies as vendors left the market due to the lack of business.

To address the growing need for more sophisticated and reliable targets for the future BMDS test program, MDA was developing a new family of targets called the FTF, which was originally intended to be a family of new short, medium, and long-range targets with ground-, air-, and sea- launch capabilities. MDA embarked on this major development without estimating the cost to develop the family of target missiles. MDA proceeded to develop and even to produce some FTF targets without a sound business case and, consequently, their acquisition has not gone as planned. The funds required for the FTF were spent sooner than expected and were insufficient for the development. Getting the FTF target’s components through the qualification process, however, was more difficult and costly than the program expected. For example, MDA originally planned to launch the first FTF target—a 72-inch LV-2—in a 2008 STSS flight test, but the test was rescheduled due to delays in satellite integration and target affordability and availability. While many of the target missile’s components are found on existing systems, their form, fit, function, and the environment they fly in have been changed for the 72- inch LV-2 target. Consequently, many critical components initially failed shock and vibration testing and other qualification tests and had to be redesigned. The process was recently scheduled to be complete in early October 2008 but, after several delays, was not finished until December 2008. Despite this, MDA expects the target to be complete and ready for its first launch in a third quarter fiscal year 2009 Aegis BMD flight test (FTM- 15).

We recently reported that the FTF has been delayed, costs have increased and exceeded $1 billion, and the program’s scope has been reduced. Work on all but one of the FTF target variants, the 72-inch LV-2, was canceled in June 2008, including plans for development and production of the second type of FTF target, the 52-inch, originally scheduled to launch in 2009. With guidance from the Missile Defense Executive Board, MDA is currently conducting a comprehensive review of the targets program to determine the best acquisition strategy for future BMDS targets. It is expected to be completed in mid-2009. Whether or not MDA decides to restart the acquisition of the 52-inch targets, or other FTF variants, and the nature of those targets depends on the results of this review.

Currently, MDA has narrowed its FTF development efforts, focusing on a single vehicle, the 72-inch LV-2 ground-launched target. The first launch was supposed to determine the viability of the FTF concept and the feasibility of discontinuing the use of existing targets. However, rather than first conducting a separate developmental test to confirm the target’s capability, MDA has chosen a much riskier approach. The first launch of the new LV-2 target will be in an Aegis BMD intercept test. Aegis BMD originally planned to use this new target in a fiscal year 2008 flight test; however, because the target was not ready, the test is delayed until at least the third quarter of fiscal year 2009.

Repeated target problems and test cancellations have also affected the development of capabilities needed to discriminate the real target from countermeasures. Without opportunities to test the functionality of the software, there now is a system-level shortfall in BMDS progress toward developing a target discrimination capability against more sophisticated countermeasures in the midcourse phase of flight. In order to improve the effectiveness of the BMDS against evolving threats, MDA elements are developing advanced discrimination software in their component’s sensors. The advanced discrimination software is critical to distinguish the threat re-entry vehicle from associated countermeasures and debris. Target failures during tests prevented opportunities to gather data to assess how well discrimination software performs in an operational environment.

Overall Performance of BMDS Can Not Yet Be Assessed

MDA’s modeling and simulation program enables MDA to assess the capabilities and limitations of how BMDS performs under a wider variety of conditions than can be accomplished through the limited number of flight tests conducted. Flight tests alone are insufficient because they only demonstrate a single collection data point of element and system performance. Flight tests are, however, an essential tool used to both validate performance of the BMDS and to anchor the models and simulations to ensure that they accurately reflect real performance. Computer models of individual elements replicate how those elements function. These models are then combined into various configurations that simulate the BMDS engagement of enemy ballistic missiles.

To ensure confidence in the accuracy of modeling and simulation in representing BMDS capability, the program goes through a process called accreditation. Element models are validated individually using flight and other test data and accredited for their intended use. MDA intends to group these models into system-level representations according to user needs. One such grouping is the annual performance assessment, a system-level end-to-end simulation that assesses the performance of the current BMDS configuration. The performance assessment integrates element-specific models into a coherent representation of the BMDS. Performance assessments are used to: assess objectives from MDA’s Deputy of Engineering and the BMDS Operational Test Agency, support MDA decisions about engagement sequence group capability deliveries, and support MDA decisions about BMDS fielding and declaring capabilities.

Fundamentally, performance assessments anchored by flight and ground tests are a comprehensive means to fully understand the performance capabilities and limitations of the BMDS.

Developing an end-to-end system-level model and simulation has been difficult. BMDS Operational Test Agency officials told us that they do not anticipate a fully accredited, system-level model and simulation capability to be available until 2011. MDA’s first effort to bring together different element models and simulations to produce a fully accredited, end-to-end model and simulation for Performance Assessment 2007 was unsuccessful primarily because of inadequate data for verification and validation to support accreditation and a lack of common threat and environment input data among element models. Though Performance Assessment 2007 was a success in establishing a capability for integrated modeling and simulation in a short time frame, it was unsuitable to assess system-level performance due to low confidence from a lack of accreditation. Consequently, acting on a joint recommendation between MDA and the Operational Test Agency, MDA officials canceled their 2008 performance assessment efforts in April 2008 because of developmental risks associated with modeling and simulations, focusing instead on testing and models for Performance Assessment 2009. MDA officials believe that the refocused efforts will increase the chances for success during Performance Assessment 2009.

According to the BMDS Operational Test Agency’s January 2009 Modeling and Simulation Accreditation Report, confidence in MDA’s modeling and simulation efforts remains low although progress was made during the year. MDA is now exercising stronger central leadership to provide guidance and resources as it coordinates the development of verified and validated models and simulations, as recommended by a 2004 Defense Science Board study. MDA and element officials are now working more closely with the BMDS Operational Test Agency. For example, MDA and the BMDS Operational Test Agency have agreed on performance parameters and criteria used to validate element models and simulations. Nonetheless, BMDS Operational Test Agency officials stated that there are several weaknesses in the BMDS testing program such as: Insufficient consideration of modeling and simulation requirements in MDA flight test plans, though they emphasized that MDA is finalizing a list of such parameters for future flight test plans, Use of artificialities in flight tests which limit the realism of scenarios for anchoring models and simulations, and Inadequate test planning for comprehensive modeling of weather conditions.

MDA intends to verify and validate models and simulations by December 2009 for Performance Assessment 2009. However, BMDS Operational Test Agency officials stated that there is a high risk that the Performance Assessment 2009 analysis will be delayed because of remaining challenges and MDA’s slow progress in accreditation, as follows: The compressed schedule of ground and flight tests leaves little time for data analysis that is essential to anchor models to those tests, particularly for a complete analysis supporting MDA’s Performance Assessment 2009.

Out of 40 models, the BMDS Operational Test Agency recommended in January 2009 full accreditation for only 6 models, partial accreditation for 9 models, and no accreditation for 25 models.

Because MDA canceled the follow-on Performance Assessment 2008, the BMDS Operational Test Agency did not receive verification and validation data that would have been included in the modeling and simulation portion of its 2008 operational assessment.

BMDS Operational Test Agency officials told us that MDA also does not adequately plan for the collection of flight test data and post-flight reconstruction to support anchoring MDA models and simulations, even though post-flight reconstruction is needed to validate that models and simulations are adequate representations of the real world for their intended purpose. MDA guidance emphasizes that one of the primary objectives of the MDA ground and flight test program is to anchor BMDS models and simulations. Additionally, this guidance requires MDA’s testing program to work with the MDA engineers to define a test program that anchors these models and simulations across the operating spectrum. According to BMDS Operational Test Agency officials, the first full post- flight reconstruction was conducted in December 2008.

Despite the guidance delineating responsibilities for test data, MDA test plans currently do not include enough detail to allocate and synchronize resources in order to anchor models and simulations. MDA recently initiated a three-phase review of the entire BMDS test program. According to MDA, this three-phase review will emphasize the need for basing BMDS test planning and test design on critical factors that have not been proven to date and will drive target selection requirements. One outcome of the review will be to create integrated campaigns of ground and flight tests to efficiently collect data needed to validate the models and simulations. MDA intends to complete all three phases of the review by May 2009, after which MDA intends to have a date when all MDA models and simulations will be verified and validated. However, the current lack of flight test data for MDA’s and BMDS Operational Test Agency analysis prevents the timely validation of models and simulations that are used to build the 2009 end- to-end performance assessment.

Production, Fielding, and Declaration of Capabilities Proceed despite Delays in Testing and Assessments

In fiscal year 2008, MDA met most of its delivery goals. However, it continued to pursue a concurrent development, manufacturing and fielding strategy in which assets are produced and fielded before they are fully demonstrated through testing and modeling. Although flight tests and modeling and simulation produced less validation of performance than planned, MDA continued manufacturing untested components and declaring capabilities ready for fielding. For example, 10 of the new configuration kill vehicles for the GBI will have been manufactured and delivered before being flight-tested. MDA also declared that it had fielded 9 of 22 BMDS capabilities planned for 2008 (postponing 13), but due to test cancellations and performance assessment delays, it had to change the basis of these declarations, often relying on previous, less realistic testing.

MDA Met Most 2008 Asset Delivery Goals

MDA achieved four of the five delivery goals it set for fiscal year 2008 as shown in the table 8.

The agency planned to deliver the Sea-based X-band radar and three additional GBIs for Block 1.0, 20 additional SM-3 missiles for its Block 2.0 capability, a C2BMC site for fielding and activation for its Blocks 3.1/3.2 and 5.0 capabilities, and an additional AN/TPY- 2 radar. Although partial capability for the Sea-based X-band radar will not be declared until at least fiscal year 2009, it was approved for Early Capability Delivery in fiscal year 2008. The agency delivered the Aegis BMD SM-3 missiles, the AN/TPY-2 radar, and the C2BMC site in fiscal year 2008 as planned, but was unable to deliver the GBIs because the GMD element encountered development challenges with components for the CE-II EKV. In addition, the Navy Commander, Operational Test and Evaluation Force declared the Aegis BMD 3.6 system as operationally suitable and effective in October 2008. This decision signifies that 18 Aegis BMD-equipped ships and 90 SM-3 missiles are ready for transition to the Navy.

Production and Fielding of BMDS Systems Getting Ahead of Testing

Despite developmental problems, test delays and MDA’s inability to complete all fiscal year 2008 Director’s test knowledge points, manufacturing, production, and fielding have proceeded close to schedule. In some cases fielding has gotten ahead of testing. For example, Aegis BMD expected to assess the ability of the SM-3 Block 1A missile to engage and intercept a long-range ballistic target to satisfy a Director’s test knowledge point. Even though that test has been delayed until the third quarter of fiscal year 2009, MDA purchased 20 SM-3 Block 1As in fiscal year 2008.

Furthermore, MDA intended to assess, through flight tests, the CE-I EKV’s capability against scenarios that included complex target scenes with countermeasures. However, due to the frequent restructuring of its test plan and a target failure during its most recent flight test, the fielded configuration for GMD has not completed a test against countermeasures. According to MDA, no more CE-I flight tests have been approved, although the agency is considering additional flight testing of the CE-I EKV in the future. Moreover, earlier ground and flight testing, along with manufacturing discoveries prompted the GMD program to initiate a refurbishment program for the kill vehicles and the boosters. Refurbishment consists of: (1) reliability improvements to address high priority risks and to support the development and understanding of GBI reliability and (2) surveillance of aging through the examination of removed components. Consequently, the capability of the CE-I, including improvements designed to mitigate risk, as well as understand its capabilities and limitations against targets employing countermeasures may not be flight-tested, yet all 24 interceptors with this configuration are already emplaced and declared operational.

More importantly, the GMD program continues to experience test delays, causing fielding to outpace flight tests as shown in figure 4.

For example, the program has only been able to conduct two intercepts since 2006 for verifying the fielded configuration yet the production of interceptors continues. According to GMD’s September 2006 flight test plan, for fiscal years 2007 and 2008, and the first quarter of fiscal year 2009 it was going to conduct seven flight tests, including a test that would utilize 2 GBIs against a single target—known as a salvo test —and field 16 new GBIs. By January 2009 GMD had changed its plan, removing the salvo test and conducting two flight tests, yet it fielded 13 GBIs.

Similarly, GMD had planned to conduct an intercept test to assess the enhanced version of the EKV called the Capability Enhancement II in the first quarter of fiscal year 2008, months before emplacing any interceptors with this configuration. However, developmental problems with the new configuration’s inertial measurement unit and the target delayed the first flight test with the CE-II configuration—FTG-06—until at least the fourth quarter of fiscal year 2009. Despite these delays, MDA expects to have emplaced five CE-II interceptors before this flight test. MDA indicated that these five interceptors will not be declared operational until the satisfactory completion of the test and the Program Change Board declares their status. However, MDA projects that 10 CE-II EKVs will have been manufactured and delivered before that first flight test demonstrates the CE-II capability. This amounts to over half of the CE-II EKV deliveries that are currently on contract.

MDA did not emplace the three GBIs it needed to meet its fiscal year 2008 fielding goals. MDA will have to emplace twice as many GBIs than planned in fiscal year 2009 before Block 1.0 can be declared complete. As of January 2009, the agency had emplaced two and must emplace four more in order to complete Block 1.0 as planned.

Major defense and acquisition programs must complete operational test and evaluation before entering full-rate production. Because MDA considers the assets it has fielded to be developmental, it has not advanced BMDS elements to DOD’s acquisition cycle or begun full-rate production. Therefore, MDA has not yet triggered the requirement for an operational test and evaluation prior to fielding. However, MDA’s concurrent approach to developing and fielding assets has led to testing problems and concerns about the performance of some fielded assets. After two flight test failures in 2005, MDA undertook a Mission Readiness Task Force to establish confidence in GMD’s ability to reliably hit its target, establish credibility in setting and meeting test event dates, build increasing levels of operationally realistic test procedures and scenarios, raise confidence in successful outcomes of flight missions, and conduct the next flight test as soon as practical within acceptable risk bounds. However, GMD accelerated the objectives for its test program after the first Mission Readiness Task Force flight test and the program continues to experience developmental problems with key interceptor components. MDA also separately established a refurbishment program designed to replace questionable interceptor parts and increase reliability of GBIs. Since 2006, we have reported that the performance of some fielded GBIs was uncertain.

Despite MDA’s previous efforts to build confidence in its test program, MDA continues to pursue a risky approach for fielding BMDS assets under its new block structure. In March 2008, we reported that MDA’s new block structure did not address whether it would continue its practice of concurrently developing and fielding BMDS elements and components. However, in 2008 the agency continued to field assets without adequate knowledge. MDA emplaced GBIs during the year although its refurbishment program was barely underway, meaning that the risks of rework continue. To date, 26 GBIs have been emplaced—many of which may contain unreliable parts—and only a few have been refurbished since the initiation of the refurbishment program in 2007. According to program officials, some improvements have already been introduced into the manufacturing flow and demonstrated during flight testing.

While it is always a concern when tests are eliminated or the complexity of a planned test is reduced, the concern is heightened for a system of systems such as the BMDS because of the complex interaction of components within an element, and between that element and the other elements within the BMDS. Consequently, the need to synchronize the development and testing of different capabilities is crucial before fielding begins. For example, for certain engagement scenarios, the ground-based interceptor will launch based on information provided by an entirely separate element such as an Aegis cruiser or destroyer. If a problem is discovered during these flight tests, post-flight reconstruction using models needs to be conducted, the root-cause must be determined, a solution or mitigation must be developed and implemented, and a new test to confirm the effectiveness of the solution or mitigation must be performed.

Reduced Testing Has Delayed Some Capability Declarations and Weakened the Basis for Others

When MDA determines that a capability can be considered for operational use, it does so through a formal declaration. MDA uses an incremental declaration process to designate BMDS capability for its blocks in three levels—early, partial and full. The first two levels allow these BMDS features to play a limited role in system operations before they have attained their full level of capability. Each capability designation in the delivery schedule represents upgraded capacity to support the overall function of BMDS in its mission as well as the level of MDA confidence in the system’s performance. Capability declarations are important because MDA uses them to assess progress toward block completion. MDA guidance calls for an orderly sequence of events that lead to declaring that a fielded capability has been achieved and is ready for consideration for operational use.

MDA bases its declarations on, among other things, a combination of models and simulations—such as end-to-end performance assessments— and ground tests all anchored to flight test data. Because performance assessments analyze the BMDS as an entire system in a variety of ways, they provide more comprehensive information than flight tests alone. These events and assessments build on each other every year as MDA adds capabilities by improving hardware and software. MDA decision makers would then declare the achievement of capability goals for engagement sequence groups based on performance assessments. While in some instances, declarations of capability have been deferred, in other instances MDA has declared capabilities despite shortfalls in testing, modeling and simulation, and performance assessments. The agency declared the delivery of nine capabilities during fiscal year 2008 as shown in figure 5 below. It declared three early capabilities for Block 1.0 engagement sequence groups, three early as well as one full capability for Block 2.0 engagement sequence groups, and one early capability and one partial capability for Block 3.1/3.2 engagement sequence groups.

GBI Launch-on COBRA DANE radar (Beale)

GBI Ene-on ea-baed X-band radar GBI Launch-on ea-baed X-band radar SM- Ene-on hipboard Ae radar SM- Launch on remote hipboard Ae radar SM-2 Ene-on hipboard Ae radar THAAD Ene-on AN/TPY-2 radar (terminal mode)

GBI Ene on COBRA DANE radar Mod 1 (Fylindale, UK; forward-baed (AN/TPY-2 radar))

GBI Launch on hipboard Ae radar Mod 1 (Fylindale, UK; ea-baed X-band radar)

MDA had intended to use the results of a flight test (FTG-04), that was later canceled; a distributed ground test (GTD-03), that was delayed into fiscal year 2009; and the results of Performance Assessments 2007 and 2008 to determine if capabilities were ready for declaration in fiscal year 2008. Consequently, these shortfalls in knowledge led MDA to reduce the basis for declaring capability goals. Performance Assessment 2007— identified by MDA as a key source to assess capabilities during fiscal year 2008—achieved only limited accreditation. This less-than-full accreditation indicated that MDA could not rely on the assessment’s results to gauge end-to-end BMDS performance. Subsequently, MDA officials decided to cancel Performance Assessment 2008 because they needed time to address problems and prepare for Performance Assessment 2009.

While MDA officials declared these capabilities during fiscal year 2008, they did so after mostly reducing the basis for the declarations. They reverted back in several cases to older ground and flight tests, though MDA in a few cases added some newer flight and ground tests as well. For example, MDA declared early Block 1.0 capability for three engagement sequence groups in fiscal year 2008 without the planned results from Performance Assessment 2007. In all cases, though MDA had intended to use the final results from comprehensive performance assessments, after revising the basis for declaring capability goals it eliminated them entirely. Specifically, MDA dropped some sources of data it expected to use, such as the canceled Performance Assessment 2008, and shifted from flight and ground tests planned to occur in fiscal year 2008 to older flight and ground tests. For example, in Block 2.0 MDA declared full capability during fiscal year 2008 for one engagement sequence group, Aegis BMDS engage on its shipboard radar, even though Performance Assessment 2008 had been canceled. MDA instead based its decision on integrated and distributed ground tests (GTI-02 and GTD-02) conducted in calendar year 2007 as well as prior flight tests during fiscal years 2006 through 2008. However, the BMDS Operational Testing Agency raised concerns about the comprehensiveness of the GTI-02 scenarios, specifically, the incorrect configuration of U.S. satellites and threat data.

MDA also deferred 13 capability goals scheduled to occur in fiscal year 2008 to the end of fiscal year 2009, as shown in figure 6 below.

MDA intended to declare all Block 1.0 engagement sequence groups as fully capable by the middle of fiscal year 2009. However, as MDA encountered test delays and technical challenges, it had to defer full capability declaration for these engagement sequence groups until the end of fiscal year 2009. For Block 2.0, MDA also deferred declaring full capability for one of the two planned full capability declarations for fiscal year 2008. This declaration is contingent upon the review of a ground test that has been rescheduled to the second quarter of fiscal year 2009 and a flight test rescheduled to the third quarter of fiscal year 2009. MDA also deferred one full and two early capability declarations for Block 3.1/3.2 beyond the end of fiscal year 2009.

In response to the limitations of Performance Assessment 2007, the cancellation of Performance Assessment 2008 and FTG-04, and the delayed GTD-03 and FTG-05 flight tests, MDA is planning to rely on older ground and flight tests; a sensor flight test, FTX-03, instead of intercept flight tests; and the initial quick look review of Performance Assessment 2009 instead of the previously planned full analysis. Appendix IV provides a detailed layout for the reduced basis of capability declarations for fiscal years 2008 and 2009.

Since MDA was only able to declare a few of the capabilities it planned for fiscal year 2008, the schedule for fiscal year 2009 and subsequent years will be compressed if the agency plans to maintain the schedule it has set for its blocks. For example, MDA may need to declare three times as many capabilities than originally planned for fiscal year 2009 in order to meet the 2009 capability declaration schedule. In addition, if the schedule cannot be maintained, MDA will likely have to make further adjustments to mitigate additional delays in BMDS capabilities.

Increased reliance on integrated ground testing will provide less knowledge than a complete analysis of capabilities from a performance assessment. Integrated ground testing involves less robust conditions than distributed ground testing, which involves operational systems in the field. The MDA master fielding plan indicates that the agency originally intended to take a more comprehensive approach upon which to base capability declarations. Reliance on an upcoming Performance Assessment 2009 quick look for Block 1.0 completion is a particular concern because the knowledge it provides may be limited with respect to testing, according to BMDS Operational Test Agency officials. For example, officials told us that a quick look may indicate anomalies from a test but will not analyze their causes. In contrast, MDA originally planned to have a complete analysis from the Performance Assessment 2009 models, simulations, and tests.

Limited Progress Made in Improving Transparency and Accountability

In March 2008, we reported that efforts were underway to improve BMDS management, transparency, accountability, and oversight including a new executive board outside of MDA and a new block structure along with other improvements within MDA. Since that time, the executive board that was established in 2007 has acted with increased oversight. MDA’s efforts, however, have not made the expected progress. In particular, MDA has decided to retain the option of deferring work from one block to another; cost baselines have not been established; test baselines remain relatively unstable; and requesting procurement funds for some assets, as directed by Congress, will not occur until fiscal year 2010.

To accomplish its mission, in 2002 the Secretary of Defense gave MDA requirements, acquisition, and budget flexibilities and relief from some oversight mechanisms and reporting responsibilities. The flexibility granted to MDA has allowed concurrent development, testing, manufacturing, and fielding. MDA used this flexibility to quickly develop and field the first increment of capability in 2005. In August 2008, in response to Congressional direction to assess the current and future missions, roles, and structure of MDA, an independent study group agreed that there is a need to move MDA toward more normal acquisition processes. However, the group noted that the continuous evolution of the BMDS requires that the approach to setting requirements for, developing, and fielding increments of capability should remain as special authorities with oversight of the Missile Defense Executive Board (MDEB). Further, in regards to budget flexibilities, the independent group concluded that while these flexibilities may have been deemed necessary at the time, it should not have been expected that all the special authorities granted to MDA would continue or would have a need to continue in full force beyond achieving the President’s goal of deploying a set of initial capabilities.

Missile Defense Executive Board’s Oversight Role More Active in 2008

During 2008, the MDEB appeared to act with an increased level of authority in providing oversight of MDA and the BMDS. For example, the board took on a major role in making key decisions regarding the transition of elements to military services. We previously reported that MDA and the military services had been negotiating the transition of responsibilities for the sustainment of fielded BMDS elements, but this process had been proven to be arduous and time consuming. However, in 2008, with the influence of the MDEB, a lead military service designation was appointed for one BMDS asset—the Sea-based X-band radar.

In March 2008, we reported that the MDEB could play a key role in the Joint Requirements Oversight Council’s proposal to return the BMDS to the Joint Capabilities Integration and Development System requirements process—a formal DOD procedure followed by most DOD programs that defines acquisition requirements and evaluation criteria for future defense programs. In responding to the proposal, the Acting Under Secretary of Defense for Acquisition, Technology, and Logistics recommended that the Deputy Secretary of Defense delay his approval of the Joint Staff’s proposal until the MDEB could review the proposal and provide a recommendation. According to Acquisition, Technology and Logistics officials, no decision has been made regarding returning the BMDS to the requirements process. However, the Deputy Secretary of Defense, in September 2008, appeared to strengthen the oversight role of the MDEB, clarifying the roles of the MDEB as well as MDA, the Office of the Secretary of Defense, Combatant Commands, and Military Departments. With respect to the role of the MDEB, he established a life cycle management process for the BMDS stating that the MDEB will recommend and oversee implementation of strategic policies and plans, program priorities, and investment options to protect our Nation and our allies from missile attack. One of the MDEB functions is to provide the Under Secretary of Defense for Acquisition, Technology, and Logistics—or Deputy Secretary of Defense, as necessary—a recommended strategic program plan and feasible funding strategy for approval. The Deputy Secretary further noted that, through the use of the BMDS Life Cycle Management Process outlined in the memo, the MDEB will oversee the annual preparation of the BMDS portfolio, including BMDS-required capabilities and a program plan to meet the requirements with Research, Development Test & Evaluation, procurement, operations and maintenance, and military construction resources in defense-wide accounts.

To further increase BMDS transparency and oversight of the BMDS, the Under Secretary of Defense Acquisition, Technology, and Logistics plans to hold program reviews for several BMDS elements commensurate with the authority granted to the MDEB by the Deputy Secretary of Defense. According to Under Secretary of Defense for Acquisition, Technology, and Logistics officials, the MDEB conducted its first of such reviews in November 2008 of the THAAD program. This review covered production and the element’s contract schedule. Under Secretary of Defense for Acquisition, Technology, and Logistics officials told us that these reviews are designed to provide the Deputy Director Acquisition, Technology, and Logistics with comprehensive information that will be used as the basis for MDEB recommendations for the BMDS business case and baseline processes— a process which, according to these officials, is similar to the traditional Defense Acquisition Board process for reviewing other major acquisition programs. However, it is unclear whether the information provided to the MDEB will be comparable to that produced for other major acquisition program reviews as most of the information appears to be derived or presented by MDA as opposed to independent sources as required for traditional major defense acquisition programs.

Efforts to Improve Transparency of MDA’s Work Have Not Progressed as Planned

Deferral of Work

In 2007, MDA redefined its block structure to better communicate its plans and goals to Congress. The agency’s new structure is based on fielding capabilities that address particular threats instead of the biennial time periods previously used to develop and field the BMDS. Last year, we reported that MDA’s new block plans included many positive changes. However, MDA, with its submission of its Fiscal Year 2008 Statement of Goals, reversed some of the positive aspects of the new block structure. For example, we previously reported that the new block structure would improve the transparency of each block’s actual cost by disallowing the deferral of work from one block to another. Under its prior block structure, MDA deferred work from one block to another; but it did not track the cost of the deferred work so that it could be attributed to the block that it benefited. Because MDA did not track the cost of the deferred work, the agency was unable to adjust the cost of its blocks to accurately capture the cost of each. This weakened the link between budget funds and the work performed. Last year, MDA officials told us that under its new block approach, MDA would no longer transfer work under any circumstances to a different block. However, MDA officials recently said that they are retaining the option to move work from one block to another as long as it is accompanied by a rebaseline. This change allows the agency to continue the practice of moving work from one block to another, which thereby reduces the transparency of the new block structure and undermines any baselines that are established.

Use of Procurement Funds

In March 2007, we reported that the majority of MDA’s funding comes from the Research, Development, Test, and Evaluation appropriation account, another flexibility provided by law. In past years, Congress authorized MDA to pay for assets incrementally using research and development funds. This allowed MDA to fund the purchase of assets over multiple years. Congress recently restricted MDA’s authority and required MDA to purchase certain assets with procurement funds and directed that for any year after fiscal year 2009, MDA’s budget materials must delineate between funds needed for research, development, test, and evaluation, procurement, operations and maintenance, and military construction. Requiring MDA to request funding in these appropriation categories will mean that it will be required to follow the funding policies for each category. For example, using procurement funds will mean that MDA will be required to ensure that assets are fully funded in the year of their purchase, rather than incrementally funded over several years.

Congress directed in the 2008 National Defense Authorization Act, for any year after fiscal year 2009, that MDA’s budget materials delineate between funds needed for research, development, test, and evaluation, procurement, operations and maintenance, and military construction. We have previously reported that using procurement funds will mean that MDA generally will be required to adhere to congressional policy that assets be fully funded in the year of their purchase, rather than incrementally funded over several years. The Congressional Research Service reported in 2006 that “incremental funding fell out of favor because opponents believed it could make the total procurement costs of weapons and equipment more difficult for Congress to understand and track, create a potential for DOD to start procurement of an item without necessarily stating its total cost to Congress, permit one Congress to ‘tie the hands’ of future Congresses, and increase weapon procurement costs by exposing weapons under construction to uneconomic start-up and stop costs.”

In the 2008 National Defense Authorization Act, Congress also provided MDA with the authority to use procurement funds for fiscal years 2009 and 2010 to field its BMDS capabilities on an incremental funding basis, without any requirement for full funding. Congress has granted similar authority to other DOD programs. In the conference report accompanying the Act, Conferees cautioned DOD that additional authority will be considered on a limited case-by-case basis and noted that they expect that future missile defense programs will be funded in a manner more consistent with other DOD acquisition programs.

MDA did not request any procurement funds for fiscal year 2009. During our audit, the agency had not yet released the 2010 budget request to include such funding categories. However, MDA officials told us that the agency plans to request procurement funds for some of its assets in its fiscal year 2010 request, but could not elaborate on its plans to do so. Given that data was unavailable, it is unclear for which assets procurement funds will be requested or the extent to which the request will meet the direction given by Congress. According to MDA officials, information regarding its plans to request procurement funding will not be released until spring 2009.

Baselines

Baselines represent starting points against which actual progress can be measured. They are thus used to provide indications of when a program is diverting from a plan. Baselines can be established to gauge progress in different areas, including cost, schedule and testing. Overall, the BMDS does not have baselines that are useful for oversight. With regard to cost, we have already discussed the lack of total and unit cost baselines for missile defense as well as the frequency of changes in contract baselines. MDA made some progress with developing a schedule baseline for its blocks and their associated capabilities. The agency’s annual Statement of Goals identifies its schedule baseline as the fiscal year dates for early, partial, and full capability deliveries of hardware and functionality for a block. Thus, while MDA has changed its schedule for making declarations, the effect of the change can be determined by comparison with the original schedule.

MDA does not have test baselines for its blocks. The agency does however, have baselines for its test program, but revises them frequently. They are therefore not effective for oversight. The agency identified its Integrated Master Test Plan as the test baseline for the BMDS. However, as depicted in table 9, the agency has made a number of changes to the content of the baseline.

The official approved test baseline changes every year and there are numerous more informal changes that happen more frequently. Most of the annual revisions to the test baseline occur either because MDA has changed the substance of test, changed the timing of tests, or added tests to the baseline.

The Integrated Master Test Plan establishes the executable test program for the current fiscal year and extends through the following fiscal year. According to MDA, the test plan is updated quarterly based upon decisions from MDA leadership or formal decision-making forums such as the Program Control Board, and is also coordinated annually with external agencies such as the Office of the Director of Operational Test and Evaluation. However, as shown in table 9, there are several versions for a given quarter and as many as seven versions have been developed since the fiscal year 2008 baseline was established. It is unclear which Integrated Master Test Plan version MDA manages to at any given time. For example, in November 2008, we requested the latest version of the Integrated Master Test Plan and were told that the latest approved version was 8.4, which was revised in July 2008. However, the signature page for that version is from a prior version—version 8.1. Since there is no signature page referring to version 8.4, it appears that this version is unapproved though MDA officials told us that it was being used to manage the BMDS.

Agency officials maintain that the document is used to manage tests and associated requirements for key test events. However, it is unclear how well the baseline is integrated with other key aspects of testing such as the acquisition of targets needed to execute tests. For example, in some instances, targets are placed on contract for two or more years in advance of the planned tests. Yet, the test baseline—the Integrated Master Test Plan—does not appear to include events beyond the following fiscal year that are key to the BMDS test program. As we reported in September 2008, MDA officials acknowledged that its target contracts did not capture all element testing requirements and target baselines were not always established before targets contracts were signed.

Conclusions

In 2002, MDA was given unusual authorities to expedite the fielding of an initial BMDS capability. As this initial capability was fielded in 2005, it showed the benefits of these flexibilities. MDA has improved on this capability in the ensuing years, including 2008, the focus of this report. Today, the program is still operating at a fast pace, as production and fielding of assets outstrips the ability to test and validate them. A collateral effect of these flexibilities has been reduced visibility into actual versus planned progress. Some fundamental questions of an oversight nature are not yet answerable. For example, a comparison of actual versus planned costs at the system or asset level is not yet possible, nor is an assessment of the performance of the fielded system as a whole. Beginning in 2007, MDA began efforts to improve visibility into its actual performance, beginning with the new way of defining blocks, coupled with DOD’s creation of the MDEB. However, progress has been slow in some areas and value for money cannot be satisfactorily assessed. Delays are especially important in a program of this size, as a year delay in establishing cost baselines means another $8 billion to $10 billion may be spent in the meantime.

With the transition to a new administration, the deployment and subsequent improvement of an initial capability, a new agency Director, and a new block structure for managing the BMDS, an opportunity exists to revisit and strengthen the processes by which MDA operates. Looking to the future, decision makers in Congress and DOD face multi-billion dollar investment decisions in allocating funds both within MDA and between MDA and other DOD programs. At this point, a better balance must still be struck between the information Congress and DOD need to conduct oversight of the BMDS and the flexibility MDA needs to manage across the portfolio of elements that collectively constitute the system’s capability.

At this point, the balance does not provide sufficient information for effective oversight. In particular: Total cost and unit cost baselines have not been set and contract baselines are subject to frequent changes. Even if such baselines are set as planned, they will only capture about 26 percent of MDA’s work.

Less testing is conducted than planned, thus delaying the validation of the models and simulations needed to assess the overall performance of the BMDS. Moreover, test plans do not hold and are revised often, in many cases due to the poor performance of target missiles. The current test plan is at risk given its ambitious scope.

Manufacturing, production, and fielding are outpacing testing, modeling, and validation. Consequently, fielding decisions and capability declarations are being made with limited understanding of system effectiveness.

Recommendations for Executive Action

We recommend that the Secretary of Defense direct the MDEB to assess how the transparency and accountability of MDA’s acquisitions can be strengthened to enhance oversight, such as by adopting relevant aspects of DOD’s normal requirements, acquisition and budgeting processes, without losing the beneficial features of MDA’s existing flexibility.

In the near term we recommend that the Secretary of Defense direct MDA to undertake the following 10 actions: In the area of cost: 1. Complete total cost baselines before requesting additional funding for Blocks 2.0 and 3.0 and commit to a date when baselines for all blocks will be established. 2. Ensure that transfers of work from one block to another are transparent and reported as cost variances. 3. Provide additional unit costs reports, beyond flyaway unit costs, that incorporate both procurement and research and development funding so that there is a more comprehensive understanding of the progress of the acquisitions.

In the area of testing and performance: 4. Expand the BMDS test baseline to include tests scheduled beyond the first succeeding year of the plan to ensure its synchronization with BMDS contracts. 5. Ensure that DOT&E is consulted before making significant changes to the test baseline so that the tests planned provide DOT&E with sufficient data to assess the performance of the BMDS elements. 6. Ensure that planned test objectives include concrete data requirements anchoring models and simulations to real-world tests, synchronized with flight and ground test plans and that the effects on models and simulations of test cancellations, delays or problems are clearly identified and reported. 7. Reassess the flight tests scheduled for the end of fiscal year 2009 to ensure that they can be reasonably conducted and analyzed given targets and other constraints.

In the area of knowledge-based decisions: 8. Synchronize the development, manufacturing, and fielding schedules of BMDS assets with the testing and validation schedules to ensure that items are not manufactured for fielding before their performance has been validated through testing. 9. Conduct a flight test of the CE-I EKV against a complex target scene with countermeasures to complete MDA’s previous testing goal of understanding the performance capabilities of the first 24 fielded GBIs. 10. Strengthen the capability declarations by using the complete analysis from annual performance assessments as the basis for declaring engagement sequence groups as fully capable and block development as fully complete; otherwise, indicate the limitations of the capabilities and steps that MDA will take to reduce the risks.

Agency Comments and Our Evaluation

DOD provided written comments on a draft of this report. These comments are reprinted in appendix I. DOD also provided technical comments, which were incorporated as appropriate.

DOD fully concurred with 10 of our 11 recommendations and partially concurred with our recommendation that the Secretary of Defense direct MDA to synchronize the development, manufacturing, and fielding schedules of BMDS assets with testing and validation schedules to ensure that items are not manufactured for fielding before their performance has been validated through testing. Yet, even DOD’s response to this recommendation appears to be, in substance, concurrence.

DOD concurred with our recommendation that the Secretary of Defense direct MDA to ensure that transfers of work from one block to another are transparent and reported as cost variances. DOD noted in its response that MDA will report block baselines and variances annually to Congress in the BMDS Accountability Report. The Department further noted that for the purposes of unit cost reporting, MDA has defined a cost variance as a confirmed increase of 10 percent or more in block or unit costs when compared to the current cost baseline or 20 percent or more compared to the original cost baseline, stating that transfers of work creating such cost variances will be reported. The intent of our recommendation is to increase visibility into transfers of work between blocks regardless of the amount of the increase or the baseline status of the blocks. The trigger for reporting the variances selected by DOD will not necessarily provide that visibility. Given that only between 2 and 26 percent of BMDS block and capability development costs from fiscal year 2010 to 2013 will be baselined initially, visibility into transfers into blocks that are not yet baselined may not occur. Further, an increase may not be reported in the baselined block from which work is transferred because the transfer would actually yield a decrease in the cost of the baselined block. An increase would also not be reported in the receiving block if that block is not baselined or if the transfer did not increase costs above the threshold. MDA may need to consider a different approach to reporting that captures meaningful transfers of work into and out of blocks regardless of whether any of the blocks are baselined. MDA should work with Congress to determine what constitutes a meaningful or significant cost increase.

DOD also concurred with our recommendation that the Secretary of Defense strengthen the capability declarations by using the complete analysis from annual performance assessments. In responding to our recommendation, DOD noted that if there is limited performance assessment data, the overall capability assessment will factor in the knowledge gained from ground tests and flight tests against the identified risks. While we recognize that MDA is not always able to complete all of its planned tests in a given time period, when MDA decides to change the planned basis for its capability declarations to a different or reduced set of data it is important for the agency to clearly report the limitations that affect the capability declaration as well as the mitigation steps it is taking.

We are sending copies of this report to the Secretary of Defense and to the Director, MDA. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you, or your staff have any questions concerning this report, please contact me at (202) 512-4841. Contact Points for our offices of Congressional Relations and Public Affairs may be found on the last page of this report. The major contributors are listed in appendix VI.

Appendix I: Comments from the Department of Defense

Appendix II: BMDS Prime Contractors Exceed Budgeted Cost and Schedule Performance during Fiscal Year 2008

Based on our analysis of 14 Ballistic Missile Defense System (BMDS) elements’ prime contractor earned value management performance, we determined that collectively the contractors overran budgeted cost by $152.4 million and were behind schedule by approximately $107.4 million during the fiscal year. Our insight of the dollar gained or lost for each dollar invested is based on monthly earned value reports which are required of each BMDS program office’s prime contractor. These reports compare monthly progress to the cost or schedule performance baseline to reveal whether the work scheduled is being completed on time and if the work is being completed at the cost budgeted. For example, if the contractor was able to complete more work than scheduled and for less cost than budgeted, the contractor reports a positive schedule and cost variance. Alternatively, if the contractor was not able to complete the work in the scheduled time period and spent more than budgeted, the contractor reports both a negative schedule and cost variance. The results can also be mixed by, for example, completing the work under cost (a positive cost variance) but taking longer than scheduled to do so (a negative schedule variance).

We also used contract performance report data to base predictions of likely overrun or underrun of each prime contractor’s budgeted cost at completion. Our predictions of final contract cost are based on the assumption that the contractor will continue to perform in the future as it has in the past. In addition, since they provide the basis for our projected overruns, we also provide the total budgeted contract cost at completion for each contract we assessed in this appendix. However, the budgeted costs at completion, in some cases, have grown significantly over time. For example, in one case the budgeted cost at completion increased by approximately five times its original value. Since our assessment does not reveal, as cost growth, the difference between the original and current budgeted costs at completion it would be inappropriate to compare the underruns or overruns for MDA programs with cost growth on major defense acquisition programs since those major defense acquisition programs have established their full scope of work as well as developed total cost baselines, while these have not been developed for MDA programs.

Aegis BMD Contractors Experienced Mixed Performance during the Fiscal Year

The Aegis Ballistic Missile Defense (Aegis BMD) program manages two prime contractors for work on its two main components—the Aegis BMD Weapon System and the Standard Missile-3 (SM-3). We report on work under one of the two separate Aegis BMD SM-3 contract’s contract line item numbers (CLIN)on which we received sufficient performance data during fiscal year 2008. The first Aegis BMD SM-3 contract’s CLIN 9 was for the production of 20 Block 1A missiles which began in February 2007 and finished deliveries in August 2008. Deliveries were completed $7.5 million under budget on the contractor’s total budgeted cost of $179.0 million. The other Aegis BMD SM-3 contract’s CLIN 1 is for a fourth lot of 27 Block 1A missiles and began reporting performance data in August 2007 for work that is still ongoing. The weapon system contractor experienced cost growth and schedule delays while the SM-3 contractor for the ongoing CLIN 1 for 27 Block 1A missiles had mixed performance. Neither of these CLINs experienced a realignment during fiscal year 2008.

The Aegis BMD weapon system contractor experienced cumulative cost growth and schedule delays throughout the year. The Aegis BMD weapon system contractor overran budgeted cost and schedule during the fiscal year by $7 million and $5.1 million respectively. Although cumulative cost performance remains positive at $16 thousand, cumulative schedule performance continued to decline to negative $8.4 million. The negative cumulative schedule variance is driven by late engineering data, delays to qualification efforts, and the need to return components experiencing issues back to the vendor which required more time than originally planned. See figure 7 for cumulative cost and schedule performance during the fiscal year.

The program attributes the fiscal year cost and schedule overruns mainly to the additional time and testing needed to ensure that the weapon system fielded capability was what was originally promised to the warfighter. To account for some of the overruns, the program performed fewer risk reduction efforts for a future weapon system capability release. If the contractor continues to perform as it has during the fiscal year, we project that at contract completion in September 2010, the contractor will overrun its budgeted cost of $1.2 billion by between $1.9 million and $12.2 million.

The Aegis BMD SM-3 contractor, producing another lot of 27 Block 1A missiles under its CLIN 1, ended the fiscal year by underrunning budgeted costs by $3.0 million. The Aegis BMD SM-3 contractor for CLIN 1 work also ended the year with a negative $7.6 million schedule variance, which means that the contractor was unable to accomplish $7.6 million worth of planned work. Since reporting began in August 2007, cumulative and fiscal year variances are nearly equal with cumulative cost variances at a positive $3.3 million and cumulative schedule variances at negative $7.0 million. See figure 8 for a graphic representation of the cumulative cost and schedule variances during fiscal year 2008.

The contractor was able to accomplish fiscal year 2008 work for $3.0 million less than originally planned in part due to adjustments made in program management, labor efficiencies, reductions in vendor cost, and material transfers in the missile’s fourth stage component. The unaccomplished work in negative $7.6 million worth of fiscal year schedule variances is largely in the first, second, and fourth stages portion of work. In the first stage booster, the contractor attributes some of the negative schedule variance to more than a year delay in testing the first stage due to rework needed to correct errors in the original drawing packages. In addition, the contractor cites second stage component delivery delays as drivers for the negative schedule variance. Vendors were unable to deliver these components due to holdups in approving waivers, achieving recertification after test equipment failures, and property damage to facilities. Lastly, the contractor experienced delays in components for the fourth stage which also contributed to the unfavorable schedule variance. If the contractor continues to perform as it did through September 2008, our analysis predicts that, at completion in April 2010, the work under the contract could cost from $6.6 million less to $0.7 million more than the budgeted cost of $237.5 million.

ABL Contractor Overran Budgeted Fiscal Year Cost

For fiscal year 2008, the Airborne Laser (ABL) contractor overran fiscal year budgeted costs by $10.6 million but had a positive fiscal year schedule variance of $2.2 million. Despite some gains in its schedule variance during the fiscal year, the program still maintains negative cumulative cost and schedule variances of $84.8 million and $23.6 million respectively. The contractor mostly attributes the negative cumulative variances in cost and schedule to late beam control/fire control hardware deliveries. Despite a replan in June 2007, the ABL contractor did not perform any type of realignment during fiscal year 2008. Figure 9 shows cumulative variances at the beginning of fiscal year 2008 along with a depiction of the contractor’s cost and schedule performance throughout the fiscal year.

Technical issues with key components of the Beam Control Fire Control system that required new hardware or refurbishment of existing components as well as late deliveries of key laser system components are the primary drivers of the unfavorable fiscal year cost variance of $10.6 million. These issues have caused delays in integration and test activities for the overall ABL weapon system. Based on the contractor’s performance up through fiscal year 2008, we estimate that, at completion in February 2010, the contractor will overrun its budgeted cost of $3.6 billion by between $89.7 million and $95.4 million.

C2BMC Program Incurred Negative Cumulative and Fiscal Year Variances

Our analysis of the Command, Control, Battle Management, and Communications’ (C2BMC) cumulative contract performance indicates that the prime contractor’s performance declined during fiscal year 2008. The contractor overran its fiscal year 2008 budget by $9.8 million and did not perform $3.6 million of work on schedule. By September 2008, this resulted in an unfavorable cumulative cost variance of $24.3 million and an unfavorable cumulative schedule variance of $7.1 million. The main drivers for the negative cumulative cost variances were costs associated with unplanned work, increased technical complexity, and reduction to cost efficiency due to losing key staff. The contractor attributes the unfavorable cumulative schedule variances to software issues related to the global engagement manager and components of test training operations. Although the C2BMC contractor performed a replan in November 2006, the contractor did not perform any type of realignment during fiscal year 2008. Trends in cost and schedule performance during the fiscal year are depicted in figure 10.

The negative fiscal year cost variance of $9.8 million is driven mainly by problems in the performance of work under Part 4 and Part 5 of the contract. The Part 4 effort, which began in December 2005, includes completing several spiral capabilities, upgrading spiral suites, and implementing initial global engagement capabilities at an operations center. The Part 5 effort began in December 2007 and covers operations and sustainment support for fielded C2BMC; the delivery of spiral hardware, software, and communications; as well as development, planning, and testing for other spiral capabilities. The contractor was able to use reserves to cover some of its Part 4 unfavorable fiscal year cost variances.

The Part 5 fiscal year cost variance’s primary drivers are unexpected complexities with the network design, unplanned work that required more resources for developing the planner, and the extension of efforts past the completion date on the global engagement management portion of work. The unfavorable fiscal year schedule variance of $3.6 million is attributable to the Part 5 portion of work and primarily caused by an unexpected reallocation of resources off of the global engagement management portion of work to other areas, delays in requesting material procurement also for global engagement management, and a lagging schedule for building out a testing lab. If the contractor continues to perform as it has in the past, we predict that the contractor will overrun its budgeted cost of $1.0 billion at completion in December 2009 by between $37.1 million and $76.8 million.

GMD Contractor Maintained Negative Cumulative Cost and Schedule Variances throughout the Fiscal Year

The government and the Ground-based Midcourse Defense (GMD) contractor began a contract restructuring during the fiscal year to rephase and rescope on-going efforts to refine capability requirements and to adjust program content as well as perform weapon system integration, perform flight test planning, and work to develop the two-stage booster among other tasks. The ongoing realignment includes a proposal to add between $350 million and $580 million to the cost of the work under contract and to add 39 months to the period of performance.

The GMD contractor reports a cumulative negative cost variance of more than $1.0 billion that it attributes to technical challenges with its Exoatmospheric Kill Vehicle (EKV) as well as supplier component quality problems. The contractor also carries a total unfavorable cumulative schedule variance of $130.3 million, the bulk of which the contractor attributes to the technical issues connected with the Ground-based Interceptor (GBI), particularly the EKV. For example, during the fiscal year the program experienced difficulties in manufacturing the Capability Enhancement II (CE-II) EKVs. Although the CE-II EKVs are expected to provide better performance, the contractor produced the kill vehicles before completing developmental tests, discovered problems during manufacturing, incorporated a new design, and continued manufacturing them. Although these issues contributed unfavorable fiscal year cost variances of $42.7 million, the program was able to make up for these losses in other areas. The variances, depicted in figure 11 represent the GMD contractor’s cumulative cost and schedule performance over fiscal year 2008.

The GMD contractor did have a favorable fiscal year cost variance of $53.9 million, which it attributed to labor efficiencies in the ground system as well as less field maintenance support required than planned, and labor efficiencies in the deployment and sustainment portion of the work under the contract. However, the GMD element’s underruns occurred partially because the contractors delayed or eliminated some planned work. For example, the GMD program did not accomplish the emplacement of three GBIs, or conduct either of its two planned flight tests. As a result, it employed less labor than originally intended. The program also reports an unfavorable fiscal year schedule variance of $77.4 million which it attributes to an administrative error that occurred in September 2007. This error incorrectly adjusted the baseline to the booster effort in September which was then updated in October. However, it should also be noted that Missile Defense Agency (MDA) officials believe that ongoing adjustments to the GMD element’s baseline have skewed recent variances to such a degree that they should not be used to predict future costs. We did perform analysis based on the contractor’s reported performance through fiscal year 2008, and our analysis estimates that at contract end planned for December 2011, the contractor could overrun its budgeted cost of $14.9 billion by between $950.2 million and $1.25 billion.

KEI Cost and Schedule Performance Continued to Decline after Replan

Despite a replan in April 2007 and again in April 2008, the Kinetic Energy Interceptors (KEI) contractor continued to experience declining cost and schedule performance during the fiscal year. Although the contractor began the year with a positive cost variance, the contractor overran fiscal year 2008 budgeted costs by $8.3 million, ending the year with an unfavorable cumulative cost variance of $2.6 million. In addition, the program was unable to accomplish $8.5 million worth of work which added to an unfavorable cumulative schedule variance of $21.3 million. Cumulative cost and schedule variances were mainly driven by costs associated with delays to booster drawing releases, delays in procurement, and unexpected costs and rework related to issues with the second stage. Figure 12 depicts the cost and schedule performance for the KEI contractor during fiscal year 2008.

The KEI contractor attributes its unfavorable fiscal year cost and schedule variances of $8.3 million and $8.5 million, respectively, to issues with its interceptor booster. Problems initially arose in fiscal year 2007 with a motor case failure during acceptance testing which led to unexpected redesigns. In October 2007, the program experienced several issues with the nozzle during a second stage ground test and also experienced a deviation in measured performance from pre-test predictions. These issues added costly redesigns and delays to its knowledge point, a booster flight test. The program performed a replan of its work in April 2008 because of these issues in order to realign the schedule with their booster flight test knowledge point which was delayed from August 2008 to April 2009. Since the replan, the booster flight test has been further delayed to the fourth quarter of fiscal year 2009. As a result of the replan, the program zeroed out some schedule variances from the baseline to reflect the program’s progress toward the newly defined schedule. Despite this replan in April, our analysis shows that the replan has not improved overall performance as cumulative cost and schedule variances continue their downward trend. We were unable to estimate whether the total work under the contract is likely to be completed within budgeted cost since trends cannot be developed until at least 15 percent of the work under the contract is completed.

Limited Contractor Data Prevented Analysis of All MKV Task Orders

The Multiple Kill Vehicles (MKV) program began utilizing an indefinite delivery indefinite quantity contract in January 2004. Since then, the program has initiated eight task orders, five of which were open during fiscal year 2008—Task Orders 4 through 8. Task Order 4 provided insufficient data to complete full earned value analysis for the fiscal year. In addition, Task Order 5 was completed shortly after the fiscal year began, without providing enough data to show performance trends. Therefore we performed analysis for Task Orders 6, 7, and 8 as shown below. None of the task orders were realigned during the fiscal year.

MKV Task Order 6 began in November 2006 for the component development and testing of a prototype carrier vehicle seeker (a long- range sensor). According to the task order, this seeker for the carrier vehicle will assign individual kill vehicles for target destruction. This task will culminate in a demonstration planned for fiscal year 2010. As shown in figure 13 below, performance data over the course of the fiscal year illustrates declining cost and schedule performance. Although it began the fiscal year with slightly positive cumulative cost and schedule variances, the program ended the year with slightly negative cumulative cost and schedule variances of $1.1 million and $0.6 million respectively. In addition, the contractor has unfavorable fiscal year cost and schedule variances of $1.4 million and $1.5 million, respectively. The program attributes its negative cumulative cost and schedule variances to increased work necessary to resolve software development issues, unplanned efforts as a result of late hardware arrivals, and a government-directed change in vendors for hardware resulting in additional design work. Based on our analysis and the assumption that the contractor will continue to perform as it has through fiscal year 2008, we predict that at its contract completion in May 2009, the contractor on Task Order 6 will overrun its budgeted cost of $19.3 million by between $1.6 million and $2.5 million.

The MKV Task Order 7 is for the development and testing of engagement management algorithms and the test bed in which it will be demonstrated. These algorithms will be a critical capability of the carrier vehicle to manage the kill vehicle engagements relying on target information from the BMDS sensors and the carrier vehicle long-range sensor. The contractor on this task order performed positively during the fiscal year, both in terms of its cumulative and fiscal year cost and schedule variances. The program had a favorable fiscal year cost variance of $1.4 million and a positive fiscal year schedule variance of $11 thousand, adding to its favorable cumulative cost and schedule variances of $1.7 million and $0.1 million, respectively. The program attributes its cumulative cost underruns to several reasons including a programmatic decision to proceed with one approach for organizing kill vehicles in attack formation rather than funding several different approaches. In addition, the contractor experienced cost savings with greater efficiencies than expected in the kill vehicle portion of the work under the contract and less manpower than planned in other portions of the work under the contract. If the contractor continues to perform as it has in the past, we estimate that at completion in May 2010 the work under the contract could cost between $3.2 million and $3.9 million less than the expected $43.9 million budgeted for the work under the contract. See figure 14 below for an illustration of cumulative cost and schedule performance during fiscal year 2008.

MKV Task Order 8 was awarded in January 2007 and began reporting full performance data in July 2007. The task order is for the development and testing of a hover test bed and hover test vehicle. This hover test bed will allow the program to integrate and test key components of the system in a repeatable ground-based free flight environment as their technologies reach maturity. The program experienced a continuing schedule performance decline as seen in figure 15.

Although the contractor began the year with a positive cumulative cost variance, overruns during the fiscal year of $10.7 million led the program to a total cumulative cost overrun of $10.3 million. The element’s fiscal year schedule variance was slightly negative at an unfavorable $15 thousand, leaving its cumulative schedule variance largely unchanged at a favorable $0.3 million. The program attributes the cumulative cost variances to increased labor, procurement, and material costs as well as increased hardware and engineering drawings, and management oversight to resolve subcontractor inefficiencies. In addition, the program increased expenditures to resolve technical and schedule issues associated with the development of avionics subsystems. The planned date for the task order’s main effort—completing the hover test—was delayed 2 months from its original date to December 2008 in part because of technical issues associated with the test vehicle’s power unit and a software anomaly. These issues were resolved prior to the hover test being conducted. Based on its prior performance, the MKV contractor could overrun the budgeted cost of $48.0 million for the work under the contract at completion in January 2009 by between $5.7 million and $13.8 million.

Sensors’ Radar Experienced Fiscal Year Cost and Schedule Growth

As of September 2008, the Sensor’s contractor had overrun its fiscal year budget by $2.2 million and was behind in completing $27.4 million worth of work. Considering prior years’ performance, the contractor is performing under budget with a favorable cumulative cost variance of $22.0 million. However, the contractor has a cumulative unfavorable schedule variance of $9.6 million. The contractor reports the cumulative schedule variance is driven by delays in the manufacturing of the sixth radar and a software capability release that is 2 to 3 months behind schedule. Additionally, the contractor reports that its favorable cumulative cost variance is attributable to efficiencies in the second radar’s manufacturing, design, development, and software. The Sensors contractor has not performed a realignment of its work since contract start in April 2003. See figure 16 for trends in the contractor’s cost and schedule performance during the fiscal year.

The contractor reports that its unfavorable fiscal year schedule variance of $27.4 million is due to a decrease of previously earned positive schedule variances reaped from the manufacturing efficiencies leveraged from the Terminal High Altitude Area Defense radar hardware design. In addition, late delivery of components also contributed to the negative fiscal year schedule variances. The negative fiscal year cost variance of $2.2 million is largely due to a contract change related to its incentive fee. Our analysis predicts that if the contractor continues to perform as it has through fiscal year 2008, the work under the contract could cost from $25 million less to $9.1 million more than the budgeted cost of $1.1 billion at completion currently planned for December 2010.

Technical Issues Drove STSS Cost Growth during the Fiscal Year

After a replan of work in October 2007, the Space Tracking and Surveillance System (STSS) contractor experienced an unfavorable cost variance of $87.9 million during the fiscal year. The replan was undertaken in order to extend the period of performance and delay the launch date of its demonstrator satellite. Despite fiscal year cost overruns, the contractor was able to make gains on the cumulative schedule variance by accomplishing $1.9 million more worth of work than was originally planned. Cumulatively, the program has both unfavorable cost and schedule variances at $319.3 million and $17.8 million, respectively. The program attributes cumulative cost variances and schedule variances to continual launch date schedule slippages. In addition, problems in the space segment portion of work also added to the cumulative cost variances. Figure 17 shows both cost and schedule trends during fiscal year 2008.

Program cost variances during the fiscal year were driven primarily by technical issues with hardware installed on the second space vehicle. These issues included an overheating flight communications box, a leak on the propulsion side of the satellite, and problems with the spacecraft processor that failed to send a critical command to the onboard computer. To resolve the issues with the processor, the program office initially recommended the removal of the entire computer from the spacecraft. However, after extensive research and testing, the program manager determined that the event with the spacecraft is an unverifiable failure with a low probability of occurrence and low mission impact and decided not to remove the computer from the spacecraft to resolve the issue. We estimate that if the contractor continues to perform as it has through fiscal year 2008, the work under the contract at completion in September 2011 could exceed its budgeted cost of $1.6 billion by between $621.7 million and $1.2 billion.

Targets and Countermeasures Program’s Rebaseline Positively Affected Fiscal Year Schedule Variances

In June 2008, a delivery order under the Targets and Countermeasures’ element that is developing a new family of targets—the Flexible Target Family (FTF)—performed a rebaseline as a result of experiencing manufacturing delays to several components. The majority of the delays were from qualification failures, subsequent redesigns, and requalification efforts. The rebaseline was to realign the work under the contract to reflect realistic hardware delivery dates. This rebaseline did not affect cost variances, but did rebaseline major milestone delivery dates and, as a result, set some of the previously existing schedule variances to zero.

The Targets and Countermeasures contractor made gains with a favorable $23.2 million fiscal year schedule variance due in part to the rebaseline in June 2008. However, the contractor ended the year with an unfavorable cumulative schedule variance of $6.4 million which was primarily driven by delays in the completion of the FTF qualification program. The program also ended the year with a cumulative cost variance of $52.8 million which the contractor attributed to costs associated with the FTF’s avionics components integration and qualification issues, and more effort than expected required on motors for one of the targets in the program. See figure 18 below for an illustration of cumulative cost and schedule variances during the course of the fiscal year.

The contractor attributes its unfavorable fiscal year cost variances of $35.7 million to the increased cost of completing the first four FTF 72-inch targets. Delays in completing component qualification extended the period of performance which invariably led to higher costs. In addition, the contractor cites cost increases to the failure of one of its targets in July 2008 that added mission assurance and testing cost to the follow-on mission using the same target configuration. We estimate that if the contractor continues to perform as it has in the past, it will overrun its budgeted cost of $1.1 billion at contract’s end in December 2009 by between $63.7 million and $75.9 million.

THAAD Contractor Spent More Money and Time Than Budgeted

The Terminal High Altitude Area Defense (THAAD) program experienced target issues during fiscal year 2008. The THAAD program performed a realignment in May 2008 to extend the flight test program after experiencing several delayed target deliveries. Because of the cost impact of these delayed targets, the program will increase its value of the work under its contract by approximately $80 million. The THAAD program performed a similar realignment in December 2006 as a result of delayed target deliveries as well. As a result of this realignment, the program extended its flight test program and added an estimated $121 million to the value of work under its contract.

The THAAD contractor experienced downward trends in its cost and schedule performance during fiscal year 2008. The program overran its budgeted costs for the fiscal year by $33.5 million. It was also unable to accomplish $7.4 million worth of work during the fiscal year. Both of these unfavorable variances added to the negative cumulative cost and schedule variances of $228.7 million and $16.5 million, respectively, as shown in figure 19.

The THAAD prime contractor’s fiscal year cost overrun of $33.5 million was primarily caused by the radar, missile, and launcher portions of work. Design problems delayed the prime power unit design review and slowed parts production, causing the radar’s negative cost trend. In addition, the missile’s negative cost trend for this same period was driven by design complexity, ongoing rework/retest of subsystems, unexpected qualification discoveries, and unfavorable labor variances at key subcontractors. Lastly, the launcher variances were driven by hardware and software complexities and higher-than-expected costs for transitioning a portion of this effort to a different facility for production.

The contractor reports that its unfavorable fiscal year schedule variance of $7.4 million is primarily driven by the radar and missile components. The radar’s negative schedule variance is associated with vendor delays in delivering trailers for both of the system’s prime power units. The late delivery of the trailers has subsequently delayed delivery of the prime power units. Missile rework due to qualification test discoveries also negatively affected schedule performance. If the contractor continues to perform as it has through fiscal year 2008, we project that at the contract’s completion currently scheduled for September 2009, the contractor could overrun its budgeted cost of $4.6 billion by between $252 million to $274 million.

Appendix III: FTG-04 Flight Test Cancellation

On May 23, 2008, the Senate Armed Services Committee requested that we review the reasons behind the cancellation of a GMD flight test designated FTG-04. Initially, on May 1, 2008, the Director, MDA decided to delay this test due to problems discovered in a telemetry device, the Pulse Code Modulation Encoder (PCME). This device does not affect operational performance, but rather is a critical component needed to transmit flight test data only. The PCME problems were due in large part to manufacturing defects, which the manufacturers and MDA concluded likely affected all the PCMEs. However, on May 8, the Director of MDA instead decided to cancel this flight test entirely, resulting in one less GMD end-to-end intercept flight test. MDA told us that delaying the flight test until the PCMEs could be repaired would cause delays in future tests since various test assets were shared. MDA officials therefore decided to cancel FTG-04 and transfer some test objectives to other tests, including a new non-intercept flight test, FTX-03, and an already planned intercept flight test, FTG-05. Also, for some remaining objectives not captured in FTG-05 and FTX-03, MDA stated that it planned a third intercept test, FTG-X. We were asked to investigate this test cancellation and answer the following questions: Why did the MDA change its initial decision to delay FTG-04 until November 2008 and decide to cancel FTG-04 instead and what deliberative process did MDA follow in deciding to cancel FTG-04?

When and how, if ever, will each of the specific test objectives previously planned for FTG-04 be accomplished?

What are the implications of canceling this flight test on the ensuing test program, on demonstrating the capability of the GMD system, and on other programmatic decisions?

Faulty Telemetry Component Caused Delay and Subsequent Cancellation of FTG- 04

MDA initially delayed the FTG-04 flight test because of defects in the PCME, a telemetry component in the Exoatmospheric Kill Vehicle (EKV) only needed to gather test data. Although the PCME does not affect operational performance, it is needed for test assets to determine if design and operational issues have been resolved. The FTG-04 had four prior delays and was originally scheduled for the first quarter of fiscal year 2007. In responding to these delays, multiple tests over several years were affected.

Several defects contributed to the problem, the first three of which are presumed to affect all PCMEs manufactured up to that point and all 24 fielded Test Bed/Capability Enhancement (CE) I EKVs: The PCMEs experienced gold embrittlement due to lack of pretinning.

Insufficient oscillator stand-off height increased thermal stress.

Circuit board deflection caused by three washers missing from the board.

In addition to these manufacturing defects, there were stress fractures in the solder of three PCMEs caused by the removal and replacement of a chip on the device. This chip was removed because a clock on a chip was asynchronous with another component’s clock. It was estimated that there was an 18 to 48 percent chance of the loss of telemetry data at some point during a flight test due to the asynchronous chip problem. Again, all 24 fielded Test Bed/CE-I EKVs have the chip with this problem. This chip does not affect operational performance, but rather is a critical component needed to transmit flight test data only.

See table 10 for timeline of events related to this cancellation.

The contractor, Boeing, and the subcontractors, Raytheon and the manufacturer of the component, L-3, took actions to mitigate the problem. They eliminated the gold embrittlement problem by sending the oscillator out for pretinning, they designed custom washers for two already produced PCMEs and raised three bosses for new PCMEs to eliminate the need for washers, and they tightened tolerances on the board to eliminate the deflection issue. These first three PCME manufacturing improvements were finalized on May 16, 2008. They also made changes to correct the chip with the asynchronous clock problem in all newly manufactured PCMEs. None of the previously fielded GBIs will be refurbished with improved PCMEs needed for flight tests, but the GBIs emplaced starting in October 2008 and thereafter have the improved PCME.

On May 1, 2008, MDA’s Program Change Board considered five options. 1. Execute FTG-04 as scheduled, using payload (32) “as is” 2. Continue diagnostic testing of payload 32, but if decision was made that it was not ready, substitute payload 33, leading to a delay in the test schedule 3. Refurbish payload 32, but if it did not improve, substitute payload 4. Immediately replace payload 32 with 33 without further testing 5. Immediately return payload 32 for repair The Director, MDA chose option 5, delaying the FTG-04 into the November to December 2008 timeframe, but keeping the program on track to provide this intercept data as planned. MDA consulted the test community, Director, Operational Test and Evaluation (DOT&E) and the BMDS Operational Test Agency, on this initial decision to delay the test and both agreed with this decision. According to MDA, the Director also asked for options for a sensor test in the summer of 2008.

On May 8, 2008, MDA’s Program Change Board reconvened to consider three options for a sensor test (FTX-03) and canceling instead of delaying FTG-04: 1. Conduct FTX-03, with a baseline like the planned FTG-04, but without a live intercept attempt. 2. Conduct FTX-03 with a baseline like the FTG-05, an intercept flight test to be conducted in December 2008. 3. Similar to option 2, but with more sensor data collected.

The Director MDA changed the May 1 decision to delay, refurbish and fly the planned FTG-04 test and chose instead to cancel FTG-04 and pursue the modified option 3 above. Choosing option 3 resulted in restructuring the intercept test into a test designed to assess multiple sensor integration capability. This new test benefited sensor modeling and simulation as post-flight reconstruction could occur now on two missions.

According to MDA, it canceled the FTG-04 at the May 8, 2008 meeting instead of delaying it, in part, because rescheduling FTG-04 would have caused a major delay in another test, the Distributed Ground Test-03 (GTD-03). GTD-03 and FTG-04 required many of the same assets, so conducting FTG-04 would have delayed GTD-03 and thus the delivery of this new capability by four months. Ground tests assess the increased BMDS capability to be fielded next and GTD-03 was to provide the means by which a more realistic simulation of threats and scenarios and the means by which new software capability could be declared ready to move into the operational baseline. MDA consulted with BMDS Operational Test Agency officials on this decision and they supported it. DOT&E was not consulted on this decision and expressed concern that the elimination of any intercept test reduced the opportunity to gather additional data that might have increased confidence in models and simulations. DOT&E has repeatedly expressed concerns over the lack of test data needed to validate MDA’s models and simulations.

Most FTG-04 Test Objectives Will Be Allocated to Follow- on Tests

According to MDA, all FTG-04 test objectives were allocated to other flight tests. However, partly due to differences in how MDA describes test objectives, it is unclear whether all planned FTG-04 test objectives will be accomplished in follow-on tests. The loss of a primary objective, an intercept of a complex target scene, will slow MDA’s efforts to build confidence in the EKV’s ability to consistently achieve intercepts, unless an additional intercept is scheduled. In August 2008, MDA informed Congress that it planned to conduct a new intercept test called FTG-X in fiscal year 2009. However, in January 2009 MDA stated that the FTG-X intercept test was never formally approved and is no longer planned.

In addition, some test objectives related to modeling and simulations have been redefined so it is unclear whether they will be fully tested. Models and simulations are critical to understanding and assessing the performance of the BMDS because flight tests are limited by their cost, complexity, and range safety constraints. Modeling and simulation is therefore the primary way to fully assess the overall performance of the BMDS and its various components. According to DOT&E, cancellation of FTG-04 reduced interceptor and EKV data available for modeling, leaving only two intercepts (FTG-3a and FTG-05) that have provided complete sets of information. In October 2008, MDA stated that modification of FTG-04 into a sensor test eliminated a second opportunity to anchor the models of EKV-fielded software. Test objectives in MDA planning documents describe modeling and simulation objectives at a high level. However, it is difficult to determine whether modeling and simulation objectives are addressed in the near term because the objectives are defined differently for each test. For example, the FTX-03 and FTG-05 objectives do not distinguish between primary and secondary objectives while the FTG-04 does. One objective that seems to be absent in the FTG-05 and FTX-03 is to collect data to support validation and anchoring of system-level (vs. element-level) simulations, MDA stated that the exclusion from the test objectives was inadvertent and it will be addressed by the tests.

BMDS Operational Test Agency objectives related to the GBI engagement were not met, although the test agency officials indicate the majority of their non-intercept sensor related objectives for FTG-04 were met in the FTX-03 test. However, BMDS Operational Test Agency officials state that some of their intercept objectives may be addressed through a combination of previous intercept test, FTG-03a, and recently conducted FTG-05. Finally, several warfighter objectives for FTG-04, related to tactics, techniques, and procedures, will be met through the ground tests instead because, according to the warfighter representative at the BMDS Operational Test Agency, flight tests do not offer the best opportunity to assess this kind of objective.

Cancellation Eliminates One of Few Opportunities to Demonstrate GMD Capabilities

The cancellation has increased the strain on the ensuing test program. GMD’s current plans call for two intercept attempts in fiscal year 2009— FTG-05, which was conducted in December 2008, and FTG-06—and one booster verification test. This is an ambitious schedule as GMD has been able to conduct only one intercept flight test per year—FTG-02 in September 2006, FTG-03a in September 2007 and FTG-05 in December 2008. MDA had planned to conduct five intercept tests with varying stresses to assess the EKV capability between February 2007 and December 2008. Flight test failures and test plan revisions caused MDA to only carry out two intercept tests in that period—FTG-03a and FTG-05— both of which resulted in an intercept.

In addition, the number of future flight tests planned has been reduced. MDA has not funded or scheduled an intercept replacement for FTG-04. In January 2008 MDA decided to merge two intercept tests—FTG-06 and FTG-07—into one single intercept attempt. This merger removes another opportunity to gather end-game EKV performance data needed to assess capability. In January 2008 MDA also decided to accelerate a two-stage verification non-intercept test required to assess the European component.

The cancellation of FTG-04 removed one chance to obtain end-game performance data needed to develop GMD models and to assess the capability of the CE-I EKV. The repetition of intercept-related objectives is important to build confidence in the intercept capability. These models are the primary way to fully assess the overall system performance, since flight tests are limited by their cost, complexity and range safety concerns. MDA planned to test the CE-I EKV against a dynamic target scene with countermeasures in both FTG-04 and FTG-05. FTG-04 was canceled and an FTG-05 target anomaly affected this objective. According to MDA, no more CE-I EKV flight tests have been approved, although it is considering whether to conduct an intercept test using a CE-I EKV in the future. GMD developed some mitigations to various developmental issues, but realistic flight testing is needed to anchor the models and to determine the effectiveness of these mitigations.

The test cancellation and target problems have reduced the knowledge that MDA expected to use for its upcoming end-to-end performance assessment. Performance assessments are annual system-level assessments to test, evaluate, and characterize the operational capability of the BMDS as of the end of the calendar year. Currently, MDA has only completed one—Performance Assessment 2007. Furthermore, acting on a joint recommendation between MDA and the Operational Test Agency, MDA officials canceled their 2008 performance assessment efforts in April 2008 because of developmental risks associated with modeling and simulations. Instead, MDA is focusing on testing and models for Performance Assessment 2009. However, the planned performance information available for Performance Assessment 2009 will be reduced. The FTG-04 cancellation reduced one set of data that was expected to be available. In addition, both FTX-03 and FTG-05 will be used to anchor data for Performance Assessment 2009, but target anomalies in each test precluded the completion of all planned test objectives. Neither target presented the complexity needed for advanced algorithm development.

Manufacturing and emplacement continue unabated by reductions and delays in tests. Twenty-four CE-I GBIs have been fielded and the new CE-II GBIs are now being fielded without important knowledge about the systems capabilities expected to be gained through tests. The first CE-II GBI emplacement occurred prior to any flight testing of this configuration. The first flight test is FTG-06 currently scheduled to occur no earlier than the fourth quarter of fiscal year 2009. According to MDA, these CE-II GBIs will not be declared operational until after the successful completion of FTG-06. FTG-04 was also identified as a key source of data supporting a number of capabilities declarations. The cancellation of FTG-04, plus other testing delays, prompted MDA to defer some capabilities and to declare others based on previous tests.

Conclusions

The cancellation of the FTG-04 flight test increases the risk to the GMD program and to the overall BMDS capability, since the lack of adequate intercept data adversely affects confidence that the system could perform as intended in a real-world situation. The GMD program has reduced its plans to assess operational performance of the fielded configuration between February 2007 and December 2008 from five to two intercept tests, leaving gaps in knowledge about the repeatability of the performance of fielded assets. In addition, the opportunity to obtain additional intercept data vital to the anchoring of models and simulations has been lost, unless the FTG-X flight test is conducted, adding to an existing concern expressed by DOT&E. Despite test reductions and effects on assessing system-level performance, production and fielding of assets continues as planned.

Appendix IV: Reduced Basis for Capability Declarations

The two tables below list test events supporting MDA capability declarations during fiscal year 2008 for certain engagement sequence groups in Blocks 1.0 through 3.0 (see table 11), as well as for the full completion of Bock 1.0 by the end of fiscal year 2009 (see table 12). Both tables illustrate that MDA reduced the basis for declaring certain engagement sequence groups as early or fully capable. The basis for declaring an early, partial, or full capability includes flight and ground tests as well as performance assessments.

Appendix V: Scope and Methodology

To examine the progress MDA made in fiscal year 2008 toward its cost, schedule, testing, and performance goals, we examined the efforts of 10 BMDS elements that MDA is developing and fielding. The elements included in our review collectively accounted for 80 percent of MDA’s fiscal year 2008 research and development budget requests. In assessing each element, we examined the BMDS Fiscal Year 2008 Statement of Goals, Program Execution Reviews, test plans and reports, production plans, Contract Performance Reports, MDA briefings, and earned value management data. We developed data collection instruments that were completed by MDA and each element program office. The instruments gathered detailed information on planned and completed program activities including tests, design reviews, prime contracts, estimates of element performance, and challenges facing the elements. In addition, we discussed fiscal year 2008 progress and performance with officials in MDA’s Agency Operations Office, each element program office, as well as the Office of DOD’s Director, Operational Test and Evaluation, and DOD’s Operational Test Agency. To assess each element’s progress toward its cost goals, we reviewed Contract Performance Reports and, when available, the Defense Contract Management Agency’s analyses of these reports. We applied established earned value management techniques to data captured in Contract Performance Reports to determine trends and used established earned value management formulas to project the likely costs of prime contracts at completion.

To evaluate the sufficiency of MDA’s modeling and simulation practices, we reviewed DOD and MDA policies, memos, flight and test plans related to modeling and simulations, the Acquisition Modeling and Simulation Master plan, as well as verification, validation and accreditation plans and reports for various elements, and MDA white papers discussing modeling and simulation techniques. We also interviewed officials in element program offices to discuss modeling and simulation plans and procedures particular to each.

In assessing MDA’s accountability, transparency, and management controls, we interviewed officials from the Office of the Under Secretary of Defense’s Office for Acquisition, Technology, and Logistics, as well officials in the MDA Agency Operations Directorate. We also reviewed an Institute for Defense Analysis study, two Congressional Research Service reports, a Congressional Budget Office report, U.S. Code, DOD acquisition system policy, various DOD directives, the Missile Defense Executive Board charter, and various MDA statements and documents related to the agency’s block structure.

To ensure that MDA-generated data used in our assessment are reliable, we evaluated the agency’s management control processes. We discussed these processes with MDA senior management. In addition, we confirmed the accuracy of MDA-generated data with multiple sources within MDA and, when possible, with independent experts. To assess the validity and reliability of prime contractors’ earned value management systems and reports, we interviewed officials and analyzed audit reports prepared by the Defense Contract Audit Agency. Finally, we assessed MDA’s internal accounting and administrative management controls by reviewing MDA’s Federal Manager’s Financial Integrity Report for Fiscal Years 2003, 2004, 2005, 2006, 2007, and 2008.

Our work was performed primarily at MDA headquarters in Arlington, Virginia. At this location, we met with officials from the Aegis Ballistic Missile Defense Program Office; Airborne Laser Program Office; Command, Control, Battle Management, and Communications Program Office; MDA’s Agency Operations Office; DOD’s Office of the Director, Operational Test and Evaluation; and the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics. In addition, in Huntsville, Alabama, we met with officials from the Ground-based Midcourse Defense Program Office, the Sensors Program Office, the Terminal High Altitude Area Defense Project Office, the Kinetic Energy Interceptors Program Office, the BMDS Kill Vehicles Program Office, the Targets and Countermeasures Program Office, and the Office of the Director for BMDS Tests. We also met with Space Tracking and Surveillance System officials in El Segundo, California.

In December 2007, the conference report accompanying the National Defense Authorization Act for Fiscal Year 2008 noted the importance of DOD and MDA providing information to GAO in a timely and responsive manner to facilitate the review of ballistic missile defense programs. During the course this audit, we experienced significant delays in obtaining information from MDA. During the audit, MDA did not provide GAO staff with expeditious access to requested documents which delayed some audit analysis and contributed to extra staff-hours. Of the documents we requested, we received approximately 19 percent within the 10-15 business day protocols that were agreed upon with MDA. Pre-existing documentation took MDA on average about 50 business days to provide and many pre-existing documents took over 100 business days to be provided to GAO. Notwithstanding these delays, we were able to obtain the information needed to satisfy our objectives in accordance with generally accepted government auditing standards.

We conducted this performance audit from May 2008 to March 2009 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix VI: GAO Contact and Staff Acknowledgments

Acknowledgments

In addition to the contact named above, David Best, Assistant Director; LaTonya Miller; Beverly Breen; Ivy Hübler; Tom Mahalek; Steven Stern; Claire Cyrnak; Isabella Johnson; Meredith Allen Kimmett; Kenneth E. Patton; Karen Richey; and Alyssa Weir made key contributions to this report.