Defense Acquisitions: Production and Fielding of Missile Defense Components Continue with Less Testing and Validation Than Planned
Published: 2009-03-13T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Missile Defense Agency (MDA) has spent about $56 billion and will spend about $50 billion more through 2013 to develop a Ballistic Missile Defense System (BMDS). GAO was directed to assess the annual progress MDA made in developing the BMDS as well as improvements in accountability and transparency in agency operations, management processes, and the new block strategy. To accomplish this, GAO reviewed contractor cost, schedule, and performance; tests completed; and the assets fielded during 2008. GAO also reviewed pertinent sections of the U.S. Code, acquisition policy, and the activities of the new Missile Defense Executive Board (MDEB). An appendix on the effect the cancellation of a Ground-based Midcourse Defense flight test had on BMDS development is also included.
section_title: What GAO Found
paragraphs: Cost: MDA has not yet established baselines for total costs or unit costs, both fundamental markers most programs use to measure progress. Consequently, for the sixth year, GAO has not been able to assess MDA's actual costs against a baseline of either total costs or unit costs. MDA planned to establish such baselines in 2008 in response to past GAO recommendations, but has delayed this until 2009. GAO was able to assess the cost performance on individual contracts, and project an overrun at completion of between $2 billion and $3 billion. However, because in some cases the budgeted costs at completion--the basis for our projection--has changed significantly over time as adjustments were made, this projection does not capture as cost growth the difference between the original and current budgeted costs at completion. In one case, these costs increased by approximately five times its original value. Performance and Testing: While MDA completed several key tests that demonstrated enhanced performance of the BMDS, all elements of the system had test delays and shortfalls. Overall, testing achieved less than planned. For example, none of the six Director's test knowledge points established by MDA for 2008 were achieved. Poor performing target missiles have been a persistent problem. Testing shortfalls have slowed the validation of models and simulations, which are needed to assess the system's overall performance. Consequently, the performance of the BMDS as a whole can not yet be determined. Schedule: Although fewer tests have been conducted than planned, the production and fielding of assets has proceeded closer to schedule. Except for no ground-based interceptors being delivered, all other radars, standard missiles, and software were delivered as planned. However, some deliveries, such as enhanced Exoatmospheric Kill Vehicles, will now precede test results. In most cases, MDA has also reduced the bases it planned to use to declare when capabilities are operational in the field. Thus, fielding decisions are being made with a reduced understanding of system effectiveness. Transparency, Accountability, and Oversight: Improvement in this area has been limited. The Missile Defense Executive Board (MDEB) has acted with increased authority in providing oversight of MDA and the BMDS. However, transparency and accountability into MDA's work is limited by the management fluidity afforded through the lack of cost baselines, an unstable test baseline, continued use of development funds to produce assets for fielding, and renewed potential for transferring work from one predefined block to another. A better balance must still be struck between the information Congress and the Department of Defense need to conduct oversight of the BMDS and the flexibility MDA needs to manage across the portfolio of assets that collectively constitute the system's capability. At this point, the balance does not provide sufficient information for effective oversight.

section_title: Letter
paragraphs: The Missile Defense Agency (MDA) has spent almost $56 billion since its  initiation in 2002 on developing and fielding a Ballistic Missile Defense  System (BMDS) and is on course to spend about $50 billion more over the  next 5 years. In 2002, the President directed the Department of Defense  (DOD) to “deploy a set of initial missile defense capabilities beginning in  2004”. MDA began delivering an initial capability in late 2004, as directed,  and deployed an initial capability in 2005 by concurrently developing and  fielding BMDS assets. Though this approach facilitated the rapid  deployment of an initial BMDS capability, as MDA has proceeded beyond  that initial capability, it has been less successful in fostering adequate  knowledge of system capabilities prior to manufacturing and fielding  BMDS assets.
In its fiscal year 2002, 2007, and 2008 National Defense Authorization Acts,  Congress directed GAO to assess the cost, schedule, testing, and  performance progress that MDA is making in developing the BMDS. We  have delivered assessments covering fiscal years 2003 through 2007. This  report assesses the progress made during fiscal year 2008 toward BMDS  goals as well as the progress MDA made in improving accountability and  transparency through its agency operations, management, processes, and  new block strategy. This report also includes an appendix that addresses  the Senate Armed Services Committee’s request that we review the  reasons behind and the effects on BMDS development of the cancellation  of a Ground-based Midcourse Defense flight test designated FTG-04.
To assess progress during fiscal year 2008, we examined the  accomplishments of 10 BMDS elements that MDA is developing and  fielding: the Aegis Ballistic Missile Defense (Aegis BMD); Airborne Laser  (ABL); BMDS Sensors; Command, Control, Battle Management, and  Communications (C2BMC); Ground-based Midcourse Defense (GMD);  Kinetic Energy Interceptors (KEI); Multiple Kill Vehicles (MKV); Space  Tracking and Surveillance System (STSS); Targets and Countermeasures;  and Terminal High Altitude Area Defense (THAAD). These elements  collectively account for about 80 percent of MDA’s research and  development budget. We also examined MDA’s Fiscal Year 2008 Statement  of Goals, Program Execution Reviews, test plans and reports, production  plans, and Contract Performance Reports. We interviewed officials within  program offices and within MDA functional directorates, such as the  Directorate for Cost Estimating. In addition, we discussed each element’s  test program and its results with the BMDS Operational Test Agency and  DOD’s Office of the Director, Operational Test and Evaluation (DOT&E).
In assessing progress made toward improving accountability and  transparency, we held discussions with officials in MDA’s Directorate of  Business Operations to determine whether its new block structure  improved accountability and transparency of the BMDS. In addition, we  reviewed pertinent sections of the U.S. Code to compare MDA’s current  level of accountability with federal acquisition laws. We also interviewed  officials from the Office of the Under Secretary of Defense for Acquisition,  Technology and Logistics to discuss the oversight role of the Missile  Defense Executive Board. Additionally, we reviewed the board’s charter to  determine its oversight responsibility. Our scope and methodology is  discussed in more detail in appendix V.
We conducted this performance audit from May 2008 to March 2009 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
subsections: section_title: Background
paragraphs: MDA’s mission is to develop an integrated and layered system to defend  the United States and its deployed forces, friends, and allies against  ballistic missile attacks. The BMDS aims to engage all ranges of ballistic  missiles during all phases of flight. This challenging expectation requires  complex coordination within an integrated system of defensive  components—space-based sensors, surveillance and tracking radars,  advanced interceptors, and a battle management, command, control, and  communication component.
A typical engagement scenario to defend against an intercontinental  ballistic missile would occur as follows:  Infrared sensors aboard early-warning satellites detect the hot plume  of a missile launch and alert the command authority of a possible  attack.
Upon receiving the alert, land- or sea-based radars are directed to  track the various objects released from the missile and, if so designed,  to identify the warhead from among spent rocket motors, decoys, and  debris.
When the trajectory of the missile’s warhead has been adequately  established, an interceptor—consisting of a kill vehicle mounted atop a  booster—is launched to engage the threat. The interceptor boosts  itself toward a predicted intercept point and releases the kill vehicle.
The kill vehicle uses its onboard sensors and divert thrusters to detect,  identify, and steer itself into the warhead. With a combined closing  speed of up to 10 kilometers per second (22,000 miles per hour), the  warhead is destroyed above the atmosphere through a “hit to kill”  collision with the kill vehicle.
Some interceptors use sensors to steer themselves into the inbound  ballistic missile. Inside the atmosphere, these systems kill the ballistic  missile using a range of mechanisms such as direct collision between  the missile and the inbound ballistic missile or killing it with the  combined effects of a blast fragmentation warhead (heat, pressure,  and grains/shrapnel) in cases where a direct hit does not occur.
Table 1 provides a brief description of 10 BMDS elements currently under  development by MDA.
To manage BMDS development, MDA uses an acquisition strategy defined  by a block structure. From its inception in 2002 through 2007, MDA  developed BMDS capability in biennial increments, ultimately delivering  two blocks—Block 2004 and Block 2006. These 2-year blocks each built on  preceding blocks and enhanced the development and capability of the  BMDS. However, in response to recommendations from GAO, in  December 2007 MDA announced a new block structure that was intended  to improve the program’s transparency, accountability, and oversight. The  new blocks are not based on biennial time periods, but instead focus on  fielding capabilities that address particular threats. Because the new block  structure is not aligned to regular time periods, multiple blocks are  underway concurrently. Table 2 details the current blocks and categories  included in the BMDS block structure.
MDA uses a Statement of Baselines and Goals to report modifications to  established block baselines. For those blocks that are currently or will  soon be underway, block baselines are created to make a firm  commitment to Congress. The Statement of Goals also includes the  following:    BMDS Baseline Capabilities – Assets and engagement sequence  groups that will be made available for fielding for a particular block.  During 2008, cost baselines were under development for Block 2.0 and  Block increments 3.1/3.2. MDA established schedule and performance  baselines for Blocks 1.0, 2.0, 3.1, and 3.2 in 2008.
BMDS Capability Goals – Assets and engagement sequence groups  expected to be made available for future blocks.
Adversary Benchmarks – Adversary missile systems used for block  performance estimates.
BMDS Budget Breakdowns – Detailed fielding, development, and  integration budgets for each block and BMDS Capability Development  activity.
MDA also uses an incremental declaration process to designate BMDS  capability for its blocks. Three capability designations are applied to all  BMDS elements, their hardware and software components, and  engagement sequence groups. This allows these BMDS features to play a  limited role in system operations before they have attained their expected  level of capability. Each capability designation in the delivery schedule  represents upgraded capacity to support the overall function of BMDS in  its mission as well as the level of MDA confidence in the system’s  performance. The designations are defined as follows:    Early Capability Delivery signifies readiness for contingency use. At  this point, MDA has determined that the capability can be utilized by  the BMDS. When integrated, the capability must be adequately  demonstrated to build sufficient confidence that it will safely perform  as intended without degrading the existing capabilities of the BMDS.
Partial Capability Delivery is an intermediate state of maturity  indicating that a capability has been shown through testing to perform  as intended in certain scenarios. At this point, MDA is sufficiently  confident that the capability can support the warfighter’s partially  mission-capable objectives and logistics support is adequate to achieve  defensive operations.
Full Capability Delivery is the point at which a capability satisfies the  BMDS block objectives and is considered to be completely mature and  ready for full operational use.
MDA’s capability goals for fiscal year 2008 for Blocks 1.0, 2.0, 3.1, and 3.2  are shown in table 3.
section_title: Cost Tracking Deficiencies Hinder Assessment of Cost Performance
paragraphs: MDA has not yet established baselines for total costs or unit costs, both  fundamental markers that most programs use to measure progress. MDA  had planned to establish total cost baselines at the element and block  levels in 2008, but the initial set of total cost baselines will not be available  until the spring of 2009. Similarly, MDA has not established unit costs for  selected assets, such as GBIs. Consequently, for the sixth year, we have  been unable to assess MDA’s overall progress on total or unit cost. While  MDA plans to establish some total cost baselines in 2009, most efforts will  not be captured in a baseline. MDA also plans to establish unit costs,  another improvement, but is considering a narrower definition of unit cost  than is used by other weapon system programs. MDA’s definition will  report a subset of procurement costs called flyaway costs, which only  includes the major piece of equipment and excludes all research and  development as well as some procurement costs—those for support  equipment and spares. Moreover, these unit costs will only be tracked  within those blocks that are baselined, which will represent a minority of  those assets being produced and fielded. Without total cost baselines in  place, the BMDS concept continually evolves, as indicated by the number  of realignments to the program of work at the individual contract level.  While the changing nature of the BMDS and the lack of total cost baselines  precludes analysis of total cost progress, we were able to analyze  contractor fiscal year performance on the current work under the  contract. We were also able to project overruns or underruns at  completion for BMDS contracts using the contracts’ current budgeted  costs at completion as a basis for our projections. However, in some cases,  the current budgeted cost at completion changed significantly over time.  In one case, the budgeted cost at completion increased by approximately  five times its original value. Our analysis of fiscal year 2008 progress  shows that several prime contractors exceeded budgeted costs.
subsections: section_title: Absence of Cost Baselines Prevents Assessment of System-Level Costs
paragraphs: To provide accountability, major defense acquisition programs are  required by statute to document program goals in an acquisition program  baseline. MDA is not yet required to establish an acquisition program  baseline because of the acquisition flexibilities it has been granted.  However, Congress has enacted legislation requiring MDA to establish  some baselines. Baselines serve an important discipline both by ensuring  that the full cost commitment is considered before embarking on major  development efforts and by identifying cost growth as a program  proceeds. Since we began annual reporting on missile defense in 2004, we  have been unable to assess overall progress on cost—that is, comparing  BMDS baselined costs with actual costs. For example, under the prior  block structure, we reported that BMDS costs grew by at least $1 billion,  but the total cost growth could not be determined because MDA did not  account for all costs for a given block.
In response to recommendations we made in March 2008, MDA agreed to  develop cost estimates and to provide independent verification for blocks  outlined under its new approach. Upon conclusion of those estimates,  MDA will develop cost baselines. In addition, on April 1, 2008, the  Director, MDA, testified before a Senate Armed Services Subcommittee  that cost baselines for the new block structure would be available by the  end of 2008. As of January 2009, the agency had not yet developed full cost  baselines for any blocks. MDA plans to have these cost baselines for  Blocks 2.0, 3.1 and 3.2 completed, independently reviewed by DOD’s Cost  Analysis Improvement Group, and released by the spring of 2009. The only  information that was available for this report was limited to budget  projections for BMDS blocks and capability development for fiscal years  2008 through 2013, totaling approximately $42 billion.
Even with the release of some block cost estimates in 2009, all block costs  will not be baselined and no date has been established for when the  remaining block costs will be baselined. MDA does not plan to baseline  Block 1.0 costs, but will provide the actual costs of the block since it is  near completion. Full cost baselines for Block 2.0 are anticipated to be  available in the spring of 2009, but only portions of Block 3.0 and none of  Block 4.0 and 5.0 costs will be baselined at this time. As figure 1 shows, if  MDA does complete baselines as planned, they will only cover about 26  percent of its block and capability development costs.
At this point, MDA plans to baseline between 2 and 26 percent of BMDS  block and capability development costs from fiscal years 2010 to 2013 as  depicted above. MDA has not determined when other blocks will be  baselined. If other blocks were to be baselined before 2013, the percentage  of funding baselined would be increased. The rapid decline in percentage  of baselined funds also shows that initial baselines are being set late in a  block’s duration. For example, Block 2.0 will be completed within 2 years  of its baseline being set. If baselines are to facilitate management and  oversight, they will have to be set sooner for Block 3.3 and beyond.  Additionally, agency officials stated that although cost estimates will be  developed for individual capability development efforts—such as ABL,  KEI, and MKV—the agency does not plan to baseline their costs until these  elements are matured and moved into a defined block. MDA may  eventually baseline these elements as part of a block once a firm  commitment can be made to Congress. The budgets for capability  development elements account for approximately $22 billon—or more  than half of MDA’s fiscal year 2008 6-year Future Years Defense Plan for  BMDS blocks and capability development.
section_title: Planned Unit Cost Reporting Will Not Be Comprehensive
paragraphs: Major defense acquisition programs are required by statute to report  certain unit costs to Congress, track unit cost growth against their  original and current baseline estimates, and perform an additional  assessment of the program if certain cost growth thresholds are reach This cost monitoring mechanism helps ensure that programs are being  held accountable. MDA is not yet required to report these unit costs  because of the acquisition flexibilities it has been granted by DOD, but  Congress has enacted legislation requiring MDA to provide unit cost  reporting data for certain BMDS elements, and MDA does plan to develop  and report unit costs for some of its assets in the spring of 2009. The  agency has also established thresholds for reporting cost growth.  However, the approach MDA is taking, while an improvement, provides a  much less comprehensive assessment of unit cost compared to the  traditional acquisition costs that are typically reported for major defense  acquisition programs. Normally, unit costs are reported in two ways: (1)  program acquisition unit cost, which is the total cost for the development  and procurement of, and system-specific military construction for, the  acquisition program divided by the number of fully configured end items  to be produced, or (2) average procurement unit cost, which is the total of all funds programmed to be available for obligation for procurement  divided by the number of end items to be procured.   ed.
MDA’s development of the BMDS outside of DOD’s normal acquisition process makes it difficult to compare the actual unit co st of a delivered  asset with its planned unit cost. For example, MDA plans to only report  recurring unit flyaway costs for the blocks that are baselined. Figure 2 reveals the significant reduction in standard areas of costs covered by  MDA’s approach compared to that normally reported for major defense  acquisition programs.
MDA’s decision to report only flyaway unit costs will not capture research  and development costs associated with BMDS assets—which account for  more than 97 percent of the nearly $56 billion MDA costs to date. In  addition, the procurement costs for initial spares and support equipment  are not included. Thus, while the flyaway cost baseline will provide  visibility into changes in recurring manufacturing costs, it will not provide  a basis for comparison with the costs of other DOD programs. If a cost  increase occurs in research and development or nonrecurring  procurement, it will not be reported in MDA’s unit cost.
Agency officials told us that the reason for using flyaway unit c the new MDA block structure. They further explained that within the  block structure, there are many cases where MDA procures and delivers a  weapon system for more than one block and, in some cases, the same  configuration in more than one block. For example, THAAD deliveries are  included in Blocks 2.0 and 5.0. Agency officials cite this as a key difference  between MDA and other defense programs. For MDA, most of the  development costs are assigned to the first block where a capability is  delivered and very little of the development cost is assigned to subsequent  blocks. MDA officials further stated that if the agency were to use the  standard unit cost methodology, it would show very dissimilar unit costs  between the first and subsequent block deliveries and the difference could  not be explained by learning curves and manufacturing efficiencies. MDA  officials also told us that they chose unit flyaway cost for unit cost  reporting because flyaway cost provides a better measure of what the  individual system components cost to procure. However, MDA is not  precluded from also determining and reporting unit costs by taking the  entire cost of the asset being developed without regard to the capability or  block for which it is originally developed.
section_title: Frequent Realignments Indicate That Full Scope Is Not Yet Determined
paragraphs: Without a firm cost commitment or baseline in place for the full scope of  the BMDS, in some cases the work currently under contract changes  frequently. These changes manifest themselves in realignments that often  add scope, cost and time to the value of the work under the contract.  Since contracts began on the BMDS, MDA has performed 31 realignm where, in some cases, the period of stability between these major  realignments average only between 1 to 2 years. These frequent chang indicate that the total BMDS effort has not been fully determined and is  likely to grow. While we have been able to make an assessment of  contractor costs, that assessment is limited to the current approved  program. es  Until total and unit cost baselines are established, the only tool for us to  use in assessing BMDS costs is the costs reported on individual contracts  under BMDS’s Earned Value Management System. All BMDS contracts  that we assessed have a cost and schedule baseline against which  progress, measured by cost and schedule performance, can be measured.  It is appropriate for a program to realign its current status with the  remaining contractual effort when officials conclude that the baseline no  longer provides valid performance assessment information. A program  can realign its current status with the remaining contractual effort through rebaselines, replans, and restructures.
A rebaseline is a more general term  to describe a major realignment of  the performance measurement baseline used to better correlate the  work plan with the baseline budget, scope, and schedule and can refer  to replans and restructures as well.  A replan is a reallocation of schedule or budget for the remaining  effort within the existing constraints of the contract.
A restructure includes adding funds to the performance manageme budget that exceed the value of the negotiated contra contract modification.
section_title: MDA Contractors Overran Fiscal Year Cost and Schedule
paragraphs: Our analysis of contractor costs indicates that during fiscal year 2008,  MDA contractors collectively overran budgeted costs by $152.4 million.  These overruns occurred in 11 of 14 MDA contracts we reviewed, with the  STSS contract accounting for more than 50 percent of the total. Based on  cost performance during the fiscal year and using formulas accepted  within the cost community, we estimate that at completion the cumulative  overrun in the contractors’ budgeted costs could be from about $2.0 billion  to $3.0 billion. Our projections are based on the current budgeted costs at  completion for each contract we assessed, which represents the total  current planned value of the contract. However, the budgeted costs at  completion, in some cases, have grown significantly over time. For  example, the ABL contractor reported budgeted costs at completion  totaling about $724 million in 1997, but as depicted in table 5, that cost has  since grown to about $3.6 billion. Our assessment only reveals the overrun  or underrun since the latest adjustment to the budget at completion. It  does not capture, as cost growth, the difference between the original and  current budgeted costs at completion. As a result, comparing the  underruns or overruns for MDA programs in table 5 with cost growth on  major defense acquisition programs is not appropriate because those  major defense acquisition programs have established their full scope of  work as well as developed total cost baselines, while these have not been  developed for MDA programs. Our analysis is presented in table 5.  Appendix II provides further details on the cost and schedule performance  outlined in the table.
Technical difficulties caused most elements to overrun their fiscal year  2008 budgeted costs. For example, STSS attributed most of its overrun of  approximately $87.9 million to hardware problems on the program’s  second space vehicle including the flight communication box and the main  spacecraft computer. The box overheated during testing which required a  thorough test of the unit. Upon successful completion of this testing, it  was determined it did not require a replacement. In addition, the program  had a failure in the main spacecraft computer for which the program office  initially recommended the removal of the entire computer from the  spacecraft. However, after extensive research and testing, the program  manager determined that the event with the spacecraft was an unverifiable  failure with a low probability of occurrence and low mission impact and  decided not to remove the computer from the spacecraft to resolve the  issue. However, as a result of these issues, the launch was delayed from  April 2008 to at least the third quarter of fiscal year 2009.
The MKV Task Order 6 and ABL contractors also experienced technical  difficulties. The MKV contractor for Task Order 6 reported cost overruns  during the fiscal year of $1.4 million due mostly to software development  issues and late delivery of government-furnished components. ABL’s fiscal  year cost overruns of $10.6 million were mainly related to late deliveries of  key laser system components and the acquisition or refurbishment of the  Beam Control/Fire Control system components. For example, a major  component of the laser system required redesign and fabrication, delaying  planned delivery and installation onto the aircraft. Also, multiple Beam  Control/Fire Control hardware components either were not refurbished to  specification or failed initial testing, delaying delivery and integration  testing. The overall effect was an approximate 1-month slip that the  contractor believes will be made up in time to make the current lethality  demonstration planned for the end of fiscal year 2009.
Three elements’ contracts—Aegis BMD’s contract for 27 SM-3 Block 1A  missiles, MKV Task Order 7, and GMD—performed below their fiscal year  budgeted costs by nearly $58.3 million with the GMD element accounting  for approximately $53.9 million of that. The GMD element’s underruns  occurred partially because the contractors delayed or eliminated some  planned work. For example, the GMD program did not emplace the three  GBIs it expected to in fiscal year 2008 or conduct either of its two planned  flight tests as scheduled during the fiscal year. As a result, it employed less  labor than originally intended. Drivers for the MKV Task Order 7 contract’s  fiscal year cost underruns include a restructuring of the effort and  decisions to use one rather than several approaches for coordinated  attack, and using less manpower than originally planned in its  procurement and software efforts. Lastly, the Aegis BMD contract for 27  SM-3 Block 1A missiles underran its fiscal year 2008 budget by  approximately $3 million due in part to spending less than planned for  engineering efforts with the missile’s third stage component as well as  adjustments made in program management, labor efficiencies, and  material transfers in the missile’s fourth stage component.
section_title: While Some Tests Succeeded, Others Were Deferred; Overall System Performance Cannot Yet Be Assessed
paragraphs: Although several tests showed progress in individual elements and some  system-level capabilities, all BMDS elements experienced test delays and  shortfalls, in part due to problems with the availability and performance of  target missiles. Most significantly, GMD was unable to conduct either of its  planned intercept attempts during fiscal year 2008, however it was able to  conduct one delayed intercept test in December 2008. As a result, key  performance capabilities of the current configuration of the GMD kill  vehicles may not be demonstrated and the new configuration is being  fielded prior to flight testing. As a consequence of testing problems, none  of the six MDA Director’s test knowledge points for 2008 were achieved.  Poor performance of targets continues to be a problem that caused several  tests to either fail in part or in whole. Shortfalls in testing have delayed  validating the models and simulations that are used to assess the overall  performance of the BMDS as a whole. Consequently, comprehensive  assessments of the capabilities and limitations of the BMDS are not  currently possible and therefore MDA still does not have the capability to  model or simulate BMDS capability from enemy missile launch to its  engagement.
subsections: section_title: Test, Targets, and Performance Challenges Continued during Fiscal Year 2008 for Several Elements
paragraphs: During fiscal year 2008, all BMDS elements experienced delays in  conducting tests, most were unable to accomplish all objectives, and  performance challenges continued for many. Moreover, the inability of  MDA to conduct its full fiscal year 2008 flight test campaign as planned  precluded the agency from collecting key information specified by the  Director, MDA—known as Director’s test knowledge points—to make  certain decisions at critical points in some BMDS programs. Table 6 below  summarizes test results and target performance for BMDS elements during  the fiscal year.
As a result of test delays, MDA restructured its flight test plan for fiscal  year 2009, increasing the number of tests and compressing the amount of  time to analyze and prepare for subsequent tests. For example, MDA plans  to conduct 14 of 18 flight tests in the third and fourth quarter of fiscal year  2009. MDA’s past performance raises questions about whether this is  realistic. In fiscal year 2008, MDA had planned to conduct 18 flight tests,  but it only accomplished 10, plus it had several flight tests delayed into  2009 from 2008. An MDA official acknowledged that the 2009 plan is  aggressive, but stated that it can be achieved. Specifics of each element’s  testing experience during fiscal year 2008 follow.
According to Aegis BMD officials, budgetary constraints prompted the  Aegis BMD element to delay some tests, reducing the number of tests  planned for 2008. However, the program was able to successfully complete  its first test involving two non-separating targets, conduct a short-range  ballistic missile intercept, and participate in a THAAD intercept during the  fiscal year. The program also planned to participate in a BMDS-level  ground test during the year, but the test was delayed until at least the  second quarter of fiscal year 2009 because of real-world events. Finally,  Aegis BMD standard missile flight tests showed that interoperability issues  persist between THAAD and Aegis BMDS with respect to correlation and  object reporting.
ABL experienced delays during fiscal year 2008, but achieved all of its  primary test objectives. The program planned to complete the installation  of its high energy laser on the aircraft by June 2008 in preparation for  testing. However, it was not completed until August 2008 because of  problems with activating some of the laser’s subsystems. The program  delayed the final testing of the laser until the problems could be resolved.  Once the problems were resolved, the ABL program was able to complete  functionality testing of the laser in September 2008.
C2BMC experienced delays in conducting tests, but achieved several test  objectives. For example, software upgrade verification testing slipped  from fiscal year 2008 to 2009 but the program was able to participate in  many other system-level ground and flight tests during the year that  enabled the program to demonstrate multiple capabilities, including  situational awareness and sensor management. The C2BMC element  extended development for its next software release, 6.4, by more than a  year because of delays in system-level BMDS testing and challenges in  developing the network server for version 6.2, as well as unplanned work  to incorporate effects from earth rotation in the 6.4 C2BMC planning  architecture. C2BMC added earth rotation effects to address a requirement  that Spiral 6.2 and later releases have the ability to model the true extent  of ranges for long-range threats. Finally, C2BMC is still developing its  capability to generate a single track from multiple sensors through a new  resource management function, the Global Engagement Manager. For  example, the development team for this function had to modify the design  of this function’s new track processing that experienced an unacceptable  level of delays when processing data.
In fiscal year 2008, the GMD program was unable to conduct either of its  two planned intercept attempts—FTG-04 and FTG-05. MDA first delayed  and then later canceled the FTG-04 test in May 2008 due to a problem with  a telemetry component in the interceptor’s Exoatmospheric Kill Vehicle  (EKV) needed to gather test data. MDA also delayed FTG-05 from fiscal  year 2008 and conducted it in December 2008. Over the past two years  MDA had expected to conduct seven GMD interceptor flight tests by the  end of the first quarter of fiscal year 2009. However, MDA was only able to  conduct two, as shown in figure 3.
The cancellation of FTG-04 raised concerns within the test community and  members of Congress. FTG-04 was at first delayed and then canceled.  MDA replaced it with a test to assess sensor capability—FTX-03. The  sensor test allowed GMD to verify fire control software and integration  with multiple sensors. The DOT&E was not consulted on the decision to  cancel FTG-04 and expressed concern that the elimination of any intercept  test reduced the opportunity to gather data that might have increased  confidence in the models and simulations. In the conference report  accompanying the National Defense Authorization Act for Fiscal Year  2008, conferees expressed concern about the loss of the FTG-04 flight test  and requested that we review the circumstances and the effects on the  BMDS. Details of our review of the FTG-04 flight test cancellation appear  in appendix III.
Because GMD conducted FTG-05 in December 2008, there are only two  full sets of GMD intercept data to date available for analysis which limits  the ability to verify and validate the models and simulations. Additionally,  FTG-04 and the subsequent test—FTG-05—were planned to present  different stresses to the kill vehicle which would provide critical data  needed to further verify the fielded configuration of the kill vehicle. The  cancellation and subsequent restructuring of the first test caused a delay  in FTG-05 from the third quarter of fiscal year 2008 until December 2008.  In the FTG-05 test, the interceptor hit its intended target. However, MDA  judged the target as a failure because it failed to release its  countermeasures as planned. Consequently, all primary test objectives  were not achieved.
Looking forward to the next GMD intercept flight test—FTG-06 in at least  the fourth quarter of fiscal year 2009—MDA is accepting a higher level of  risk than it previously expected in conducting this first test of the CE-II  EKV because it will contain several objectives that had planned to be  previously tested, but have not been. MDA had set up an approach to test  one new major component change at a time. For example, MDA had  planned to test the CE-I EKV first against simple targets, then the CE-I  against a complex target, and once that had been proven MDA planned to  test the CE-II EKV against a complex target. However, MDA was not able  to test the CE-I EKV against a complex target due to a target failure. Due  to testing problems, GMD has only been able to assess the CE-I EKV with a  target without countermeasures. As a result, the FTG-06 flight test will be  the first GMD test assessing both a CE-II EKV and a complex target scene.  Adding to the risk, this will be only the second test using a newly  developed FTF LV-2 target.
During fiscal year 2008, the KEI program experienced problems during  testing that required it to rework components which, in turn, caused a  delay to subsequent testing. More importantly, due to technical issues  experienced by the program over the past two years, the first booster  flight test—a key decision point for the program—has been delayed by  nearly a year and is not scheduled to occur until at least the fourth quarter  of fiscal year 2009. Technical difficulties delayed the MKV program’s fiscal  year 2008 hover test until fiscal year 2009. This hover test will allow the  program to integrate and test key components of the system in a  repeatable ground-based free flight environment as their technologies  reach maturity. Although originally planned for the fourth quarter of fiscal  year 2008, the test was successfully conducted in December 2008. The  STSS program encountered problems during testing that forced the  program to delay the launch of its demonstration satellites from April 2008  to at least the third quarter of fiscal year 2009. The program continued to  experience technical difficulties with its space vehicles. For example,  during testing, the program experienced problems with its main spacecraft  computer as well as an overheating flight communications box. After  extensive testing, the program determined that these components were  acceptable for flight.
Similarly, in fiscal year 2008, the Sensors element also experienced flight  test delays as well as difficulties in achieving planned objectives due to  target performance, but met some primary objectives. The element  successfully participated in other tests during the fiscal year which  demonstrated the ability for the sensors to acquire and track a target. One  test event, FTX-03, provided the first opportunity for four key sensors— Sea-based X-band radar, AN/TPY-2, Upgraded Early Warning Radar, and an  Aegis BMD radar—to operate in a more operationally realistic test  scenario. This test demonstrated the capability for the sensors to  correlate target information in order to conduct an intercept test.  However, the target failed to release its countermeasures as planned. This  failure precluded sensors from assessing capability against a dynamic  lethal target scene with countermeasures. As a result, the sensors could  not collect all of the expected data, which delayed the element’s ability to  develop algorithms needed for the discrimination capability. These  objectives will need to be addressed in future testing. The BMDS  Operational Test Agency has had ongoing concerns regarding the  formatting, tracking, and accounting of messages from GMD sensors.  The timely reception of messages from sensors to weapon systems is key  to support decisions and achieve effective intercepts. Since 2000 the BMDS  Operational Test Agency has reported these concerns to MDA about poor  data collection and management practices involving sensors affecting its  assessment of tests. These data management problems prevented the  analysis of message data, according to BMDS Operational Test Agency  officials. In response, the contractor proposed a message monitoring  system among communications nodes. Consequently, MDA recommended  that this issue be closed out, but the BMDS Operational Test Agency still  considers the matter to be open because GMD has not funded the  monitoring system.
THAAD planned to conduct three intercept attempts, but due to a target  failure, it was only able to conduct two. The program could not complete  its final flight test of the fiscal year because the target experienced an  anomaly during flight. The test was planned to be a BMDS-level event and  was designated as a developmental test/operational test mission utilizing  multiple BMDS elements and operationally realistic criteria. The program  also expected to demonstrate that it could launch more than one THAAD  interceptor during the engagement. Program officials rescheduled this test  for the second quarter of fiscal year 2009. In addition, THAAD’s radar data  collection test, RDC-2, was planned for 2008 but was deleted due to target  availability and funding. As a result, program officials told us that these  test objectives will be covered in the future with hardware-in-the-loop  simulations and other radar events. The program successfully completed  its first two planned tests for the fiscal year. In October 2007, THAAD  successfully demonstrated an intercept of a target outside the earth’s  atmosphere. This was the first time THAAD had successfully conducted an  intercept outside of the atmosphere since 1999. Additionally, in June 2008,  THAAD completed a successful intercept of a separating target. This  intercept utilized warfighter procedures developed by the U.S. Army Air  Defense School.
section_title: Key MDA Test Knowledge Points Not Achieved
paragraphs: As a consequence of flight test delays as well as a delay in a key ground  test, MDA was unable to achieve any of the Director’s test knowledge  points scheduled for fiscal year 2008 as shown in table 7.
In May 2007, the Director, MDA, established key system-level and element- level knowledge points to provide critical information for making key  decisions regarding the BMDS. According to MDA, these knowledge points  are unique management approaches chosen to manage MDA’s critical  program risks. Each knowledge point is based on an event that provides  critical information—or knowledge—for a key MDA decision requiring the  Director’s approval.
In fiscal year 2008, among the Director’s test knowledge points delayed,  MDA had to defer the confirmation of the 72” target performance due to  delays in qualifying components. Additionally, MDA had to delay the  confirmation of the booster for the KEI program as problems were  encountered during testing of the nozzle.
section_title: Poor Target Missile Performance Continues to Hamper BMDS Testing
paragraphs: While targets have caused problems in fiscal year 2008 testing, poor  performance of targets is not new. Targets’ reliability and availability  problems have significantly affected BMDS development and testing since  2006, and issues have grown even more problematic in recent years.  Although target anomalies and failures have affected many of the missile  defense elements, THAAD and GMD have been most affected. In 2006, the  THAAD program was unable to achieve its first intercept attempt (FTT-04)  because the target did not function properly. In 2007, two THAAD radar  characterization tests (RDC-1c and RDC-1d) were unsuccessful due to  target anomalies. These tests flew targets with characteristics needed for  radar observation in support of advanced discrimination algorithm  development. However, target problems prevented an opportunity for the  radar to exercise all of the planned algorithms, causing a loss of expected  data. In addition to target failure issues, the THAAD program deferred  some flight tests because targets were not available, which cost the  program about $201 million. GMD also experienced similar long-term  effects on its flight test schedule when it was unable to achieve primary  test objectives in a 2007 intercept attempt (FTG-03) due to a target failure.
MDA’s existing targets are becoming less capable of meeting requirements  for near-term flight tests. These targets are aging and likely to grow even  less reliable with time; some components, such as the rocket motors, are  more than 40 years old. Among other things, MDA’s Targets and  Countermeasures program office has also had problems incorporating  requirements into contracts and has experienced problems obtaining  supplies as vendors left the market due to the lack of business.
To address the growing need for more sophisticated and reliable targets  for the future BMDS test program, MDA was developing a new family of  targets called the FTF, which was originally intended to be a family of new  short, medium, and long-range targets with ground-, air-, and sea- launch  capabilities. MDA embarked on this major development without  estimating the cost to develop the family of target missiles. MDA  proceeded to develop and even to produce some FTF targets without a  sound business case and, consequently, their acquisition has not gone as  planned. The funds required for the FTF were spent sooner than  expected and were insufficient for the development. Getting the FTF  target’s components through the qualification process, however, was more  difficult and costly than the program expected. For example, MDA  originally planned to launch the first FTF target—a 72-inch LV-2—in a 2008  STSS flight test, but the test was rescheduled due to delays in satellite  integration and target affordability and availability. While many of the  target missile’s components are found on existing systems, their form, fit,  function, and the environment they fly in have been changed for the 72- inch LV-2 target. Consequently, many critical components initially failed  shock and vibration testing and other qualification tests and had to be  redesigned. The process was recently scheduled to be complete in early  October 2008 but, after several delays, was not finished until December  2008. Despite this, MDA expects the target to be complete and ready for its  first launch in a third quarter fiscal year 2009 Aegis BMD flight test (FTM- 15).
We recently reported that the FTF has been delayed, costs have increased  and exceeded $1 billion, and the program’s scope has been reduced.  Work on all but one of the FTF target variants, the 72-inch LV-2, was  canceled in June 2008, including plans for development and production of  the second type of FTF target, the 52-inch, originally scheduled to launch  in 2009. With guidance from the Missile Defense Executive Board, MDA is  currently conducting a comprehensive review of the targets program to  determine the best acquisition strategy for future BMDS targets. It is  expected to be completed in mid-2009. Whether or not MDA decides to  restart the acquisition of the 52-inch targets, or other FTF variants, and the  nature of those targets depends on the results of this review.
Currently, MDA has narrowed its FTF development efforts, focusing on a  single vehicle, the 72-inch LV-2 ground-launched target. The first launch  was supposed to determine the viability of the FTF concept and the  feasibility of discontinuing the use of existing targets. However, rather  than first conducting a separate developmental test to confirm the target’s  capability, MDA has chosen a much riskier approach. The first launch of  the new LV-2 target will be in an Aegis BMD intercept test. Aegis BMD  originally planned to use this new target in a fiscal year 2008 flight test;  however, because the target was not ready, the test is delayed until at least  the third quarter of fiscal year 2009.
Repeated target problems and test cancellations have also affected the  development of capabilities needed to discriminate the real target from  countermeasures. Without opportunities to test the functionality of the  software, there now is a system-level shortfall in BMDS progress toward  developing a target discrimination capability against more sophisticated  countermeasures in the midcourse phase of flight. In order to improve the  effectiveness of the BMDS against evolving threats, MDA elements are  developing advanced discrimination software in their component’s  sensors. The advanced discrimination software is critical to distinguish the  threat re-entry vehicle from associated countermeasures and debris.  Target failures during tests prevented opportunities to gather data to  assess how well discrimination software performs in an operational  environment.
section_title: Overall Performance of BMDS Can Not Yet Be Assessed
paragraphs: MDA’s modeling and simulation program enables MDA to assess the  capabilities and limitations of how BMDS performs under a wider variety  of conditions than can be accomplished through the limited number of  flight tests conducted. Flight tests alone are insufficient because they only  demonstrate a single collection data point of element and system  performance. Flight tests are, however, an essential tool used to both  validate performance of the BMDS and to anchor the models and  simulations to ensure that they accurately reflect real performance.  Computer models of individual elements replicate how those elements  function. These models are then combined into various configurations that  simulate the BMDS engagement of enemy ballistic missiles.
To ensure confidence in the accuracy of modeling and simulation in  representing BMDS capability, the program goes through a process called  accreditation. Element models are validated individually using flight and  other test data and accredited for their intended use. MDA intends to  group these models into system-level representations according to user  needs. One such grouping is the annual performance assessment, a  system-level end-to-end simulation that assesses the performance of the  current BMDS configuration. The performance assessment integrates  element-specific models into a coherent representation of the BMDS.  Performance assessments are used to:  assess objectives from MDA’s Deputy of Engineering and the BMDS  Operational Test Agency,   support MDA decisions about engagement sequence group capability  deliveries, and  support MDA decisions about BMDS fielding and declaring  capabilities.
Fundamentally, performance assessments anchored by flight and ground  tests are a comprehensive means to fully understand the performance  capabilities and limitations of the BMDS.
Developing an end-to-end system-level model and simulation has been  difficult. BMDS Operational Test Agency officials told us that they do not  anticipate a fully accredited, system-level model and simulation capability  to be available until 2011. MDA’s first effort to bring together different  element models and simulations to produce a fully accredited, end-to-end  model and simulation for Performance Assessment 2007 was unsuccessful  primarily because of inadequate data for verification and validation to  support accreditation and a lack of common threat and environment input  data among element models. Though Performance Assessment 2007 was a  success in establishing a capability for integrated modeling and simulation  in a short time frame, it was unsuitable to assess system-level performance  due to low confidence from a lack of accreditation. Consequently, acting  on a joint recommendation between MDA and the Operational Test  Agency, MDA officials canceled their 2008 performance assessment efforts  in April 2008 because of developmental risks associated with modeling  and simulations, focusing instead on testing and models for Performance  Assessment 2009. MDA officials believe that the refocused efforts will  increase the chances for success during Performance Assessment 2009.
According to the BMDS Operational Test Agency’s January 2009 Modeling  and Simulation Accreditation Report, confidence in MDA’s modeling and  simulation efforts remains low although progress was made during the  year. MDA is now exercising stronger central leadership to provide  guidance and resources as it coordinates the development of verified and  validated models and simulations, as recommended by a 2004 Defense  Science Board study. MDA and element officials are now working more  closely with the BMDS Operational Test Agency. For example, MDA and  the BMDS Operational Test Agency have agreed on performance  parameters and criteria used to validate element models and simulations.  Nonetheless, BMDS Operational Test Agency officials stated that there are  several weaknesses in the BMDS testing program such as:  Insufficient consideration of modeling and simulation requirements in  MDA flight test plans, though they emphasized that MDA is finalizing a  list of such parameters for future flight test plans,    Use of artificialities in flight tests which limit the realism of scenarios  for anchoring models and simulations, and  Inadequate test planning for comprehensive modeling of weather  conditions.
MDA intends to verify and validate models and simulations by December  2009 for Performance Assessment 2009. However, BMDS Operational Test  Agency officials stated that there is a high risk that the Performance  Assessment 2009 analysis will be delayed because of remaining challenges  and MDA’s slow progress in accreditation, as follows:    The compressed schedule of ground and flight tests leaves little time  for data analysis that is essential to anchor models to those tests,  particularly for a complete analysis supporting MDA’s Performance  Assessment 2009.
Out of 40 models, the BMDS Operational Test Agency recommended in  January 2009 full accreditation for only 6 models, partial accreditation  for 9 models, and no accreditation for 25 models.
Because MDA canceled the follow-on Performance Assessment 2008,  the BMDS Operational Test Agency did not receive verification and  validation data that would have been included in the modeling and  simulation portion of its 2008 operational assessment.
BMDS Operational Test Agency officials told us that MDA also does not  adequately plan for the collection of flight test data and post-flight  reconstruction to support anchoring MDA models and simulations, even  though post-flight reconstruction is needed to validate that models and  simulations are adequate representations of the real world for their  intended purpose. MDA guidance emphasizes that one of the primary  objectives of the MDA ground and flight test program is to anchor BMDS  models and simulations. Additionally, this guidance requires MDA’s testing  program to work with the MDA engineers to define a test program that  anchors these models and simulations across the operating spectrum.  According to BMDS Operational Test Agency officials, the first full post- flight reconstruction was conducted in December 2008.
Despite the guidance delineating responsibilities for test data, MDA test  plans currently do not include enough detail to allocate and synchronize  resources in order to anchor models and simulations. MDA recently  initiated a three-phase review of the entire BMDS test program. According  to MDA, this three-phase review will emphasize the need for basing BMDS  test planning and test design on critical factors that have not been proven  to date and will drive target selection requirements. One outcome of the  review will be to create integrated campaigns of ground and flight tests to  efficiently collect data needed to validate the models and simulations.  MDA intends to complete all three phases of the review by May 2009, after  which MDA intends to have a date when all MDA models and simulations  will be verified and validated. However, the current lack of flight test data  for MDA’s and BMDS Operational Test Agency analysis prevents the timely  validation of models and simulations that are used to build the 2009 end- to-end performance assessment.
section_title: Production, Fielding, and Declaration of Capabilities Proceed despite Delays in Testing and Assessments
paragraphs: In fiscal year 2008, MDA met most of its delivery goals. However, it  continued to pursue a concurrent development, manufacturing and  fielding strategy in which assets are produced and fielded before they are  fully demonstrated through testing and modeling. Although flight tests and  modeling and simulation produced less validation of performance than  planned, MDA continued manufacturing untested components and  declaring capabilities ready for fielding. For example, 10 of the new  configuration kill vehicles for the GBI will have been manufactured and  delivered before being flight-tested. MDA also declared that it had fielded  9 of 22 BMDS capabilities planned for 2008 (postponing 13), but due to test  cancellations and performance assessment delays, it had to change the  basis of these declarations, often relying on previous, less realistic testing.
subsections: section_title: MDA Met Most 2008 Asset Delivery Goals
paragraphs: MDA achieved four of the five delivery goals it set for fiscal year 2008 as  shown in the table 8.
The agency planned to deliver the Sea-based X-band radar and three  additional GBIs for Block 1.0, 20 additional SM-3 missiles for its Block 2.0  capability, a C2BMC site for fielding and activation for its Blocks 3.1/3.2  and 5.0 capabilities, and an additional AN/TPY- 2 radar. Although partial  capability for the Sea-based X-band radar will not be declared until at least  fiscal year 2009, it was approved for Early Capability Delivery in fiscal year  2008. The agency delivered the Aegis BMD SM-3 missiles, the AN/TPY-2  radar, and the C2BMC site in fiscal year 2008 as planned, but was unable  to deliver the GBIs because the GMD element encountered development  challenges with components for the CE-II EKV. In addition, the Navy  Commander, Operational Test and Evaluation Force declared the Aegis  BMD 3.6 system as operationally suitable and effective in October 2008.  This decision signifies that 18 Aegis BMD-equipped ships and 90 SM-3  missiles are ready for transition to the Navy.
section_title: Production and Fielding of BMDS Systems Getting Ahead of Testing
paragraphs: Despite developmental problems, test delays and MDA’s inability to  complete all fiscal year 2008 Director’s test knowledge points,  manufacturing, production, and fielding have proceeded close to schedule.  In some cases fielding has gotten ahead of testing. For example, Aegis  BMD expected to assess the ability of the SM-3 Block 1A missile to engage  and intercept a long-range ballistic target to satisfy a Director’s test  knowledge point. Even though that test has been delayed until the third  quarter of fiscal year 2009, MDA purchased 20 SM-3 Block 1As in fiscal  year 2008.
Furthermore, MDA intended to assess, through flight tests, the CE-I EKV’s  capability against scenarios that included complex target scenes with  countermeasures. However, due to the frequent restructuring of its test  plan and a target failure during its most recent flight test, the fielded  configuration for GMD has not completed a test against countermeasures.  According to MDA, no more CE-I flight tests have been approved, although  the agency is considering additional flight testing of the CE-I EKV in the  future. Moreover, earlier ground and flight testing, along with  manufacturing discoveries prompted the GMD program to initiate a  refurbishment program for the kill vehicles and the boosters.  Refurbishment consists of: (1) reliability improvements to address high  priority risks and to support the development and understanding of GBI  reliability and (2) surveillance of aging through the examination of  removed components. Consequently, the capability of the CE-I, including  improvements designed to mitigate risk, as well as understand its  capabilities and limitations against targets employing countermeasures  may not be flight-tested, yet all 24 interceptors with this configuration are  already emplaced and declared operational.
More importantly, the GMD program continues to experience test delays,  causing fielding to outpace flight tests as shown in figure 4.
For example, the program has only been able to conduct two intercepts  since 2006 for verifying the fielded configuration yet the production of  interceptors continues. According to GMD’s September 2006 flight test  plan, for fiscal years 2007 and 2008, and the first quarter of fiscal year 2009  it was going to conduct seven flight tests, including a test that would  utilize 2 GBIs against a single target—known as a salvo test —and field 16  new GBIs. By January 2009 GMD had changed its plan, removing the salvo  test and conducting two flight tests, yet it fielded 13 GBIs.
Similarly, GMD had planned to conduct an intercept test to assess the  enhanced version of the EKV called the Capability Enhancement II in the  first quarter of fiscal year 2008, months before emplacing any interceptors  with this configuration. However, developmental problems with the new  configuration’s inertial measurement unit and the target delayed the first  flight test with the CE-II configuration—FTG-06—until at least the fourth  quarter of fiscal year 2009. Despite these delays, MDA expects to have  emplaced five CE-II interceptors before this flight test. MDA indicated that  these five interceptors will not be declared operational until the  satisfactory completion of the test and the Program Change Board  declares their status. However, MDA projects that 10 CE-II EKVs will have  been manufactured and delivered before that first flight test demonstrates  the CE-II capability. This amounts to over half of the CE-II EKV deliveries  that are currently on contract.
MDA did not emplace the three GBIs it needed to meet its fiscal year 2008  fielding goals. MDA will have to emplace twice as many GBIs than planned  in fiscal year 2009 before Block 1.0 can be declared complete. As of  January 2009, the agency had emplaced two and must emplace four more  in order to complete Block 1.0 as planned.
Major defense and acquisition programs must complete operational test  and evaluation before entering full-rate production. Because MDA  considers the assets it has fielded to be developmental, it has not  advanced BMDS elements to DOD’s acquisition cycle or begun full-rate  production. Therefore, MDA has not yet triggered the requirement for an  operational test and evaluation prior to fielding. However, MDA’s  concurrent approach to developing and fielding assets has led to testing  problems and concerns about the performance of some fielded assets.  After two flight test failures in 2005, MDA undertook a Mission Readiness  Task Force to establish confidence in GMD’s ability to reliably hit its  target, establish credibility in setting and meeting test event dates, build  increasing levels of operationally realistic test procedures and scenarios,  raise confidence in successful outcomes of flight missions, and conduct  the next flight test as soon as practical within acceptable risk bounds.  However, GMD accelerated the objectives for its test program after the  first Mission Readiness Task Force flight test and the program continues  to experience developmental problems with key interceptor components.  MDA also separately established a refurbishment program designed to  replace questionable interceptor parts and increase reliability of GBIs.  Since 2006, we have reported that the performance of some fielded GBIs  was uncertain.
Despite MDA’s previous efforts to build confidence in its test program,  MDA continues to pursue a risky approach for fielding BMDS assets under  its new block structure. In March 2008, we reported that MDA’s new block  structure did not address whether it would continue its practice of  concurrently developing and fielding BMDS elements and components.  However, in 2008 the agency continued to field assets without adequate  knowledge. MDA emplaced GBIs during the year although its  refurbishment program was barely underway, meaning that the risks of  rework continue. To date, 26 GBIs have been emplaced—many of which  may contain unreliable parts—and only a few have been refurbished since  the initiation of the refurbishment program in 2007. According to program  officials, some improvements have already been introduced into the  manufacturing flow and demonstrated during flight testing.
While it is always a concern when tests are eliminated or the complexity of  a planned test is reduced, the concern is heightened for a system of  systems such as the BMDS because of the complex interaction of  components within an element, and between that element and the other  elements within the BMDS. Consequently, the need to synchronize the  development and testing of different capabilities is crucial before fielding  begins. For example, for certain engagement scenarios, the ground-based  interceptor will launch based on information provided by an entirely  separate element such as an Aegis cruiser or destroyer. If a problem is  discovered during these flight tests, post-flight reconstruction using  models needs to be conducted, the root-cause must be determined, a  solution or mitigation must be developed and implemented, and a new test  to confirm the effectiveness of the solution or mitigation must be  performed.
subsections: section_title: Reduced Testing Has Delayed Some Capability Declarations and Weakened the Basis for Others
paragraphs: When MDA determines that a capability can be considered for operational  use, it does so through a formal declaration. MDA uses an incremental  declaration process to designate BMDS capability for its blocks in three  levels—early, partial and full. The first two levels allow these BMDS  features to play a limited role in system operations before they have  attained their full level of capability. Each capability designation in the  delivery schedule represents upgraded capacity to support the overall  function of BMDS in its mission as well as the level of MDA confidence in  the system’s performance. Capability declarations are important because  MDA uses them to assess progress toward block completion. MDA  guidance calls for an orderly sequence of events that lead to declaring that  a fielded capability has been achieved and is ready for consideration for  operational use.
MDA bases its declarations on, among other things, a combination of  models and simulations—such as end-to-end performance assessments— and ground tests all anchored to flight test data. Because performance  assessments analyze the BMDS as an entire system in a variety of ways,  they provide more comprehensive information than flight tests alone.  These events and assessments build on each other every year as MDA adds  capabilities by improving hardware and software. MDA decision makers  would then declare the achievement of capability goals for engagement  sequence groups based on performance assessments. While in some  instances, declarations of capability have been deferred, in other instances  MDA has declared capabilities despite shortfalls in testing, modeling and  simulation, and performance assessments. The agency declared the  delivery of nine capabilities during fiscal year 2008 as shown in figure 5  below. It declared three early capabilities for Block 1.0 engagement  sequence groups, three early as well as one full capability for Block 2.0  engagement sequence groups, and one early capability and one partial  capability for Block 3.1/3.2 engagement sequence groups.
GBI Launch-on COBRA  DANE radar (Beale)
GBI Ene-on ea-baed  X-band radar GBI Launch-on ea-baed  X-band radar SM- Ene-on  hipboard Ae radar SM- Launch on remote  hipboard Ae radar SM-2 Ene-on  hipboard Ae radar THAAD Ene-on  AN/TPY-2 radar  (terminal mode)
GBI Ene on  COBRA DANE radar  Mod 1 (Fylindale,  UK; forward-baed  (AN/TPY-2 radar))
GBI Launch on hipboard  Ae radar Mod 1  (Fylindale, UK; ea-baed  X-band radar)
MDA had intended to use the results of a flight test (FTG-04), that was  later canceled; a distributed ground test (GTD-03), that was delayed into  fiscal year 2009; and the results of Performance Assessments 2007 and  2008 to determine if capabilities were ready for declaration in fiscal year  2008. Consequently, these shortfalls in knowledge led MDA to reduce the  basis for declaring capability goals. Performance Assessment 2007— identified by MDA as a key source to assess capabilities during fiscal year  2008—achieved only limited accreditation. This less-than-full  accreditation indicated that MDA could not rely on the assessment’s  results to gauge end-to-end BMDS performance. Subsequently, MDA  officials decided to cancel Performance Assessment 2008 because they  needed time to address problems and prepare for Performance  Assessment 2009.
While MDA officials declared these capabilities during fiscal year 2008,  they did so after mostly reducing the basis for the declarations. They  reverted back in several cases to older ground and flight tests, though  MDA in a few cases added some newer flight and ground tests as well. For  example, MDA declared early Block 1.0 capability for three engagement  sequence groups in fiscal year 2008 without the planned results from  Performance Assessment 2007. In all cases, though MDA had intended to  use the final results from comprehensive performance assessments, after  revising the basis for declaring capability goals it eliminated them entirely.  Specifically, MDA dropped some sources of data it expected to use, such  as the canceled Performance Assessment 2008, and shifted from flight and  ground tests planned to occur in fiscal year 2008 to older flight and ground  tests. For example, in Block 2.0 MDA declared full capability during fiscal  year 2008 for one engagement sequence group, Aegis BMDS engage on its  shipboard radar, even though Performance Assessment 2008 had been  canceled. MDA instead based its decision on integrated and distributed  ground tests (GTI-02 and GTD-02) conducted in calendar year 2007 as well  as prior flight tests during fiscal years 2006 through 2008. However, the  BMDS Operational Testing Agency raised concerns about the  comprehensiveness of the GTI-02 scenarios, specifically, the incorrect  configuration of U.S. satellites and threat data.
MDA also deferred 13 capability goals scheduled to occur in fiscal year  2008 to the end of fiscal year 2009, as shown in figure 6 below.
MDA intended to declare all Block 1.0 engagement sequence groups as  fully capable by the middle of fiscal year 2009. However, as MDA  encountered test delays and technical challenges, it had to defer full  capability declaration for these engagement sequence groups until the end  of fiscal year 2009. For Block 2.0, MDA also deferred declaring full  capability for one of the two planned full capability declarations for fiscal  year 2008. This declaration is contingent upon the review of a ground test  that has been rescheduled to the second quarter of fiscal year 2009 and a  flight test rescheduled to the third quarter of fiscal year 2009. MDA also  deferred one full and two early capability declarations for Block 3.1/3.2  beyond the end of fiscal year 2009.
In response to the limitations of Performance Assessment 2007, the  cancellation of Performance Assessment 2008 and FTG-04, and the  delayed GTD-03 and FTG-05 flight tests, MDA is planning to rely on older  ground and flight tests; a sensor flight test, FTX-03, instead of intercept  flight tests; and the initial quick look review of Performance Assessment  2009 instead of the previously planned full analysis. Appendix IV provides  a detailed layout for the reduced basis of capability declarations for fiscal  years 2008 and 2009.
Since MDA was only able to declare a few of the capabilities it planned for  fiscal year 2008, the schedule for fiscal year 2009 and subsequent years  will be compressed if the agency plans to maintain the schedule it has set  for its blocks. For example, MDA may need to declare three times as many  capabilities than originally planned for fiscal year 2009 in order to meet  the 2009 capability declaration schedule. In addition, if the schedule  cannot be maintained, MDA will likely have to make further adjustments  to mitigate additional delays in BMDS capabilities.
Increased reliance on integrated ground testing will provide less  knowledge than a complete analysis of capabilities from a performance  assessment. Integrated ground testing involves less robust conditions than  distributed ground testing, which involves operational systems in the field.  The MDA master fielding plan indicates that the agency originally intended  to take a more comprehensive approach upon which to base capability  declarations. Reliance on an upcoming Performance Assessment 2009  quick look for Block 1.0 completion is a particular concern because the  knowledge it provides may be limited with respect to testing, according to  BMDS Operational Test Agency officials. For example, officials told us  that a quick look may indicate anomalies from a test but will not analyze  their causes. In contrast, MDA originally planned to have a complete  analysis from the Performance Assessment 2009 models, simulations, and  tests.
section_title: Limited Progress Made in Improving Transparency and Accountability
paragraphs: In March 2008, we reported that efforts were underway to improve BMDS  management, transparency, accountability, and oversight including a new  executive board outside of MDA and a new block structure along with  other improvements within MDA. Since that time, the executive board  that was established in 2007 has acted with increased oversight. MDA’s  efforts, however, have not made the expected progress. In particular, MDA  has decided to retain the option of deferring work from one block to  another; cost baselines have not been established; test baselines remain  relatively unstable; and requesting procurement funds for some assets, as  directed by Congress, will not occur until fiscal year 2010.
To accomplish its mission, in 2002 the Secretary of Defense gave MDA  requirements, acquisition, and budget flexibilities and relief from some  oversight mechanisms and reporting responsibilities. The flexibility  granted to MDA has allowed concurrent development, testing,  manufacturing, and fielding. MDA used this flexibility to quickly develop  and field the first increment of capability in 2005. In August 2008, in  response to Congressional direction to assess the current and future  missions, roles, and structure of MDA, an independent study group agreed  that there is a need to move MDA toward more normal acquisition  processes. However, the group noted that the continuous evolution of the  BMDS requires that the approach to setting requirements for, developing,  and fielding increments of capability should remain as special authorities  with oversight of the Missile Defense Executive Board (MDEB). Further,  in regards to budget flexibilities, the independent group concluded that  while these flexibilities may have been deemed necessary at the time, it  should not have been expected that all the special authorities granted to  MDA would continue or would have a need to continue in full force  beyond achieving the President’s goal of deploying a set of initial  capabilities.
subsections: section_title: Missile Defense Executive Board’s Oversight Role More Active in 2008
paragraphs: During 2008, the MDEB appeared to act with an increased level of  authority in providing oversight of MDA and the BMDS. For example, the  board took on a major role in making key decisions regarding the  transition of elements to military services. We previously reported that  MDA and the military services had been negotiating the transition of  responsibilities for the sustainment of fielded BMDS elements, but this  process had been proven to be arduous and time consuming. However, in  2008, with the influence of the MDEB, a lead military service designation  was appointed for one BMDS asset—the Sea-based X-band radar.
In March 2008, we reported that the MDEB could play a key role in the  Joint Requirements Oversight Council’s proposal to return the BMDS to  the Joint Capabilities Integration and Development System requirements  process—a formal DOD procedure followed by most DOD programs that  defines acquisition requirements and evaluation criteria for future defense  programs. In responding to the proposal, the Acting Under Secretary of  Defense for Acquisition, Technology, and Logistics recommended that the  Deputy Secretary of Defense delay his approval of the Joint Staff’s  proposal until the MDEB could review the proposal and provide a  recommendation. According to Acquisition, Technology and Logistics  officials, no decision has been made regarding returning the BMDS to the  requirements process. However, the Deputy Secretary of Defense, in  September 2008, appeared to strengthen the oversight role of the MDEB,  clarifying the roles of the MDEB as well as MDA, the Office of the  Secretary of Defense, Combatant Commands, and Military Departments.  With respect to the role of the MDEB, he established a life cycle  management process for the BMDS stating that the MDEB will recommend  and oversee implementation of strategic policies and plans, program  priorities, and investment options to protect our Nation and our allies  from missile attack. One of the MDEB functions is to provide the Under  Secretary of Defense for Acquisition, Technology, and Logistics—or  Deputy Secretary of Defense, as necessary—a recommended strategic  program plan and feasible funding strategy for approval. The Deputy  Secretary further noted that, through the use of the BMDS Life Cycle  Management Process outlined in the memo, the MDEB will oversee the  annual preparation of the BMDS portfolio, including BMDS-required  capabilities and a program plan to meet the requirements with Research,  Development Test & Evaluation, procurement, operations and  maintenance, and military construction resources in defense-wide  accounts.
To further increase BMDS transparency and oversight of the BMDS, the  Under Secretary of Defense Acquisition, Technology, and Logistics plans  to hold program reviews for several BMDS elements commensurate with  the authority granted to the MDEB by the Deputy Secretary of Defense.  According to Under Secretary of Defense for Acquisition, Technology, and  Logistics officials, the MDEB conducted its first of such reviews in  November 2008 of the THAAD program. This review covered production  and the element’s contract schedule. Under Secretary of Defense for  Acquisition, Technology, and Logistics officials told us that these reviews  are designed to provide the Deputy Director Acquisition, Technology, and  Logistics with comprehensive information that will be used as the basis for  MDEB recommendations for the BMDS business case and baseline  processes— a process which, according to these officials, is similar to the  traditional Defense Acquisition Board process for reviewing other major  acquisition programs. However, it is unclear whether the information  provided to the MDEB will be comparable to that produced for other  major acquisition program reviews as most of the information appears to  be derived or presented by MDA as opposed to independent sources as  required for traditional major defense acquisition programs.
section_title: Efforts to Improve Transparency of MDA’s Work Have Not Progressed as Planned
subsections: section_title: Deferral of Work
paragraphs: In 2007, MDA redefined its block structure to better communicate its plans  and goals to Congress. The agency’s new structure is based on fielding  capabilities that address particular threats instead of the biennial time  periods previously used to develop and field the BMDS. Last year, we  reported that MDA’s new block plans included many positive changes.  However, MDA, with its submission of its Fiscal Year 2008 Statement of  Goals, reversed some of the positive aspects of the new block structure.  For example, we previously reported that the new block structure would  improve the transparency of each block’s actual cost by disallowing the  deferral of work from one block to another. Under its prior block  structure, MDA deferred work from one block to another; but it did not  track the cost of the deferred work so that it could be attributed to the  block that it benefited. Because MDA did not track the cost of the deferred  work, the agency was unable to adjust the cost of its blocks to accurately  capture the cost of each. This weakened the link between budget funds  and the work performed. Last year, MDA officials told us that under its  new block approach, MDA would no longer transfer work under any  circumstances to a different block. However, MDA officials recently said  that they are retaining the option to move work from one block to another  as long as it is accompanied by a rebaseline. This change allows the  agency to continue the practice of moving work from one block to  another, which thereby reduces the transparency of the new block  structure and undermines any baselines that are established.
section_title: Use of Procurement Funds
paragraphs: In March 2007, we reported that the majority of MDA’s funding comes  from the Research, Development, Test, and Evaluation appropriation  account, another flexibility provided by law. In past years, Congress  authorized MDA to pay for assets incrementally using research and  development funds. This allowed MDA to fund the purchase of assets over  multiple years. Congress recently restricted MDA’s authority and required  MDA to purchase certain assets with procurement funds and directed that  for any year after fiscal year 2009, MDA’s budget materials must delineate  between funds needed for research, development, test, and evaluation,  procurement, operations and maintenance, and military construction.  Requiring MDA to request funding in these appropriation categories will  mean that it will be required to follow the funding policies for each  category. For example, using procurement funds will mean that MDA will  be required to ensure that assets are fully funded in the year of their  purchase, rather than incrementally funded over several years.
Congress directed in the 2008 National Defense Authorization Act, for any  year after fiscal year 2009, that MDA’s budget materials delineate between  funds needed for research, development, test, and evaluation,  procurement, operations and maintenance, and military construction. We  have previously reported that using procurement funds will mean that  MDA generally will be required to adhere to congressional policy that  assets be fully funded in the year of their purchase, rather than  incrementally funded over several years. The Congressional Research  Service reported in 2006 that “incremental funding fell out of favor  because opponents believed it could make the total procurement costs of  weapons and equipment more difficult for Congress to understand and  track, create a potential for DOD to start procurement of an item without  necessarily stating its total cost to Congress, permit one Congress to ‘tie  the hands’ of future Congresses, and increase weapon procurement costs  by exposing weapons under construction to uneconomic start-up and stop  costs.”
In the 2008 National Defense Authorization Act, Congress also provided  MDA with the authority to use procurement funds for fiscal years 2009 and  2010 to field its BMDS capabilities on an incremental funding basis,  without any requirement for full funding. Congress has granted similar  authority to other DOD programs. In the conference report accompanying  the Act, Conferees cautioned DOD that additional authority will be  considered on a limited case-by-case basis and noted that they expect that  future missile defense programs will be funded in a manner more  consistent with other DOD acquisition programs.
MDA did not request any procurement funds for fiscal year 2009. During  our audit, the agency had not yet released the 2010 budget request to  include such funding categories. However, MDA officials told us that the  agency plans to request procurement funds for some of its assets in its  fiscal year 2010 request, but could not elaborate on its plans to do so.  Given that data was unavailable, it is unclear for which assets  procurement funds will be requested or the extent to which the request  will meet the direction given by Congress. According to MDA officials,  information regarding its plans to request procurement funding will not be  released until spring 2009.
section_title: Baselines
paragraphs: Baselines represent starting points against which actual progress can be  measured. They are thus used to provide indications of when a program is  diverting from a plan. Baselines can be established to gauge progress in  different areas, including cost, schedule and testing. Overall, the BMDS  does not have baselines that are useful for oversight. With regard to cost,  we have already discussed the lack of total and unit cost baselines for  missile defense as well as the frequency of changes in contract baselines.  MDA made some progress with developing a schedule baseline for its  blocks and their associated capabilities. The agency’s annual Statement of  Goals identifies its schedule baseline as the fiscal year dates for early,  partial, and full capability deliveries of hardware and functionality for a  block. Thus, while MDA has changed its schedule for making declarations,  the effect of the change can be determined by comparison with the  original schedule.
MDA does not have test baselines for its blocks. The agency does however,  have baselines for its test program, but revises them frequently. They are  therefore not effective for oversight. The agency identified its Integrated  Master Test Plan as the test baseline for the BMDS. However, as depicted  in table 9, the agency has made a number of changes to the content of the  baseline.
The official approved test baseline changes every year and there are  numerous more informal changes that happen more frequently. Most of  the annual revisions to the test baseline occur either because MDA has  changed the substance of test, changed the timing of tests, or added tests  to the baseline.
The Integrated Master Test Plan establishes the executable test program  for the current fiscal year and extends through the following fiscal year.  According to MDA, the test plan is updated quarterly based upon decisions  from MDA leadership or formal decision-making forums such as the  Program Control Board, and is also coordinated annually with external  agencies such as the Office of the Director of Operational Test and  Evaluation. However, as shown in table 9, there are several versions for a  given quarter and as many as seven versions have been developed since  the fiscal year 2008 baseline was established. It is unclear which Integrated  Master Test Plan version MDA manages to at any given time. For example,  in November 2008, we requested the latest version of the Integrated Master  Test Plan and were told that the latest approved version was 8.4, which  was revised in July 2008. However, the signature page for that version is  from a prior version—version 8.1. Since there is no signature page  referring to version 8.4, it appears that this version is unapproved though  MDA officials told us that it was being used to manage the BMDS.
Agency officials maintain that the document is used to manage tests and  associated requirements for key test events. However, it is unclear how  well the baseline is integrated with other key aspects of testing such as the  acquisition of targets needed to execute tests. For example, in some  instances, targets are placed on contract for two or more years in advance  of the planned tests. Yet, the test baseline—the Integrated Master Test  Plan—does not appear to include events beyond the following fiscal year  that are key to the BMDS test program. As we reported in September 2008,  MDA officials acknowledged that its target contracts did not capture all  element testing requirements and target baselines were not always  established before targets contracts were signed.
section_title: Conclusions
paragraphs: In 2002, MDA was given unusual authorities to expedite the fielding of an  initial BMDS capability. As this initial capability was fielded in 2005, it  showed the benefits of these flexibilities. MDA has improved on this  capability in the ensuing years, including 2008, the focus of this report.  Today, the program is still operating at a fast pace, as production and  fielding of assets outstrips the ability to test and validate them. A collateral  effect of these flexibilities has been reduced visibility into actual versus  planned progress. Some fundamental questions of an oversight nature are  not yet answerable. For example, a comparison of actual versus planned  costs at the system or asset level is not yet possible, nor is an assessment  of the performance of the fielded system as a whole. Beginning in 2007,  MDA began efforts to improve visibility into its actual performance,  beginning with the new way of defining blocks, coupled with DOD’s  creation of the MDEB. However, progress has been slow in some areas  and value for money cannot be satisfactorily assessed. Delays are  especially important in a program of this size, as a year delay in  establishing cost baselines means another $8 billion to $10 billion may be  spent in the meantime.
With the transition to a new administration, the deployment and  subsequent improvement of an initial capability, a new agency Director,  and a new block structure for managing the BMDS, an opportunity exists  to revisit and strengthen the processes by which MDA operates. Looking  to the future, decision makers in Congress and DOD face multi-billion  dollar investment decisions in allocating funds both within MDA and  between MDA and other DOD programs. At this point, a better balance  must still be struck between the information Congress and DOD need to  conduct oversight of the BMDS and the flexibility MDA needs to manage  across the portfolio of elements that collectively constitute the system’s  capability.
At this point, the balance does not provide sufficient information for  effective oversight. In particular:    Total cost and unit cost baselines have not been set and contract  baselines are subject to frequent changes. Even if such baselines are  set as planned, they will only capture about 26 percent of MDA’s work.
Less testing is conducted than planned, thus delaying the validation of  the models and simulations needed to assess the overall performance  of the BMDS. Moreover, test plans do not hold and are revised often, in  many cases due to the poor performance of target missiles. The  current test plan is at risk given its ambitious scope.
Manufacturing, production, and fielding are outpacing testing,  modeling, and validation. Consequently, fielding decisions and  capability declarations are being made with limited understanding of  system effectiveness.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Defense direct the MDEB to assess  how the transparency and accountability of MDA’s acquisitions can be  strengthened to enhance oversight, such as by adopting relevant aspects of  DOD’s normal requirements, acquisition and budgeting processes, without  losing the beneficial features of MDA’s existing flexibility.
In the near term we recommend that the Secretary of Defense direct MDA  to undertake the following 10 actions:  In the area of cost:  1.  Complete total cost baselines before requesting additional funding  for Blocks 2.0 and 3.0 and commit to a date when baselines for all  blocks will be established.  2.  Ensure that transfers of work from one block to another are  transparent and reported as cost variances.  3.  Provide additional unit costs reports, beyond flyaway unit costs,  that incorporate both procurement and research and development  funding so that there is a more comprehensive understanding of  the progress of the acquisitions.
In the area of testing and performance:  4.  Expand the BMDS test baseline to include tests scheduled beyond  the first succeeding year of the plan to ensure its synchronization  with BMDS contracts.  5.  Ensure that DOT&E is consulted before making significant changes  to the test baseline so that the tests planned provide DOT&E with  sufficient data to assess the performance of the BMDS elements.  6.  Ensure that planned test objectives include concrete data  requirements anchoring models and simulations to real-world tests,  synchronized with flight and ground test plans and that the effects  on models and simulations of test cancellations, delays or  problems are clearly identified and reported.  7.  Reassess the flight tests scheduled for the end of fiscal year 2009 to  ensure that they can be reasonably conducted and analyzed given  targets and other constraints.
In the area of knowledge-based decisions:  8.  Synchronize the development, manufacturing, and fielding  schedules of BMDS assets with the testing and validation  schedules to ensure that items are not manufactured for fielding  before their performance has been validated through testing.  9.  Conduct a flight test of the CE-I EKV against a complex target  scene with countermeasures to complete MDA’s previous testing  goal of understanding the performance capabilities of the first 24  fielded GBIs.  10. Strengthen the capability declarations by using the complete  analysis from annual performance assessments as the basis for  declaring engagement sequence groups as fully capable and block  development as fully complete; otherwise, indicate the limitations  of the capabilities and steps that MDA will take to reduce the risks.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD provided written comments on a draft of this report. These  comments are reprinted in appendix I. DOD also provided technical  comments, which were incorporated as appropriate.
DOD fully concurred with 10 of our 11 recommendations and partially  concurred with our recommendation that the Secretary of Defense direct  MDA to synchronize the development, manufacturing, and fielding  schedules of BMDS assets with testing and validation schedules to ensure  that items are not manufactured for fielding before their performance has  been validated through testing. Yet, even DOD’s response to this  recommendation appears to be, in substance, concurrence.
DOD concurred with our recommendation that the Secretary of Defense  direct MDA to ensure that transfers of work from one block to another are  transparent and reported as cost variances. DOD noted in its response that  MDA will report block baselines and variances annually to Congress in the  BMDS Accountability Report. The Department further noted that for the  purposes of unit cost reporting, MDA has defined a cost variance as a  confirmed increase of 10 percent or more in block or unit costs when  compared to the current cost baseline or 20 percent or more compared to  the original cost baseline, stating that transfers of work creating such cost  variances will be reported. The intent of our recommendation is to  increase visibility into transfers of work between blocks regardless of the  amount of the increase or the baseline status of the blocks. The trigger for  reporting the variances selected by DOD will not necessarily provide that  visibility. Given that only between 2 and 26 percent of BMDS block and  capability development costs from fiscal year 2010 to 2013 will be  baselined initially, visibility into transfers into blocks that are not yet  baselined may not occur. Further, an increase may not be reported in the  baselined block from which work is transferred because the transfer  would actually yield a decrease in the cost of the baselined block. An  increase would also not be reported in the receiving block if that block is  not baselined or if the transfer did not increase costs above the threshold.  MDA may need to consider a different approach to reporting that captures  meaningful transfers of work into and out of blocks regardless of whether  any of the blocks are baselined. MDA should work with Congress to  determine what constitutes a meaningful or significant cost increase.
DOD also concurred with our recommendation that the Secretary of  Defense strengthen the capability declarations by using the complete  analysis from annual performance assessments. In responding to our  recommendation, DOD noted that if there is limited performance  assessment data, the overall capability assessment will factor in the  knowledge gained from ground tests and flight tests against the identified  risks. While we recognize that MDA is not always able to complete all of  its planned tests in a given time period, when MDA decides to change the  planned basis for its capability declarations to a different or reduced set of  data it is important for the agency to clearly report the limitations that  affect the capability declaration as well as the mitigation steps it is taking.
We are sending copies of this report to the Secretary of Defense and to the  Director, MDA. In addition, the report will be available at no charge on the  GAO Web site at http://www.gao.gov.
If you, or your staff have any questions concerning this report, please  contact me at (202) 512-4841. Contact Points for our offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. The major contributors are listed in appendix VI.
section_title: Appendix I: Comments from the Department of Defense
section_title: Appendix II: BMDS Prime Contractors Exceed Budgeted Cost and Schedule Performance during Fiscal Year 2008
paragraphs: Based on our analysis of 14 Ballistic Missile Defense System (BMDS)  elements’ prime contractor earned value management performance, we  determined that collectively the contractors overran budgeted cost by  $152.4 million and were behind schedule by approximately $107.4 million  during the fiscal year. Our insight of the dollar gained or lost for each  dollar invested is based on monthly earned value reports which are  required of each BMDS program office’s prime contractor. These reports  compare monthly progress to the cost or schedule performance baseline  to reveal whether the work scheduled is being completed on time and if  the work is being completed at the cost budgeted. For example, if the  contractor was able to complete more work than scheduled and for less  cost than budgeted, the contractor reports a positive schedule and cost  variance. Alternatively, if the contractor was not able to complete the  work in the scheduled time period and spent more than budgeted, the  contractor reports both a negative schedule and cost variance. The results  can also be mixed by, for example, completing the work under cost (a  positive cost variance) but taking longer than scheduled to do so (a  negative schedule variance).
We also used contract performance report data to base predictions of  likely overrun or underrun of each prime contractor’s budgeted cost at  completion. Our predictions of final contract cost are based on the  assumption that the contractor will continue to perform in the future as it  has in the past. In addition, since they provide the basis for our projected  overruns, we also provide the total budgeted contract cost at completion  for each contract we assessed in this appendix. However, the budgeted  costs at completion, in some cases, have grown significantly over time. For  example, in one case the budgeted cost at completion increased by  approximately five times its original value. Since our assessment does not  reveal, as cost growth, the difference between the original and current  budgeted costs at completion it would be inappropriate to compare the  underruns or overruns for MDA programs with cost growth on major  defense acquisition programs since those major defense acquisition  programs have established their full scope of work as well as developed  total cost baselines, while these have not been developed for MDA  programs.
subsections: section_title: Aegis BMD Contractors Experienced Mixed Performance during the Fiscal Year
paragraphs: The Aegis Ballistic Missile Defense (Aegis BMD) program manages two  prime contractors for work on its two main components—the Aegis BMD  Weapon System and the Standard Missile-3 (SM-3). We report on work  under one of the two separate Aegis BMD SM-3 contract’s contract line  item numbers (CLIN)on which we received sufficient performance data  during fiscal year 2008. The first Aegis BMD SM-3 contract’s CLIN 9 was  for the production of 20 Block 1A missiles which began in February 2007  and finished deliveries in August 2008. Deliveries were completed $7.5  million under budget on the contractor’s total budgeted cost of $179.0  million. The other Aegis BMD SM-3 contract’s CLIN 1 is for a fourth lot of  27 Block 1A missiles and began reporting performance data in August 2007  for work that is still ongoing. The weapon system contractor experienced  cost growth and schedule delays while the SM-3 contractor for the ongoing  CLIN 1 for 27 Block 1A missiles had mixed performance. Neither of these  CLINs experienced a realignment during fiscal year 2008.
The Aegis BMD weapon system contractor experienced cumulative cost  growth and schedule delays throughout the year. The Aegis BMD weapon  system contractor overran budgeted cost and schedule during the fiscal  year by $7 million and $5.1 million respectively. Although cumulative cost  performance remains positive at $16 thousand, cumulative schedule  performance continued to decline to negative $8.4 million. The negative  cumulative schedule variance is driven by late engineering data, delays to  qualification efforts, and the need to return components experiencing  issues back to the vendor which required more time than originally  planned. See figure 7 for cumulative cost and schedule performance  during the fiscal year.
The program attributes the fiscal year cost and schedule overruns mainly  to the additional time and testing needed to ensure that the weapon  system fielded capability was what was originally promised to the  warfighter. To account for some of the overruns, the program performed  fewer risk reduction efforts for a future weapon system capability release.  If the contractor continues to perform as it has during the fiscal year, we  project that at contract completion in September 2010, the contractor will  overrun its budgeted cost of $1.2 billion by between $1.9 million and $12.2  million.
The Aegis BMD SM-3 contractor, producing another lot of 27 Block 1A  missiles under its CLIN 1, ended the fiscal year by underrunning budgeted  costs by $3.0 million. The Aegis BMD SM-3 contractor for CLIN 1 work  also ended the year with a negative $7.6 million schedule variance, which  means that the contractor was unable to accomplish $7.6 million worth of  planned work. Since reporting began in August 2007, cumulative and fiscal  year variances are nearly equal with cumulative cost variances at a  positive $3.3 million and cumulative schedule variances at negative $7.0  million. See figure 8 for a graphic representation of the cumulative cost  and schedule variances during fiscal year 2008.
The contractor was able to accomplish fiscal year 2008 work for $3.0  million less than originally planned in part due to adjustments made in  program management, labor efficiencies, reductions in vendor cost, and  material transfers in the missile’s fourth stage component. The  unaccomplished work in negative $7.6 million worth of fiscal year  schedule variances is largely in the first, second, and fourth stages portion  of work. In the first stage booster, the contractor attributes some of the  negative schedule variance to more than a year delay in testing the first  stage due to rework needed to correct errors in the original drawing  packages. In addition, the contractor cites second stage component  delivery delays as drivers for the negative schedule variance. Vendors  were unable to deliver these components due to holdups in approving  waivers, achieving recertification after test equipment failures, and  property damage to facilities. Lastly, the contractor experienced delays in  components for the fourth stage which also contributed to the unfavorable  schedule variance. If the contractor continues to perform as it did through  September 2008, our analysis predicts that, at completion in April 2010, the  work under the contract could cost from $6.6 million less to $0.7 million  more than the budgeted cost of $237.5 million.
section_title: ABL Contractor Overran Budgeted Fiscal Year Cost
paragraphs: For fiscal year 2008, the Airborne Laser (ABL) contractor overran fiscal  year budgeted costs by $10.6 million but had a positive fiscal year schedule  variance of $2.2 million. Despite some gains in its schedule variance during  the fiscal year, the program still maintains negative cumulative cost and  schedule variances of $84.8 million and $23.6 million respectively. The  contractor mostly attributes the negative cumulative variances in cost and  schedule to late beam control/fire control hardware deliveries. Despite a  replan in June 2007, the ABL contractor did not perform any type of  realignment during fiscal year 2008. Figure 9 shows cumulative variances  at the beginning of fiscal year 2008 along with a depiction of the  contractor’s cost and schedule performance throughout the fiscal year.
Technical issues with key components of the Beam Control Fire Control  system that required new hardware or refurbishment of existing  components as well as late deliveries of key laser system components are  the primary drivers of the unfavorable fiscal year cost variance of $10.6  million. These issues have caused delays in integration and test activities  for the overall ABL weapon system. Based on the contractor’s  performance up through fiscal year 2008, we estimate that, at completion  in February 2010, the contractor will overrun its budgeted cost of $3.6  billion by between $89.7 million and $95.4 million.
section_title: C2BMC Program Incurred Negative Cumulative and Fiscal Year Variances
paragraphs: Our analysis of the Command, Control, Battle Management, and  Communications’ (C2BMC) cumulative contract performance indicates  that the prime contractor’s performance declined during fiscal year 2008.  The contractor overran its fiscal year 2008 budget by $9.8 million and did  not perform $3.6 million of work on schedule. By September 2008, this  resulted in an unfavorable cumulative cost variance of $24.3 million and an  unfavorable cumulative schedule variance of $7.1 million. The main  drivers for the negative cumulative cost variances were costs associated  with unplanned work, increased technical complexity, and reduction to  cost efficiency due to losing key staff. The contractor attributes the  unfavorable cumulative schedule variances to software issues related to  the global engagement manager and components of test training  operations. Although the C2BMC contractor performed a replan in  November 2006, the contractor did not perform any type of realignment  during fiscal year 2008. Trends in cost and schedule performance during  the fiscal year are depicted in figure 10.
The negative fiscal year cost variance of $9.8 million is driven mainly by  problems in the performance of work under Part 4 and Part 5 of the  contract. The Part 4 effort, which began in December 2005, includes  completing several spiral capabilities, upgrading spiral suites, and  implementing initial global engagement capabilities at an operations  center. The Part 5 effort began in December 2007 and covers operations  and sustainment support for fielded C2BMC; the delivery of spiral  hardware, software, and communications; as well as development,  planning, and testing for other spiral capabilities. The contractor was able  to use reserves to cover some of its Part 4 unfavorable fiscal year cost  variances.
The Part 5 fiscal year cost variance’s primary drivers are unexpected  complexities with the network design, unplanned work that required more  resources for developing the planner, and the extension of efforts past the  completion date on the global engagement management portion of work.  The unfavorable fiscal year schedule variance of $3.6 million is  attributable to the Part 5 portion of work and primarily caused by an  unexpected reallocation of resources off of the global engagement  management portion of work to other areas, delays in requesting material  procurement also for global engagement management, and a lagging  schedule for building out a testing lab. If the contractor continues to  perform as it has in the past, we predict that the contractor will overrun its  budgeted cost of $1.0 billion at completion in December 2009 by between  $37.1 million and $76.8 million.
section_title: GMD Contractor Maintained Negative Cumulative Cost and Schedule Variances throughout the Fiscal Year
paragraphs: The government and the Ground-based Midcourse Defense (GMD)  contractor began a contract restructuring during the fiscal year to rephase  and rescope on-going efforts to refine capability requirements and to  adjust program content as well as perform weapon system integration,  perform flight test planning, and work to develop the two-stage booster  among other tasks. The ongoing realignment includes a proposal to add  between $350 million and $580 million to the cost of the work under  contract and to add 39 months to the period of performance.
The GMD contractor reports a cumulative negative cost variance of more  than $1.0 billion that it attributes to technical challenges with its  Exoatmospheric Kill Vehicle (EKV) as well as supplier component quality  problems. The contractor also carries a total unfavorable cumulative  schedule variance of $130.3 million, the bulk of which the contractor  attributes to the technical issues connected with the Ground-based  Interceptor (GBI), particularly the EKV. For example, during the fiscal  year the program experienced difficulties in manufacturing the Capability  Enhancement II (CE-II) EKVs. Although the CE-II EKVs are expected to  provide better performance, the contractor produced the kill vehicles  before completing developmental tests, discovered problems during  manufacturing, incorporated a new design, and continued manufacturing  them. Although these issues contributed unfavorable fiscal year cost  variances of $42.7 million, the program was able to make up for these  losses in other areas. The variances, depicted in figure 11 represent the  GMD contractor’s cumulative cost and schedule performance over fiscal  year 2008.
The GMD contractor did have a favorable fiscal year cost variance of $53.9  million, which it attributed to labor efficiencies in the ground system as  well as less field maintenance support required than planned, and labor  efficiencies in the deployment and sustainment portion of the work under  the contract. However, the GMD element’s underruns occurred partially  because the contractors delayed or eliminated some planned work. For  example, the GMD program did not accomplish the emplacement of three  GBIs, or conduct either of its two planned flight tests. As a result, it  employed less labor than originally intended. The program also reports an  unfavorable fiscal year schedule variance of $77.4 million which it  attributes to an administrative error that occurred in September 2007. This  error incorrectly adjusted the baseline to the booster effort in September  which was then updated in October. However, it should also be noted that  Missile Defense Agency (MDA) officials believe that ongoing adjustments  to the GMD element’s baseline have skewed recent variances to such a  degree that they should not be used to predict future costs. We did  perform analysis based on the contractor’s reported performance through  fiscal year 2008, and our analysis estimates that at contract end planned  for December 2011, the contractor could overrun its budgeted cost of  $14.9 billion by between $950.2 million and $1.25 billion.
section_title: KEI Cost and Schedule Performance Continued to Decline after Replan
paragraphs: Despite a replan in April 2007 and again in April 2008, the Kinetic Energy  Interceptors (KEI) contractor continued to experience declining cost and  schedule performance during the fiscal year. Although the contractor  began the year with a positive cost variance, the contractor overran fiscal  year 2008 budgeted costs by $8.3 million, ending the year with an  unfavorable cumulative cost variance of $2.6 million. In addition, the  program was unable to accomplish $8.5 million worth of work which  added to an unfavorable cumulative schedule variance of $21.3 million.  Cumulative cost and schedule variances were mainly driven by costs  associated with delays to booster drawing releases, delays in procurement,  and unexpected costs and rework related to issues with the second stage.  Figure 12 depicts the cost and schedule performance for the KEI  contractor during fiscal year 2008.
The KEI contractor attributes its unfavorable fiscal year cost and schedule  variances of $8.3 million and $8.5 million, respectively, to issues with its  interceptor booster. Problems initially arose in fiscal year 2007 with a  motor case failure during acceptance testing which led to unexpected  redesigns. In October 2007, the program experienced several issues with  the nozzle during a second stage ground test and also experienced a  deviation in measured performance from pre-test predictions. These issues  added costly redesigns and delays to its knowledge point, a booster flight  test. The program performed a replan of its work in April 2008 because of  these issues in order to realign the schedule with their booster flight test  knowledge point which was delayed from August 2008 to April 2009. Since  the replan, the booster flight test has been further delayed to the fourth  quarter of fiscal year 2009. As a result of the replan, the program zeroed  out some schedule variances from the baseline to reflect the program’s  progress toward the newly defined schedule. Despite this replan in April,  our analysis shows that the replan has not improved overall performance  as cumulative cost and schedule variances continue their downward trend.  We were unable to estimate whether the total work under the contract is  likely to be completed within budgeted cost since trends cannot be  developed until at least 15 percent of the work under the contract is  completed.
section_title: Limited Contractor Data Prevented Analysis of All MKV Task Orders
paragraphs: The Multiple Kill Vehicles (MKV) program began utilizing an indefinite  delivery indefinite quantity contract in January 2004. Since then, the  program has initiated eight task orders, five of which were open during  fiscal year 2008—Task Orders 4 through 8. Task Order 4 provided  insufficient data to complete full earned value analysis for the fiscal year.  In addition, Task Order 5 was completed shortly after the fiscal year  began, without providing enough data to show performance trends.  Therefore we performed analysis for Task Orders 6, 7, and 8 as shown  below. None of the task orders were realigned during the fiscal year.
MKV Task Order 6 began in November 2006 for the component  development and testing of a prototype carrier vehicle seeker (a long-  range sensor). According to the task order, this seeker for the carrier  vehicle will assign individual kill vehicles for target destruction. This task  will culminate in a demonstration planned for fiscal year 2010. As shown  in figure 13 below, performance data over the course of the fiscal year  illustrates declining cost and schedule performance. Although it began the  fiscal year with slightly positive cumulative cost and schedule variances,  the program ended the year with slightly negative cumulative cost and  schedule variances of $1.1 million and $0.6 million respectively. In  addition, the contractor has unfavorable fiscal year cost and schedule  variances of $1.4 million and $1.5 million, respectively. The program  attributes its negative cumulative cost and schedule variances to increased  work necessary to resolve software development issues, unplanned efforts  as a result of late hardware arrivals, and a government-directed change in  vendors for hardware resulting in additional design work. Based on our  analysis and the assumption that the contractor will continue to perform  as it has through fiscal year 2008, we predict that at its contract  completion in May 2009, the contractor on Task Order 6 will overrun its  budgeted cost of $19.3 million by between $1.6 million and $2.5 million.
The MKV Task Order 7 is for the development and testing of engagement  management algorithms and the test bed in which it will be demonstrated.  These algorithms will be a critical capability of the carrier vehicle to  manage the kill vehicle engagements relying on target information from  the BMDS sensors and the carrier vehicle long-range sensor. The  contractor on this task order performed positively during the fiscal year,  both in terms of its cumulative and fiscal year cost and schedule variances.  The program had a favorable fiscal year cost variance of $1.4 million and a  positive fiscal year schedule variance of $11 thousand, adding to its  favorable cumulative cost and schedule variances of $1.7 million and $0.1  million, respectively. The program attributes its cumulative cost underruns  to several reasons including a programmatic decision to proceed with one  approach for organizing kill vehicles in attack formation rather than  funding several different approaches. In addition, the contractor  experienced cost savings with greater efficiencies than expected in the kill  vehicle portion of the work under the contract and less manpower than  planned in other portions of the work under the contract. If the contractor  continues to perform as it has in the past, we estimate that at completion  in May 2010 the work under the contract could cost between $3.2 million  and $3.9 million less than the expected $43.9 million budgeted for the  work under the contract. See figure 14 below for an illustration of  cumulative cost and schedule performance during fiscal year 2008.
MKV Task Order 8 was awarded in January 2007 and began reporting full  performance data in July 2007. The task order is for the development and  testing of a hover test bed and hover test vehicle. This hover test bed will  allow the program to integrate and test key components of the system in a  repeatable ground-based free flight environment as their technologies  reach maturity. The program experienced a continuing schedule  performance decline as seen in figure 15.
Although the contractor began the year with a positive cumulative cost  variance, overruns during the fiscal year of $10.7 million led the program  to a total cumulative cost overrun of $10.3 million. The element’s fiscal  year schedule variance was slightly negative at an unfavorable $15  thousand, leaving its cumulative schedule variance largely unchanged at a  favorable $0.3 million. The program attributes the cumulative cost  variances to increased labor, procurement, and material costs as well as  increased hardware and engineering drawings, and management oversight  to resolve subcontractor inefficiencies. In addition, the program increased  expenditures to resolve technical and schedule issues associated with the  development of avionics subsystems. The planned date for the task order’s  main effort—completing the hover test—was delayed 2 months from its  original date to December 2008 in part because of technical issues  associated with the test vehicle’s power unit and a software anomaly.  These issues were resolved prior to the hover test being conducted. Based  on its prior performance, the MKV contractor could overrun the budgeted  cost of $48.0 million for the work under the contract at completion in  January 2009 by between $5.7 million and $13.8 million.
section_title: Sensors’ Radar Experienced Fiscal Year Cost and Schedule Growth
paragraphs: As of September 2008, the Sensor’s contractor had overrun its fiscal year  budget by $2.2 million and was behind in completing $27.4 million worth  of work. Considering prior years’ performance, the contractor is  performing under budget with a favorable cumulative cost variance of  $22.0 million. However, the contractor has a cumulative unfavorable  schedule variance of $9.6 million. The contractor reports the cumulative  schedule variance is driven by delays in the manufacturing of the sixth  radar and a software capability release that is 2 to 3 months behind  schedule. Additionally, the contractor reports that its favorable cumulative  cost variance is attributable to efficiencies in the second radar’s  manufacturing, design, development, and software. The Sensors  contractor has not performed a realignment of its work since contract  start in April 2003. See figure 16 for trends in the contractor’s cost and  schedule performance during the fiscal year.
The contractor reports that its unfavorable fiscal year schedule variance of  $27.4 million is due to a decrease of previously earned positive schedule  variances reaped from the manufacturing efficiencies leveraged from the  Terminal High Altitude Area Defense radar hardware design. In addition,  late delivery of components also contributed to the negative fiscal year  schedule variances. The negative fiscal year cost variance of $2.2 million is  largely due to a contract change related to its incentive fee. Our analysis  predicts that if the contractor continues to perform as it has through fiscal  year 2008, the work under the contract could cost from $25 million less to  $9.1 million more than the budgeted cost of $1.1 billion at completion  currently planned for December 2010.
section_title: Technical Issues Drove STSS Cost Growth during the Fiscal Year
paragraphs: After a replan of work in October 2007, the Space Tracking and  Surveillance System (STSS) contractor experienced an unfavorable cost  variance of $87.9 million during the fiscal year. The replan was undertaken  in order to extend the period of performance and delay the launch date of  its demonstrator satellite. Despite fiscal year cost overruns, the contractor  was able to make gains on the cumulative schedule variance by  accomplishing $1.9 million more worth of work than was originally  planned. Cumulatively, the program has both unfavorable cost and  schedule variances at $319.3 million and $17.8 million, respectively. The  program attributes cumulative cost variances and schedule variances to  continual launch date schedule slippages. In addition, problems in the  space segment portion of work also added to the cumulative cost  variances. Figure 17 shows both cost and schedule trends during fiscal  year 2008.
Program cost variances during the fiscal year were driven primarily by  technical issues with hardware installed on the second space vehicle.  These issues included an overheating flight communications box, a leak on  the propulsion side of the satellite, and problems with the spacecraft  processor that failed to send a critical command to the onboard computer.  To resolve the issues with the processor, the program office initially  recommended the removal of the entire computer from the spacecraft.  However, after extensive research and testing, the program manager  determined that the event with the spacecraft is an unverifiable failure  with a low probability of occurrence and low mission impact and decided  not to remove the computer from the spacecraft to resolve the issue. We  estimate that if the contractor continues to perform as it has through fiscal  year 2008, the work under the contract at completion in September 2011  could exceed its budgeted cost of $1.6 billion by between $621.7 million  and $1.2 billion.
section_title: Targets and Countermeasures Program’s Rebaseline Positively Affected Fiscal Year Schedule Variances
paragraphs: In June 2008, a delivery order under the Targets and Countermeasures’  element that is developing a new family of targets—the Flexible Target  Family (FTF)—performed a rebaseline as a result of experiencing  manufacturing delays to several components. The majority of the delays  were from qualification failures, subsequent redesigns, and requalification  efforts. The rebaseline was to realign the work under the contract to  reflect realistic hardware delivery dates. This rebaseline did not affect cost  variances, but did rebaseline major milestone delivery dates and, as a  result, set some of the previously existing schedule variances to zero.
The Targets and Countermeasures contractor made gains with a favorable  $23.2 million fiscal year schedule variance due in part to the rebaseline in  June 2008. However, the contractor ended the year with an unfavorable  cumulative schedule variance of $6.4 million which was primarily driven  by delays in the completion of the FTF qualification program. The  program also ended the year with a cumulative cost variance of $52.8  million which the contractor attributed to costs associated with the FTF’s  avionics components integration and qualification issues, and more effort  than expected required on motors for one of the targets in the program.  See figure 18 below for an illustration of cumulative cost and schedule  variances during the course of the fiscal year.
The contractor attributes its unfavorable fiscal year cost variances of $35.7  million to the increased cost of completing the first four FTF 72-inch  targets. Delays in completing component qualification extended the period  of performance which invariably led to higher costs. In addition, the  contractor cites cost increases to the failure of one of its targets in July  2008 that added mission assurance and testing cost to the follow-on  mission using the same target configuration. We estimate that if the  contractor continues to perform as it has in the past, it will overrun its  budgeted cost of $1.1 billion at contract’s end in December 2009 by  between $63.7 million and $75.9 million.
section_title: THAAD Contractor Spent More Money and Time Than Budgeted
paragraphs: The Terminal High Altitude Area Defense (THAAD) program experienced  target issues during fiscal year 2008. The THAAD program performed a  realignment in May 2008 to extend the flight test program after  experiencing several delayed target deliveries. Because of the cost impact  of these delayed targets, the program will increase its value of the work  under its contract by approximately $80 million. The THAAD program  performed a similar realignment in December 2006 as a result of delayed  target deliveries as well. As a result of this realignment, the program  extended its flight test program and added an estimated $121 million to the  value of work under its contract.
The THAAD contractor experienced downward trends in its cost and  schedule performance during fiscal year 2008. The program overran its  budgeted costs for the fiscal year by $33.5 million. It was also unable to  accomplish $7.4 million worth of work during the fiscal year. Both of these  unfavorable variances added to the negative cumulative cost and schedule  variances of $228.7 million and $16.5 million, respectively, as shown in  figure 19.
The THAAD prime contractor’s fiscal year cost overrun of $33.5 million  was primarily caused by the radar, missile, and launcher portions of work.  Design problems delayed the prime power unit design review and slowed  parts production, causing the radar’s negative cost trend. In addition, the  missile’s negative cost trend for this same period was driven by design  complexity, ongoing rework/retest of subsystems, unexpected  qualification discoveries, and unfavorable labor variances at key  subcontractors. Lastly, the launcher variances were driven by hardware  and software complexities and higher-than-expected costs for  transitioning a portion of this effort to a different facility for production.
The contractor reports that its unfavorable fiscal year schedule variance of  $7.4 million is primarily driven by the radar and missile components. The  radar’s negative schedule variance is associated with vendor delays in  delivering trailers for both of the system’s prime power units. The late  delivery of the trailers has subsequently delayed delivery of the prime  power units. Missile rework due to qualification test discoveries also  negatively affected schedule performance. If the contractor continues to  perform as it has through fiscal year 2008, we project that at the contract’s  completion currently scheduled for September 2009, the contractor could  overrun its budgeted cost of $4.6 billion by between $252 million to $274  million.
section_title: Appendix III: FTG-04 Flight Test Cancellation
paragraphs: On May 23, 2008, the Senate Armed Services Committee requested that we  review the reasons behind the cancellation of a GMD flight test designated  FTG-04. Initially, on May 1, 2008, the Director, MDA decided to delay this  test due to problems discovered in a telemetry device, the Pulse Code  Modulation Encoder (PCME). This device does not affect operational  performance, but rather is a critical component needed to transmit flight  test data only. The PCME problems were due in large part to  manufacturing defects, which the manufacturers and MDA concluded  likely affected all the PCMEs. However, on May 8, the Director of MDA  instead decided to cancel this flight test entirely, resulting in one less GMD  end-to-end intercept flight test. MDA told us that delaying the flight test  until the PCMEs could be repaired would cause delays in future tests since  various test assets were shared. MDA officials therefore decided to cancel  FTG-04 and transfer some test objectives to other tests, including a new  non-intercept flight test, FTX-03, and an already planned intercept flight  test, FTG-05. Also, for some remaining objectives not captured in FTG-05  and FTX-03, MDA stated that it planned a third intercept test, FTG-X. We  were asked to investigate this test cancellation and answer the following  questions:    Why did the MDA change its initial decision to delay FTG-04 until  November 2008 and decide to cancel FTG-04 instead and what  deliberative process did MDA follow in deciding to cancel FTG-04?
When and how, if ever, will each of the specific test objectives  previously planned for FTG-04 be accomplished?
What are the implications of canceling this flight test on the ensuing  test program, on demonstrating the capability of the GMD system, and  on other programmatic decisions?
subsections: section_title: Faulty Telemetry Component Caused Delay and Subsequent Cancellation of FTG- 04
paragraphs: MDA initially delayed the FTG-04 flight test because of defects in the  PCME, a telemetry component in the Exoatmospheric Kill Vehicle (EKV)  only needed to gather test data. Although the PCME does not affect  operational performance, it is needed for test assets to determine if design  and operational issues have been resolved. The FTG-04 had four prior  delays and was originally scheduled for the first quarter of fiscal year 2007.  In responding to these delays, multiple tests over several years were  affected.
Several defects contributed to the problem, the first three of which are  presumed to affect all PCMEs manufactured up to that point and all 24  fielded Test Bed/Capability Enhancement (CE) I EKVs:    The PCMEs experienced gold embrittlement due to lack of pretinning.
Insufficient oscillator stand-off height increased thermal stress.
Circuit board deflection caused by three washers missing from the  board.
In addition to these manufacturing defects, there were stress fractures in  the solder of three PCMEs caused by the removal and replacement of a  chip on the device. This chip was removed because a clock on a chip was  asynchronous with another component’s clock. It was estimated that there  was an 18 to 48 percent chance of the loss of telemetry data at some point  during a flight test due to the asynchronous chip problem. Again, all 24  fielded Test Bed/CE-I EKVs have the chip with this problem. This chip  does not affect operational performance, but rather is a critical component  needed to transmit flight test data only.
See table 10 for timeline of events related to this cancellation.
The contractor, Boeing, and the subcontractors, Raytheon and the  manufacturer of the component, L-3, took actions to mitigate the problem.  They eliminated the gold embrittlement problem by sending the oscillator  out for pretinning, they designed custom washers for two already  produced PCMEs and raised three bosses for new PCMEs to eliminate the  need for washers, and they tightened tolerances on the board to eliminate  the deflection issue. These first three PCME manufacturing improvements  were finalized on May 16, 2008. They also made changes to correct the  chip with the asynchronous clock problem in all newly manufactured  PCMEs. None of the previously fielded GBIs will be refurbished with  improved PCMEs needed for flight tests, but the GBIs emplaced starting in  October 2008 and thereafter have the improved PCME.
On May 1, 2008, MDA’s Program Change Board considered five options.  1.  Execute FTG-04 as scheduled, using payload (32) “as is”  2.  Continue diagnostic testing of payload 32, but if decision was made  that it was not ready, substitute payload 33, leading to a delay in  the test schedule  3.  Refurbish payload 32, but if it did not improve, substitute payload  4.  Immediately replace payload 32 with 33 without further testing  5.  Immediately return payload 32 for repair  The Director, MDA chose option 5, delaying the FTG-04 into the November  to December 2008 timeframe, but keeping the program on track to provide  this intercept data as planned. MDA consulted the test community,  Director, Operational Test and Evaluation (DOT&E) and the BMDS  Operational Test Agency, on this initial decision to delay the test and both  agreed with this decision. According to MDA, the Director also asked for  options for a sensor test in the summer of 2008.
On May 8, 2008, MDA’s Program Change Board reconvened to consider  three options for a sensor test (FTX-03) and canceling instead of delaying  FTG-04:  1.  Conduct FTX-03, with a baseline like the planned FTG-04, but  without a live intercept attempt.  2.  Conduct FTX-03 with a baseline like the FTG-05, an intercept flight  test to be conducted in December 2008.  3.  Similar to option 2, but with more sensor data collected.
The Director MDA changed the May 1 decision to delay, refurbish and fly  the planned FTG-04 test and chose instead to cancel FTG-04 and pursue  the modified option 3 above. Choosing option 3 resulted in restructuring  the intercept test into a test designed to assess multiple sensor integration  capability. This new test benefited sensor modeling and simulation as  post-flight reconstruction could occur now on two missions.
According to MDA, it canceled the FTG-04 at the May 8, 2008 meeting  instead of delaying it, in part, because rescheduling FTG-04 would have  caused a major delay in another test, the Distributed Ground Test-03  (GTD-03). GTD-03 and FTG-04 required many of the same assets, so  conducting FTG-04 would have delayed GTD-03 and thus the delivery of  this new capability by four months. Ground tests assess the increased  BMDS capability to be fielded next and GTD-03 was to provide the means  by which a more realistic simulation of threats and scenarios and the  means by which new software capability could be declared ready to move  into the operational baseline. MDA consulted with BMDS Operational Test  Agency officials on this decision and they supported it. DOT&E was not  consulted on this decision and expressed concern that the elimination of  any intercept test reduced the opportunity to gather additional data that  might have increased confidence in models and simulations. DOT&E has  repeatedly expressed concerns over the lack of test data needed to  validate MDA’s models and simulations.
section_title: Most FTG-04 Test Objectives Will Be Allocated to Follow- on Tests
paragraphs: According to MDA, all FTG-04 test objectives were allocated to other flight  tests. However, partly due to differences in how MDA describes test  objectives, it is unclear whether all planned FTG-04 test objectives will be  accomplished in follow-on tests. The loss of a primary objective, an  intercept of a complex target scene, will slow MDA’s efforts to build  confidence in the EKV’s ability to consistently achieve intercepts, unless  an additional intercept is scheduled. In August 2008, MDA informed  Congress that it planned to conduct a new intercept test called FTG-X in  fiscal year 2009. However, in January 2009 MDA stated that the FTG-X  intercept test was never formally approved and is no longer planned.
In addition, some test objectives related to modeling and simulations have  been redefined so it is unclear whether they will be fully tested. Models  and simulations are critical to understanding and assessing the  performance of the BMDS because flight tests are limited by their cost,  complexity, and range safety constraints. Modeling and simulation is  therefore the primary way to fully assess the overall performance of the  BMDS and its various components. According to DOT&E, cancellation of  FTG-04 reduced interceptor and EKV data available for modeling, leaving  only two intercepts (FTG-3a and FTG-05) that have provided complete sets  of information. In October 2008, MDA stated that modification of FTG-04  into a sensor test eliminated a second opportunity to anchor the models of  EKV-fielded software. Test objectives in MDA planning documents  describe modeling and simulation objectives at a high level. However, it is  difficult to determine whether modeling and simulation objectives are  addressed in the near term because the objectives are defined differently  for each test. For example, the FTX-03 and FTG-05 objectives do not  distinguish between primary and secondary objectives while the FTG-04  does. One objective that seems to be absent in the FTG-05 and FTX-03 is to  collect data to support validation and anchoring of system-level (vs.  element-level) simulations, MDA stated that the exclusion from the test  objectives was inadvertent and it will be addressed by the tests.
BMDS Operational Test Agency objectives related to the GBI engagement  were not met, although the test agency officials indicate the majority of  their non-intercept sensor related objectives for FTG-04 were met in the  FTX-03 test. However, BMDS Operational Test Agency officials state that  some of their intercept objectives may be addressed through a  combination of previous intercept test, FTG-03a, and recently conducted  FTG-05. Finally, several warfighter objectives for FTG-04, related to  tactics, techniques, and procedures, will be met through the ground tests  instead because, according to the warfighter representative at the BMDS  Operational Test Agency, flight tests do not offer the best opportunity to  assess this kind of objective.
section_title: Cancellation Eliminates One of Few Opportunities to Demonstrate GMD Capabilities
paragraphs: The cancellation has increased the strain on the ensuing test program.  GMD’s current plans call for two intercept attempts in fiscal year 2009— FTG-05, which was conducted in December 2008, and FTG-06—and one  booster verification test. This is an ambitious schedule as GMD has been  able to conduct only one intercept flight test per year—FTG-02 in  September 2006, FTG-03a in September 2007 and FTG-05 in December  2008. MDA had planned to conduct five intercept tests with varying  stresses to assess the EKV capability between February 2007 and  December 2008. Flight test failures and test plan revisions caused MDA to  only carry out two intercept tests in that period—FTG-03a and FTG-05—  both of which resulted in an intercept.
In addition, the number of future flight tests planned has been reduced.  MDA has not funded or scheduled an intercept replacement for FTG-04. In  January 2008 MDA decided to merge two intercept tests—FTG-06 and  FTG-07—into one single intercept attempt. This merger removes another  opportunity to gather end-game EKV performance data needed to assess  capability. In January 2008 MDA also decided to accelerate a two-stage  verification non-intercept test required to assess the European component.
The cancellation of FTG-04 removed one chance to obtain end-game  performance data needed to develop GMD models and to assess the  capability of the CE-I EKV. The repetition of intercept-related objectives is  important to build confidence in the intercept capability. These models are  the primary way to fully assess the overall system performance, since  flight tests are limited by their cost, complexity and range safety concerns.  MDA planned to test the CE-I EKV against a dynamic target scene with  countermeasures in both FTG-04 and FTG-05. FTG-04 was canceled and an  FTG-05 target anomaly affected this objective. According to MDA, no more  CE-I EKV flight tests have been approved, although it is considering  whether to conduct an intercept test using a CE-I EKV in the future. GMD  developed some mitigations to various developmental issues, but realistic  flight testing is needed to anchor the models and to determine the  effectiveness of these mitigations.
The test cancellation and target problems have reduced the knowledge  that MDA expected to use for its upcoming end-to-end performance  assessment. Performance assessments are annual system-level  assessments to test, evaluate, and characterize the operational capability  of the BMDS as of the end of the calendar year. Currently, MDA has only  completed one—Performance Assessment 2007. Furthermore, acting on a  joint recommendation between MDA and the Operational Test Agency,  MDA officials canceled their 2008 performance assessment efforts in April  2008 because of developmental risks associated with modeling and  simulations. Instead, MDA is focusing on testing and models for  Performance Assessment 2009. However, the planned performance  information available for Performance Assessment 2009 will be reduced.  The FTG-04 cancellation reduced one set of data that was expected to be  available. In addition, both FTX-03 and FTG-05 will be used to anchor data  for Performance Assessment 2009, but target anomalies in each test  precluded the completion of all planned test objectives. Neither target  presented the complexity needed for advanced algorithm development.
Manufacturing and emplacement continue unabated by reductions and  delays in tests. Twenty-four CE-I GBIs have been fielded and the new CE-II  GBIs are now being fielded without important knowledge about the  systems capabilities expected to be gained through tests. The first CE-II  GBI emplacement occurred prior to any flight testing of this configuration.  The first flight test is FTG-06 currently scheduled to occur no earlier than  the fourth quarter of fiscal year 2009. According to MDA, these CE-II GBIs  will not be declared operational until after the successful completion of  FTG-06. FTG-04 was also identified as a key source of data supporting a  number of capabilities declarations. The cancellation of FTG-04, plus other  testing delays, prompted MDA to defer some capabilities and to declare  others based on previous tests.
section_title: Conclusions
paragraphs: The cancellation of the FTG-04 flight test increases the risk to the GMD  program and to the overall BMDS capability, since the lack of adequate  intercept data adversely affects confidence that the system could perform  as intended in a real-world situation. The GMD program has reduced its  plans to assess operational performance of the fielded configuration  between February 2007 and December 2008 from five to two intercept  tests, leaving gaps in knowledge about the repeatability of the  performance of fielded assets. In addition, the opportunity to obtain  additional intercept data vital to the anchoring of models and simulations  has been lost, unless the FTG-X flight test is conducted, adding to an  existing concern expressed by DOT&E. Despite test reductions and effects  on assessing system-level performance, production and fielding of assets  continues as planned.
section_title: Appendix IV: Reduced Basis for Capability Declarations
paragraphs: The two tables below list test events supporting MDA capability  declarations during fiscal year 2008 for certain engagement sequence  groups in Blocks 1.0 through 3.0 (see table 11), as well as for the full  completion of Bock 1.0 by the end of fiscal year 2009 (see table 12). Both  tables illustrate that MDA reduced the basis for declaring certain  engagement sequence groups as early or fully capable. The basis for  declaring an early, partial, or full capability includes flight and ground  tests as well as performance assessments.
section_title: Appendix V: Scope and Methodology
paragraphs: To examine the progress MDA made in fiscal year 2008 toward its cost,  schedule, testing, and performance goals, we examined the efforts of 10  BMDS elements that MDA is developing and fielding. The elements  included in our review collectively accounted for 80 percent of MDA’s  fiscal year 2008 research and development budget requests. In assessing  each element, we examined the BMDS Fiscal Year 2008 Statement of  Goals, Program Execution Reviews, test plans and reports, production  plans, Contract Performance Reports, MDA briefings, and earned value  management data. We developed data collection instruments that were  completed by MDA and each element program office. The instruments  gathered detailed information on planned and completed program  activities including tests, design reviews, prime contracts, estimates of  element performance, and challenges facing the elements. In addition, we  discussed fiscal year 2008 progress and performance with officials in  MDA’s Agency Operations Office, each element program office, as well as  the Office of DOD’s Director, Operational Test and Evaluation, and DOD’s  Operational Test Agency. To assess each element’s progress toward its  cost goals, we reviewed Contract Performance Reports and, when  available, the Defense Contract Management Agency’s analyses of these  reports. We applied established earned value management techniques to  data captured in Contract Performance Reports to determine trends and  used established earned value management formulas to project the likely  costs of prime contracts at completion.
To evaluate the sufficiency of MDA’s modeling and simulation practices,  we reviewed DOD and MDA policies, memos, flight and test plans related  to modeling and simulations, the Acquisition Modeling and Simulation  Master plan, as well as verification, validation and accreditation plans and  reports for various elements, and MDA white papers discussing modeling  and simulation techniques. We also interviewed officials in element  program offices to discuss modeling and simulation plans and procedures  particular to each.
In assessing MDA’s accountability, transparency, and management  controls, we interviewed officials from the Office of the Under Secretary  of Defense’s Office for Acquisition, Technology, and Logistics, as well  officials in the MDA Agency Operations Directorate. We also reviewed an  Institute for Defense Analysis study, two Congressional Research Service  reports, a Congressional Budget Office report, U.S. Code, DOD acquisition  system policy, various DOD directives, the Missile Defense Executive  Board charter, and various MDA statements and documents related to the  agency’s block structure.
To ensure that MDA-generated data used in our assessment are reliable,  we evaluated the agency’s management control processes. We discussed  these processes with MDA senior management. In addition, we confirmed  the accuracy of MDA-generated data with multiple sources within MDA  and, when possible, with independent experts. To assess the validity and  reliability of prime contractors’ earned value management systems and  reports, we interviewed officials and analyzed audit reports prepared by  the Defense Contract Audit Agency. Finally, we assessed MDA’s internal  accounting and administrative management controls by reviewing MDA’s  Federal Manager’s Financial Integrity Report for Fiscal Years 2003, 2004,  2005, 2006, 2007, and 2008.
Our work was performed primarily at MDA headquarters in Arlington,  Virginia. At this location, we met with officials from the Aegis Ballistic  Missile Defense Program Office; Airborne Laser Program Office;  Command, Control, Battle Management, and Communications Program  Office; MDA’s Agency Operations Office; DOD’s Office of the Director,  Operational Test and Evaluation; and the Office of the Under Secretary of  Defense for Acquisition, Technology and Logistics. In addition, in  Huntsville, Alabama, we met with officials from the Ground-based  Midcourse Defense Program Office, the Sensors Program Office, the  Terminal High Altitude Area Defense Project Office, the Kinetic Energy  Interceptors Program Office, the BMDS Kill Vehicles Program Office, the  Targets and Countermeasures Program Office, and the Office of the  Director for BMDS Tests. We also met with Space Tracking and  Surveillance System officials in El Segundo, California.
In December 2007, the conference report accompanying the National  Defense Authorization Act for Fiscal Year 2008 noted the importance of  DOD and MDA providing information to GAO in a timely and responsive  manner to facilitate the review of ballistic missile defense programs.  During the course this audit, we experienced significant delays in  obtaining information from MDA. During the audit, MDA did not provide  GAO staff with expeditious access to requested documents which delayed  some audit analysis and contributed to extra staff-hours. Of the documents  we requested, we received approximately 19 percent within the 10-15  business day protocols that were agreed upon with MDA. Pre-existing  documentation took MDA on average about 50 business days to provide  and many pre-existing documents took over 100 business days to be  provided to GAO. Notwithstanding these delays, we were able to obtain  the information needed to satisfy our objectives in accordance with  generally accepted government auditing standards.
We conducted this performance audit from May 2008 to March 2009 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: Acknowledgments
paragraphs: In addition to the contact named above, David Best, Assistant Director;  LaTonya Miller; Beverly Breen; Ivy Hübler; Tom Mahalek; Steven Stern;  Claire Cyrnak; Isabella Johnson; Meredith Allen Kimmett; Kenneth E.  Patton; Karen Richey; and Alyssa Weir made key contributions to this  report.