Missile Defense: Mixed Progress in Achieving Acquisition Goals and Improving Accountability
Released: 2014-04-01T00:00:00

Letter

Background

Ballistic missiles have different ranges—short, medium, intermediate, and intercontinental—as well as different speeds, sizes, and performance characteristics. Short-range ballistic missiles have a range of less than 621 miles; medium-range ballistic missiles have a range from 621 to 1,864 miles; intermediate-range ballistic missiles have a range from 1,864 to 3,418 miles; and intercontinental ballistic missiles have a range greater than 3,418 miles. As a result, MDA is developing a variety of systems that, when integrated, provide multiple opportunities to destroy ballistic missiles in flight for the strategic defense of the United States and regional defense of its deployed forces and allies. The BMDS includes space-based sensors; ground- and sea-based radars; ground- and sea- based interceptor missiles; and a command and control system that provides communication links to the sensors and interceptor missiles.

Once a ballistic missile has been launched, these sensors and interceptors track and engage the threat missile during its flight as shown in figure 1.

MDA’s Flexible Acquisition Approach

When MDA was established in 2002, the Secretary of Defense granted it exceptional flexibility to set requirements and manage the acquisition of the BMDS in order to meet a presidential directive to deliver an initial defensive capability against ballistic missiles in 2004. This flexibility allows MDA to develop BMDS elements outside of DOD’s standard acquisition process until they are mature enough to be handed over to a military service for production and deployment. Because the BMDS’s entrance into DOD’s acquisition process is deferred, certain laws and policies that generally require major defense acquisition programs to take certain steps at certain phases in the DOD acquisition process do not yet apply to MDA. For example, before a major defense acquisition program begins the product development phase, it must document key performance, cost, and schedule goals in an acquisition baseline that has been approved by a higher-level DOD official. This acquisition baseline is used to measure a program’s performance as it progresses. Specifically, as implemented by DOD, major defense acquisition programs’ baselines provide decision makers with key goals such as the program’s total cost for an increment of work, key dates associated with acquiring a capability, and the weapon’s intended performance. Additionally, once a baseline has been approved, DOD’s major defense acquisition programs are required to measure performance against their baseline and report certain changes to Congress. For instance, they are required to report certain increases in unit cost (cost divided by the quantity produced) measured from the original and the current program baseline.

While this flexibility allows MDA latitude to manage the BMDS and enable it to rapidly develop and field new systems, we have previously reported that the agency has used this flexibility to employ acquisition strategies with high levels of concurrency (that is, overlapping activities such as testing and production), which increases the risk for performance shortfalls, costly retrofits, and test problems. this flexibility has hampered oversight and accountability.MDA officials, MDA has taken some steps to identify and track concurrency in their programs.

We have also reported that Congress has taken steps to improve the transparency and accountability of BMDS development efforts. For example, in the National Defense Authorization Act for Fiscal Year 2008, Congress required MDA to establish cost, schedule, and performance baselines for certain BMDS elements. MDA first reported baselines for several BMDS elements to Congress in its June 2010 BAR and has continued to report baselines annually. Table 1 describes the six acquisition baselines MDA established and reports in its BAR for individual BMDS elements or major portions of such elements.

GAO-12-486. 10 U.S.C. § 2432 defines, with respect to a major defense acquisition program, procurement unit cost as the amount equal to (1) the total of all funds programmed to be available for obligation for procurement for the program divided by (2) the number of fully configured end items to be procured. Program acquisition unit cost is defined as the amount equal to (1) the total cost for development and procurement of, and system-specific military construction for, the acquisition program divided by (2) the number of fully configured end items to be produced for the acquisition program.

Most recently, the National Defense Authorization Act for Fiscal Year 2012 amended MDA’s baseline reporting requirements.law currently requires MDA to report to the congressional defense committees certain changes or variances in the current baselines from the baselines presented in the prior year’s report and from when the baselines were initially established. Additionally, the act allows MDA to revise an initial baseline, which the agency refers to as a “revised initial baseline.”

Specifically, the In 2010, MDA also established an acquisition process that continues to guide the development of the BMDS. Table 2 identifies the five life cycle phases of MDA’s acquisition process.

The agency has documented the key knowledge that is needed prior to the technology development, product development, initial production, and production phases. For example, prior to entering initial production, an element must demonstrate that its design and manufacturing processes are stable, planned quantities are affordable, and developmental and operational test results show that the user’s needs will be met. In general, developmental testing is aimed at determining whether the system design will satisfy the desired capabilities, while operational testing is aimed at determining whether the system is effective, survivable, and suitable in the hands of the user under realistic conditions. Additionally, according to DOD policy, programs entering initial production or production require approval from DOD’s Under Secretary of Defense for Acquisition, Technology, and Logistics.

Description of BMDS Elements

Table 3 describes the BMDS elements and programs assessed in this report and their current MDA acquisition phase.

Missile Defense Acquisition Changes in Fiscal Year 2013

In March 2013, in response to a growing threat from Iran and North Korea, the Secretary of Defense announced steps that affected the acquisition of the BMDS including deploying 14 additional ground-based interceptors in Fort Greely, deploying a second AN/TPY-2 radar to Japan; and shifting the resources from development of the Aegis BMD SM-3 Block IIB interceptor, which was planned to be deployed after 2020 to defend against intercontinental ballistic missiles, to fund the additional ground-based interceptors, as well as develop advanced technology to improve the performance of current and future versions of BMDS interceptors.

In 2013, DOD canceled the Aegis BMD SM-3 Block IIB and the Precision Tracking Space System citing concerns with the programs’ high-risk acquisition strategies and technical challenges that GAO has previously raised. We have previously reported that MDA did not consider a broad range of alternatives or fully assess program or technical risks before committing to either program.

In addition, MDA altered its fiscal year 2013 acquisition plan to offset a funding reduction of $568 million (6.8 percent) in its total available budget for fiscal year 2013. For example, in consultation with key stakeholders such as the Operational Test Agency and DOD’s Office of the Director, Operational Test and Evaluation, MDA revised its test plan by combining, delaying, and deleting tests to cut costs. MDA also delayed some development activities for various elements into fiscal year 2014 and beyond.

BMDS Flight Tests Successfully Executed, but Development Challenges Continue

In fiscal year 2013, MDA successfully executed several flight tests that demonstrated key BMDS capabilities and modifications made to resolve prior development issues, but continued to experience failures and delays resulting in less testing and production than planned. For the first time in September 2013, the Aegis BMD and Terminal High Altitude Area Defense (THAAD) programs participated in an operational flight test that resulted in a near simultaneous engagement. Additionally, the Aegis BMD program also successfully conducted flight tests with the SM-3 Block IB missile. However, according to officials from the DOD’s Office of the Director, Operational Testing and Evaluation, Aegis BMD experienced a SM-3 Block IB missile failure that is currently being investigated and could result in a modification to a component that is common between the Aegis BMD SM-3 Block IA and IB. Lastly, GMD also successfully conducted a non-intercept test of its upgraded interceptor that is currently in development, but experienced an intercept test failure of the fielded interceptor, the cause of which is still unknown. Further details on the operational test are provided after table 4. In addition, because ongoing testing and development challenges increase the potential to affect the production of the SM-3 Block IB missile as well as delaying understanding of the capabilities and limitations of GMD’s fielded interceptor, we also provide additional details for these programs after the table.

Table 4 presents a summary of key accomplishments and challenges for BMDS elements and programs that are in the BAR.

First System-level Operational Test of the BMDS

After more than 11 years of development of the BMDS, MDA conducted the first system-level operational missile defense flight test called Flight Test Operational-01 (FTO-01) in September 2013. During the test, warfighters from several combatant commands employed multiple missile defense systems including Aegis BMD and THAAD to demonstrate the regional capabilities of U.S. missile defense. This is a significant achievement because it is the first operational test that involved multiple elements working simultaneously. To conduct this test, MDA invested in range assets and conducted other activities to ensure it could test multiple elements at once. For example, MDA conducted its first integrated system-level flight test, known as Flight Test Integrated-01, in October 2012 as a risk-reduction exercise for the operational test. During FTO-01, MDA launched two nearly simultaneous threat- representative medium-range ballistic missile targets including its air- launched extended-medium range ballistic missile (eMRBM) target for the first time.because of development problems associated with the eMRBM target. MDA also had to make some adjustments to the FTO-01 test plan because of fiscal year 2013 sequestration. Although MDA preserved its primary objective to demonstrate the interoperability of BMDS elements, it reduced the number of targets included in the test from five to two and removed participation of more mature elements such as the Patriot Advance Capability-3.

This test was delayed for approximately one year in part The BMDS elements successfully engaged the targets during the test, but according to independent testing officials, full system integration was not achieved. Specifically, according to the DOD’s Director, Operational Test and Evaluation, the Aegis ship successfully intercepted one of the targets with a SM-3 Block IA and THAAD successfully intercepted a medium- range target for the second time. In addition, as a planned demonstration of its layered defense, THAAD launched a second interceptor at the target intercepted by the Aegis ship as a contingency in event the SM-3 Block IA did not achieve an intercept. However, DOD’s Director, Operational Test and Evaluation, also found that the test failed to achieve full integration between all systems due to challenges with system networks, limitations in elements’ ability to work together and component failures. For example, the test uncovered several issues with communication networks that are needed for interoperability between all elements. Interoperability is important because it can improve the missile defense effectiveness and enhance individual systems performance beyond operating alone.

Aegis BMD SM-3 Block IB

The Aegis BMD SM-3 Block IB program largely overcame previous development challenges and successfully intercepted all targets in its last three flight tests as shown in table 5. These tests are required for a full production decision—the last key production authorization by the Under Secretary of Defense, Acquisition, Technology, and Logistics that would allow MDA to produce the remaining 415 interceptors. However, a missile failure of the second interceptor launched during the September 2013 test could increase production risk if design changes are needed.

As we found in April 2013, the SM-3 Block IB production line has been repeatedly disrupted since 2011 due to flight test anomalies caused by malfunctions in two separate sections of the third-stage rocket motor, and development challenges with the throttleable divert and attitude control system—components that maneuver the interceptor in its later stages of flight. These challenges delayed the SM-3 Block IB full production authorization by more than two years to fiscal year 2015. Largely resolving these previous challenges, in fiscal year 2013 the program received permission to procure 33 additional initial production missiles. Although MDA initially planned to award a contract for 29 SM-3 Block IB missiles in fiscal year 2013, it bought four additional missiles in August 2013 to recover an earlier reduction. That reduction occurred to provide funds to resolve technical and production issues. Based on successful intercepts of the last three flight tests, the program also received permission to buy 52 more interceptors in fiscal year 2014.

Despite the three successful intercepts, the effect of the missile failure in September 2013 on the upcoming full production decision remains unclear. Before the program enters into full production, MDA’s acquisition management instruction requires it to demonstrate to the Under Secretary of Defense, Acquisition, Technology, and Logistics that there are no significant risks to production and that the planned production quantities are affordable and fully funded. The permission to enter full production is also based on independent assessments of the weapon’s effectiveness and suitability by the DOD’s Director, Operational Test and Evaluation and the Navy’s Commander Operational Test & Evaluation Force. Although the failure investigation is ongoing, preliminary results indicate that the failure occurred in the third-stage rocket motor, a component common to the SM-3 Block IA, which is nearing the end of its production. Different issues with that same component have contributed to previous SM-3 Block IB schedule delays and production disruptions. While the precise cause of the September 2013 failure is under review, MDA documentation indicates that it could potentially result in design changes to the third-stage rocket motor and changes to manufacturing processes. Additionally, retrofits may be required for SM-3 Block IB and SM-3 Block IA interceptors that were already produced. If design changes are necessary, program documentation indicates that they will not be flight tested until the fourth quarter of fiscal year 2015, just prior to the planned deployment of the SM-3 Block IB to support the regional defense of Europe and 6 months after its planned full production decision. Consequently, until the program thoroughly understands the extent of needed modifications, if any, and their effects on performance as demonstrated though testing, its production strategy is at risk of cost growth and schedule delays. MDA has experienced these consequences in other elements when it pursued design changes concurrently with production.

Although the GMD program made progress in resolving a prior CE-II intercept failure, test failures and development challenges continue to disrupt the program and increase the cost to demonstrate the new CE-II. The GMD program first attempted to demonstrate the CE-II interceptor in January 2010 but subsequently experienced a number of setbacks in both the CE-II and the fielded CE-I, as seen in table 6 below.

Developing a mitigation to the FTG-06a failure has proven more difficult than initially expected. The program initially planned to conduct FTG-06b in the third quarter of fiscal year 2012 but the test has since been delayed to at least the third quarter of fiscal year 2014 because of challenges resolving test failures. For example, while initial results from CTV-01 indicated the redesigned guidance system component could be used to resolve the problem that caused the FTG-06a failure, subsequent ground testing revealed that only one-third of those produced could be used in future interceptor production or flight tests because the component’s performance was uncertain. The program mitigated the issue by implementing software and hardware modifications and delivered the redesigned component for kill vehicle integration in October 2013. However, according to MDA, the program experienced further delays in the FTG-06b test while it implemented changes based on assessments from the ongoing FTG-07 failure review. Consequently, confirmation that the CE-II design works as intended has been delayed by nearly seven years and costs have increased by over $1 billion because of the CE-II development challenges and test failures.

In July 2013, MDA conducted the FTG-07 developmental test to understand the performance of the fielded CE-I against a longer range target in more challenging conditions and assess the performance of upgrades. This interceptor was fielded before completing developmental testing, leading MDA to undertake retrofit efforts and upgrades to fix issues identified during testing. According to acquisition best practices, developmental testing should be complete before beginning production and fielding in order to, among other reasons, avoid the need for retrofits and upgrades to fix issues discovered during testing. The test failed, delaying understanding of the capabilities and limitations of upgrades to the fielded CE-I. Shortly after the test failure, the Director, MDA stated a failure review was initiated to not only identify the root cause of the failure, but also provide a comprehensive review of potential CE-I failures and identify any correlations with the CE-II. Since then, according to program officials, MDA has identified a kill vehicle component common to both interceptors that could be associated with the FTG-07 failure. However, it remains unclear what, if any, design changes, retrofits, or other corrective actions to the CE-I or CE-II are necessary since the failure review is not complete. According to MDA officials, they have not determined if they will re-conduct the flight test. If the CE-I is not flight tested again, the warfighter will not have a full understanding of the capabilities and limitations of the upgrades to the CE-I interceptor, the original purpose of the FTG-07 test.

Overall, GMD’s ongoing testing issues in conjunction with concurrent acquisition practices have caused—and will likely continue to cause— major disruptions to the program. We previously found that, in 2004, MDA committed to a highly concurrent development, production, and fielding strategy for the CE-II interceptor and began delivering interceptors in 2008. Because MDA moved forward with CE-I and CE-II interceptor production before completing its flight testing program, test failures have exacerbated disruptions to the program. For example, because the program has delivered approximately three-fourths of the interceptors for fielding, the program faces difficult and costly decisions on how it will implement corrections from prior test failures. Also, the program has had to add tests that were previously not planned and delay tests that are necessary to understand the system’s capabilities and limitations. As a result of these development challenges, the GMD program will likely continue to experience delays, disruptions, and cost growth.

MDA Made Its Resource and Schedule Baselines Clearer, but Further Actions Needed to Enhance Their Use for Oversight

MDA has taken some steps to improve the clarity of its resource and schedule baselines, but issues with the content and presentation of these baselines continue to limit the usefulness of the information available to decision makers for oversight of BMDS development efforts. Since 2011, we have found deficiencies in the quality of the cost estimates that underpin MDA’s resource baselines and reported on the efforts MDA has undertaken to improve those estimates. In 2013, we found the agency made little progress addressing the underlying quality issues with those cost estimates that we raised. As a result, this is the fourth year we have found that the cost estimates that support MDA’s resource baselines are not sufficiently reliable to support oversight. However, according to MDA officials, the agency is taking steps to improve the quality of its cost estimates to support the resource baselines it plans to report in its 2014 BAR. Assessing MDA’s progress in achieving its schedule goals is also difficult because MDA’s schedule baselines are not presented in a way that allows decision makers to understand or easily monitor progress. Until MDA improves the quality and comprehensiveness of its cost estimates and the clarity of its schedule information, its baselines may not be useful for decision makers.

MDA Made Limited Improvements to Its Resource and Schedule Baselines

In its 2013 BAR, MDA continued to make useful changes to its reported resource and schedule baselines. We found in March 2011 that MDA’s schedule and resource baselines had several shortcomings that limited their usefulness for oversight, such as not explaining variances or significant changes in the baselines. Additionally, we found in April 2013 that, in its 2012 BAR, MDA only reported annual progress by comparing its current estimates for unit cost and scheduled activities against the prior year’s estimate and adjusted the content of the baselines from year to year in such a way that they were no longer comparable. As a result, MDA’s baselines were not useful for tracking longer term progress or for holding the agency accountable.

MDA took some action to improve the completeness and clarity of the BAR baselines by: identifying the date of the initial baseline and, if applicable, the date when the initial baseline was most recently revised for each element or major portion of an element reported in the BAR; explaining most of the significant cost and schedule changes from the current baseline estimates against both the estimates reported in the prior year’s BAR and the latest initial baseline; and making the baselines easier to read by removing cluttered formatting such as strikethroughs and highlights that made some of the events listed in past BARs unreadable.

MDA’s 2013 Resource Baselines Remain Unreliable

GAO-11-372 In March 2011, we assessed MDA’s life-cycle cost estimates using the GAO Cost Estimating and Assessment Guide, which is based on best practices in cost estimating and identifies key criteria for establishing high-quality cost estimates. Our review found that the estimates we assessed were not comprehensive, lacked documentation, were not completely accurate, or were not sufficiently credible. We recommended that MDA (1) take steps to ensure its cost estimates are high quality, reliable cost estimates that are documented to facilitate external review and (2) obtain independent cost estimates for each baseline. resource baselines reported in the 2013 BAR. We have found that completing these steps could further improve the quality of MDA’s cost estimates.

GAO-13-432 In April 2013, we recommended that the Secretary of Defense direct the MDA Director to include in its resource baseline cost estimates all life cycle costs, specifically the operations and support costs, from the military services in order to provide decision makers with the full costs of ballistic missile defense systems.

Additionally, MDA has made little progress improving the comprehensiveness of the cost estimates that support its resource baselines. Similar to past years, the cost estimates reported in the 2013 BAR also do not include the operation and support costs funded by the individual military services, which we concluded in April 2013 may result in significantly understated life cycle costs for some BMDS elements. In response to our April 2013 recommendation, DOD agreed that decision makers should have insight into the full life cycle costs of DOD programs, but the department stated that the BAR should only include content for which MDA is responsible. Because MDA already reports the estimated acquisition costs and some of the operation and support costs for the acquisitions in the annual BAR, we concluded that annual document to be the most appropriate way to report the full costs to Congress. Additionally, we concluded that good budgeting requires that the full costs of a project be considered when making decisions to provide resources and, therefore, both DOD and Congress would benefit from a comprehensive understanding of the full costs of MDA’s acquisition programs. Until MDA’s resource baselines are based on reliable information and are comprehensive, they will not be useful for decision makers to understand progress or make well-informed investment decisions.

In the National Defense Authorization Act for Fiscal Year 2014, Congress took steps to address concerns over MDA’s cost estimates by requiring MDA to report to the congressional defense committees on its efforts to improve the quality of the cost estimates included in its acquisition For example, the act requires MDA to report on a description baselines.of and schedule for planned actions to improve its cost estimates, as well as an assessment of how the planned improvements align with GAO’s cost estimating best practices. We are also required to provide our views on the content of MDA’s report. Additionally, the act requires that the life cycle cost estimate included in the agency’s acquisition baselines include a description of the operations and support functions and costs for which the military services are responsible, in addition to the costs borne by MDA.

MDA’s Schedule Baseline Reporting Makes Assessing Progress Difficult

MDA’s schedule baselines are presented in a way that makes it difficult for decision makers to understand a program’s planned activities and therefore hold programs accountable for their performance. According to GAO’s Schedule Assessment Guide, a reliable program schedule includes all activities required to complete a project, but the schedule should not be too detailed to interfere with its use.presenting decision makers with a high-level summary of the schedule is a best practice because schedules that include too many milestones or have too much detail make it difficult to manage progress. Additionally, the activities included on the schedule should have descriptive names that clearly communicate the work required. MDA’s 2013 BAR schedule baselines include numerous events but provide very little information about them, making it difficult to understand what the events are and why they are important. For example, the milestones identifying significant increases in performance for C2BMC Spiral 8.2 are numbered with no description of the capabilities they represent. Additionally, several of the events reported for Aegis modernized weapon system software are titled with abbreviations that are not explained in the BAR.

In addition, MDA does not present any comparisons of event dates with previously reported dates. In contrast, DOD’s major defense acquisition programs report a comparison of current schedule estimates against their original and current schedule goals. According to GAO’s Schedule Assessment Guide, comparing the current schedule to the baseline schedule to track deviations from the plan provides decision makers valuable insight into program risk and can help identify where corrective action may be needed. While removing the formatting that identified changes in prior BARs made the schedule baselines easier to read, doing so removed the ability for decision makers to see if the planned dates for events had changed. As a result, decision makers must consult past versions of the BAR to identify any changes in the planned schedule for a specific event, which can be difficult or impossible in some cases. For example, we found in April 2013 that we were unable to compare the current estimated dates for the activities presented in the Aegis Ashore schedule baseline to the dates baselined in the 2010 BAR because activities were split into multiple events, renamed, eliminated, or moved to several other Aegis BMD schedule baselines.audit we raised this issue with MDA and, according to agency officials, MDA is open to considering alternative formats for presenting the schedule baseline in future version of the BARs. Until MDA improves the content of its schedule baselines, decision makers will not be able to assess how a program is performing over time.

Conclusions

During fiscal year 2013, MDA was able to make some significant acquisition progress, including the first operational system-level flight test involving multiple BMDS elements, but it continued to experience difficulties achieving its goals for testing. This has resulted in delaying progress on individual elements, delaying understanding of the overall performance of the BMDS, and fielding assets before all testing is complete. The most significant acquisition effects have been experienced on the Aegis BMD SM-3 Block IB and GMD program where testing and development challenges have led to failure investigations and increased the risk of continued cost growth and schedule delays. Both programs conducted flight tests and made progress in resolving design flaws in 2013, but still have further development, testing, and production issues to address. For Aegis BMD SM-3 Block IB, the failure of its interceptor in a September 2013 flight test means that a key component may need to be redesigned and that change confirmed to work in additional flight testing. For GMD, the failure of the deployed CE-I interceptor in a July 2013 flight test compounds its challenge because the program did not gain the expected understanding of the effectiveness of software upgrades planned for the operational fleet and now must determine the cause of the failure. As a result, for both programs, to the extent that software or hardware changes are necessary to resolve the cause of these failures, new flight tests will likely be needed to demonstrate both the effectiveness of any resolutions and, for GMD, understand the performance of the software upgrades that were the original purpose of the test.

Additionally, for over a decade, we have reported that MDA provides Congress with only limited insight into the acquisition progress for individual programs. While MDA has taken steps to improve the clarity of the baselines it reports to Congress, the agency’s cost and schedule reporting still lacks the quality, completeness, and clarity necessary to track actual cost or schedule growth over time. Specifically, the agency has not addressed all of the critical gaps in the quality of its underlying cost estimates used to develop its resource baselines that we have identified over the years. Until corrective actions are implemented and substantial improvements are made to MDA’s cost estimates, its reported resource baselines will not be useful for decision makers to hold MDA accountable for its performance or make informed decisions on how best to allocate limited resources. Congress recently amended the requirements for the cost estimates MDA must report in its baselines, which may enhance the transparency into MDA’s cost estimating processes. As a result, we do not make any new recommendations regarding cost at this time. However, additional actions can be taken in the schedule baselines to improve the ability of decision makers to understand what program events are most critical and identify whether the dates for those critical events have changed. Until improvements are made, the schedule baselines will not be a useful tool for providing oversight of the BMDS.

Recommendations for Executive Action

We recommend that the Secretary of Defense take the following three actions to strengthen MDA’s acquisitions and help support oversight. 1. To the extent that MDA determines hardware or software modifications are required to address the September 2013 Aegis BMD SM-3 Block IB failure, we recommend that the Secretary of Defense direct, a) the Director of the MDA to verify the changes work as intended through subsequent flight testing, and b) the Under Secretary of Defense, Acquisitions, Technology, and Logistics to delay the decision to approve the program’s full production until such testing demonstrates that the redesigned missile is effective and suitable. 2. To demonstrate the CE-I’s effectiveness against a longer range target in more challenging conditions and to confirm the design changes implemented to improve performance, as well as any changes needed to resolve the July 2013 CE-I flight test failure work as intended, we recommend that the Secretary of Defense direct MDA’s Director to conduct a flight test of the CE-I interceptor once the cause of the failure has been determined and any mitigations have been developed. 3. To improve the content of the schedule baselines it reports to Congress for monitoring program performance, we recommend that the Secretary of Defense direct MDA’s Director to take the following actions as MDA implements other improvements required by the Congress: a) Focus the information included in the schedule baselines to highlight critical events. b) For each event included in the schedule baseline, provide a description of the event explaining what it entails and why it is important. c) Present the schedule baseline in a format that allows decision makers to identify any changes made from the current estimated date to the date reported in not only the prior year’s BAR but also to the date established in the initial baseline.

Agency Comments and Our Evaluation

DOD provided written comments on a draft of this report. These comments are reprinted in appendix II. DOD also provided technical comments, which were incorporated as appropriate.

DOD partially concurred with our first recommendation, non-concurred with our second recommendation and concurred with our third recommendation. The department partially concurred with our first recommendation to flight test any modifications that may be required to the Aegis BMD SM-3 Block IB as a result of September 2013 failure, before the Under Secretary of Defense, Acquisitions, Technology, and Logistics approves full production. In its comments, DOD acknowledged that if modifications are required they will be tested, but added that the type of testing—flight or ground testing—will depend on the magnitude of such modifications. The department also believes that the component currently tied to the failure, has a successful testing history and thus expects to meet the reliability requirement needed for the full production decision in fiscal year 2015. However, there have now been three flight test anomalies associated with this component over the last three years. According to Aegis BMD officials, they are considering design changes for this component. Since the fiscal year 2015 full production decision is the commitment by the Under Secretary of Defense, Acquisitions, Technology, and Logistics to produce several hundred missiles, this decision should be supported by an assessment of the final product under operational mission conditions to ensure that it is effective and suitable. As such, we maintain our recommendation that before the program is approved for full production, flight testing should demonstrate that any modifications work as intended.

DOD did not concur with our second recommendation to complete the original purpose of the July 2013 CE-I flight test once the cause of that failure has been determined and any mitigations have been developed. In its response, DOD stated that the decision to flight test a CE-I interceptor will be made by the Director, MDA, based on the judgment of stakeholders from the Office of the Secretary of Defense and combatant commands on the need to perform a test. The DOD response focused almost exclusively on the steps it is taking to identify the cause of the July 2013 failure and mitigate it and did not address the main part of our recommendation—determining the effectiveness of the CE-I under more challenging conditions and confirming that design changes previously made improve performance. These were the objectives of FTG-07. In our view, resolving these performance questions remains important. Since the FTG-07 failure review is still ongoing, we cannot assess whether DOD should conduct a CE-I test for the sole purpose of demonstrating corrective actions, to the extent any are needed, to address the cause of the failure. While we acknowledge that DOD must balance several competing GMD priorities, including which flight tests to conduct, and conducting another CE-I flight test may not be feasible in the immediate future, we also maintain that demonstrating CE-I intercept capabilities should continue to be a priority for DOD since the CE-I interceptor constitutes a multi-billion dollar investment by DOD and serves as the primary defense of the United States homeland against enemy ballistic missile attacks.

In addition to responding to our recommendations, the department’s letter raised additional concerns about our draft report. First, DOD disagreed with our statement that because the BMDS entrance into DOD’s acquisition process is deferred, it is exempt from certain acquisition laws and policies that generally provide oversight of major defense acquisition programs. DOD stated that MDA is not exempt from acquisition-related laws because, while it is not captured by several statutes, Congress has provided legislation specific to MDA to ensure oversight and accountability. We clarified the language in our report to remove the term exempt. However, because of the acquisition flexibility it has been granted, MDA is not yet required to apply certain laws and policies to the BMDS. We have found that while the flexibility allows MDA latitude to manage the BMDS and to rapidly develop and field new systems, we have also found that this flexibility has hampered oversight and accountability. Our report recognizes the actions Congress has taken to improve the transparency and accountability of the BMDS development efforts through legislation specific to MDA, particularly to require MDA to report baselines to Congress. However, there are a number of requirements that are triggered by phases of the DOD acquisition process that are important to sound acquisition management. For example, we have previously found that MDA is not yet required to conduct an analysis of alternatives to compare potential solutions and determine the most cost effective weapon system to acquire nor is MDA yet required to obtain an independent cost estimate prior to beginning product development.

Second, DOD stated that it disagreed with our assessment that MDA’s cost estimates are not sufficiently reliable to support oversight, suggesting that the report be revised to include more of MDA’s efforts to improve the quality of its cost estimates. Since 2011, we have found that there are issues with the cost estimates and baseline reporting, including incomplete cost estimates due to the exclusion of military service operation and support costs as well as instability in the content of the baselines, which makes assessing progress difficult or impossible. While the draft report was being reviewed by DOD, we met with MDA officials who discussed more of their efforts to improve their cost estimates; however, we were not provided sufficient information to change our determination. We did clarify the report to better reflect the efforts they have undertaken. DOD stated in its response that MDA has included previously unreported costs in the baselines it provides to Congress, which we have previously found is an improvement to the amount of information reported to Congress, but which does not demonstrate that the quality of the cost estimates themselves have improved. DOD stated in its response that it has provided joint operation and support costs documentation for two programs reported in the 2013 BAR. However, we did not assess the joint operation and support costs because they were not included in the 2013 resource baselines reported to Congress. DOD also stated in its response that it received an assessment from us on cost estimate documentation for a third program. We did not perform a formal assessment of the third program because it was cancelled and not included in the 2013 BAR. In order to assist MDA in improving its cost estimates, we did informally assess the third program’s cost estimate, but reached no conclusion as to its quality. However, we noted several issues in that informal review. For example, we concluded that because MDA did not provide a cost model to support the estimate we were unable to check the cost estimate for accuracy. Finally, DOD stated in its response that MDA has published and implemented a cost estimating handbook. We have previously found that fully implementing that handbook could improve the quality of MDA’s cost estimates. However, during this review, we specifically asked to review the cost estimate documentation supporting MDA’s fiscal year 2013 BAR baselines in order to assess its progress in implementing that handbook. An MDA senior cost official told us that the agency was working to fill in documentation gaps on existing cost estimates and that the estimates were not ready for us to review. In the course of our work, we concluded and informed MDA that the cost estimating process defined in that handbook has not been applied to any systems that are currently baselined or part of the BMDS. Until MDA is able to provide us with documentation that supports the actual baselines reported to Congress so we can independently assess the quality of the cost estimates, we have no basis to change our assessment.

Third, DOD disputed that MDA has not obtained independent cost estimates from DOD’s Office of the Director of Cost Assessment and Program Evaluation for any of the elements GAO reviewed since 2010. In response, we clarified the language in the report so that it specifically refers to the lack of independent cost estimates completed for any of the resource baselines reported to Congress in the 2013 BAR. We also clarified in the report that DOD’s Office of the Director of Cost Assessment and Program Evaluation has assessed other BMDS costs. DOD was unable to provide us with documentation of independent cost estimates completed for MDA’s BAR baselines, therefore, we have no basis to change our determination.

Lastly, DOD identified 35 “technical and factual errors” in its technical comments. However, upon review we found that 29 were not technical or factual errors, but rather different conclusions, errors in DOD’s comments, or required additional substantiation that was not provided. We determined that 6 were actual technical or factual errors and therefore made the appropriate changes in those circumstances.

We are sending copies of this report to the Secretary of Defense and to the Director, MDA. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

To assess the Missile Defense Agency’s (MDA) progress and any challenges associated with developing, testing, and producing the ballistic missile defense system (BMDS) during fiscal year 2013, we examined the acquisition accomplishments of several missile defense elements and MDA’s targets program. Specifically, we reviewed the Aegis Ballistic Missile Defense (Aegis BMD) with Standard Missile-3 (SM-3) Block IB; Aegis Ashore; Aegis Modernized Weapon System Software; Army/Navy Transportable Radar Surveillance and Control Model 2 (AN/TPY-2); Command, Control, Battle Management, and Communications (C2BMC); Ground-based Midcourse Defense (GMD) System; Targets and Countermeasures; and Terminal High Altitude Area Defense (THAAD) elements because, as reported in the 2013 BMDS Accountability Report (BAR), these elements or programs have entered MDA’s product development, initial production, or production acquisition phase, but are not yet mature enough to be transferred to a military service and enter the We formal DOD acquisition cycle for full-rate production and deployment.reviewed key management documents for fiscal year 2013 including Program and Baseline Execution Reviews, which detailed program accomplishments and areas of concerns, and interviewed program element officials. We also examined MDA’s master test plan and flight test reports, and discussed the element- and BMDS-level test programs and test results with the BMDS Operational Test Agency and the Department of Defense’s (DOD) Office of the Director, Operational Test and Evaluation and office of Developmental Test and Evaluation. In addition, we also met with officials from MDA’s functional directorates including the Engineering Directorate to discuss the agency’s process for delivering and integrating BMDS capabilities as well as the Directorates for Acquisition and Operations to discuss significant internal and external events and decisions that occurred in fiscal year 2013, such as sequestration, that affected the agency’s overall acquisition of the BMDS.

To assess the progress made as well as any remaining challenges MDA faces in reporting resource and schedule baselines that support oversight, we examined MDA’s reported baselines in the 2010, 2011, 2012, and 2013 BARs. To be consistent with last year, we focused our assessment on the resource and schedule baselines as they continue to be the only reported baselines that have measurable goals, such as cost estimates and dates of program events, and separately explain when current estimates have deviated to a certain extent from the baselines set in prior BARs. We also examined the National Defense Authorization Act for Fiscal Year 2012, which required MDA to establish and maintain baselines for program elements or major portions of such program elements and outlined the information to be included in MDA’s baselines, as well as interviewed officials within MDA’s general counsel’s office. We also interviewed officials in MDA’s Acquisitions Directorate about how the agency establishes and manages its acquisition baselines and met with MDA officials in the Operations Directorate to discuss their progress in adopting best practices in cost estimating based on our Cost Guide. We also reviewed findings and recommendations from several of our past reports to see if MDA had made progress in improving the completeness, clarity, and stability of its reported resource and schedule baselines. In addition, we examined DOD acquisition policy such as the Interim DOD Instruction 5000.02 issued in November 2013 and the Defense Acquisition University’s Defense Acquisition Guidebook to discern how other major defense acquisition programs are required to report baselines and measure program progress. We also reviewed GAO’s cost and schedule guides, which outline best practices for establishing and managing program cost and schedule estimates.

To gauge the extent to which MDA reported changes or variances in the current baselines from the baselines presented in the prior year’s BAR and from when the baselines were initially established, we compared the 2013 BAR resource and schedule baselines for each BMDS element in our review to the baselines presented in the 2012 and 2010 BARs. In order to compare unit costs calculated in different years, there were instances where it was necessary to convert prior cost estimates to match the base year of the estimates presented in the 2013 BAR. We performed these conversions using indexes published by the Office of the Secretary of Defense (Comptroller) in the National Defense Budget Estimates, commonly referred to as the “Green Book.”

The National Defense Authorization Act for Fiscal Year 2013 directed GAO to provide separate assessments on several other missile defense related issues. Specifically, GAO was required to provide briefings on our views and to submit reports as soon as practicable to the congressional defense committees our assessments of DOD reports on (1) a comprehensive evaluation of alternatives for the Precision Tracking Space System and its conformance with GAO best practices for analyses of alternatives; (2) the Ground-based Midcourse Defense system’s test plan; (3) the status and progress of regional missile defense programs, including the adequacy of MDA’s existing and planned efforts to deploy a U.S. missile defense in Europe; and (4) the status of efforts to improve the homeland defense capability of the United States. additional mandated work covers the details on many BMDS elements, we do not include appendixes on each of the individual elements as we have done in prior reports under this mandate.

Pub. L. No. 112-239, § 224 (e), § 231 (e), § 229 (c), and § 228 (c). For our assessments of DOD’s reports, see GAO, Missile Defense: Precision Tracking Space System Evaluation of Alternatives, GAO-13-747R (Washington, D.C.: July 25, 2013) and Regional Missile Defense: DOD’s Report Provided Limited Information; Assessment of Acquisition Risks is Optimistic, GAO-14-248R (Washington, D.C.: Mar. 14, 2014). At the time of publication of this report, our work on DOD’s reports on the Ground-based Midcourse Defense system’s test plan and homeland defense is ongoing. officials from the Aegis BMD SM-3 Block IB, Aegis Ashore, and Aegis Modernized Weapon System Software program offices. In Huntsville, we interviewed program officials for BMDS Sensors, C2BMC, GMD, and THAAD as well as officials in MDA’s Acquisition and Cost Directorates.

We also visited several contractor facilities that were working on programs covered in our review. These facilities were located in Huntsville and Courtland, Alabama as well as Tucson and Chandler, Arizona. In Huntsville, we discussed the manufacturing of the Aegis BMD SM-3 Block IB interceptor with Raytheon officials and met with GMD’s prime contractor, Boeing, to discuss progress in resolving development challenges and their plans to deliver additional interceptors. In Courtland, we met with officials from Lockheed Martin to discuss the production of the extended medium-range ballistic missile target, which was used in flight test operational 01 on September 10, 2013. In Tucson and Chandler, Arizona, we met with GMD’s subcontractors Raytheon and Orbital to discuss their progress in resolving development challenges with the interceptor, flight testing, and future development efforts. We also interviewed officials from various testing agencies located in Arlington, Virginia and Huntsville, Alabama. In Arlington, we met with officials from DOD’s Director, Operational Test and Evaluation, as well as DOD’s Director of Developmental Test and Evaluation, to discuss MDA’s test plans and results from recent tests. Lastly, in Huntsville, we spoke with officials from the BMDS Operational Test Agency to discuss MDA’s performance assessment.

We conducted this performance audit from April 2013 to April 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to the contact named above, David Best, Assistant Director; Aryn Ehlow; Meredith Allen Kimmett; Wiktor Niewiadomski; Kenneth E. Patton; John H. Pendleton; Karen Richey; Steven Stern; Roxanna Sun, Robert Swierczek; Brian Tittle; Hai V. Tran; and Alyssa Weir made key contributions to this report.