Missile Defense: Mixed Progress in Achieving Acquisition Goals and Improving Accountability
Published: 2014-04-01T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Since 2002, MDA has spent approximately $98 billion and has requested $38 billion more through fiscal year 2018 to develop, test, and field a system to defend against enemy ballistic missiles. The BMDS is comprised of a command and control system, sensors that identify incoming threats, and intercepting missiles. GAO is mandated by law to assess the extent to which MDA has achieved its acquisition goals and objectives, as reported to Congress through its acquisition baselines, and to report on other issues as appropriate. This report examines the agency's progress and any challenges in fiscal year 2013 associated with (1) developing, flight testing, and producing individual systems, which MDA refers to as BMDS elements; and (2) reporting resource and schedule baselines that support oversight. To support this effort, GAO examined MDA's acquisition and test reports, analyzed two of MDA's acquisition baselines—resource and schedule—to discern progress, and interviewed a wide range of DOD and contractor officials.
section_title: What GAO Found
paragraphs: In fiscal year 2013, the Missile Defense Agency (MDA) made mixed progress in achieving its acquisition goals to develop, test, and produce elements of the Ballistic Missile Defense System (BMDS). For the first time, MDA conducted an operational flight test that involved warfighters from several combatant commands using multiple BMDS elements simultaneously. The agency also successfully conducted several developmental flight tests that demonstrated key capabilities and modifications made to resolve prior production issues. However, the Aegis BMD and Ground-based Midcourse Defense (GMD) continued to experience testing and development challenges.
Aegis BMD—while the program successfully conducted three intercept flight tests with the Standard Missile (SM)-3 Block IB missile in support of a full production decision planned for fiscal year 2015, a missile failed during one of these tests. Although the cause of failure is not known, the program plans to move forward with missile production in 2014. The program is also determining whether a key component that is common with the already fielded SM-3 Block IA missile will need to be redesigned.
GMD—although the program successfully conducted a non-intercept flight test of its upgraded interceptor, the program is nearing a seven year delay in completing its first successful intercept. Until this upgraded interceptor is demonstrated in an intercept test, expected to be conducted in the third quarter of fiscal year 2014, manufacturing and deliveries remain on hold. In July 2013, the GMD program also failed a flight test of its fielded interceptor. This flight test was designed to assess the fielded interceptor under more challenging conditions and to confirm design changes to resolve prior issues. MDA has not yet made a decision on how to proceed since the cause of failure has not been determined.
MDA has improved the clarity of its resource and schedule baselines since it first submitted them to Congress in 2010. However, issues with the content and presentation of these baselines continue to limit the usefulness of the information available to decision makers for oversight. First, as the agency is still in the process of improving the quality and comprehensiveness of the cost estimates that support its resource baselines, for the fourth year, GAO has found that MDA's cost estimates are unreliable. For example, MDA's 2013 cost estimates still do not include operations and support costs for military services which may significantly understate total costs. Congress has recently required MDA to include these costs in future acquisition baselines which may improve transparency. Second, MDA's schedule baselines are presented in a way that makes it difficult to assess progress. Specifically, MDA's 2013 schedule baselines include numerous events but provide very little information about them, making it difficult to understand what the events are and why they are important. Additionally, the 2013 schedule baselines do not compare the current event dates with previously reported dates, so decision makers cannot easily assess how the program is performing over time. Until MDA improves the quality and comprehensiveness of its cost estimates and the content of its schedule information, its baselines will not be useful for decision makers to gauge progress.
section_title: What GAO Recommends
paragraphs: GAO recommends (1) any changes to the SM-3 Block IB be flight tested before DOD approves full production; (2) retest the fielded GMD interceptor to demonstrate performance; and (3) improve the content of its schedule baselines. DOD partially concurred with the first, non-concurred with the second, and concurred with the third, stating that the production and testing decisions will be made using the proper DOD processes. GAO believes both recommendations are valid as discussed in this report.

section_title: Letter
paragraphs: For over half a century, the Department of Defense (DOD) has been  funding efforts to develop a system to detect, track, and defeat enemy  ballistic missiles. The current system—the Ballistic Missile Defense  System (BMDS)—includes a diverse collection of land-, sea-, and space- based assets located around the globe. Since 2002, the Missile Defense  Agency (MDA)—the agency charged with developing an integrated  BMDS—has spent over $98 billion to develop and deploy this highly  complex group of systems and has requested an additional $38 billion for  fiscal years 2014 through 2018 to continue its efforts. For over a decade,  we have reported on MDA’s progress and challenges in developing and  fielding BMDS capabilities as well as transparency, accountability, and  oversight issues.
Since 2002, we have been mandated to prepare annual assessments of  MDA’s progress toward its acquisition goals. To date, we have delivered  reviews of MDA’s progress covering fiscal years 2003 through 2012 and  are currently mandated to issue our final report in 2016. The National  Defense Authorization Act for Fiscal Year 2012 mandated that we report  our assessment of the extent to which MDA has achieved its stated  acquisition goals and objectives, as reported through its acquisition  baselines, and also to include any other findings and recommendations   This  on MDA acquisition programs and accountability as appropriate.report provides our assessment of MDA’s acquisition progress in fiscal  year 2013. Specifically, it highlights MDA’s progress and any challenges  associated with (1) developing, flight testing, and producing individual  systems, which MDA refers to as BMDS elements; and (2) reporting  resource and schedule baselines that support oversight.
To assess MDA’s progress and any challenges with developing, testing,  and producing the BMDS, we examined the acquisition accomplishments  of several missile defense elements and MDA’s targets program. We reviewed key management documents including Program Execution  Reviews and Baseline Execution Reviews, which detail program  accomplishments and areas of concern, and interviewed program  element officials. We also examined MDA’s master test plan and flight  test reports, and discussed the BMDS testing and results with the BMDS  Operational Test Agency; DOD’s Office of the Director, Operational Test  and Evaluation; and DOD’s Office of Developmental Test and Evaluation.  We also visited several contractor facilities that were working on  programs covered in our review.
To assess the progress as well as any remaining challenges MDA faces  in reporting resource and schedule baselines that support oversight, we  examined the baselines presented in the BMDS Accountability Reports  (BAR) for fiscal years 2010 through 2013. To be consistent with last year,  we focused our assessment on the resource and schedule baselines as  they continue to be the only reported baselines that separately explain  when current estimates have deviated to a certain extent from the  baselines set in prior BARs and have measurable goals, such as cost  estimates and dates of program events. We interviewed officials in MDA’s  Operations Directorate to discuss their progress in adopting best  practices in cost estimating based on our Cost Guide. We also met with  officials from MDA’s Acquisition Directorate to discuss significant internal  and external events and decisions that occurred in fiscal year 2013, such  as sequestration, that affected the agency’s overall acquisition of the  BMDS, as well as how the agency establishes and manages its   We examined MDA’s statutory reporting  acquisition baselines.requirements and interviewed officials within MDA’s general counsel’s  office. In addition, we reviewed DOD policy and guidance to discern how  other major defense acquisition programs are required to report baselines  and measure program progress.
In addition to this assessment, the National Defense Authorization Act for  Fiscal Year 2013 mandated GAO to provide separate assessments on  several other missile defense related issues. Specifically, GAO was  required to provide briefings on our views and to submit reports as soon  as practicable to the congressional defense committees our assessments  of DOD reports on (1) a comprehensive evaluation of alternatives for the  Precision Tracking Space System and its conformance with GAO best  practices for analyses of alternatives; (2) the Ground-based Midcourse  Defense system’s test plan; (3) the status and progress of regional  missile defense programs, including the adequacy of MDA’s existing and  planned efforts to deploy a U.S. missile defense in Europe; and (4) the  status of efforts to improve the homeland defense capability of the United  States. Because this additional mandated work covers the details on  many BMDS elements, we do not include appendixes on each of the  individual elements as we have done in prior reports under this mandate.  For more details on our scope and methodology, see appendix I.
We conducted this performance audit from April 2013 to April 2014 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Ballistic missiles have different ranges—short, medium, intermediate, and  intercontinental—as well as different speeds, sizes, and performance  characteristics. Short-range ballistic missiles have a range of less than  621 miles; medium-range ballistic missiles have a range from 621 to  1,864 miles; intermediate-range ballistic missiles have a range from 1,864  to 3,418 miles; and intercontinental ballistic missiles have a range greater  than 3,418 miles. As a result, MDA is developing a variety of systems  that, when integrated, provide multiple opportunities to destroy ballistic  missiles in flight for the strategic defense of the United States and  regional defense of its deployed forces and allies. The BMDS includes  space-based sensors; ground- and sea-based radars; ground- and sea- based interceptor missiles; and a command and control system that  provides communication links to the sensors and interceptor missiles.
Once a ballistic missile has been launched, these sensors and  interceptors track and engage the threat missile during its flight as shown  in figure 1.
subsections: section_title: MDA’s Flexible Acquisition Approach
paragraphs: When MDA was established in 2002, the Secretary of Defense granted it  exceptional flexibility to set requirements and manage the acquisition of  the BMDS in order to meet a presidential directive to deliver an initial  defensive capability against ballistic missiles in 2004. This flexibility  allows MDA to develop BMDS elements outside of DOD’s standard  acquisition process until they are mature enough to be handed over to a  military service for production and deployment. Because the BMDS’s  entrance into DOD’s acquisition process is deferred, certain laws and  policies that generally require major defense acquisition programs to take  certain steps at certain phases in the DOD acquisition process do not yet  apply to MDA. For example, before a major defense acquisition program  begins the product development phase, it must document key  performance, cost, and schedule goals in an acquisition baseline that has  been approved by a higher-level DOD official. This acquisition baseline  is used to measure a program’s performance as it progresses.  Specifically, as implemented by DOD, major defense acquisition  programs’ baselines provide decision makers with key goals such as the  program’s total cost for an increment of work, key dates associated with  acquiring a capability, and the weapon’s intended performance.  Additionally, once a baseline has been approved, DOD’s major defense  acquisition programs are required to measure performance against their  baseline and report certain changes to Congress. For instance, they are  required to report certain increases in unit cost (cost divided by the  quantity produced) measured from the original and the current program  baseline.
While this flexibility allows MDA latitude to manage the BMDS and enable  it to rapidly develop and field new systems, we have previously reported  that the agency has used this flexibility to employ acquisition strategies  with high levels of concurrency (that is, overlapping activities such as  testing and production), which increases the risk for performance  shortfalls, costly retrofits, and test problems. this flexibility has hampered oversight and accountability.MDA officials, MDA has taken some steps to identify and track  concurrency in their programs.
We have also reported that  Congress has taken steps to improve the transparency and accountability  of BMDS development efforts. For example, in the National Defense  Authorization Act for Fiscal Year 2008, Congress required MDA to  establish cost, schedule, and performance baselines for certain BMDS  elements. MDA first reported baselines for several BMDS elements to  Congress in its June 2010 BAR and has continued to report baselines  annually. Table 1 describes the six acquisition baselines MDA established  and reports in its BAR for individual BMDS elements or major portions of  such elements.
GAO-12-486.   10 U.S.C. § 2432 defines, with respect to a major defense acquisition program, procurement unit  cost as the amount equal to (1) the total of all funds programmed to be available for obligation for  procurement for the program divided by (2) the number of fully configured end items to be procured.  Program acquisition unit cost is defined as the amount equal to (1) the total cost for development and  procurement of, and system-specific military construction for, the acquisition program divided by (2)  the number of fully configured end items to be produced for the acquisition program.
Most recently, the National Defense Authorization Act for Fiscal Year  2012 amended MDA’s baseline reporting requirements.law currently requires MDA to report to the congressional defense  committees certain changes or variances in the current baselines from  the baselines presented in the prior year’s report and from when the  baselines were initially established. Additionally, the act allows MDA to  revise an initial baseline, which the agency refers to as a “revised initial  baseline.”
Specifically, the  In 2010, MDA also established an acquisition process that continues to  guide the development of the BMDS. Table 2 identifies the five life cycle  phases of MDA’s acquisition process.
The agency has documented the key knowledge that is needed prior to  the technology development, product development, initial production, and  production phases. For example, prior to entering initial production, an  element must demonstrate that its design and manufacturing processes  are stable, planned quantities are affordable, and developmental and  operational test results show that the user’s needs will be met. In general,  developmental testing is aimed at determining whether the system design  will satisfy the desired capabilities, while operational testing is aimed at  determining whether the system is effective, survivable, and suitable in  the hands of the user under realistic conditions. Additionally, according to  DOD policy, programs entering initial production or production require  approval from DOD’s Under Secretary of Defense for Acquisition,  Technology, and Logistics.
section_title: Description of BMDS Elements
paragraphs: Table 3 describes the BMDS elements and programs assessed in this  report and their current MDA acquisition phase.
section_title: Missile Defense Acquisition Changes in Fiscal Year 2013
paragraphs: In March 2013, in response to a growing threat from Iran and North  Korea, the Secretary of Defense announced steps that affected the  acquisition of the BMDS including deploying 14 additional ground-based interceptors in Fort Greely, deploying a second AN/TPY-2 radar to Japan; and  shifting the resources from development of the Aegis BMD SM-3  Block IIB interceptor, which was planned to be deployed after 2020 to  defend against intercontinental ballistic missiles, to fund the additional  ground-based interceptors, as well as develop advanced technology  to improve the performance of current and future versions of BMDS  interceptors.
In 2013, DOD canceled the Aegis BMD SM-3 Block IIB and the Precision  Tracking Space System citing concerns with the programs’ high-risk  acquisition strategies and technical challenges that GAO has previously  raised. We have previously reported that MDA did not consider a broad  range of alternatives or fully assess program or technical risks before  committing to either program.
In addition, MDA altered its fiscal year 2013 acquisition plan to offset a  funding reduction of $568 million (6.8 percent) in its total available budget  for fiscal year 2013. For example, in consultation with key stakeholders  such as the Operational Test Agency and DOD’s Office of the Director,  Operational Test and Evaluation, MDA revised its test plan by combining,  delaying, and deleting tests to cut costs. MDA also delayed some  development activities for various elements into fiscal year 2014 and  beyond.
section_title: BMDS Flight Tests Successfully Executed, but Development Challenges Continue
paragraphs: In fiscal year 2013, MDA successfully executed several flight tests that  demonstrated key BMDS capabilities and modifications made to resolve  prior development issues, but continued to experience failures and delays  resulting in less testing and production than planned. For the first time in  September 2013, the Aegis BMD and Terminal High Altitude Area  Defense (THAAD) programs participated in an operational flight test that  resulted in a near simultaneous engagement. Additionally, the Aegis BMD  program also successfully conducted flight tests with the SM-3 Block IB  missile. However, according to officials from the DOD’s Office of the  Director, Operational Testing and Evaluation, Aegis BMD experienced a  SM-3 Block IB missile failure that is currently being investigated and could  result in a modification to a component that is common between the Aegis  BMD SM-3 Block IA and IB. Lastly, GMD also successfully conducted a  non-intercept test of its upgraded interceptor that is currently in  development, but experienced an intercept test failure of the fielded  interceptor, the cause of which is still unknown. Further details on the  operational test are provided after table 4. In addition, because ongoing  testing and development challenges increase the potential to affect the  production of the SM-3 Block IB missile as well as delaying understanding  of the capabilities and limitations of GMD’s fielded interceptor, we also  provide additional details for these programs after the table.
Table 4 presents a summary of key accomplishments and challenges for  BMDS elements and programs that are in the BAR.
subsections: section_title: First System-level Operational Test of the BMDS
paragraphs: After more than 11 years of development of the BMDS, MDA conducted  the first system-level operational missile defense flight test called Flight  Test Operational-01 (FTO-01) in September 2013. During the test,  warfighters from several combatant commands employed multiple missile  defense systems including Aegis BMD and THAAD to demonstrate the  regional capabilities of U.S. missile defense. This is a significant  achievement because it is the first operational test that involved multiple  elements working simultaneously. To conduct this test, MDA invested in  range assets and conducted other activities to ensure it could test  multiple elements at once. For example, MDA conducted its first  integrated system-level flight test, known as Flight Test Integrated-01, in  October 2012 as a risk-reduction exercise for the operational test.  During FTO-01, MDA launched two nearly simultaneous threat- representative medium-range ballistic missile targets including its air- launched extended-medium range ballistic missile (eMRBM) target for the  first time.because of development problems associated with the eMRBM target.  MDA also had to make some adjustments to the FTO-01 test plan  because of fiscal year 2013 sequestration. Although MDA preserved its  primary objective to demonstrate the interoperability of BMDS elements, it  reduced the number of targets included in the test from five to two and  removed participation of more mature elements such as the Patriot  Advance Capability-3.
This test was delayed for approximately one year in part  The BMDS elements successfully engaged the targets during the test, but  according to independent testing officials, full system integration was not  achieved. Specifically, according to the DOD’s Director, Operational Test  and Evaluation, the Aegis ship successfully intercepted one of the targets  with a SM-3 Block IA and THAAD successfully intercepted a medium- range target for the second time. In addition, as a planned demonstration  of its layered defense, THAAD launched a second interceptor at the  target intercepted by the Aegis ship as a contingency in event the SM-3  Block IA did not achieve an intercept. However, DOD’s Director,  Operational Test and Evaluation, also found that the test failed to achieve  full integration between all systems due to challenges with system  networks, limitations in elements’ ability to work together and component  failures. For example, the test uncovered several issues with  communication networks that are needed for interoperability between all  elements. Interoperability is important because it can improve the missile  defense effectiveness and enhance individual systems performance  beyond operating alone.
section_title: Aegis BMD SM-3 Block IB
paragraphs: The Aegis BMD SM-3 Block IB program largely overcame previous  development challenges and successfully intercepted all targets in its last  three flight tests as shown in table 5. These tests are required for a full  production decision—the last key production authorization by the Under  Secretary of Defense, Acquisition, Technology, and Logistics that would  allow MDA to produce the remaining 415 interceptors. However, a missile  failure of the second interceptor launched during the September 2013 test  could increase production risk if design changes are needed.
As we found in April 2013, the SM-3 Block IB production line has been  repeatedly disrupted since 2011 due to flight test anomalies caused by  malfunctions in two separate sections of the third-stage rocket motor, and  development challenges with the throttleable divert and attitude control  system—components that maneuver the interceptor in its later stages of  flight. These challenges delayed the SM-3 Block IB full production  authorization by more than two years to fiscal year 2015. Largely  resolving these previous challenges, in fiscal year 2013 the program  received permission to procure 33 additional initial production missiles.  Although MDA initially planned to award a contract for 29 SM-3 Block IB  missiles in fiscal year 2013, it bought four additional missiles in August  2013 to recover an earlier reduction. That reduction occurred to provide  funds to resolve technical and production issues. Based on successful  intercepts of the last three flight tests, the program also received  permission to buy 52 more interceptors in fiscal year 2014.
Despite the three successful intercepts, the effect of the missile failure in  September 2013 on the upcoming full production decision remains  unclear. Before the program enters into full production, MDA’s acquisition  management instruction requires it to demonstrate to the Under Secretary  of Defense, Acquisition, Technology, and Logistics that there are no  significant risks to production and that the planned production quantities  are affordable and fully funded. The permission to enter full production is  also based on independent assessments of the weapon’s effectiveness  and suitability by the DOD’s Director, Operational Test and Evaluation  and the Navy’s Commander Operational Test & Evaluation Force.  Although the failure investigation is ongoing, preliminary results indicate  that the failure occurred in the third-stage rocket motor, a component  common to the SM-3 Block IA, which is nearing the end of its production.  Different issues with that same component have contributed to previous  SM-3 Block IB schedule delays and production disruptions. While the  precise cause of the September 2013 failure is under review, MDA  documentation indicates that it could potentially result in design changes  to the third-stage rocket motor and changes to manufacturing processes.  Additionally, retrofits may be required for SM-3 Block IB and SM-3 Block  IA interceptors that were already produced. If design changes are  necessary, program documentation indicates that they will not be flight  tested until the fourth quarter of fiscal year 2015, just prior to the planned  deployment of the SM-3 Block IB to support the regional defense of  Europe and 6 months after its planned full production decision.  Consequently, until the program thoroughly understands the extent of  needed modifications, if any, and their effects on performance as  demonstrated though testing, its production strategy is at risk of cost  growth and schedule delays. MDA has experienced these consequences  in other elements when it pursued design changes concurrently with  production.
Although the GMD program made progress in resolving a prior CE-II  intercept failure, test failures and development challenges continue to  disrupt the program and increase the cost to demonstrate the new CE-II.  The GMD program first attempted to demonstrate the CE-II interceptor in  January 2010 but subsequently experienced a number of setbacks in  both the CE-II and the fielded CE-I, as seen in table 6 below.
Developing a mitigation to the FTG-06a failure has proven more difficult  than initially expected. The program initially planned to conduct FTG-06b  in the third quarter of fiscal year 2012 but the test has since been delayed  to at least the third quarter of fiscal year 2014 because of challenges  resolving test failures. For example, while initial results from CTV-01  indicated the redesigned guidance system component could be used to  resolve the problem that caused the FTG-06a failure, subsequent ground  testing revealed that only one-third of those produced could be used in  future interceptor production or flight tests because the component’s  performance was uncertain. The program mitigated the issue by  implementing software and hardware modifications and delivered the  redesigned component for kill vehicle integration in October 2013.  However, according to MDA, the program experienced further delays in  the FTG-06b test while it implemented changes based on assessments  from the ongoing FTG-07 failure review. Consequently, confirmation that  the CE-II design works as intended has been delayed by nearly seven  years and costs have increased by over $1 billion because of the CE-II  development challenges and test failures.
In July 2013, MDA conducted the FTG-07 developmental test to  understand the performance of the fielded CE-I against a longer range  target in more challenging conditions and assess the performance of  upgrades. This interceptor was fielded before completing developmental  testing, leading MDA to undertake retrofit efforts and upgrades to fix  issues identified during testing. According to acquisition best practices,  developmental testing should be complete before beginning production  and fielding in order to, among other reasons, avoid the need for retrofits  and upgrades to fix issues discovered during testing. The test failed,  delaying understanding of the capabilities and limitations of upgrades to  the fielded CE-I. Shortly after the test failure, the Director, MDA stated a  failure review was initiated to not only identify the root cause of the failure,  but also provide a comprehensive review of potential CE-I failures and  identify any correlations with the CE-II. Since then, according to program  officials, MDA has identified a kill vehicle component common to both  interceptors that could be associated with the FTG-07 failure. However, it  remains unclear what, if any, design changes, retrofits, or other corrective  actions to the CE-I or CE-II are necessary since the failure review is not  complete. According to MDA officials, they have not determined if they  will re-conduct the flight test. If the CE-I is not flight tested again, the  warfighter will not have a full understanding of the capabilities and  limitations of the upgrades to the CE-I interceptor, the original purpose of  the FTG-07 test.
Overall, GMD’s ongoing testing issues in conjunction with concurrent  acquisition practices have caused—and will likely continue to cause— major disruptions to the program. We previously found that, in 2004, MDA  committed to a highly concurrent development, production, and fielding  strategy for the CE-II interceptor and began delivering interceptors in  2008. Because MDA moved forward with CE-I and CE-II interceptor  production before completing its flight testing program, test failures have  exacerbated disruptions to the program. For example, because the  program has delivered approximately three-fourths of the interceptors for  fielding, the program faces difficult and costly decisions on how it will  implement corrections from prior test failures. Also, the program has had  to add tests that were previously not planned and delay tests that are  necessary to understand the system’s capabilities and limitations. As a  result of these development challenges, the GMD program will likely  continue to experience delays, disruptions, and cost growth.
section_title: MDA Made Its Resource and Schedule Baselines Clearer, but Further Actions Needed to Enhance Their Use for Oversight
paragraphs: MDA has taken some steps to improve the clarity of its resource and  schedule baselines, but issues with the content and presentation of these  baselines continue to limit the usefulness of the information available to  decision makers for oversight of BMDS development efforts. Since 2011,  we have found deficiencies in the quality of the cost estimates that  underpin MDA’s resource baselines and reported on the efforts MDA has  undertaken to improve those estimates. In 2013, we found the agency  made little progress addressing the underlying quality issues with those  cost estimates that we raised. As a result, this is the fourth year we have  found that the cost estimates that support MDA’s resource baselines are  not sufficiently reliable to support oversight. However, according to MDA  officials, the agency is taking steps to improve the quality of its cost  estimates to support the resource baselines it plans to report in its 2014  BAR. Assessing MDA’s progress in achieving its schedule goals is also  difficult because MDA’s schedule baselines are not presented in a way  that allows decision makers to understand or easily monitor progress.  Until MDA improves the quality and comprehensiveness of its cost  estimates and the clarity of its schedule information, its baselines may not  be useful for decision makers.
subsections: section_title: MDA Made Limited Improvements to Its Resource and Schedule Baselines
paragraphs: In its 2013 BAR, MDA continued to make useful changes to its reported  resource and schedule baselines. We found in March 2011 that MDA’s  schedule and resource baselines had several shortcomings that limited  their usefulness for oversight, such as not explaining variances or  significant changes in the baselines. Additionally, we found in April 2013  that, in its 2012 BAR, MDA only reported annual progress by comparing  its current estimates for unit cost and scheduled activities against the  prior year’s estimate and adjusted the content of the baselines from year  to year in such a way that they were no longer comparable. As a result,  MDA’s baselines were not useful for tracking longer term progress or for  holding the agency accountable.
MDA took some action to improve the completeness and clarity of the  BAR baselines by:  identifying the date of the initial baseline and, if applicable, the date  when the initial baseline was most recently revised for each element  or major portion of an element reported in the BAR; explaining most of the significant cost and schedule changes from the  current baseline estimates against both the estimates reported in the  prior year’s BAR and the latest initial baseline; and making the baselines easier to read by removing cluttered formatting  such as strikethroughs and highlights that made some of the events  listed in past BARs unreadable.
section_title: MDA’s 2013 Resource Baselines Remain Unreliable
paragraphs: GAO-11-372  In March 2011, we assessed MDA’s life-cycle  cost estimates using the GAO Cost Estimating  and Assessment Guide, which is based on  best practices in cost estimating and identifies  key criteria for establishing high-quality cost  estimates. Our review found that the  estimates we assessed were not  comprehensive, lacked documentation, were  not completely accurate, or were not  sufficiently credible. We recommended that  MDA (1) take steps to ensure its cost  estimates are high quality, reliable cost  estimates that are documented to facilitate  external review and (2) obtain independent  cost estimates for each baseline. resource baselines reported in the 2013 BAR. We have found that  completing these steps could further improve the quality of MDA’s cost  estimates.
GAO-13-432  In April 2013, we recommended that the  Secretary of Defense direct the MDA Director  to include in its resource baseline cost  estimates all life cycle costs, specifically the  operations and support costs, from the military  services in order to provide decision makers  with the full costs of ballistic missile defense  systems.
Additionally, MDA has made little progress improving the  comprehensiveness of the cost estimates that support its resource  baselines. Similar to past years, the cost estimates reported in the 2013  BAR also do not include the operation and support costs funded by the  individual military services, which we concluded in April 2013 may result  in significantly understated life cycle costs for some BMDS elements. In  response to our April 2013 recommendation, DOD agreed that decision  makers should have insight into the full life cycle costs of DOD programs,  but the department stated that the BAR should only include content for  which MDA is responsible. Because MDA already reports the estimated  acquisition costs and some of the operation and support costs for the  acquisitions in the annual BAR, we concluded that annual document to be  the most appropriate way to report the full costs to Congress. Additionally,  we concluded that good budgeting requires that the full costs of a project  be considered when making decisions to provide resources and,  therefore, both DOD and Congress would benefit from a comprehensive  understanding of the full costs of MDA’s acquisition programs. Until  MDA’s resource baselines are based on reliable information and are  comprehensive, they will not be useful for decision makers to understand  progress or make well-informed investment decisions.
In the National Defense Authorization Act for Fiscal Year 2014, Congress  took steps to address concerns over MDA’s cost estimates by requiring  MDA to report to the congressional defense committees on its efforts to  improve the quality of the cost estimates included in its acquisition    For example, the act requires MDA to report on a description  baselines.of and schedule for planned actions to improve its cost estimates, as well  as an assessment of how the planned improvements align with GAO’s  cost estimating best practices. We are also required to provide our views  on the content of MDA’s report. Additionally, the act requires that the life  cycle cost estimate included in the agency’s acquisition baselines include  a description of the operations and support functions and costs for which  the military services are responsible, in addition to the costs borne by  MDA.
section_title: MDA’s Schedule Baseline Reporting Makes Assessing Progress Difficult
paragraphs: MDA’s schedule baselines are presented in a way that makes it difficult  for decision makers to understand a program’s planned activities and  therefore hold programs accountable for their performance. According to  GAO’s Schedule Assessment Guide, a reliable program schedule  includes all activities required to complete a project, but the schedule  should not be too detailed to interfere with its use.presenting decision makers with a high-level summary of the schedule is  a best practice because schedules that include too many milestones or  have too much detail make it difficult to manage progress. Additionally,  the activities included on the schedule should have descriptive names  that clearly communicate the work required. MDA’s 2013 BAR schedule  baselines include numerous events but provide very little information  about them, making it difficult to understand what the events are and why  they are important. For example, the milestones identifying significant  increases in performance for C2BMC Spiral 8.2 are numbered with no  description of the capabilities they represent. Additionally, several of the  events reported for Aegis modernized weapon system software are titled  with abbreviations that are not explained in the BAR.
In addition, MDA does not present any comparisons of event dates with  previously reported dates. In contrast, DOD’s major defense acquisition  programs report a comparison of current schedule estimates against their  original and current schedule goals. According to GAO’s Schedule  Assessment Guide, comparing the current schedule to the baseline  schedule to track deviations from the plan provides decision makers  valuable insight into program risk and can help identify where corrective  action may be needed. While removing the formatting that identified  changes in prior BARs made the schedule baselines easier to read, doing  so removed the ability for decision makers to see if the planned dates for  events had changed. As a result, decision makers must consult past  versions of the BAR to identify any changes in the planned schedule for a  specific event, which can be difficult or impossible in some cases. For  example, we found in April 2013 that we were unable to compare the  current estimated dates for the activities presented in the Aegis Ashore  schedule baseline to the dates baselined in the 2010 BAR because  activities were split into multiple events, renamed, eliminated, or moved to  several other Aegis BMD schedule baselines.audit we raised this issue with MDA and, according to agency officials,  MDA is open to considering alternative formats for presenting the  schedule baseline in future version of the BARs. Until MDA improves the  content of its schedule baselines, decision makers will not be able to  assess how a program is performing over time.
section_title: Conclusions
paragraphs: During fiscal year 2013, MDA was able to make some significant  acquisition progress, including the first operational system-level flight test  involving multiple BMDS elements, but it continued to experience  difficulties achieving its goals for testing. This has resulted in delaying  progress on individual elements, delaying understanding of the overall  performance of the BMDS, and fielding assets before all testing is  complete. The most significant acquisition effects have been experienced  on the Aegis BMD SM-3 Block IB and GMD program where testing and  development challenges have led to failure investigations and increased  the risk of continued cost growth and schedule delays. Both programs  conducted flight tests and made progress in resolving design flaws in  2013, but still have further development, testing, and production issues to  address. For Aegis BMD SM-3 Block IB, the failure of its interceptor in a  September 2013 flight test means that a key component may need to be  redesigned and that change confirmed to work in additional flight testing.  For GMD, the failure of the deployed CE-I interceptor in a July 2013 flight  test compounds its challenge because the program did not gain the  expected understanding of the effectiveness of software upgrades  planned for the operational fleet and now must determine the cause of the  failure. As a result, for both programs, to the extent that software or  hardware changes are necessary to resolve the cause of these failures,  new flight tests will likely be needed to demonstrate both the  effectiveness of any resolutions and, for GMD, understand the  performance of the software upgrades that were the original purpose of  the test.
Additionally, for over a decade, we have reported that MDA provides  Congress with only limited insight into the acquisition progress for  individual programs. While MDA has taken steps to improve the clarity of  the baselines it reports to Congress, the agency’s cost and schedule  reporting still lacks the quality, completeness, and clarity necessary to  track actual cost or schedule growth over time. Specifically, the agency  has not addressed all of the critical gaps in the quality of its underlying  cost estimates used to develop its resource baselines that we have  identified over the years. Until corrective actions are implemented and  substantial improvements are made to MDA’s cost estimates, its reported  resource baselines will not be useful for decision makers to hold MDA  accountable for its performance or make informed decisions on how best  to allocate limited resources. Congress recently amended the  requirements for the cost estimates MDA must report in its baselines,  which may enhance the transparency into MDA’s cost estimating  processes. As a result, we do not make any new recommendations  regarding cost at this time. However, additional actions can be taken in  the schedule baselines to improve the ability of decision makers to  understand what program events are most critical and identify whether  the dates for those critical events have changed. Until improvements are  made, the schedule baselines will not be a useful tool for providing  oversight of the BMDS.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Defense take the following three  actions to strengthen MDA’s acquisitions and help support oversight.  1.  To the extent that MDA determines hardware or software  modifications are required to address the September 2013 Aegis BMD  SM-3 Block IB failure, we recommend that the Secretary of Defense  direct,  a)  the Director of the MDA to verify the changes work as intended  through subsequent flight testing, and  b)  the Under Secretary of Defense, Acquisitions, Technology, and  Logistics to delay the decision to approve the program’s full  production until such testing demonstrates that the redesigned  missile is effective and suitable.  2.  To demonstrate the CE-I’s effectiveness against a longer range target  in more challenging conditions and to confirm the design changes  implemented to improve performance, as well as any changes needed  to resolve the July 2013 CE-I flight test failure work as intended, we  recommend that the Secretary of Defense direct MDA’s Director to  conduct a flight test of the CE-I interceptor once the cause of the  failure has been determined and any mitigations have been  developed.  3.  To improve the content of the schedule baselines it reports to  Congress for monitoring program performance, we recommend that  the Secretary of Defense direct MDA’s Director to take the following  actions as MDA implements other improvements required by the  Congress:  a)  Focus the information included in the schedule baselines to  highlight critical events.  b)  For each event included in the schedule baseline, provide a  description of the event explaining what it entails and why it is  important.  c)  Present the schedule baseline in a format that allows decision  makers to identify any changes made from the current estimated  date to the date reported in not only the prior year’s BAR but also  to the date established in the initial baseline.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD provided written comments on a draft of this report. These  comments are reprinted in appendix II. DOD also provided technical  comments, which were incorporated as appropriate.
DOD partially concurred with our first recommendation, non-concurred  with our second recommendation and concurred with our third  recommendation. The department partially concurred with our first  recommendation to flight test any modifications that may be required to  the Aegis BMD SM-3 Block IB as a result of September 2013 failure,  before the Under Secretary of Defense, Acquisitions, Technology, and  Logistics approves full production. In its comments, DOD acknowledged  that if modifications are required they will be tested, but added that the  type of testing—flight or ground testing—will depend on the magnitude of  such modifications. The department also believes that the component  currently tied to the failure, has a successful testing history and thus  expects to meet the reliability requirement needed for the full production  decision in fiscal year 2015. However, there have now been three flight  test anomalies associated with this component over the last three years.  According to Aegis BMD officials, they are considering design changes  for this component. Since the fiscal year 2015 full production decision is  the commitment by the Under Secretary of Defense, Acquisitions,  Technology, and Logistics to produce several hundred missiles, this  decision should be supported by an assessment of the final product under  operational mission conditions to ensure that it is effective and suitable.  As such, we maintain our recommendation that before the program is  approved for full production, flight testing should demonstrate that any  modifications work as intended.
DOD did not concur with our second recommendation to complete the  original purpose of the July 2013 CE-I flight test once the cause of that  failure has been determined and any mitigations have been developed. In  its response, DOD stated that the decision to flight test a CE-I interceptor  will be made by the Director, MDA, based on the judgment of  stakeholders from the Office of the Secretary of Defense and combatant  commands on the need to perform a test. The DOD response focused  almost exclusively on the steps it is taking to identify the cause of the July  2013 failure and mitigate it and did not address the main part of our  recommendation—determining the effectiveness of the CE-I under more  challenging conditions and confirming that design changes previously  made improve performance. These were the objectives of FTG-07. In our  view, resolving these performance questions remains important. Since  the FTG-07 failure review is still ongoing, we cannot assess whether DOD  should conduct a CE-I test for the sole purpose of demonstrating  corrective actions, to the extent any are needed, to address the cause of  the failure. While we acknowledge that DOD must balance several  competing GMD priorities, including which flight tests to conduct, and  conducting another CE-I flight test may not be feasible in the immediate  future, we also maintain that demonstrating CE-I intercept capabilities  should continue to be a priority for DOD since the CE-I interceptor  constitutes a multi-billion dollar investment by DOD and serves as the  primary defense of the United States homeland against enemy ballistic  missile attacks.
In addition to responding to our recommendations, the department’s letter  raised additional concerns about our draft report. First, DOD disagreed  with our statement that because the BMDS entrance into DOD’s  acquisition process is deferred, it is exempt from certain acquisition laws  and policies that generally provide oversight of major defense acquisition  programs. DOD stated that MDA is not exempt from acquisition-related  laws because, while it is not captured by several statutes, Congress has  provided legislation specific to MDA to ensure oversight and  accountability. We clarified the language in our report to remove the term  exempt. However, because of the acquisition flexibility it has been  granted, MDA is not yet required to apply certain laws and policies to the  BMDS. We have found that while the flexibility allows MDA latitude to  manage the BMDS and to rapidly develop and field new systems, we  have also found that this flexibility has hampered oversight and  accountability. Our report recognizes the actions Congress has taken to  improve the transparency and accountability of the BMDS development  efforts through legislation specific to MDA, particularly to require MDA to  report baselines to Congress. However, there are a number of  requirements that are triggered by phases of the DOD acquisition process  that are important to sound acquisition management. For example, we  have previously found that MDA is not yet required to conduct an analysis  of alternatives to compare potential solutions and determine the most cost  effective weapon system to acquire nor is MDA yet required to obtain an  independent cost estimate prior to beginning product development.
Second, DOD stated that it disagreed with our assessment that MDA’s  cost estimates are not sufficiently reliable to support oversight, suggesting  that the report be revised to include more of MDA’s efforts to improve the  quality of its cost estimates. Since 2011, we have found that there are  issues with the cost estimates and baseline reporting, including  incomplete cost estimates due to the exclusion of military service  operation and support costs as well as instability in the content of the  baselines, which makes assessing progress difficult or impossible. While  the draft report was being reviewed by DOD, we met with MDA officials  who discussed more of their efforts to improve their cost estimates;  however, we were not provided sufficient information to change our  determination. We did clarify the report to better reflect the efforts they  have undertaken. DOD stated in its response that MDA has included  previously unreported costs in the baselines it provides to Congress,  which we have previously found is an improvement to the amount of  information reported to Congress, but which does not demonstrate that  the quality of the cost estimates themselves have improved. DOD stated  in its response that it has provided joint operation and support costs  documentation for two programs reported in the 2013 BAR. However, we  did not assess the joint operation and support costs because they were  not included in the 2013 resource baselines reported to Congress. DOD  also stated in its response that it received an assessment from us on cost  estimate documentation for a third program. We did not perform a formal  assessment of the third program because it was cancelled and not  included in the 2013 BAR. In order to assist MDA in improving its cost  estimates, we did informally assess the third program’s cost estimate, but  reached no conclusion as to its quality. However, we noted several issues  in that informal review. For example, we concluded that because MDA did  not provide a cost model to support the estimate we were unable to check  the cost estimate for accuracy. Finally, DOD stated in its response that  MDA has published and implemented a cost estimating handbook. We  have previously found that fully implementing that handbook could  improve the quality of MDA’s cost estimates.  However, during this  review, we specifically asked to review the cost estimate documentation  supporting MDA’s fiscal year 2013 BAR baselines in order to assess its  progress in implementing that handbook. An MDA senior cost official told  us that the agency was working to fill in documentation gaps on existing  cost estimates and that the estimates were not ready for us to review. In  the course of our work, we concluded and informed MDA that the cost  estimating process defined in that handbook has not been applied to any  systems that are currently baselined or part of the BMDS. Until MDA is  able to provide us with documentation that supports the actual baselines  reported to Congress so we can independently assess the quality of the  cost estimates, we have no basis to change our assessment.
Third, DOD disputed that MDA has not obtained independent cost  estimates from DOD’s Office of the Director of Cost Assessment and  Program Evaluation for any of the elements GAO reviewed since 2010. In  response, we clarified the language in the report so that it specifically  refers to the lack of independent cost estimates completed for any of the  resource baselines reported to Congress in the 2013 BAR. We also  clarified in the report that DOD’s Office of the Director of Cost  Assessment and Program Evaluation has assessed other BMDS costs.  DOD was unable to provide us with documentation of independent cost  estimates completed for MDA’s BAR baselines, therefore, we have no  basis to change our determination.
Lastly, DOD identified 35 “technical and factual errors” in its technical  comments. However, upon review we found that 29 were not technical or  factual errors, but rather different conclusions, errors in DOD’s comments,  or required additional substantiation that was not provided.  We  determined that 6 were actual technical or factual errors and therefore  made the appropriate changes in those circumstances.
We are sending copies of this report to the Secretary of Defense and to  the Director, MDA. In addition, the report is available at no charge on the  GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-4841 or chaplainc@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: To assess the Missile Defense Agency’s (MDA) progress and any  challenges associated with developing, testing, and producing the ballistic  missile defense system (BMDS) during fiscal year 2013, we examined the  acquisition accomplishments of several missile defense elements and  MDA’s targets program. Specifically, we reviewed the Aegis Ballistic  Missile Defense (Aegis BMD) with Standard Missile-3 (SM-3) Block IB;  Aegis Ashore; Aegis Modernized Weapon System Software; Army/Navy  Transportable Radar Surveillance and Control Model 2 (AN/TPY-2);  Command, Control, Battle Management, and Communications (C2BMC);  Ground-based Midcourse Defense (GMD) System; Targets and  Countermeasures; and Terminal High Altitude Area Defense (THAAD)  elements because, as reported in the 2013 BMDS Accountability Report  (BAR), these elements or programs have entered MDA’s product  development, initial production, or production acquisition phase, but are  not yet mature enough to be transferred to a military service and enter the   We  formal DOD acquisition cycle for full-rate production and deployment.reviewed key management documents for fiscal year 2013 including  Program and Baseline Execution Reviews, which detailed program  accomplishments and areas of concerns, and interviewed program  element officials. We also examined MDA’s master test plan and flight  test reports, and discussed the element- and BMDS-level test programs  and test results with the BMDS Operational Test Agency and the  Department of Defense’s (DOD) Office of the Director, Operational Test  and Evaluation and office of Developmental Test and Evaluation. In  addition, we also met with officials from MDA’s functional directorates  including the Engineering Directorate to discuss the agency’s process for  delivering and integrating BMDS capabilities as well as the Directorates  for Acquisition and Operations to discuss significant internal and external  events and decisions that occurred in fiscal year 2013, such as  sequestration, that affected the agency’s overall acquisition of the BMDS.
To assess the progress made as well as any remaining challenges MDA  faces in reporting resource and schedule baselines that support  oversight, we examined MDA’s reported baselines in the 2010, 2011,  2012, and 2013 BARs. To be consistent with last year, we focused our  assessment on the resource and schedule baselines as they continue to  be the only reported baselines that have measurable goals, such as cost  estimates and dates of program events, and separately explain when  current estimates have deviated to a certain extent from the baselines set  in prior BARs. We also examined the National Defense Authorization Act  for Fiscal Year 2012, which required MDA to establish and maintain  baselines for program elements or major portions of such program  elements and outlined the information to be included in MDA’s baselines,  as well as interviewed officials within MDA’s general counsel’s office. We  also interviewed officials in MDA’s Acquisitions Directorate about how the  agency establishes and manages its acquisition baselines and met with  MDA officials in the Operations Directorate to discuss their progress in  adopting best practices in cost estimating based on our Cost Guide. We  also reviewed findings and recommendations from several of our past  reports to see if MDA had made progress in improving the completeness,  clarity, and stability of its reported resource and schedule baselines. In  addition, we examined DOD acquisition policy such as the Interim DOD  Instruction 5000.02 issued in November 2013 and the Defense  Acquisition University’s Defense Acquisition Guidebook to discern how  other major defense acquisition programs are required to report baselines  and measure program progress. We also reviewed GAO’s cost and  schedule guides, which outline best practices for establishing and  managing program cost and schedule estimates.
To gauge the extent to which MDA reported changes or variances in the  current baselines from the baselines presented in the prior year’s BAR  and from when the baselines were initially established, we compared the  2013 BAR resource and schedule baselines for each BMDS element in  our review to the baselines presented in the 2012 and 2010 BARs. In  order to compare unit costs calculated in different years, there were  instances where it was necessary to convert prior cost estimates to match  the base year of the estimates presented in the 2013 BAR. We  performed these conversions using indexes published by the Office of the  Secretary of Defense (Comptroller) in the National Defense Budget  Estimates, commonly referred to as the “Green Book.”
The National Defense Authorization Act for Fiscal Year 2013 directed  GAO to provide separate assessments on several other missile defense  related issues. Specifically, GAO was required to provide briefings on our  views and to submit reports as soon as practicable to the congressional  defense committees our assessments of DOD reports on (1) a  comprehensive evaluation of alternatives for the Precision Tracking  Space System and its conformance with GAO best practices for analyses  of alternatives; (2) the Ground-based Midcourse Defense system’s test  plan; (3) the status and progress of regional missile defense programs,  including the adequacy of MDA’s existing and planned efforts to deploy a  U.S. missile defense in Europe; and (4) the status of efforts to improve  the homeland defense capability of the United States. additional mandated work covers the details on many BMDS elements,  we do not include appendixes on each of the individual elements as we  have done in prior reports under this mandate.
Pub. L. No. 112-239, § 224 (e), § 231 (e), § 229 (c), and § 228 (c). For our assessments  of DOD’s reports, see GAO, Missile Defense: Precision Tracking Space System  Evaluation of Alternatives, GAO-13-747R (Washington, D.C.: July 25, 2013) and Regional  Missile Defense: DOD’s Report Provided Limited Information; Assessment of Acquisition  Risks is Optimistic, GAO-14-248R (Washington, D.C.: Mar. 14, 2014). At the time of  publication of this report, our work on DOD’s reports on the Ground-based Midcourse  Defense system’s test plan and homeland defense is ongoing. officials from the Aegis BMD SM-3 Block IB, Aegis Ashore, and Aegis  Modernized Weapon System Software program offices. In Huntsville, we  interviewed program officials for BMDS Sensors, C2BMC, GMD, and  THAAD as well as officials in MDA’s Acquisition and Cost Directorates.
We also visited several contractor facilities that were working on  programs covered in our review. These facilities were located in  Huntsville and Courtland, Alabama as well as Tucson and Chandler,  Arizona. In Huntsville, we discussed the manufacturing of the Aegis BMD  SM-3 Block IB interceptor with Raytheon officials and met with GMD’s  prime contractor, Boeing, to discuss progress in resolving development  challenges and their plans to deliver additional interceptors. In Courtland,  we met with officials from Lockheed Martin to discuss the production of  the extended medium-range ballistic missile target, which was used in  flight test operational 01 on September 10, 2013. In Tucson and  Chandler, Arizona, we met with GMD’s subcontractors Raytheon and  Orbital to discuss their progress in resolving development challenges with  the interceptor, flight testing, and future development efforts. We also  interviewed officials from various testing agencies located in Arlington,  Virginia and Huntsville, Alabama. In Arlington, we met with officials from  DOD’s Director, Operational Test and Evaluation, as well as DOD’s  Director of Developmental Test and Evaluation, to discuss MDA’s test  plans and results from recent tests. Lastly, in Huntsville, we spoke with  officials from the BMDS Operational Test Agency to discuss MDA’s  performance assessment.
We conducted this performance audit from April 2013 to April 2014 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the contact named above, David Best, Assistant Director;  Aryn Ehlow; Meredith Allen Kimmett; Wiktor Niewiadomski; Kenneth E.  Patton; John H. Pendleton; Karen Richey; Steven Stern; Roxanna Sun,  Robert Swierczek; Brian Tittle; Hai V. Tran; and Alyssa Weir made key  contributions to this report.