Missile Defense: Additional Knowledge Needed in Developing System for Intercepting Long-Range Missiles
Published: 2003-08-21T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: A number of countries hostile to the United States and its allies have or will soon have missiles capable of delivering nuclear, biological, or chemical weapons. To counter this threat, the Department of Defense's (DOD's) Missile Defense Agency (MDA) is developing a system to defeat ballistic missiles. MDA expects to spend $50 billion over the next 5 years to develop and field this system. A significant portion of these funds will be invested in the Ground-based Midcourse Defense (GMD) element. To field elements as soon as practicable, MDA has adopted an acquisition strategy whereby capabilities are upgraded as new technologies become available and is implementing it in 2-year blocks. Given the risks inherent to this strategy, GAO was asked to determine when MDA plans to demonstrate the maturity of technologies critical to the performance of GMD's Block 2004 capability and to identify the estimated costs to develop and field the GMD element and any significant risks with the estimate.
section_title: What GAO Found
paragraphs: GMD is a sophisticated weapon system being developed to protect the United States against limited attacks by long-range ballistic missiles. It consists of a collection of radars and a weapon component--a three-stage booster and exoatmospheric kill vehicle--integrated by a centralized control system that formulates battle plans and directs the operation of GMD components. Successful performance of these components is dependent on 10 critical technologies. MDA expects to demonstrate the maturity of most of these technologies before fielding the GMD element, which is scheduled to begin in September 2004. However, the agency has accepted higher cost and schedule risks by beginning integration of the element's components before these technologies have matured. So far, MDA has matured two critical GMD technologies. If development and testing progress as planned, MDA expects to demonstrate the maturity of five other technologies by the second quarter of fiscal year 2004. The radar technologies are the least mature. MDA intends to demonstrate the maturity of an upgraded early warning radar in California in the first quarter of fiscal year 2005 and a sea-based radar in the Pacific Ocean in the fourth quarter of that year. Although MDA does not plan to demonstrate the maturity of the technology of the early warning radar in Alaska, which will serve as the primary fire control radar, through its own integrated flight tests, it may be able to do so through the anticipated launch of foreign test missiles. MDA estimates that it will spend about $21.8 billion between 1997 and 2009 to develop the GMD element. This estimate includes $7.8 billion to develop and field the GMD Block 2004 capability. For example, the funds will be used to install interceptors at two sites, upgrade existing radars and testing infrastructure, and develop the sea-based X-band radar. We found that MDA has incurred a greater risk of cost growth because for more than a year the agency was not able to rely fully on data from its primary tool for monitoring whether the GMD contractor has been performing work within cost and on schedule. In February 2002, MDA modified the prime contract to reflect an increased scope of work for developing GMD. It was not until July 2003 that the agency completed a review to ensure that the data was fully reliable.

section_title: Letter
paragraphs: Hostile states, including those that sponsor terrorism, are investing  significant resources to develop and deploy ballistic missiles of increasing  range and sophistication that could be used against the United States, our  deployed forces, and our allies. At least 25 countries now have, or are in  the process of acquiring, missiles capable of delivering nuclear, biological,  or chemical weapons. To counter this threat, the President of the United  States in December 2002, directed the Department of Defense (DOD) to  begin fielding a ballistic missile defense system in 2004.
The Missile Defense Agency (MDA) within DOD is responsible for  developing this system, including the Ground-based Midcourse Defense  (GMD) element, which is being developed to protect the United States  against long-range ballistic missiles. MDA is also building an integrated  testing infrastructure—or “test bed”—with the newly designated GMD  element as its centerpiece. MDA expects to spend nearly $50 billion in  research and development funds between fiscal years 2004 and 2009 to  develop and field a ballistic missile defense system. A significant  percentage of the $50 billion will be invested in the GMD element.
GMD is a sophisticated weapon system that will rely on state-of-the-art  technologies that have been under development for a number of years.  GMD will use space-based sensors to provide early warning of missile  launches; ground-based radars to identify and refine the tracks of  threatening warheads and associated objects; ground-based interceptors  (each consisting of a three-stage booster and exoatmospheric kill vehicle)  to destroy warheads; and a centralized control system that formulates  battle plans and directs the operation of GMD components for carrying out  the missile defense mission.
To meet the technical challenge of developing both the integrated  system and the GMD element, MDA has adopted a “capabilities-based”  acquisition strategy and is implementing it in 2-year development blocks.  This approach is designed to field elements as soon as practicable and  to improve the effectiveness of fielded elements by upgrading their  capability as new technologies become available or as the threat warrants.  Block 2004 will be the first block fielded, followed by Blocks 2006 and  2008. Although GMD’s Block 2004 capability is expected to be fielded  beginning in September 2004, MDA plans to upgrade that capability  through the end of 2005.
Because development and fielding of GMD involves substantial technical  challenges and a major investment, you asked us to review technical and  cost issues related to the GMD element. Specifically, we determined when  MDA plans to demonstrate the maturity of technologies critical to the  performance of GMD’s Block 2004 capability. We also identified the  estimated costs to develop and field the GMD element and any significant  risks associated with the estimate.
Our scope and methodology are included in appendix I. Although we  assessed the maturity of specific GMD critical technologies, the scope of  this review did not include an evaluation of MDA’s test plans for  demonstrating GMD’s ability to operate as a system overall. Our detailed  assessment of GMD system-level testing is included in a classified report  that we issued in June 2003 to other congressional requesters.
subsections: section_title: Background
paragraphs: The concept of using a missile to destroy another missile (hit-to-kill) has  been explored since the mid-1950’s, but it was not until 1984 that the first  such intercept achieved its objective. Between the mid-1980’s and  late-1990’s the United States conducted a number of experiments designed  to demonstrate that it was possible to hit one missile with another. In  1997, the Ballistic Missile Defense Organization (BMDO) established the  National Missile Defense (NMD) Joint Program Office. The program office  was directed to demonstrate by 1999 a system that could protect the  United States from attacks of intercontinental ballistic missiles and to  be in a position to deploy the system if the threat warranted by 2003. The  initial system consisted of space- and ground-based sensors, early warning  radars, interceptors, and battle management functions.
The program underwent additional changes as the new decade began.  In September 2000, the President decided to defer deployment of the  NMD system, but development of the system continued with the goal of  being ready to deploy the system when directed. This action was followed  in 2001 by BMDO’s redirection of the prime contractor’s efforts from  developing and deploying an NMD system to developing an integrated test  bed with the newly designated GMD system as its centerpiece. The  Secretary of Defense, in January 2002, renamed BMDO as MDA and  consolidated all ballistic missile defense programs under the new agency.  Former missile defense acquisition programs became elements of a single  ballistic missile defense system. These changes were followed in  December 2002, by the President’s directive to begin fielding in 2004 a  ballistic missile defense system, which included components of the GMD  element already under development.
The GMD element is intended to protect the United States against long- range ballistic missiles in the midcourse phase of their flight. This is the  point outside the atmosphere where the motors that boost an enemy  missile into space have stopped burning and the deployed warhead  follows a predictable path toward its target. Compared to the boost and  terminal phases, this stage of flight offers the largest window of  opportunity for interception and allows the GMD element a longer time to  track and engage a target.
As illustrated in figure 1, GMD will rely on a broad array of components  to track and intercept missiles. Figure 2 provides a notional concept of  how these components will operate once they are fully integrated into the  GMD element.
section_title: MDA Expects to Demonstrate the Maturity of Most GMD Technologies before September 2004
paragraphs: MDA is gaining the knowledge it needs to have confidence that  technologies critical to the GMD Block 2004 capability will work as  intended. Two of the ten technologies essential to the Block 2004  capability have already been incorporated into actual prototype hardware  and have been demonstrated to function as expected in an operational  environment. Other technologies are reaching this level of maturity. If  development and testing proceed as planned, MDA will demonstrate the  maturity of five additional technologies by the second quarter of fiscal  year 2004 and two critical radar technologies during fiscal year 2005. MDA  believes that its best opportunity to demonstrate the maturity of the tenth  technology, technology critical to GMD’s primary radar, may come  through the anticipated flight tests of foreign missiles.
Our work over the years has found that making a decision to begin system  integration of a capability before the maturity of all critical technologies  have been demonstrated increases the program’s cost, schedule, and  performance risks. Because the President directed DOD to begin fielding  a ballistic missile defense system in 2004, MDA began GMD system  integration with technologies whose maturity has not been demonstrated.  As a result, there is a greater likelihood that critical technologies will not  work as intended in planned flight tests. If this occurs, MDA may have to  spend additional funds in an attempt to identify and correct problems by  September 2004 or accept a less capable system.
subsections: section_title: Importance of Maturing Technology
paragraphs: Successful developers follow “knowledge-based acquisition” practices  to get quality products to the customer as quickly and cost effectively as  possible. As a part of meeting this goal, developers focus their technology  programs on maturing technologies that have the realistic potential for  being incorporated into the product under consideration. Accordingly,  successful developers spend time to mature technology in a technology  setting, where costs are typically not as great, and they do not move  forward with product development—the initiation of a program to fully  design, integrate, and demonstrate a product for production—until  essential technologies are sufficiently mature.
An analytical tool—which has been used by DOD and the National  Aeronautics and Space Administration, called technology readiness levels  (TRLs), —can assess the maturity level of technology as well as the risk  that technology poses if it is included in a product’s development. The nine  readiness levels are associated with progressing levels of technological  maturity and demonstrated performance relative to a particular  application—starting with paper studies of applied scientific principles  (TRL 1) and ending with a technology that has been “flight proven” on an  actual system through successful mission operations (TRL 9). Additional  details on TRLs are shown in appendix III.
TRLs provide a gauge of how much knowledge the program office has  on the progress or status of a particular technology and are based on  two principal factors: (1) the fidelity of demonstration hardware,  including design maturity and level of functionality achieved; and  (2) the extent and realism of the environment in which the technology  has been demonstrated.
MDA recognizes the value of beginning system integration with mature  technology and of using TRLs to assess the maturity of technology  proposed for a block configuration. In particular, MDA prefers to include  new technology in a block configuration only if the technology has  reached a TRL 7; that is, only if prototype hardware with the desired form,  fit, and function has been proved in an operational environment. However;  MDA retains the flexibility to include less mature technology in a block  configuration if that technology offers a significant benefit in performance  and the risk of retaining it is acceptable and properly managed.
section_title: Readiness Levels of GMD Element Technologies
paragraphs: Through technical discussions with the GMD joint program office and  its prime contractor, we identified ten critical GMD technologies and  jointly assessed the readiness level of each. The critical technologies  are resident in the exoatmospheric kill vehicle, the boosters, the battle  management, command, and control component, and in the element’s  radars. In 7 of 10 cases, we agreed with the program office and the  GMD prime contractor on the maturity level of the element’s critical  technologies. The differences in the remaining three cases, as discussed in  detail below, were primarily due to interpretation of TRL definitions. The  program office and its contractor rated the two booster technologies and  one radar technology at higher readiness levels than, in our opinion, MDA  had demonstrated.
Most critical GMD technologies are currently at TRLs 5 and 6. At TRL 5,  the technology’s development is nearing completion, but it has not been  applied or fitted for the intended product. At this point, the technology has  been incorporated into a high-fidelity breadboard that has been tested in  a laboratory or relevant environment. Although this demonstrates the  functionality of the technology to some extent, the hardware is not  necessarily of the form and fit (configuration) that would be integrated  into the final product. A new application of existing technology is usually  assessed at a TRL 5, because the technology has not been demonstrated in  the relevant environment for the new application. TRL 6 begins the true  “fitting” or application of the technology to the intended product. To reach  this level, technology must be a part of a representative prototype that is  very close to the form, fit, and function of that needed for the intended  product. Reaching a TRL 6 requires a major step in a technology’s  demonstrated readiness, that is, the prototype must be tested in a high- fidelity laboratory environment or demonstrated in a restricted but  relevant environment.
Two of the ten GMD technologies were assessed at a TRL 7, the level that  successful developers insist upon before initiating product development.  To reach this level, a pre-production prototype of the technology must be  demonstrated to its expected functionality in an operational environment.  If development and testing proceed as planned by MDA, we judge that  most of the technologies (7 of 10) will be at a TRL 7 after the completion  of integrated flight test (IFT)-14, which is scheduled for the second  quarter of fiscal year 2004. Table 1 summarizes our assessment of the TRL  for each critical technology as of June 2003 and the date at which MDA  anticipates each technology will reach TRL 7. A detailed discussion of  each critical technology follows.
section_title: Exoatmospheric Kill Vehicle Technologies
paragraphs: The exoatmospheric kill vehicle is the weapon component of the GMD  interceptor that attempts to detect and destroy the threat reentry vehicle  through a hit-to-kill impact. The prime contractor identified three critical  technologies pertaining to the operation of the exoatmospheric kill  vehicle. They include the following:  Infrared seeker, which is the “eyes” of the kill vehicle. The seeker is  designed to support kill vehicle functions like tracking and target  discrimination. The primary subcomponents of the seeker are the infrared  sensors, a telescope, and the cryostat that cools down the sensors.
On-board discrimination, which is needed to identify the true warhead  from among decoys and associated objects. Discrimination is a critical  function of the hit-to-kill mission that requires the successful execution of  a sequence of functions, including target detection, target tracking, and the  estimation of object features. As such, successful operation of the infrared  seeker is a prerequisite for discrimination.
Guidance, navigation, and control subsystem, which is a combination  of hardware and software that enables the kill vehicle to track its  position and velocity in space and to physically steer itself into the  designated target.
All three kill vehicle technologies have been demonstrated to some extent  in actual integrated flight tests on near-production-representative kill  vehicles. The infrared seeker has reached a TRL 7, because a configuration  very much like that to be fielded has been demonstrated in previous  integrated flight tests, and only minor design upgrades are planned to  reach the Block 2004 configuration. The remaining two kill vehicle  technologies are at a TRL 6, because their functionality is being upgraded  and the technologies have yet to be incorporated into the kill vehicle and  demonstrated in an operational environment.
The on-board discrimination technology has not yet reached TRL 7  because MDA has not tested a “knowledge database” that is expected to  increase the kill vehicle’s discrimination capability. The purpose of the  database is to enable the kill vehicle to distinguish characteristics of  threatening from non threatening objects. MDA expects to test the  database for the first time in IFT-14.
As a software-intensive technology, on-board discrimination performance  under all flight conditions can only be evaluated through ground testing,  but flight-testing is needed to validate the software’s operation in a real  world environment. The discrimination capability that will be tested in  IFT-14 is expected to be fielded as part of the Block 2004 capability.
Therefore, IFT-14 should demonstrate the technology’s maturity if the test  shows that the kill vehicle achieves its discrimination objective.
Similarly, the guidance, navigation, and control technology will also  increase to a TRL 7 if the technology achieves its objectives in IFT-14.  The inertial measurement unit, an important component of the guidance,  navigation, and control subsystem that enables the kill vehicle to track its  position and velocity, has not yet been tested in the severe environments  (e.g., vibrations and accelerations) induced by the operational booster.  This will be first attempted when one of the new operational boosters  is used in IFT-14. In addition to testing the inertial measurement unit,  IFT-14 will also test the upgraded divert hardware (used to actively  steer the kill vehicle to its target) that is expected to be part of the  Block 2004 configuration.
section_title: Booster Technologies
paragraphs: The integrated booster stack is the part of the GMD interceptor that is  composed of rocket motors needed to deliver and deploy the kill vehicle  into a desired intercept trajectory. For all flight tests to date, a two-stage  surrogate booster called the payload launch vehicle has been used.
In July 1998, the GMD prime contractor began developing a new  three-stage booster for the GMD program, known as the “Boost Vehicle”,  from commercial off-the-shelf components. However, the contractor  encountered difficulty. By the time the booster was flight tested in  August 2001, it was already about 18 months behind schedule. The first  booster flight test met its objectives, but the second booster tested drifted  off course and had to be destroyed 30 seconds after launch.
Subsequently, MDA altered its strategy for acquiring a new booster for  the interceptor. Instead of relying on a single contractor, MDA authorized  the GMD prime contractor to develop a second source for the booster  by awarding a subcontract to another contractor. If development of the  boosters proceeds as planned, both boosters will be part of the Block 2004  capability. One booster is known as BV+ and the other as “OSC Lite.”
subsections: section_title: The BV+ Booster
paragraphs: The prime contractor ultimately transferred development of the boost  vehicle to a subcontractor who is currently developing a variant—known  as “BV+”—for the GMD element. The program office and GMD  contractor rated the BV+ at a TRL 7. The prime contractor reasoned  that the extent of the legacy program and its one successful flight test  should allow for this rating. However, given the limited testing to date, we  assessed the BV+ booster currently at a TRL 6; that is, the technology has  been demonstrated in a restricted flight environment using hardware close  in form, fit, and function to that which will be fielded in 2004. We believe  the contractor’s assessment is too high at this time, because the step from  TRL 6 to TRL 7 is significant in terms of the fidelity of the demonstration  environment. However, the first test of a full configuration BV+ booster  will occur with IFT-13A, which is scheduled for the first quarter of fiscal  year 2004. In our opinion, the BV+ booster will reach TRL 7 at this time if  the booster works as planned.
section_title: The “OSC Lite” Booster
paragraphs: The second booster under development is referred to as “OSC Lite”. This  booster, which is essentially the Taurus Lite missile that carries satellites  into low-earth orbit, will be reconfigured for the GMD element. Despite the  fact that the booster was recently tested under restricted flight conditions,  GMD’s prime contractor believes that the legacy development of the  Taurus Lite missile is sufficient to prove that the OSC Lite has reached  TRL 7. However, in our opinion, because the test was conducted with  hardware configured as it was in the Taurus missile, not as it will be  configured for GMD’s Block 2004, the booster’s maturity level is  comparable to that of the BV+. The first flight test of a full configuration  OSC Lite booster is scheduled for IFT-13B in the first quarter of fiscal year  2004. We believe that if the booster performs as intended in this test, it will  reach TRL 7.
section_title: Battle Management Command, Control, and Communications Technologies
paragraphs: The battle management component is the integrating and controlling  component of the GMD element. Prime contractor officials identified and  assessed the following sub-components as critical technologies:    GMD fire control software, which analyzes the threat, plans engagements,  and tasks components of the GMD element to execute a mission.  In-flight interceptor communications system, which enables the GMD  fire control component to communicate with the exoatmospheric kill  vehicle while in flight.
The two battle management technologies have been demonstrated to some  extent in actual integrated flight tests, and both are near their Block 2004  design. We determined that the GMD fire control software has currently  achieved a TRL 7 and the in-flight interceptor communications system has  reached a TRL 6. Prime contractor officials concur with our assessment.
The fire control software is nearing expected functionality and prior  software builds have been demonstrated in GMD flight tests. Only minor  design changes will be made to address interfacing issues (linking the  fire control component with other GMD components) before the  software reaches the operational configuration of Block 2004. As a  software-intensive technology, the performance of the fire control  software throughout the entire “flight envelope” can only be evaluated  through ground testing. Ground testing is well underway at both the Joint  National Integration Center at Schriever Air Force Base, Colorado, and at  the prime contractor’s integration laboratory in Huntsville, Alabama.
The second technology associated with the battle management component  is the in-flight interceptor communications system. Even though the  pointing accuracy and communications capability of this technology  were demonstrated in previous flight tests, the operational hardware to  be fielded by 2004 is expected to operate at a different uplink frequency  than the legacy hardware used in these past flight tests. Accordingly, we  assessed the in-flight interceptor communications system at a TRL 6.  The first integrated flight test to include an operational-like build of this  technology is IFT-14, and if the technology meets its objectives in this  flight test, TRL 7 would be achieved.
section_title: Radar Technologies
paragraphs: The GMD contractor initially identified the sea-based X-band radar as  the only radar-related critical technology. Since its initial assessment in  September 2002, the contractor has now agreed with us that the Beale  upgraded early warning radar and the Cobra Dane radar are also critical  technologies of the GMD element. The contractor and the GMD program  office assessed the Beale and Cobra Dane radars at a TRL 5, because the  technology, especially mission software, is still under development and  has not yet been demonstrated in a relevant flight environment. The  contractor assessed the sea-based X-band radar at a TRL 6. As discussed  below, we agree with their assessment of the Beale and Cobra Dane radars  but rated the sea-based X-band radar as a TRL 5.
The early warning radar at Beale Air Force Base has participated in  integrated flight tests in a missile-defense role using legacy hardware  and developmental software. Design and development of operational  builds of the software are progressing, but such builds have only been  tested in a simulated environment. Therefore, we assessed the Beale radar  technology at a TRL 5—an assessment driven by software considerations.  The conversion of the early warning radar at Beale to an upgraded early  warning radar, which consists of minor hardware and significant software  upgrades, is planned for completion sometime during the middle of fiscal  year 2004. After this time, the Beale radar can take part in flight-testing  in its upgraded configuration. MDA currently plans to demonstrate the  upgraded Beale technology in a non intercept flight test, known as a  radar certification flight, in the first quarter of fiscal year 2005. The Beale  radar will be demonstrated at a TRL 7 if the objectives of this flight test  are achieved.
The Cobra Dane radar is currently being used in a surveillance mode  to collect data on selected intercontinental ballistic missile test launches  out of Russia and does not require real-time data processing and  communications capabilities. To achieve a defensive capability by  September 2004, the Cobra Dane radar is being upgraded to perform  both of these tasks. This upgrade, which requires a number of software  modifications, is designed to enable Cobra Dane to detect and track  enemy targets much as the Beale upgraded early warning radar does.  Although the hardware component of the Cobra Dane radar is mature and  will undergo only minor updating, Cobra Dane’s mission software is being  revised for this application. The revision includes reuse of existing  software and development of new software so that the Cobra Dane radar  can be integrated into the GMD architecture.
Upgrades to the Cobra Dane radar are due to be completed at the  beginning of 2004. After the software is developed and ground tested, the  radar can reach a TRL 6, but it is uncertain when the radar will reach a  TRL 7. Because of other funding and scheduling priorities, MDA has no  plans through fiscal year 2007 for using this radar in integrated flight tests;  such tests would require air- or sea-launched targets that are not currently  part of the test program. Unless the current test program is modified, the  only opportunities for demonstrating Cobra Dane in an operational  environment would come from flight tests of foreign missiles. MDA  officials anticipate that such opportunities will occur. However, it is not  clear that testing Cobra Dane in this manner will provide all of the  information that a dedicated test provides because MDA will not control  the configuration of the target or the flight environment.
The sea-based X-band radar is being built as part of the Block 2004  capability and scheduled for completion in 2005. It will be built from  demonstrated technologies—a sea-based platform and the prototype  X-band radar currently being used in the GMD test program. Prime  contractor officials told us that they consider the risk associated with  the construction and checkout of the radar as primarily a programmatic,  rather than technical risk, and believe that the sea-based X-band radar has  reached a TRL 6. The contractor also stated that the initial operational  build of the radar software is developed and currently being tested at the  contractor’s integration laboratory. We assessed the sea-based X-band  radar as a TRL 5 because the radar has not yet been built and because  constructing a radar from an existing design and placing it on a sea-based  platform is a new application of existing technology. For example, severe  wind and sea conditions may affect the radar’s functionality—conditions  that cannot be replicated in a laboratory. As a result, developers cannot be  sure that the sea-based X-band radar will work as intended until it is  demonstrated in this new environment. However, both we and the  contractor agree that the maturity level of the sea-based X-band radar will  increase to a TRL 7 if it achieves its test objectives in IFT-18 (scheduled  for the fourth quarter of fiscal year 2005).
section_title: MDA Has Risked Cost Growth Because It Could Not Fully Rely on Data from Its System for Monitoring Contractor Performance
paragraphs: From the program’s inception in 1997 through 2009, MDA expects to  spend about $21.8 billion to develop the GMD element. About $7.8 billion  of the estimated cost will be needed between 2002 and 2005 to develop  and field the Block 2004 GMD capability and to develop the GMD portion  of the test bed. However, MDA has incurred a greater risk of cost  increases because for more than a year MDA was not sure that it could  rely fully upon data from the prime contractor’s Earned Value  Management (EVM) system, which provides program managers and  others with early warning of problems that could cause cost and schedule  growth.
subsections: section_title: GMD Development Costs
paragraphs: Before the restructuring of the GMD program in 2002, about $6.2 billion  was spent (between 1997 and 2001) to develop a ground-based defense  capability. MDA estimates it will need an additional $7.8 billion between  2002 and 2005 to, among other tasks, install interceptors at Fort Greely,  Alaska, and at Vandenberg Air Force Base, California; upgrade existing  radars and test bed infrastructure; and develop the sea-based X-band radar  that will be added in the fourth quarter of fiscal year 2005. In addition,  MDA will invest an additional $7.8 billion between fiscal year 2004 and  2009 to continue efforts begun under Block 2004, such as enhancing  capability and expanding the test bed. Table 2, below, provides details on  the funding requirements by block and by fiscal year, and figure 3 provides  examples of specific Block 2004 tasks.
MDA did not include the following costs is its Block 2004 estimate:    The cost to recruit, hire, and train military personnel to operate the initial  defensive capability and provide site security at various locations, which  MDA estimates to be an additional $13.4 million (half in fiscal year 2003  and half in 2004 each), will be needed to operate GMD and provide  physical security. Additional costs to cover these personnel throughout  the life of the program beginning in 2005 and beyond were also omitted.
The cost to maintain equipment and facilities was not included.    Systems engineering and national team costs—which benefit all elements,  including GMD and cannot be divided among the elements—were not  included in MDA’s budget.
section_title: MDA’s Insight into Potential Cost Growth Was Limited by the Agency’s Inability to Rely Fully on Data from Earned Value Management System
paragraphs: Because a significant portion of MDA’s Block 2004 GMD cost estimate is  the cost of work being performed by the element’s prime contractor,  MDA’s ability to closely monitor its contractor’s performance is critical to  controlling costs. The tool that MDA, and many DOD entities, have chosen  for this purpose is the EVM system. This system uses contractor reported  data to provide program managers and others with timely information on a  contractor’s ability to perform work within estimated cost and schedule. It  does so by examining variances reported in contractor cost performance  reports between the actual cost and time of performing work tasks and the  budgeted or estimated cost and time. While this tool can provide insightful  information to managers, MDA’s use of it has been hampered by several  factors. Principally, although major contract modifications were made in  February 2002, it took until July 2003 for MDA to complete a review to  confirm the reliability of data from the EVM system. An earlier review of a  similar nature revealed significant deficiencies in the contractor’s  formulation and collection of EVM data. Until a new review was  completed, MDA could not be sure about its ability to rely fully upon this  data to identify potential problems in time to prevent significant cost  growth and schedule delays.
subsections: section_title: Baseline Revised over 13-Month Period
paragraphs: An accurate, valid, and current performance management baseline is  needed to perform useful analyses using EVM. The baseline identifies and  defines work tasks, designates and assigns organizational responsibility  for each task, schedules the work task in accordance with established  targets, and allocates budget to the scheduled work. According to DOD  guidance, a performance management baseline should be in place as  early as possible after the contractor is authorized to proceed. Although  the guidance does not define how quickly the contractor should establish  a baseline, experts generally agree that it should be in place, on average,  within 3 months after a contract is awarded or modified.
About a year before the Secretary of Defense directed MDA to adopt an  evolutionary acquisition strategy, the agency awarded a new contract for  the development of a National Missile Defense system. In February 2002,  MDA modified this contract to redirect the contractor’s efforts. Instead of  developing a missile defense system that met all of the requirements of the  war fighter, as the initial contract required, the modification directed the  contractor to develop the first GMD increment, or block, which was to be  a ballistic missile test bed with GMD as its centerpiece.
Following the contract’s modification, the contractor in June 2002  established an interim baseline. This baseline was developed by adding  budgets for near-term new work to the original baseline. Because the cost  of the work being added to the baseline had not yet been negotiated, the  contractor based the budgets on the cost proposed to MDA, as directed by  DOD guidelines. The contractor implemented the baseline almost within  the 3-month time frame recommended by experts. In the time between the  modification and the development of the interim baseline, MDA authorized  the contractor to begin work and spend a specified amount of money, and  MDA paid the contractor about $390 million during this period.
An option that MDA could have used to help validate the interim baseline  was to have the Defense Contract Management Agency (DCMA) verify  contractor work packages and track the movement of funds between the  unpriced work account and the baseline. However, neither MDA nor  DCMA initiated these actions. In its technical comments on a draft of this  report, DOD pointed out that during the negotiation process, MDA reviews  prime and subcontractor proposal data that include engineering labor  hours, material, and cost estimates. DOD further noted that these  estimates eventually form a basis for the work packages that make up the  data for the performance management baseline. We agree that these costs  will eventually be associated with the work packages that make up the  baseline. However, a joint contractor and MDA review of the initial GMD  baseline concluded that even though these costs were otherwise fair and  reasonable, some work packages that the contractor developed for the  original contract’s baseline did not correctly reflect the work directed by  MDA. An independent review of work packages included in the interim  baseline would have increased the likelihood that the work packages were  being properly developed and that their budget and schedule were  appropriate.
The contractor completed all revisions to the baseline for the prime  contractor and all five subcontractors by March 2003, 3 months  after negotiating the cost of the modification and 13 months after  authorizing the work to begin. The contracting officer explained that it  took until December 2002 to negotiate the 2002 contract change because  the additional work was extremely complex, and, as a result, the  modification needed to be vetted through many subcontractors that  support the prime.
section_title: Baseline Review Completed in July 2003
paragraphs: The DOD guidance states that an integrated baseline review (IBR) is to be  conducted within 6 months of award of a new contract or major change  to an existing contract. The review verifies the technical content of the  baseline. It also ensures that contractor personnel understand and have  been adequately trained to collect EVM data. The review also verifies the  accuracy of the related budget and schedules, ensures that risks have been  properly identified, assesses the contractor’s ability to implement properly  EVM, and determines if the work identified by the contractor meets the  program’s objectives. The government’s program manager and technical  staff carry out this review with their contractor counterparts.
Completing an IBR of the new baseline has been of particular importance  because the July 2001 IBR for the initial contract identified more than  300 deficiencies in the contractor’s formulation and execution of the  baseline. For example, the contractor had not defined a critical path for  the overall effort, many tasks did not have sufficient milestones that would  allow the contractor to objectively measure performance, and contractor  personnel who were responsible for reporting earned value were making  mistakes in measuring actual performance against the baseline.
MDA began a review in March 2003 of the contractor’s new baseline,  which reflected the contract modification,. Completing this IBR took  until July 2003 because of the complexity of the program and the many  subcontractors that were involved. Although the review team found fewer  problems with the contractor’s formulation and execution of the new  baseline, problems were identified. For example, the IBR showed that in  some cases the baseline did not reflect the new statement of work. Also,  both the prime contractor and subcontractors improperly allocated budget  to activities that indirectly affect a work product (known as level of effort  activities) when they could have associated these activities with a discrete  end product. Because of the way these activities are accounted for, this  designation could mask true cost variances.
section_title: Management Reserve Used to Offset Expected Cost Overruns at Contract Completion
paragraphs: Before the IBR was underway, DCMA recognized another problem with  the contractor’s EVM reports. In its December 2002 cost performance  report, the contractor reported that it expected no cost overrun at  contract completion. This implied that the program was not experiencing  any problems that could result in significant cost or schedule growth.  However, DCMA stated that October 2002 was the second month in a row  that the contractor had used management reserve funds to offset a  significant negative cost variance. DCMA emphasized that this is not the  intended purpose of management reserves. (Management reserves are a  part of the total project budget intended to be used to fund work  anticipated but not currently defined.) DCMA officials told us that while  this is not a prohibited practice most programs wait until their work is  almost completed, that is 80 to 90 percent complete, before making a  judgment that the management reserve would not be needed for additional  undefined work and could be applied to unfavorable contract cost  variances.
section_title: Conclusions
paragraphs: Because of the President’s direction to begin fielding a ballistic missile  defense system in 2004, the MDA took a higher risk approach by beginning  GMD system integration before knowing whether its critical technologies  were mature. If development and testing progress as planned, however,  MDA expects to have demonstrated the maturity of 7 of the 10 critical  GMD technologies before the element is initially fielded in September 2004  and 2 others during fiscal year 2005. If technologies do not achieve their  objectives during testing, MDA may have to spend additional funds in an  attempt to identify and correct problems by September 2004 or accept a  less capable system.
Because of other funding and scheduling priorities, MDA does not plan  to demonstrate through integrated flight tests whether the Cobra Dane  radar’s software can process and communicate data on the location of  enemy missiles in “real time.” Although tests using sea- or air-launched  targets before September 2004 would provide otherwise unavailable  information on the software’s performance, we recognize those tests  would be costly and funds have not been allocated for that purpose. We  also recognize that the most cost efficient means of testing the Cobra Dane  radar is through launches involving foreign test missiles. However, we  believe it would be useful for MDA to consider whether the increased  confidence provided by a planned test event outweighs other uses for  those funds.
MDA is investing a significant amount of money to achieve an operational  capability during the first block of GMD’s development, and the agency  expects to continue investing in the element’s improvement over the next  several years. Because MDA is also developing other elements and must  balance its investment in each, it needs an accurate GMD cost estimate.  If it is used as intended, the EVM system can be an effective means of  monitoring one of GMD’s largest costs, the cost of having a contractor  develop the GMD system. It is understandable that the dynamic changes in  MDA’s acquisition strategy led to major contract modifications, which  made it more difficult for the contractor to establish a stable baseline.  However, in this environment, it is even more important that MDA find  ways to ensure the integrity of the interim baselines and to quickly  determine that revised baselines can be fully relied on to identify potential  problems before they significantly affect the program’s cost.
section_title: Recommendations for Executive Action
paragraphs: To increase its confidence that the Ground-based Midcourse Defense  element fielded in 2004 will operate as intended, we recommend that the  Secretary of Defense direct the Director, Missile Defense Agency, to  explore its options for demonstrating the upgraded Cobra Dane radar in its  new ballistic missile defense role in a real-world environment before  September 2004.
To improve MDA’s oversight of the GMD element and to provide the  Congress with the best available information for overseeing the program,  we recommend that the Secretary of Defense direct the Director, Missile  Defense Agency, to:     ensure that when a contractor is authorized to begin new work before a  price is negotiated that DCMA validate the performance measurement  baseline to the extent possible by (1) tracking the movement of budget  from the authorized, unpriced work account into the baseline, (2) verify  that the work packages accurately reflect the new work directed, and  (3) report the results of this effort to MDA; and    strive to initiate and complete an integrated baseline review (IBR) of any  major contract modifications within 6 months.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD’s comments on our draft report are reprinted in appendix II.  DOD concurred with our first recommendation. DOD stated that MDA is  exploring its options for demonstrating, prior to 2004, the upgraded  Cobra Dane radar in a real-world environment. However, DOD noted that  because it takes considerable time to develop and produce targets and to  conduct safety and environmental assessments, completing a Cobra  Dane radar test before September 2004 would be very challenging. DOD  concluded that “targets of opportunity” (flight tests of foreign missiles)  and ground testing may provide the best means to demonstrate the radar’s  maturity in the near term.
DOD partially concurred with our second recommendation. In responding  to the first part of recommendation two, DOD stated that MDA and the  DCMA will jointly determine the feasibility of tracking the budget for  authorized, unpriced work into the baseline and will concurrently  assess work package data while establishing the formal performance  measurement baseline. DOD also stated that a selected portion of this  work is already being accomplished by DCMA. We continue to believe in  the feasibility of our recommendation. DCMA officials told us that they  could monitor the movement of budget into the baseline and verify the  work packages associated with the budget. In addition, the guidelines  state that surveillance may be accomplished through sampling of internal  and external data. We believe that if DCMA sampled the data as it is  transferred into the baseline, the implementation of this recommendation  should not be burdensome.
In responding to the second part of recommendation two, DOD stated that  MDA will continue to adhere to current DOD policy by starting an IBR of  any major contract modification within 6 months. MDA correctly pointed  out that DOD’s Interim Defense Acquisition Guidebook only requires a  review be initiated within 6 months (180 days) after a contract is awarded  or a major modification is issued. However, DOD’s Earned Value  Management Implementation Guide states that such a review is  conducted within 6 months. Similar language is found in the applicable  clause from the GMD contract, which states that such reviews shall be  scheduled as early as practicable and should be conducted within 180  calendar days after the incorporation of major modifications. While we  understand the difficulty of conducting reviews within 180 days when the  contract is complex and many subcontractors are involved, we believe  that it is important for the government to complete an IBR as soon as  possible to ensure accurate measurement of progress toward the  program’s cost, schedule, and performance goals.
DOD also provided technical comments to this report, which we  considered and implemented as appropriate. In its technical comments,  for example, DOD expressed particular concern that our draft report  language asserting MDA’s inability to rely on the EVM system was  unsupported and misleading. DOD also stated that its prime contractor’s  EVM system is reliable. It stated, for example, that MDA has reviewed, and  continues to review on a monthly basis, the contractor’s cost performance  reports and that the prime contractor’s EVM system and accounting  systems have been fully certified and validated by DCMA. We modified our  report to better recognize MDA’s ability to use and trust the EVM system.  However, we still believe that MDA would benefit from taking additional  measures to increase its confidence in the accuracy of its interim  baselines. Also, when the revised baseline is in place, a review of its  formulation and execution is necessary before MDA can confidently and  fully rely on data from the EVM system.
We conducted our review from December 2001 through August 2003 in  accordance with generally accepted government auditing standards. As  arranged with your staff, unless you publicly announce its contents earlier,  we plan no further distribution of this report until 30 days from its issue  date. At that time, we plan to provide copies of this report to interested  congressional committees, the Secretary of Defense, and the Director,  Missile Defense Agency. We will make copies available to others upon  request. In addition, the report will be available at no charge on the GAO  Web site at http://www.gao.gov/.
If you or your staff have any questions concerning this report, please  contact me on (202) 512-4841. Major contributors to this report are listed  in appendix V.
section_title: Appendix I: Scope and Methodology
paragraphs: To determine when MDA plans to demonstrate the maturity of  technologies critical to the performance of GMD’s Block 2004 capability,  we reviewed their critical technologies using technology readiness levels  (TRLs) developed by the National Aeronautics and Space Administration  and used by DOD. We did so by asking contractor officials at the Boeing  System Engineering and Integration Office in Arlington, Virginia, to  identify the most critical technologies and to assess the level of maturity of  each technology using definitions developed by the National Aeronautics  and Space Administration. We reviewed these assessments along with  program documents, such as the results of recent flight tests and  discussed the results with contractor and agency officials in order to reach  a consensus, where appropriate, on the readiness level for each  technology and identify the reasons for any disagreements.
In reviewing the agency’s current cost estimate to develop the first block  of the GMD element and its test bed, we reviewed and analyzed budget  backup documents, cost documents, and selected acquisition reports for  the GMD program extending over a period of several years. We also met  with program officials responsible for managing the development and  fielding of the GMD Block 2004 capability. For example, we met with  officials from the GMD Joint Program Office in Arlington, Virginia, and  Huntsville, Alabama; and the Office of the Deputy Assistant for Program  Integration at the MDA, Arlington, Virginia.
To determine whether there were any significant risks associated with the  estimate, we met with agency officials responsible for determining the  cost of the GMD element to find out if there were costs that were omitted,  but should have been included, in the estimate. We also analyzed data  from cost performance reports that the GMD contractor developed for the  MDA. We reviewed data from the GMD element and contracting officials  and conducted interviews to discuss the data. Although we did not  independently verify the accuracy of the cost performance reports we  received from MDA, the data were assessed independently by DCMA.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: Technology Readiness Level Assessment Matrix
paragraphs: Appendix III: Technology Readiness Level  Assessment Matrix  Lowest level of technology  readiness. Scientific  research begins to be  translated into applied  research and development.  Examples might include  paper studies of a  technology’s basic  properties.
None (paper studies and  analysis)
Invention begins. Once  basic principles are  observed, practical  applications can be  invented. The application is  speculative, and there is no  proof or detailed analysis to  support the assumption.  Examples are still limited to  paper studies.
None (paper studies and  analysis)
Active research and  development is initiated.  This includes analytical  studies and laboratory  studies to physically  validate analytical  predictions of separate  elements of the technology.  Examples include  components that are not  yet integrated or  representative.
Analytical studies and  demonstration of nonscale  individual components  (pieces of subsystem).
Low fidelity breadboard.  Integration of nonscale  components to show  pieces will work together.  Not fully functional or form  or fit but representative of  technically feasible  approach suitable for flight  articles.
Basic technological  components are integrated  to establish that the pieces  will work together. This is  relatively “low fidelity”  compared to the eventual  system. Examples include  integration of “ad hoc”  hardware in a laboratory.
Fidelity of breadboard  technology increases  significantly. The basic  technological components  are integrated with  reasonably realistic  supporting elements so that  the technology can be  tested in a simulated  environment. Examples  include “high fidelity”  laboratory integration of  components.
High fidelity breadboard.  Functionally equivalent but  not necessarily form and/or  fit (size, weight, materials,  etc). Should be  approaching appropriate  scale. May include  integration of several  components with  reasonably realistic support  elements/subsystems to  demonstrate functionality.
Lab demonstrating functionality but  not form and fit. May include flight- demonstrating breadboard in  surrogate aircraft.
Technology ready for detailed  design studies.
Representative model or  prototype system, which is  well beyond the breadboard  tested for TRL 5, is tested  in a relevant environment.  Represents a major step up  in a technology’s  demonstrated readiness.  Examples include testing a  prototype in a high fidelity  laboratory environment or  in simulated operational  environment.
Prototype. Should be very  close to form, fit, and  function. Probably includes  the integration of many  new components and  realistic supporting  elements/subsystems if  needed to demonstrate full  functionality of the  subsystem.
High-fidelity lab demonstration or  limited/restricted flight  demonstration for a relevant  environment.
Integration of technology is well  defined.
Prototype near or at  planned operational  system. Represents a  major step up from TRL 6,  requiring the demonstration  of an actual system  prototype in an operational  environment, such as in an  aircraft, on a vehicle or in  space. Examples include  testing the prototype in a  test bed aircraft.
Prototype. Should be form,  fit and function integrated  with other key supporting  elements/subsystems to  demonstrate full  functionality of subsystem.
Flight demonstration in  representative operational  environment such as flying test bed  or demonstrator aircraft.
Technology is well substantiated  with test data.
Technology has been  proven to work in its final  form and under expected  conditions. In almost all  cases, this TRL represents  the end of true system  development. Examples  include developmental test  and evaluation of the  system in its intended  weapon system to  determine if it meets design  specifications.
Actual application of the  technology in its final form  and under mission  conditions, such as those  encountered in operational  test and evaluation. In  almost all cases, this is the  end of the last “bug fixing”  aspects of true system  development. Examples  include using the system  under operational mission  conditions.
section_title: Appendix IV: Importance of Earned Value Management
paragraphs: Pulling together essential cost, schedule, and technical information in a  meaningful, coherent fashion is always a challenge for any program.  Without this information, management of the program will be fragmented,  presenting a distorted view of program status. For several decades, DOD  has compared the value of work performed to the work’s actual cost. This  measurement is referred to as Earned Value Management (EVM). Earned  value goes beyond the two-dimensional approach of comparing budgeted  costs to actuals. It attempts to compare the value of work accomplished  during a given period with the work scheduled for that period. By using  the value of completed work as a basis for estimating the cost and time  needed to complete the program, the earned value concept should alert  program managers to potential problems early in the program.
In 1996, in response to acquisition reform initiatives, DOD reemphasized  the importance of earned value in program management and adopted  32 criteria for evaluating the quality of management systems. These  32 criteria are organized into 5 basic categories: organization, planning and  budgeting, accounting considerations, analysis and management reports,  and revisions and data maintenance. The 32 criteria are listed in table 1. In  general terms, the criteria require contractors to (1) define the contractual  scope of work using a work breakdown structure; (2) identify  organizational responsibility for the work; (3) integrate internal  management subsystems; (4) schedule and budget authorized work;  (5) measure the progress of work based on objective indicators; (6) collect  the cost of labor and materials associated with the work performed;  (7) analyze any variances from planned cost and schedules; (8) forecast  costs at contract completion; and (9) control changes.
The criteria have become the standard for EVM and have also been  adopted by major US government agencies, industry, and the governments  of Canada and Australia. The full application of EVM system criteria is  appropriate for large cost reimbursable contracts where the government  bears the cost risk. For such contracts, the management discipline  described by the criteria is essential. In addition, data from an EVM system  have been proven to provide objective reports of contract status, allowing  numerous indices and performance measures to be calculated. These can  then be used to develop accurate estimates of anticipated costs at  completion, providing early warning of impending schedule delays and  cost overruns.
The standard format for tracking earned value is through a Cost  Performance Report (CPR). The CPR is a monthly compilation of cost,  schedule and technical data which displays the performance measurement  baseline, any cost and schedule variances from that baseline, the amount  of management reserve used to date, the portion of the contract that is  authorized unpriced work, and the contractor’s latest revised estimate to  complete the program.
As a result, the CPR can be used as an effective management tool because  it provides the program manager with early warning of potential cost and  schedule overruns. Using data from the CPR, a program manager can  assess trends in cost and schedule performance. This information is useful  because trends tend to continue and can be difficult to reverse. Studies  have shown that once programs are 15 percent complete the performance  indicators are indicative of the final outcome. For example, a CPR  showing a negative trend for schedule status would indicate that the  program is behind schedule. By analyzing the CPR, one could determine  the cause of the schedule problem such as delayed flight tests, changes in  requirements, or test problems because the CPR contains a section that  describes the reasons for the negative status. A negative schedule  condition is a cause for concern, because it can be a predictor of later cost  problems since additional spending is often necessary to resolve  problems. For instance, if a program finishes 6 months later than planned,  additional costs will be expended to cover the salaries of personnel and  their overhead beyond what was originally expected. CPR data provides  the basis for independent assessments of a program’s cost and schedule  status and can be used to project final costs at completion in addition to  determining when a program should be completed.
Examining a program’s management reserve is another way that a  program can use a CPR to determine potential issues early on.
Management reserves, which are funds that may be used as needed,  provide flexibility to cope with problems or unexpected events. EVM  experts agree that transfers of management reserve should be tracked and  reported because they are often problem indicators. An alarming situation  arises if the CPR shows that the management reserve is being used at a  faster pace than the program is progressing toward completion. For  example, a problem would be indicated if a program has used 80 percent  of its management reserve but only completed 40 percent of its work. A  program’s management reserve should contain at least 10 percent of the  cost to complete a program so that funds will always be available to cover  future unexpected problems that are more likely to surface as the program  moves into the testing and evaluation phase.
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the individual named above Yvette Banks, Myra Watts  Butler, Cristina Chaplain, Roger Corrado, Jennifer Echard, Dayna Foster,  Matt Lea, Karen Richey, and Randy Zounes made key contributions to this  report.