Missile Defense: Review of Results and Limitations of an Early National Missile Defense Flight Test
Released: 2002-03-04T00:00:00

Letter

Disclosure of Key Results and Limitations

Boeing and TRW disclosed the key results and limitations of Integrated Flight Test 1A in written reports released between August 13, 1997, and April 1, 1998. The contractors explained in a report issued 60 days after the June 1997 test that the test achieved its primary objectives, but that some sensor abnormalities were noted. For example, while the report explained that the sensor detected the deployed targets and collected some usable target signals, the report also stated that some sensor components did not operate as desired and the sensor often detected targets where there were none. In December 1997, the contractors documented other test anomalies. According to briefing charts prepared for a December meeting, the Boeing sensor tested in Integrated Flight Test 1A had a low probability of detection; the sensor’s software was not always confident that it had correctly identified some target objects; the software significantly increased the rank of one target object toward the end of the flight; and in-flight calibration of the sensor was inconsistent. Additionally, on April 1, 1998, the contractors submitted an addendum to an earlier report that noted two more problems. In this addendum, the contractors disclosed that their claim that TRW’s software successfully distinguished a mock warhead from decoys during a post-flight analysis was based on tests of the software using about one-third of the target signals collected during Integrated Flight Test 1A. The contractors also noted that TRW reduced the software’s reference data so that it would correspond to the collected target signals being analyzed. Project office and Nichols Research officials said that in late August 1997, the contractors orally communicated to them all problems and limitations that were subsequently described in the December 1997 briefing and the April 1998 addendum. However, neither project officials nor contractors could provide us with documentation of these communications.

Although the contractors reported the test’s key results and limitations, they described the results using some terms that were not defined. For example, one written report characterized the test as a “success” and the sensor’s performance as “excellent.” We found that the information in the contractors’ reports, in total, enabled officials in the Ground Based Interceptor Project Management Office and Nichols Research to understand the key results and limitations of the test. However, because such terms are qualitative and subjective rather than quantitative and objective, their use increased the likelihood that test results would be interpreted in different ways and might even be misunderstood. As part of our ongoing review of missile defense testing, we are examining the need for improvements in test reporting.

Appendix I provides details on the test and the information disclosed.

Project Office Reliance on Various Sources for Contractor Oversight

The Ground Based Interceptor Project Management Office relied on an on- site engineer and Nichols Research Corporation to provide insight into Boeing’s work. The project office also relied on Boeing to oversee the performance of its subcontractor, TRW. Oversight was limited by the ongoing competition between Boeing and another contractor competing for the exoatmospheric kill vehicle contract because the Ground Based Interceptor Project Management Office and its support contractors had to be careful not to affect competition by assisting one contractor more than another. Project officials said that they relied more on “insight” into the contractors’ work rather than oversight of that work. Nichols gained program insight by attending technical meetings, assessing test reports, and sometimes evaluating technologies proposed by Boeing and TRW.

For more information on how the project office exercised oversight over its contractors’ technical performance, see appendix II.

Distinguishable Differences in Objects Deployed in Space

Boeing and TRW reported that post-flight testing and analysis of data collected during Integrated Flight Test 1A showed that deployed target objects displayed distinguishable features when observed by an infrared sensor. The contractors reported the test also showed that Boeing’s exoatmospheric kill vehicle sensor could collect target signals from which TRW’s software could extract distinguishable features and that the software could identify the mock warhead from other objects by comparing the extracted features to the features that it had been told to expect each object to display. However, there has been no independent verification of these claims.

We talked with Dr. Mike Munn, who was, during the 1980s, the Chief Scientist for missile defense programs at Lockheed Missiles and Space Company. He agreed that a warhead and decoys deployed in the exoatmosphere likely display distinguishable differences in the infrared spectrum. However, the differences may not be fully understood or there may not presently be methods to predict the differences. Dr. Munn added that the key was in the ability to make both accurate and precise measurements and also to predict signatures accurately. He emphasized that robust discrimination depends on the ability to predict signatures and then to match in-space measurements with those predictions. The Phase One Engineering Team and Nichols Research Corporation have noted that TRW's software used prior knowledge of warhead and decoy differences, to the maximum extent available, to discriminate one object from the other and cautioned such knowledge may not always be available in the real world.

Decoy Reduction in Later Tests

National Missile Defense program officials said that after considerable debate among themselves and contractors, the program manager reduced the number of decoys planned for intercept flight tests in response to a recommendation by an independent panel, known as the Welch Panel.The panel, established to reduce risk in ballistic missile defense flight test programs, viewed a successful hit-to-kill engagement as a difficult task that should not be further complicated in early tests by the addition of decoys. After contemplating the advice of the Welch panel and considering the opinions of program officials and contractors who disagreed over the number and complexity of decoys that should be deployed in future tests, the program manager decided that early tests should include only one decoy, a large balloon.

See appendix III for more information on the reduction of decoys in later tests.

Evaluation of TRW’s Discrimination Software

The Phase One Engineering Team was tasked by the National Missile Defense Joint Program Office to assess the performance of TRW’s software and to complete the assessment within 2 months using available data. The team's methodology included determining if TRW’s software was based on sound mathematical, engineering, and scientific principles and testing the software’s critical modules using data from Integrated Flight Test 1A.

The team reported that although the software had weaknesses, it was well designed and worked properly, with only some changes needed to increase the robustness of the discrimination function. Further, the team reported that the results of its test of the software using Integrated Flight Test 1A data produced essentially the same results as those reported by TRW. Based on its analysis, team members predicted that the software would perform successfully in a future intercept test if target objects deployed as expected.

Because the Phase One Engineering Team did not process the raw data from Integrated Flight Test 1A or develop its own reference data, the team cannot be said to have definitively proved or disproved TRW’s claim that its software successfully discriminated the mock warhead from decoys using data collected from Integrated Flight Test 1A. A team member told us its use of Boeing- and TRW-provided data was appropriate because the former TRW employee had not alleged that the contractors tampered with the raw test data or used inappropriate reference data.

Appendix IV provides additional details on the Phase One Engineering Team evaluation.

Agency Comments and Our Evaluation

In commenting on a draft of this report, the Department of Defense concurred with our findings. It also suggested technical changes, which we incorporated as appropriate. The Department's comments are reprinted in appendix VII.

We conducted our review from August 2000 through February 2002 in accordance with generally accepted government auditing standards. Appendix VI provides details on our scope and methodology. The National Missile Defense Joint Program Office’s process for releasing documents significantly slowed our work. For example, the program office took approximately 4 months to release key documents such as the Phase One Engineering Team’s response to the professor’s allegations. We requested these and other documents on September 14, 2000, and received them on January 9, 2001.

As arranged with your staff, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from its issue date. At that time, we plan to provide copies of this report to the Chairmen and Ranking Minority Members of the Senate Committee on Armed Services; the Senate Committee on Appropriations, Subcommittee on Defense; the House Committee on Armed Services; and the House Committee on Appropriations, Subcommittee on Defense; and the Secretary of Defense; and the Director, Missile Defense Agency. We will make copies available to others upon request.

If you or your staff have any questions concerning this report, please contact Bob Levin, Director, Acquisition and Sourcing Management, on (202) 512-4841; Jack Brock, Managing Director, on (202) 512-4841; or Keith Rhodes, Chief Technologist, on (202) 512-6412. Major contributors to this report are listed in appendix VIII.

Appendix I: Disclosure of Flight Test’s Key Results and Limitations

Boeing and TRW disclosed the key results and limitations of an early sensor flight test, known as Integrated Flight Test 1A, to the Ground Based Interceptor Project Management Office. The contractors included some key results and limitations in written reports submitted soon after the June 1997 test, but others were not included in written reports until December 1997 or April 1998. However, according to project office and Nichols officials, all problems and limitations included in the written reports were communicated orally to the project management office in late August 1997. The deputy project office manager said his office did not report these verbal communications to others within the Program Office or the Department of Defense because the project office was the office within the Department responsible for the Boeing contract.

One problem that was included in initial reports to program officials was a malfunctioning cooling mechanism that did not lower the sensor’s temperature to the desired level. Boeing characterized the mechanism’s performance as somewhat below expectations but functioning well enough for the sensor’s operation. We hired experts to determine the extent to which the problem could affect the sensor’s performance. The experts found that the cooling problem degraded the sensor’s performance in a number of ways, but would not likely result in extreme performance degradation. The experts studied only how increased noise affected the sensor’s performance regarding comparative strengths of the target signals and the noise (signal to noise ratio). The experts did not evaluate discrimination performance, which is dependent on the measurement accuracy of the collected infrared signals. The experts’ findings are discussed in more detail later in this appendix.

The Test

Integrated Flight Test 1A, conducted in June 1997, was a test of the Boeing sensor—a highly sensitive, compact, infrared device, consisting of an array of silicon detectors, that is normally mounted on the exoatmospheric kill vehicle. However, in this test, a surrogate launch vehicle carried the sensor above the earth’s atmosphere to view a cluster of target objects that included a mock warhead and various decoys. When the sensor detected the target cluster, its silicon detectors began to make precise measurements of the infrared radiation emitted by the target objects. Over the tens of seconds that the target objects were within its field of view, the sensor continuously converted the infrared radiation into an electrical current, or signal, proportional to the amount of energy collected by the detectors. The sensor then digitized the signal (converted the signals into numerical values), completed a preliminary part of the planned signal processing, and formatted the signal so that it could be transmitted via a data link to a recorder on the ground. After the test, Boeing processed the signals further and formatted them so that TRW could input the signals into its discrimination software to assess its capability to distinguish the mock warhead from decoys. In post-flight ground testing, the software analyzed the processed data and identified the key characteristics, or features, of each signal. The software then compared the features it extracted to the expected features of various types of target objects. Based on this comparison, the software ranked each item according to its likelihood of being the mock warhead. TRW reported that the highest- ranked object was the mock warhead.

The primary objective of Integrated Flight Test 1A was to reduce risk in future flight tests. Specifically, the test was designed to determine if the sensor could operate in space; to examine the extent to which the sensor could detect small differences in infrared emissions; to determine if the sensor was accurately calibrated; and to collect target signature data for post-mission discrimination analysis. In addition, Boeing established quantitative requirements for the test. For example, the sensor was expected to acquire the target objects at a specified distance. According to a Nichols’ engineer, Boeing established these requirements to ensure that its exoatmospheric kill vehicle, when fully developed, could destroy a warhead with the single shot precision (expressed as a probability) required by the Ground Based Interceptor Project Management Office. The engineer said that in Integrated Flight Test 1A, Boeing planned to measure its sensor’s performance against these lower-level requirements so that Boeing engineers could determine which sensor elements, including the software, required further refinement. However, the engineer told us that because of the various sensor problems, of which the contractor and project office were aware, Boeing determined before the test that it would not use most of these requirements to judge the sensor’s performance. (Although Boeing did not judge the performance of its sensor against the requirements as it originally planned, Boeing did, in some cases, report the sensor’s performance in terms of these requirements. For a summary of selected test requirements and the sensor’s performance as reported by Boeing and TRW in their August 22, 1997, report, see app. V.)

Reported Key Results and Limitations

Table 1 provides details on the key results and limitations of Integrated Flight Test 1A that contractors disclosed in various written reports and briefing charts.

Although the contractors disclosed the key results and limitations of the flight test in written reports and in discussions, the written reports described the results using some terms that were not defined. For example, in their August 22, 1997, report, Boeing and TRW described Integrated Flight Test 1A as a “success” and the performance of the Boeing sensor as “excellent.” We asked the contractors to explain their use of these terms. We asked Boeing, for example, why it characterized its sensor’s performance as “excellent” when the sensor’s silicon detector array did not cool to the desired temperature, the sensor’s power supply created excess noise, and the sensor detected numerous false targets. Boeing said that even though the silicon detector array operated at temperatures 20 to 30 percent higher than desired, the sensor produced useful data. Officials said they knew of no other sensor that would be capable of producing any useful data under those conditions. Boeing officials went on to say that the sensor continuously produced usable, and, much of the time, excellent data in “real-time” during flight. In addition, officials said the sensor component responsible for suppressing background noise in the silicon detector array performed perfectly in space and the silicon detectors collected data in more than one wave band. Boeing concluded that the sensor’s performance allowed the test to meet all mission objectives.

Based on our review of the reports and discussions with officials in the Ground Based Interceptor Project Management Office and Nichols Research, we found that the contractors’ reports, in total, contained information for those officials to understand the key results and limitations of the test. However, because terms such as “success” and “excellent” are qualitative and subjective rather than quantitative and objective, we believe their use increases the likelihood that test results would be interpreted in different ways and could even be misunderstood. As part of our ongoing review of missile defense testing, we are examining the need for improvements in test reporting.

The August 13 Report

This report, sometimes referred to as the 45-day report, was a series of briefing charts. In it, contractors reported that Integrated Flight Test 1A achieved its principal objectives of reducing risks for subsequent flight tests, demonstrating the performance of the exoatmospheric kill vehicle’s sensor, and collecting target signature data. In addition, the report stated that TRW’s software successfully distinguished a mock warhead from accompanying decoys.

The August 22 Report

The August 22 report, known as the 60-day report, was a lengthy document that disclosed much more than the August 13 report. As discussed in more detail below, the report explained that some sensor abnormalities were observed during the test, that some signals collected from the target objects were degraded, that the launch vehicle carrying the sensor into space adversely affected the sensor’s ability to collect target signals, and that the sensor sometimes detected targets where there were none. These problems were all noted in the body of the report, but the report summary stated that review and analysis subsequent to the test confirmed the “excellent” performance and nominal operation of all sensor subsystems.

Some Sensor Abnormalities Were Observed During the Test

Boeing disclosed in the report that sensor abnormalities were observed during the test and that the sensor experienced a higher than expected false alarm rate. These abnormalities were (1) a cooling mechanism that did not bring the sensor’s silicon detectors to the intended operating temperature, (2) a power supply unit that created excess noise, and (3) software that did not function as designed because of the slow turnaround of the surrogate launch vehicle.

In the report’s summary, Boeing characterized the cooling mechanism’s performance as somewhat below expectations but functioning well enough for the sensor’s operation. In the body of the report, Boeing said that the fluctuations in temperature could lead to an apparent decrease in sensor performance. Additionally, Boeing engineers told us that the cooling mechanism’s failure to bring the silicon detector array to the required temperature caused the detectors to be noisy. Because the discrimination software identifies objects as a warhead or a decoy by comparing the features of a target’s signal with those it expects a warhead or decoy to display, a noisy signal may confuse the software. Boeing and TRW engineers said that they and program office officials were aware that there was a problem with the sensor’s cooling mechanism before the test was conducted. However, Boeing believed that the sensor would perform adequately at higher temperatures. According to contractor documents, the sensor did not perform as well as expected, and some target signals were degraded more than anticipated. Boeing disclosed in the report that sensor abnormalities were observed during the test and that the sensor experienced a higher than expected false alarm rate. These abnormalities were (1) a cooling mechanism that did not bring the sensor’s silicon detectors to the intended operating temperature, (2) a power supply unit that created excess noise, and (3) software that did not function as designed because of the slow turnaround of the surrogate launch vehicle.

In the report’s summary, Boeing characterized the cooling mechanism’s performance as somewhat below expectations but functioning well enough for the sensor’s operation. In the body of the report, Boeing said that the fluctuations in temperature could lead to an apparent decrease in sensor performance. Additionally, Boeing engineers told us that the cooling mechanism’s failure to bring the silicon detector array to the required temperature caused the detectors to be noisy. Because the discrimination software identifies objects as a warhead or a decoy by comparing the features of a target’s signal with those it expects a warhead or decoy to display, a noisy signal may confuse the software. Boeing and TRW engineers said that they and program office officials were aware that there was a problem with the sensor’s cooling mechanism before the test was conducted. However, Boeing believed that the sensor would perform adequately at higher temperatures. According to contractor documents, the sensor did not perform as well as expected, and some target signals were degraded more than anticipated.

Power Supply Creates Noise

The report also referred to a problem with the sensor’s power supply unit and its effect on target signals. An expert we hired to evaluate the sensor’s performance at higher than expected temperatures found that the power supply, rather than the temperature, was the primary cause of excess noise early in the sensor’s flight. Boeing engineers told us that they were aware that the power supply was noisy before the test, but, as shown by the test, it was worse than expected.

Payload Launch Vehicle Affected Software’s Ability to Remove Background Noise

The report explained that, as expected before the flight, the slow turnaround of the massive launch vehicle on which the sensor was mounted in Integrated Flight Test 1A caused the loss of some target signals. Engineers explained to us that the sensor would eventually be mounted on the lighter, more agile exoatmospheric kill vehicle, which would move back and forth to detect objects that did not initially appear in the sensor’s field of view. The engineers said that Boeing designed software that takes into account the kill vehicle’s normal motion to remove the background noise, but the software’s effectiveness depended on the fast movement of the kill vehicle. Boeing engineers told us that, because of the slow turnaround of the launch vehicle used in the test, the target signals detected during the turnaround were particularly noisy and the software sometimes removed not only the noise but the entire signal as well.

Sensor Sometimes Detected False Targets

The report mentioned that the sensor experienced more false alarms than expected. A false alarm is a detection of a target that is not there. According to the experts we hired, during Integrated Flight Test 1A, the Boeing sensor often mistakenly identified noise produced by the power supply as signals from actual target objects. In a fully automated discrimination software program, a high false alarm rate could overwhelm the tracking software. Because the post-flight processing tools were not fully developed at the time of the August 13 and August 22, 1997, reports, Boeing did not rely upon a fully automated tracking system when it processed the Integrated Flight Test 1A data. Instead, a Boeing engineer manually tracked the target objects. The contractors realized, and reported to the Ground Based Interceptor Project Management Office, that numerous false alarms could cause problems in future flight tests, and they identified software changes to reduce their occurrence.

December 11 Briefing

On December 11, 1997, Boeing and TRW briefed officials from the Ground Based Interceptor Project Management Office and one of its support contractors on various anomalies observed during Integrated Flight Test 1A. The contractors’ briefing charts explained the effect the anomalies could have on Integrated Flight Test 3, the first planned intercept test for the Boeing exoatmospheric kill vehicle, identified potential causes of the anomalies, and summarized the solutions to mitigate their effect. While some of the anomalies included in the December 11 briefing charts were referred to in the August 13 and August 22 reports, others were being reported in writing for the first time.

The anomalies referenced in the briefing charts included the sensor’s high false alarm rate, the silicon detector array’s higher-than-expected temperature, the software’s low confidence factor that it had correctly identified two target objects correctly, the sensor’s lower than expected probability of detection, and the software’s elevation in rank of one target object toward the end of the test. In addition, the charts showed that an in-flight attempt to calibrate the sensor was inconsistent. According to the charts, actions to prevent similar anomalies from occurring or impacting Integrated Flight Test 3 had in most cases already been implemented or were under way.

Contractors Report Further on False Alarms

The contractors again recognized that a large number of false alarms occurred during Integrated Flight Test 1A. According to the briefing charts, false alarms occurred during the slow turnarounds of the surrogate launch vehicle. Additionally, the contractors hypothesized that some false alarms resulted from space-ionizing events. By December 11, engineers had identified solutions to reduce the number of false alarms in future tests.

Briefing Charts Include Observations on Higher Detector Array Temperature

As they had in the August 22, 1997, report, the contractors recognized that the silicon detector array did not cool properly during Integrated Flight Test 1A. The contractors reported that higher silicon detector array temperatures could cause noisy signals that would adversely impact the detector array’s ability to estimate the infrared intensity of observed objects. Efforts to eliminate the impact of the higher temperatures, should they occur in future tests, were on-going at the time of the briefing.

Some Software Confidence Factors Lower Than Expected

Contractors observed that the confidence factor produced by the software was small for two target objects. The software equation that makes a determination as to how confident the software should be to identify a target object correctly, did not work properly for the large balloon or multiple-service launch vehicle. Corrections to the equation had been made by the time of the briefing.

Sensor’s Probability of Detection Is Lower Than Expected

The charts state that the Integrated Flight Test 1A sensor had a lower than anticipated probability of detection and a high false alarm rate. Because a part of the tracking, fusion, and discrimination software was designed for a sensor with a high probability of detection and a low false alarm rate, the software did not function optimally and needed revision. Changes to prevent this from happening in future flight tests were under way.

Software Increases the Rank of One Object Near Test’s End

The briefing charts showed that TRW’s software significantly increased the rank of one target object just before target objects began to leave the sensor’s field of view. Although a later Integrated Flight Test 1A report stated the mock warhead was consistently ranked as the most likely target, the charts show that if in Integrated Flight Test 3 the same object’s rank began to increase, the software could select the object as the intercept target. In the briefing charts, the contractors reported that TRW made a software change in the model that is used to generate reference data. When reference data was generated with the software change, the importance of the mock warhead increased, and it was selected as the target. Tests of the software change were in progress as of December 11.

In-Flight Calibration Was Inconsistent

The Boeing sensor measures the infrared emissions of target objects by converting the collected signals into intensity with the help of calibration data obtained from the sensor prior to flight. However, the sensor was not calibrated at the higher temperature range that was experienced during Integrated Flight Test 1A. To remedy the problem, the sensor viewed a star with known infrared emissions. The measurement of the star’s intensity was to have helped fill the gaps in calibration data that was essential to making accurate measurements of the target object signals. Boeing disclosed that the corrections based on the star calibration were inconsistent and did not improve the match of calculated and measured target signatures. Boeing subsequently told us that the star calibration corrections were effective for one of the wavelength bands, but not for another, and that the inconsistency referred to in the briefing charts was in how these bands behaved at temperatures above the intended operating range. Efforts to find and implement solutions were in progress.

April 1, 1998, Report

On April 1, 1998, Boeing submitted a revised addendum to replace an addendum that had accompanied the August 22, 1997, report. This revised addendum was prepared in response to comments and questions submitted by officials from the Ground Based Interceptor Project Management Office, Nichols Research Corporation, and the Defense Criminal Investigative Service concerning the August 22 report. In this addendum, the contractors referred in writing to three problems and limitations that had not been addressed in earlier written test reports or the December 11 briefing. Contractors noted that a gap-filling module, which was designed to replace noisy or missing signals, did not operate as designed. They also disclosed that TRW’s analysis of its discrimination software used target signals collected during a selected portion of the flight timeline and used a portion of the Integrated Flight Test 1A reference data that corresponded to this same timeline.

Gap-Filling Software Module Did Not Perform As Designed

The April 1 addendum reported that a gap-filling module that was designed to replace portions of noisy or missing target signals with expected signal values did not operate as designed. TRW officials told us that the module’s replacement values were too conservative and resulted in a poor match between collected signals and the signals the software expected the target objects to display.

Assessment Uses Selected Target Signals

The April 1, 1998, addendum also disclosed that the August 13 and August 22 reports, in which TRW conveyed that its software successfully distinguished the mock warhead from decoys, were based on tests of the software using about one-third of the target signals collected during Integrated Flight Test 1A. We talked to TRW officials who told us that Boeing provided several data sets to TRW, including the full data set. The officials said that Boeing provided target signals from the entire timeline to a TRW office that was developing a prototype version of the exoatmospheric kill vehicle’s tracking, fusion, and discrimination software, which was not yet operational. However, TRW representatives said that the test bed version of the software that TRW was using so that it could submit its analysis within 60 days of Integrated Flight Test 1A could not process the full data set. The officials said that shortly before the August 22 report was issued, the prototype version of the tracking, fusion, and discrimination software became functional and engineers were able to use the software to assess the expanded set of target signals. According to the officials, this assessment also resulted in the software’s selecting the mock warhead as the most likely target. In our review of the August 22 report, we found no analysis of the expanded set of target signals. The April 1, 1998, report, did include an analysis of a few additional seconds of data collected near the end of Integrated Flight Test 1A, but did not include an analysis of target signals collected at the beginning of the flight.

Most of the signals that were excluded from TRW's discrimination analysis were collected during the early part of the flight, when the sensor’s temperature was fluctuating. TRW told us that their software was designed to drop a target object’s track if the tracking portion of the software received no data updates for a defined period. This design feature was meant to reduce false tracks that the software might establish if the sensor detected targets where there were none. In Integrated Flight Test 1A, the fluctuation of the sensor’s temperature caused the loss of target signals. TRW engineers said that Boeing recognized that this interruption would cause TRW’s software to stop tracking all target objects and restart the discrimination process. Therefore, Boeing focused its efforts on processing those target signals that were collected after the sensor’s temperature stabilized and signals were collected continuously.

Some signals collected during the last seconds of the sensor’s flight were also excluded. The former TRW employee alleged that these latter signals were excluded because during this time a decoy was selected as the target. The Phase One Engineering Team cited one explanation for the exclusion of the signals. The team said that TRW stopped using data when objects began leaving the sensor’s field of view. Our review did not confirm this explanation. We reviewed the target intensities derived from the infrared frames covering that period and found that several seconds of data were excluded before objects began to leave the field of view. Boeing officials gave us another explanation. They said that target signals collected during the last few seconds of the flight were streaking, or blurring, because the sensor was viewing the target objects as it flew by them. Boeing told us that streaking would not occur in an intercept flight because the kill vehicle would have continued to approach the target objects. We could not confirm that the test of TRW’s discrimination software, as explained in the August 22, 1997, report, included all target signals that did not streak. We noted that the April 1, 1998, addendum shows that TRW analyzed several more seconds of target signals than is shown in the August 22, 1997, report. It was in these additional seconds that the software began to increase the rank of one decoy as it assessed which target object was most likely the mock warhead. However, the April 1, 1998, addendum also shows that even though the decoy’s rank increased the software continued to rank the mock warhead as the most likely target. But, because not all of the Integrated Flight Test 1A timeline was presented in the April 1 addendum, we could not determine whether any portion of the excluded timeline might have been useful data and if there were additional seconds of useful data whether a target object other than the mock warhead might have been ranked as the most likely target.

Corresponding Portions of Reference Data Excluded

The April 1 addendum also documented that portions of the reference data developed for Integrated Flight Test 1A were also excluded from the discrimination analysis. Nichols and project office officials told us the software identifies the various target objects by comparing the target signals collected from each object at a given point in their flight to the target signals it expects each object to display at that same point in the flight. Therefore, when target signals collected during a portion of the flight timeline are excluded, reference data developed for the same portion of the timeline must be excluded.

Information Provided Verbally to Project Office

Officials in the National Missile Defense Joint Program Office’s Ground Based Interceptor Project Management Office and Nichols Research told us that soon after Integrated Flight Test 1A the contractors orally disclosed all of the problems and limitations cited in the December 11, 1997, briefing and the April 1, 1998, addendum. Contractors made these disclosures to project office and Nichols Research officials during meetings that were held to review Integrated Flight Test 1A results sometime in late August 1997. The project office and contractors could not, however, provide us with documentation of these disclosures.

The current Ground Based Interceptor Project Management Office deputy manager said that the problems that contractors discussed with his office were not specifically communicated to others within the Department of Defense because his office was the office within the Department responsible for the Boeing contract. The project office’s assessment was that these problems did not compromise the reported success of the mission, were similar in nature to problems normally found in initial developmental tests, and could be easily corrected.

Effect of Cooling Failure on Sensor’s Performance

Because we questioned whether Boeing’s sensor could collect any usable target signals if the silicon detector array was not cooled to the desired temperature, we hired sensor experts at Utah State University’s Space Dynamics Laboratory to determine the extent to which the sub-optimal cooling degraded the sensor’s performance. These experts concluded that the higher temperature of the silicon detectors degraded the sensor’s performance in a number of ways, but did not result in extreme degradation. For example, the experts said the higher temperature reduced by approximately 7 percent the distance at which the sensor could detect targets. The experts also said that the rapid temperature fluctuation at the beginning and at the end of data acquisition contributed to the number of times that the sensor detected a false target. However, the experts said the major cause of the false alarms was the power supply noise that contaminated the electrical signals generated by the sensor in response to the infrared energy. When the sensor signals were processed after Integrated Flight Test 1A, the noise appeared as objects, but they were actually false alarms.

Additionally, the experts said that the precision with which the sensor could estimate the infrared energy emanating from an object based on the electrical signal produced by the energy was especially degraded in one of the sensor’s two infrared wave bands. In their report, the experts said that the Massachusetts Institute of Technology’s Lincoln Laboratory analyzed the precision with which the Boeing sensor could measure infrared radiation and found large errors in measurement accuracy. The Utah State experts said that their determination that the sensor’s measurement capability was degraded in one infrared wave band might partially explain the errors found by Lincoln Laboratory.

Although Boeing’s sensor did not cool to the desired temperature during Integrated Flight Test 1A, the experts found that an obstruction in gas flow rather than the sensor’s design was at fault. These experts said the sensor’s cooling mechanism was properly designed and Boeing’s sensor design was sound.

Appendix II: Project Office Reliance on Various Sources for Contractor Oversight

The Ground Based Interceptor Project Management Office used several sources to monitor the contractors’ technical performance, but oversight activities were limited by the ongoing exoatmospheric kill vehicle contract competition between Boeing and Raytheon. Specifically, the project office relied on an engineer and a System Engineering and Technical Analysis contractor, Nichols Research Corporation, to provide insight into Boeing’s work. The project office also relied on Boeing to oversee TRW’s performance.

The deputy manager of the Ground Based Interceptor Project Management Office told us that competition between Boeing and Raytheon limited oversight to some extent. He said that because of the ongoing competition, the project office monitored the two contractors’ progress but was careful not to affect the competition by assisting one contractor more than the other. The project office primarily ensured that the contractors abided by their contractual requirements. The project office deputy manager told us that his office relied on “insight” into the contractors’ work rather than oversight of that work.

The project office gained insight by placing an engineer on-site at Boeing and tasking Nichols Research Corporation to attend technical meetings, assess test reports, and, in some cases, evaluate Boeing’s and TRW’s technologies. The on-site engineer was responsible for observing the performance of Boeing and TRW and relaying any problems back to the project office. He did not have authority to provide technical direction to the contractors. According to the Ground Based Interceptor Project Management Office deputy manager, Nichols essentially “looked over the shoulder” of Boeing and TRW. We observed evidence of Nichols’ insight in memorandums that Nichols’ engineers submitted to the project office suggesting questions that should be asked of the contractors, memorandums documenting engineer’s comments on various contractor reports, and trip reports recorded by the engineers after various technical meetings.

Boeing said its oversight of TRW’s work complied with contract requirements. The contract between the Department of Defense and Boeing required Boeing to declare that “to the best of its knowledge and belief, the technical data delivered is complete, accurate, and complies with all requirements of the contract.” With regard to Integrated Flight Test 1A, Boeing officials said that they complied with this provision by selecting a qualified subcontractor, TRW, to develop the discrimination concepts, software, and system design in support of the flight tests, and by holding weekly team meetings with subcontractor and project office officials. Boeing officials stated that they were not required to verify the validity of their subcontractor’s flight test analyses; rather, they were only required to verify that the analyses seemed reasonable. According to Boeing officials, both they and the project office shared the belief that TRW possessed the necessary technical expertise in threat phenomenology modeling, discrimination, and target tracking, and both relied on TRW’s expertise.

Appendix III: Reduced Test Complexity

National Missile Defense Joint Program Office officials said that they reduced the number of decoys planned for intercept flight tests in response to a recommendation by an independent panel, known as the Welch Panel. The panel, established to reduce risk in ballistic missile defense flight test programs, viewed a successful hit-to-kill engagement as a difficult task that should not be further complicated in early tests by the addition of decoys. In contemplating the panel’s advice, the program manager discussed various target options with other program officials and the contractors competing to develop and produce the system’s exoatmospheric kill vehicle. The officials disagreed on the number of decoys that should be deployed in the first intercept flight tests. Some recommended using the same target set deployed in Integrated Flight Test 1A and 2, while others wanted to eliminate some decoys. After considering the differing viewpoints, the program manager decided to deploy only one decoy—a large balloon—in early intercept tests.

Decoys in Early Intercept Tests

As flight tests began in 1997, the National Missile Defense Joint Program Office was planning two sensor tests—Integrated Flight Test 1A and 2— and 19 intercept tests. The primary objective of the sensor flight tests was to reduce risk in future flight tests. Specifically the tests were designed to determine if the sensor could operate in space; to examine the extent to which the sensor could detect small differences in infrared emissions; to determine if the sensor was accurately calibrated; and to collect target signature data for post-mission discrimination analysis.

Initially, the next two flight tests were to demonstrate the ability of the competing kill vehicles to intercept a mock warhead. Integrated Flight Test 3 was to test the Boeing kill vehicle and Integrated Flight Test 4 was to test the Raytheon kill vehicle. Table 1 shows the number of target objects deployed in the two sensor tests, the number of objects originally planned to be deployed in the first two intercept attempts, and the number of objects actually deployed in the intercept attempts.

By the time Integrated Flight Tests 3 and 4 were actually conducted, Boeing had become the National Missile Defense Lead System Integrator and had selected Raytheon’s exoatmospheric kill vehicle for use in the National Missile Defense system. Boeing conducted Integrated Flight Test 3 (in October 1999) and Integrated Flight Test 4 (in January 2000) with the Raytheon kill vehicle. However, both of these flight tests used only the mock warhead and one large balloon, rather than the nine objects originally planned. Integrated Flight Test 5 (flown in July 2000) also used only the mock warhead and one large balloon.

Program officials told us that the National Missile Defense Program Manager decided to reduce the number of decoys used in Integrated Flight Tests 3, 4, and 5, based on the findings of an expert panel. This panel, known as the Welch Panel, reviewed the flight test programs of several Ballistic Missile Defense Organization programs, including the National Missile Defense program. The resulting report,which was released shortly after Integrated Flight Test 2, found that U.S. ballistic missile defense programs, including the National Missile Defense program, had not yet demonstrated that they could reliably intercept a ballistic missile warhead using the technology known as “hit-to-kill.” Numerous failures had occurred for several of these programs and the Welch Panel concluded that the National Missile Defense program (as well as other programs using "hit-to-kill" technology) needed to demonstrate that it could reliably intercept simple targets before it attempted to demonstrate that it could hit a target accompanied by decoys. The panel reported again 1 month after Integrated Flight Test 3 and came to the same conclusion.

The Director of the Ballistic Missile Defense Organization testified at a congressional hearing that the Welch Panel advocated removing all decoys from the initial flight tests, but that the Ballistic Missile Defense Organization opted to include a limited discrimination requirement with the use of one decoy. Nevertheless, he said that the primary purpose of the tests was to demonstrate the system’s “hit-to-kill” capability.

Opinions on Decoys

Program officials said there was disagreement within the Joint Program Office and among the key contractors as to how many targets to use in the early intercept flight tests. Raytheon and one high-ranking program official wanted Integrated Flight Tests 3, 4, and 5 to include target objects identical to those deployed in the sensor flight tests. Boeing and other program officials wanted to deploy fewer target objects. After considering all options, the Joint Program Office decided to deploy a mock warhead and one decoy—a large balloon.

Raytheon officials told us that they discussed the number of objects to be deployed in Integrated Flight Tests 3, 4, and 5 with program officials and recommended using the same target set as deployed in Integrated Flight Tests 1A and 2. Raytheon believed that this approach would be less risky because it would not require revisions to be made to the kill vehicle’s software. Raytheon and program officials told us that Raytheon was confident that it could successfully identify and intercept the mock warhead even with this larger target set.

One high-ranking program official said that she objected to reducing the number of decoys used in Integrated Flight Test 3, because there was a need to more completely test the system. However, other program officials lobbied for a smaller target set. One program official said that his position was based on the Welch Panel’s findings and on the fact that the program office was not concerned at that time about discrimination capability. He added that the National Missile Defense program was responding to the threat of “nations of concern,” which could only develop simple targets, rather than major nuclear powers, which were more likely to be able to deploy decoys.

The Boeing/TRW team also wanted to reduce the number of decoys used in the first intercept tests. In a December 1997 study, the companies recommended that Integrated Flight Test 3 be conducted with a total of four objects—the mock warhead, the two small balloons, and the large balloon. (The multi-service launch system was not counted as one of the objects.) The study cited concerns about the inclusion of decoys that were not part of the initially expected threat and about the need to reduce risk. Boeing said that the risk increased significantly that the exoatmospheric kill vehicle would not intercept the mock warhead if the target objects did not deploy from the test missile as expected.

According to Boeing/TRW, as the types and number of target objects increased, the potential risk that the target objects would be different in some way from what was expected also increased. Specifically, the December 1997 study noted that the medium balloons had been in inventory for some time and had not deployed as expected in other tests, including Integrated Flight Test 1A. In that test, one medium balloon only partially inflated and was not positioned within the target cluster as expected. The study also found that the medium rigid light replicas are the easiest to misdeploy and the small canisterized light replica moved differently than expected during Integrated Flight Test 1A.

Appendix IV: Phase One Engineering Team’s Evaluation of TRW’s Software

In 1998, the National Missile Defense Joint Program Office asked the Phase One Engineering Team to conduct an assessment, using available data, of TRW’s discrimination software even though Nichols Research Corporation had already concluded that it met the requirements established by Boeing. The program office asked for the second evaluation because the Defense Criminal Investigative Service lead investigator was concerned about the ability of Nichols to provide a truly objective evaluation.

The Phase One Engineering Team developed a methodology to (1) determine if TRW’s software was consistent with scientific, mathematical, and engineering principles; (2) determine whether TRW accurately reported that its software successfully discriminated a mock warhead from decoys using data collected during Integrated Flight Test 1A; and (3) predict the performance of TRW’s basic discrimination software against Integrated Flight Test 3 scenarios. The key results of the team’s evaluation were that the software was well designed; the contractors accurately reported the results of Integrated Flight Test 1A; and the software would likely perform successfully in Integrated Flight Test 3. The primary limitation was that the team used Boeing- and TRW- processed target data and TRW-developed reference data in determining the accuracy of TRW reports for Integrated Flight Test 1A.

Phase One Engineering Team’s Methodology

The team began its work by assuring itself that TRW’s discrimination software was based on sound scientific, engineering, and mathematical principles and that those principles had been correctly implemented. It did this primarily by studying technical documents provided by the contractors and the program office. Next, the team began to look at the software’s performance using Integrated Flight Test 1A data. The team studied TRW’s August 13 and August 22, 1997, test reports to learn more about discrepancies that the Defense Criminal Investigative Service said it found in these reports. Team members also received briefings from the Defense Criminal Investigative Service, Boeing, TRW, and Nichols Research Corporation.

Team members told us that they did not replicate TRW’s software in total. Instead, the team emulated critical functions of TRW’s discrimination software and tested those functions using data collected during Integrated Flight Test 1A. To test the ability of TRW’s software to extract the features of each target object’s signal, the team designed a software routine that mirrored TRW’s feature-extraction design. The team received Integrated Flight Test 1A target signals that had been processed by Boeing and then further processed by TRW. These signals represented about one-third of the collected signals. Team members input the TRW-supplied target signals into the team’s feature-extraction software routine and extracted two features from each target signal. The team then compared the extracted features to TRW’s reports on these same features and concluded that TRW’s software-extraction process worked as reported by TRW. Next, the team acquired the results of 200 of the 1,000 simulations that TRW had run to determine the features that target objects deployed in Integrated Flight Test 1A would likely display.Using these results, team members developed reference data that the software could compare to the features extracted from Integrated Flight Test 1A target signals. Finally, the team wrote software that ranked the different observed target objects in terms of the probability that each was the mock warhead. The results produced by the team’s software were then compared to TRW’s reported results.

The team did not perform any additional analysis to predict the performance of the Boeing sensor and its software in Integrated Flight Test 3. Instead, the team used the knowledge that it gained from its assessment of the software’s performance using Integrated Flight Test 1A data to estimate the software’s performance in the third flight test.

The Phase One Engineering Team’s Key Results

In its report published on January 25, 1999, the Phase One Engineering Team reported that even though it noted some weaknesses, TRW’s discrimination software was well designed and worked properly, with only some refinement or redesign needed to increase the robustness of the discrimination function. In addition, the team reported that its test of the software using data from Integrated Flight Test 1A produced essentially the same results as those reported by TRW. The team also predicted that the Boeing sensor and its software would perform well in Integrated Flight Test 3 if target objects deployed as expected.

Weaknesses in TRW’s Software

The team's assessment identified some software weaknesses. First, the team reported that TRW’s use of a software module to replace missing or noisy target signals was not effective and could actually hurt rather than help the performance of the discrimination software. Second, the Phase One Engineering Team pointed out that while TRW proposed extracting several features from each target-object signal, only a few of the features could be used.

The Phase One Engineering Team also reported that it found TRW’s software to be fragile because the software was unlikely to operate effectively if the reference data—or expected target signals—did not closely match the signals that the sensor collected from deployed target objects. The team warned that the software’s performance could degrade significantly if incorrect reference data were loaded into the software. Because developing good reference data is dependent upon having the correct information about target characteristics, sensor-to-target geometry, and engagement timelines, unexpected targets might challenge the software. The team suggested that very good knowledge about all of these parameters might not always be available.

Accuracy of Contractors’ Integrated Flight Test 1A Reports

The Phase One Engineering Team reported that the results of its evaluation using Integrated Flight Test 1A data supported TRW’s claim that in post-flight analysis its software accurately distinguished a mock warhead from decoys. The report stated that TRW explained why there were differences in the discrimination analysis included in the August 13, 1997, Integrated Flight Test 1A test report and that included in the August 22, 1997, report. According to the report, one difference was that TRW mislabeled a chart in the August 22 report. Another difference was that the August 22 discrimination analysis was based on target signals collected over a shorter period of time (see app. I for more information regarding TRW’s explanation of report differences). Team members said that they found TRW’s explanations reasonable.

Predicted Success in Integrated Flight Test 3

The Phase One Engineering Team predicted that if the targets deployed in Integrated Flight Test 3 performed as expected, TRW's discrimination software would successfully identify the warhead as the target. The team observed that the targets proposed for the flight test had been viewed by Boeing’s sensor in Integrated Flight Test 1A and that target-object features collected by the sensor would be extremely useful in constructing reference data for the third flight test. The team concluded that given this prior knowledge, TRW’s discrimination software would successfully select the correct target even in the most stressing Integrated Flight Test 3 scenario being considered, if all target objects deployed as expected. However, the team expressed concern about the software’s capabilities if objects deployed differently, as had happened in previous flight tests.

Limitations of the Team’s Evaluation

The Phase One Engineering Team’s conclusion that TRW’s software successfully discriminated is based on the assumption that Boeing’s and TRW’s input data were accurate. The team did not process the raw data collected by the sensor’s silicon detector array during Integrated Flight Test 1A or develop their own reference data by running hundreds of simulations. Instead, the team used target signature data extracted by Boeing and TRW and developed reference data from a portion of the simulations that TRW ran for its own post-flight analysis. Because it did not process the raw data from Integrated Flight Test 1A or develop its own reference data, the team cannot be said to have definitively proved or disproved TRW’s claim that its software successfully discriminated the mock warhead from decoys using data collected from Integrated Flight Test 1A. A team member told us its use of Boeing- and TRW-provided data was appropriate because the former TRW employee had not alleged that the contractors tampered with the raw test data or used inappropriate reference data.

Appendix V: Boeing Integrated Flight Test 1A Requirements and Actual Performance as Reported by Boeing and TRW

The table below includes selected requirements that Boeing established before the flight test to evaluate sensor performance and the actual sensor performance characteristics that Boeing and TRW discussed in the August 22 report.

Appendix VI: Scope and Methodology

We determined whether Boeing and TRW disclosed key results and limitations of Integrated Flight Test 1A to the National Missile Defense Joint Program Office by examining test reports submitted to the program office on August 13, 1997, August 22, 1997, and April 1, 1998, and by examining the December 11, 1997, briefing charts. We also held discussions with and examined various reports and documents prepared by Boeing North American, Anaheim, California; TRW Inc., Redondo Beach, California; the Raytheon Company, Tucson, Arizona; Nichols Research Corporation, Huntsville, Alabama; the Phase One Engineering Team, Washington, D.C.; the Massachusetts Institute of Technology/Lincoln Laboratory, Lexington, Massachusetts; the National Missile Defense Joint Program Office, Arlington, Virginia, and Huntsville, Alabama; the Office of the Director, Operational Test and Evaluation, Washington D.C.; the U.S. Army Space and Missile Defense Command, Huntsville, Alabama; the Defense Criminal Investigative Service, Mission Viejo, California, and Arlington, Virginia; and the Institute for Defense Analyses, Alexandria, Virginia.

We held discussions with and examined documents prepared by Dr. Theodore Postol, Massachusetts Institute of Technology, Cambridge, Massachusetts; Dr. Nira Schwartz, Torrance, California; Mr. Roy Danchick, Santa Monica, California; and Dr. Michael Munn, Benson, Arizona.

In addition, we hired the Utah State University Space Dynamics Laboratory, Logan, Utah, to examine the performance of the Boeing sensor because we needed to determine the effect the higher operating temperature had on the sensor’s performance. We did not replicate TRW’s assessment of its software using target signals that the Boeing sensor collected during the test. This would have required us to make engineers and computers available to verify TRW’s software, format raw target signals for input into the software, develop reference data, and run the data through the software. We did not have these resources available, and we, therefore, cannot attest to the accuracy of TRW’s discrimination claims.

We also examined the methodologies, findings, and limitations of the review conducted by the Phase One Engineering Team of TRW’s discrimination software. To accomplish this task, we analyzed the Phase One Engineering Team’s “Independent Review of TRW EKV Discrimination Techniques” dated January 1999. In addition, we held discussions with Phase One Engineering Team members, officials from the National Missile Defense Joint Program Office, and contractor officials.

We did not replicate the evaluations conducted by the Phase One Engineering Team and cannot attest to the accuracy of their reports.

We reviewed the decision by the National Missile Defense Joint Program Office to reduce the complexity of later flight tests by comparing actual flight test information with information in prior plans and by discussing these differences with program and contractor officials. We held discussions with and examined documents prepared by the National Missile Defense Joint Program Office, the Institute for Defense Analyses, Boeing North American, and the Raytheon Company.

Our work was conducted from August 2000 through February 2002 according to generally accepted government auditing standards. The length of time the National Missile Defense Joint Program Office required to release documents to us significantly slowed our review. For example, the Program Office required approximately 4 months to release key documents such as the Phase One Engineering Team’s response to the professor’s allegations. We requested these and other documents on September 14, 2000, and received them on January 9, 2001.

Appendix VIII: Major Contributors

General Counsel