Missile Defense: Knowledge-Based Decision Making Needed to Reduce Risks in Developing Airborne Laser
Released: 2002-07-16T00:00:00

Letter

Background

The effort to develop the Airborne Laser is based on over 25 years of scientific development in the Departments of Defense and Energy. It evolved primarily from Airborne Laser laboratory research to develop applications for high-energy lasers. This research culminated in a demonstration that showed that a low-power, short-range laser was capable of destroying a short-range air-to-air missile. Although this demonstration was considered militarily insignificant because of the laser’s low power and short range, it did succeed in identifying technologies that were necessary for the development of an operational Airborne Laser system. The research showed that an operational system would need optics that could compensate for the atmospheric turbulence that weakens and scatters a laser beam, optical devices that could withstand the heat produced by a high-energy laser without the added weight of water-cooling devices, and a new chemical laser with higher energy levels that would produce a stronger laser beam.

In 1996, the Air Force launched the Airborne Laser program to develop a defensive system that could destroy enemy missiles from a distance of several hundred kilometers. Engineers determined that if they were to meet this requirement, the system would need a 14-module oxygen iodine laser. They also determined that the system would need a beam control/fire control assembly that could (1) safely move the laser beam through the aircraft, (2) shape the beam so that it was not scattered or weakened by the atmosphere, and (3) hold the beam on target, despite the movement of the aircraft. In addition, engineers determined that the system would need a battle management and control system capable of planning and executing an engagement.

The Air Force planned to have the science and technology community develop extensive knowledge about the laser and beam control/fire control technologies before it launched an Airborne Laser acquisition program. However, according to the retired manager of the science and technology project, the budgets for technology efforts were limited, and the science and technology community could not fund the technology maturation effort. The Air Force knew that a program office was more likely to command the large budget needed to fully mature technologies, so it launched an acquisition program and assigned the program manager responsibility for both technology and product development. The program manager planned to demonstrate critical Airborne Laser technologies by first building a six-module version of the oxygen iodine laser, installing it along with other system components aboard a Boeing 747 aircraft (see fig. 1), and testing the capability of this scaled system in system-level flight tests. The tests would conclude in 2003 with an attempt to shoot down a short-range ballistic missile target at a distance of 100 kilometers. If this final test were successful, the Airborne Laser would have moved into product development.

Original Cost and Schedule Goals Are Based on Inadequate Knowledge

The Air Force launched the Airborne Laser acquisition program and identified cost and schedule goals before officials had the knowledge to make realistic projections. In 1996, when the program was launched, Department of Defense regulation 5000.2 required, and still requires today, that when a military service initiates a major acquisition program, it must establish cost and schedule goals. However, the Air Force could not make realistic estimates when it began the program because it had no way of knowing how much engineering effort would be needed to complete the development of technology critical to the system. Even today, some critical technologies that the system’s design depends upon remain immature, making it very difficult for analysts to determine how long it will take and how much it will cost to develop and produce the system.

Technologies Were Immature at Program Launch

At the time the Airborne Laser program was launched, the laser and beam control/fire control technologies needed to develop the Airborne Laser system was immature. The Department of Defense’s science and technology community was actively researching and developing the laser and had produced a weak beam in a laboratory setting—but this major component had not reached the level of maturity needed to proceed into product development. The technology necessary to develop the beam control/fire control was even less advanced. Most of the scientists’ work was limited to analytical studies, wherein a few tests of laboratory hardware were linked together to work somewhat like the intended component.

Because technology development is a process of discovery, the Air Force soon learned that there were too many unknowns regarding the development of Airborne Laser technology to make good cost and schedule estimates. As the technology development progressed, unanticipated technical challenges affected the program’s cost and schedule. Department of Defense analysts reported that the Airborne Laser program experienced cost and schedule growth because the program and its contractors underestimated the complexity of (1) designing laser components, (2) the system’s engineering analysis and design effort, and (3) engineering the system to fit on board the aircraft. As system development progressed and the Air Force gained a better understanding of the technical complexity of the system, the Air Force increased its cost and schedule estimates.

Some Critical Technologies Remain Immature

The Air Force has made some progress in developing the Airborne Laser’s critical technologies, but many remain immature. We asked the Airborne Laser program office to determine the technologies most critical to the Airborne Laser system and to use technology readiness levels to assess the maturity of each. The officials determined that if the Airborne Laser is to meet the requirements established by the war fighters, then engineers must mature technologies in six areas, all of which are needed to successfully design the system. These technologies are devices that stabilize the laser system aboard the aircraft so that the beam can be maintained firmly on the target, optics—mirrors and windows—that focus and control the laser beam and allow it to pass safely through the aircraft, optical coatings that enhance the optics’ ability to pass laser energy through the system and to reflect the laser energy, hardware that works in tandem with computer software to actively track devices that measure atmospheric turbulence and compensate for it so that it does not scatter or weaken the laser beam, and safety systems that automatically shut down the high energy laser in the event of an emergency.

At our request, the program office also assessed the maturity of the oxygen iodine laser.

As figure 2 shows, program officials assessed the optical coatings at level five and the safety systems, atmospheric compensation, and target- tracking components at level six. At technology readiness level five, the technology being tested is incorporated into hardware whose form and fit are coming closer to that needed for an operational component and integrated with reasonable realistic supporting elements so that the technology can be tested in a simulated environment. At level six, the technology is incorporated into a prototype and tested in a high-fidelity laboratory environment or in a simulated operational environment. The program officials identified the optics and stabilizing devices as the least mature—at level four. At this level, engineers have shown that a technology is technically feasible but have not shown whether the technology will have the form, fit, or function required in the operational system. We agreed with all but one of the program officials’ assessments for these technologies.

Our one disagreement centered on the maturity of the laser component of the system. While the program office assessed it at a technology readiness level of six, we consider the laser technology to be at level four because tests have been conducted only for a one-module laser in a controlled laboratory environment using surrogate components. For example, the tests used a stable laser resonator, rather than the unstable resonator that will be used in system-level flight tests. We also found that during tests of the one-module laser, the resonator was operating in multimode rather than single-mode. The resonator in the operational system will operate in single mode. Furthermore, the chemical storage and delivery subcomponents used in these tests were not representative of those that will be incorporated into the system’s design. According to program office officials, conducting a more realistic test would have cost time and money that were not available.

Documents summarizing the tests of the one-module laser stated that the tests were successful in reducing the technical risks associated with the one-module system but that a new set of technical risks linked with developing a multimodule system must still be addressed during testing of the six-module system. In our opinion, the program office will demonstrate the laser technology in a relative environment (technology readiness level six) when the six-module system is integrated and successfully tested at full power within the high-fidelity laboratory environment of the Airborne Laser Systems Integration Laboratory, currently under construction at Edwards Air Force Base, California. According to the program office, this type of demonstration will not occur until February 2003.

New Strategy Incorporates Some Knowledge-Based Practices, but Additional Practice Would Reduce Program Risk

The Missile Defense Agency’s new strategy for developing the Airborne Laser incorporates some of the knowledge-based practices that characterize successful programs, but the agency would benefit from adopting another that would add greater discipline to its acquisition process. The new strategy allows more flexibility in setting requirements, makes time and facilities available to mature and test the critical technologies, and collects information needed to match the war fighters’ requirements to demonstrated technology. However, the agency has not established decision points with associated knowledge-based criteria for moving forward from (1) technology development to system integration, (2) system integration to system demonstration, and (3) system demonstration to production. At each of these points, the agency would stop to assess its knowledge and decide whether investment in the program’s next phase is warranted.

New Strategy Introduces Knowledge-Based Practices

The first new practice allows the Missile Defense Agency to refine requirements on the basis of the results of system engineering. The Department of Defense ordinarily faces significant hurdles in matching requirements to resources. The fundamental problem is twofold. First, under the department’s traditional process, requirements must be set before a program can be approved and a program must be approved before the product developer conducts systems engineering. Second, the competition for funding encourages requirements that will make the desired weapon system stand out from others. Consequently, many of the department’s product development programs include unrealistic requirements set by the user before the product developer has conducted the system engineering necessary to identify the time, technology, and money necessary to develop a product capable of meeting requirements.

A second practice that is likely to improve the Airborne Laser’s development is making the time and facilities available to mature and test critical technologies. To implement this practice, the agency increased the time available to test the six-module laser system and is building a new test facility. Instead of following the Air Force’s plan to complete system- level flight tests of the six-module system in the last quarter of fiscal year 2003, the agency has delayed the demonstration to the first quarter of fiscal year 2005. This delay will allow additional time to learn from and correct problems discovered during system-level tests that are scheduled to begin in the last quarter of fiscal year 2003 and end with the fiscal year 2005 demonstration. In addition, the agency plans to increase the Airborne Laser’s ground-testing capability by awarding a contract in 2003 for what the agency is calling an “iron bird,” which is essentially an aircraft hull with installed laser equipment. The “iron bird” is expected to allow testing of a fully integrated Airborne Laser system on the ground so that technologies for future blocks can be evaluated before being installed in an aircraft.

The information gained from testing informs the requirements process. Because testing allows developers to gauge the progress being made in translating an idea into a weapon system, it enables the developer to make a more informed decision as to whether a technology is ready to be incorporated into a system’s design. With this knowledge, the developer can determine whether the technology is so important to the system’s design that additional time and money should be spent to mature the technology or whether the system’s initial performance requirements should be reduced.

A third practice that the agency plans to adopt is matching requirements to available technology. According to the Missile Defense Agency’s Technical Director, the agency defines the war fighters’ requirement as a system that has the capability to destroy some threat ballistic missiles during their boost phase at a range representative of an operational scenario. The Technical Director told us that the agency will attain the knowledge to determine if it has the technology in-hand to meet this requirement by examining each block’s capabilities during simulated and system-level flight test and comparing those capabilities with data derived from intelligence sources on the likely launch points and types of missiles that the system could encounter. Our previous work with successful development programs shows that once the technology is in-hand to meet the customer’s requirements, the developer can make more accurate initial estimates of the cost and time needed to develop and produce an operational system.

Successful Developers Recognize Need for Knowledge-Based Decision Points

Successful developers have instilled discipline in their acquisition processes by requiring that certain criteria for attaining knowledge are met as an acquisition program moves forward. (See fig. 3.) They recognize that the focus and cost of activities change over time and that less rework is required if all activities with the same focus are completed before beginning other activities.

In successful development programs, decisions are made when the knowledge is available to support those decisions. The first decision point, or knowledge point, occurs when the focus of a developer’s activities changes from technology development to system integration—the first phase of product development. The criterion for deciding to move forward is having the knowledge to match requirements and available resources (time, technology, and funds). The second knowledge point occurs between system integration and system demonstration when the developer has successfully integrated subsystems and components into a stable design that not only meets the customer’s performance requirements but also is optimized for reproducibility, maintainability, and reliability. The decision criterion used here is usually having completed about 90 percent of the engineering drawings. The third knowledge point separates system demonstration from production. The decision to invest in production is generally based on a determination that the product performs as required during testing and that the manufacturing processes will produce a product within cost, schedule, and quality targets.

The cost of a program’s activities increases as it moves closer to production. In commercial acquisitions, product development is typically much more costly than technology development. During technology development, small teams of technologists work to perfect the application of scientific knowledge to a practical problem. As product development begins, developers begin to make larger investments in human capital, bringing on a large engineering force to design and manufacture the product. In addition, product development requires significant investments in facilities and materials. These investments increase continuously as the product approaches the point of manufacture. In fact, industry experts estimate that identifying and resolving a problem during product development can cost 10 times more than correcting that problem during technology development and that correcting the problem during manufacturing is even more costly.

Knowledge-Based Decisions Missing from Airborne Laser Strategy

We examined the Airborne Laser’s acquisition strategy and determined that it does not include decision points at which officials would use knowledge-based criteria to determine if the program is ready to move from technology development to system integration, system integration to system demonstration, and system demonstration to production. We found that the agency’s process has three phases: development, transition, and production.

Development includes all developmental activities and system-level demonstrations of military utility.

Transition will involve preparation of the operational requirements document by the appropriate armed service and conducting operational testing.

Production will involve producing and fielding the final weapon system.

The agency’s strategy also calls for developing the Airborne Laser incrementally, rather than trying to initially develop a system with all desired capabilities. In the near term, the agency plans to complete the six- module laser system aircraft, now known as block 2004, and use it to demonstrate critical Airborne Laser technologies. Beginning in March 2003, the agency intends to begin developing another demonstration aircraft, known as block 2008, which will incorporate new capabilities and technologies. The Airborne Laser program manager told us that blocks 2004 and 2008 are primarily test assets for the purpose of technology demonstration. While some of the block 2008 activities are focused on improving subsystems and components, such as reducing the weight of laser components and improving optics, other activities are focused on the integration of these pieces into a block 2008 design.

The agency expects to develop subsequent blocks, or system configurations to introduce additional capabilities. If system-level tests show that any one of these configurations performs at a level that merits fielding, the Air Force will prepare a requirements document based on the configuration’s demonstrated capabilities and make plans for operational testing and production. This “baseline” capability would be improved in subsequent blocks as more advanced technology becomes available and as the threat warrants.

We did not find that the agency’s strategy includes a disciplined process that separates technology development, system integration, system demonstration, and production with decision points supported by knowledge-based criteria. Instead, the agency has put in place a decision point for moving from the development to the transition phase. According to the agency’s strategy, when the agency determines that it has the technology in-hand to produce a system that merits fielding, it will begin to transition the system over to the appropriate military service. Also, at the end of the transition phase, a system would enter the formal Department of Defense acquisition process at Milestone C—the point at which the decision is made to enter low rate initial production. We did not find, however, an established set of decision points with associated criteria that would enable the agency to make a knowledge-based decision on whether to invest in system integration and, subsequently, system demonstration and production. That is, even though the agency might know that it has the technology in-hand to develop a useful military capability, it has not established a first decision point where it would determine the cost and time needed to move the program forward and whether the program should proceed into a system integration phase during which the design would be matured and optimized for reproducibility, maintainability, and reliability. Neither does the agency’s strategy include a second decision point that would allow agency officials to use the knowledge they have attained regarding the design’s maturity to determine whether to invest further to demonstrate that the system meets requirements and that manufacturing processes are in place to repeatedly produce a quality product. Only after the agency successfully moves the program through all of these decision points and successfully demonstrates the system’s capabilities and manufacturing processes would the agency’s production decision be fully knowledge based. Without this disciplined process, the agency would be accepting greater cost and schedule risks and is much less likely to realize the full potential benefits of its new approach to developing missile defense systems.

Conclusion

The revolutionary nature of missile defense weapon systems demands cutting-edge technology. Although there is no one approach that ensures that a developer can deal successfully with the unknowns inherent in developing a product from such technology, the knowledge-based process has proven to yield good results within cost and schedule estimates. The Missile Defense Agency has implemented practices that are part of the knowledge-based approach, and these practices are likely to improve the agency’s ability to gather the knowledge it needs to develop an Airborne Laser capability acceptable to the war fighter. However, the agency has the opportunity to make its acquisition process more disciplined. By establishing knowledge-based decision points at key junctures, the agency would be in a better position to decide whether to move from one development phase to the next. Also, the agency would be better able to hold system developers accountable for planning all of the activities required to develop a quality product, approaching those activities in a systematic manner so that no important steps are skipped and problems are resolved sooner rather than later, and making cost and schedule projections when they have the knowledge to make realistic estimates. With this disciplined process in place, the agency is much more likely to achieve a needed capability for the war fighter within established cost and schedule goals.

Recommendations for Executive Action

To make its acquisition process more disciplined and provide better information for decision makers as additional investments in the Airborne Laser are considered, we recommend that the Secretary of Defense direct the Director of the Missile Defense Agency to establish decision points separating technology development from system integration, system integration from system demonstration, and system demonstration from production. For each decision point, we recommend that the Secretary instruct the Director to establish knowledge-based criteria and use those criteria to determine where additional investments should be made in the program.

Agency Comments and Our Evaluation

In commenting on a draft of this report, the Department of Defense partially concurred with our recommendations (see appendix II). The department stated that Secretary of Defense direction is not needed to implement our recommendations, the Missile Defense Agency’s acquisition process for ballistic missile defense already uses tailored versions of the knowledge-based practices recommended by us, and the agency intends to expand the use of knowledge-based criteria in the future.

The Department of Defense has not fully implemented the knowledge- based process recommended in our reports. Effective product development depends on gaining sufficient knowledge about technology, design, and manufacturing processes at key points in a system’s development. At those points, using metrics--such as technology readiness levels to measure the maturity of technology--that are commonly understood allow informed trade-offs to be made between resources, including cost and time, and performance. We have found that product development activities, such as building engineering prototypes of an integrated system and then demonstrating that the system can be manufactured to acceptable cost and quality standards, are ineffective unless the technologies needed to meet the product’s intended capabilities are fully matured and ready for system integration. Virtually every world- class product developer we have spoken with agrees with this.

The Airborne Laser program does not appear to have established this type of decision-making process. The Missile Defense Agency appears to have set up a development phase that combines maturing technologies with establishing a stable design. It does not include any visible decision points or standards to clearly indicate when technology development is concluded and system integration work to establish a design begins. Thus, it appears to us that this acquisition process forces the agency to manage significant risk from immature technologies simultaneously with trying to build a stable product design during this phase. Further, separating system integration from system demonstration and system demonstration from production and using common metrics in deciding to move forward will enhance the future likelihood that decisions on the Airborne Laser will be cost-effective. Such a process will also enhance decision-makers’ ability across the range of missile defense elements by facilitating comparisons across elements. Therefore, we have retained our recommendations.

Scope and Methodology

To address our objectives, we reviewed the contractor’s monthly cost performance reports, Defense Contract Management Agency analyses of those reports, and Defense Acquisition Executive Summaries and Selected Acquisition Reports prepared by the Airborne Laser program office. We also discussed cost and schedule problems with Airborne Laser program officials, Kirtland Air Force Base, New Mexico; and contractor officials at the Boeing Company, Seattle, Washington; Lockheed Martin, Sunnyvale, California; and TRW, Los Angeles, California. In addition, we obtained a technology readiness level analysis of the system’s critical technologies from the Airborne Laser program office. We compared this analysis with information obtained during our prior review to determine if progress had been made in maturing the critical technologies to higher technology readiness levels. We obtained detailed briefings from program office personnel and Missile Defense Agency officials, Arlington, Virginia; and from the contractors about the status of critical technologies and the problems associated with maturing the technologies required for the laser, the beam control/fire control system, and the required aircraft modifications. We also obtained detailed briefings from program office and Missile Defense Agency officials regarding the new Missile Defense Agency acquisition process and the implementation of this process within the Airborne Laser program. We conducted our review from July 2001 through May 2002 in accordance with generally accepted government auditing standards.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the congressional defense committees; the Secretary of Defense; the Director, Missile Defense Agency, the Secretary of the Air Force; and the Director, Office of Management and Budget. We will also make copies available to other interested parties upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

Please contact me at (202) 512-4841 if you or your staff have any questions concerning this report. Key contributors to this report are identified in appendix III.

Appendix I: Technology Readiness Level Assessment Matrix

2. Technology concept and/or application

Description Lowest level of technology readiness. Scientific research begins to be translated into applied research and development. Examples might include paper studies of a technology’s basic properties. Invention begins. Once basic principles are observed, practical applications can be invented. The application is speculative, and there is no proof or detailed analysis to support the assumption. Examples are still limited to paper studies. Active research and development is initiated. This includes analytical studies and laboratory studies to physically validate analytical predictions of separate elements of the technology. Examples include components that are not yet integrated or representative Basic technological components are integrated to establish that the pieces will work together. This is relatively “low fidelity” compared with the eventual system. Examples include integration of “ad hoc” hardware in a laboratory. Fidelity of breadboard technology increases significantly. The basic technological components are integrated with reasonably realistic supporting elements so that the technology can be tested in a simulated environment. Examples include “high fidelity” laboratory integration of components. Representative model or prototype system, which is well beyond the breadboard tested for technology readiness level five, is tested in a relevant environment. Represents a major step up in a technology’s demonstrated readiness. Examples include testing a prototype in a high-fidelity laboratory environment or in simulated operational environment. Prototype near or at planned operational system. Represents a major step up from technology readiness level six, requiring the demonstration of an actual system prototype in an operational environment, such as in an aircraft, vehicle, or space. Examples include testing the prototype in a test bed aircraft. Technology has been proven to work in its final form and under expected conditions. In almost all cases, this technology readiness level represents the end of true system development. Examples include developmental test and evaluation of the system in its intended weapon system to determine if it meets design specifications. Actual application of the technology in its final form and under mission conditions, such as those encountered in operational test and evaluation. In almost all cases, this is the end of the last “bug fixing” aspects of true system development. Examples include using the system under operational mission conditions.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to the contact named above, Christina Chaplain, Marcus Ferguson, Tom Gordon, Subrata Ghoshroy, Barbara Haynes, Matt Lea, Hai Tran, Adam Vodraska, and John Warren made key contributions to this report.