Report to the Chairman, Subcommittee

on National Security, Veterans’ Affairs,

and International Relations, Committee

on Government Reform, House of

Representatives

United States General Accounting Office

GAO

July 2002 MISSILE DEFENSE

Knowledge-Based

Decision Making

Needed to Reduce

Risks in Developing

Airborne Laser

GAO-02-631
Page i GAO-02-631 Missile Defense

Letter 1

Results in Brief 2

Background 3

Original Cost and Schedule Goals Are Based on Inadequate

Knowledge 5

New Strategy Incorporates Some Knowledge-Based Practices, but

Additional Practice Would Reduce Program Risk 10

Conclusion 15

Recommendations for Executive Action 16

Agency Comments and Our Evaluation 16

Scope and Methodology 17

Appendix I Technology Readiness Level Assessment Matrix 19

Appendix II Comments from the Department of Defense 20

Appendix III GAO Contact and Staff Acknowledgments 23

GAO Contact 23

Acknowledgments 23

Figures

Figure 1: Airborne Laser Aboard Boeing 747 Aircraft 5

Figure 2: Current Airborne Laser Technology Readiness Levels 8

Figure 3: The Knowledge-Based Process 12

Contents
Page 1 GAO-02-631 Missile Defense

July 12, 2002

The Honorable Christopher Shays

Chairman, Subcommittee on National Security,

 Veterans’ Affairs, and International Relations

Committee on Government Reform

House of Representatives

Dear Mr. Chairman:

In 1996, the Department of Defense, through the Air Force, launched an

acquisition program to develop and produce a revolutionary laser weapon

system. The system, known as the Airborne Laser because it is being

developed for installation in a modified Boeing 747 aircraft, is intended to

destroy enemy ballistic missiles almost immediately after their launch (in

the so-called “boost phase”) before they pose a threat to civilian

populations and military assets. The Air Force originally estimated

development costs at $2.5 billion and projected fielding of the system to

begin in 2006. However, by August 2001, the Air Force determined that

maturing the technologies and developing the system would cost about

50 percent more and take another 4 years. The development cost estimate

increased to $3.7 billion, and the fielding date slipped to 2010.

Against this backdrop of cost increases and schedule delays, the

Department of Defense transferred responsibility for the Airborne Laser in

October 2001 from the Air Force to the Ballistic Missile Defense

Organization, which shared responsibility with the armed services for

developing ballistic missile defense systems. Subsequently, in January

2002, the Secretary of Defense announced broad changes to the

department’s strategy for developing and acquiring missile defense

systems. Specifically, the Defense Secretary designated the Ballistic

Missile Defense Organization as the Missile Defense Agency and granted

the agency expanded responsibility and authority. The Secretary directed

the agency to develop an integrated system with various elements that

have the capability to attack enemy missiles in all phases of their flight,

transition responsibility for the production and fielding of systems to the

individual services, and encourage incremental improvements by inserting

United States General Accounting Office

Washington, DC 20548
Page 2 GAO-02-631 Missile Defense

new technologies through a series of block upgrades.1 The Airborne Laser

is one of many systems affected by the new strategy.

Concerned about significant cost and schedule problems associated with

developing the Airborne Laser, you asked us to determine (1) why the

system’s development is costing more and taking longer than the Air Force

originally estimated and (2) whether the Missile Defense Agency’s new

strategy for developing the Airborne Laser incorporates the practices that

characterize successful programs.

The Air Force was unable to meet the Airborne Laser’s original cost and

schedule goals because it established those goals before it fully

understood the level of effort that would be required to develop the

critical system technology needed to meet the user’s requirements. When

the Air Force launched the Airborne Laser acquisition program,

Department of Defense policy required that program cost and schedule

goals be established.2

 In 1996, at program launch, the Air Force did not

have enough knowledge about the technology challenges facing the

program. As a result, the Air Force underestimated the complexity of the

engineering task at hand and misjudged the amount of time and money

that the program would need. Some critical technologies that the system’s

design depends upon remain immature, making it very difficult, even

today, for analysts to establish realistic cost and schedule goals.

The Missile Defense Agency’s new strategy for developing the Airborne

Laser incorporates some knowledge-based practices that characterize

successful programs. For example, one practice that the agency

implemented is a requirements process that gives the agency the flexibility

to develop a system that has some capability without being held to

requirements that cannot be met with currently available technology. A

second knowledge-based practice is the provision of additional time and

facilities for testing. Increased testing allows agency officials to reach a

better understanding of the capabilities of the technology so that they can

establish more realistic requirements and ultimately more accurate

 1

 The Missile Defense Agency plans to develop a series of Airborne Laser configurations,

which are referred to as “blocks.” It is expected that each block will include improved

technology that was not available in the prior block.

2

 This policy implements statutory planning and reporting requirements for major defense

acquisition programs.

Results in Brief
Page 3 GAO-02-631 Missile Defense

estimates of the time and money needed to meet those requirements. A

third practice is the collection of the types of information needed to

determine whether the technology is “in-hand” to give war fighters an

Airborne Laser with some, if not all, desired capabilities. For example, the

agency intends to compare developed capabilities with data derived from

intelligence sources on the likely launch points and types of missiles that

the system could encounter.

However, the agency has not established knowledge-based decision points

and associated criteria for moving forward from technology development

to product development and on to production. Separating technology

development from product development has been a critical determinant

for successful program outcomes. Without decision points and criteria, the

agency risks beginning new and more costly activities before it has the

knowledge to determine the money and time required to complete the

activities and whether additional investment in those activities is

warranted. Also, the agency risks beginning the activities before it has the

knowledge to complete them without the need for expensive rework.

We are making recommendations that are intended to make the Missile

Defense Agency’s acquisition process more disciplined and provide better

information for decision makers as additional investments in the Airborne

Laser are considered.

In commenting on a draft of this report, the Department of Defense

partially concurred with our recommendations. The department stated

that Secretary of Defense direction is not needed to implement our

recommendations, the Missile Defense Agency’s acquisition process for

ballistic missile defense already uses tailored versions of the knowledgebased practices recommended by us, and the agency intends to expand the

use of knowledge-based criteria in the future. The Missile Defense

Agency’s acquisition process separates acquisition into three phases—

development, transition, and production. While the process definitely

separates product development from production, it clearly does not

separate technology development from product development. Also, it does

not establish the knowledge-based criteria characteristic of successful

programs at any of these decision points. Because we have not seen

evidence in the Airborne Laser’s strategy that such decision points and

criteria are in place, we have retained our recommendations.

The effort to develop the Airborne Laser is based on over 25 years of

scientific development in the Departments of Defense and Energy. It Background
Page 4 GAO-02-631 Missile Defense

evolved primarily from Airborne Laser laboratory research to develop

applications for high-energy lasers. This research culminated in a

demonstration that showed that a low-power, short-range laser was

capable of destroying a short-range air-to-air missile. Although this

demonstration was considered militarily insignificant because of the

laser’s low power and short range, it did succeed in identifying

technologies that were necessary for the development of an operational

Airborne Laser system. The research showed that an operational system

would need optics that could compensate for the atmospheric turbulence

that weakens and scatters a laser beam, optical devices that could

withstand the heat produced by a high-energy laser without the added

weight of water-cooling devices, and a new chemical laser with higher

energy levels that would produce a stronger laser beam.

In 1996, the Air Force launched the Airborne Laser program to develop a

defensive system that could destroy enemy missiles from a distance of

several hundred kilometers. Engineers determined that if they were to

meet this requirement, the system would need a 14-module3

 oxygen iodine

laser. They also determined that the system would need a beam

control/fire control assembly that could (1) safely move the laser beam

through the aircraft, (2) shape the beam so that it was not scattered or

weakened by the atmosphere, and (3) hold the beam on target, despite the

movement of the aircraft. In addition, engineers determined that the

system would need a battle management and control system capable of

planning and executing an engagement.

The Air Force planned to have the science and technology community

develop extensive knowledge about the laser and beam control/fire

control technologies before it launched an Airborne Laser acquisition

program. However, according to the retired manager of the science and

technology project, the budgets for technology efforts were limited, and

the science and technology community could not fund the technology

maturation effort. The Air Force knew that a program office was more

likely to command the large budget needed to fully mature technologies,

so it launched an acquisition program and assigned the program manager

responsibility for both technology and product development. The program

manager planned to demonstrate critical Airborne Laser technologies by

first building a six-module version of the oxygen iodine laser, installing it

 3

 The chemical reaction that generates the laser energy occurs in the laser modules. The

amount of laser energy produced increases as the number of laser modules increases.
Page 5 GAO-02-631 Missile Defense

along with other system components aboard a Boeing 747 aircraft (see

fig. 1), and testing the capability of this scaled system in system-level flight

tests. The tests would conclude in 2003 with an attempt to shoot down a

short-range ballistic missile target at a distance of 100 kilometers. If this

final test were successful, the Airborne Laser would have moved into

product development.

Figure 1: Airborne Laser Aboard Boeing 747 Aircraft

Source: Airborne Laser Program Office.

The Air Force launched the Airborne Laser acquisition program and

identified cost and schedule goals before officials had the knowledge to

make realistic projections. In 1996, when the program was launched,

Department of Defense regulation 5000.2 required, and still requires today,

that when a military service initiates a major acquisition program, it must

establish cost and schedule goals. However, the Air Force could not make

realistic estimates when it began the program because it had no way of

knowing how much engineering effort would be needed to complete the

development of technology critical to the system. Even today, some

critical technologies that the system’s design depends upon remain

immature, making it very difficult for analysts to determine how long it

will take and how much it will cost to develop and produce the system.

At the time the Airborne Laser program was launched, the laser and beam

control/fire control technologies needed to develop the Airborne Laser

system was immature. The Department of Defense’s science and

technology community was actively researching and developing the laser

and had produced a weak beam in a laboratory setting—but this major

Original Cost and

Schedule Goals Are

Based on Inadequate

Knowledge

Technologies Were

Immature at Program

Launch
Page 6 GAO-02-631 Missile Defense

component had not reached the level of maturity needed to proceed into

product development. The technology necessary to develop the beam

control/fire control was even less advanced. Most of the scientists’ work

was limited to analytical studies, wherein a few tests of laboratory

hardware were linked together to work somewhat like the intended

component.

Because technology development is a process of discovery, the Air Force

soon learned that there were too many unknowns regarding the

development of Airborne Laser technology to make good cost and

schedule estimates. As the technology development progressed,

unanticipated technical challenges affected the program’s cost and

schedule. Department of Defense analysts reported that the Airborne

Laser program experienced cost and schedule growth because the

program and its contractors underestimated the complexity of

(1) designing laser components, (2) the system’s engineering analysis and

design effort, and (3) engineering the system to fit on board the aircraft. As

system development progressed and the Air Force gained a better

understanding of the technical complexity of the system, the Air Force

increased its cost and schedule estimates.

The Air Force has made some progress in developing the Airborne Laser’s

critical technologies, but many remain immature. We asked the Airborne

Laser program office to determine the technologies most critical to the

Airborne Laser system and to use technology readiness levels4

 to assess

the maturity of each. The officials determined that if the Airborne Laser is

to meet the requirements established by the war fighters, then engineers

must mature technologies in six areas, all of which are needed to

successfully design the system. These technologies are

• devices that stabilize the laser system aboard the aircraft so that the beam

can be maintained firmly on the target,

• optics—mirrors and windows—that focus and control the laser beam and

allow it to pass safely through the aircraft,

• optical coatings that enhance the optics’ ability to pass laser energy

through the system and to reflect the laser energy,

 4

 Technology readiness levels were developed by the National Aeronautics and Space

Agency and are recommended for use by the Department of Defense and the military

services. (See appendix I for their definition.)

Some Critical Technologies

Remain Immature
Page 7 GAO-02-631 Missile Defense

• hardware that works in tandem with computer software to actively track

the target missile,

• devices that measure atmospheric turbulence and compensate for it so

that it does not scatter or weaken the laser beam, and

• safety systems that automatically shut down the high energy laser in the

event of an emergency.

At our request, the program office also assessed the maturity of the

oxygen iodine laser.

As figure 2 shows, program officials assessed the optical coatings at level

five and the safety systems, atmospheric compensation, and targettracking components at level six. At technology readiness level five, the

technology being tested is incorporated into hardware whose form and fit

are coming closer to that needed for an operational component and

integrated with reasonable realistic supporting elements so that the

technology can be tested in a simulated environment. At level six, the

technology is incorporated into a prototype and tested in a high-fidelity

laboratory environment or in a simulated operational environment. The

program officials identified the optics and stabilizing devices as the least

mature—at level four. At this level, engineers have shown that a

technology is technically feasible but have not shown whether the

technology will have the form, fit, or function required in the operational

system. We agreed with all but one of the program officials’ assessments

for these technologies.
Page 8 GAO-02-631 Missile Defense

Figure 2: Current Airborne Laser Technology Readiness Levels

Note: Appendix I contains information describing technology readiness levels.

Source: GAO’s analysis.

Our one disagreement centered on the maturity of the laser component of

the system. While the program office assessed it at a technology readiness

level of six, we consider the laser technology to be at level four because

tests have been conducted only for a one-module laser in a controlled

laboratory environment using surrogate components. For example, the

tests used a stable laser resonator, rather than the unstable resonator that
Page 9 GAO-02-631 Missile Defense

will be used in system-level flight tests.5 We also found that during tests of

the one-module laser, the resonator was operating in multimode rather

than single-mode.6

 The resonator in the operational system will operate in

single mode. Furthermore, the chemical storage and delivery

subcomponents used in these tests were not representative of those that

will be incorporated into the system’s design. According to program office

officials, conducting a more realistic test would have cost time and money

that were not available.

Documents summarizing the tests of the one-module laser stated that the

tests were successful in reducing the technical risks associated with the

one-module system but that a new set of technical risks linked with

developing a multimodule system must still be addressed during testing of

the six-module system. In our opinion, the program office will

demonstrate the laser technology in a relative environment (technology

readiness level six) when the six-module system is integrated and

successfully tested at full power within the high-fidelity laboratory

environment of the Airborne Laser Systems Integration Laboratory,

currently under construction at Edwards Air Force Base, California.

According to the program office, this type of demonstration will not occur

until February 2003.

 5

 A resonator consists of two mirrors placed at opposite ends of a laser cavity. As the

reaction of chemicals within the laser cavity produces photons of light, the photons are

reflected back and forth between the two mirrors, which generates additional photons and

creates a state of high energy within the cavity. In a stable resonator, one mirror is fully

reflective while the other is partially reflective and partially transmissive. Energy that

escapes from the laser cavity through the transmissive portion of the mirror in a stable

resonator forms a high-energy beam. In an unstable resonator, both mirrors are fully

reflective, and one is much smaller in diameter. As the photons are reflected from the

larger mirror in the direction of the smaller mirror, energy escapes from the laser cavity

around the edges of the smaller mirror and forms a doughnut-shaped beam.

6

 As photons are generated in a laser resonator, the photons oscillate or move in different

ways. A resonator operating in single-mode suppresses all photons except those oscillating

at a certain frequency so that the beam produced can be directed at one spot on the target.

A resonator operating in multimode does not suppress any photons, regardless of their

frequency. While a multimode resonator directs more energy toward the target, all of that

energy will not be focused on one area of the target.
Page 10 GAO-02-631 Missile Defense

The Missile Defense Agency’s new strategy for developing the Airborne

Laser incorporates some of the knowledge-based practices that

characterize successful programs, but the agency would benefit from

adopting another that would add greater discipline to its acquisition

process. The new strategy allows more flexibility in setting requirements,

makes time and facilities available to mature and test the critical

technologies, and collects information needed to match the war fighters’

requirements to demonstrated technology. However, the agency has not

established decision points with associated knowledge-based criteria for

moving forward from (1) technology development to system integration,

(2) system integration to system demonstration, and (3) system

demonstration to production. At each of these points, the agency would

stop to assess its knowledge and decide whether investment in the

program’s next phase is warranted.

The first new practice allows the Missile Defense Agency to refine

requirements on the basis of the results of system engineering. The

Department of Defense ordinarily faces significant hurdles in matching

requirements to resources. The fundamental problem is twofold. First,

under the department’s traditional process, requirements must be set

before a program can be approved and a program must be approved

before the product developer conducts systems engineering. Second, the

competition for funding encourages requirements that will make the

desired weapon system stand out from others. Consequently, many of the

department’s product development programs include unrealistic

requirements set by the user before the product developer has conducted

the system engineering necessary to identify the time, technology, and

money necessary to develop a product capable of meeting requirements.

A second practice that is likely to improve the Airborne Laser’s

development is making the time and facilities available to mature and test

critical technologies. To implement this practice, the agency increased the

time available to test the six-module laser system and is building a new

test facility. Instead of following the Air Force’s plan to complete systemlevel flight tests of the six-module system in the last quarter of fiscal year

2003, the agency has delayed the demonstration to the first quarter of

fiscal year 2005. This delay will allow additional time to learn from and

correct problems discovered during system-level tests that are scheduled

to begin in the last quarter of fiscal year 2003 and end with the fiscal year

2005 demonstration. In addition, the agency plans to increase the Airborne

Laser’s ground-testing capability by awarding a contract in 2003 for what

the agency is calling an “iron bird,” which is essentially an aircraft hull

New Strategy

Incorporates Some

Knowledge-Based

Practices, but

Additional Practice

Would Reduce

Program Risk

New Strategy Introduces

Knowledge-Based

Practices
Page 11 GAO-02-631 Missile Defense

with installed laser equipment. The “iron bird” is expected to allow testing

of a fully integrated Airborne Laser system on the ground so that

technologies for future blocks can be evaluated before being installed in

an aircraft.

The information gained from testing informs the requirements process.

Because testing allows developers to gauge the progress being made in

translating an idea into a weapon system, it enables the developer to make

a more informed decision as to whether a technology is ready to be

incorporated into a system’s design. With this knowledge, the developer

can determine whether the technology is so important to the system’s

design that additional time and money should be spent to mature the

technology or whether the system’s initial performance requirements

should be reduced.

A third practice that the agency plans to adopt is matching requirements to

available technology. According to the Missile Defense Agency’s Technical

Director, the agency defines the war fighters’ requirement as a system that

has the capability to destroy some threat ballistic missiles during their

boost phase at a range representative of an operational scenario. The

Technical Director told us that the agency will attain the knowledge to

determine if it has the technology in-hand to meet this requirement by

examining each block’s capabilities during simulated and system-level

flight test and comparing those capabilities with data derived from

intelligence sources on the likely launch points and types of missiles that

the system could encounter. Our previous work with successful

development programs shows that once the technology is in-hand to meet

the customer’s requirements, the developer can make more accurate initial

estimates of the cost and time needed to develop and produce an

operational system.
Page 12 GAO-02-631 Missile Defense

Successful developers have instilled discipline in their acquisition

processes by requiring that certain criteria for attaining knowledge are met

as an acquisition program moves forward. (See fig. 3.) They recognize that

the focus and cost of activities change over time and that less rework is

required if all activities with the same focus are completed before

beginning other activities.

Figure 3: The Knowledge-Based Process

Source: GAO’s analysis.

In successful development programs, decisions are made when the

knowledge is available to support those decisions. The first decision point,

or knowledge point, occurs when the focus of a developer’s activities

changes from technology development to system integration—the first

phase of product development. The criterion for deciding to move forward

is having the knowledge to match requirements and available resources

(time, technology, and funds). The second knowledge point occurs

between system integration and system demonstration when the developer

has successfully integrated subsystems and components into a stable

design that not only meets the customer’s performance requirements but

also is optimized for reproducibility, maintainability, and reliability. The

Successful Developers

Recognize Need for

Knowledge-Based Decision

Points
Page 13 GAO-02-631 Missile Defense

decision criterion used here is usually having completed about 90 percent

of the engineering drawings. The third knowledge point separates system

demonstration from production. The decision to invest in production is

generally based on a determination that the product performs as required

during testing and that the manufacturing processes will produce a

product within cost, schedule, and quality targets.

The cost of a program’s activities increases as it moves closer to

production. In commercial acquisitions, product development is typically

much more costly than technology development. During technology

development, small teams of technologists work to perfect the application

of scientific knowledge to a practical problem. As product development

begins, developers begin to make larger investments in human capital,

bringing on a large engineering force to design and manufacture the

product. In addition, product development requires significant investments

in facilities and materials. These investments increase continuously as the

product approaches the point of manufacture. In fact, industry experts

estimate that identifying and resolving a problem during product

development can cost 10 times more than correcting that problem during

technology development and that correcting the problem during

manufacturing is even more costly.

We examined the Airborne Laser’s acquisition strategy and determined

that it does not include decision points at which officials would use

knowledge-based criteria to determine if the program is ready to move

from technology development to system integration, system integration to

system demonstration, and system demonstration to production. We found

that the agency’s process has three phases: development, transition, and

production.

• Development includes all developmental activities and system-level

demonstrations of military utility.

• Transition will involve preparation of the operational requirements

document by the appropriate armed service and conducting operational

testing.

• Production will involve producing and fielding the final weapon system.

The agency’s strategy also calls for developing the Airborne Laser

incrementally, rather than trying to initially develop a system with all

Knowledge-Based

Decisions Missing from

Airborne Laser Strategy
Page 14 GAO-02-631 Missile Defense

desired capabilities. In the near term, the agency plans to complete the sixmodule laser system aircraft, now known as block 2004,7

 and use it to

demonstrate critical Airborne Laser technologies. Beginning in March

2003, the agency intends to begin developing another demonstration

aircraft, known as block 2008, which will incorporate new capabilities and

technologies. The Airborne Laser program manager told us that blocks

2004 and 2008 are primarily test assets for the purpose of technology

demonstration. While some of the block 2008 activities are focused on

improving subsystems and components, such as reducing the weight of

laser components and improving optics, other activities are focused on the

integration of these pieces into a block 2008 design.8

The agency expects to develop subsequent blocks, or system

configurations to introduce additional capabilities. If system-level tests

show that any one of these configurations performs at a level that merits

fielding, the Air Force will prepare a requirements document based on the

configuration’s demonstrated capabilities and make plans for operational

testing and production. This “baseline” capability would be improved in

subsequent blocks as more advanced technology becomes available and as

the threat warrants.

We did not find that the agency’s strategy includes a disciplined process

that separates technology development, system integration, system

demonstration, and production with decision points supported by

knowledge-based criteria. Instead, the agency has put in place a decision

point for moving from the development to the transition phase. According

to the agency’s strategy, when the agency determines that it has the

technology in-hand to produce a system that merits fielding, it will begin to

transition the system over to the appropriate military service. Also, at the

end of the transition phase, a system would enter the formal Department

of Defense acquisition process at Milestone C—the point at which the

 7

 The six-module system is referred to as block 2004 because testing will conclude in

December 2004. Testing of a second configuration, known as block 2008, will be completed

in December 2008.

8

 One of the major technical challenges is accommodating the laser’s weight. Engineers

determined that the six-module system would weigh 180,000 pounds, but the original

system requirement was that the system must weigh no more than 175,000 pounds with

14 laser modules. Because each additional module weighs about 6,000 pounds, the agency

intends to redesign some components to reduce their mass and redistribute the weight

using a passenger version of the Boeing 747 as the block 2008 aircraft. The passenger

version of the 747 can accommodate the crew on an upper deck, thereby allowing the

laser’s weight to be moved forward where it places less stress on the aircraft frame.
Page 15 GAO-02-631 Missile Defense

decision is made to enter low rate initial production. We did not find,

however, an established set of decision points with associated criteria that

would enable the agency to make a knowledge-based decision on whether

to invest in system integration and, subsequently, system demonstration

and production. That is, even though the agency might know that it has the

technology in-hand to develop a useful military capability, it has not

established a first decision point where it would determine the cost and

time needed to move the program forward and whether the program

should proceed into a system integration phase during which the design

would be matured and optimized for reproducibility, maintainability, and

reliability. Neither does the agency’s strategy include a second decision

point that would allow agency officials to use the knowledge they have

attained regarding the design’s maturity to determine whether to invest

further to demonstrate that the system meets requirements and that

manufacturing processes are in place to repeatedly produce a quality

product. Only after the agency successfully moves the program through all

of these decision points and successfully demonstrates the system’s

capabilities and manufacturing processes would the agency’s production

decision be fully knowledge based. Without this disciplined process, the

agency would be accepting greater cost and schedule risks and is much

less likely to realize the full potential benefits of its new approach to

developing missile defense systems.

The revolutionary nature of missile defense weapon systems demands

cutting-edge technology. Although there is no one approach that ensures

that a developer can deal successfully with the unknowns inherent in

developing a product from such technology, the knowledge-based process

has proven to yield good results within cost and schedule estimates. The

Missile Defense Agency has implemented practices that are part of the

knowledge-based approach, and these practices are likely to improve the

agency’s ability to gather the knowledge it needs to develop an Airborne

Laser capability acceptable to the war fighter. However, the agency has

the opportunity to make its acquisition process more disciplined. By

establishing knowledge-based decision points at key junctures, the agency

would be in a better position to decide whether to move from one

development phase to the next. Also, the agency would be better able to

hold system developers accountable for planning all of the activities

required to develop a quality product, approaching those activities in a

systematic manner so that no important steps are skipped and problems

are resolved sooner rather than later, and making cost and schedule

projections when they have the knowledge to make realistic estimates.

With this disciplined process in place, the agency is much more likely to

Conclusion
Page 16 GAO-02-631 Missile Defense

achieve a needed capability for the war fighter within established cost and

schedule goals.

To make its acquisition process more disciplined and provide better

information for decision makers as additional investments in the Airborne

Laser are considered, we recommend that the Secretary of Defense direct

the Director of the Missile Defense Agency to establish decision points

separating technology development from system integration, system

integration from system demonstration, and system demonstration from

production. For each decision point, we recommend that the Secretary

instruct the Director to establish knowledge-based criteria and use those

criteria to determine where additional investments should be made in the

program.

In commenting on a draft of this report, the Department of Defense

partially concurred with our recommendations (see appendix II). The

department stated that Secretary of Defense direction is not needed to

implement our recommendations, the Missile Defense Agency’s acquisition

process for ballistic missile defense already uses tailored versions of the

knowledge-based practices recommended by us, and the agency intends to

expand the use of knowledge-based criteria in the future.

The Department of Defense has not fully implemented the knowledgebased process recommended in our reports. Effective product

development depends on gaining sufficient knowledge about technology,

design, and manufacturing processes at key points in a system’s

development. At those points, using metrics--such as technology readiness

levels to measure the maturity of technology--that are commonly

understood allow informed trade-offs to be made between resources,

including cost and time, and performance. We have found that product

development activities, such as building engineering prototypes of an

integrated system and then demonstrating that the system can be

manufactured to acceptable cost and quality standards, are ineffective

unless the technologies needed to meet the product’s intended capabilities

are fully matured and ready for system integration. Virtually every worldclass product developer we have spoken with agrees with this.

The Airborne Laser program does not appear to have established this type

of decision-making process. The Missile Defense Agency appears to have

set up a development phase that combines maturing technologies with

establishing a stable design. It does not include any visible decision points

Recommendations for

Executive Action

Agency Comments

and Our Evaluation
Page 17 GAO-02-631 Missile Defense

or standards to clearly indicate when technology development is

concluded and system integration work to establish a design begins. Thus,

it appears to us that this acquisition process forces the agency to manage

significant risk from immature technologies simultaneously with trying to

build a stable product design during this phase. Further, separating system

integration from system demonstration and system demonstration from

production and using common metrics in deciding to move forward will

enhance the future likelihood that decisions on the Airborne Laser will be

cost-effective. Such a process will also enhance decision-makers’ ability

across the range of missile defense elements by facilitating comparisons

across elements. Therefore, we have retained our recommendations.

To address our objectives, we reviewed the contractor’s monthly cost

performance reports, Defense Contract Management Agency analyses of

those reports, and Defense Acquisition Executive Summaries and Selected

Acquisition Reports prepared by the Airborne Laser program office. We

also discussed cost and schedule problems with Airborne Laser program

officials, Kirtland Air Force Base, New Mexico; and contractor officials at

the Boeing Company, Seattle, Washington; Lockheed Martin, Sunnyvale,

California; and TRW, Los Angeles, California. In addition, we obtained a

technology readiness level analysis of the system’s critical technologies

from the Airborne Laser program office. We compared this analysis with

information obtained during our prior review to determine if progress had

been made in maturing the critical technologies to higher technology

readiness levels. We obtained detailed briefings from program office

personnel and Missile Defense Agency officials, Arlington, Virginia; and

from the contractors about the status of critical technologies and the

problems associated with maturing the technologies required for the laser,

the beam control/fire control system, and the required aircraft

modifications. We also obtained detailed briefings from program office

and Missile Defense Agency officials regarding the new Missile Defense

Agency acquisition process and the implementation of this process within

the Airborne Laser program. We conducted our review from July 2001

through May 2002 in accordance with generally accepted government

auditing standards.

As agreed with your office, unless you publicly announce the contents of

this report earlier, we plan no further distribution until 30 days from the

report date. At that time, we will send copies of this report to the

congressional defense committees; the Secretary of Defense; the Director,

Missile Defense Agency, the Secretary of the Air Force; and the Director,

Scope and

Methodology
Page 18 GAO-02-631 Missile Defense

Office of Management and Budget. We will also make copies available to

other interested parties upon request. In addition, the report will be

available at no charge on the GAO Web site at http://www.gao.gov.

Please contact me at (202) 512-4841 if you or your staff have any questions

concerning this report. Key contributors to this report are identified in

appendix III.

Sincerely yours,

R. E. Levin

Director, Acquisition and

 Sourcing Management
Appendix I: Technology Readiness Level

Assessment Matrix

Page 19 GAO-02-631 Missile Defense

Technology readiness level Description

1. Basic principles observed and

reported.

Lowest level of technology readiness. Scientific research begins to be translated into

applied research and development. Examples might include paper studies of a

technology’s basic properties.

2. Technology concept and/or application

formulated.

Invention begins. Once basic principles are observed, practical applications can be

invented. The application is speculative, and there is no proof or detailed analysis to

support the assumption. Examples are still limited to paper studies.

3. Analytical and experimental critical

function and/or characteristic proof of

concept.

Active research and development is initiated. This includes analytical studies and

laboratory studies to physically validate analytical predictions of separate elements of the

technology. Examples include components that are not yet integrated or representative

4. Component and/or breadboard.

Validation in laboratory environment.

Basic technological components are integrated to establish that the pieces will work

together. This is relatively “low fidelity” compared with the eventual system. Examples

include integration of “ad hoc” hardware in a laboratory.

5. Component and/or breadboard

validation in relevant environment.

Fidelity of breadboard technology increases significantly. The basic technological

components are integrated with reasonably realistic supporting elements so that the

technology can be tested in a simulated environment. Examples include “high fidelity”

laboratory integration of components.

6. System/subsystem model or prototype

demonstration in a relevant

environment.

Representative model or prototype system, which is well beyond the breadboard tested

for technology readiness level five, is tested in a relevant environment. Represents a

major step up in a technology’s demonstrated readiness. Examples include testing a

prototype in a high-fidelity laboratory environment or in simulated operational

environment.

7. System prototype demonstration in an

operational environment.

Prototype near or at planned operational system. Represents a major step up from

technology readiness level six, requiring the demonstration of an actual system prototype

in an operational environment, such as in an aircraft, vehicle, or space. Examples include

testing the prototype in a test bed aircraft.

8. Actual system completed and “flight

qualified” through test and

demonstration.

Technology has been proven to work in its final form and under expected conditions. In

almost all cases, this technology readiness level represents the end of true system

development. Examples include developmental test and evaluation of the system in its

intended weapon system to determine if it meets design specifications.

9. Actual system “flight proven” through

successful mission operations.

Actual application of the technology in its final form and under mission conditions, such as

those encountered in operational test and evaluation. In almost all cases, this is the end

of the last “bug fixing” aspects of true system development. Examples include using the

system under operational mission conditions.

Appendix I: Technology Readiness Level

Assessment Matrix
Appendix II: Comments from the Department

of Defense

Page 20 GAO-02-631 Missile Defense

Appendix II: Comments from the Department

of Defense
Appendix II: Comments from the Department

of Defense

Page 21 GAO-02-631 Missile Defense
Appendix II: Comments from the Department

of Defense

Page 22 GAO-02-631 Missile Defense
Appendix III: GAO Contact and Staff

Acknowledgments

Page 23 GAO-02-631 Missile Defense

R. E. Levin (202) 512-4841

In addition to the contact named above, Christina Chaplain, Marcus

Ferguson, Tom Gordon, Subrata Ghoshroy, Barbara Haynes, Matt Lea, Hai

Tran, Adam Vodraska, and John Warren made key contributions to this

report.

Appendix III: GAO Contact and Staff

Acknowledgments

GAO Contact

Acknowledgments

(120079)
The General Accounting Office, the investigative arm of Congress, exists to

support Congress in meeting its constitutional responsibilities and to help

improve the performance and accountability of the federal government for the

American people. GAO examines the use of public funds; evaluates federal

programs and policies; and provides analyses, recommendations, and other

assistance to help Congress make informed oversight, policy, and funding

decisions. GAO’s commitment to good government is reflected in its core values

of accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost is

through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and fulltext files of current reports and testimony and an expanding archive of older

products. The Web site features a search engine to help you locate documents

using key words and phrases. You can print these documents in their entirety,

including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and

correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site

daily. The list contains links to the full-text document files. To have GAO e-mail

this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily

E-mail alert for newly released products” under the GAO Reports heading.

The first copy of each printed report is free. Additional copies are $2 each. A

check or money order should be made out to the Superintendent of Documents.

GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a

single address are discounted 25 percent. Orders should be sent to:

U.S. General Accounting Office

441 G Street NW, Room LM

Washington, D.C. 20548

To order by Phone: Voice: (202) 512-6000

TDD: (202) 512-2537

Fax: (202) 512-6061

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800

U.S. General Accounting Office, 441 G Street NW, Room 7149

Washington, D.C. 20548

GAO’s Mission

Obtaining Copies of

GAO Reports and

Testimony

Order by Mail or Phone

To Report Fraud,

Waste, and Abuse in

Federal Programs

Public Affairs