GAO

United States General Accounting Office

Report to the Chairman and Ranking 

Minority Member, Subcommittee on 

Readiness and Management Support, 

Committee on Armed Services, U.S. 

Senate

July 1999 BEST PRACTICES

Better Management of 

Technology 

Development Can 

Improve Weapon 

System Outcomes

GAO/NSIAD-99-162

United States General Accounting Office

Washington, D.C. 20548

National Security and

International Affairs Division

B-280233 Letter

July 30, 1999

The Honorable James Inhofe

Chairman

The Honorable Charles Robb

Ranking Minority Member

Subcommittee on Readiness and Management Support

Committee on Armed Services

United States Senate

As you requested, this report assesses how best practices offer improvements to the way the 

Department of Defense (DOD) incorporates new technology into weapon system programs. It also 

assesses the factors that can make it difficult to mature technologies before they are included on 

weapon system programs and what can be done about them. We make recommendations to the 

Secretary of Defense on how advanced technologies can be better managed so they pose less risk 

when they are included in weapon system designs.

We are sending copies of this report to the Honorable William S. Cohen, Secretary of Defense; the 

Honorable Louis Caldera, Secretary of the Army; the Honorable Richard Danzig, Secretary of the 

Navy; the Honorable F. Whitten Peters, Acting Secretary of the Air Force; the Honorable Jacob J. Lew, 

Director, Office of Management and Budget; and to interested congressional committees. We will also 

make copies available to others upon request.

If you have any questions regarding this report, please call me at (202) 512-4841. Other key contacts 

are listed in appendix III.

Katherine V. Schinasi

Associate Director

Defense Acquisitions Issues

Letr
Page 2 GAO/NSIAD-99-162 Best Practices

Executive Summary

Purpose The Department of Defense (DOD) plans to increase its investment in new 

weapons to about $60 billion in fiscal year 2001—a 40-percent increase 

over fiscal year 1997. DOD has high expectations from this investment: that 

new weapons will be better and less expensive than their predecessors and 

will be developed in half the time. With its traditional management 

approach—which has produced superior weapons, but at much greater 

cost and time than planned—DOD will not meet these expectations. 

Leading commercial firms have changed their practices for developing 

products and have achieved the kinds of results DOD seeks. Maturing new 

technology before it is included in products is one of the main determinants 

of these firms’ successes. This practice holds promise for DOD, for 

immature technologies have been a main source of problems on weapon 

systems. In response to a request from the Chairman and the Ranking 

Minority Member, Subcommittee on Readiness and Management Support, 

Senate Committee on Armed Services, GAO assessed (1) the impact of 

technology maturity on product outcomes, (2) best practices for managing 

new technologies and incorporating them into products, and (3) ways DOD 

can adapt these practices to get better outcomes on weapon system 

programs.

Background GAO reviewed commercial and DOD experiences in incorporating

23 different technologies into new product and weapon system designs. 

The technologies were drawn from (1) six commercial firms recognized for 

their success in developing technically advanced products more quickly 

than the products’ predecessors and (2) five DOD weapon system 

programs that incorporated advanced technologies, including some that 

did not encounter problems and some that did. GAO asked the managers of 

these technologies to assess the maturity of the technologies at the point 

they were included in product development by applying a tool, referred to 

as technology readiness levels (TRLs). The National Aeronautics and Space 

Administration and the Air Force Research Laboratory use TRLs to 

determine the readiness of technologies to be incorporated into a weapon 

or another type of system. Readiness levels are measured along a scale of 

one to nine, starting with paper studies of the basic concept, proceeding 

with laboratory demonstrations, and ending with a technology that has 

proven itself on the intended product. 

The Air Force Research Laboratory considers TRL 6 an acceptable risk for 

a weapon system entering the program definition stage, the point at which 

DOD launches its weapon programs, and TRL 7 an acceptable risk for the 

Letr
Executive Summary

Page 3 GAO/NSIAD-99-162 Best Practices

engineering and manufacturing development stage. This is an important 

distinction because leading commercial firms launch a new product later 

than DOD, after technology development is complete. They refer to this 

point as the beginning of product development, the point at which they 

commit to developing and manufacturing the product. Typically, 

technology is still being developed when weapon system programs are 

launched; the point at which a weapon system is far enough along to 

compare to a commercial product development is likely to be at or after the 

start of engineering and manufacturing development.

Results in Brief The experiences of DOD and commercial technology development cases 

GAO reviewed indicate that demonstrating a high level of maturity before 

new technologies are incorporated into product development programs 

puts those programs in a better position to succeed. The TRLs, as applied 

to the 23 technologies, reconciled the different maturity levels with 

subsequent product development experiences. They also revealed when 

gaps occurred between a technology’s maturity and the intended product’s 

requirements. For technologies that were successfully incorporated into a 

product, the gap was recognized and closed before product development 

began, improving the chances for successful cost and schedule outcomes. 

The closing of the gap was a managed result. It is a rare program that can 

proceed with a gap between product requirements and the maturity of key 

technologies and still be delivered on time and within costs. 

Two conditions were critical to closing the maturity gap. First, the right 

environment for maturing technologies existed. Key to this environment 

was making a science and technology organization, rather than the 

program or product development manager, responsible for maturing 

technologies to a high TRL. When a maturity gap persisted, managers were 

given the flexibility to take the time to mature the technology or decrease 

product requirements so that they could use another, already mature 

technology. Second, both technology and product managers were 

supported with the disciplined processes, readily available information, 

readiness standards, and authority to ensure technology was ready for 

products. This support enabled these managers to safeguard product 

development from undue technology risks. On the other hand, immature 

technologies were sometimes incorporated into products for reasons such 

as inflexible performance requirements, increasing the likelihood of cost 

overruns and delays in product development. Product managers had little 

choice but to accept the technologies and hope that they would mature 

Letr
Executive Summary

Page 4 GAO/NSIAD-99-162 Best Practices

successfully. However, the pressures of product development made for an 

environment less conducive to maturing technology.

For several reasons, DOD is likely to move technologies to product 

development programs before they are mature. Science and technology 

organizations, which traditionally operate within fixed budget levels, do 

not necessarily have the funds to mature technology to the higher TRLs. 

Programs are more able to command the large budgets necessary for 

reaching these levels. The pressures exerted on new programs to offer 

unique performance at low cost encourage acceptance of unproven 

technologies. The technologies GAO reviewed indicate these conditions 

can be overcome on individual cases. DOD has several initiatives 

underway, such as advanced technology demonstrations, that could make 

it more feasible for science and technology organizations to mature 

technology before it is moved to product development programs. The 

challenge will be whether the lessons learned from these cases and 

initiatives offer an approach that has a DOD-wide application. 

GAO makes recommendations to the Secretary of Defense on ways to 

pursue advanced technologies while lessening their potential for causing 

problems on weapon system programs.

Principal Findings

Maturity of Technology at 

Program Start Is an 

Important Determinant of 

Success

The 23 technologies GAO reviewed spanned a wide range of readiness 

levels—from a low of TRL 2 to a high of TRL 9—when they were included 

in product development programs. Programs with key technologies at 

readiness levels 6 to 8 at the time of program launch met or were meeting 

cost, schedule, and performance requirements. All of the commercial 

technologies and a few of the DOD technologies fell into this category. For 

example, Ford managed its voice-activated control technology to TRL 8—a 

10-year effort—before introducing it on the 1999 Jaguar. Similarly, the 

Defense Advanced Research Projects Agency matured a revolutionary 

periscope technology to TRL 9 before it was included on the Virginia class 

attack submarine. DOD programs that accepted technologies at a readiness 

level of 5 or less experienced significant cost and schedule increases due, 

in part, to problems with the technologies. DOD’s acceptance of 

technologies at level 4 or lower was not unusual. For example, the key 

technologies for the Army’s brilliant antiarmor submunition were at levels 2 
Executive Summary

Page 5 GAO/NSIAD-99-162 Best Practices

and 3 when weapon system development began. At these levels, DOD had a 

significant gap in technology maturity at the start of the program. The gap 

was not closed until well into the development program, and problems with 

the technologies were a main contributor to the program’s 88-percent cost 

growth and 62-percent slip in schedule. 

Controllable Conditions 

Affect How Well a 

Technology’s Inclusion on a 

Product Can Be Managed

Closing the technology development gap before beginning product 

development was the result of good technology maturation practices and 

sound methods for moving technologies to products. The more successful 

of the 23 technologies were managed by science and technology 

organizations until they reached at least TRL 6 and more, often TRL 8 or 

higher. This environment was an important condition for successfully 

maturing technologies, as it allowed room for unexpected results such as 

test “failures,” which are considered normal events in developing 

technologies. To match technology maturity and product requirements, 

managers also had the option of waiting until technologies matured or 

changing product requirements so that an already mature technology could 

be used. For example, Hughes deferred the development of the HS-702 

satellite until critical solar cell technology had matured—a process that 

took over 10 years. Also, Navy managers accepted an existing weapon 

ejection system on the Virginia class attack submarine when technology 

failed to mature as expected. In contrast, performance requirements for the 

Comanche helicopter were inflexible; requirements mandated the inclusion 

of advanced sensors and avionics technologies, despite their immaturity. 

The Comanche program has experienced cost growth and schedule delays, 

partly attributable to the inclusion of these technologies.

In the more successful cases, technology and product managers were given 

the authority and tools to move technology only when it was at high 

readiness levels. Disciplined processes provided managers credible 

information on the status of technologies and high standards for assessing 

readiness. Science and technology managers developed technologies to 

standards acceptable to product managers who could reject those 

technologies that fell short. For example, Ford’s science and technology 

managers use agreed-upon standards for judging technology readiness, and 

all new technologies follow the same maturation process. Ford’s product 

managers are also empowered to say no when technologies are not deemed 

mature. Recently, the Jaguar vehicle team rejected night vision technology 

at TRL 8 because it did not meet cost objectives. DOD program managers 

that had to accept immature technologies had less information available to 

guide them. For example, key technologies for the brilliant antiarmor 
Executive Summary

Page 6 GAO/NSIAD-99-162 Best Practices

submunition program bypassed Army science and technology 

organizations, forcing the program manager to accept the technologies 

with little information about their readiness. Often, the tools used to assess 

the technologies’ status failed to identify high risks; the TRLs indicate that 

risks on the problematic technologies were often high. Also, the greater 

pressures to meet cost and schedule goals in product development 

provided a less forgiving environment for fledgling technologies. 

Impediments to Adopting 

Best Practices for 

Technology Inclusion in 

DOD Are Surmountable

Leading commercial firms have put the organizations, tools, and other 

practices in place to foster technology development and improve the 

outcomes of product developments as a matter of necessity. The large 

investment required for a new product and the risks to that investment if 

the product does not meet customer needs reinforce these practices. The 

DOD cases that followed a similar approach—the Advanced Amphibious 

Assault Vehicle and the Virginia class attack submarine—have so far 

avoided problems with key technologies. Yet these cases are not the norm 

for DOD programs. DOD programs operate under conditions that make it 

more difficult—and less rewarding—to separate technology from product 

development and to allow technology to reach high maturity before being 

included in an acquisition program.

It is easier for weapon system programs to fund technology development at 

higher readiness levels because they attract much bigger budgets than 

science and technology projects. DOD typically does not fund science and 

technology organizations to take technology past the feasibility stage—

TRL 5. As a practical matter, it is often necessary to move immature 

technology to a weapon system program to get needed funds and 

management support. New programs are pressured to include immature 

technologies that offer significant performance gains. These pressures 

come from the user’s perception of the threat, technologists that see the 

program as an opportunity to apply a new technology, and funding 

competition that rewards weapon systems with unique features.

DOD and the services have several initiatives for improving the technology 

development process and reducing weapon system cycle times. These 

include defense technology objectives, advanced technology 

demonstrations, advanced concept technology demonstrations, the Army’s 

new scout/cavalry vehicle, and the Air Force’s Integrated High 

Performance Turbine Engine Technology Program. These initiatives are 

aimed at putting the science and technology organizations and funding in 
Executive Summary

Page 7 GAO/NSIAD-99-162 Best Practices

place to bring technologies to higher readiness levels before they are 

included in weapon system programs.

Recommendations GAO recommends that the Secretary of Defense adopt a disciplined and 

knowledge-based approach of assessing technology maturity, such as 

TRLs, DOD-wide, and establish the point at which a match is achieved 

between key technologies and weapon system requirements as the proper 

point for committing to the development and production of a weapon 

system. GAO also recommends that the Secretary (1) require that 

technologies needed to meet a weapon’s requirements reach a high 

readiness level (analogous to TRL 7) before making that commitment,

(2) extract lessons from successful technology inclusion cases for 

application to future technology inclusion efforts, and (3) empower 

program managers to refuse to accept key technologies with low levels of 

maturity by making decisions on individual programs that reinforce a best 

practice approach to technology maturation and inclusion. These 

recommendations appear in full in chapter 5.

Agency Comments DOD generally agreed with the report and its recommendations. A detailed 

discussion of DOD’s comments appear in appendix I.
Page 8 GAO/NSIAD-99-162 Best Practices

Contents

Letter

Executive Summary 2

Chapter 1

Introduction

Separating Technology Development From Product Development

Is a Best Practice 12

Technology and Product Development Conducted at the Same Time

Within DOD 15

Shorter Acquisition Cycle Times Are Needed 17

Objectives, Scope, and Methodology 18

Chapter 2

Maturity of Technology 

at Program Start Is an 

Important Determinant 

of Success

Technology Maturity Can Be Measured and Its Consequences for

Products Can Be Forecast 22

Technologies With High Readiness Levels at Launch Were Better

Able to Meet Product Objectives 25

Chapter 3

Controllable 

Conditions Affect How 

Well a Technology’s 

Inclusion on a Product 

Can Be Managed

Providing the Right Environment Is Critical to the Successful

Maturation of Technology 35

Good Technology Handoff Decisions Depend on the Tools and

Authority Given to Managers 41

Chapter 4

Impediments to 

Adopting Best 

Practices for 

Technology Inclusion 

in DOD Are 

Surmountable

Several Factors Make It Difficult to Mature Technologies Before

They Are Included on Weapon Systems 50

Services Encouraged to Use Best Practices 54

Two Unique DOD Projects May Provide Lessons on How to Enable

S&T Organizations to Manage Technology Further 56
Contents

Page 9 GAO/NSIAD-99-162 Best Practices

Chapter 5

Conclusions and 

Recommendations

Conclusions 61

Recommendations 63

Agency Comments and Our Evaluation 65

Appendixes Appendix I: Technology Readiness Levels and Their Definitions 68

Appendix II: Comments From the Department of Defense 69

Appendix III: GAO Contacts and Staff Acknowledgments 73

Related GAO Products 76

Tables Table 2.1: Cost and Schedule Experiences on Product

Developments 27

Table 3.1: TRLs of Technologies Managed by S&T Organizations 36
Contents

Page 10 GAO/NSIAD-99-162 Best Practices

Figures Figure 1.1: Cycle for Providing Users a Product With Better

Capabilities 13

Figure 1.2: DOD’s Weapon System Acquisition Cycle 16

Figure 1.3: Allocation of DOD’s Fiscal Year 1999 Research and

Development Funds 17

Figure 2.1: Using TRLs to Match Technology With Product Launch 

Requirements 24

Figure 2.2: Readiness Levels of Technologies at the Time They Were 

Included in Product Designs 26

Figure 2.3: Time Line for Ford’s Development of Voice Activated

Controls Technology 28

Figure 2.4: Jaguar 29

Figure 2.5: Virginia Class Attack Submarine 30

Figure 2.6: Brilliant Anti-Armor Submunition 32

Figure 2.7: Comanche Helicopter 33

Figure 3.1: Hughes Solar Cell Arrays 38

Figure 3.2: Integrated Avionics for Comanche Helicopter 40

Figure 3.3: Process for Closing the Gap Between the Readiness of

Adaptive Cruise Control Technology and Jaguar Requirements 43

Figure 3.4: Process for Closing the Gap Between the Readiness of 

Propulsion Technologies and AAAV Requirements 46

Figure 3.5: AAAV 47

Figure 3.6: Assimilation of New Technology Into the BAT Program 48

Figure 4.1: Airborne Laser 52

Figure 4.2: Comparison of Traditional Technology Development

Process With the Army’s Fast Track Approach 57

Figure 4.3: Future Scout and Cavalry System 59
Page 11 GAO/NSIAD-99-162 Best Practices

Abbreviations

AAAV Advanced Amphibious Assault Vehicle

ABL Airborne Laser

ACTD Advanced Concept Technology Demonstration

ATD Advanced Technology Demonstration

BAT Brilliant Anti-Armor Subminition

DARPA Defense Advanced Research Projects Agency

DEAL Deliverables Agreement Log

DOD Department of Defense

DTO defense technology objective

FLIR forward-looking infrared

NASA National Aeronautics and Space Administration

S&T science and technology

TRL technology readiness level
Page 12 GAO/NSIAD-99-162 Best Practices

Chapter 1

Introduction Chapte1r

A central piece of the National Military Strategy is the military capability 

represented by advanced weaponry. The Department of Defense (DOD) 

plans to increase its annual investment in new weapons to about $60 billion 

by fiscal year 2001—a 40-percent increase over fiscal year 1997. DOD has 

high expectations from this investment: that new weapons will be better 

and less expensive than their predecessors and will be developed in half 

the time. These expectations frame a great challenge for managers of 

programs. The traditional management approach—which has produced 

superior weapons but at much greater cost and time than planned—will not 

meet these expectations. Cycle times—the time to develop a new 

weapon—can be so long that the technology a weapon is designed with 

becomes obsolete before it can be produced. Costs of new weapons have 

reached the point that significantly fewer can be bought than planned. 

These are not new issues, but they have become more pressing as the pace 

and sophistication of foreign and commercial technology have increased, 

complicating a national security environment of unknown threats.

Leading commercial firms have changed the way they develop products 

and have achieved the kinds of results DOD seeks, often yielding more 

sophisticated products in half the time formerly needed. Industry experts 

estimate that resolving technology problems before product development 

begins results in 10 times the savings compared to correcting problems 

afterward. In this sense, technology maturity breeds product success. The 

practices leading firms use to mature and transition technology to products 

hold promise for DOD, for immature technologies have been main sources 

of problems on weapon systems. We have previously reported on the 

different elements of knowledge firms insist on to get better products to 

market faster. Of these, no element is more important than having 

technology, advanced enough to meet requirements but also mature 

enough to be predictably managed, available at the start of the product 

development cycle. Maturing new technology before it is included on a 

product is perhaps the most important determinant of the success of the 

eventual product—or weapon system. It is the topic of this report.

Separating Technology 

Development From 

Product Development 

Is a Best Practice 

The cycle for placing better capabilities in the hands of users—both 

military and commercial—can be described as consisting of technology
Chapter 1

Introduction

Page 13 GAO/NSIAD-99-162 Best Practices

development, product development, and production. In a 1998 report,1 we 

characterized the knowledge needed on a new product as consisting of 

three knowledge points: when a match is made between a customer’s 

requirements and the available technology; when the product’s design is 

determined to be capable of meeting performance requirements; and when 

the product is determined to be producible within cost, schedule, and 

quality targets (see fig. 1.1). We found that this knowledge, when obtained 

at the right time and in the right sequence—technology, design, and 

manufacturing—was a best practice. This practice lowered product 

development risks, reduced cycle times and costs, and resulted in 

smoother production programs. 

Figure 1.1: Cycle for Providing Users a Product With Better Capabilities

Leading commercial firms recognize a distinct difference between 

technology development and product development; accordingly, they 

develop technology before introducing it into product development 

programs. They minimize risk, improve cost and schedule outcomes, 

reduce cycle time, and improve quality during product development by 

1

Best Practices: Successful Application to Weapon Acquisitions Requires Changes in DOD’s

Environment (GAO/NSIAD-98-56, Feb. 24, 1998).

Technology 

Development

Product

Development Production

Technology 

readiness

Design

maturity

Knowledge

Point 1

Knowledge

Point 2

Knowledge

Point 3

Product

launch

Control of

manufacturing

process
Chapter 1

Introduction

Page 14 GAO/NSIAD-99-162 Best Practices

gaining significant knowledge about a technology before launching the 

product development. Scientists and technologists—different people than 

those that manage product developments—manage the development of 

technology until it is ready to be included in the design of a product. 

Program launch is the point at which a firm defines a product’s 

performance, cost, and schedule estimates and begins making a large 

investment in human capital, facilities, and materials—an investment that 

increases continuously as the product approaches the point of 

manufacture. It includes a commitment to manufacture the product. 

Therefore, program launch and the start of product development are 

synonymous within commercial firms. Protecting this investment provides 

a strong incentive for firms to minimize the potential for technology 

development problems during the product phase and cause delays. 

Confining delays in maturing technology to a time prior to launch—in an 

environment where small teams of technologists work in laboratories and 

are dedicated to perfecting the technology—is critical to saving time and 

money. If delays occur during product development, when a large 

engineering force is in place to design and manufacture the product, they 

would be much more costly. In fact, industry experts estimate that 

identifying and resolving a problem before product development can reap a 

10-fold savings compared to correcting the problem after launch and that 

correcting the same problem in the manufacturing stage would be even 

more costly. 

Leading commercial firms have found that managing technology 

development separately from and before product development is a major 

reason they have been able to reduce product cycle times. As a whole,

50 to 70 percent reductions in cycle times are not unrealistic achievements 

by leading commercial firms. For instance, leaders in the automobile 

industry have reduced cycle times from 7 years to 2 years, or by about

70 percent. The consumer electronics industry has recently reduced its 

cycle time from 2 years to 6 months, and the commercial aircraft industry 

has achieved reductions of 50 percent. Leading commercial firms have 

found that reducing the product development cycle time brings products to 

market faster, results in an increased market share, and helps to keep 

products from becoming technologically obsolete. 
Chapter 1

Introduction

Page 15 GAO/NSIAD-99-162 Best Practices

Technology and 

Product Development 

Conducted at the Same 

Time Within DOD 

DOD’s process for developing and manufacturing weapon systems is 

described as a cycle consisting of phases. These phases are concept 

exploration, program definition and risk reduction, engineering and 

manufacturing development, and production and fielding. The basic 

process of gathering knowledge about technology, design, and 

manufacturing is followed, but in practice, the DOD cycle does not make a 

clear distinction between technology development and product 

development. The launch of a program in DOD usually takes place several 

years before the beginning of product development does in leading 

commercial firms. 

In fact, a new weapon system program is normally launched at the start of 

the program definition and risk reduction phase, which is often in the midst 

of technology development, while most product development activities do 

not begin until the engineering and manufacturing development phase. 

Consequently, technology, design, and manufacturing knowledge is 

attained concurrently--in the higher cost environment that characterizes 

product development--throughout the weapon system phases. In our 

February 1998 report, we noted that such technology development 

problems are a major cause of cost increases and schedule delays on DOD 

weapon system programs. The phases in DOD’s weapon system acquisition 

cycle and the knowledge gathering process, as it is typically followed, are 

shown in figure 1.2.
Chapter 1

Introduction

Page 16 GAO/NSIAD-99-162 Best Practices

Figure 1.2: DOD’s Weapon System Acquisition Cycle

DOD’s process also has organizational and budgetary implications. 

Activities accomplished in the first three phases of the acquisition cycle use 

research and development funds whereas production programs use 

procurement funds. Generally, DOD’s science and technology (S&T) 

community is responsible for basic research, applied research, and 

advanced technology development to produce generic, rather than 

weapon-specific, technologies. Its goal is to conduct research, develop 

technology, and farm these efforts for potential military application, such 

as a weapon system. The S&T community also uses research and 

development funds, but its work generally precedes the acquisition cycle. 

Weapon system program managers, who receive most of DOD’s research 

and development budget, apply generic technologies to specific weapon 

systems. However, they often become responsible for completing 

development of generic technologies as well. The allocation of DOD’s fiscal 

year 1999 research and development funds to these categories is shown in 

figure 1.3.

Concept 

exploration

Program 

definition and

risk reduction

Engineering and

manufacturing

development

Production 

and fielding

Program launch Begin product 

development

Technology maturity

Design maturity

Manufacturing processes controlled

Knowledge

attainment
Chapter 1

Introduction

Page 17 GAO/NSIAD-99-162 Best Practices

Figure 1.3: Allocation of DOD’s Fiscal Year 1999 Research and Development Funds

Source: DOD

S&T officials stated their role is to show that technology is feasible through 

laboratory experiments or demonstrations. It is often at this point that the 

technology’s military potential will be identified and the technology will be 

harvested for inclusion on a weapon system. Because the technology is still 

not mature, its development will be completed as part of the weapon 

system’s design and development, under the authority of the weapon 

system manager and apart from the S&T community. 

Shorter Acquisition 

Cycle Times Are 

Needed

DOD’s weapon acquisition cycle times average between 10 to 15 years—far 

longer than the cycle time for commercial products. To an extent, DOD’s 

cycle times are longer because they start earlier than commercial cycles 

and often entail more complex products. Compounding the length of the 

weapon system development cycle is its unpredictability. Over the years, 

we have issued numerous reports highlighting cost overruns and schedule 

delays during the product development cycle, for which technology 

development problems were a major cause. These problems require 

additional technology development activities to take place at a time when 

3%

79%

9%

9%

Weapon specific

development - $29.6 billion

Basic research - $1.1 billion

Applied research - $3.2 billion

Advanced technology 

development - $3.5 billion
Chapter 1

Introduction

Page 18 GAO/NSIAD-99-162 Best Practices

the product should be undergoing design and manufacturing development. 

As a result, the pace of technology advances outruns the time to develop a 

weapon system and some of the more mature components designed into a 

weapon system become obsolete before the weapon is manufactured. For 

example, the F-22 will have almost 600 obsolete components by fiscal year 

2000 while the aircraft is still in development.

The longer a weapon system’s development cycle, the more prone the 

program is to management and funding changes. According to DOD, an 

11-year development program historically encounters a 30-percent cost 

growth over time. Based on historical averages, DOD calculates that the 

typical program will have four different program managers, eight defense 

acquisition executives, and seven Secretaries of Defense—all of who are 

major influences and decisionmakers on the program. In addition, the 

program will have gone through 11 annual budget cycles in which funding 

changes could have occurred and affected the program’s content.

The Under Secretary of Defense for Acquisition and Technology has stated 

that cycle time reduction is necessary to meet DOD’s goals of delivering 

emerging technologies to warfighters in less time and at lower costs. The 

Under Secretary has set a goal to reduce the average acquisition cycle time 

for all program starts in fiscal year 1999 and beyond by 50 percent over 

historical averages. Reductions in cycle times will (1) allow for earlier 

fielding of increased capabilities, (2) reduce costs, (3) free up funds for 

more programs, (4) reduce the potential for components becoming 

obsolete, and (5) take more frequent advantage of technology advances 

found in the commercial world. An emphasis on shorter cycle times may 

also reduce the tendency to add technological advances that are unproven 

and immature into weapon acquisition programs. To help achieve this goal, 

DOD is working on several efforts such as Defense Acquisition Pilot 

Programs, the Defense Reform Initiatives, and many acquisition reform 

projects. The Under Secretary has also advocated adopting the practices of 

leading commercial firms and taking a more evolutionary approach to 

developing weapon systems, which would lessen the amount of technology 

development initially attempted within a weapon system program.

Objectives, Scope, and 

Methodology

The Chairman and the Ranking Minority Member, Subcommittee on 

Readiness and Management Support, Senate Committee on Armed 

Services, requested that we examine various aspects of the acquisition 

process to determine whether the application of best practices can improve 

program outcomes. To date, we have issued reports on advanced quality 
Chapter 1

Introduction

Page 19 GAO/NSIAD-99-162 Best Practices

concepts, earned value management, management of a product from 

development to production, and management of key suppliers (see related 

GAO products). This report covers the inclusion of technology into weapon 

system programs, and is, in a sense, a prequel to our report on product 

development. Our overall objective was to determine whether best 

practices offer methods to improve the way DOD matures new technology 

so that it can be assimilated into weapon system programs with less 

disruption. Specifically, we assessed (1) the impact of technology maturity 

on product outcomes, (2) best practices for managing new technologies 

and incorporating them into products, and (3) ways DOD can adapt best 

practices to achieve better outcomes on weapon system programs.

Our methodology consisted of analyzing 23 commercial and DOD 

technologies that had transitioned or attempted to transition into product 

development programs. The technologies were drawn from six commercial 

firms recognized for their success in developing technically advanced 

products more quickly than their predecessors and five weapon system 

programs that incorporated advanced technologies, including some that 

did not encounter problems and some that did. We asked the managers of 

these technologies to apply a tool, referred to as technology readiness 

levels (TRLs), for our analysis. The managers used TRLs to judge the 

maturity of the technologies at the time they had entered product 

development or were included in programs. The National Aeronautics and 

Space Administration (NASA) originally developed TRLs, and the Air Force 

Research Laboratory uses them to determine when technologies are ready 

to be handed off from S&T managers to product development managers. 

We held discussions with the DOD and NASA users of TRLs to better 

understand their applicability to our review. They stated that TRLs can be 

used as general indicators of a technology’s readiness level and associated 

risk of including the technology into a product development program, given 

its TRL at that time. TRLs are more fully explained in chapter 2. 

To understand the best practices the commercial sector used to include 

technologies in product development programs, we conducted literature 

searches and focused those searches as the review progressed. On the 

basis of the searches, we identified a number of commercial firms with 

innovative technology development processes for including new or 

advanced technologies into new products. We used structured interview 

questions sent in advance of our visits to gather uniform and consistent 

information about each firm’s process and practice and the results 

achieved. In addition, we examined four specific technology cases—Ford’s 

night vision, adaptive cruise control, and voice activated controls and 
Chapter 1

Introduction

Page 20 GAO/NSIAD-99-162 Best Practices

Hughes’ solar cell array—to better understand their processes and 

practices. The commercial firms we visited were

• Ethicon-Endo Surgery (medical device manufacturer), Division of 

Johnson and Johnson, Cincinnati, Ohio;

• Ford Motor Company (automobile manufacturer), Dearborn, Michigan; 

• Harris Semiconductor (semiconductor manufacturer), Melbourne, 

Florida;

• Hughes Space and Communications (satellite and spacecraft 

manufacturer), Los Angeles, California;

• 3M (commercial products manufacturer), St. Paul, Minnesota; and

• Motorola Corporate Research Headquarters (communications 

technology manufacturer), Schaumburg, Illinois, and Motorola Land 

Mobile Products Sector, Plantation, Florida.

We also attended and participated in conferences and workshops with 

recognized leaders in the acquisition field to obtain information on how 

organizations are improving their acquisition processes. Finally, we 

interviewed officials from trade organizations concerning the application 

of commercial practices to DOD operations.

To better understand DOD’s technology inclusion process, we selected 19 

advanced technologies that had been included in 5 DOD weapon system 

programs that were in various stages of the acquisition process. We 

collected technical reports, acquisition management, and risk management 

documentation about the technologies. In addition, we interviewed S&T 

and acquisition program management officials about each technology’s 

development history, costs, and current status. The technologies and 

programs reviewed were 

• acoustic sensor, infrared seeker, inertial measurement unit, tandem 

shaped charge warhead, and processor technologies from the Army’s 

Brilliant Anti-Armor Submunition (referred to as BAT) Program;

• rotor, engine, integrated avionics, forward looking infrared, and helmet 

mounted display technologies from the Army’s Comanche Helicopter 

Program;

• nonpenetrating periscope and weapon ejection system technologies 

from the Navy’s Virginia class attack submarine program; 

• high speed planing craft, power dense diesel engine, lightweight 

composite armor, high power water jet, moving map and advanced 

navigation technologies from the Marines’ Advanced Amphibious 

Assault Vehicle (AAAV) Program; and
Chapter 1

Introduction

Page 21 GAO/NSIAD-99-162 Best Practices

• laser and beam control technologies from the Air Force’s Airborne Laser 

(ABL) Program.

To determine relevant DOD policy and initiatives, we obtained documents 

and interviewed officials of the Office of the Secretary of Defense; the 

Defense Advanced Research Projects Agency (DARPA); and Army, Navy 

and Air Force Science and Technology organizations. We also had 

discussions with former DOD officials and industry experts about DOD 

acquisition policies and practices.

Even though we selected firms with product lines of varying complexity, 

we did not concentrate only on firms whose products had the most in 

common with weapon systems. Such an approach would have limited our 

ability to include firms recognized as the best at including new, advanced 

technologies into programs. In our analysis, we concentrated on the 

criteria and knowledge used to support technology readiness decisions. 

Although the approach from product to product may vary, the basic 

processes and standards leading commercial firms applied to technology 

inclusion decisions were consistent. We were limited, however, in our 

ability to obtain and present some relevant data that commercial 

companies considered proprietary in nature. This information included 

funding amounts for investing in technology development, details on 

technological innovations, and some specific data from recent technology 

inclusion successes. Our report highlights the best commercial practices 

for including technology into product development programs. As such, they 

are not intended to describe all commercial industry practices or to suggest 

that commercial firms do not have any flaws.

We conducted our review between March 1998 and June 1999 in 

accordance with generally accepted government auditing standards.
Page 22 GAO/NSIAD-99-162 Best Practices

Chapter 2

Maturity of Technology at Program Start Is an 

Important Determinant of Success Chapte2r

The experiences of the DOD and commercial technology development 

cases we reviewed indicate that demonstrating a high level of maturity 

before allowing new technologies into product development programs puts 

those programs in a better position to succeed. Simply put, the more 

mature technology is at the start of the program, the more likely the 

program will succeed in meeting its objectives. Technologies that were 

included in a product development before they were mature later 

contributed to cost increases and schedule delays in those products.

We found an analytical tool—TRLs—that can assess the maturity level of 

technology as well as the risk that maturity poses if the technology is 

included in a product development. The tool associates different TRLs with 

different levels of demonstrated performance, ranging from paper studies 

to proven performance on the intended product. The value of using the tool 

is that it can presage the likely consequences of incorporating a technology 

at a given level of maturity into a product development, enabling 

decisionmakers to make informed choices. TRLs proved to be reliable 

indicators of the relative maturity of the 23 technologies reviewed, both 

commercial and military, and their eventual success after they were 

included in product development programs. 

Technology Maturity 

Can Be Measured and 

Its Consequences for 

Products Can Be 

Forecast

Successful technologies progress from initial concept to proven 

performance, whether they are developed in the laboratory or in the 

factory, by commercial industry or DOD. The Air Force Research 

Laboratory has adapted and uses TRLs to measure the key steps in this 

progression toward inclusion into weapon systems. TRLs are measured 

along a scale of one to nine, starting with paper studies of the basic concept 

and ending with a technology that has proven itself in actual usage on the 

intended product. A detailed description of TRLs is provided in

appendix II, but the following hypothetical example about an airborne 

communications radio can illustrate the readiness levels.

First, the idea for a new radio is conceived. The idea reaches TRL 3 when 

analytical studies and some tests of the technology’s elements, such as a 

circuit, back it up. When initial hand-built versions of all of the radio’s basic 

elements are connected and tested together, the radio reaches TRL 5. This 

is sometimes referred to as a “breadboard” article; although it may function 

like a radio, it does not look like one because the individual parts are 

attached to plywood and hand-wired together. When the technology is built 

into a generic model, which is well beyond the breadboard tested in TRL 5, 

and demonstrated in a laboratory environment, the radio reaches TRL 6. 
Chapter 2

Maturity of Technology at Program Start Is 

an Important Determinant of Success

Page 23 GAO/NSIAD-99-162 Best Practices

This model represents the last level of demonstration before the radio 

becomes tailored for application to a specific aircraft. When the 

components are assembled inside a case that resembles the final radio 

design and are demonstrated aboard a surrogate for the intended aircraft, 

the radio reaches TRL 7. TRL 8 is reached when the radio is put in its final 

form, installed in the intended aircraft’s cockpit, and tested in conjunction 

with the other aircraft equipment with which it must interface. TRL 9 is 

achieved when the radio is successfully operated on the aircraft through 

several test missions. Unexpected problems can arise at every level, and 

effort must be expended to overcome them. This effort takes time and can 

delay the progress to the next readiness level.

Once a technology’s readiness level has been established, the risks of 

including that technology in a product development can be assessed. 

Unlike S&T projects, for which the main objective is to develop knowledge, 

a product development’s objective is to deliver products that meet strict 

cost, schedule, and performance targets. We found that most leading 

commercial firms, after they had translated their own methods of assessing 

risk into TRLs, determined that a TRL 8 was required before they allowed a 

new technology into a product development.1

 DOD launches a program in 

the program definition and risk reduction phase—much earlier than the 

leading commercial firms do. According to the Air Force Research 

Laboratory, a TRL 6 is required for a technology to be an acceptable risk for 

a program in that phase. When weapon system development reaches the 

engineering and manufacturing development phase, it more nearly 

approximates the point at which a commercial product development 

program would start. The Air Force Research Laboratory depicts a 

technology at TRL 7 as an acceptable risk for this phase—technologies at 

lower levels would be considered high risks.

The lower the level of technology readiness, the more ground must be 

covered to bring the technology to the point at which it can meet the 

intended product’s cost, schedule, and performance requirements with 

little risk (see fig. 2.1).

1

An exception to this is space systems technology. Space-based technologies are generally included on a 

development program once they have been prototyped and ground tested—a TRL 6, the highest level 

attainable short of space operation. 
Chapter 2

Maturity of Technology at Program Start Is 

an Important Determinant of Success

Page 24 GAO/NSIAD-99-162 Best Practices

Figure 2.1: Using TRLs to Match Technology With Product Launch Requirements

The gap between the maturity of the technology and the product’s 

requirements represents the risks or unknowns about the technology. As 

each succeeding level of readiness is demonstrated, unknowns are 

replaced by knowledge and the gap becomes smaller. Ideally, the gap is 

closed before a new technology is included in a new product’s design, 

although the Air Force Research Laboratory accepts the amount of risk at 

TRL 7 for a program entering engineering and manufacturing development. 

Technologies that reach TRL 7 or higher at the start of product 

development allow product managers to focus their attention on 

integrating the technologies and proving out the product design. 

Technologies that are included at lower maturity levels require more of the 

product managers’ attention and resources, as basic knowledge about 

those technologies must still be gained.

Thus, a major purpose served by TRLs is to reveal the gap between a 

technology’s maturity and the maturity demanded for successful inclusion 

in the intended product. With TRLs as guides, the options available to 

decisionmakers can be framed. Given that a key determinant of achieving 

cost and schedule outcomes for a product development is the technology’s 

maturity at product launch, decisionmakers can either (1) delay product 

Product

Requirements

TR L

High risk for

product launch

Low risk for

product launch

1 2 3

4 5

 6 7

8 9

Risks or

unknowns
Chapter 2

Maturity of Technology at Program Start Is 

an Important Determinant of Success

Page 25 GAO/NSIAD-99-162 Best Practices

development until the technology is matured to a high enough readiness 

level or (2) reduce the product’s requirements so that a less advanced, but 

more mature, technology can suffice. If it is perceived that the 

requirements of the product cannot be lowered and the product launch 

cannot be delayed until the requisite technology is of a sufficient readiness 

level, then the remaining option is to launch the product development with 

the immature technology. If this option is chosen, then the success of the 

product development will depend heavily on the product manager’s ability 

to simultaneously close the technology maturity gap and develop the 

product for manufacture, which is a very challenging task.

TRLs do not represent strictures that must be adhered to without 

exception. According to the people in DOD who have used TRLs, there are 

occasions when a lower than expected TRL can be accepted, such as when 

the product development’s schedule and resources are generous enough 

that the technology will have enough time to mature. In other instances, a 

higher than expected TRL may be required, such as if the technology in 

question is the linchpin for the entire product. Nonetheless, we found that 

TRLs ably reconciled the different maturity levels and product experiences 

of the 23 technologies reviewed. 

Technologies With 

High Readiness Levels 

at Launch Were Better 

Able to Meet Product 

Objectives 

The 23 technologies reviewed spanned a wide range of readiness levels at 

the time they were included in product development programs. The least 

mature reached TRL 2 at the time it was included in a product 

development, while the most mature had reached TRL 9 at the point of 

inclusion. We observed a general relationship between TRLs and the 

technologies’ inclusion on the intended product developments. Those 

products whose technologies reached high TRLs at the time they were 

included were better able to meet cost, schedule, and performance 

requirements. In fact, commercial firms informed us that maturing the 

technology separately from and ahead of the product was a main reason 

they were able to reduce cycle times on their products. An official from one 

of the firms termed the approach as “moving discovery to the left.”

Those technologies with low TRLs at inclusion encountered maturation 

difficulties and contributed to problems the products experienced. Other 

problems, such as funding and schedule changes unrelated to the 

technologies, also contributed to problems in the product developments. 

Figure 2.2 shows the TRLs when each of the 23 technologies was included 

in a product design, whether at product development launch (for 

commercial technologies) or at program launch (for DOD technologies).
Chapter 2

Maturity of Technology at Program Start Is 

an Important Determinant of Success

Page 26 GAO/NSIAD-99-162 Best Practices

Figure 2.2: Readiness Levels of Technologies at the Time They Were Included in Product Designs

1 2 3 4 5 6 7 8 9

Solar cell array

Non-penetrating periscope

Adaptive cruise control

Night vision

Voice activated control

High-speed planing craft

High power water jet

Weapon ejection system

Diesel engine

Helicopter rotor

Lightweight composite armor

Helicopter engine

Moving map and navigation

Hemical oxygen iodin laser

Beam control system

Helmet mounted display

Helicopter forward linking infrared

Integrated avionics

Data processor

Inertial measurement unit

Warhead

Infrared seeker

Acousting targeting sensor

Technology Readiness Levels

Commercial technologies

DoD technologies
Chapter 2

Maturity of Technology at Program Start Is 

an Important Determinant of Success

Page 27 GAO/NSIAD-99-162 Best Practices

The cost and schedule experiences of some of the products or programs 

that inherited the technologies are shown in table 2.1. 

Table 2.1: Cost and Schedule Experiences on Product Developments 

a

The Comanche, in particular, has experienced a great deal of cost growth and schedule slippage for 

many reasons, of which technology immaturity is only one. Other factors, such as changing the scope, 

funding, and pace of the program for affordability reasons, have also contributed.

Data for three weapon system development programs, the Virginia class 

attack submarine, AAAV, and the ABL, were not included in the table 

because they had not been in the product development phase long enough 

to report actual consequences. To date, AAAV and the submarine have 

stayed within 15 percent of their cost and schedule estimates for 

development. The ABL, for which key technologies were much less mature 

at program launch, still faces challenges with these technologies. Ford’s 

night vision technology was excluded because the firm decided not to 

include the technology on a product. Details on the Comanche, BAT, the 

Virginia class attack submarine, and Ford technology and product 

experiences follow. 

Product development

Product development and 

associated technologies

TRL at

program

launch Cost growth Schedule slippage

Comanche helicopter 

 Engine

 Rotor

 Forward looking infrared

 Helmet mounted display

 Integrated avionics

5

5

3

3

3

101 percenta 120 percenta

BAT

 Acoustic sensor

 Infrared seeker

 Warhead

 Inertial measurement unit

 Data processors 

2

3

3

3

3

88 percent 62 percent

Hughes HS-702 satellite

 Solar cell array 6

None None

Ford Jaguar

 Adaptive cruise control

 Voice activated controls

8

8

None None
Chapter 2

Maturity of Technology at Program Start Is 

an Important Determinant of Success

Page 28 GAO/NSIAD-99-162 Best Practices

Technology and Product 

Experiences on Ford and 

Virginia Class Attack 

Submarine 

The key technologies for the Ford Jaguar and the Virginia class attack 

submarine followed the pattern of increasing TRLs until they demonstrated 

a low risk for transition to the product. Two examples are Ford’s voice 

activated controls development and DARPA’s nonpenetrating periscope 

development for the submarine. In both cases, the technologies were 

validated, operational prototypes demonstrated, and the technologies had 

demonstrated the form, fit, and function of the final article by the beginning 

of product development. 

Ford’s voice activated controls technology, which allows a driver to control 

certain functions such as windows and the radio through verbal 

commands, was under development in the technology base for over

10 years, being pushed by the firm’s technology leaders. It was not until 

1993 that Ford found that (1) other complementary technologies, such as 

processor speeds and low cost memory, had become available and

(2) customers wanted more features and functions but less distractions 

from driving. Given this market information, Ford decided to pursue voice 

technology as a strategic technology in terms of product differentiation, 

recognizing the importance of being first to market with this enabling 

technology. Figure 2.3 shows the time line for developing this technology.

Figure 2.3: Time Line for Ford’s Development of Voice Activated Controls Technology

Between 1993 and 1994, based on discussions with customers, Ford 

developed cost and performance requirements for the technology. Ford has 

1983

Ford decides to pursue

voice activated controls

technology. Technology

under early development

in technology base.

TRL 3 - 5

1993

Technology is linked

to a specific vehicle.

Cost and performance

requirements are

defined.

TRL 6 - 7

Technology is ready to

transition into a product

development program.

Technology meets all

cost and schedule targets

for the product.

TRL 8

1995

Technology featured

on model year 1999

Jaguar designs.

TRL 9

1999
Chapter 2

Maturity of Technology at Program Start Is 

an Important Determinant of Success

Page 29 GAO/NSIAD-99-162 Best Practices

never relaxed them. By September 1995, when Ford allowed the technology 

into the development program for a new Jaguar design, voice activated 

controls had been demonstrated as an integrated system in the appropriate 

form and fit for the Jaguar. Ford officials stated that the product has met all 

cost and cycle time targets established at the outset of its development. 

Figure 2.4 shows the Jaguar.

Figure 2.4: Jaguar

Ford demonstrated voice activated control technology in the appropriate form and fit before 

incorporating it into the Jaguar.

Source: Ford Motor Company.

DARPA began developing the nonpenetrating periscope technology as part 

of its submarine technology development efforts after recognizing, in 1988, 

along with the Navy, that the nonpenetrating technology would enhance 

operator visibility, provide greater submarine design flexibility, and be 

stealthier than conventional masts and periscopes. At the time, the Virginia 

class attack submarine program had not been initiated. Once the decision 

was made to include the nonpenetrating periscope, it became a key feature 

of the submarine and was a major design driver for the submarine’s overall 

configuration.

Nonpenetrating refers to the fact that the periscope is essentially a group of 

sensors that are linked to the submarine via fiber optic and other cables. 

This technology uses infrared imaging and advanced sensors to replace 

conventional periscopes and frees up physical space compared with a 

conventional periscope. A conventional periscope relies on a series of 

telescoping shafts and reflecting surfaces to see above the water’s surface. 
Chapter 2

Maturity of Technology at Program Start Is 

an Important Determinant of Success

Page 30 GAO/NSIAD-99-162 Best Practices

When the periscope is retracted, the shafts take up a column of space from 

the top of the submarine to the bottom, through all decks. Its location 

virtually dictates the design and placement of the control and other rooms. 

If the nonpenetrating periscope technology did not become available, then 

the submarine would have to be drastically redesigned to accommodate the 

space required by a conventional periscope. 

The new nonpenetrating periscope and photonics mast technology 

underwent land testing in 1991—a TRL 5. The Navy actually tested the new 

technology at sea on the U.S.S. Memphis in 1992 and 1993. According to 

program officials, these sea trials demonstrated the highest level of 

technology readiness: proving the actual system through successful 

mission operations. This readiness equated to a TRL 9. Yet, this technology 

was not included in the Virginia class attack submarine requirements until 

1995. Figure 2.5 shows an artist’s concept of the Virginia class attack 

submarine. 

Figure 2.5: Virginia Class Attack Submarine

The Navy demonstrated a key technology at the highest readiness level before including it as a 

requirement for the Virginia class attack submarine.

Source: DOD. 

The high readiness level of the nonpenetrating periscope afforded the Navy 

the opportunity to develop an improved version of the periscope to a

TRL 9. This was a relatively low-risk endeavor as the baseline periscope 

was sufficient to meet the submarine’s requirements. Program officials 

believe that having knowledge about key technologies, such as the 

nonpenetrating periscope, for the Virginia class attack submarine at 

program launch made a short program definition and risk reduction phase 

possible. This phase for the Virginia class attack submarine was about

75 percent shorter than those of previous acquisition programs. Based on 

its demonstrated maturity, we anticipate that the nonpenetrating periscope 

to be less likely to impact the cost and schedule of the submarine’s 

development program. There are, however, several other technologies that 
Chapter 2

Maturity of Technology at Program Start Is 

an Important Determinant of Success

Page 31 GAO/NSIAD-99-162 Best Practices

are critical to the submarine program. We did not examine these 

technologies and cannot predict their likely outcomes.

BAT and Comanche Cases Key technologies for the BAT and Comanche programs had much lower 

readiness levels at the time the product developments were launched. 

Consequently, they did not reduce the gap between their demonstrated 

maturity and the maturity needed to meet product requirements until after 

program launch. For some technologies, the gap was not closed until well 

into the product development program. For others, the gap has still not 

been closed. 

Five key technologies included in the Army’s BAT program had low TRLs 

when they were included on the program. The level of readiness for most of 

these technologies at program launch was characterized by the program 

office as experimental in nature but with major uncertainty remaining—a 

TRL 3. The acoustic targeting technology was the most important enabling 

technology needed to meet the weapon’s performance requirements. This 

technology provides BAT the capability to locate targets from great 

distances based on the sounds generated by the target, such as moving 

tanks and vehicles. At the time the program was launched, the Army knew 

little about the feasibility of using this technology on this program. In fact, 

the technology was still being defined in paper studies—a TRL 2. The 

Army did not prototype this technology until after the program had entered 

the engineering and manufacturing development phase, more than 6 years 

after program launch. As of December 1998, the BAT had experienced 

significant development cost and schedule increases, which program 

officials attribute at least, in part, to unknowns about the new technologies. 

Figure 2.6 shows the BAT.
Chapter 2

Maturity of Technology at Program Start Is 

an Important Determinant of Success

Page 32 GAO/NSIAD-99-162 Best Practices

Figure 2.6: Brilliant Anti-Armor Submunition

At the time the program was launched, the Army knew relatively little about the performance of several 

key technologies for the BAT

Source: DOD.

Two technologies key to meeting the Comanche helicopter’s requirements–

integrated avionics and forward-looking infrared (FLIR) technologies—

were included on the program when they were still conceptual in nature. 

The integrated avionics technology replaces individual radios, navigation, 

and other communication equipment with a modular system that shares a 

common processor. The FLIR is a second-generation version that uses 

infrared sensors to improve the pilot’s ability to see at night and in bad 
Chapter 2

Maturity of Technology at Program Start Is 

an Important Determinant of Success

Page 33 GAO/NSIAD-99-162 Best Practices

weather. Program officials stated that both had TRLs of 3 when the 

helicopter program was started. Despite the low readiness levels of the 

technologies, the Army included the technologies on the program to meet 

weight, cost, and performance requirements. 

The development of these technologies has taken longer than the Army 

expected it would. The contractor for the integrated avionics has had 

difficulties in getting the multiple avionics modules to work simultaneously 

within required size and weight parameters, and the FLIR technology has 

undergone several design and performance requirement changes. As of 

September 1998—approximately 10 years after program launch—neither 

the integrated avionics nor the FLIR technology had advanced past a TRL 5. 

Problems with the maturation of these technologies have contributed to 

the program’s cost and schedule increases. In contrast, the advanced rotor 

and engine technologies, which were the most mature of the Comanche 

technologies we reviewed, have experienced fewer problems in maturation 

and have not contributed significantly to the program’s cost and schedule 

increases. Figure 2.7 shows the Comanche.

Figure 2.7: Comanche Helicopter

The Army included two key technologies in the Comanche when they were still considered conceptual 

to meet weight, cost, and performance requirements

Source: DOD.
Page 34 GAO/NSIAD-99-162 Best Practices

Chapter 3

Controllable Conditions Affect How Well a 

Technology’s Inclusion on a Product Can Be 

Managed Chapte3r

Closing the gap between technology maturity and product requirements 

before a product is launched—and baselines are set—distinguished the 

more successful cases. Notably, closing the gap before product launch was 

a managed result; it put product managers in a better position to succeed. 

Two conditions were critical to achieving this kind of result. First was an 

environment that put the primary responsibility for maturing technology in 

the hands of S&T managers and provided them considerable flexibility to 

make decisions. Second was having the quality information and standards 

needed to make good technology handoff decisions, coupled with giving 

the product manager the authority to refuse new technology that did not 

meet product requirements. When these conditions were not present, the 

handoff to the product manager was compromised, with negative 

consequences for both technology and product. 

In each of the successful cases, S&T organizations played major roles in 

bridging the gap between technology maturity and product requirements. 

Flexibility provided by requirements communities and resource providers 

enabled S&T and product managers to delay the inclusion of technology if 

it was not ready or to reduce product requirements to match what mature 

technology could deliver. This environment was better suited to the 

unexpected results and delays that accompany technology development. 

Moreover, technology maturation was managed within a disciplined 

process that provided good information to be judged against clear and high 

standards, like TRLs. Armed with the tools and the authority to make 

technology inclusion decisions, both S&T and product managers 

functioned as gatekeepers to safeguard the product development.

In the more problematic cases, S&T organizations disengaged much earlier, 

and product managers had little choice but to accept immature 

technologies. Accordingly, less information about the technologies was 

available at the point of inclusion. Often, the tools used to assess the 

technologies’ status failed to identify high risks. In retrospect, TRLs 

indicated that risks were in fact high and perhaps unacceptable from a 

product standpoint. Also, pressures to meet cost and schedule estimates in 

product development provided a less forgiving environment for 

technologies in the discovery process.
Chapter 3

Controllable Conditions Affect How Well a 

Technology’s Inclusion on a Product Can Be 

Managed

Page 35 GAO/NSIAD-99-162 Best Practices

Providing the Right 

Environment Is Critical 

to the Successful 

Maturation of 

Technology

While most new technologies—commercial and military—are initially 

managed by the S&T community, the more successful cases we reviewed 

continued to be managed by S&T organizations until they reached at least 

TRL 6 and more often TRL 8 or higher. These technologies were provided 

the environmental advantages an S&T project has over a product 

development. This environment availed S&T managers and product 

managers of the less risky options of waiting or trading to get the match 

between technology maturity and product requirements—rather than 

forcing the product launch and gambling on the completion of technology 

maturity. In contrast, the more problematic technologies did not have as 

benign an environment. Often, the technologies were handed off early by 

S&T organizations because inflexible performance requirements for the 

product demanded their inclusion. Product development managers 

launched the product development and hoped that the technology 

development would succeed. 

Once in a product development environment, external pressures to keep 

the program moving become dominant, such as preserving cost and 

schedule estimates to secure budget approval. For example, DOD policies 

require that a program be funded in the current year and that funds be 

made available over the next 6 years in the DOD planning cycle. If, during 

the program definition phase, a program manager were to decide that an 

additional year was needed to overcome unexpected technology problems 

to reach the desired level of maturity, the delay could push the start of 

engineering and manufacturing development back. This delay could 

jeopardize the funding for that phase, thus risking the funding support for 

the entire program. Consequently, the program manager may be more likely 

to accept the risk of not getting the technology to the desired level of 

maturity and starting the engineering and manufacturing development 

phase as planned, rather than risk the rest of the program. These conditions 

compete with and detract from the needs of technology development. One 

acquisition official stated that these conditions cause the weapon system 

program “to pull double duty,” inventing new technology while integrating 

it into a product. In general, he believed there is an equal amount of 

difficulty in both tasks. 

Technologies Matured by 

S&T Organizations Made 

Smooth Transitions into 

Product Developments

In the most successful cases that we reviewed, S&T organizations bridged 

the gap between immature technology and the maturity needed for either 

program start in DOD (TRL 6) or product development (TRL 7 or higher). 

These cases and the responsible organizations are shown in table 3.1. 
Chapter 3

Controllable Conditions Affect How Well a 

Technology’s Inclusion on a Product Can Be 

Managed

Page 36 GAO/NSIAD-99-162 Best Practices

Table 3.1: TRLs of Technologies Managed by S&T Organizations

Despite the different circumstances between the commercial and DOD 

sectors and among the DOD cases themselves, the results were similar: 

having S&T organizations bridge the maturity gap reduced 

technology-related problems in the products. For the leading commercial 

firms we visited, it is standard practice to have S&T organizations 

responsible for the bridge. In the DOD cases shown in table 3.1, the S&T 

organizations played atypical roles in managing the bridge between 

technology and product by delivering the technology to a TRL 6 or higher. 

Different pressures and incentives that are brought to bear on the 

commercial and DOD product developments explain why DOD product 

managers become responsible for more technology development than their 

commercial counterparts. These influences are discussed in chapter 4.

Having an S&T organization manage a technology to maturation means 

more than just having a different group of people involved than a product 

development. S&T projects operate in a different environment than 

product developments. The process of developing technology culminates 

in discovery and must, by its nature, allow for unexpected results. S&T 

provides a more forgiving environment in which events—such as test 

“failures,” new discoveries, and delays in the attainment of knowledge—are 

considered normal. It is also a less costly environment, making external 

pressures to develop knowledge on a schedule less keenly felt. On the 

Technology

TRL at

handoff

Responsible

S&T organization

Receiving product 

development program

Nonpenetrating 

periscope

9 DARPA Virginia class attack 

submarine program

Adaptive cruise control 8 Ford Advanced Vehicle 

Technology Office

Jaguar vehicle team

Voice activated controls 8 Ford Advanced Vehicle 

Technology Office

Jaguar vehicle team

Solar cell array 6 Hughes Laboratories HS-702 satellite 

program

Weapon ejection 

system

6 Office of Naval 

Research

Virginia class attack 

submarine program

Diesel powered engine 6 Office of Naval 

Research

AAAV program

High-speed planing 

craft

6 Office of Naval 

Research

AAAV program

High-power water jet 6 Office of Naval 

Research

AAAV program
Chapter 3

Controllable Conditions Affect How Well a 

Technology’s Inclusion on a Product Can Be 

Managed

Page 37 GAO/NSIAD-99-162 Best Practices

other hand, the process of developing a product culminates in delivery, and 

thus gives great weight to design and production. The same events and 

unexpected results that are considered normal for technology development 

represent problems in the product environment; they can jeopardize 

achievement of cost and schedule objectives and draw criticism to the 

product. The ups and downs and the resource changes associated with the 

technology discovery process do not mesh well with a program’s need to 

meet cost, schedule, and performance goals. This situation has been 

described as attempting to “schedule inventions.”

Successful Cases Afforded 

Flexibility to 

Decisionmakers

In the early 1980s, Hughes Space and Communications began developing 

dual junction solar cell technology that had the potential of greatly 

increasing the electrical power on satellites. By 1985, a Hughes laboratory 

had demonstrated the technology by ground testing prototypes, a TRL 6, 

which is considered an acceptable level of demonstration for space-based 

technology. Nonetheless, Hughes was not satisfied that the supporting 

infrastructure (materials, reactors, and test equipment) was mature enough 

to sustain development and production of the new technology on a 

satellite. The infrastructure was seen as critical to meeting the cost and 

schedule requirements of a product. As a result, Hughes did not hand off 

the technology to a product. Instead, the firm kept it in a research 

environment, away from cost and schedule pressures. 

In the early 1990s, Hughes established requirements for a new satellite—

the HS-702—that would use the solar cell technology to leapfrog the 

competition. After a laboratory demonstration in 1993, Hughes successfully 

used the new technology on a high-powered version of its existing HS-601 

satellite before it began product development on the HS-702 satellite. By 

1994, it had determined that the business base was available to sustain 

development and production of the HS-702 satellite. In all, the firm waited 

10 years for the demonstrated technology to meet the requirements. This 

experience closely resembled that of Ford’s voice activated control 

technology because, in both cases, the new technology took 10 years to 

mature enough for product readiness. Thus, the firms’ approach was not to 

accelerate technology development but to shorten product development by 

maturing the technology first. Figure 3.1 shows the solar cell arrays 

installed on the HS-702.
Chapter 3

Controllable Conditions Affect How Well a 

Technology’s Inclusion on a Product Can Be 

Managed

Page 38 GAO/NSIAD-99-162 Best Practices

Figure 3.1: Hughes Solar Cell Arrays

Hughes successfully proved solar cell array technology on a predecessor satellite before beginning 

product development of the HS-702

Source: Hughes Space and Communications.
Chapter 3

Controllable Conditions Affect How Well a 

Technology’s Inclusion on a Product Can Be 

Managed

Page 39 GAO/NSIAD-99-162 Best Practices

The Navy made trade-offs in choosing a technology for the weapon ejection 

system, which is used to deploy weapons like torpedoes, of the Virginia 

class attack submarine. Because of quietness, weight, and cost 

requirements, the Navy preferred a new elastomeric (rubber-based) 

technology. However, this technology failed endurance testing, and product 

managers determined that the technology was too risky to be included in 

the first product. Product managers could have declined this technology 

and its attendant risk without delaying the submarine’s schedule because 

the Navy accepted marginal increases to the cost and weight requirements 

for the system so that the proven Seawolf ejection technology could be 

used as a substitute. Using proven technology on the first submarine has 

allowed the Navy S&T community to continue developing the elastomeric 

technology, which is to be incorporated into the new system on the fourth 

production submarine. As discussed earlier, decisionmakers also had the 

flexibility to wait for the nonpenetrating periscope technology to reach 

TRL 9 before including it in the submarine’s requirements.

Problematic Cases Provided 

Little Flexibility to 

Managers 

According to Army officials, the FLIR and integrated avionics technologies 

required for the Comanche helicopter were critical for providing an 

increased operational capability over existing Army helicopters. The 

advanced FLIR technology was needed to meet the user’s requirements for 

increased targeting range and for improved piloting capabilities in bad 

weather and at night. It represented a quantum leap from existing 

capabilities. Integrated avionics technology was expected to replace 

separate radios, navigation systems, and other communication equipment 

on the helicopter with a modular system that uses central processors. 

These technologies were needed to meet weight and size requirements for 

the aircraft as well as improve communications. Both were critical 

elements of a mission equipment package that was supposed to reduce the 

pilot’s workload while improving capabilities. Requirements were 

inflexible. Thus, requirements managers informed us they were unwilling 

to accept the product manager’s request to trade requirements that was 

prompted by his concerns that the technologies could not advance in time 

to meet the program’s schedule. They believed the product manager to be 

too risk averse and said they would not take no for an answer. Not only did 

the user consider the technologies nontradable, they became even more 

confined by weight and cost restrictions that were placed on the program. 

For example, a more mature FLIR technology that could possibly meet 

performance requirements but also weighed more was rejected. 
Chapter 3

Controllable Conditions Affect How Well a 

Technology’s Inclusion on a Product Can Be 

Managed

Page 40 GAO/NSIAD-99-162 Best Practices

The technological solutions that could meet the strict requirements were 

limited. According to Army officials, the only viable option was to develop 

the new technologies, which were in a very immature state, to the required 

performance levels because no suitable back-up technologies existed. 

When the Comanche acquisition program was launched, the FLIR and 

integrated avionics technologies had a TRL 3, barely demonstrated in a 

laboratory. This level placed the burden on the Comanche program 

manager to complete their development during the acquisition program. 

The only ways for the program manager was to slip the schedule or 

increase development costs. Figure 3.2 shows an early model of the 

integrated avionics component for the Comanche.

Figure 3.2: Integrated Avionics for Comanche Helicopter

The Army launched the Comanche program with immature technologies, placing the burden on the 

program manager to complete technology development

Source: DOD.
Chapter 3

Controllable Conditions Affect How Well a 

Technology’s Inclusion on a Product Can Be 

Managed

Page 41 GAO/NSIAD-99-162 Best Practices

Similarly, the acoustic sensor technology on the BAT was critical to the 

submunition’s performance because it provided breakthrough 

improvements in the capability for precision attack of targets at ranges of 

up to 500 kilometers and in most weather. There was no flexibility for the 

program manager to ease requirements to substitute a more mature 

technology because the Army had no existing capability to perform this 

mission. Thus, the technology, which had a TRL 2 at program launch, was 

the only solution for locating and acquiring targets. Its feasibility was based 

on an engineering analysis in the form of studies. Key challenges for the 

acoustic sensor were to reduce noise to an acceptable level, develop 

microphones with sufficient range, and reduce the size of the sensor so it 

would fit into the BAT delivery system. The technology development that 

was necessary to have the sensor meet requirements had to be 

accomplished during the schedule-driven, delivery-oriented product 

development program. The development program encountered technical 

problems that left the program manager with no choice but to slip the 

schedule and increase the cost. By the start of the engineering and 

manufacturing development phase, program officials stated that the 

acoustic sensor had a TRL 5—still a high risk using the Air Force Research 

Laboratory’s criteria.

Good Technology 

Handoff Decisions 

Depend on the Tools 

and Authority Given to 

Managers

With the right environment as a precondition, managers on the successful 

cases benefited from disciplined technology development processes that 

linked the technologies to products and provided credible information on 

the status of technologies. They also had standards that were both clear 

and high for assessing readiness. Once a technology’s feasibility and 

usefulness were demonstrated, it was linked to a product through an early 

agreement with the product developer to use it if it could be fully 

developed. Ideally, as technologies approached the higher readiness levels 

associated with the bridge, S&T managers and receiving product managers 

agreed to more specific terms for accepting or rejecting a technology. 

These agreements were early links to the product that were needed for the 

technology to succeed. If a product manager was not willing to make such 

an agreement, then the investment to bring the technology to higher 

readiness levels might not be made. S&T managers were responsible for 

ensuring that information at key junctures was sufficient and that the 

technology was ready for inclusion on a product. They saw their role as to 

screen and develop technologies to standards acceptable to product 

managers. Product managers were responsible for ensuring that the 

product could be developed and brought to market within cost and 

performance targets. They saw as their role to encourage the successful 
Chapter 3

Controllable Conditions Affect How Well a 

Technology’s Inclusion on a Product Can Be 

Managed

Page 42 GAO/NSIAD-99-162 Best Practices

development of new technology but to decline the handoff if it did not meet 

product performance, cost, and schedule requirements. 

When an S&T organization disengages from a technology at a low TRL, the 

S&T manager gives up much of the ability to be a gatekeeper. In the event 

that unyielding requirements or other pressures force product managers to 

accept technologies before they have matured, they are weakened in their 

ability to safeguard the product development from technology risks. For 

the cases in which technologies had problems transitioning to products, 

decisionmakers were disadvantaged by the incomplete information 

available to them, yet were not empowered to say no to the handoff. Their 

situation was further degraded by risk assessments that embodied lower 

standards for accepting undemonstrated technology readiness. In the case 

of the BAT, the S&T community was bypassed altogether, as the weapon 

system and its enabling technologies were proposed by a contractor and 

assigned directly to a program manager. 

Successful Cases Benefited 

From Strong Gatekeepers, 

Disciplined Processes, and 

High Maturity Standards

All new technologies at Ford, regardless of whether they are proposed by 

inside or outside sources, take essentially the same path and gates into 

products. Initially, technology proposals pass through a process that 

prioritizes them according to customer needs. The proposals are then 

passed on to the Advanced Vehicle Technology Office, an S&T organization 

that determines the readiness of the proposed technology and fits it into 

Ford’s path of technology demonstration. Once approved, the technology 

follows a structured process that includes two development phases: 

concept ready and implementation ready. This process results in a smooth 

transition from the technology development environment into a product, 

once the technology is mature. Ford’s adaptive cruise control technology 

went through this process, as shown in figure 3.3. 
Chapter 3

Controllable Conditions Affect How Well a 

Technology’s Inclusion on a Product Can Be 

Managed

Page 43 GAO/NSIAD-99-162 Best Practices

Figure 3.3: Process for Closing the Gap Between the Readiness of Adaptive Cruise Control Technology and Jaguar 

Requirements

According to Ford officials, technologies for adaptive cruise control 

existed as separate projects in the technology base from about 1993 to 

1995, when the Jaguar vehicle team identified a strong demand for the 

capability. Ford’s S&T community inventoried the ongoing projects and 

demonstrated the technology as a laboratory breadboard—a TRL 5. By 

August 1996, the technologists had built a prototype that could 

demonstrate the technology in a relevant environment—a TRL 7. This work 

comprised the concept ready phase, in which the technology was taken 

from concept to where its feasibility was demonstrated to potential users. 

At the end of this phase, S&T representatives proved that it could work, 

and cost, schedule, and performance targets were established. Also, a 

target product and sponsor were identified, linking the technology to a 

product. The sponsor agreed that it would accept this technology if specific 

cost, quality, schedule, and performance targets were met. The medium for 

this acceptance was the Deliverables Agreement Log (DEAL), which was 

signed in September 1996 by Jaguar’s chief engineer based on the prototype 

demonstration.

Ford uses the DEAL as a tool to maintain visibility over a new technology 

as it progresses through the development process and to assess its 

readiness and acceptability for inclusion in a vehicle program before 

handing it to a sponsor, the vehicle center, or team. The DEAL formalizes 

Advanced vehicle technology office

Technology

feasibility

Product

development

Bridge between technology

and product

Jaguar vehicle team

Enabling technologies

pursued in the technology

base

Interest expressed

by vehicle team

TRL 5

Agreement signed

between S&T and

vehicle team

 TRL 7

1997

Product launch

TRL 8

1993 1995 1999

Model introduced

1996
Chapter 3

Controllable Conditions Affect How Well a 

Technology’s Inclusion on a Product Can Be 

Managed

Page 44 GAO/NSIAD-99-162 Best Practices

the content of the two development phases and establishes agreements 

between the technologists managing the project and those with authority to 

accept the technology into a product. According to Ford officials, the DEAL 

is important to this process because it is a contract between the parties that 

addresses the technology’s performance, cost, quality, weight, producibility, 

and maintainability targets that must be met before the end of each phase. 

It has been invaluable in getting parties to agree on what is expected by the 

giver and receiver of a technology during the process. 

Once these targets are established, the technology moves to the 

implementation ready phase. For the Jaguar, the Advanced Vehicle 

Technology Office matured the technology to a high level of readiness by 

prototyping it in demonstrator vehicles—a TRL 8. The technology passed 

the implementation ready milestone in February 1997. At that point, the 

vehicle team accepted the technology for inclusion on a Jaguar product 

development. 

Ford used this decision-making process to develop the night vision 

technology, but with a different result. Since 1991, Ford has been working 

on this technology to provide a wide field of view and depth perception for 

the driver at night, similar to that provided by a FLIR. By 1998, the 

Advanced Vehicle Technology Office brought the technology to a TRL 8. 

However, the vehicle center did not agree to include the technology on a 

product because the technology did not meet the cost targets established in 

the DEAL. 

Other companies we visited had similar practices for supporting 

technology inclusion decisions. For example, 3M takes technology from its 

technology base when it believes it has a customer need. The gatekeeper 

responsible for moving technology into a concept phase—analogous to 

TRL 3 or 4—is the S&T organization of a business unit. That business unit 

monitors the technology’s progress until a new product requirement is 

identified and decides whether there is interest from a product center to 

“pull” it. If an interest exists, it begins a feasibility phase that refines 

requirements through quality functional deployment and builds working 

prototypes of the new product—a stage that would be analogous to TRL 7 

or 8. This phase culminates with an agreement between the technologists 

and the product developers—the receivers—as to the specific cost and 

schedule targets that must be met for the technology to be included into a 

product. To help facilitate the transition, 3M establishes a product 

development team that includes people from research and development, 
Chapter 3

Controllable Conditions Affect How Well a 

Technology’s Inclusion on a Product Can Be 

Managed

Page 45 GAO/NSIAD-99-162 Best Practices

marketing, manufacturing, and other functions that transfer with the new 

technology and ensure it is integrated into the new product. 

3M also has high standards for measuring the readiness of a technology 

before the product developer accepts it. For example, 3M officials told us 

that they are developing a fuel cell technology for which they have built 15 

prototypes for testing purposes—a TRL 7 or higher. However, because the 

technology has not yet met all of the cost, schedule, and performance 

targets for product development, they have not allowed it to be included on 

a new product, despite demand from the marketplace. 

Among the DOD cases, the process followed and the roles played on the 

AAAV program had several features that enabled good technology inclusion 

decisions. For almost 3 decades, the Marine Corps has stated a need for an 

amphibious vehicle with far greater capabilities than the current vehicle. 

Specifically, the requirement to achieve a speed of 20 to 25 knots in the 

open ocean made advances in propulsion technology key enablers for the 

AAAV program. For a vehicle of the planned size and weight of the AAAV, 

this requirement meant achieving 2,700 horsepower with a relatively 

compact engine that must operate on land and in water. The Corps had 

been exploring propulsion technologies for such a vehicle in its technology 

base for many years. Despite this, the Office of Naval Research, an S&T 

organization, assessed the propulsion technology and advised that it was 

not mature enough to warrant inclusion on a program. Based on this 

assessment, Marine Corps and Navy decisionmakers delayed program 

launch from 1991 to 1995, until the technology could be brought to higher 

readiness levels. Figure 3.4 illustrates the process used to transition this 

technology.
Chapter 3

Controllable Conditions Affect How Well a 

Technology’s Inclusion on a Product Can Be 

Managed

Page 46 GAO/NSIAD-99-162 Best Practices

Figure 3.4: Process for Closing the Gap Between the Readiness of Propulsion Technologies and AAAV Requirements

The S&T community and the product managers agreed on what had to be 

done before the program could be launched. The S&T community then 

took the lead in maturing the engine to a TRL 6—a level the Air Force 

Research Laboratory considers acceptable for starting the program 

definition and risk reduction phase. Thus, the assessment by the Office of 

Naval Research provided both the information and the criteria that enabled 

decisionmakers to say no to launching the program given the low readiness 

of the propulsion technology. This was coupled with the flexibility to wait 

for the technology to mature and the decision to give an S&T organization 

responsibility for managing the bridge to product readiness. Figure 3.5 

shows the AAAV.

Office of Naval Research

Technology

feasibility

Product

development

Bridge between

technology and program

AAAV program office

Engine proof

of concept

demonstrated

TRL 3

Agreements reached between

S&T and program office

Program launched in

program definition

phase

TRL 6

1988 1991 1999 1995

Prototype

engine

demonstrated

TRL 7
Chapter 3

Controllable Conditions Affect How Well a 

Technology’s Inclusion on a Product Can Be 

Managed

Page 47 GAO/NSIAD-99-162 Best Practices

Figure 3.5: AAAV

The Marine Corps and Navy delayed program launch by 4 years to develop key technologies to a 

higher readiness level

Source: DOD.

Even with an urgent need for the AAAV, the Marine Corps remained 

disciplined in its development approach, allowing the technology to mature 

to the level of the requirement. Two years before program launch, a Navy 

S&T organization demonstrated the technology in a full-scale prototype 

engine. By program launch in 1995, the required 2,700 horsepower was 

demonstrated by a near prototype engine—a TRL 6. The remaining risk was 

limited to marginal weight and size reductions, although the demonstrator 

engine could be used as a backup if the size and weight reductions could 

not be obtained. In early 1999, the AAAV program office demonstrated a 
Chapter 3

Controllable Conditions Affect How Well a 

Technology’s Inclusion on a Product Can Be 

Managed

Page 48 GAO/NSIAD-99-162 Best Practices

prototype engine at 2,700 horsepower that met size and weight 

requirements—a TRL 7. 

Technology Handoffs Were 

Compromised When 

Managers Had Limited 

Information and Authority 

In the BAT program, neither the S&T community nor the product manager 

had the opportunity to act as a gatekeeper between product requirements 

and the maturity of enabling technologies. All of the technologies for the 

BAT came to the program after the contractor, in 1985, had proposed a 

weapon concept for carrying out unmanned, deep strike missions to attack 

enemy armored vehicles. Army leadership accepted the concept and 

drafted requirements for the BAT, and the acquisition program was 

launched after the proposal was accepted. Thus, the technology for the 

weapon came directly from the contractor’s technology base into the 

acquisition program, with little or no review by the Army’s S&T 

organization. The process, information, and standards that were critical to 

successful technology inclusion decisions in other cases were not 

employed on the BAT. The process followed is shown in figure 3.6.

Figure 3.6: Assimilation of New Technology Into the BAT Program

The program office accepted the acoustic sensor, infrared seeker, and 

navigation technologies included on the BAT program. In retrospect, the 

levels of demonstration at the time posed high risks to the product 

development because the acoustic sensor technology had a TRL of 2 and 

Contractor technology base

Technology concept Technology maturation

and product development

BAT program office

Program launched in

program definition phase

TRL 2-3

Start engineering,

manufacturing and

development phase

TRL 5

1985 1991
Chapter 3

Controllable Conditions Affect How Well a 

Technology’s Inclusion on a Product Can Be 

Managed

Page 49 GAO/NSIAD-99-162 Best Practices

the infrared seeker and navigation technologies had TRLs of 3. Program 

officials stated that a significant amount of technology development was 

required during product development due to the lack of visibility over 

technology readiness before program launch. As a result, the development 

program’s cost and schedule significantly increased over original estimates.

An interesting sidelight to the BAT experience concerns the inertial 

measurement unit, a navigation component of the submunition. When the 

contractor first proposed the BAT concept, the design included a mature 

inertial measurement unit in production on other systems. However, after 

the program was launched, the contractor substituted a new quartz rate 

technology. At the request of the BAT program manager, the Army’s Missile 

Research and Development Engineering Center, an S&T organization, 

assessed the maturity of the quartz rate technology. The Center concluded 

that the new technology had not demonstrated a high enough level of 

readiness and recommended that a more proven existing technology be 

used in the program. Eventually, the new technology was dropped, and an 

existing technology that was at a higher readiness level was used.

We observed additional cases in which decisionmakers relied on 

comparatively low standards for including technologies. The Army 

assessed the FLIR, integrated avionics, and helmet mounted display 

technologies as having moderate risk when they were included in the 

Comanche program. Army officials stated that they required only the 

existence of an ongoing S&T technology project as acceptable, as long as 

the technology was projected to be ready by the engineering and 

manufacturing development phase. According to program officials, 

demonstrated maturity was considered but not required; proof that the 

projects were progressing as scheduled was enough. These technologies, 

however, had TRLs of 3 at the time of launch—a high risk for the program 

definition and risk reduction phase. This risk assessment is more 

consistent with the actual experience of the technologies’ maturation in the 

program. 

The standards used for accepting the laser technology into the ABL 

program also appeared low when compared with the standards used on the 

more successful cases. While the Air Force had established demonstration 

standards for the laser to meet prior to program launch, these standards 

were met if scale models of the laser technology in a laboratory 

demonstrated they had the potential to produce the energy needed for an 

operational system. This level of technical demonstration equated to a TRL 

of 4, representing a high risk for inclusion into an acquisition program. 
Page 50 GAO/NSIAD-99-162 Best Practices

Chapter 4

Impediments to Adopting Best Practices for 

Technology Inclusion in DOD Are 

Surmountable Chapte4r

Although product developments—commercial or defense—fare better 

when key technologies are matured before they are included in the product 

design, the more traditional approach within DOD is to mature technology 

during a product’s development. Rational explanations are behind this 

tradition. S&T organizations, operating within fixed budget levels, are not 

necessarily accustomed or equipped to manage the bridge between 

technology feasibility and product readiness. Programs are more able to 

command the large budgets necessary for reaching higher levels of 

technology readiness than S&T projects. Also, pressures are exerted on 

new programs to offer unique performance and acceptable cost and 

schedule projections, which encourage premature acceptance of unproven 

technologies. 

The Under Secretary of Defense for Acquisition and Technology not only 

supports shorter cycle times and a more aggressive pursuit of technology 

outside of programs, but also use of commercial best practices to get these 

results. DOD has several initiatives underway that could make conditions 

more favorable for S&T organizations to mature a technology further 

before it is included in a product development. One Army project calls for 

an S&T organization to manage all technology maturation and integration 

tasks for a new combat vehicle up to the engineering and manufacturing 

development phase. Other initiatives may make the S&T community a more 

integral participant in matching user requirements with technology and 

tying S&T projects more closely to product development paths. Whether 

these efforts are effective and can be applied on a broader scale remains to 

be seen.

Several Factors Make 

It Difficult to Mature 

Technologies Before 

They Are Included on 

Weapon Systems 

Budgetary, organizational, and other factors within DOD make it difficult to 

bring technologies to high readiness levels before being included in 

weapon systems. These factors encourage S&T organizations to disengage 

from technology development too soon and weapon system program 

managers to accept immature technology. Factors other than these 

encourage leading commercial firms to keep technology development out 

of the product developers’ hands and in those of S&T organizations. The 

differences in these factors and in the management of technology 

development stem from differences in what helps commercial and DOD 

programs to succeed. They do not stem from capabilities commercial firms 

possess that DOD does not.
Chapter 4

Impediments to Adopting Best Practices for 

Technology Inclusion in DOD Are 

Surmountable

Page 51 GAO/NSIAD-99-162 Best Practices

Budget and Organizational 

Factors 

Budget realities within DOD—the fact that weapon system programs 

attract higher levels of funding than S&T projects—make these programs a 

more advantageous setting for funding technology development to the 

higher readiness levels. As a practical matter, it is often necessary to move 

immature technology to a weapon system program to get needed funds and 

management support for maturation. Normally, DOD S&T organizations do 

not see their role as going beyond demonstrating the feasibility of a 

technology for generic—versus product specific—application (a TRL 5). 

However, as seen in several of the cases we reviewed, even this level often 

is not reached before a product development organization takes over. The 

S&T organizations that helped to bridge the gap from technology feasibility 

to product readiness on the more successful cases had gone beyond their 

typical role. 

One of the reasons that S&T organizations disengage relatively early is that 

S&T work is traditionally funded as a percentage of the overall DOD 

research and development budget. S&T organizations receive about

$8 billion annually, or about 20 percent, of DOD’s research and 

development budget. This money funds several thousand projects, 

providing less than $1 million per project on average. As a result, a project 

needing $100 million or more to mature technology to higher readiness 

levels than normal—not unreasonable sums—would command a fairly 

large share of an S&T organization’s budget, thereby reducing funds 

available for other projects. Under the current scenario, the remaining

80 percent of DOD’s research and development funds, approximately

$30 billion, is spread out over a much smaller number of specific weapon 

programs. A typical weapon system program can receive several hundred 

million dollars annually and occasionally over $1 billion to fund 

development. A major program, such as the F-22, can command $15 billion 

or more in total for product development, receiving sometimes more than 

$2 billion in a year.

Events on the Air Force’s ABL program illustrate these realities. Originally, 

the Air Force had planned the ABL as a technology development project to 

be managed to high readiness levels by an S&T organization. The project 

was started in 1992 as an advanced technology transition demonstration to 

design, fabricate, and test a single demonstrator weapon system and was to 

take 8 years to complete. The pacing technologies, the laser and the beam 

control, were to be matured to a high level—equivalent to TRL 6 or 7—

before being included in a product development program. Requirements 

had not been fixed. In other words, the planned approach resembled what 
Chapter 4

Impediments to Adopting Best Practices for 

Technology Inclusion in DOD Are 

Surmountable

Page 52 GAO/NSIAD-99-162 Best Practices

we have described as the more successful cases in our review. Figure 4.1 

shows the ABL.

Figure 4.1: Airborne Laser

The Air Force used relatively low readiness level standards to include a key technology into ABL

Source: DOD.

In 1996, the Air Force abandoned this approach and decided to launch ABL 

as a weapon system development program, not because technologies were 

sufficiently mature but because of funding and sponsorship concerns. At 

this time, the two key technologies were at TRLs 3 and 4. According to the 

retired manager of the S&T project, a product development program was 

deemed necessary to make the technology development effort appear real 

to the users and not a scientific curiosity. Within the Air Force, the 

perceived lack of support by the users placed the project in a constant state 

of funding jeopardy. This perception was important because the S&T 

project was costly, with a total estimated cost of $800 million, with some 

annual funding requirements approaching $200 million. The annual funding 

requirements would encompass a large percentage of the Air Force’s

S&T budget unless additional funds were made available from weapon 

system budgets or elsewhere. By transitioning to a weapon system program 

linked to user requirements, the ABL was more likely to get these funding 

levels.

This approach was successful—the program won user support and the 

desired funding. However, sacrifices were made in technology 

development. According to the former project manager, the new program 

focused less on the elemental technology hurdles and more on meeting all 
Chapter 4

Impediments to Adopting Best Practices for 

Technology Inclusion in DOD Are 

Surmountable

Page 53 GAO/NSIAD-99-162 Best Practices

user requirements. More expensive demonstrations were necessary to 

meet these broader requirements without necessarily doing more to 

demonstrate basic technology readiness. It became a more traditional 

program with technology and product development proceeding at the same 

time, with attendant higher risks. In March 1999, we reported that, while 

the ABL has made progress in developing these technologies, it still faced 

technical challenges.1

Other Incentives Pressures exerted on weapon system programs can make it advantageous 

to include in their design immature technologies that offer significant 

performance gains. One traditional source has been the perceived threat. 

Users can demand performance improvements that necessitate the 

application of unproven technologies, particularly when a fielding date is 

mandated, to stay ahead of the threat. Another source is technologists, 

whether from S&T organizations or contractors, who see a new weapon 

system as an opportunity to apply a new technology. Also, the competition 

for funds can encourage performance features—and requisite 

technologies—that distinguish the new weapon system from competitors. 

The F-22 was justified as being faster, stealthier, and more lethal than other 

fighters, such as the F-15 and F-117, were. As a result, the F-22 is being 

designed with several advanced technologies, including a very 

sophisticated suite of avionics that is critical to its performance features 

that distinguish it from the other fighters. However, at the time the F-22 

program was launched in 1986, the avionics technologies were immature; 

they have since been a source of problems on the program. We recently 

reported that the development of the F-22’s integrated avionics systems 

continues to experience cost growth and schedule delays, more than

12 years into the program.2

 

A different set of incentives causes leading commercial firms to make their 

S&T organizations responsible for maturing technologies to higher 

readiness levels. Commercial firms are aware of the risks associated with 

the high investment that product development requires. They have a strong 

incentive in the realization that if a product is late, costs more, or performs 

1

Defense Acquisitions: DOD Efforts to Develop Laser Weapons for Theater Defense (GAO/NSIAD-99-50, 

Mar. 31, 1999).

2

F-22 Aircraft: Issues in Achieving Engineering and Manufacturing Development Costs

(GAO/NSIAD-99-55, Mar. 15, 1999).
Chapter 4

Impediments to Adopting Best Practices for 

Technology Inclusion in DOD Are 

Surmountable

Page 54 GAO/NSIAD-99-162 Best Practices

less than expected, the customer could walk away from the product and 

the investment would be lost. Minimizing the possibility of technology 

being the cause of such problems is thus a top priority. Having their S&T 

organizations reduce those risks is essential to putting product 

developments in the best position to succeed. DOD does not have the same 

incentives. DOD programs are not penalized if a product is late, costs more, 

or performs less than expected, because the customer does not walk away. 

Services Encouraged 

to Use Best Practices

Over the past several years, DOD has encouraged the services to use best 

practices to streamline the current process for acquiring new weapon 

systems in order to make them faster, cheaper, and better. Shorter 

acquisition cycle times are seen as critical to making the best use of 

advances in technology. To encourage change, DOD has set a goal to reduce 

the average acquisition cycle time for all program starts in fiscal year 1999 

and beyond by 50 percent over historical averages. DOD has several 

initiatives to improve its technology development process and to move 

technologies to the warfighter faster and less expensively than the 

traditional means. The initiatives also attempt to put the organizations and 

funding in place to bring technologies to higher readiness levels before they 

are included in programs. These initiatives—defense technology 

objectives, advanced technology demonstrations, and advanced concept 

technology demonstrations—call for S&T organizations to play a bigger 

role in managing technologies closer to the point of product readiness, 

matching requirements to technology projects, and making better use of 

demonstration standards. 

Defense Technology 

Objectives

Defense technology objectives (DTO) are used to bring more discipline to 

S&T projects and to link them more closely with weapon system 

development programs. A DTO typically involves a particular technology 

advance, such as high temperature materials for turbine engines and high 

fidelity infrared sensors. It can also group several technologies into a larger 

demonstration. Each DTO identifies a specific technology advancement 

that will be developed or demonstrated, the anticipated date of the 

technology availability, the ultimate customer, and the specific benefits 

resulting from the technology. It places a corporate attention and 

commitment on the technology project by having the technologists, 

product developer, and customer involved in the project. 

According to DOD, the focus of its S&T investment is enhanced and guided 

through DTOs. Each DTO must go through a formal review and approval 
Chapter 4

Impediments to Adopting Best Practices for 

Technology Inclusion in DOD Are 

Surmountable

Page 55 GAO/NSIAD-99-162 Best Practices

process within DOD and must be directly related to advancing the 

operational concepts depicted in DOD’s “Joint Vision 2010” planning 

document. According to DOD officials, those requirements have helped to 

eliminate instances in which technologists work on projects of particular 

interest to them, but with no military application, because the projects 

should be linked to a specific warfighter need. For fiscal year 1999, DOD 

established approximately 350 DTOs, which accounted for $3 billion, or 

less than 50 percent, of the funds DOD had allocated to S&T projects. The 

remaining funds were allocated to projects under the jurisdiction of each 

military service or other defense agencies and did not go through the same 

review and approval process. 

Advanced Technology 

Demonstrations 

Advanced technology demonstrations (ATD) are intended to more rapidly 

evolve and demonstrate new technologies so they can be incorporated into 

a product, if warranted. An ATD has four characteristics that distinguish it 

from a conventional S&T project. They (1) require large-scale resources; 

(2) involve the user; (3) use specific cost, schedule, and performance 

metrics; and (4) identify a target product for inclusion. An ATD is managed 

by an S&T organization and should conclude with an operational 

demonstration of the potential capabilities of the technology, equating to a 

TRL 5 or 6. The original approach to the ABL was essentially an ATD 

approach. Most ATDs use laboratory hardware to demonstrate the 

potential capability of nonproduct specific technologies and not prototype 

hardware. If the technology is determined to be feasible and provides some 

military use, then it may proceed to the program definition and risk 

reduction phase of an acquisition program. From that point, the product 

developer completes the technology development for a specific product.

Advanced Concept 

Technology Demonstrations 

In 1994, DOD initiated Advanced Concept Technology Demonstrations 

(ACTD) to help expedite the transition of mature technologies from the 

developers to the warfighters. ACTDs are intended to help the DOD 

acquisition process adapt to budget constraints while developing 

technology more rapidly. The purpose of an ACTD is to assess the military 

use of a capability, such as a weapon, comprised of mature technologies. 

Typically, ACTDs last 2 to 4 years and consist of building and 

demonstrating a prototype to provide a warfighter the opportunity to 

assess a prototype’s capability in realistic operational scenarios. From this 

demonstration, the warfighter can refine operational requirements, develop 

an initial concept of operation, and determine the military use of the 

technology before it proceeds to the product development process. 
Chapter 4

Impediments to Adopting Best Practices for 

Technology Inclusion in DOD Are 

Surmountable

Page 56 GAO/NSIAD-99-162 Best Practices

According to DOD, ACTDs, which are managed by S&T organizations, will 

be a key mechanism to ensure technology development is separated from 

product development. In related work on unmanned air vehicles, we found 

that ACTDs provided decisionmakers credible data that they used to 

terminate efforts or transition the demonstrator to an acquisition program. 

In these cases, ACTDs put decisionmakers in a better position to be 

gatekeepers. However, we have reported that the ACTD program needs to 

be improved.3

 We found that DOD’s process for selecting program 

candidates does not include adequate criteria for assessing the maturity of 

proposed technology and has resulted in the approval of projects that 

included immature technologies. We found that the use of specific criteria 

for determining maturity was a best practice in the most successful 

technology development cases we examined. 

Two Unique DOD 

Projects May Provide 

Lessons on How to 

Enable S&T 

Organizations to 

Manage Technology 

Further 

Two DOD projects are using S&T organizations to manage technology 

development to higher readiness levels. One, the Army’s Future Scout and 

Cavalry System, is using a modified ATD to mature technologies and make 

performance trade-offs in the more flexible environment provided by

S&T. The other is a joint government and industry program, which the Air 

Force Research Laboratory is managing to reduce the risks associated with 

new jet engine technologies. These projects may provide insights on how

S&T organizations could routinely play a bigger role in maturing 

technologies enough for safe inclusion on weapon system programs. They 

may also clarify the concern that playing a bigger role in technology 

maturation could cause S&T organizations to do less basic research and 

technology development.

Future Scout and Cavalry 

System

In fiscal year 1997, the Army began piloting a variation of an ATD that is 

designed to help bridge the gap between technology development and 

product development by expanding the S&T community’s role in managing 

technologies further into the development cycle. The Army’s initiative, 

called Fast Track, is intended to reduce cost and cycle time by bypassing 

the program definition and risk reduction phase of the DOD acquisition 

process. The Army is testing this concept with its Future Scout and Cavalry 

System project. In this project, the Army will design, develop, and build a 

demonstrator vehicle to show the technical feasibility of the weapon. All of 

3

Defense Acquisition: Advanced Concept Technology Demonstration Program Can Be Improved

(GAO/NSIAD-99-4, Oct. 15, 1998).
Chapter 4

Impediments to Adopting Best Practices for 

Technology Inclusion in DOD Are 

Surmountable

Page 57 GAO/NSIAD-99-162 Best Practices

these tasks will be done under the management of the Army’s S&T 

organization. The Army believes that a more extensive S&T project will 

make the program definition phase unnecessary and estimates that this 

concept will reduce the development process by as much as 4 years and 

save about $400 million. We did not review the Future Scout project in 

terms of its affordability, feasibility, or any impacts it may have on the 

Army’s S&T budget. Figure 4.2 compares the Fast Track development 

process with the traditional approach.

Figure 4.2: Comparison of Traditional Technology Development Process With the Army’s Fast Track Approach

While we do not necessarily agree that the first phase of the acquisition 

cycle can be omitted, so far the Future Scout project is emulating 

technology development practices like those we observed in the successful 

cases. First, it has established demonstration criteria that must be met 

before the technology enters product development. Second, it has also 

S&T organization

Technology feasibility Technology maturation

and product development

Weapon system program office

Program launched in

program definition phase

TRL 3-5

Production

Traditional

approach 9 years

Army S&T organization Weapon system program office Future Scout

and Cavalry

System

Technology feasibility Technology maturation

and product development

Bridge built to

increase technology

maturity

Program office

staff joins S&T

project office

Program launch

in engineering,

manufacturing and

development phase.

TRL 6

4.5 years

Production
Chapter 4

Impediments to Adopting Best Practices for 

Technology Inclusion in DOD Are 

Surmountable

Page 58 GAO/NSIAD-99-162 Best Practices

established forums that involve key players on the technology path to keep 

them informed of the technology’s development progress. For example, the 

acquisition program manager will be integrated into the development 

project during the final 1.5 years of the S&T program. This should provide a 

good link between the technology development and product development, 

allowing the program manager to fully understand the technology before 

product development begins. Finally, by allowing an S&T organization the 

flexibility to manage technologies further into the development cycle, 

Army officials believe they will be able to make trade-offs among cost, 

schedule, and performance requirements before program launch, without 

raising concerns about the state of the project or breaching baselines that 

had been set without enough knowledge.

While this concept comes closer to the most successful technology 

development cases we reviewed, it still embodies greater technical risk. 

The Army expects to demonstrate some performance capabilities of the 

vehicle before the product development phase begins. However, the 

demonstrator vehicle will only be about 75 percent of a complete 

prototype, which means some key technologies will not be demonstrated 

to high readiness levels before that phase begins. Nonetheless, the project 

manager equated the expected overall technical maturity of the vehicle at 

transition to a TRL 6. The Army considers this a medium or acceptable 

level of risk, and it is willing to enter product development with some 

immature technologies. If, however, product development begins at 

engineering and manufacturing development, this risk could be assessed as 

high, based on TRLs. Figure 4.3 shows an artist’s concept of the Future 

Scout and Cavalry System.
Chapter 4

Impediments to Adopting Best Practices for 

Technology Inclusion in DOD Are 

Surmountable

Page 59 GAO/NSIAD-99-162 Best Practices

Figure 4.3: Future Scout and Cavalry System

An Army S&T organization is maturing technologies and proposing performance trade-offs for the 

Future Scout and Cavalry System before program launch

Source: DOD.

Integrated High 

Performance Turbine 

Engine Technology Program

The Integrated High Performance Turbine Engine Technology program—a 

joint government and industry effort—is focused on developing 

technologies for more affordable and higher performance turbine engines 

for both missiles and aircraft. It is a technology validation program and is 

managed by an S&T organization to perform demonstrations of various 

engine technologies to higher readiness levels than most S&T projects. 

After the demonstrations, the technologies enter a product development 

program. The program takes the technology through a series of tests that 

range from individual component tests to full-scale engine demonstrations. 

The program has established strong links with the acquisition programs for 

which the technologies are intended. For example, Air Force Research 

Laboratory officials informed us that they established formal technology 

transition plans with the F-22 and Joint Strike Fighter programs that 

document agreements on what technology development activities will be 

performed to support the programs. Representatives from each program 

office are invited to all technology demonstrations and are kept informed 

about demonstrated progress. 
Chapter 4

Impediments to Adopting Best Practices for 

Technology Inclusion in DOD Are 

Surmountable

Page 60 GAO/NSIAD-99-162 Best Practices

As part of the program, the Air Force developed a set of standards to assess 

the readiness levels of technologies similar to NASA’s TRLs. According to 

the Air Force, the S&T organization uses these standards to determine 

when the project has been completed. These standards were the first 

application of readiness levels by the Air Force Research Laboratory. There 

are five technology readiness levels ranging from component-level tests in 

a laboratory to the highest level involving actual flight tests of engines. The 

program typically does not take technologies to the highest readiness level 

(flight test) because of the high cost. The program stops when it has been 

determined the technology is well defined within acceptable boundaries 

and a good correlation exists between test results and engineering 

predictions. This readiness level would translate to a TRL of 5 or 6, as used 

in this report. The final step of the technology development is left to the 

product developer who determines if the technology can be packaged and 

integrated into the final product.
Page 61 GAO/NSIAD-99-162 Best Practices

Chapter 5

Conclusions and Recommendations Chapte5r

Conclusions Clearly, DOD’s continued advancement of new technologies is essential to 

the continued superiority of its weaponry. The leading edge military 

capabilities the United States possesses today, such as stealth aircraft, 

precision munitions, and intelligence-gathering satellites, bear witness to 

the effects of such technical advances. At the same time, the incorporation 

of advanced technologies before they are mature has been a major source 

of cost increases, schedule delays, and performance problems on weapon 

systems. As DOD contemplates increasing its annual investment in new 

weapons to $60 billion, the expectations on program managers are great: 

they must develop and field weapons of superior capability more quickly 

and less expensively than in the past. The way advanced technologies are 

matured and included in weapon systems will play a central role in meeting 

these expectations. Although different ways to better assimilate new 

technologies into weapons are legitimate topics for debate, that it has to be 

done better is not. 

The leading commercial firms’ practices have produced results that 

resemble those sought by DOD: more technically advanced, higher quality 

products, developed in significantly less time, and less expensively than 

their predecessors. Managing the development of advanced technology 

differently--and separately--from the development of a product has been 

key to these results. The firms insist that advanced technology reach a high 

level of maturity, the point at which the knowledge about that technology is 

essentially complete, before allowing it into a product development. By 

separating the two, the firms lessen the product manager’s burden and 

place that person in a better position to succeed in delivering the product. 

These practices may not necessarily accelerate the pace at which 

technology matures. In fact, several of the commercial technologies we 

reviewed took 10 years or more to get to market. The clear beneficiaries of 

the practices are the product developments, for which the investments are 

much larger, and time translates into significantly more resources than in a 

technology project. Adapting these practices on its weapon system 

programs can help DOD to reduce costs and the time from product launch 

to fielding, and use technology advances as they become available more 

frequently.

Separating technology development from product development calls for a 

new approach to managing technology development. Two conditions are 

essential to such an approach. First, the right environment for maturing 

technologies must exist. A practice that is instrumental in providing this 

environment is maturing technology to achieve product readiness before it 
Chapter 5

Conclusions and Recommendations

Page 62 GAO/NSIAD-99-162 Best Practices

is constrained by the rules of an acquisition program. In the successful 

DOD cases we reviewed, this environment was provided by S&T 

organizations or a team of S&T and product developers who managed 

technologies to high readiness levels before they were included in an 

acquisition program. These organizations provided an environment more 

conducive to the ups and downs normally associated with the discovery 

process. A corollary practice is agreeing on what level of knowledge is 

needed about a new technology before it is considered for inclusion in a 

product design. When that knowledge level does not exist, the flexibility for 

S&T organizations and product managers to either take the time to mature 

the technology or trade off product requirements until they can be met with 

mature technology is essential. It is a rare program that can proceed with a 

gap between product requirements and maturity of key technologies and 

still be delivered on time and within costs. Second, S&T and product 

managers must be provided with the disciplined processes, information, 

standards, and authority to make good handoffs of technology to product. 

Prepared with the tools and authority to make sound handoff decisions, 

both S&T and product managers can function as gatekeepers to safeguard 

the product development from undue technology risks.

Leading commercial firms have adopted this approach as a matter of 

necessity and have used the organizations, tools, and other practices to 

foster technology development and improve the outcomes of product 

developments. The high stakes stemming from the large investment 

required for a new product and the risks if the product does not meet 

customer needs reinforce this approach in leading commercial firms. The 

DOD cases that followed a similar approach were realizing better program 

outcomes, at least in the sense that the programs avoided key technology 

development problems. Yet, these cases are not the norm for DOD 

programs for several reasons. 

• More typically, the commitment to develop and produce a weapon 

system is made before a match between technology and weapon system 

requirements exists. 

• DOD programs operate under different conditions that make it more 

difficult—and less rewarding—to separate technology development 

from product development. 

• Budget realities make it more difficult for S&T organizations to carry 

technologies to the high readiness levels needed to meet product 

requirements; such resources are more available within product 

developments. 
Chapter 5

Conclusions and Recommendations

Page 63 GAO/NSIAD-99-162 Best Practices

• The pressures to show the unequalled performance necessary to win 

funding encourage including promising, but immature, technologies in 

weapon system designs. 

It will take procedural, organizational, and cultural changes within DOD’s 

acquisition process to foster an environment in which (1) technologies can 

be successfully matured outside the purview of weapon system programs, 

(2) programs can be relieved of the pressures to include immature 

technologies and the unrealistic expectations that the technologies will 

conform to tight cost and schedule projections, and (3) technology 

advances will not stall due to inadequate funding or lack of identification 

with a product in the later, more expensive stages of demonstration. 

Experience has shown that such an approach can work within DOD on 

individual cases. DARPA played a primary role in managing the transition 

of the nonpenetrating photonics mast technology to the Virginia class 

attack submarine. The Integrated High Performance Turbine Engine 

Technology program has carried advanced jet engine technologies to TRLs 

of between 5 and 6 for successful inclusion into programs. In the Future 

Scout program, an Army S&T organization, augmented by product 

development staff, is managing an ATD to lower the risk of key 

technologies before a product development program is launched. However, 

it remains to be seen whether the Army will be successful in using large and 

expensive S&T projects, such as the Future Scout program, without 

affecting other Army S&T projects. A challenge for DOD will be whether 

the lessons learned from these individual cases offer an approach that has 

DOD-wide application. Meeting this challenge is essential to fielding 

technologically superior weapons more quickly and within predicted costs.

Recommendations We have previously recommended that DOD separate technology 

development from weapon system programs. That recommendation was 

made without prejudice toward the necessity of technology development 

but rather with the intent that programs could be better managed if such 

development was conducted outside of a program manager’s purview. 

Similarly, the recommendations that follow are made without prejudice 

toward—or the intention of compromising—the basic research and other 

activities that S&T organizations perform. We recognize that 

implementation of these recommendations will have organizational, 

funding, and process implications and will require the cooperation of the 

Congress.
Chapter 5

Conclusions and Recommendations

Page 64 GAO/NSIAD-99-162 Best Practices

To help ensure that new technologies are vigorously pursued and 

successfully moved into weapon system programs, we recommend that the 

Secretary of Defense adopt a disciplined and knowledge-based method for 

assessing technology maturity, such as TRLs, DOD-wide. This practice 

should employ standards for assessing risks of handoff to program 

managers that are based on a technology’s level of demonstration and its 

criticality to meeting the weapon system’s requirements.

With these tools in hand, we recommend that the Secretary (1) establish 

the place at which a match is achieved between key technologies and 

weapon system requirements as the proper time for committing to the cost, 

schedule, and performance baseline for developing and producing that 

weapon system and (2) require that key technologies reach a high maturity 

level—analogous to TRL 7—before making that commitment. This would 

approximate the launch point for product development as practiced by 

leading commercial firms. 

We recommend that the Secretary find ways to ensure that the managers 

responsible for maturing the technologies and designing weapon systems 

before product development are provided the more flexible environment 

that is suitable for the discovery of knowledge, as distinct from the delivery 

of a product. Providing more flexibility will require the cooperation of 

requirements managers and resource managers so that rigid requirements 

or the threat of jeopardizing the funding planned to start product 

development will not put pressure on program managers to accept 

immature technologies. Such an environment may not be feasible if the 

program definition and risk reduction phase remains the effective launch 

point for an entire weapon system program.

An implication of these recommendations is that S&T organizations will 

have to play a greater role in maturing technologies to higher levels and 

should be funded accordingly. Therefore, we recommend that the Secretary 

of Defense evaluate the different ways S&T organizations can play a greater 

role in helping technologies reach high levels of maturity before product 

development begins. For example, given that a technology has sufficient 

potential for application to a weapon system, at a minimum, an S&T 

organization should be responsible for taking a technology to TRL 6 before 

it is handed off to a program office at the program definition and risk 

reduction phase. During this phase, the program manager would be 

responsible for maturing the technology to TRL 7 before it is included in an 

engineering and manufacturing development program. In a situation where 

a single, design-pacing technology is to be developed for a known 
Chapter 5

Conclusions and Recommendations

Page 65 GAO/NSIAD-99-162 Best Practices

application—like the nonpenetrating periscope—an S&T organization 

should be required to mature that technology to TRL 7 before it is turned 

over to a product development manager. S&T organizations could play a 

similar role when a significant new technology is being prepared for 

insertion into an existing weapon system. Finally, when multiple new 

technologies are to be merged to create a weapon system, S&T 

organizations should be required to bring key technologies to TRL 6 and 

then become part of a hybrid organization with product developers to 

integrate the technologies and bring them to TRL 7 before handing full 

responsibility to a product development manager. 

To help guard against the possibility that the more basic research and 

technology development activities would be compromised by having S&T 

organizations routinely take key technologies to TRL 6 or higher, we 

recommend that the Secretary extract lessons from the nonpenetrating 

periscope, the AAAV, and the Army’s Future Scout programs, and other 

ATD and ACTD programs. Specifically, the Secretary should assess whether 

the resources needed to enable S&T organizations to play a leading role in 

the development of technologies and, in some cases, preliminary system 

design, detracted from or displaced more basic research and technology 

development programs.

Finally, we recommend that the Secretary empower managers of product 

development programs to refuse to accept key technologies with low levels 

of demonstrated maturity. The Secretary can encourage this behavior 

through supportive decisions on individual programs, such as by denying 

proposals to defer the development of key technologies and by favoring 

proposals to lengthen schedules or lessen requirements to reduce 

technological risk early.

Agency Comments and 

Our Evaluation

DOD generally concurred with a draft of this report and its 

recommendations, noting that the traditional path to new weapon system 

development is no longer affordable or necessary (see app. I). DOD stated 

that it has embarked upon a “Revolution in Business Affairs” that will 

enable new technologies to be developed more efficiently and effectively. It 

believes that the first steps in this direction have already been taken but 

agrees that more progress needs to be made. DOD agreed that TRLs are 

necessary in assisting decisionmakers in deciding on when and where to 

insert new technologies into weapon system programs and that weapon 

system managers should ensure that technology is matured to a TRL 7 

before insertion occurs. DOD concurred that S&T organizations should be 
Chapter 5

Conclusions and Recommendations

Page 66 GAO/NSIAD-99-162 Best Practices

involved in maturing technologies to high levels, such as TRL 6, before 

transitioning to the engineering and manufacturing development phase and 

agreed to assess the impact of this involvement on other S&T resources. 

We note that the best practice is to mature technology to at least a TRL 7 

before starting the engineering and manufacturing development phase, 

whether the technology is managed by an S&T organization, a weapon 

system program manager, or a hybrid of the two organizations.

DOD noted that while TRLs are important and necessary, the increasing 

projected life for new weapon systems, total ownership costs, and urgency 

based upon threat assessments are also important considerations for 

system development decisions. We agree and note that our 

recommendations are not intended to cover all aspects of weapon system 

development decisions or to suggest that technology maturity is the only 

factor in such decisions. Rather, the recommendations are in keeping with 

the purpose of the report, “to determine whether best practices offer 

methods to improve the way DOD matures new technology so that it can be 

assimilated into weapon system programs with less disruption.” We believe 

that a knowledge-based approach to maturing technology, such as TRLs, 

can benefit other considerations as well. For example, decisions on what 

technologies to include in a weapon system and when to include them can 

have a significant bearing on its total ownership costs. 

DOD stated that there should be an established point for the transition of 

technologies and that it plans to supplement its milestone review process 

with additional guidance in the next revisions to DOD Directive 5000.2R. It 

also stated that its policy on the evolutionary approach to weapon 

acquisitions should be developed in consonance with the technology 

transition strategy. We cannot comment on the revisions to the directive or 

the evolutionary acquisition policy because they have yet to be published. 

However, under the current milestone review process, the pressures placed 

on a program during the program definition and risk reduction phase—

when much technology development occurs—can operate against the 

flexibility and judgments that are needed to mature technologies. If the 

revisions to the directive supplement the current milestones without 

relieving the pressures brought to bear on programs as they are launched in 

the program definition and risk reduction phase, it will remain difficult to 

discourage the acceptance of immature technologies in the design of new 

weapon systems. To relieve these pressures, we encourage DOD, as it 

develops the directive and the evolutionary acquisition policy, to separate 

technology development from product development and to redefine the 

launch point for a program as the point at which enough knowledge has 
Chapter 5

Conclusions and Recommendations

Page 67 GAO/NSIAD-99-162 Best Practices

been gained to ensure that a match is reached between the maturity of key 

technologies and weapon system requirements.

DOD also stated that program managers already have the ability to reject 

inappropriately mature technologies, and to the extent technology 

immaturity affects acquisition baselines, to advise acquisition executives of 

feasible alternatives. We did not find this to be the case in our review. 

Rather, we found that the program managers’ ability to reject immature 

technologies is hampered by (1) untradable requirements that force 

acceptance of technologies despite their immaturity and (2) reliance on 

tools for judging technology maturity that fail to alert the managers of the 

high risks that would prompt such a rejection. As noted in the report, once 

a weapon system program begins, the environment becomes inflexible and 

deviations to program baselines can attract unwanted attention. This 

reality limits the program managers’ ability to reject immature 

technologies. 
Page 68 GAO/NSIAD-99-162 Best Practices

Appendix I

Technology Readiness Levels and Their 

Definitions ApendxIi

Technology readiness level Description

1. Basic principles observed and reported. Lowest level of technology readiness. Scientific research begins to be translated into 

applied research and development. Examples might include paper studies of a 

technology’s basic properties.

2. Technology concept and/or application 

formulated.

Invention begins. Once basic principles are observed, practical applications can be 

invented. The application is speculative and there is no proof or detailed analysis to 

support the assumption. Examples are still limited to paper studies.

3. Analytical and experimental critical function 

and/or characteristic proof of concept.

Active research and development is initiated. This includes analytical studies and 

laboratory studies to physically validate analytical predictions of separate elements of 

the technology. Examples include components that are not yet integrated or 

representative. 

4. Component and/or breadboard validation in 

laboratory environment.

Basic technological components are integrated to establish that the pieces will work 

together. This is relatively “low fidelity” compared to the eventual system. Examples 

include integration of “ad hoc” hardware in a laboratory. 

5. Component and/or breadboard validation in 

relevant environment.

Fidelity of breadboard technology increases significantly. The basic technological 

components are integrated with reasonably realistic supporting elements so that the 

technology can be tested in a simulated environment. Examples include “high fidelity” 

laboratory integration of components. 

6. System/subsystem model or prototype 

demonstration in a relevant environment.

Representative model or prototype system, which is well beyond the breadboard 

tested for TRL 5, is tested in a relevant environment. Represents a major step up in a 

technology’s demonstrated readiness. Examples include testing a prototype in a high 

fidelity laboratory environment or in simulated operational environment.

7. System prototype demonstration in an 

operational environment.

Prototype near or at planned operational system. Represents a major step up from 

TRL 6, requiring the demonstration of an actual system prototype in an operational 

environment, such as in an aircraft, vehicle or space. Examples include testing the 

prototype in a test bed aircraft.

8. Actual system completed and “flight qualified” 

through test and demonstration.

Technology has been proven to work in its final form and under expected conditions. 

In almost all cases, this TRL represents the end of true system development. 

Examples include developmental test and evaluation of the system in its intended 

weapon system to determine if it meets design specifications.

9. Actual system “flight proven” through 

successful mission operations. 

Actual application of the technology in its final form and under mission conditions, 

such as those encountered in operational test and evaluation. In almost all cases, this 

is the end of the last “bug fixing” aspects of true system development. Examples 

include using the system under operational mission conditions.
Page 69 GAO/NSIAD-99-162 Best Practices

Appendix II

Comments From the Department of Defense ApendxIi
Appendix II

Comments From the Department of Defense

Page 70 GAO/NSIAD-99-162 Best Practices

Now on pp. 7 and 63-64.

Now on pp. 7 and 63-64.

Now on pp. 7 and 63-64.
Appendix II

Comments From the Department of Defense

Page 71 GAO/NSIAD-99-162 Best Practices

Now on pp. 7 and 63-64.
Appendix II

Comments From the Department of Defense

Page 72 GAO/NSIAD-99-162 Best Practices

Now on pp. 7 and 65.

Now on pp. 7 and 65.
Page 73 GAO/NSIAD-99-162 Best Practices

Appendix III

GAO Contacts and Staff Acknowledgments ApenIdxi

GAO Contacts Louis Rodrigues, (202) 512-4841

Paul Francis, (202) 512-2811

Acknowledgments In addition to those named above, Michael Sullivan, Jeffrey Hunter, 

Matthew Lea, Maria Santos, Rae Ann Sapp, and Katrina Taylor made key 

contributions to this report.
Page 74 GAO/NSIAD-99-162 Best Practices
Page 75 GAO/NSIAD-99-162 Best Practices
Page 76 GAO/NSIAD-99-162 Best Practices

Related GAO Products

Best Practices: Commercial Quality Assurance Practices Offer

Improvements for DOD (GAO/NSIAD-96-162, Aug. 26,1996).

Major Acquisitions: Significant Changes Underway in DOD’s Earned Value

Management Process (GAO/NSIAD-97-108, May 5, 1997).

Best Practices: Successful Application to Weapon Acquisitions Requires

Changes in DOD’s Environment (GAO/NSIAD-98-56, Feb. 24, 1998).

Best Practices: DOD Can Help Suppliers Contribute More to Weapon

System Programs (GAO/NSIAD-98-87, Mar. 17, 1998).

Defense Acquisition: Improved Program Outcomes Are Possible

(GAO/T-NSIAD-98-123, Mar. 18, 1998).

Defense Acquisitions: Best Commercial Practices Can Improve Program

Outcomes (GAO/T-NSIAD-99-116, Mar. 17, 1999).

(707336) Letr
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the 

following address, accompanied by a check or money order made 

out to the Superintendent of Documents, when necessary, VISA and 

MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are 

discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000

or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and 

testimony. To receive facsimile copies of the daily list or any list 

from the past 30 days, please call (202) 512-6000 using a touchtone 

phone. A recorded menu will provide information on how to obtain 

these lists.

For information on how to access GAO reports on the INTERNET, 

send an e-mail message with “info” in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at: 

http://www.gao.gov
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. GI00