United States General Accounting Office

GAO Report to the Subcommittee on

Acquisition and Technology, Committee

on Armed Services, U.S. Senate

February 1998 BEST PRACTICES

Successful Application

to Weapon Acquisitions

Requires Changes in

DOD’s Environment

GAO/NSIAD-98-56

GAO United States

General Accounting Office

Washington, D.C. 20548

National Security and

International Affairs Division

B-275842 

February 24, 1998

The Honorable Rick Santorum

Chairman

The Honorable Joseph I. Lieberman

Ranking Minority Member

Subcommittee on Acquisition and Technology

Committee on Armed Services

Unites States Senate

As you requested, this report assesses whether commercial product development practices

offer ways to improve the process used at the Department of Defense (DOD) to prepare weapon

systems for production. It also determines how differences in commercial and DOD

environments for developing new products affect the corresponding practices. We make

recommendations to the Secretary of Defense and present matters for congressional

consideration, all of which are intended to create the necessary environment to ensure DOD’s

future success in improving the transition of weapons from development to production.

We are sending copies of this report to other congressional committees; the Secretaries of

Defense, the Army, the Navy, and the Air Force; and the Director, Office of Management and

Budget. We will also make copies available to others on request.

This report was prepared under the direction of Katherine V. Schinasi, Associate Director of

Defense Acquisitions Issues, who may be reached at (202) 512-4841 if you or your staff have any

questions. Other major contributors to this report are listed in appendix II.

Henry L. Hinton, Jr.

Assistant Comptroller General
Executive Summary

Purpose The Department of Defense (DOD) continues to state a need to modernize

weapons for the armed forces at a faster pace within relatively level funds.

It has a budget of over $40 billion in fiscal year 1998 to acquire and

upgrade weapons, and may not reasonably expect to receive much more

than that amount in future years. Therefore, it must find new ways to

modernize more economically. The challenge of the Soviet Union has been

replaced by a challenge to maintain weapons that have a leading

technological edge; however, such technologies are increasingly being

developed in the commercial sector. In this period of lessened tensions,

DOD has an opportunity to revamp the practices it uses to acquire weapon

systems, avoid the technological obsolescence that comes with 15-year

and longer product development cycles, speed the pace of modernization

in the face of budgetary pressures, and meet defense needs with sufficient

flexibility. The practices employed by some commercial firms to reduce

the time and money to develop new products—by as much as

50 percent—can illuminate ways for DOD to make similar improvements.

In response to a request from the Chairman and Ranking Minority

Member, Subcommittee on Acquisition and Technology, Committee on

Armed Services, GAO assessed whether commercial practices offer ways to

improve DOD’s process for transitioning weapons from development to

production. Specifically, this report (1) compares DOD’s practices for

preparing a weapon system for production with best commercial

practices, (2) determines how differences in commercial and DOD

environments for developing new products affect practices, and

(3) discusses environmental changes that are key to the success of DOD

initiatives for improving the transition of weapons from development to

production.

Background Commercial firms make a distinction between product and technology

development. Product development entails the design and manufacture of

a product, such as an airplane, a car, or a satellite, as an end item for

delivery to a customer. The basic features of product development include

a definable market in terms of customer needs or wants, the ability to

design a product for that market opportunity, the ability to produce that

product, and the investment capital needed to fund the development

effort. Technology development fosters technological advances for

potential application to a product development. In DOD, the acquisition

cycle for weapons includes technology development, product

development, and production. Thus, the distinction between product and

technology development on an individual program is much less clear.

Page 2 GAO/NSIAD-98-56 Best Practices
Executive Summary

Among the key sources of information GAO relied on in this review were

individual DOD acquisition programs and several firms recognized as being

among the best in developing and manufacturing new products. Much

commonality existed among these firms’ practices, even though their

products spanned a range of technological sophistication. In this report,

GAO highlights the best commercial practices in product development

based on its fieldwork. As such, they are not intended to describe all

commercial industry or suggest that commercial firms are without flaws.

Results in Brief Commercial firms gained more knowledge about a product’s technology,

performance, and producibility much earlier in the product development

process than DOD. Product development in commercial ventures was a

clearly defined undertaking for which firms insisted on having the

technology in hand to meet customer requirements before starting. Once

underway, these firms demanded—and got—specific knowledge about a

new product before production began. The process of discovery—the

accumulation of knowledge and the elimination of unknowns—was

completed for the best commercial programs well ahead of production.

Not having this knowledge when demanded constituted a risk the firms

found unacceptable. Immature or undeveloped technology could not meet

these demands and was kept out of commercial product development

programs; this technology was managed separately until it could meet the

demands for product development.

In contrast, DOD programs allowed more technology development to

continue into product development. Consequently, the programs

proceeded with much less knowledge—and thus more risk—about

required technologies, design capability, and producibility. The programs’

discovery process persisted much longer, even after the start of

production. Turbulence in program outcomes—in the form of production

problems and associated cost and schedule increases—was the

predictable consequence as the transition to production was made.

Although DOD accepted more unknowns on its programs than commercial

firms, it understated the risks present.

The commercial and defense environments created different incentives

and elicited different behaviors from the people managing the programs.

Specific practices took root and were sustained because they helped a

program succeed in its environment—not because they were textbook

solutions. The success of commercial product developments was

determined when production items were sold. Until that point, the firm’s

Page 3 GAO/NSIAD-98-56 Best Practices
Executive Summary

own money was at risk. Failure was seen as both clear and painful if the

customer walked away. This definition of success, coupled with the

realization that production startup was relatively close at hand, made

production concerns a key factor in program start and subsequent

decisions and provided strong incentives to identify risks both early and

realistically. DOD programs began without needed technology in hand;

rather, they were encouraged to include undeveloped technology. Because

these programs ran much longer, production concerns did not play as big a

role and were not as critical to success in the early stages. The definition

of success was more complicated in DOD. During most of product

development, success was defined as getting DOD and the Congress, as the

customers, to fund the development annually. Optimistic assessments of

performance and cost helped ensure this kind of success; realistic risk

assessments did not.

Commercial practices for gaining knowledge and assessing risks can help

produce better outcomes on DOD acquisitions. Indeed, DOD has several

commercial-like initiatives underway, such as using cost or price as a

means for forcing technology tradeoffs, that it believes are having

promising results. For such practices to work on a broad scale, however,

the DOD environment must be conducive to such practices. At least two

factors are critical to fostering such an environment. First, program launch

decisions must be relieved of the need to overpromise on technical

performance and resource estimates. The pressure to amass broad support

at launch creates incentives for new programs to embrace far more

technology development than commercial programs. Although technology

development clearly must be undertaken by DOD and supported by the

Congress, its objectives, as well as what can be demanded of its

knowledge and estimates, differ from those of product development.

Second, once a program is underway, it must become acceptable for

program managers to identify unknowns as high risks so that they can be

aggressively worked on earlier in development. Currently, identifying a

high risk on a DOD program is perceived as inviting criticism and the loss of

funding. Studies sponsored by DOD and the defense industry call for the

kinds of changes that could help shape a better environment for managing

weapon acquisitions. The challenge for DOD and the Congress is to foster

the environment that provides program managers with incentives for

applying best practices.

Page 4 GAO/NSIAD-98-56 Best Practices
Executive Summary

Principal Findings

Leading Commercial Firms

Attain More Knowledge

Earlier in Product

Development

To minimize the amount of technology development that occurs during

product development, the companies GAO visited employ a disciplined

process to match requirements with technological capability before the

product development process begins. This process is grounded in

production realities that demand proof that the technology will work and

can be produced at an acceptable cost, on schedule, and with high quality.

The companies bring proven technological knowledge to the requirements

process in the form of current, high fidelity information from predecessor

programs, people with first-hand experience on those programs, and new

technologies deemed mature as a result of having “graduated” from a

disciplined technology development and screening process. Also, they

communicate extensively with customers to match their wants and needs

to the firms’ available technology and ability to manufacture the desired

product. They do not stray far from their technological foundation.

For the programs GAO reviewed, DOD did not get as good a match between

what customers require and what technology could confidently deliver.

Even though DOD examines the potential for available technology to meet

product requirements, it allows requirements to make technology reach

beyond what is proven. DOD has less knowledge about the ability of the

design to deliver required performance and be produced at this point in

the program than commercial firms. The knowledge that the match has

been achieved between requirements and technology is often not attained

until testing is completed, late in development or after production begins.

Commercial and DOD decisions made on a lightweight aircraft

material—aluminum lithium—are illustrative of the differences between

the two sectors. Boeing had initially decided to use the alloy on its 777-200

aircraft but rejected its use early in development because it was expensive,

its manufacturing processes were not well understood, and its availability

was limited. DOD accepted these risks and used the alloy on the C-17

aircraft. The alloy is now being phased out of the program as some of its

unknowns became actual problems.

Before the halfway point of product development, the commercial firms

GAO visited achieved near certainty that their product designs would meet

customer requirements and had gone a long way to ensure that the

product could be produced. Both DOD and commercial firms hold a critical

design review to determine whether the design is mature and essentially

Page 5 GAO/NSIAD-98-56 Best Practices
Executive Summary

complete for production purposes. Engineering drawings, which

document the schematics of the product, along with its performance,

required materials and production processes, comprise much of the

source material for the critical design review. Commercial firms typically

had over 90 percent of these drawings available for the review. In

comparison, the C-17 program had less than 60 percent and the F-22

program less than one-third of the drawings available for the review. Over

one-fifth of the C-17’s drawings became available after production began,

and the aircraft experienced a number of problems in production as

difficulties with the design were worked out. Several key technologies are

still unproven on the F-22, and some will not be proven until after 40

aircraft have entered production. Nonetheless, the risks of proceeding

with the rest of development as planned at the time of the critical design

reviews for both programs were deemed acceptable.

The companies GAO visited reached the point at which they knew that

manufacturing processes would produce a new product conforming to

cost, quality, and schedule targets before they began fabricating

production articles. Reaching this point meant more than knowing the

product could be manufactured; it meant that all key processes were

under control, such that the quality, volume, and cost of their output were

proven and acceptable. The DOD programs demanded less proof of a

design’s producibility. The C-17 program began production in 1989 and still

has less than 13 percent of its key manufacturing processes in control. The

F-22 program is currently faring better than the C-17: the contractor

believes it has almost 40 percent of its key manufacturing processes in

control, 2 years before production is scheduled to begin. However, the

program does not plan to have all key processes in control until about 

4 years into production.

Differences in Military and

Commercial Practices

Reflect Different

Environments

The commercial firms GAO contacted launch a product development only

when a solid business case can be made. The business case basically

revolves around the ability to produce a product with the right features to

meet the market opportunity on schedule, with limited investment capital,

and at a predictable unit cost so that the product will sell well enough to

make an acceptable return on investment. Because success is determined

in production, the business case for launching a program considers

production realities and builds in natural curbs to overreaching for

performance, cost, or schedule. A company demands considerable proof

that the product will fulfill all of the business case factors and then

provides full support for the program to succeed. The business case

Page 6 GAO/NSIAD-98-56 Best Practices
Executive Summary

provides a very solid decision-making framework from the outset and

throughout the program. Commercial companies build relatively short

cycle times, keyed to meeting market demands, into their decisions to

begin a product’s development. These short timeframes, together with the

responsibility for protecting the business case, encourage program

managers to identify risks and enable them to say “no” to pressures to

accept unknowns. The companies are conservative in their estimates and

aggressive in reducing risk. The abundance of reliable data and

experienced people from predecessor programs provides a solid factual

basis for defining unknowns and assessing risks.

Because DOD launches programs earlier, the knowledge commercial firms

insist on is generally not available for a DOD program until years later.

Some knowledge, such as high-fidelity information from numerous closely

related predecessor programs, is simply unavailable. Even though less

information about a new DOD product development is available at the time

of launch, the competition for funding requires detailed projections to be

made from the information that does exist. Although DOD is attempting to

ease the technical requirements of programs, a new product development

must possess performance features that distinguish it from either the

systems to be replaced or rival candidates. Consequently, aspiring DOD

programs are encouraged to include performance features and design

characteristics that rely on immature technologies. Untempered by

knowledge to the contrary, the risks associated with these technologies

are deemed acceptable. Production realities, critical to matching

technological capabilities with customer requirements on commercial

programs, are too far away from the DOD launch decision to have the same

curbing effect on technology decisions.

In this environment, risks in the form of ambitious technology

advancements and tight cost and schedule estimates are accepted as

necessary for a successful launch. Problems or indications that the

estimates are decaying do not help sustain the program in subsequent

years, and thus their admission is implicitly discouraged. An optimistic

production cost estimate makes it easier to launch a product development

and sustain annual approval; admission that costs are likely to be higher

could invite failure. There are few rewards for discovering and recognizing

potential problems early in the DOD product development, given the

amount of external scrutiny the programs receive. The behavior of

tolerating unknowns and not designating them the same risk value as in

the commercial environment is manifested in the DOD environment

because there is little incentive to admit to high risks before it is

Page 7 GAO/NSIAD-98-56 Best Practices
Executive Summary

absolutely necessary. This is not to say that DOD program managers and

other sponsors act irrationally or with bad intentions. Rather, they see the

programs under their purview as aligned with the national interest and do

what they believe is right, given the pressures they face.

The Right Environment

Will Be Key to the Success

of DOD Initiatives to

Improve Weapon

Acquisitions

DOD has embarked on several acquisition reform initiatives that draw

lessons from commercial practices. These initiatives include using cost or

price targets as vehicles for forcing requirements or technology tradeoffs,

evaluating contractors’ past performance as a factor in selecting sources

for new contracts, removing specifications that told contractors how to

design or build a product in favor of specifying what performance was

required of the end item, and using multidisciplinary teams to make

decisions and tradeoffs on individual weapon systems. DOD reports that

programs are showing potential reductions in cycle time and staff

resources as a result of these initiatives. DOD has also recently set up a

funding reserve to offset unexpected cost growth to mitigate the effect of

unknowns on programs. In past acquisitions, changing the mechanics of a

weapon’s development, without changing aspects of its environment that

determine its incentives, did not produce desired results. Nonetheless,

DOD’s current initiatives could help the transition of weapons into

production if the environment for launching programs and appraising risks

can be changed to better approximate the commercial environment.

Studies sponsored by DOD and the defense industry call for changes that

could help shape such an environment. A May 1996 study by the Defense

Science Board recommended replacing the current weapon acquisition

process with one that emphasizes incremental technology advancement,

coupled with much shorter product development cycle times. The study

also noted that technology development should be aggressively pursued

outside the realm of weapon system development programs. In April 1996,

the National Center for Advanced Technologies also proposed a change in

the DOD weapons acquisition process to reduce cycle time to the 3- to

5-year range by following an incremental technology advancement

approach. The report noted that although concepts such as DOD’s

initiatives are constructive, there is no reason to assume that these

concepts should succeed when previous good ideas did not.

A December 1994 study by the Defense Systems Management College

made several recommendations to create an environment for weapon

systems development to encourage realism in reporting program status. A

significant recommendation is that the individual military services transfer

Page 8 GAO/NSIAD-98-56 Best Practices
Executive Summary

control of their acquisition organizations and people to the Under

Secretary of Defense for Acquisition and Technology. The study noted that

the Under Secretary would then be empowered to reward candor and

realism in reporting through the use of assignments and promotions.

Recommendations GAO makes several recommendations to the Secretary of Defense. These

recommendations are intended to (1) ensure that sound standards for the

timing and quality of production-related knowledge are applied to

individual programs, (2) separate technology development from weapon

system programs to enable DOD to meet higher knowledge standards on

those programs, and (3) use decisions on individual weapon systems to

foster the right environment for identifying risks early and send the right

message to other programs concerning what practices will work in that

environment. These recommendations appear in full in chapter 5.

Matters for

Congressional

Consideration

Because of its critical role in creating the environment for what

constitutes program success and which practices will work, the Congress

may wish to consider supporting the Secretary of Defense’s efforts to

create such an environment through changes to the acquisition process

that provide program managers clear incentives for gaining sufficient

knowledge at key points in weapon acquisition programs. The best

commercial practices described in this report suggest what may constitute

“sufficient” levels of knowledge. If DOD takes steps to manage technology

development efforts outside the bounds of individual weapon system

programs, the Congress may wish to consider providing the funds needed

for such efforts. The Congress could also help create the right incentives

on individual programs by favorably considering DOD funding requests to

mitigate high risks early in a program and cautiously considering late

requests for funds to resolve problems that should have been addressed

earlier.

Agency Comments DOD agreed with the report and all of the recommendations. A discussion

of DOD’s actions appears on pages 75-77. DOD’s comments appear in

appendix I.

Page 9 GAO/NSIAD-98-56 Best Practices
Contents

Executive Summary 2

Chapter 1 

Introduction

12

Product Development in the Commercial and Defense Sectors 12

The Case for a Smoother and Faster Development to Production

Cycle

16

Objectives, Scope, and Methodology 17

Chapter 2 

Knowledge at Key

Junctures Is Critical

to a Successful

Transition to

Production

21

Commercial Firms Develop Knowledge Sooner in the Product

Development Cycle

22

Knowledge Point 1: Requirements and Technological Capability

Are Matched

24

Knowledge Point 2: The Design Will Perform as Required 33

Knowledge Point 3: Production Units Will Meet Cost, Quality, and

Schedule Objectives

37

Chapter 3 

Differences in Military

and Commercial

Practices Reflect

Different

Environments

44

Commercial Practices Are Driven by the Customer’s Acceptance

of the Finished Product

45

DOD Practices Reflect the Need to Succeed in Funding and

Managing the Development Effort

53

Chapter 4 

Creating the Right

Environment Is Key to

the Success of

Initiatives to Improve

Weapon Acquisitions

60

DOD’s Risk Reduction Policy Is Consistent With Commercial

Practices

60

DOD Initiatives Adapt Some Commercial Practices 63

Experience on Past Programs Underscores the Need to Match

Technique With the Environment

66

Recent Studies Promote Shorter Cycle Times and Other

Environmental Changes

70

Chapter 5 

Conclusions and

Recommendations

72

Conclusions 72

Recommendations 74

Matters for Congressional Consideration 75

Agency Comments and Our Evaluation 75

Page 10 GAO/NSIAD-98-56 Best Practices
Contents

Appendixes Appendix I: Comments From the Department of Defense 78

Appendix II: Major Contributors to This Report 81

Figures Figure 1.1: DOD’s Acquisition Process 14

Figure 2.1: Comparison of Three Key Knowledge Points for

Commercial and Military Product Developments

23

Figure 2.2: Boeing 777 Airliner 27

Figure 2.3: C-17 Cargo Aircraft 32

Figure 2.4: Comparison of When Commercial and DOD Programs

Achieve Knowledge About Their Product’s Design

34

Figure 2.5: Comparison of When Commercial and DOD Programs

Achieve Knowledge That Processes Can Produce An Acceptable

Product

38

Figure 2.6: Hughes HS-702 Satellite 40

Figure 3.1: Plymouth Prowler 50

Figure 3.2: Lincoln Navigator 52

Figure 3.3: F-22 Fighter Plane 55

Figure 4.1: DOD Guidance on Preparing Weapons for Production 61

Figure 4.2: Joint Direct Attack Munitions System 64

Figure 4.3: AIM-9X Missile 69

Abbreviations

AIM Air Intercept Missile

CAIV cost as an independent variable

CATIA computer-aided three-dimensional interactive application

CDR critical design review

DOD Department of Defense

EMD engineering and manufacturing development

GAO General Accounting Office

IPT integrated product team

JDAM Joint Direct Attack Munitions

SPC statistical process control

Page 11 GAO/NSIAD-98-56 Best Practices
Chapter 1 

Introduction

The Department of Defense (DOD) continues to state a need to modernize

weapons for the armed forces at a faster pace within relatively level funds.

It has a budget of over $40 billion in fiscal year 1998 to acquire and

upgrade weapons, and may not reasonably expect to receive much more

than that amount in future years. Therefore, it must find new ways to

modernize more economically. The challenge of the Soviet Union has been

replaced by a challenge to maintain weapons that have a leading

technological edge; however, such technologies are increasingly being

developed in the commercial sector. In this period of lessened tensions,

DOD has an opportunity to revamp the practices it uses to acquire weapon

systems, avoid the technological obsolescence that comes with 15-year

and longer product development cycles, speed the pace of modernization

in the face of budgetary pressures, and meet defense needs with sufficient

flexibility. When dealing with some similar problems, the best commercial

firms have focused on the early stages of the product development cycle to

speed up product developments, get them into production sooner, and

achieve cost goals.

Product Development

in the Commercial

and Defense Sectors

For commercial firms, developing a product entails the design and

manufacture of an end item, such as an airplane, car, or satellite, for

delivery to a customer. The basic features of product development include

a definable market in terms of customer needs or wants, the ability to

design a product for that market opportunity, the ability to produce that

product, and the investment capital needed to fund development. Product

development is therefore very concerned with production realities and

cost and schedule targets. It is distinct from technology development,

which furthers the advancement of technology to see if it has potential

application to a product. For example, a firm that makes satellites may

study new materials and their properties to see if they can reduce the

weight and size of solar array batteries, thus improving a satellite’s

carrying capacity. If the technology can be perfected and is feasible to use,

it is ready to be included in a new product development. More failures are

expected in a technology development, since it is a process of discovery

that deals with many unknowns.

In DOD, the same functions take place on the weapon systems DOD

develops, but they occur within the broader context of the acquisition

cycle. The entire cycle includes technology development, product

development, and production. DOD’s development process is designed to

manage a program through sequential phases, all followed by major

milestone decisions in which decisionmakers approve or disapprove the

Page 12 GAO/NSIAD-98-56 Best Practices
Chapter 1 

Introduction

acquisition strategy for a program and its move into the next phase based

on progress made as presented by the program manager. The phases of the

acquisition cycle, as shown in figure 1.1, are: concept exploration; program

definition and risk reduction; engineering and manufacturing development

(EMD); and production, fielding/deployment, and operational support. The

concept exploration phase decides what kind of weapon, if any, is the best

solution to a military need. A new program actually starts with the

program definition and risk reduction phase.

Page 13 GAO/NSIAD-98-56 Best Practices
Chapter 1 

Introduction

Figure 1.1: DOD’s Acquisition Process

PHASE I: PROGRAM DEFINITION & RISK REDUCTION

PHASE II: ENGINEERING & MANUFACTURING DEVELOPMENT

PHASE III: PRODUCTION, FIELDING/DEPLOYMENT, AND OPERATIONAL SUPPORT

Program is defined as various concepts and technologies are pursued. Risk assessments of 

each concept are refined. Prototyping and demonstrations are considered to reduce technical

and manufacturing risks. Cost drivers, cost/performance trades, and acquisition strategies

are considered.

The most promising design is chosen and translated into a stable, producible, cost-effective 

design; manufacturing processes are validated; and testing begins to demonstrate system 

capabilities. Low rate initial production begins to produce test articles, establish a 

production base, and permit orderly increase to the production rate. 

MILESTONE II

MILESTONE III

Operational capability that satisfies mission needs is satisfied. Deficiencies encountered 

as a result of operational testing are resolved and fixes are verified.

Decision based on -All of above, updated

-Specific milestone II exit criteria

-Limited production quantities

-All of the above, updated

-Specific milestone III exit criteria

-Completion of live fire test & evaluation

Decision based on

Do phase I results warrant

program continuation?

Authority to enter into

full production.

PHASE 0: CONCEPT EXPLORATION

Decision based on

Competitive studies are done to define and evaluate the feasibility of alternative concepts and

provide an assessment of risk for decisionmakers. Analysis of alternatives is used to compare

concepts, which are defined by broad cost, schedule, and performance objectives as well as

opportunities for tradeoffs.

Do phase 0 results warrant

new acquisition? MILESTONE I

-Acquisition Management Plan

-Cost/Performance Trade Studies

-Acquisition Program Baseline

-Test Plan (unless waived)

-Any specific mandated exit criteria

Source: DOD.

Page 14 GAO/NSIAD-98-56 Best Practices
Chapter 1 

Introduction

Within the acquisition cycle for a given weapon system, technology

development, product development, and production activities take place

over a number of years. The delineation of these activities is not always

clearly discernable. Thus, even though the product development activities

described for the commercial sector would occur largely in a weapon’s

EMD phase, technology development and production activities also occur

in this phase. Likewise, some product development activities for a weapon

system take place during production. The acquisition process is embodied

in DOD’s Regulation 5000.2, Mandatory Procedures for Major Defense

Acquisition Programs, which was revised in 1996. In addition, DOD’s 1985

Manual 4245.7-M, Transition From Development to Production, contains

risk-reduction guidance based on best practices.

At any given time, DOD has hundreds of products in various stages of

development and production, each with a program manager from one of

the three services or other procuring agencies within DOD. Although DOD

maintains oversight for all of these acquisitions and controls their funding,

the actual management practices employed are specific to the program,

the contractors, and the relevant government organization. We have issued

a number of reports on weapon system acquisitions since the 1970s.

The program manager and staff for a new product’s development in either

sector face a formidable task. They are responsible for balancing the

requirements for the product’s performance against its established cost

and schedule targets. This responsibility includes preparing an acquisition

strategy for funding, testing, and manufacturing the product and ensuring

that the strategy is carried out. Throughout the development process, the

program manager is expected to identify and manage high-risk areas,

which means making decisions about tradeoffs among cost, schedule, and

performance to deliver a product that will satisfy the customer. The

program manager and staff must therefore have expertise in many areas,

such as estimating techniques, the budget process, simulation, testing,

engineering, logistics, and production. The program manager is the one

person who ultimately must know whether there is a match among

financial, time, and personnel resources provided by upper management

and what the product development effort can deliver. In product

development, the financial investment is substantial and the stakes are

high; program failure is not taken lightly.

Page 15 GAO/NSIAD-98-56 Best Practices
Chapter 1 

Introduction

The Case for a

Smoother and Faster

Development to

Production Cycle

Products that proceed more quickly and smoothly through development

into production require lower financial investment costs to develop. Also,

a smoother and faster process enhances the product’s impact on

customers or users and benefits the producer by providing improved

profitability and market share. Commercial firms that have been

recognized as industry leaders have significantly reduced the time it takes

to complete the product development cycle. They have put new products

into production more quickly within unit production cost targets and

improved the products’ performance features and quality. For example,

Boeing will reduce development time by 40 percent on its 777-300 airplane

compared with its 777-200 development, Hughes develops satellites in 

26 months and will soon reduce the time to 12 months, and Chrysler has

reduced its new vehicle development times from over 5 years to less than

30 months.

If DOD made reductions of the same magnitude in the time it takes to

develop a weapon, it could reap similar benefits. A compelling reason for

DOD to pursue such reductions is to resolve long-standing difficulties in

controlling cost and schedule outcomes of major weapon system

programs. According to cost and schedule data from 93 major acquisitions

started since 1975, the acquisitions overran original schedule estimates by

an average of 24 percent. Recent DOD studies have concluded that weapon

system development programs typically overrun costs by 20 to 40 percent

and that acquisitions average 16 to 18 years. A 1993 Rand study concluded

that there had been no improvement in controlling cost growth on the

average weapon system. These results have persisted despite the

implementation of various initiatives to mitigate cost risk and growth,

including significant risk management guidelines DOD instituted in 1985 to

improve the transition to production.

Another reason for DOD pursuing faster and smoother development to

production cycles is to lower its investment costs, enabling the services to

modernize at a faster pace within existing funding levels. According to

DOD, modernization in recent years has been sacrificed to improve

readiness in the forces. DOD believes that modernization should proceed

more quickly and wants to increase the annual investment in procurement

by $20 billion, but such increases have not materialized. Shorter

development cycles would also help mitigate the possibility that

technology will become obsolete while a weapon is still in development.

Technological obsolescence has become a significant challenge for DOD

acquisition programs whose developments span many years. Finally, faster

Page 16 GAO/NSIAD-98-56 Best Practices
Chapter 1 

Introduction

product developments would deliver improved products and better

capabilities to the military forces sooner.

DOD recognizes the importance of reducing product development costs and

cycle times. DOD’s 1996 Technology Area Plan recognizes that the

prevailing acquisition environment made affordability and rapid cycle

times for acquisition of new systems key to maintaining an appropriate

mix of systems and forces that are ready to respond to the defense

missions of the future. The plan recognized that short cycle times

invariably create cost reductions that would have clear implications for

readiness and modernization. It also noted that world-class commercial

companies have demonstrated overall product development and

production cost reductions as well as cycle time reductions of about

50 percent. The plan states that, despite the major differences in products

and requirements, cost and cycle time savings in that same range are

feasible for military programs as well.

DOD is attempting to improve the product development process by

introducing acquisition reform initiatives that mirror world-class

companies’ product development practices. DOD believes that these

initiatives will result in better outcomes. A recent Defense Science Board

study1

 concluded that the research and development phase of military

systems should be revamped and should adopt best commercial practices.

The aerospace industry has also offered suggestions based on commercial

practices for improving DOD’s development process for weapons as a basis

for change.

Objectives, Scope,

and Methodology

The Chairman and the Ranking Minority Member, Subcommittee on

Acquisition and Technology, Senate Committee on Armed Services,

requested that we examine various aspects of the acquisition process to

identify best practices that might be useful or should be strengthened. This

report covers one aspect of the acquisition process, the transition from

development to production. Our overall objective was to determine

whether commercial practices offer ways to improve DOD’s product

development process as it relates to the transition to production.

Specifically, this report (1) compares DOD’s practices for preparing a

weapon system for production with best commercial practices,

(2) determines how differences in the commercial and DOD environments

for developing new products affect specific practices, and (3) discusses

1

A Streamlined Approach to Weapon Systems Research, Development and Acquisition: The

Application of Commercial Practices, a Report from the Defense Science Board Task Force on

Defense Acquisition Reform, May 1996.

Page 17 GAO/NSIAD-98-56 Best Practices
Chapter 1 

Introduction

environmental changes that are key to the success of DOD initiatives for

improving the transition of weapons from development to production.

Even though we selected firms with product lines of varying complexity,

we did not concentrate only on firms whose products had the most in

common with weapon systems. Such an approach would have limited our

ability to include firms recognized as the best at getting new products into

production. In our analysis, we concentrated on the criteria and

knowledge used to support product decisions. Although the approach

from product to product may vary, the basic processes and standards the

best commercial firms applied to product decisions were consistent. We

were limited in our ability to obtain and present some relevant data that

commercial companies considered proprietary in nature. This information

included funding amounts for investing in product development or for

recurring production costs; the exact number and cost of changes to

engineering drawings on a product; and specific costs of scrap, rework,

and repair on products.

To determine historical cost and schedule outcomes on past DOD

acquisitions, we examined Rand’s database of Selected Acquisition

Reports for over 200 acquisitions since the 1960s. To determine current

DOD policy and practices and identify new initiatives, we interviewed and

obtained documents from officials of the Office of the Secretary of

Defense in Washington, D.C. For information from acquisitions on specific

risk management practices and results, we interviewed officials and

reviewed acquisition and risk management documentation from the

following two major DOD acquisition programs, which are in different

stages of the process:

• The C-17 aircraft, an air refuelable, four-engine jet transport, is being

developed by McDonnell Douglas Corporation (since acquired by Boeing).

It received full-rate production authority in November 1995. The Air Force

estimates the cost to develop and produce 120 C-17 aircraft at about $43

billion, amounting to a program unit cost of $358 million. Procurement

unit cost is estimated by the Air Force at $298 million.

• The F-22 fighter, the next-generation air superiority fighter, is being

developed by Lockheed Martin Aeronautical Systems Company. It is

currently in EMD. The Air Force estimates the cost to develop and produce

339 F-22s at about $62.2 billion, amounting to a program unit cost of

$183 million. Procurement unit cost is estimated at $127 million.

Page 18 GAO/NSIAD-98-56 Best Practices
Chapter 1 

Introduction

We also reviewed the following newer DOD acquisition programs that have

implemented some of DOD’s acquisition reform initiatives; because these

programs are newer, our review was more limited:

• The AIM-9X Sidewinder Missile, a joint Navy/Air Force, launch-and-leave air

intercept missile, is being developed by Hughes Corporation. The program

is currently in EMD. The services estimate the cost to develop and produce

10,049 AIM-9X missiles at about $3.2 billion, amounting to a program unit

cost of $320,000. Procurement unit cost is estimated at $264,000.

• The Joint Direct Attack Munitions (JDAM), a joint Navy/Air Force tail kit to

be attached to a 2,000-pound free-fall bomb and converting it to a guided

munition, is being developed by McDonnell Douglas Corporation (since

acquired by Boeing). The JDAM program is currently in EMD. The services

estimate the cost to develop and produce 87,500 JDAM kits at about

$3.39 billion, amounting to a program unit cost of $38,700. Procurement

unit cost is estimated at $32,900.

To understand the product development process and best practices from

the commercial sector, we conducted general literature searches and

focused those searches as the assignment progressed. We also met with

several experts in the area of product development, including

representatives of Indiana University, Bloomington, Indiana; the University

of Michigan, Ann Arbor, Michigan; and the Navy’s Center of Excellence for

Best Manufacturing Practices, College Park, Maryland. On the basis of our

literature searches and discussions with experts, we identified a number

of commercial firms as having innovative development processes and

practices that resulted in successful transitions to production. We

developed a data collection instrument to assist us in gathering uniform,

quantifiable measurements about each firm’s product development

process and practices and the results they accomplished. We visited the

following commercial firms, all identified in our literature searches, and

followed the same agenda with each one:

• Boeing Commercial Airplane Group (airplane manufacturer), Everett,

Washington;

• Chrysler Corporation (automobile manufacturer), Auburn Hills, Michigan;

• Cummins Engine Company (engine manufacturer), Columbus, Indiana;

• Ford Motor Company (automobile manufacturer), Dearborn, Michigan;

• Honda Motor Company (automobile manufacturer), Raymond, Ohio; and

• Hughes Space and Communications (satellite and spacecraft

manufacturer), Los Angeles, California.

Page 19 GAO/NSIAD-98-56 Best Practices
Chapter 1 

Introduction

Our report highlights the best commercial practices in product

development based on our fieldwork. As such, they are not intended to

describe all commercial industry and practices or suggest that commercial

firms are without flaws.

We conducted our review between October 1996 and September 1997 in

accordance with generally accepted government auditing standards.

Page 20 GAO/NSIAD-98-56 Best Practices
Chapter 2 

Knowledge at Key Junctures Is Critical to a

Successful Transition to Production

Product development can be characterized as the reduction of risk and

resolution of unknowns through the acquisition of knowledge. Compared

with the DOD programs we reviewed, commercial firms gain more

knowledge about a product’s performance, producibility, and ability to

meet customer requirements much earlier in the product development

process. The best firms will not launch a new product development unless

they have high confidence that they have achieved a match between what

the customer wants and what the firms can deliver. The firms’ keen

knowledge of their technological capabilities and limitations frames the

design features they are willing to offer. Before the midway point in

product development, the firms attain enough knowledge to ensure that

the product works. By the end of product development, but before

fabrication of production items begins, commercial firms prove that their

production methods will yield the desired volume and quality of the

product within the desired unit cost. Throughout product development,

the firms adhere to their own proven standards as to what constitutes an

acceptable level of knowledge. By the time actual production begins, few

risks or unknowns remain.

The DOD programs we reviewed allowed more technology development to

carry over into product development. Consequently, they proceeded

through product development with much less knowledge about required

technologies, design capability, and producibility than the commercial

firms we visited. Two programs—the C-17 and the F-22—did not or will

not attain the same level of knowledge of the design’s ability to perform or

be produced until late in development or early production. Attaining the

match between product design and customer requirements will not be

certain until testing is completed late in development, a practice that is not

acceptable in commercial product developments. Proof that production

methods can yield the desired volume and quality within the desired unit

cost will occur after production begins. As a result, the DOD programs tend

to have more unknowns and greater risks to manage when production

fabrication begins. DOD’s review mechanisms tended to overstate

knowledge or understate risks, as borne out by problems or unknowns

discovered later in product development. Two programs we

reviewed—JDAM and the AIM-9X—have attempted to make a closer match

up front between demonstrated technology and customer requirements.

They are thus in a better position to gain critical knowledge sooner in the

remainder of their development phases.

Page 21 GAO/NSIAD-98-56 Best Practices
Chapter 2 

Knowledge at Key Junctures Is Critical to a

Successful Transition to Production

Commercial Firms

Develop Knowledge

Sooner in the Product

Development Cycle

The successful management of cost, schedule, and performance risk in

transitioning a new product from development to production is related to

how soon full knowledge about key dimensions of the product is attained.

In this sense, knowledge is not the same as information. Knowledge, in

this context, means that program managers and decisionmakers have

reached virtual certainty about an aspect of the product being developed,

such as a critical manufacturing process. It is the inverse of risk.

Information is essential to attaining knowledge, but not all information

produces the same level of certainty. For example, a study that shows a

new manufacturing process should work does not produce the same

degree of certainty that actual use of the process on another product does.

For the purposes of comparing commercial and DOD product development

cycles, we have characterized the point at which virtual certainty of some

aspect of a product is achieved as a “knowledge point.”

The commercial and military programs we reviewed did not all follow the

same processes or steps in their development cycles. However, at some

point, full knowledge was or will be achieved about a completed product,

regardless of what development approach was taken. Knowledge on

product developments can be broken down into three points or junctures:

when a match is made between the customer’s requirements and the

available technology; when the product’s design is determined to be

capable of meeting performance requirements; and when the product is

determined to be producible within cost, schedule, and quality targets.

Program launch or start is the point at which organizations define a

product’s performance, cost, and schedule estimates and commit to

making the financial investment needed to complete development and

bring the product into production. Figure 2.1 illustrates the three

knowledge points and the differences between the commercial and

military product developments in terms of when they attain knowledge.

Page 22 GAO/NSIAD-98-56 Best Practices
Chapter 2 

Knowledge at Key Junctures Is Critical to a

Successful Transition to Production

Figure 2.1: Comparison of Three Key Knowledge Points for Commercial and Military Product Developments

| |

| |

Knowledge Point 1

Knowledge Point 2

Knowledge Point 3

Commercial Product Development

Program Launch Production Start

DOD Product Development

Program Launch Production Start

Knowledge Point 1

Knowledge Point 2

Knowledge Point 3

Unknowns Knowns

Knowledge Point 1: Knowledge that a match exists between technology and requirements.

Knowledge Point 2: Knowledge that the design will work as required.

Knowledge Point 3: Knowledge that the design can be produced within cost, schedule, and 

quality targets.

Page 23 GAO/NSIAD-98-56 Best Practices
Chapter 2 

Knowledge at Key Junctures Is Critical to a

Successful Transition to Production

Because military programs tend to start product development with more

unknowns, it takes them additional time, sometimes until well after

production begins, to actually discover and capture enough solid

information to attain full product knowledge and thereby virtually

eliminate risk. The programs continue to resolve risks about the

performance of the product design and its producibility after they begin

production. Commercial firms attain full product knowledge before

fabrication of production units begins.

Decisions made in matching requirements with technology have a direct

impact on what is required to reach subsequent knowledge points. Synergy

exists when a decision early in product development, such as establishing

product requirements, reduces the amount of unknowns that have to be

resolved before full knowledge of product design performance and

producibility is attained. Conversely, a decision to reach for a

technological advance can increase the effort required to attain full design

performance knowledge, which, in turn, can delay the attainment of

producibility knowledge.

Knowledge Point 1:

Requirements and

Technological

Capability Are

Matched

To minimize the amount of technology development that occurs during

product development, the commercial companies we visited employ

disciplined processes to match requirements with technological capability

before product development begins. The process reflects production

realities and demands proof that the technology will work and can be

produced at an acceptable cost, on schedule, and with high quality. The

companies bring solid technological knowledge to the requirements

process in the form of current, high-fidelity information from predecessor

programs, people with first-hand experience on those programs, or new

technologies deemed mature as a result of having “graduated” from a

disciplined technology development and screening process. In addition,

the companies communicate extensively with customers to match their

wants and needs to the firm’s available technology and its ability to

manufacture an appropriate product. They do not stray far from their

technological foundation. In the commercial world, this process is

sometimes referred to as Quality Functional Deployment, a technique used

to convert complex or unclear customer requirements into design and

manufacturing requirements.

For the programs we reviewed, DOD did not get as good a match between

customer requirements and the technology that could be confidently

delivered. Usually, the individual services identify a need for a capability

Page 24 GAO/NSIAD-98-56 Best Practices
Chapter 2 

Knowledge at Key Junctures Is Critical to a

Successful Transition to Production

and translate that need into specific weapon performance objectives and

thresholds. Even though DOD examines the potential for available

technology to meet product requirements, it allows requirements to drive

technology and make it reach beyond what is proven. DOD has less

knowledge than commercial firms—in the form of predecessor programs,

people with experience on those programs, and technology

maturity—about the ability of the design to deliver required performance

and be produced at the point it starts a program. The knowledge that the

match has been achieved between required performance and the

capability of technology is often not attained until after production begins,

when operational testing is completed.

Commercial Practices We found examples of best commercial practices for matching

requirements to technology at Boeing, Hughes, and Ford. Boeing

communicated with the airlines to set achievable requirements for the

777-200 airplane and tested the design early. Hughes used a technology

development process that graduated new technologies from concept into a

product development program, enabling the firm to make what it saw as

quantum performance increases with mature technologies. Ford uses its

technology deployment process to separate immature technology from a

new product’s development. It enforces a decision point in product

development in which new technology must show proof that it will meet

cost and performance expectations to be accepted onto a product’s

development program.

Boeing established requirements for the 777-200 airplane after extensive

communications with several airlines over an 18-month period before

beginning product development. Originally, the airlines thought that a 767

derivative would serve their purposes. However, during the course of

discussions, Boeing and the airlines realized that a “family” of airplanes

that would provide the airlines’ range and payload flexibility was required.

During that time, Boeing matched customer’s needs to its technological

capability using an airline working group made up of airline

representatives to establish market and product requirements and

extensive performance testing to these requirements. Boeing tested all of

the airplane’s various systems together as a single, integrated entity in

simulated flight conditions very early in product development. In addition,

Boeing completed an initial design review that ensured the design’s

performance, identified all risks, made plans for their resolution, and

selected a final configuration.

Page 25 GAO/NSIAD-98-56 Best Practices
Chapter 2 

Knowledge at Key Junctures Is Critical to a

Successful Transition to Production

The diligence Boeing applied to matching requirements and technological

capability on the 777 was a practice that resulted from painful experiences

on two predecessor programs. On one program, Boeing had erred on the

side of proposing too many new technologies and design features on a

new aircraft design. Boeing halted the program before the launch point,

after one airline customer complained that it contained “too much

technology for technology’s sake.” In retrospect, Boeing was relieved

because the design contained too much risk to make the sound business

case needed for the launch decision. On another program, Boeing had

erred too much on the side of allowing customers to add requirements

after program launch. This program was completed, and aircraft were

delivered; however, delays occurred in the development phase due to

reworking the design, and the early production aircraft experienced

problems with reliability rates. The requirements process for the 777-200

applied the lessons learned from both programs. Figure 2.2 shows the

Boeing 777 airliner.

Page 26 GAO/NSIAD-98-56 Best Practices
Chapter 2 

Knowledge at Key Junctures Is Critical to a

Successful Transition to Production

Figure 2.2: Boeing 777 Airliner

Lessons learned from previous programs helped Boeing match technology with requirements

before launching the 777.

Source: Boeing Commercial Airplane Group.

Because of early communication and testing, Boeing met the airlines’

requirement with plans for a family of airplanes in the medium-lift

category that would meet the airlines’ needs in a timely manner. The

777-200 model was first, followed by models that would increase capacity

and range. Boeing also found that it could meet requirements for the

777-200 using mostly existing technologies. When new technologies were

needed, such as digital avionics and advanced materials, they had to be

demonstrated with laboratory testing, simulation, and modeling before

Page 27 GAO/NSIAD-98-56 Best Practices
Chapter 2 

Knowledge at Key Junctures Is Critical to a

Successful Transition to Production

introducing them to the development program. Boeing also decided

against some new technology on the 777 as the result of its matching of

requirements to capability. For example, Boeing originally designed the

airplane using a new, lightweight material—aluminum lithium—for some

portions. However, at the design review that preceded the start of

development, manufacturing members on the design/build team argued

that the material was expensive and hard to find and that the fabrication

processes for it were not yet fully understood. Despite the increase in

weight that would result, the team decided against using the material.

Boeing made the customers aware of the problem and worked with them

to reduce the weight. Ultimately, the final 777-200 configuration was 

1,500 pounds under its originally proposed weight.

According to Hughes Space and Communications, its newest product, the

HS-702 satellite, was a quantum leap in satellite technology. It provided

twice as much power and capacity as its predecessor; however, it was

made with mature technology. Hughes used its technology development

process to demonstrate new technology before inserting it into the

HS-702’s development. Specifically, it uses Technology Roadmaps to

prioritize technology against business needs and corporate strategies, and

investments are made accordingly. Product development managers will

not use the resultant technology until it proves capable of meeting product

performance, cost, schedule, and quality requirements and manufacturing

processes are in place and in control.

Hughes identified three technologies it considered high risk when it began

development of the HS-702 satellite: the gallium arsenide solar cells and

solar concentrators, the xenon-ion propulsion system, and the deployable

radiator needed to radiate additional heat resulting from the increased

power. The new solar cell and propulsion system technologies had been

through the corporate technology development process and had been

applied on predecessor programs, so there was significant knowledge

about them at the outset. The deployable radiator was high risk because

Hughes did not believe it had the technology for this capability. Hughes

found a Russian supplier that was already producing the required

technology, greatly reducing technical risk. At Hughes, technology that is

not proven through predecessor programs or the development process is

not allowed onto a new product. For example, Hughes wanted to include

lithium ion batteries and advanced receiver technologies on the HS-702

satellite to decrease weight and improve its capacity and power, but it did

not include them because these technologies had not met cost, schedule,

performance, and quality targets.

Page 28 GAO/NSIAD-98-56 Best Practices
Chapter 2 

Knowledge at Key Junctures Is Critical to a

Successful Transition to Production

Ford reduces risk from new technology during product development with

its Technology Deployment Process. This process prioritizes, matures, and

introduces new or emerging technologies for product development,

reducing risk from the time the concept is proposed to its ultimate

implementation on a new product. During this process, technologists

agree to deliver new technology to vehicle centers for product

development. Managers of product development are customers to

technologists and can agree or refuse to sponsor technology, which

creates a shared understanding of deliverables between the technologist

and the product manager. Each technology project has a concept

readiness milestone, which includes a demonstration of the hardware and

its required manufacturing processes, as well as testing and analysis

results. Also, each technology has an implementation readiness milestone,

which includes proof that the technology is ready for product

development with identified and manageable risks. At this point, the new

technology’s cost, schedule, and quality targets must be met.

Ford enforces a “Wall of Invention” at the start of each new product’s

development. Beyond this point, no new technologies or design features

are allowed to be added to the vehicle. From then on, the product manager

takes responsibility for successfully developing and producing the new

vehicle. Product managers are veterans of previous production programs,

and their production knowledge makes them very discriminating in

deciding what new technology will be allowed onto the vehicle.

DOD Practices DOD programs did not attain a match between technology and

requirements at the time of launch. DOD accepted varying—but

consistently higher—degrees of technological risk on four of the product

development programs we reviewed. Technologies ranged from

comparatively high-risk advances for the F-22’s avionics and low

observability to low-risk advances for mostly off-the-shelf components

being used to develop the AIM-9X missile and JDAM. The C-17 program was

developed using mostly existing technology, with some demanding

features required of the airframe design. DOD practices for matching

product requirements to available technology on the AIM-9X and JDAM

appeared to do a better job of reducing risks.

The F-22 development program has reduced risk from new technologies

across most of its key design features through extensive, high-fidelity

modeling and analysis. However, the program’s product development

includes new low-observable materials, avionics, and propulsion

Page 29 GAO/NSIAD-98-56 Best Practices
Chapter 2 

Knowledge at Key Junctures Is Critical to a

Successful Transition to Production

technology whose performance is undemonstrated and raises some

concern about producibility. Low observability is one of the F-22’s most

important characteristics and is dependent on many factors, such as the

shape of the airframe, its flight characteristics, and airframe materials and

coatings. Although the program has done pole model testing of the aircraft

to determine whether it meets its low-observability requirements, the

aircraft includes 10 newly developed derivatives of existing materials that

are important to its low-observability feature. The materials will have had

3 years of flight testing before they are applied to the first production

aircraft; however, the performance and maintainability of these materials

cannot be completely verified until radar signatures can be tested in

varying climates. This testing is scheduled to be completed 2 years after

production begins.

Similarly, much remains to be proven on the F-22’s avionics technology

before it is certain that the technology is a match for the performance

requirements. The avionics software features a level of integration not

previously achieved in a fighter. One major technical challenge is

perfecting the way in which various avionics subsystems will interface.

The program has incorporated a series of risk reduction activities

designed to mitigate these software integration problems. For example, it

has defined and replicated all of the avionics interfaces at this point in

development, but this testing will not prove that the various subsystems

will interact in identifying and prioritizing threats. The program has begun

testing in an avionics integration laboratory and on a flying avionics test

bed mounted on a Boeing 757 and will further test the avionics capabilities

for prioritizing threats at very high densities and conditions once the test

facility is ready. However, until that type of testing and flight testing begin

to produce results—in 1999 at the earliest—there are significant

unknowns concerning the capability of integrated avionics on the F-22.

The program plans three incremental updates to the software, completing

its development in 2002. By this time, Lockheed plans to have at least 

40 production F-22s, an investment of at least $8 billion, finished or in the

manufacturing process.

The F-22 engines have many advanced features to meet aggressive

performance requirements. One such requirement is for supercruise—the

ability to sustain supersonic speeds without the use of afterburners. Some

of these advanced features resulted in components that are more difficult

to produce than originally anticipated and are resulting in

higher-than-planned recurring production costs. The producibility risk is

largely driven by unknowns concerning the hollow fan blades, some

Page 30 GAO/NSIAD-98-56 Best Practices
Chapter 2 

Knowledge at Key Junctures Is Critical to a

Successful Transition to Production

components fabricated from composite materials, the turbine exhaust

case, and a few features of the exhaust nozzle. A product cost reduction

process has been implemented to mitigate these risks, but program

officials acknowledged that the user’s overall performance requirements

for the engines remain very challenging, and it is still unclear whether the

engines will meet all of the requirements.

For the most part, the C-17 was developed using nondevelopmental items

or commercial parts. However, performance requirements for short

landing and takeoff, payload, and range required design innovations that

significantly increased technical risk on the program. A new technology

that added risk to the program was aluminum lithium, a new, unproven

alloy that the program used for its strength and weight savings. The first

production aircraft contained 2,200 pounds of the alloy. Its application

was unsuccessful due to a lack of knowledge about the technology’s

characteristics. The program discontinued its use because of its potential

to warp during handling, cracking and chipping during installation, the

need to keep it separate from other materials, and concerns over sources

of supply. It is being phased out of the program and the 51st production

airplane will be free of it. The C-17 program replaced aluminum lithium

with aluminum alloy 7050, a less expensive and easier-to-handle material

with several sources of supply. Boeing made the same decision at the

outset of its 777-200 development program based on similar conclusions

reached by its design/build team. This is a good illustration of different

knowledge standards applied in making design decisions. Figure 2.3 shows

the C-17 aircraft.

Page 31 GAO/NSIAD-98-56 Best Practices
Chapter 2 

Knowledge at Key Junctures Is Critical to a

Successful Transition to Production

Figure 2.3: C-17 Cargo Aircraft

Although the C-17 was developed mostly with nondevelopmental items and commercial parts,

performance requirements significantly increased technical risk.

Source: DOD.

Also, the software development effort to meet avionics performance

requirements turned out to be significantly more complex than the Air

Force thought. When the C-17 development program began in 1985, for

example, the Air Force had identified 4 subsystems with about 164,000

lines of code that had to be developed. By 1990, this number had increased

to 56 subsystems and about 1,356,000 lines of code, including

approximately 643,000 newly developed lines of code.

Page 32 GAO/NSIAD-98-56 Best Practices
Chapter 2 

Knowledge at Key Junctures Is Critical to a

Successful Transition to Production

The JDAM and AIM-9X development programs have been more attentive to

matching customer requirements with technological capabilities. Both

programs chose designs that relied strongly on predecessor data, and both

programs emphasized affordability as a key product requirement. JDAM

used modified variants of proven product lines for its guidance component

and global positioning system. It also used mature, existing components

from other proven manufacturing processes for its own system for

controlling tail fin movements. The designs for the battery and the tail

housing both use mature technology and will be built using mostly existing

tooling and processes. The program office and contractor have been able

to further minimize risk by conducting extensive development,

qualification, and flight testing to demonstrate performance, quality, and

reliability on some key subsystems.

Except for its tracking technology, the AIM-9X air-to-air missile will use

technology largely from manufacturing processes already in place or from

the United Kingdom’s Advanced Short-Range Air-to-Air Missile production

program. For example, the sensor is currently being developed for the

United Kingdom missile, and over 2,500 sensors will have been produced

before the first AIM-9X is assembled. The AIM-9X airframe is adapted from a

past Air Force flight test program and is made up of proven technology.

The rocket motor, engine, warhead, and fuse are all taken directly from

AIM-9M missiles and are already in production. The AIM-9X tracking system

is the primary area of new technology. It is a sophisticated, computerized

component that translates the light image obtained by the seeker into

electronic signals that can be viewed by the pilot, discriminate against

countermeasures, and guide the missile to a particular point on the target

aircraft. It provides the seeker a wider, higher fidelity view in which to find

targets. This new technology has been in development for some time and

is made up of off-the-shelf components. Hughes believes this risk is

manageable because of modeling and simulation that preceded the

development program.

Knowledge Point 2:

The Design Will

Perform as Required

The completion of engineering drawings and their release to

manufacturing organizations signify that program managers are confident

in their knowledge that the design performs acceptably and can be

considered mature. The drawings are critical to documenting this

knowledge because they are not only precision schematics of the entire

product and all of its component parts—they also reflect the results of

testing and describe the materials and manufacturing processes to be used

to make each component. Both DOD and commercial companies consider

Page 33 GAO/NSIAD-98-56 Best Practices
Chapter 2 

Knowledge at Key Junctures Is Critical to a

Successful Transition to Production

the design to be essentially complete when about 90 percent of the

engineering drawings are completed. Both sectors schedule a critical

design review (CDR) to review the drawings, confirm the design is mature,

and “freeze” it to minimize changes in the future. Figure 2.4 compares

what knowledge, in the form of released drawings, was in hand at the time

of CDR for the commercial and DOD programs we reviewed.

Figure 2.4: Comparison of When Commercial and DOD Programs Achieve Knowledge About Their Product’s Design

Hughes HS-702

Boeing 777-200

DOD F-22

DOD C-17

Program Launch Production Start

CDR and Knowledge Point 2

CDR and Knowledge Point 2

CDR Knowledge Point 2

CDR Knowledge Point 2

Unknowns Knowns

Program Launch Production Start

Commercial Practices The commercial firms we visited released over 90 percent of the products’

engineering drawings at CDR. At that point—about midway through

development—the firms had near certainty that their product designs

Page 34 GAO/NSIAD-98-56 Best Practices
Chapter 2 

Knowledge at Key Junctures Is Critical to a

Successful Transition to Production

would meet requirements and had gone a long way to ensure that the

product could be produced.

Boeing had full knowledge of the 777-200 design’s capability at its final

CDR. It began releasing engineering drawings after a preliminary design

review when product development began in 1990 and completed this

release process in 1992, less than 2 years later. CDR at Boeing is generally

done by peer organizations, such as engineers from another program.

Outside consultants are sometimes used. Boeing officials told us that, once

the design drawings were released to the manufacturing process, they

averaged less than two changes per drawing, well below their average on

past programs and exceeding their goal for the 777-200 program. Once CDR

was complete, Boeing strictly enforced the design freeze for the 777-200.

For example, Boeing incorporated a customer requirement to include

folding wingtips, along with the supporting bulkheads, into the 777 design

at a cost of nearly $40 million. Later, the customer decided the folding

wingtips were not necessary; however, Boeing left the bulkheads in the

wings anyway because all of the engineering drawings were completed

and the risk of introducing changes, even though the changes would have

saved weight, was considered too high relative to cost and schedule

targets.

Hughes, which designs both military and commercial satellites in the same

facility, completed two design reviews for development of the HS-702.

First, it passed an internal review to satisfy the corporation that its basic

design was producible. Second, it reiterated design reviews on individual,

customized satellites as they were ordered by customers. During the

internal development process, Hughes began releasing engineering

drawings at preliminary design review and completed releasing drawings

at CDR, about 15 months into a 26-month product development process.

Hughes’ goal was to complete and release all engineering drawings at CDR;

however, representatives told us the drawings were close to 95 percent

released at that point.

Chrysler described several practices for ensuring mature designs. It uses

information from predecessor products to quickly develop focused

updates and innovative derivatives to existing products. This predecessor

data, along with its use of platform teams, allows Chrysler to take

advantage of existing information and expertise in new product

developments. It is able to simultaneously develop and test the product’s

design and its required processes, which it described as true concurrent

engineering at the right time. Chrysler uses prototypes to refine the

Page 35 GAO/NSIAD-98-56 Best Practices
Chapter 2 

Knowledge at Key Junctures Is Critical to a

Successful Transition to Production

product and capture design knowledge early. It also employs a pilot

production process and extensive testing facilities at its technical center to

prove the design’s performance and producibility before it is released to

manufacturing facilities.

DOD Practices The C-17 and the F-22 programs had less knowledge—in the form of test

results or engineering drawings—about their designs than commercial

companies did at the time they held their CDRs. The programs did not get

or were not projected to get to the same level of completion on the

drawings until later in the development cycle, which placed greater

reliance on the lesser information available at CDR. This is important

because the review is a major event that represents a point of departure

from detail design to manufacture of the product. The risks of proceeding

with CDR and the rest of development as planned are increased without the

requisite drawings in hand. This was deemed acceptable for both

programs.

The C-17 program had released 10,229 of its engineering

drawings—56 percent—at CDR in August 1988, just past the midway point

in development. After CDR, an additional 8,161 drawings were released,

over 4,000 of which were released after assembly of the first production

aircraft had started. We estimate that the C-17 program did not release

95 percent of its initial drawings until December 1991, more than 3 years

after CDR and after seven production aircraft had been delivered to DOD.

The C-17 encountered numerous technical problems during testing that

resulted in redesigns, cost increases, and schedule delays. For example,

flight tests in 1991 revealed that an innovative system of blown wing

flaps—essential to the C-17’s ability to fly steep landing approaches at

slow speeds to access shorter runways—suffered heat damage and

acoustical cracks. To correct this problem, the contractor had to

completely redesign the slats to include titanium skin and substructure.

This knowledge was not obtained until the program was at least two-thirds

into development. Also, in 1992 both wings on the static test article failed

at approximately 128 percent during a 150-percent limit load test and had

to be redesigned.

Software development and management problems also contributed to C-17

cost and schedule problems. The Air Force lacked specific knowledge

about software development problems as they occurred. The first C-17

aircraft achieved first flight 19 months later than planned and did not

include many of the mission-critical software functions required for a fully

Page 36 GAO/NSIAD-98-56 Best Practices
Chapter 2 

Knowledge at Key Junctures Is Critical to a

Successful Transition to Production

operational aircraft. Problems such as these caused completion of testing

on the C-17 to be delayed by well over 1 year.

The F-22 program had released 3,070 initial structures and systems

drawings, less than one-third of the eventual drawings, at its CDR held at

the midway point in development. Since then, an additional 6,032 drawings

have been completed and released. The drawing release process is not yet

complete for the F-22, so we cannot determine if the program has yet

released 95 percent of its drawings. F-22 production is scheduled to begin

in June 1999.

Even though it is still too early to predict outcomes on the AIM-9X missile

and the JDAM development programs, their prospects appear promising

because they have chosen mostly proven technology from existing

programs to achieve performance requirements. They appear to have

robust, mature designs that will use manufacturing processes already

identified and demonstrated to be stable or in control. JDAM held its CDR in

August 1995. At that time, it had released 65 percent of its drawings. In

addition, it demonstrated how the product would meet requirements using

data, analysis, and a physical display of the product. The time it took to

build up and load a JDAM was also demonstrated. AIM-9X has not yet held its

CDR, but program officials indicated that they will have already built a

prototype before CDR and hope to freeze the design based on that

prototype.

Knowledge Point 3:

Production Units Will

Meet Cost, Quality,

and Schedule

Objectives

The companies we visited reached the point at which they knew that

manufacturing processes would produce a new product according to cost,

quality, and schedule targets before they began fabricating production

articles. This meant more than knowing the product could be

manufactured; it meant that all key processes were under control, so the

quality, volume, and cost of their output were proven and acceptable. The

C-17, in production for 7 years, still does not have all of its processes

under control. The F-22 program is not scheduled to have all of its

processes under control until the 4th year of production. The AIM-9X and

JDAM programs appear to be doing better in this regard. Both programs are

using the same approach as commercial firms to attempt to get their

critical manufacturing processes in control before the programs enter

production. Figure 2.5 illustrates the difference between the commercial

HS-702 and 777-200 programs and the military C-17 and F-22 programs

concerning when full knowledge about the production processes was

achieved.

Page 37 GAO/NSIAD-98-56 Best Practices
Chapter 2 

Knowledge at Key Junctures Is Critical to a

Successful Transition to Production

Figure 2.5: Comparison of When Commercial and DOD Programs Achieve Knowledge That Processes Can Produce an

Acceptable Product

Knowledge Point 3

Knowledge Point 3

Knowledge Point 3 has 

not yet been reached.

Hughes HS-702

Boing 777-200 

DOD F-22 

Program Launch Production Start

Unknowns Knowns

Knowledge Point 3 will

be reached well after

production begins.

Program Launch Production Start

DOD C-17

Commercial Practices The commercial firms relied on existing manufacturing processes and

statistical process control (SPC) data to achieve this knowledge in a timely

manner and, in fact, had all their key processes under statistical control

when production began. SPC is established by monitoring processes to see

if they are consistently producing output that is within the quality

standards and tolerances set for the overall product. Statistics concerning

the quality of each process output are analyzed, and when the output is

out of tolerance, process owners search for causes. Once a process is

producing consistently high-quality output, the process is considered to be

in statistical control, and inspections can be reduced. The knowledge

gained using SPC is significant in transitioning from development to

Page 38 GAO/NSIAD-98-56 Best Practices
Chapter 2 

Knowledge at Key Junctures Is Critical to a

Successful Transition to Production

production because it helps companies ensure that cost, schedule, quality,

and reliability targets will satisfy the customer and the business case.

The ability to establish SPC for key processes before production began was

the culmination of all the practices employed to identify and reduce risk.

The criteria commercial companies had established throughout their

product development processes forced program managers to prove that

the product’s design was capable and producible early in the process. For

example, Boeing carried out more than 25 technical and manufacturing

reviews during the development process leading up to production to

ensure that the product’s design was producible. Boeing identified

computer-aided three-dimensional interactive application software (CATIA)

and its design/build teams as crucial to ensuring that processes were in

control and the transition to production went smoothly.1 CATIA software

allowed workers to share design and process information about the 777 in

real time and different locations. Manufacturing engineers could check on

a part’s producibility at any time in the development process.

Hughes told us that it knew that all of the processes to manufacture the

HS-702 were in control because the processes were either already in place

from previous programs and under SPC or they had been proven to be in

control before being released to product development by demonstrations

during the corporation’s overall technology development process. 

Figure 2.6 shows the Hughes HS-702 satellite. Ford had similar practices to

Hughes, often demonstrating SPC on technology before it was inserted into

a product development, even before production begins.

1

CATIA allows designers and manufacturers the opportunity to view design drawings as three

dimensional and provides real-time information to everyone in the process about the impact a change

to a drawing will have on the overall design and manufacture of the product.

Page 39 GAO/NSIAD-98-56 Best Practices
Chapter 2 

Knowledge at Key Junctures Is Critical to a

Successful Transition to Production

Figure 2.6: Hughes HS-702 Satellite

Hughes has achieved statistical process control on the low-volume HS-702 satellite before

beginning production.

Source: Hughes Space and Communications.

Chrysler’s use of platform teams, CATIA software, and its technical center

all contribute to a smooth transition to production and ensure that

processes are in control for every new product. Perhaps as important,

synergy develops with a disciplined product development process that

accumulates so much knowledge up to this point. All of the companies we

visited agreed that knowledge about technology and design up front in the

Page 40 GAO/NSIAD-98-56 Best Practices
Chapter 2 

Knowledge at Key Junctures Is Critical to a

Successful Transition to Production

process makes the control of processes possible and the transition to

production smooth.

DOD Practices The DOD programs we reviewed demanded less proof than commercial

firms of a design’s producibility before a product transitions to production.

The C-17 did not achieve SPC on all key processes before production and

will not reach that point until well into production. The F-22 is not slated

to achieve control on all key processes until after production begins.

Moreover, basic producibility problems were not discovered until late in

the F-22’s development and after production began on the C-17. These

risks went unrecognized even though both programs had established

criteria for ensuring that risks were acceptable and enough knowledge had

been gained before proceeding to the next program phase. These exit

criteria were light on production-related requirements, even at the

decision point for proceeding with production. For example, the F-22

program’s exit criteria for moving to full production does not require

information from the final production readiness review until after the

decision, and the C-17 program’s exit criteria did not require proof of

producibility. Neither set of criteria contained requirements about drawing

releases nor the schedule for achieving control on key processes.

The C-17 program began production in 1989 with less than 13 percent of its

key manufacturing processes in control, despite completing production

readiness reviews that were intended to reduce producibility problems.

The program has produced and delivered nearly 40 aircraft and has now

identified 420 key manufacturing processes. Of those, only 56 are under

statistical control. This inability to control manufacturing processes was

caused, in part, by an immature design. For example, the C-17 program

discovered major design changes of the wings, flaps, and slats were

required after CDR, as prototype aircraft were being built. In addition to

causing high rates of drawing changes, these late design changes caused

problems on the production line. They necessitated costly changes to

processes; forced the manufacturers to develop workaround plans that led

to labor inefficiencies; and resulted in high rates of scrap, rework, and

repair. The C-17 program estimates scrap, rework, and repair costs on the

34 delivered aircraft to be over $200 million.

The F-22 program is ahead of the C-17 at this point regarding SPC, perhaps

due in part to its use of CATIA. Of 926 key manufacturing processes

identified, almost 40 percent of its key processes are in control now,

2 years before production is scheduled to begin. Nonetheless, the

Page 41 GAO/NSIAD-98-56 Best Practices
Chapter 2 

Knowledge at Key Junctures Is Critical to a

Successful Transition to Production

production preparations for the F-22 illustrate the limitation of a review

mechanism when a substantial amount of knowledge is unattained. The

program’s initial production readiness review, held in 1995 when only

about one-third of the engineering drawings were released, did not report

any high risks in manufacturing or producibility, although clearly much

was unknown at the time and drawing release was slower than called for

by DOD guidance. In 1996, an independent team mandated by the Air Force

reviewed the program and discovered numerous manufacturing and

producibility problems, such as underestimated complexity in

manufacturing processes, understated labor requirements for building the

aircraft, immature definition of avionics flight test requirements, and

concerns about software integration. These and other concerns led to

increases to development and production cost estimates of $2.1 billion and

$13 billion, respectively. Air Force officials believe that these additional

costs can be offset by reprogramming existing funds and by implementing

a number of cost avoidance measures in the future.

The JDAM program office completed manufacturing assessments at the

prime contractor and all major suppliers in February 1997 in such areas as

process controls and plant capacities. These assessments found that all of

the production tooling was in place to build JDAM. The program office has

identified 84 manufacturing processes that must be in control to meet the

product’s key performance characteristics and has achieved statistical

control on 69 percent of the processes at this point in the program, about 

1 year before full-rate production begins. The AIM-9X program is still very

early in its EMD phase but has already identified all of the critical

manufacturing processes that must be in control to meet the product’s key

performance characteristics. The contractor told us that all of those

processes are stable at this point in the program, and most are in statistical

control.

Although the F-22 program may not be characterizing and controlling its

manufacturing processes as quickly as commercial firms do, it offers

perspective on some of the strides that DOD has made in preparing

weapons for production since the 1980s. In 1985, we reported that several

weapons had encountered substantial problems in early production

because they used hand-built development prototypes made in special

shops by expert personnel.2

 Production articles were built in different

facilities with different processes and people. There was no real transition

to production, but rather a sudden shift from development to production.

2

Why Some Weapon Systems Encounter Production Problems While Others Do Not: Six Case Studies

(GAO/NSIAD-85-34, May 24, 1985).

Page 42 GAO/NSIAD-98-56 Best Practices
Chapter 2 

Knowledge at Key Junctures Is Critical to a

Successful Transition to Production

One of the programs at that time, the F-16 fighter, had a much better

experience. We noted that development prototypes were built in the

production facility using as many production processes and people as

possible. This made the transition to production much more gradual and

was one of several factors that helped prepare the F-16 for production.

Although the F-22 is attempting to make a bigger technological leap than

the F-16 did, the development prototypes are being built in the production

facility using some production processes and people.

Page 43 GAO/NSIAD-98-56 Best Practices
Chapter 3 

Differences in Military and Commercial

Practices Reflect Different Environments

The differences in the practices employed by the best commercial firms

and DOD are not necessarily explainable by differences in tools, techniques,

or talent. Rather, the differences in the actual practices reflect the

different demands imposed on programs by the circumstances or

environment in which they were managed. Indeed, the way success and

failure are defined for commercial and defense product developments

differs considerably, which creates a different set of incentives and

different behaviors from the people managing the programs. Specific

practices take root and are sustained because they help a program

succeed in its environment. In this sense, practices are adopted because

they work—not because they are textbook solutions.

In general, the success of a commercial product development is

determined on the basis of production items sold; similarly, failure is

clearly defined as the customer walking away and purchasing the product

of a competitor. This definition of success, coupled with shorter cycle

times, makes production concerns a main determinant in the initial

business case to launch a commercial product development and in

subsequent decisions to manage risk and make technology tradeoffs. This

environment encourages the early identification of unknowns with

realistic risk assessments so that risks are not allowed to jeopardize

success. Strong incentives, both positive and negative, stress getting the

product development right.

DOD product developments are launched much sooner—with greater

technology, cost, and schedule unknowns—and extend much longer.

Production concerns do not play as big a role early in these programs

because production is so far off and thus is not as critical to success. The

definition of success is more complicated in DOD. During most of product

development, success is defined in large part as getting DOD and the

Congress to fund the development on an annual basis. Optimistic

assessments of design performance and cost help ensure this kind of

success; realistic risk assessments for unknowns do not. By the time

production nears, the risk of failure is much reduced because a vested

customer is not likely to walk away.

Page 44 GAO/NSIAD-98-56 Best Practices
Chapter 3 

Differences in Military and Commercial

Practices Reflect Different Environments

Commercial Practices

Are Driven by the

Customer’s

Acceptance of the

Finished Product

The commercial firms we contacted launch a product development

program only when a solid business case can be made. The business case

basically revolves around the ability to produce a product that will sell

well enough to make an acceptable return on investment. The point of sale

occurs after product development is complete; program success is

determined in production when the customer buys the finished product. If

the firm has not made a sound business case, or has been unable to deliver

on one or more of the business case factors, it faces a very real prospect of

failure. Production is a dominant concern throughout the product

development process and forces discipline and tradeoffs in the design

process. This environment encourages realistic assessments of risks and

costs; doing otherwise would threaten the business case and invite failure.

For the same reasons, the environment places a high value on knowledge

for making decisions. Incentives favor identifying unknowns early,

designating them an appropriate high risk, and aggressively eliminating

them. Practices, such as achieving statistical process control before

production, are adopted because they help ensure success.

Production Considerations

Figure Prominently in the

Program Launch Decision

For the commercial firms we contacted, the main focus of a product

development program is to produce and sell the right product at the right

time. The success of a program depends on the product’s sales and not just

on its successful development. Therefore, the business case for launching

a program considers production realities and builds in natural curbs to

overreaching for performance, cost, or schedule. Although the

corporations demand considerable proof in the business plan that the

product will meet market expectations, they then provide full

support—including funding—for the plan to succeed. This provides a

program with a very solid baseline from the outset that continues to serve

as a good decisionmaking framework as the program progresses and

problems arise. The day of reckoning—when program success or failure is

realized—comes clearly and swiftly after development when the customer

takes delivery of the product and is prominent throughout the product

development process.

Boeing described the business case for the 777-200 product development

as a “money wheel” that must be balanced across all of its factors. These

factors include a market opportunity, a product whose technical features

can satisfy the market, available investment capital, a cycle time short

enough to get the product to market on time, and a unit production cost

that will yield an acceptable return on investment. Boeing informed us that

if any factor gets out of line, either through estimating errors or changing

Page 45 GAO/NSIAD-98-56 Best Practices
Chapter 3 

Differences in Military and Commercial

Practices Reflect Different Environments

conditions, the “wheel” will not turn, and profitability—and perhaps

corporate reputation—could be lost. The program manager is judged by

these standards, unlike in DOD.

Before Boeing committed to development of the 777-200, it first

determined that a market opportunity existed for a family of at least 10,000

new, medium-lift airplanes and that the airlines would buy these airplanes

at a certain unit price. The company then had to make the business

case—that it could develop a product with the right features, in time to

meet the market opportunity, and at an acceptable unit cost. Boeing did

not commit to offer the new aircraft for sale until the commercial group’s

president presented solid evidence, on the basis of predecessor programs,

testing, simulations, and analysis, to the board of directors that all of the

business case factors were achievable. The board then approved

investment of capital based on its assessment of the risk involved with

meeting the business case. Boeing program officials pointed out, however,

that it was not the board meeting that provided the incentive to get the

business case right. Rather, they said that the commercial group president

knew that the company’s investment capital was at stake on such a major

product development. An oversold business case would lead to failure and

could have a devastating effect on the company’s future. This had a very

sobering effect on the people that constructed the factors in the business

case and curbed the tendency to overstate the case.

Hughes presented a very similar scenario for the corporate decision to go

forward with the development of the HS-702 satellite. Hughes was very

comfortable with the level of technical risk at the outset because its

technology development process had matured the required technologies

before inclusion in the program. In fact, the biggest risk in its business

case was ensuring that the market for the product was profitable. To

resolve this risk, Hughes spent considerable time educating its customers

about the benefits of the proposed program. Only after potential

customers were convinced did Hughes have a solid business case for going

forward with development.

The other companies we contacted—Chrysler, Cummins, and Ford—had

similar frameworks established for making decisions about product

development. Business realities, such as capital investment, cycle time,

execution in production, and the customer’s sensitivity to purchase price,

created a high-stakes environment. The decision to begin a new

development program was typically made at the highest corporate level.

However, the business case, with all of its competing factors, provided a

Page 46 GAO/NSIAD-98-56 Best Practices
Chapter 3 

Differences in Military and Commercial

Practices Reflect Different Environments

framework and the incentives for program managers to make practical

decisions about product development based on facts and data. This

framework held the achievement of each factor as essential to success and

committed the program managers to assess all associated risks

realistically. Getting a program approved that embodied false or weak

information was viewed as a failure because the customer might not buy

the finished product. One commercial program manager informed us that

his firm used to routinely underestimate the cost and schedule for

developing a new product and overestimate its sales volume and profit

margin. He described this practice as a “death cycle” that nearly

bankrupted the company.

Shorter Cycle Times and

Strong Incentives Foster

Behaviors That Keep

Programs on Track

Once a company decides to launch a product development, strong

incentives—both positive and negative—serve to keep the programs on

track. To meet market demands, leading commercial companies build

relatively short cycle times into decisions to begin a product’s

development. Boeing’s 777-200 went to production less than 5 years after

development began, Hughes’ HS-702 took about 26 months, and Chrysler

developed its new sport utility vehicle in less than 30 months. These short

timeframes make the day of reckoning, in terms of customer acceptance

and return on investment, close at hand. Consequently, production—on

time, at rate, at cost, and with quality—looms as a near-term reality that

continues to greatly influence subsequent design and configuration

decisions within the framework of the business case. The incentives that

operate in the commercial environment encourage program managers to

want risks identified early, be intolerant of unknowns, and not rely on

testing as the main vehicle for discovering the performance characteristics

of the product. By protecting the business case as the key to success,

program managers are conservative in their estimates and aggressive in

risk reduction. Ultimately, preserving the business case strengthens the

ability to say “no” to pressures to accept risks or unknowns.

Boeing believes a number of factors serve as incentives to keep a program

on track. First, because the fate of the program (and to a large extent the

fate of the company) is in the hands of the program’s design/build teams

and their leaders, the incentive is for them to convince themselves that the

business case is solid rather than to “sell” program estimates to corporate

management. Second, the company’s investment capital is fixed. Thus,

development cost increases on any one program that can be absorbed are

limited because, at some point, the overrun will hurt investment on other

programs. Third, the program’s ultimate performance becomes part of a

Page 47 GAO/NSIAD-98-56 Best Practices
Chapter 3 

Differences in Military and Commercial

Practices Reflect Different Environments

scorecard on the program manager’s impact on corporate profitability. A

failure that hurts the company financially will remain associated with the

program manager and other people involved. Fourth, if the product does

not live up to its performance characteristics (which include operation

and support costs), the company will lose money on product guarantees.

Finally, the program management staff understand that more is at stake

than the program at hand; a serious miscue on one program can hurt the

company’s reputation and damage other programs as well.

Although these factors could be seen as negative incentives, Boeing’s

environment also provides strong positive incentives for keeping a

program on track. The board approval to launch a product development is

a commitment to completely develop and produce the item. There is no

turning back unless something occurs that would cause a serious decay in

the business case. The investment funding is approved up front, the

corporation stands firmly behind the product development, and the

program manager has control over the product’s development process.

When problems do occur, the corporate support does not waver, and

problem solving takes place within a well-understood

framework—anchored in the business case—for making tradeoffs. For

example, Boeing realized early in the 777-200 program that investment

costs would be higher than anticipated, largely due to cultural changes it

was trying to accomplish with full use of CATIA and design/build teams.

Boeing’s approach to resolving the problem began with the premise that

the contractual commitment to deliver the aircraft, with all of its features,

on time, and at the same price, would be met. The problem was then

limited to minimizing the increase in investment cost to keep the business

case intact.

Boeing said that it is predisposed to identifying and solving problems early

and paying the up-front costs to do it rather than accept the production

cost consequences of the problems at the end. Its decisions on the use of

aluminum lithium and the bulkhead for the 777-200 wing tip are

demonstrative of behaviors that value minimizing production risks after a

program is underway. Although the aluminum lithium could have been

designed into the aircraft and offered performance improvements, it was

rejected because of the unknowns it presented for material handling on

the production line and for long-term performance under operational

conditions. Similarly, Boeing could have taken the unneeded bulkhead out

of the wing tip to reduce weight, but that idea was rejected because the

redesign would have had a ripple effect on drawings and production

preparations.

Page 48 GAO/NSIAD-98-56 Best Practices
Chapter 3 

Differences in Military and Commercial

Practices Reflect Different Environments

Chrysler officials discussed a similar environment for managing the

product development process, emphasizing the need for discipline to

come from within the program itself. Chrysler establishes a target unit

price, specific performance objectives, and cycle time as part of the

business case for each product’s development. These factors are strongly

enforced—within a fixed-price, firm schedule environment—as a

discipline to manage risk and facilitate product tradeoffs. The platform

teams operate under a “zero-sum” basis, meaning that if the price of one

component is higher than expected, then savings must be found

elsewhere. The platform teams are totally empowered to make all

decisions concerning their product as long as periodic reviews by product

assurance teams indicate that all of the performance objectives are being

met. If the objectives are not being met, the team receives increased

scrutiny and outside direction. This practice creates a strong incentive for

product managers and platform teams to keep product risk to a minimum

by using known product and process technologies and discovering

problems as early as possible.

Chrysler explained an important aspect of the program manager’s role is

to say no to things, such as immature technologies, that may disrupt the

product’s cost, schedule, or performance targets. Chrysler believes that,

without the “bottom line” discipline from the program manager, cost

targets will not work. Any facet of product development that cannot be

shown to be knowledge, rather than projection, is considered an unknown

and designated a high risk. Design and production issues are thus

equalized; a suspension component that can improve vehicle performance

and reduce complexity will be rejected if the proposing team cannot

provide proof that this component can be produced at the rate and cost

called for by the business case. Figure 3.1 shows Chrysler’s Plymouth

Prowler.

Page 49 GAO/NSIAD-98-56 Best Practices
Chapter 3 

Differences in Military and Commercial

Practices Reflect Different Environments

Figure 3.1: Plymouth Prowler

Shortening product development cycles and other incentives help Chrysler managers identify

risks and say no when necessary.

Source: Chrysler Corportation.

In this environment, every team member is considered to have high risk

until each can demonstrate otherwise. Team members are encouraged to

volunteer information early on risks or problems they are having because

the other team members will step in to help, particularly if they have

savings that can offset price increases. The team members know that if

one of them fails, they all fail because the business case decays. They also

know that if they help someone, they increase the willingness of others to

help them. By the same token, the team member that withholds problems

Page 50 GAO/NSIAD-98-56 Best Practices
Chapter 3 

Differences in Military and Commercial

Practices Reflect Different Environments

until late in the development is ostracized for putting the entire team in a

position to fail.

Ford officials believed that the product development environment at the

firm had gotten to the point at which people treated the company’s money

as if it were their own. In our discussions with these officials, we

discerned several factors that converged to keep estimates intrinsically

sound during the product development process. These factors were:

(1) the vast amount of predecessor data from previous product

developments that can be used as a reality check for a new program’s

estimates; (2) the amount and quality of the information being generated

for the program at hand, including the technique being used to generate

data; and (3) the consequences of weak or overstated data on the

program’s success. Figure 3.2 shows the Ford Motor Company’s Lincoln

Navigator.

Page 51 GAO/NSIAD-98-56 Best Practices
Chapter 3 

Differences in Military and Commercial

Practices Reflect Different Environments

Figure 3.2: Lincoln Navigator

Ford reduced the amount of unknowns on the new Lincoln Navigator by limiting product

requirements to proven technologies.

Source: Ford Motor Company.

The individual practices employed by commercial firms are consistent

with the incentives of their environment. The high reliance on CATIA

shortens cycle time and reduces risk by generating knowledge that

Page 52 GAO/NSIAD-98-56 Best Practices
Chapter 3 

Differences in Military and Commercial

Practices Reflect Different Environments

previously had to await the building and testing of physical articles.

Insistence on having near 100-percent release of drawings at the CDR drives

risks down as the production line is geared up. Achieving statistical

process control before production begins helps create certainty that

production costs, schedule, and quality will satisfy the customer. In short,

these practices are adopted because they work by helping to ensure

success in the commercial environment.

DOD Practices Reflect

the Need to Succeed

in Funding and

Managing the

Development Effort

DOD must make a defendable case before launching a program. The

program’s merits generally relate to providing a needed capability within

the limits of affordability. DOD typically defines and launches a program

years earlier in the process than a commercial product development, and

thus the case for the product is made when much less is known about

technology, cost, and schedule. In a very real sense, the point of sale

begins much earlier on a DOD program and continues throughout

development as the customer (DOD and the Congress) pays for the product

on an annual installment basis from program launch. Success, then, for

most of the product development cycle, is measured in terms of ability to

secure the next installment. Because this approval must be won every

year, it creates incentives to make the program’s case look attractive.

By the time production begins, the customer is deeply invested and

unlikely to walk away. As a result, success, in terms of program

continuance, is substantially ensured before end items are produced. For

these reasons, and because production is often many years from the

launch decision, it is difficult for production realities and concerns to

exert as much influence on a DOD product development. Instead, design

features and performance are more dominant. More unknowns are

accepted on a DOD program, and their attendant risks are often

understated. This combination, which can be devastating to a business

case, can work in a weapon development if it helps the program get

launched and survive. Practices, such as deferring prove-out of key

processes until production begins, are compatible with this definition of

success.

Early Launches and

Long-Term Programs Put

Focus on Technology

Development

For a major product development to be launched, DOD’s acquisition

guidance calls for a strong case to be made that includes a firm need,

superior product performance, affordable cost, and feasible technology. A

need can stem from several sources, including an increase in threat

capabilities, the obsolescence of an existing weapon system, or the

Page 53 GAO/NSIAD-98-56 Best Practices
Chapter 3 

Differences in Military and Commercial

Practices Reflect Different Environments

potential for a new or expanded capability made possible by technology

advances. Traditionally, needs for new weapons have been generated by

individual branches within each service. In recent years, DOD has been

experimenting with other vehicles for determining needs, such as

advanced concept technology demonstrators to evaluate the utility of

mature advanced technologies.1

 Once a need has been established, a

product development vying for launch faces intense competition for initial

funding.

A key distinction between DOD and commercial product developments is

the timing of the launch. The launch decision on a DOD product

development is often made years before that on a commercial product. In

DOD, this decision is made when the program is approved to enter the

program definition/risk reduction phase. The knowledge required to make

the business case to launch a commercial product development is

generally not available for a DOD program until well into the EMD phase. By

way of analogy, the F-22 fighter program, 11 years after launch and about

midway through the EMD phase, may now be at the point that it would be

ready for launch in the commercial environment. Thus, when a DOD

product development competes for launch funding, it is generally dealing

with far greater unknowns than its commercial counterpart. Figure 3.3

shows an F-22 fighter plane.

1

DOD defines advanced concept technology demonstrators as a means of demonstrating the use of

mature technology to address urgent military needs. The demonstrators are not acquisition programs,

but are designed to provide a residual, usable capability upon completion. If the user determines that

additional units are desired, the additional buys would constitute an acquisition program.

Page 54 GAO/NSIAD-98-56 Best Practices
Chapter 3 

Differences in Military and Commercial

Practices Reflect Different Environments

Figure 3.3: F-22 Fighter Plane

The F-22, launched earlier than commercial products, needed significant advances in propulsion,

low-observables, and avionics technologies to meet performance requirements.

Source: DOD.

Even though less information about a new DOD product development is

available at the time of launch, the competition for funding requires

detailed projections to be made from what information does exist.

Although DOD is attempting to ease the technical requirements of

programs, a new product development is encouraged to possess

performance features that distinguish it from other systems. For example,

the F-22 will be faster than the F-117 and stealthier than the F-15; the

Page 55 GAO/NSIAD-98-56 Best Practices
Chapter 3 

Differences in Military and Commercial

Practices Reflect Different Environments

RAH-66 Comanche helicopter will be faster, stealthier, and smaller than

the AH-64 Apache. Consequently, aspiring DOD programs have incentives to

include performance features and design characteristics that rely on

immature technologies.

Untempered by knowledge to the contrary, the risks associated with these

technologies are deemed acceptable. We have often reported on the

subsequent realization of such risks in the form of problems later in the

product development. In a 1993 study of seven weapon systems, Rand

found that, despite policies to the contrary, the technology underlying

several programs was clearly not fully developed and ultimately caused

substantial difficulties.2

 Production realities, critical to matching

technological capabilities with customer requirements on commercial

programs, are too far away from the DOD launch decision to have the same

curbing effect on technology decisions. Indeed, in support of the Defense

Manufacturing Council, a multi-discipline team from industry and

government pinpointed a major cause of acquisition problems in DOD as

the imbalance between product goals and the maturity of engineering and

manufacturing processes used to reach those goals.

Other pressures on DOD programs at launch make tough demands for

knowledge that does not yet exist. A product development deemed worthy

cannot be launched unless development and production funding is

available over the right time period. The product’s development and

production cost, as well as timing, must fall within available funding.

Because DOD relies largely on forecasts of cost, schedule, and performance

that are comparatively soft at this stage, success in funding competition

encourages the cost and schedule estimate to be squeezed into profiles of

available funding. Additional product requirements, such as high reliability

and maintainability, serve to make the fit even tighter.

Ultimately, the demands of successfully competing for launch funds make

for a much different business case on a DOD product development.

Compared with commercial programs, the DOD environment launches

product developments that embody more technical unknowns and less

knowledge about the performance and production risks they entail. These

unknowns place a much greater focus on demonstrating and proving out

technology during the remainder of product development than we found

on commercial programs. Equally important, the DOD program is launched

with customer funding. Even though the competition for funding will

continue throughout the program’s development, the point of sale begins

2

Barriers to Managing Risk in Large Scale Weapons System Development Programs, Rand, 1993.

Page 56 GAO/NSIAD-98-56 Best Practices
Chapter 3 

Differences in Military and Commercial

Practices Reflect Different Environments

with program launch. When a commercial product development is

launched, the customer’s willingness to buy is still undemonstrated and

thus remains a significant risk and a powerful incentive.

Incentives on DOD

Programs Encourage

Different Behaviors From

Managers

As a product development proceeds in DOD, its success is still measured in

terms of the funding it receives. Success translates into getting the funding

requested each year; failure can mean anything from a significant funding

cut to cancellation. In a sense, the day of reckoning for a DOD product

development comes every year with the budget process. The most

important issues facing the program at the time will dominate the

attention of program managers and decisionmakers. These issues tend to

be related to the successful demonstration of technology. In contrast, the

day of reckoning for commercial programs is later in the program cycle: at

the sale of the produced item. The business case with the production

conclusion sustains the commercial product development’s focus through

future decisions and problem solving. Again, shorter cycle times are key to

sustaining this focus.

Unlike commercial programs, DOD programs do not receive full corporate

support throughout development. Even though many individual programs

compete for funding at any given time, competition also exists at the

aggregate level, among the services for their portion of the budget. In

addition, DOD programs face scrutiny by service executives, the Office of

the Secretary of Defense, independent cost estimating and test agencies,

audit agencies, and several committees and subcommittees of the

Congress. Given this amount of competition and oversight, the detection

of a problem on an individual program makes that program vulnerable to

criticism and possible loss of funding support. Ironically, it is these same

pressures that encourage overreaching at the time of program launch. This

situation contrasts with that of the commercial product development,

which enjoys full corporate support once it is launched; in turn, the

program’s estimates are kept realistic by the knowledge that the program’s

success may determine the firm’s future.

The pressures and incentives in the DOD environment explain why the

behaviors of managers and other sponsors of product developments differ

from those in commercial programs. According to a 1994 study done for

the Under Secretary of Defense for Acquisition, government program

managers found their formal role of objective program management at

Page 57 GAO/NSIAD-98-56 Best Practices
Chapter 3 

Differences in Military and Commercial

Practices Reflect Different Environments

odds with their informal role of program advocates.3 According to the

study:

“A feeling of responsibility for program advocacy appears to be the primary factor causing

government managers to search aggressively and optimistically for good news relating to

their programs, and to avoid bad news, even when it means discrediting conventional

management tools that forecast significant negative deviations from plan.”

In this environment, risks in the form of ambitious technology

advancements and tight cost and schedule estimates are accepted as

necessary for a successful launch. Problems or indications that the

estimates are decaying do not help sustain the program in subsequent

years, and thus their admission is implicitly discouraged. An optimistic

production cost estimate makes it easier to launch a product development

and sustain annual approval; admission that costs are likely to be higher

could invite failure. There are few rewards for discovering and recognizing

potential problems early in the DOD product development. For commercial

product developments, an optimistic production cost estimate will mean

failure of sales or profit; admission of cost increases early invites

aggressive problem-solving behaviors to restore the business case. The

behavior of tolerating unknowns and not designating them the same risk

value as in the commercial environment is rational in the DOD environment

because there is little incentive to admit to high risks before it is

absolutely necessary, as long as the resulting estimates are accepted by

DOD and the Congress. In fact, admitting risk may doom the program.

Behaviors toward testing follow a similar logic. On commercial product

developments, much more is known about the product’s performance at

the beginning of development. Testing is used to confirm knowledge and

identify weaknesses or limits in the product. It is consistent with a firm’s

anxiety to eliminate unknowns to preclude failure in production. DOD

product developments are much more dependent on testing to discover

technical performance characteristics and answer the question of whether

the product will work. DOD tests serve more than the purpose of

discovering or confirming performance characteristics—they are

examinations on which the program must get good grades or face failure

in the form of withdrawal of support. Good test results can help a

program, whereas negative test results are equated with failure.

Unknowns, then, present a safer course of action; if testing does not occur

until late in the product development, forecasts of product performance

will serve as the only information available.

3

Critical Issues in the Defense Acquisition Culture, Defense Systems Management College—Executive

Institute, December 1994.

Page 58 GAO/NSIAD-98-56 Best Practices
Chapter 3 

Differences in Military and Commercial

Practices Reflect Different Environments

Cycle time is an important determinant of what pressures the environment

brings to bear on a product development. In a DOD product development,

cycle times can be 10 to 15 years, whereas the tenure of a program

manager is more likely to be 2 to 4 years. One senior commercial program

manager informed us that it is unreasonable to expect that a program

manager can truly focus on anything that is more than 3 years off; events

beyond that timeframe are not powerful motivators. Thus, it is hard for

production success to present the same reality for the first few managers

of a DOD program and the decisions they are involved in. To some extent,

DOD may attempt to counteract the disparity between the length of a

program manager’s tenure and that of the program being managed through

formal reviews. Production readiness reviews are an example; they do not

have a commercial counterpart because readiness for production is

intrinsic to the commercial product development. The production

readiness review does not provide the same incentives for success

because it is also prey to the pressure to accept unknowns and assess

them as tolerable risks. DOD’s practices, which allow key production

knowledge to be gained concurrent with production, are a logical

consequence of the DOD environment. If problems arise in production,

there is not nearly the risk of failure that a commercial product faces; by

the time a weapon enters production, the point of sale to the customer has

already occurred.

We did not attempt to make a formal comparison between the capabilities

of DOD and commercial program managers. We believe that commercial

program managers have one advantage in that they are likely to have more

experience with repeated product developments than a DOD program

manager. Aside from that advantage, we did not observe that commercial

managers were somehow better or more ethical than their DOD

counterparts. As we previously reported,4 DOD program managers and

other sponsors do not act irrationally or with bad intentions. Rather, they

see the acquisition of the weapons under their purview as aligned with

national interests. They do what they believe is right, given the pressures

they face. The difference is that the definition of program success

determines what is right, and success in the DOD environment is different

from success in the commercial environment.

4

Weapons Acquisition: A Rare Opportunity for Lasting Change (GAO/NSIAD-93-15, Dec. 1992).

Page 59 GAO/NSIAD-98-56 Best Practices
Chapter 4 

Creating the Right Environment Is Key to

the Success of Initiatives to Improve

Weapon Acquisitions

DOD has embarked on several initiatives that draw lessons from

commercial practices, such as cost as an independent variable (CAIV),

integrated product teams (IPT), use of past performance data, and

performance specifications. DOD has also recently set up a funding reserve

to offset unexpected cost growth to mitigate the effect of unknowns on

acquisition programs. These initiatives could have a positive effect on the

transition of weapons into production if the environment for launching

programs and appraising risks can be changed to provide the right

incentives. Studies sponsored by DOD and the defense industry call for

changes that could help shape such an environment. A 1996 study by the

Defense Science Board Task Force on Defense Acquisition Reform calls

for replacing the DOD acquisition process with a process that combines the

best elements of practices from commercial and defense-unique activities.

A major tenet of the recommended process is more emphasis on

incremental technology advancement, coupled with much shorter product

development cycle times. These themes are also echoed in a study by a

National Center for Advanced Technologies task force comprised of

defense firms.

Examples from past acquisitions serve as reminders that changing the

mechanics of a weapon’s development, without changing aspects of its

environment that determine its incentives, may not produce desired

results. DOD’s guidance for preparing weapons for a successful transition

to production, some of which is 10 years old, already has much in common

with commercial best practices. Thus, the challenge for DOD and

congressional decisionmakers may not lie so much in the “how to” aspects

of product development as in creating the incentives—or the reasons why

best practices will work for program managers.

DOD’s Risk Reduction

Policy Is Consistent

With Commercial

Practices

DOD 4245.7-M, Transition From Development to Production, a risk

reduction policy manual written in 1985, provides metrics and tools for

minimizing design and production risk during a product’s development. In

addition, DOD’s new procedures for major defense acquisition programs

contained in the new 5000 series of acquisition policies emphasize best

practices in assessing and mitigating risk during development. Figure 4.1

shows DOD’s main policy guidance on preparing weapons for production.

Page 60 GAO/NSIAD-98-56 Best Practices
Chapter 4 

Creating the Right Environment Is Key to

the Success of Initiatives to Improve

Weapon Acquisitions

Figure 4.1: DOD Guidance on Preparing Weapons for Production

DOD has provided ample guidance to program managers on how to properly manage weapon

system development to successful production.

DOD 4245.7-M contains templates that provide guidance for when and how

to perform trade studies, design reviews, and producibility reviews. It

contains criteria for trade studies that require an evaluation of new

Page 61 GAO/NSIAD-98-56 Best Practices
Chapter 4 

Creating the Right Environment Is Key to

the Success of Initiatives to Improve

Weapon Acquisitions

technology before selecting it for a product’s development and suggests

that cost, producibility, and quality should be considered equally with

performance. The templates also suggest that a CDR should be held when

95 percent of the product’s design is complete in terms of engineering

plans and drawings and that the design should be frozen and released

directly after CDR. The templates also state that CDR results and critical

drawing release schedules and approvals are essential data sources for

design release. Finally, the templates suggest that (1) the effect of the

design on current manufacturing processes should be measured during the

design process, (2) any new manufacturing processes should be “proofed”

during development to ensure producibility, (3) proofing simulates actual

production conditions and environments, and (4) SPC data are considered

essential program data sources for qualifying the manufacturing processes

during product development.

DOD revised its 5000 series procurement policies in March 1996 with the

intent of defining an acquisition environment that makes DOD a smart and

responsive buyer of the best goods and services. Several themes described

in the new 5000 series documents resonate with commercial practices:

• The commercial marketplace must be researched to determine its

potential to meet system performance requirements and results

documented in the initial operational requirements document.

• The acquisition process must consider both performance requirements

and fiscal constraints as embodied in CAIV.

• Acquisition of commercial processes and practices provides rapid and

affordable application of new technologies to meet DOD mission needs.

• Future acquisitions must take into account customary commercial best

practices in developing acquisition strategies and contracting

arrangements.

• A streamlined management structure and event-driven management

process at DOD would emphasize risk management and affordability and

would explicitly link milestone decisions to demonstrated

accomplishments.

DOD’s 5000 series documents emphasize the interrelationship between

establishing requirements, managing the development process, and

making funding decisions with the objective of translating users’ needs

into products with affordability as a key discriminator. The policy also

emphasizes the consideration of producibility early in a product’s

development. It states that producibility is key to managing risk and that

existing manufacturing processes should be capitalized on when possible.

Page 62 GAO/NSIAD-98-56 Best Practices
Chapter 4 

Creating the Right Environment Is Key to

the Success of Initiatives to Improve

Weapon Acquisitions

It also states that production should not be approved until the design has

been stabilized, manufacturing processes have been proven, and facilities

and equipment are in place. DOD has emphasized that programs should

meet users’ needs with mature technology, when possible, by using

demonstrations of technology and incremental or evolutionary product

developments, thereby taking advantage of existing information about cost

and performance.

DOD Initiatives Adapt

Some Commercial

Practices

DOD’s more recent initiatives are intended to reduce the cost and duration

of major weapon system acquisitions. In establishing the framework in

which cost is considered an independent variable, DOD is attempting to

make cost a prime reason for trading off performance requirements. Once

cost and performance tradeoffs have been made, CAIV establishes cost as

the overriding constraint for obtaining the needed military capability from

the new system. For CAIV to work properly, program managers will need

sufficient knowledge of technological capability and associated program

development and production costs, along with a sound mechanism and

forum for making tradeoffs. Whether tradeoffs are made will depend on

the incentives in place to make them. DOD points to the JDAM as an example

of how effective CAIV can be because use of this tradeoff technique

reduced expected unit production prices from $40,000 to about $14,000

per unit. Figure 4.2 shows the JDAM.

As the program proceeds through development, the framework for making

decisions is to be similar to the zero sum approach an auto manufacturer

follows after establishing a price target for a new vehicle. Today, however,

the career incentives for program managers in DOD are not consistent with

those of the auto manufacturers.

Page 63 GAO/NSIAD-98-56 Best Practices
Chapter 4 

Creating the Right Environment Is Key to

the Success of Initiatives to Improve

Weapon Acquisitions

Figure 4.2: Joint Direct Attack Munitions System

Use of cost as an independent variable and integrated product teams enabled the projected unit

price of the JDAM to be reduced from $40,000 to $14,000.

IPTs are a means to integrate the development of a product and its

manufacturing processes by using multidisciplinary teams that represent a

variety of product functions, such as engineering, manufacturing,

purchasing, and accounting. Traditionally, these functions have been

separate organizations that tended to get involved sequentially in the

product development process. Through IPTs, the functions are brought

together to make tradeoffs that will optimize the design, manufacturing,

and supportability processes for a weapon system. They are important to

making techniques such as CAIV work. IPTs have analogies in Boeing’s

design/build teams and Chrysler’s platform teams. Many defense

contractors have already adopted the IPT approach. Several factors affect

Page 64 GAO/NSIAD-98-56 Best Practices
Chapter 4 

Creating the Right Environment Is Key to

the Success of Initiatives to Improve

Weapon Acquisitions

the success of IPTs, including reasonableness of the program goals, the

knowledge the members bring to the team, the authority the teams have to

make tradeoffs to achieve the goals, and the right disciplines represented

on the teams.

DOD has an initiative to use information on the past performance of a

contractor to ensure that DOD has access to a globally competitive

industrial base capable of supplying the best value goods and services. DOD

believes that past contractor performance information is an important

strategic tool that when properly used, will allow DOD to evaluate the risk

of using poor or nonperforming contractors as well as the potential for

using excellent contractors. This information will allow DOD to make better

decisions during source selection. In the commercial sector, the use of

past performance information for suppliers was generally found to be

integral to improving product developments.

DOD has also made the decision to use performance specifications on

weapon system developments. Previously, many military specifications

told defense contractors specifically how they were to accomplish certain

tasks. Performance specifications tell the contractor only what

performance is desired of the end product and not how to get that

performance. Again, there is a commercial analogy in the sense that

customers are generally not concerned with how a product does

something, such as get better fuel economy, as long as performance is as

advertised.

DOD is tracking the effects of these acquisition reform initiatives on seven

programs, designated as Defense Acquisition Pilot Programs. These

programs were afforded early statutory and regulatory relief under the

provisions of the Federal Acquisition Streamlining Act of 1994 to set the

example for acquisition reform. JDAM is one of the pilot programs.

According to DOD, the successful application of commercial practices

enabled these programs to demonstrate significant improvements. DOD

projects that acquisition programs that have benefitted from acquisition

reform could reduce cycle time by 25 percent. Some of DOD’s pilot

programs have significantly reduced the amount of military specifications

and standards. JDAM has eliminated all 87 of the military specifications and

standards that were in its baseline contract in favor of commercial

practices. A 1997 DOD report on the pilot programs credits the use of IPTs

for reducing program office size and significantly decreasing government

contract administrative hours. The same report states that the

Page 65 GAO/NSIAD-98-56 Best Practices
Chapter 4 

Creating the Right Environment Is Key to

the Success of Initiatives to Improve

Weapon Acquisitions

improvements indicated by the pilot programs appear to be applicable

across DOD.

DOD has recently set up a reserve of funds to be used to mitigate the effect

of unforeseen technical problems that might threaten to upset an

acquisition program’s schedule. This initiative emerged from DOD’s

Quadrennial Defense Review, which stated that

“complex, technologically advanced programs all bear some risk of costing more than

planned. When unforeseeable growth in costs occurs, offsets from other programs must be

found, which in turn disrupts the overall modernization program. Our programming

process must provide sufficient flexibility in the form of program reserves to address this

risk.”

DOD plans to begin accumulating the risk reduction fund in fiscal year 2000

and expects it to grow to about $1 billion by fiscal year 2003. It will be

accumulated through contributions from the Office of the Secretary of

Defense and the services.

We have not evaluated this new funding mechanism or the way DOD plans

to implement it. Nonetheless, DOD’s use of the reserve has the potential for

communicating to program managers which practices will be encouraged

and which ones will not. For example, if the fund is used primarily to pay

for problems that are revealed in late development or early production, the

fund could reinforce existing incentives for not dealing with risks until

they become problems. Conversely, if the fund is used to resolve risks

early and preclude problems, it could encourage risks to be revealed

earlier in programs.

Experience on Past

Programs

Underscores the Need

to Match Technique

With the Environment

A discussion of lessons learned or difficulties encountered on past weapon

acquisitions must recognize that the DOD acquisition process has produced

the best weaponry in the world. Nonetheless, the experiences of some

weapon systems serve to point out the importance of making the

environmental changes necessary for good practices to work. These

techniques then will have the chance to offer better outcomes consistently

in terms of cost, schedule, and performance.

In the 1980s, the Army attempted to acquire the Sergeant York Air Defense

Gun using a strategy that included several features similar to those in the

Defense Science Board’s recommended process. These features included a

competition between two contractors to use components based on

Page 66 GAO/NSIAD-98-56 Best Practices
Chapter 4 

Creating the Right Environment Is Key to

the Success of Initiatives to Improve

Weapon Acquisitions

previously demonstrated technology and integrate them into a weapon

system, coupled with the freedom to trade off flexible performance

requirements to save costs and shorten cycle time. The resulting system

failed in operational tests after it had entered the 4th year of production

and was subsequently canceled. The risks associated with integrating the

technologies had been underestimated, and the discovery process was

protracted into production. The techniques to foster tradeoffs, reduce

risks, and give the competing contractors the freedom to design the

system did not, by themselves, change the Sergeant York’s environment

into one that encouraged the early discovery of problems.

Fluctuations in funding cause perturbations in weapon programs and are a

complication that poses less of a problem for the commercial firms we

contacted. However, stable funding, by itself, will not ensure program

success. In the 1980s, baselining a program was a technique aimed at

enhancing program stability, whereby a program office “contracted” with

top management to develop a system that met basic performance, cost,

and schedule requirements in exchange for stable funding and minimal

interference. The 1986 Packard Commission cited the B-1B bomber as an

example of a baselined system. Despite enjoying stable funding and

congressional support, the B-1B experienced significant performance

shortfalls and continues to require substantial additional funding to

correct these and other problems. Similarly, the F-117A program

experienced cost increases, schedule delays, and substantial modifications

despite stable funding, continuous congressional support, and streamlined

management. In view of the many pressures that characterized the

acquisition environment for these programs, stable funding and support

did not prevent other problems, such as those stemming from ambitious

performance requirements and concurrent schedules.

When a stated program goal is to trade performance for cost and

operational support reductions, the acquisition process must encourage

those tradeoffs. This was not the case for the Army’s Comanche

helicopter. This program was initiated in the early 1980s by senior Army

management as a family of lightweight, multipurpose helicopters whose

justification centered on practicality rather than the threat. The program

was expected to replace a fleet of Vietnam-era helicopters with new

helicopters that would be up to 50 percent less expensive to operate and

support. Within these economical confines, the new helicopters were to

offer as good a technical performance as was possible. However, specific

requirements were subsequently developed through a

performance-oriented requirements process. The program emerged as a

Page 67 GAO/NSIAD-98-56 Best Practices
Chapter 4 

Creating the Right Environment Is Key to

the Success of Initiatives to Improve

Weapon Acquisitions

threat-based program to yield the next-generation, high-performance

helicopter at a cost significantly higher than that of the Apache, the Army’s

most advanced and costly helicopter.

The lesson to be drawn from these experiences is not that a weapon’s

transition to production cannot be improved through the use of better

practices. Rather, these experiences show that attempts to improve

individual acquisitions require more than a new technique or approach;

they also require the knowledge needed to make key tradeoffs and risk

assessments, as well as the recognition by decisionmakers that such

information provided early is critical to program success. We believe that

the JDAM and AIM-9X have promise in this regard because they have utilized

mostly existing technology in setting requirements and, when they did not

use existing technology, have reduced risk through other means. JDAM

represents a new munition built with mostly commercial parts, and the

AIM-9X is an upgrade to an existing missile system that largely relies on its

predecessor’s technology. Moreover, the launch decisions for the JDAM and

the AIM-9X missile have encouraged performance tradeoffs to save time

and money. Figure 4.3 shows the AIM-9X missile.

Page 68 GAO/NSIAD-98-56 Best Practices
Chapter 4 

Creating the Right Environment Is Key to

the Success of Initiatives to Improve

Weapon Acquisitions

Figure 4.3: AIM-9X Missile

Tradeoffs early in the program may enable requirements for the AIM-9X to be met with mature

technologies.

Source: DOD.

Page 69 GAO/NSIAD-98-56 Best Practices
Chapter 4 

Creating the Right Environment Is Key to

the Success of Initiatives to Improve

Weapon Acquisitions

The challenge for these programs will be to maintain their integrity during

subsequent decisions as knowledge—and problems—are discovered. It

also remains to be seen to what extent large programs, such as new tank

or aircraft development programs, can apply these techniques in the

perhaps higher pressure environment in which they will be managed. DOD’s

1997 report on the Defense Acquisition Pilot Programs notes that, to fully

achieve the benefits of acquisition reform, a complete cultural change

must be affected. The report notes that despite significant top-level

management emphasis on changing the acquisition process, cultural

resistance to various reform initiatives has been encountered on both

industry and government teams.

Recent Studies

Promote Shorter

Cycle Times and

Other Environmental

Changes

In May 1996, the Defense Science Board completed a study that concluded

“force modernization at low cost and with short cycles—and with ability

to draw on world-class commercial firms—requires a new weapons R&D

[research and development] process.” Some of the features of the process

recommended by the study directly address key environmental differences

we noted between commercial and DOD product developments.

Specifically, the study recommended that DOD’s process

• aggressively pursue high-risk technology before inclusion in a weapon

research and development program;

• employ product solutions chosen jointly by the Under Secretary of

Defense for Acquisition and Technology and the Vice Chairman of the

Joint Chiefs of Staff;

• maintain competing alternatives for solutions throughout product design;

• make use of already developed next-generation technology, and

concentrate on evaluations of existing subsystems as the building blocks

for the concepts selected to meet the need;

• use flexible performance requirements with fixed-price and firm schedule

requirements; and

• rely on competitive forces and price-based contracting versus regulations

and cost-based contracting.

In the opinion of the Defense Science Board, adopting such an acquisition

process would supply effective hardware in small quantities, producible

and supportable at affordable cost, with cycle times reduced by one-half.

In April 1996, the National Center for Advanced Technologies also

proposed a change in the DOD weapons acquisition process to reduce cycle

time by drawing on lessons learned from successful military and

Page 70 GAO/NSIAD-98-56 Best Practices
Chapter 4 

Creating the Right Environment Is Key to

the Success of Initiatives to Improve

Weapon Acquisitions

commercial programs. The center notes that concepts such as integrated

product/process development and CAIV are constructive but that previous

good ideas did not succeed. Its proposal for change calls for a new culture

that relies on an affordable, incremental approach that could reduce

product development cycle times by 3 to 5 years. The new culture features

• an incremental approach to performance, with a threshold or minimum

performance for the initial battle group with incremental upgrades and

• requirements that would be managed through cost tradeoffs to keep

performance and cost in balance, avoid grand designs, and mitigate risk.

The December 1994 study by the Defense Systems Management College,

Critical Issues in the Defense Acquisition Culture, made several

recommendations to create an environment for weapon systems

development that encourages realism in reporting program status

information. Perhaps the most significant recommendation is that the

individual military services transfer control of their acquisition

organizations and people to the Under Secretary of Defense for

Acquisition and Technology. The study noted that, by doing so, the Under

Secretary would then be empowered to reward candor and realism in

reporting through the use of assignments, transfers, and promotions.

Page 71 GAO/NSIAD-98-56 Best Practices
Chapter 5 

Conclusions and Recommendations

Conclusions Product development in commercial ventures is a clearly defined

undertaking for which firms insist on high levels of knowledge before

starting. Once underway, these firms demand—and get—specific

knowledge about a new product before manufacturing begins. The process

of discovery—the accumulation of knowledge and the elimination of risks

or unknowns—is completed for the best commercial programs well before

production units are made. In our analysis, we characterized this

knowledge in terms of three points or junctures: the match between

requirements and technology, the ability of the design to perform as

expected, and the ability to produce the product on time and at the right

price. Not having this knowledge when demanded constitutes risks that

the firms find unacceptable. Immature or undeveloped technology cannot

meet these demands. It is essentially kept out of commercial product

development programs because of the risks it poses to a product’s price

and schedule. Increases in price and schedule devalue a commercial

product’s worth. Immature technology is managed separately until it can

meet the demands for product development.

DOD does not insist on the same level of knowledge when it decides to

begin a weapon development program. In the DOD acquisition process, a

clear delineation is not made between technology development and

product development. In fact, programs are launched in the technology

development phase, and the pursuit of new technology—the discovery

process—continues through EMD and even into production. Not having the

same level of knowledge as commercial firms explains much of the

turbulence in DOD program outcomes as the transition to production is

made. It is a predictable consequence that can be forecast early by the use

of knowledge points or other metrics. DOD has had guidance in place that

provides policies and metrics that have much in common with what

commercial firms demand in terms of knowledge. However, the way the

tools for implementing such guidance are used, such as CDRs and

production readiness reviews, understates the risks present. Ironically,

commercial firms that have weeded out unknowns and minimized the

discovery process still identified some risks as high that needed to be

resolved in their product developments. DOD programs, which accepted

more unknowns and technical advances, did not assess risks as high.

Environmental differences underlie the differences in the product

development practices we observed between the DOD programs and

commercial firms. At the core of these differences are the definition of

success and the time span of programs. In commercial programs, success

is defined by the sale of production units to the customer; product

Page 72 GAO/NSIAD-98-56 Best Practices
Chapter 5 

Conclusions and Recommendations

development activities exist only for production. Until the sale is made,

the investment money of the product developer is at risk. Product

development cycle times are short enough that production realities are a

major factor in assessing and eliminating risk in the commercial sector.

Behaviors, such as candor about risk and an intolerance for unknowns,

and practices, such as ensuring design performance and demonstration of

all critical production processes by CDR, are encouraged because they

facilitate success.

In DOD, programs are launched with immature technology, product

development cycle times are much longer, and production realities figure

less prominently in early tradeoff decisions. Success is defined more as the

ability to secure annual funding installments from the customer through

the budgetary process to sustain the development effort. Thus, the point of

sale essentially occurs when the program is launched. By the time

production begins, the sale of the product has, to a large extent, already

occurred. DOD’s acceptance of unknowns and practices that understate

risk are consistent with program success as defined in its environment.

Commercial practices for gaining knowledge and assessing risks can help

produce better outcomes on DOD acquisitions. For such practices to work,

however, the knowledge they produce must help a DOD program succeed

in its environment. Thus, the DOD environment must become conducive to

such practices. At least two factors are critical to fostering such an

environment. First, program launch decisions must be relieved of the need

to overpromise on performance and resource estimates. The pressure to

amass broad support at launch creates incentives for new programs to

embrace far more technology development than commercial programs.

The objectives of technology development, as well as what is demanded of

knowledge and estimates, differ from those of product development.

Clearly, DOD has to develop technology, particularly the technology that is

unique to military applications. However, by separating technology

development from weapon programs, DOD could insist on higher standards

for knowledge on its programs and get better results when those programs

transition to production.

Second, once a program is underway, it must become acceptable to

identify unknowns as high risks so that they can be aggressively worked

on earlier in development. Commercial firms insist on knowledge

measured against a criterion for assessing risk. Fixed prices and firm

schedules help form the basis for such assessments. Firms then make

decisions to preserve the business case by eliminating risks. The result is

Page 73 GAO/NSIAD-98-56 Best Practices
Chapter 5 

Conclusions and Recommendations

discipline provided from within. In contrast, the identification of high risk

is more likely to invite damage to the funding case on DOD programs.

Unknowns are more likely to be accepted, and knowledge is more likely to

be traded to preserve resource estimates. Releasing drawings behind

schedule indicated the attainment of less knowledge about design and

manufacturing on the F-22 program, but production readiness reviews did

not identify any areas of high risk. It took an external team—discipline

from outside—to assess production risks as having significant cost

consequences.

Decisions made by the Office of the Secretary of Defense and the services

to advance programs with significant unknowns or advance programs with

known high risks but without sufficient resources, send signals to

acquisition managers that current practices work and are acceptable.

These decisions define what success means in DOD and what practices

contribute to success. The newly created risk reserve could become one

vehicle for signaling what behaviors and practices are encouraged on

individual programs.

Recommendations To close the gap between policy and practice, we recommend that the

Secretary of Defense take steps to ensure that sound standards for the

timing and quality of production-related knowledge are applied to

individual weapon system programs and used as a basis for assessing

production risks and for making tradeoffs. These standards, which can

already be found to some extent in existing DOD guidance, should draw

from commercial practices and could include the release of engineering

drawings, identification of key production processes, demonstration of

risky or new production processes, and achievement of statistical process

control. Such standards could enable manufacturing representatives on

IPTs to flag something as high risk if it would delay drawing release and the

achievement of statistical process control beyond the standard of

acceptability. Identifying the impact of such deficits in production

knowledge could help program managers to say “no” to proposals to

accept unknowns and could force tradeoffs in the design.

To make the environment for DOD product developments more conducive

to the techniques used by commercial firms, we also recommend the

Secretary of Defense redefine the point for launching programs as the

point at which technology development ends and product development

begins. This recommendation is made without prejudice toward the

necessity of technology development but rather with the intent that

Page 74 GAO/NSIAD-98-56 Best Practices
Chapter 5 

Conclusions and Recommendations

programs could be better managed if such development was conducted

outside their purview. Thus, concomitant with defining the launch point

later in the acquisition cycle must be the willingness of decisionmakers in

DOD and the Congress to support research and development efforts that

will advance technology and establish the basis for determining which

technologies and subsystems have the mettle to meet the performance,

production, and precision estimating demands of product development. If

extenuating circumstances necessitate including technology development

in a program, this should be recognized as a departure from sound

practices, accorded a high risk, and funded and paced accordingly.

Finally, we recommend that on individual program decisions, the

Secretary of Defense or his designee send the signals that create

incentives for acquisition managers to identify unknowns and ameliorate

their risks early in development. Incentives could take the form of a

decision to fully fund one program’s efforts to mitigate a high risk

identified early or requiring another program in which risks are revealed

late to absorb the associated financial consequences.

Matters for

Congressional

Consideration

Because of its critical role in creating the environment for what

constitutes program success and which practices will work, the Congress

may wish to consider supporting the Secretary of Defense’s efforts to

create such an environment through changes to the acquisition process

that provide program managers clear incentives for gaining sufficient

knowledge at key points in weapon acquisition programs. The best

commercial practices described in this report suggest what may constitute

“sufficient” levels of knowledge. If DOD takes steps to manage technology

development efforts outside the bounds of individual weapon system

programs, the Congress may wish to consider providing the funds needed

for such efforts. The Congress could also help create the right incentives

on individual programs by favorably considering DOD funding requests to

mitigate high risks early in a program and cautiously considering late

requests for funds to resolve problems that should have been addressed

earlier.

Agency Comments

and Our Evaluation

DOD concurred with a draft of this report and all of its recommendations

(see app. I). DOD noted that, although it accepts technical risks to provide

superior weapons that often differ from risks in commercial industry, it

agreed that changes were needed in its environment to adopt practices

that industry has used to become leaner and more flexible. DOD stated that

Page 75 GAO/NSIAD-98-56 Best Practices
Chapter 5 

Conclusions and Recommendations

it was pursuing such practices through the Defense Reform Initiative, the

National Performance Review Reinvention Center, Management Reform

Memorandums, and numerous acquisition reform initiatives.

DOD also highlighted actions that it believes are consistent with our

recommendations. Regarding redefining the point for launching programs,

DOD noted that its EMD phase was equivalent to the commercial product

launch and that it was taking steps to ensure that technology development

was separated from this phase. DOD stated that, to create incentives for

acquisition managers to identify unknowns and risks early in development,

it had established goals for reducing program cycle time and obviating

cost growth. DOD said it was striving to identify metrics to be used in

assessing program risks and cited several existing sources for such

metrics. In response to the matter that the Congress favorably consider

DOD proposals for fully funding technology development efforts, DOD

observed that full funding can, in some instances, reduce the flexibility

needed to manage these risky efforts. DOD also provided technical

comments, which we incorporated where appropriate.

As DOD takes actions to improve weapon system outcomes by better

identifying and eliminating program risks early, we underscore the need

for an environment that creates incentives for such actions to succeed. It

was the right environment that gave rise to the excellent practices we

found in leading commercial firms. Given the competitive pressures new

programs face, a weapon system program that is started during the

technology development phase (Program Definition and Risk Reduction)

is encouraged to accept significant technological risks, which are not

necessarily reflected in cost and schedule projections. When that program

reaches the EMD point, it is difficult to back off of the original projections

and put the program on a footing in which requirements, available

technology, funding, and time are matched to finish development and start

production. Yet, it is from that point on, despite the defense-unique

technological risks previously accepted, that a weapon system program

should have much in common with a commercial program, including the

standards for knowledge that are demanded.

Likewise, once a program is underway, perhaps the clearest way of

encouraging the right behaviors—specifically, program managers’ and

other decisionmakers’ willingness to identify and deal with risks early—is

through signals sent by individual decisions made on individual programs.

Absent such incentives, mechanisms that already exist for identifying and

reporting risks have not worked. While we support the search for

Page 76 GAO/NSIAD-98-56 Best Practices
Chapter 5 

Conclusions and Recommendations

improved metrics for production-related knowledge, it is the effective

application of good metrics that is more pressing. Such metrics should be

applied to programs at key junctures in a way that will enforce standards

for knowledge like those we found at commercial companies. We did not

intend that providing full funding for technology development efforts

should impede the flexibility needed to manage such efforts. We have

clarified the wording on that matter.

Page 77 GAO/NSIAD-98-56 Best Practices
Appendix I 

Comments From the Department of Defense

Page 78 GAO/NSIAD-98-56 Best Practices
Appendix I 

Comments From the Department of Defense

Now on pp. 74-75.

Now on pp. 74-75.

Now on pp. 74-75.

Page 79 GAO/NSIAD-98-56 Best Practices
Appendix I 

Comments From the Department of Defense

Now on pp. 74-75.

Now on pp. 74-75.

Now on pp. 74-75.

Page 80 GAO/NSIAD-98-56 Best Practices
Appendix II 

Major Contributors to This Report

National Security and

International Affairs

Division, Washington,

D.C.

Louis J. Rodrigues

Paul L. Francis

Gordon W. Lusby

Maria J. Santos

Chicago Field Office Michael J. Sullivan

Marvin E. Bonner

Office of the General

Counsel

John A. Carter

(707188) Page 81 GAO/NSIAD-98-56 Best Practices
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address

are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (202) 512-6000 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

For information on how to access GAO reports on the INTERNET,

send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. G100