Testimony 

Before the Subcommittee on National 

Security, Emerging Threats, and 

International Relations, Government Reform 

Committee, House of Representatives

United States General Accounting Office 

GAO

For Release on Delivery 

Expected at 10:00 a.m. EDT 

Friday, April 11, 2003 BEST PRACTICES 

Better Acquisition 

Outcomes Are Possible If 

DOD Can Apply Lessons 

from F/A-22 Program 

Statement of David M. Walker, 

Comptroller General of the United States 

GAO-03-645T 
GAO’s reviews of commercial best practices have identified key enablers to 

the success of product development programs and focused on how DOD can 

better leverage its investments by shortening the time it takes to field new 

capabilities at a more predictable cost and schedule. First, commercial firms 

use an approach that evolves a product to its ultimate capabilities on the 

basis of mature technologies and available resources. This approach allows 

only the product features and capabilities achievable with available 

resources in the initial development. Further product enhancements are 

planned for subsequent development efforts when technologies are proven 

to be mature and other resources are available. Second, commercial firms 

ensure that a high level of knowledge exists at key junctures during a 

product’s development. The knowledge-based process includes three points:

• Before a program is launched, successful programs match customer 

needs and available resources—technology, engineering knowledge, 

time, and funding. 

• About midway through development, the ability of the product’s 

design is demonstrated to be stable and meet performance 

requirements. 

• Before production begins, programs must show that a product can 

be manufactured within cost, schedule, and quality targets. 

In contrast, the F/A-22 program illustrates what can happen when a 

major acquisition program is not guided by the principles of evolutionary, 

knowledge-based acquisition. When the program was started, several 

key technologies were not mature. Program managers proceeded 

through development without the requisite knowledge to effectively 

manage program risk and, at the start of production, key manufacturing 

processes were not under control. The F/A-22 program has undergone 

significant cost increases. Instead of fielding early capabilities to the war 

fighter, the development cycle has extended to 19 years, so far, and 

original quantities have been significantly reduced, raising concerns 

about the capability the program will eventually deliver. 

DOD recognizes the need to get better weapon system outcomes, and its 

newest acquisition policy emphasizes the use of evolutionary, 

knowledge-based acquisition concepts proven to produce better 

outcomes in developing new products. However, policy changes alone 

are not enough. Leadership commitment and attention to putting the 

policy into practice for individual programs is needed to avoid the 

problems of the past. DOD will have many opportunities to do so over 

the next several years with its force modernization investments. 

Over the next 5 years, DOD’s 

overall investments are expected to 

average $150 billion a year to 

modernize and transition our 

forces. In addition, DOD must 

modernize its forces amid 

competing demands for federal 

funds, such as health care and 

homeland security. Therefore, it is 

critical that DOD manage its 

acquisitions in the most cost 

efficient and effective manner 

possible. 

DOD’s newest acquisition policy 

emphasizes the use of evolutionary, 

knowledge-based concepts that 

have proven to produce more 

effective and efficient weapon 

systems outcomes. However, most 

DOD programs currently do not 

employ these practices and, as a 

result, experience cost increases, 

schedule delays, and poor product 

quality and reliability. 

This testimony compares the best 

practices for developing new 

products with the experiences of 

the F/A-22 program. 

GAO is not making 

recommendations in this 

testimony. However, in a number 

of prior reports, GAO has 

recommended that DOD adopt 

policies with metrics for 

technology, design, and 

manufacturing maturity to support 

knowledge-based decision making. 

These policies should apply when 

making decisions on individual 

weapons programs. 

www.gao.gov/cgi-bin/getrpt?GAO-03-645T.

To view the full report, including the scope 

and methodology, click on the link above. 

For more information, contact Katherine 

Schinasi at (202) 512-4841 or 

schinasik@gao.gov. 

Highlights of GAO-03-645T, a testimony 

before the Subcommittee on National 

Security, Emerging Threats, and 

International Relations, Government 

Reform Committee, House of 

Representatives 

April 2003

BEST PRACTICES 

Better Acquisition Outcomes Are 

Possible If DOD Can Apply Lessons From 

F/A-22 Program 
Page 1 GAO-03-645T Best Practices 

Mr. Chairman and Members of the Subcommittee: 

I am pleased to be here today to participate in the Subcommittee’s hearing 

on how the Department of Defense (DOD) can—and must—get better 

outcomes from its weapon system investments. DOD is on the threshold of 

several major investments in acquisition programs that are likely to 

dominate budget and doctrinal debates well into the next decade. These 

programs include such systems as the Missile Defense Agency’s suite of 

land, sea, air, and space defense systems; the Army’s Future Combat 

Systems; and the Air Force’s and Navy’s Joint Strike Fighter. Over the next 

5 years, DOD’s overall investments are expected to average $150 billion a 

year as DOD works to keep legacy systems as well as modernize and 

transform our national defense capabilities for the future. Therefore, to 

meet these challenges, it is essential that sound foundations for these and 

other weapon system investments be laid now so that the resulting 

programs can be executed within estimates of available resources. 

Any discussion of improvements to DOD’s modernization efforts must be 

set in the context of overall expected budget availability. There are 

important competing priorities. Health care costs are growing at doubledigit rates, and spending on homeland security will likely grow as we seek 

to defeat terrorism worldwide. We face an oncoming demographic tidal 

wave, and by 2035 the number of people who are 65 or older will have 

doubled, creating much larger demands on the federal budget. The 

demand of funding for entitlement programs continues to grow, creating 

increasing pressures on discretionary funding for other federal priorities 

like education and defense. Therefore, it is critical that DOD manage its 

acquisitions in the most cost efficient and effective manner possible. 

My testimony today is about improving the outcomes of major weapon 

system acquisitions by using best practices to capture and use the right 

product knowledge at the right time for better decision making during 

product development. As per your request, I will compare acquisition 

practices and decisions made for the F/A-22 with these best practices for 

developing new products. The divergence between F/A-22 experiences and 

best product development practices, we believe, largely explains why the 

F/A-22 has been in development for over 16 years and its cost has grown 

substantially. It is also a primary contributor to other performance issues 

that are currently faced by the program. My testimony will also include 

observations on what can be done at this time to limit further negative 

outcomes in the F/A-22 program. Lastly, I will discuss the need for 

enforcing DOD’s newest acquisition policy, which on paper embraces best 
Page 2 GAO-03-645T Best Practices 

practices but in operation does not always do so, if DOD really expects to 

get improved outcomes in its major weapon system acquisitions. 

Clearly, the acquisition process has produced superior weapons, but it 

does so at a high price. Weapon systems routinely take much longer time 

to field, cost more to buy, and require more support than investment plans 

provide for. These consequences reduce the buying power of the defense 

dollar, delay capabilities for the war fighter, and force unplanned—and 

possibly unnecessary—trade-offs in desired acquisition quantities and an 

adverse ripple effect among other weapons programs or defense needs. 

Because of the lengthy time to develop new weapons, many enter the field 

with outdated technologies and a diminished supply base needed for 

system support. Frequently, this requires upgrades to the capability as 

soon as the new system is fielded. As previously noted, these inefficiencies 

have often led to reduced quantities of new systems. In turn, legacy 

systems remain in the inventory for longer periods, requiring greater 

operations and support cost that pull funds from other accounts, including 

modernization. DOD is facing these problems with its tactical air force 

assets now. We believe DOD can learn lessons from the experiences with 

the F/A-22 program as it frames the acquisition environment for its many 

transformational investments. 

DOD recognizes the need to get better weapon system outcomes, and its 

newest acquisition policy emphasizes the use of evolutionary, knowledgebased acquisition concepts proven to produce more effective and efficient 

outcomes in developing new products. It incorporates the elements of a 

knowledge-based acquisition model for developing new products, which 

we have recommended in our reviews of commercial best practices. Our 

body of work focuses on how DOD can better leverage its investments by 

shortening the time it takes to field new capabilities at a more predictable 

cost and schedule. However, policy changes alone will not guarantee 

success. Unless written policies are consistently implemented in practice 

through timely and informed decisions on individual programs, outcomes 

will not change. This requires sustained leadership and commitment and 

attention to the capture and use of key product knowledge at critical 

decision points to avoid the problems of the past. 

A key enabler to the success of commercial firms is using an approach that 

evolves a product to its ultimate capabilities on the basis of mature 

technologies and available resources. This approach allows commercial 

companies to develop and produce more sophisticated products faster and 

Improving Major 

Weapon System 

Acquisition Outcomes 

The Case for an 

Evolutionary Product 

Development Environment 
Page 3 GAO-03-645T Best Practices 

less expensively than their predecessors. Commercial companies have 

found that trying to capture the knowledge required to stabilize the design 

of a product that requires significant amounts of new technical content is 

an unmanageable task, especially if the goal is to reduce development 

cycle times and get the product to the marketplace as quickly as possible. 

Therefore, product features and capabilities not achievable in the initial 

development are planned for subsequent development efforts in future 

generations of the product, but only when technologies are proven to be 

mature and other resources are available. DOD’s new policy embraces the 

idea of evolutionary acquisition. Figure 1 compares evolutionary and 

single step (“big bang”) acquisitions. 
Page 4 GAO-03-645T Best Practices 

Figure 1: Comparison of Evolutionary and Big Bang Acquisition Approaches 
Page 5 GAO-03-645T Best Practices 

An evolutionary environment for developing and delivering new products 

reduces risks and makes cost more predictable. While the customer may 

not receive an ultimate capability initially, the product is available sooner, 

with higher quality and reliability, and at lower, more predictable cost. 

Improvements are planned for future generations of the product. 

Leading commercial firms expect that their program managers will deliver 

high-quality products on time and within budgets. Doing otherwise could 

result in losing a customer in the short term and losing the company in the 

longer term. Thus, in addition to creating an evolutionary product 

development environment that brings risk in control, these firms have 

adopted practices that put their individual program managers in a good 

position to succeed in meeting these expectations on individual products. 

Collectively, these practices ensure that a high level of knowledge exists 

about critical facets of the product at key junctures during its 

development. Such a knowledge-based process enables decision makers to 

be reasonably certain about critical facets of the product under 

development when they need to be. 

The knowledge-based process followed by leading firms is shown in detail 

in table 1, but in general can be broken down into three knowledge points. 

First, a match must be made between the customer’s needs and the 

available resources—technology, engineering knowledge, time, and 

funding—before a program is launched. Second, a product’s design must 

demonstrate its ability to meet performance requirements and be stable 

about midway through development. Third, the developer must show that 

the product can be manufactured within cost, schedule, and quality targets 

and is demonstrated to be reliable before production begins. The following 

table illustrates more specifically what we have learned about how 

successful programs gather knowledge as they move through product 

development. 

The Case for KnowledgeBased Product 

Development Process 
Page 6 GAO-03-645T Best Practices 

Table 1: Highlights of Specific Best Practices for Acquisitions 

 Knowledge Point 1 (Should occur before program launch) 

Separate technology from product development. 

Have clear measures and high standards for assessing technology maturity—technology readiness levels. 

Use a disciplined systems engineering process for translating and balancing customer’s desires with product developer’s technology, 

design, and production limitations; in other words, bring the right knowledge to the table when laying down a program’s foundation. 

Identify the mismatches between desired product features and the product developer’s knowledge and either (1) delay the start of the 

new product development until knowledge deficit can be made up or (2) reduce product features to lessen their dependence on areas 

where knowledge is insufficient (evolutionary acquisition). The main opportunities for trading off design features to save time and 

money occur here, before a program is started. 

When do you know you have achieved this knowledge point? When technologies needed to meet essential product requirements 

have been demonstrated to work in their intended environment and the producer has completed a preliminary design of the product. 

Knowledge Point 2 (Should occur midway between system integration and demonstration) 

Hold a major decision review between system integration and system demonstration that determines that the product design is stable 

and includes specific criteria to move into the system demonstration phase. 

Use integrated engineering prototypes to demonstrate design stability and prove with testing that the design meets the customer’s 

requirements. It is important that this happen before initial manufacturing begins—a point when investments are increased to produce 

an item. 

Identify critical manufacturing processes and establish a plan to bring these under statistical control by the start of production; also 

establish reliability goals and a growth plan to achieve these by production. This facilitates the achievement of process control and 

reliability goals at the completion of knowledge point 3. 

When do you know you have achieved this knowledge point? When 90 percent of engineering drawings are releasable to 

manufacturing organizations. Drawings are the language used by engineers to communicate to the manufacturers the details of the

new product—what it looks like, how its components interface, how to build it and the critical materials and processes needed to 

fabricate it. This makes drawings a key measure of whether the design is stable or not. 

Knowledge Point 3 (Should occur before production) 

Demonstrate that all critical manufacturing processes are under statistical control and consistently producing items within the quality 

standards and tolerances for the overall product before production begins. This is important, since variation in one process can 

reverberate to others and result in defective parts that need to be repaired or reworked. 

Demonstrate product reliability before the start of production. This requires testing to identify the problems, design corrections, and 

retest the new design. Commercial firms consider reliability important and its achievement a measure of design maturity. 

When do you know you have achieved this knowledge point? When all key manufacturing processes have come under statistical 

control and product reliability has been demonstrated. 
Page 7 GAO-03-645T Best Practices 

DOD programs often do not employ these practices. We found that if the 

evolutionary, knowledge-based acquisition concepts were not applied, a 

cascade of negative effects became magnified in the product development 

and production phases of an acquisition program. These led to acquisition 

outcomes that included significant cost increases and schedule delays, 

poor product quality and reliability, and delays in getting new capability to 

the war fighter. This is often the case in DOD programs as shown in our 

past work on systems like F/A-22 fighter, C-17 airlifter, V-22 tiltrotor 

aircraft, PAC-3 missile, BAT antitank munition, and others. We did find 

some DOD programs that employed best practice concepts and have had 

more successful program outcomes to date. These included the Global 

Hawk unmanned vehicle, AIM-9X missile, and Joint Direct Attack 

Munitions guided bomb. Figure 3 shows a notional illustration of the 

different paths and effects of a product development. 
Page 8 GAO-03-645T Best Practices 

Figure 2: Different Paths That A Product’s Development Can Take 

It is clear that knowledge about the product’s technology, design, and 

processes captured at the right time can reduce development cycle times 

and deliver a more cost effective, reliable product to the customer sooner 

than programs that do not capture this knowledge. 

In applying the knowledge-based approach, the most leveraged decision 

point of the three, is matching the customer’s needs with the developer’s 

resources—technology, design, timing, and funding. This initial decision 
Page 9 GAO-03-645T Best Practices 

sets the stage for the eventual outcome—desirable or problematic. The 

match is ultimately achieved in every development program, but in 

successful development programs, it occurs prior to program launch. In 

successful programs, negotiations and trade-offs occur before a product 

development is launched to ensure that a match exists between customer 

expectations and developer resources. The results achieved from this 

match are balanced and achievable requirements, sufficient investment to 

complete the development, and a firm commitment to deliver the product. 

Commercial companies we have visited usually limit product development 

cycle-time to less than 5 years. 

In DOD, this match is seldom achieved. It is not unusual for DOD to 

bypass early trade-offs and negotiations, instead planning to develop a 

product based on a rigid set of requirements that are unachievable within 

a reasonable development time frame. This results in cost and schedule 

commitments that are unrealistic. Although a program can take as long as 

15 years in DOD, the program manager is expected to develop and be 

accountable for precise cost and schedule estimates made at the start of 

the program. Because of their short tenures, it normally takes several 

program managers to complete product development. Consequently, the 

program manager that commits to the cost and schedule estimate at the 

beginning of the program is not the same person responsible for achieving 

it. Therefore, program accountability is problematic. Ironically, this 

outcome is rational in the traditional acquisition environment. The 

pressures put on program managers to get programs approved encourage 

promising more than can be delivered for the time and money allotted. 

They are not put in a position to succeed. 

The differences in the practices employed by successful commercial firms 

and DOD reflect the different demands imposed on programs by the 

environments in which they are managed. Specific practices take root and 

are sustained because they help a program succeed in its environment. 

The way success and failure are defined for commercial and defense 

product developments differs considerably, which creates a different set 

of incentives and evokes different behaviors from managers. Attempts at 

reforming weapon system acquisitions have not succeeded because they 

did not change these incentives. All of the participants in the acquisition 

process play a part in creating incentives. The F/A-22 program, advertised 

as a flagship of acquisition reform in its early days, failed to establish this 

match before program launch and today we are discussing the resulting 

outcomes to-date. 
Page 10 GAO-03-645T Best Practices 

The F/A-22 provides an excellent example of what can happen when a 

major acquisition program is not guided by the principles of evolutionary, 

knowledge-based acquisition. The program failed to match requirements 

with resources and make early trade-offs and took on a number of new 

and unproven technologies. Instead of fielding early capability and then 

evolving the product to get new capabilities to the war fighter sooner, the 

Air Force chose a “big bang” product development approach that is now 

planned to take about 19 years. This created a challenging and risky 

acquisition environment that has delayed the war fighter the capabilities 

expected from this new aircraft. Program leaders did not capture the 

specific knowledge identified as key for each of the three critical 

knowledge points in product development. Instead, program managers 

proceeded through the F/A-22’s development without the requisite 

knowledge necessary for reducing program risk and achieving more 

successful program outcomes. Now the optimism underlying these 

decisions has resulted in significant cost increases, schedule delays, tradeoffs—making do with less than half the number of originally desired 

aircraft—and concerns about the capability to be delivered. 

Since the F/A-22 acquisition program was started in October 1986, the F/A22 cost and schedule estimates have grown significantly to where, today, 

the Air Force estimates the total acquisition unit cost of a single aircraft is 

$257.5 million.1

 This represents a 74 percent increase from the estimate at 

the start of development and a commensurate loss in the buying power of 

the defense dollar. Intended to replace the aging F-15 fighter, the F/A-22 

program is now scheduled to reach its initial operational capability in 

December 2005—making its development cycle about 19 years. During this 

cycle, the planned buy quantity has been reduced 63 percent from 750 to 

276 aircraft2

. In addition, since fiscal year 2001, funding for F/A-22 

upgrades has dramatically increased from $166 million to $3.0 billion, most 

of which is to provide increased ground attack capability, a requirement 

that was added late in the development program. 

 1

 All references to F/A-22 costs in this testimony are in then-year dollars in order to 

maintain consistent reporting with our prior reports on the F/A-22. 

2

 Between 1986 and the start of engineering and manufacturing development in 1991, the 

quantity was reduced from 750 to 648 aircraft. 

F/A-22 Did Not 

Employ Evolutionary 

or Knowledge-Based 

Process 

F/A-22 Program Outcomes 
Page 11 GAO-03-645T Best Practices 

The F/A-22 acquisition strategy from the outset was to achieve full 

capability in a “big bang” approach. By not using an evolutionary 

approach, the F/A-22 took on significant risk and onerous technological 

challenges. While the big bang approach may have allowed the Air Force 

to more successfully compete for early funding, it hamstrung the program 

with many new undemonstrated technologies, preventing the program 

from knowing cost and schedule ramifications throughout development. 

Cost, schedule, and performance problems resulted. The following table 

summarizes the F/A-22 program’s attainment of critical knowledge and key 

decision junctures during the development program and the changes in 

development cost and cycle time at each point. 

Table 2: Knowledge Attainment in the F/A-22 Program 

Program start—1986 Design review—1995 Production start—2001 

Best practice Attain knowledge point 1. 

Separate technology and product 

development, deliver mature 

technology, and have preliminary 

design. 

Attain knowledge point 2. 90 

percent of systems and 

structures engineering drawings 

releasable and subsystem 

design reviews completed. 

Attain knowledge point 3. 100% of 

critical manufacturing processes in 

statistical control and reliability 

goals demonstrated. 

F/A-22 practice Knowledge point 1 not attained. 

Failed to separate technology 

and product development. Three 

critical technologies immature: 

Low-observable materials, 

propulsion, and integrated 

avionics. Knowledge point 1 not 

attained until September 2000. 

Knowledge points 1 and 2 not 

attained. Only 26 percent of 

drawings released at the critical 

design review in February 1995. 

Knowledge point 2 not attained 

until September 1998, after 

delivery of second test aircraft. 

Knowledge point 3 not attained. 

Less than 50 percent of critical 

manufacturing processes in control. 

Only 22 percent of reliability goal 

demonstrated with many 

outstanding deficiencies. 

F/A-22’s estimated 

development costa

 $12.6 billion $21.2 billion 

(68 percent increase) 

$28.7 billion 

(128 percent increase) 

Estimated development 

cycle time 9.4 years 18.1 years 

(93 percent increase) 

19.2 years 

(104 percent increase) 

a

The development estimate includes all F/A-22 RDT&E costs. 

Technology—The F/A-22 did not have mature technology at the start of 

the acquisition program. The program included new low-observable 

(stealth) materials, integrated avionics, and propulsion technology that 

were not mature at this time. The Air Force did not complete an evaluation 

of stealth technology on a full-scale model of the aircraft until several 

years into development. It was not until September 2000, or 9 years into 

F/A-22 Did Not Use 

Evolutionary Acquisition 

or Capture Knowledge 

Required at Key Decision 

Junctures 
Page 12 GAO-03-645T Best Practices 

development, that the integrated avionics reached a maturity level 

acceptable to begin product development. During development, the 

integrated avionics was a source of schedule delays and cost growth. 

Since 1997, avionics software development and flight-testing have been 

delayed, and the cost of avionics development has increased by over $980 

million dollars. Today, the avionics still has problems affecting the ability 

to complete developmental testing and begin operational testing, and the 

Air Force cannot predict when a solution will be found. 

Design—The effects of immature technologies cascaded into the F/A-22 

development program, making it more difficult to achieve a stable design 

at the right time. The standard measure of design stability is 90 percent of 

design drawings releasable by the critical design review. The F/A-22 

achieved only 26 percent by this review, taking an additional 43 months to 

achieve the standard. Moving ahead in development, the program 

experienced several design and manufacturing problems described by the 

F/A-22 program office as a “rolling wave” effect throughout system 

integration and final assembly. These effects included numerous design 

changes, labor inefficiencies, parts shortages, out of sequence work, cost 

increases, and schedule delays. 

Production—At the start of production, the F/A-22 did not have 

manufacturing processes under control and was only beginning testing 

and demonstration efforts for system reliability. Initially, the F/A-22 had 

taken steps to use statistical process control data to gain control of critical 

manufacturing processes by full rate production. However, the program 

abandoned this best practice approach in 2000 with less than 50 percent of 

its critical manufacturing processes in control. In March 2002,3

 we 

recommended that the F/A-22 program office monitor the status of critical 

manufacturing process as the program proceeds toward high rate 

production. 

The reliability goal for the F/A-22 is 3 hours of flying time between 

maintenance actions. The Air Force estimated that in late 2001, when it 

entered production, it should have been able to demonstrate almost 2 

flying hours between maintenance actions. Instead, it could fly an average 

of only 0.44 hours between maintenance actions. Since then there has 

 3

 U.S. General Accounting Office, Tactical Aircraft: F-22 Delays Indicate Initial 

Production Rates Should Be Lower to Reduce Risks, GAO-02-298 (Washington, D.C.: Mar. 

5, 2002). 
Page 13 GAO-03-645T Best Practices 

been a decrease in reliability. As of November 2002, development test 

aircraft have been completing only 0.29 hours between maintenance 

actions. Additionally, the program was slow to fix and correct problems 

that had affected reliability. At the time of our review in July 2002, 

program officials had identified about 260 different types of failures and 

had identified fixes for less than 50 percent of the failures. To achieve 

reliability goals will require additional design changes, testing, and 

modifications. Therefore, additional problems and costs can be expected if 

the system is fielded with the level of reliability achieved to date. 

The F/A-22 did not take advantage of evolutionary, knowledge-based 

concepts up front and now, the best it can hope for is to limit cost 

increases and performance problems by not significantly increasing its 

production until development is complete—signified by developmental 

and operational testing and reliability demonstrations. To that end, we 

have recommended that the Air Force reconsider its decision to increase 

the aircraft production rate beyond 16 aircraft per year.4

 The program is 

nearing the end of developmental testing and plans to start initial 

operational testing in October 2003. If developmental testing goes as 

planned, which is not guaranteed, operational testing is expected to be 

completed around September 2004. By the end of this fiscal year, 51 F/A22s will be on contract as low rate production began in 2001. 

Our March 2003 report identifies various problems still outstanding that 

could have further impacts on cost, schedule, and delivered performance 

that are in addition to undemonstrated reliability goals. The problems 

identified are of particular concern, given Air Force plans to increase 

production rates and make a full rate production decision in 2004. The 

problems include: 

• unexpected shutdowns (instability) of the avionics, 

• excessive movement of the vertical tails, 

• overheating in rear portions of the aircraft, 

• separations of the horizontal tail material, 

• inability to meet airlift support requirements, and 

• excessive ground maintenance actions. 

 4

 U.S. General Accounting Office, Tactical Aircraft: DOD Should Reconsider Decision to 

Increase F/A-22 Production Rates While Development Risks Continue, GAO-03-431

(Washington, D.C.: Mar. 14, 2003). 

It Is Too Late for the F/A22 Program to Gain Full 

Benefit of a KnowledgeBased Process 
Page 14 GAO-03-645T Best Practices 

These problems are still being addressed, and not all of them have been 

solved as yet. For example, Air Force officials stated they do not yet 

understand the problems associated with the avionics instability well 

enough to predict when they would be able to resolve them, and certain 

tests to better understand the vertical tail problem have not yet begun. 

Despite remaining testing and outstanding problems, the Air Force plans 

to continue acquiring production aircraft at increasing annual rates and 

make the full rate production decision in 2004. This is a very risky strategy, 

given outstanding issues in the test program and the system’s less than 

expected reliability. The Air Force may encounter higher production costs 

as a result of acquiring significant quantities of aircraft before adequate 

testing and demonstrations are complete. In addition, remaining testing 

could identify problems that require costly modifications in order to 

achieve satisfactory performance. 

In a February 28, 2003 report to Representative John Tierney,5

 we found 

that F/A-22 production costs are likely to increase more than the latest 

$5.4 billion cost growth recently estimated by the Air Force and the Office 

of Secretary of Defense (OSD). First, the current OSD production estimate 

does not include $1.3 billion included in the latest Air Force acquisition 

plan. Second, schedule delays in developmental testing could further 

postpone the start of the first F/A-22 multiyear contract, which has already 

been delayed until fiscal year 2006. This could result in lower cost savings 

from multiyear procurement. Last, we found several risk factors that may 

increase future production costs, including the dependency of certain cost 

reduction plans on Air Force investments that are not being made to 

improve production processes, the availability of funding, and a reduction 

in funding for support costs. In addition, DOD has not informed Congress 

about the quantity of aircraft that can be procured within existing 

production cost limits, which we believe could be fewer than the 276 

currently planned. Further details on F/A-22 cost growth and the Air 

Force’s attempt to offset it are provided in appendix I. 

 5

 U.S. General Accounting Office, Tactical Aircraft: DOD Needs to Better Inform Congress 

about Implications of Continuing F/A-22 Cost Growth, GAO-03-280 (Washington, D.C.: 

Feb. 28, 2003). 
Page 15 GAO-03-645T Best Practices 

While DOD’s new acquisition policy is too late to influence the F/A-22 

program, it is not too late for other major acquisition programs like the 

Missile Defense Agency’s suite of land, sea, air, and space defense 

systems; the Army’s Future Combat Systems; and the Air Force and Navy’s 

Joint Strike Fighter. DOD’s revised acquisition policy represents tangible 

leadership action to getting better weapon system acquisition outcomes, 

but unless the policies are implemented through decisions on individual 

programs, outcomes are not likely to change. Further, unless pressures are 

alleviated in DOD to get new acquisition programs approved and funded 

on the basis of requirements that must stand out, programs will continue 

to be compromised from the outset with little to no chance of successful 

outcomes. If new policies were implemented properly, through decisions 

on individual programs, managers would face less pressure to promise 

delivery of all the ultimate capabilities of a weapon system in one “big 

bang.” 

Both form and substance are essential to getting desired outcomes. At a 

tactical level, we believe that the policies could be made more explicit in 

several areas to facilitate such decisions. First, the regulations provide 

little or no controls at key decision points of an acquisition program that 

force a program manager to report progress against knowledge-based 

metrics. Second, the new regulations, once approved, may be too general 

and may no longer provide mandatory procedures. Third, the new 

regulations may not provide adequate accountability because they may not 

require knowledge-based deliverables containing evidence of knowledge 

at key decision points. 

At a strategic level, some cultural changes will be necessary to translate 

policy into action. At the very top level, this means DOD leadership will 

have to take control of the investment dollars and to say “no” in some 

circumstances if programs are inappropriately deviating from sound 

acquisition policy. In my opinion, programs should follow a knowledgebased acquisition policy—one that embraces best practices—unless there 

is a clear and compelling national security reason not to. Other cultural 

changes instrumental to implementing change include: 

• Keeping key people in place long enough so that they can affect decisions 

and be held accountable. 

• Providing program offices with the skilled people needed to craft 

acquisition approaches that implement policy and to effectively oversee 

the execution of programs by contractors. 

Real Change in 

Acquisition Outcomes 

Requires Disciplined 

Enforcement of 

Acquisition Policy 
Page 16 GAO-03-645T Best Practices 

• Realigning responsibilities and funding between science and technology 

organizations and acquisition organizations to enable the separation of 

technology development from product development. 

• Bringing discipline to the requirements-setting process by demanding a 

match between requirements and resources. 

• Requiring readiness and operating cost as key performance parameters 

prior to beginning an acquisition. 

• Designing and implementing test programs that deliver knowledge when 

needed, including reliability testing early in design. 

Ultimately, the success of the new acquisition policy will be seen in 

individual program and resource decisions. Programs that are 

implementing knowledge-based policies in their acquisition approaches 

should be supported and resourced, presuming they remain critical to 

national needs and affordable within current and projected resource 

levels. Conversely, if programs that repeat the approaches of the past are 

approved and funded, past policies—and their outcomes—will be 

reinforced with a number of adverse implications. 

DOD will continue to face challenges in modernizing its forces with new 

demands on the federal dollar created by changing world conditions. 

Consequently, it is incumbent upon DOD to find and adopt best product 

development practices that can allow it to manage its weapon system 

programs in the most efficient and effective way. Success over the long 

term will depend not only on policies that embrace evolutionary, 

knowledge-based acquisition practices but also on DOD leadership’s 

sustaining its commitment to improving business practices and ensuring 

that those adopted are followed and enforced. 

DOD’s new acquisition policy embraces the best practice concepts of 

knowledge-based, evolutionary acquisition and represents a good first step 

toward achieving better outcomes from major acquisition programs. The 

F/A-22 program followed a different path at its beginning, a big bang, highrisk approach whose outcomes so far have been increased cost, quality 

and reliability problems, growing procurement reductions, and delays in 

getting the aircraft to the war fighter. Since this program is nearing the end 

of development and already into production, it is too late to adopt a 

knowledge approach, but it can limit further cost increases and adverse 

actions by not ramping up production beyond current levels until 

developmental and operational testing are completed and reliability goals 

have been demonstrated. Regardless of the F/A-22’s current predicament, 

the new policy can and should be used to manage all new acquisition 

Conclusions 
Page 17 GAO-03-645T Best Practices 

programs and should be adapted to those existing programs that have not 

progressed too far in development to benefit. At a minimum, the F/A-22 

should serve as a lesson learned from which to effect a change in the 

future DOD acquisition environment. The costs of doing otherwise are 

simply too high for us to tolerate. 

Mr. Chairman, this concludes my prepared statement. I would be happy to 

respond to any questions that you or other members of the Subcommittee 

may have. 
Page 18 GAO-03-645T Best Practices 

Over the last 6 years, DOD has identified about $18 billion in estimated 

production cost growth during the course of two DOD program reviews. 

As a result, the estimated cost of the production program currently 

exceeds the congressional cost limit. The Air Force has implemented cost 

reduction plans designed to offset a significant amount of this estimated 

cost growth. But the effectiveness of these cost reduction plans has varied. 

During a 1997 review, the Air Force estimated cost growth of $13.1 billion.1

The major contributing factors to this cost growth were inflation, 

increased estimates of labor costs and materials associated with the 

airframe and engine, and engineering changes to the airframe and engine. 

These factors made up about 75 percent of the cost growth identified in 

1997. 

In August 2001, DOD estimated an additional $5.4 billion in cost growth for 

the production of the F/A-22, bringing total estimated production cost to 

$43 billion. The major contributing factors to this cost growth were again 

due to increased labor costs and airframe and engine costs. These factors 

totaled almost 70 percent of the cost growth. According to program 

officials, major contractors’ and suppliers’ inability to achieve the 

expected reductions in labor costs throughout the building of the 

development and early production aircraft has been the primary reason for 

estimating this additional cost growth. 

The Air Force was able to implement cost reduction plans and offset cost 

growth by nearly $2 billion in the first four production contracts awarded. 

As shown in table 3, the total offsets for these contracts slightly exceeded 

earlier projections by about $.5 million. 

 1

 Based on a plan to procure 438 aircraft. 

Appendix I: F/A-22 Production Cost Growth 

Mixed Success With Cost 

Reduction Plans 
Page 19 GAO-03-645T Best Practices 

Table 3: Comparison of Planned Versus Implemented Cost Reduction Offsets for 

Awarded Production Contracts 

Dollars in millions 

Production lot 

Planned 

offsets 

Implemented 

offsets Difference

Fiscal year 1999 (2 aircraft) $199.0 $200.5 $1.5

Fiscal year 2000 (6 aircraft) 329.3 336.4 7.1

Fiscal year 2001 (10 aircraft) 580.2 611.1 30.9

Fiscal year 2002 (13 aircraft) 827.2 788.2 (39.0)

Total $1,935.7 $1,936.2 $0.5

Source: Air Force. 

Cost reduction plans exist but have not yet been implemented for 

subsequent production lots planned for fiscal years 2003 through 2010 

because contracts for these production lots have not yet been awarded. If 

implemented successfully, the Air Force expects these cost reduction 

plans to achieve billions of dollars in offsets to estimated cost growth and 

to allow the production program to be completed within the current 

production cost estimate of $43 billion.2

 However, this amount exceeds the 

production cost limit of $36.8 billion. 

In addition, while the Air Force has been attempting to offset costs 

through production improvement programs (PIPs), recent funding 

cutbacks for PIPs may reduce their effectiveness. PIPs focus specifically 

on improving production processes to realize savings by using an initial 

government investment. The earlier the Air Force implements PIPs, the 

greater the impact on the cost of production. Examples of PIPs previously 

implemented by the Air Force include manufacturing process 

improvements for avionics, improvements in fabrication and assembly 

processes for the airframe, and redesign of several components to enable 

lower production costs. 

As shown in figure 3, the Air Force reduced the funding available for 

investment in PIPs by $61 million for lot 1 and $26 million for lot 2 to cover 

 2

 The F/A-22 President’s budget for fiscal year 2004 would transfer $876 million in 

production funding to help fund estimated cost increases in development. As a result, the 

current production cost estimate is $42.2 billion. 
Page 20 GAO-03-645T Best Practices 

cost growth in production lots 1 and 23. As a result, it is unlikely that PIPs 

covering these two lots will be able to offset cost growth as planned. 

Figure 3: Planned Versus Actual F/A-22 Production Improvement Program 

Investment for Production Lots 1 (Fiscal Year 2001) and 2 (Fiscal Year 2002) 

Figure 4 shows the remaining planned investment in PIPs through fiscal 

year 2006 and the $3.7 billion in estimated cost growth that can potentially 

be offset through fiscal year 2010 if the Air Force invests as planned in 

these PIPs. 

 3

 Production lot 1 was awarded in fiscal year 2001 and production lot 2 was awarded in 

fiscal year 2002. 
Page 21 GAO-03-645T Best Practices 

Figure 4: Planned Offsets to Cost Growth From Investing in and Implementing PIPs 

In the past, Congress has been concerned about the Air Force’s practice of 

requesting fiscal year funding for these PIPs but then using part of that 

funding for F/A-22 airframe cost increases. 4 Recently, Congress directed 

the Air Force to submit a request if it plans to use PIP funds for an 

alternate purpose. 

 4

 Report 107-298, Nov. 19, 2001. 
Page 22 GAO-03-645T Best Practices 

Best Practices: Setting Requirements Differently Could Reduce Weapon 

Systems’ Total Ownership Costs. GAO-03-57. Washington, D.C.: February 

11, 2003. 

Best Practices: Capturing Design and Manufacturing Knowledge Early 

Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15, 

2002. 

Defense Acquisitions: DOD Faces Challenges in Implementing Best 

Practices. GAO-02-469T. Washington, D.C.: February 27, 2002. 

Best Practices: Better Matching of Needs and Resources Will Lead to 

Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8, 

2001. 

Best Practices: A More Constructive Test Approach Is Key to Better 

Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31, 

2000. 

Defense Acquisitions: Employing Best Practices Can Shape Better 

Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 

26, 2000. 

Best Practices: DOD Training Can Do More to Help Weapon System 

Programs Implement Best Practices. GAO/NSIAD-99-206. Washington, 

D.C.: August 16, 1999. 

Best Practices: Better Management of Technology Development Can 

Improve Weapon System Outcomes. GAO/NSIAD-99-162. Washington, 

D.C.: July 30, 1999. 

Best Practices: DOD Can Help Suppliers Contribute More to Weapon 

System Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998. 

Best Practices: Successful Application to Weapon Acquisition Requires 

Changes in DOD’s Environment. GAO/NSIAD-98-56. Washington, D.C.: 

February 24, 1998. 

Best Practices: Commercial Quality Assurance Practices Offer 

Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26, 

1996. 

Related GAO Products 
Page 23 GAO-03-645T Best Practices 

Tactical Aircraft: Status of the F/A-22 Program. GAO-03-603T. 

Washington, D.C.: April 2, 2003. 

Tactical Aircraft: DOD Should Reconsider Decision to Increase F/A-22 

Production Rates While Development Risks Continue. GAO-03-431. 

Washington, D.C.: March 14, 2003. 

Tactical Aircraft: DOD Needs to Better Inform Congress about 

Implications of Continuing F/A-22 Cost Growth. GAO-03-280. 

Washington, D.C.: February 28, 2003. 

Tactical Aircraft: F-22 Delays Indicate Initial Production Rates Should 

Be Lower to Reduce Risks. GAO-02-298. Washington, D.C.: March 5, 2002. 

Tactical Aircraft: Continuing Difficulty Keeping F-22 Production Costs 

within the Congressional Limitation. GAO-01-782. Washington, D.C.: July 

16, 2001. 

Tactical Aircraft: F-22 Development and Testing Delays Indicate Need 

for Low-Rate Production. GAO-01-310. Washington, D.C.: March 15, 2001. 

Defense Acquisitions: Recent F-22 Production Cost Estimates Exceeded 

Congressional Limitation. GAO/NSIAD-00-178. Washington, D.C.: August 

15, 2000. 

Defense Acquisitions: Use of Cost Reduction Plans in Estimating F-22 

Total Production Costs. GAO/T-NSIAD-00-200. Washington, D.C.: June 15, 

2000. 

Budget Issues: Budgetary Implications of Selected GAO Work for Fiscal 

Year 2001. GAO/OCG-00-8. Washington, D.C.: March 31, 2000. 

F-22 Aircraft: Development Cost Goal Achievable If Major Problems Are 

Avoided. GAO/NSIAD-00-68. Washington, D.C.: March 14, 2000. 

Defense Acquisitions: Progress in Meeting F-22 Cost and Schedule Goals.

GAO/T-NSIAD-00-58. Washington, D.C.: December 7, 1999. 

Fiscal Year 2000 Budget: DOD’s Procurement and RDT&E Programs.

GAO/NSIAD-99-233R. Washington D.C.: September 23, 1999. 

Budget Issues: Budgetary Implications of Selected GAO Work for Fiscal 

Year 2000. GAO/OCG-99-26. Washington, D.C.: April 16, 1999. 
Page 24 GAO-03-645T Best Practices 

Defense Acquisitions: Progress of the F-22 and F/A-18E/F Engineering 

and Manufacturing Development Programs. GAO/T-NSIAD-99-113. 

Washington, D.C.: March 17, 1999. 

F-22 Aircraft: Issues in Achieving Engineering and Manufacturing 

Development Goals. GAO/NSIAD-99-55. Washington, D.C.: March 15, 1999. 

F-22 Aircraft: Progress of the Engineering and Manufacturing 

Development Program. GAO/T-NSIAD-98-137. Washington D.C.: March 25, 

1998. 

F-22 Aircraft: Progress in Achieving Engineering and Manufacturing 

Development Goals. GAO/NSIAD-98-67. Washington, D.C.: March 10, 1998. 

Tactical Aircraft: Restructuring of the Air Force F-22 Fighter Program.

GAO/NSIAD-97-156. Washington, D.C.: June 4, 1997. 

Defense Aircraft Investments: Major Program Commitments Based on 

Optimistic Budget Projections. GAO/T-NSIAD-97-103. Washington, D.C.: 

March 5, 1997. 

F-22 Restructuring. GAO/NSIAD-97-100BR. Washington, D.C.: February 

28, 1997. 

Tactical Aircraft: Concurrency in Development and Production of F-22 

Aircraft Should Be Reduced. GAO/NSIAD-95-59. Washington, D.C.: April 

19, 1995. 

Tactical Aircraft: F-15 Replacement Issues. GAO/T-NSIAD-94-176. 

Washington, D.C.: May 5, 1994. 

Tactical Aircraft: F-15 Replacement Is Premature as Currently Planned.

GAO/NSIAD-94-118. Washington, D.C.: March 25, 1994. 

(120231) 
This is a work of the U.S. government and is not subject to copyright protection in the 

United States. It may be reproduced and distributed in its entirety without further 

permission from GAO. However, because this work may contain copyrighted images or 

other material, permission from the copyright holder may be necessary if you wish to 

reproduce this material separately.