This is the accessible text file for GAO report number GAO-03-645T 
entitled 'Best Practices: Better Acquisition Outcomes Are Possible If 
DOD Can Apply Lessons from F/A-22 Program' which was released on April 
11, 2003.

This text file was formatted by the U.S. General Accounting Office 

(GAO) to be accessible to users with visual impairments, as part of a 

longer term project to improve GAO products accessibility. Every 

attempt has been made to maintain the structural and data integrity of 

the original printed product. Accessibility features, such as text 

descriptions of tables, consecutively numbered footnotes placed at the 

end of the file, and the text of agency comment letters, are provided 

but may not exactly duplicate the presentation or format of the printed 

version. The portable document format (PDF) file is an exact electronic 

replica of the printed version. We welcome your feedback. Please E-mail 

your comments regarding the contents or accessibility features of this 

document to Webmaster@gao.gov.

Testimony:

Before the Subcommittee on National Security, Emerging Threats, and 

International Relations, Government Reform Committee, House of 

Representatives:

United States General Accounting Office:

GAO:

For Release on Delivery Expected at 10:00 a.m. EDT:

Friday, April 11, 2003:

Best Practices:

Better Acquisition Outcomes Are Possible If DOD Can Apply Lessons from 

F/A-22 Program:

Statement of David M. Walker,

Comptroller General of the United States:

GAO-03-645T:

GAO Highlights:

Highlights of GAO-03-645T, a testimony before the Subcommittee on 

National Security, Emerging Threats, and International Relations, 

Government Reform Committee, House of Representatives 

Why GAO Did This Study:

Over the next 5 years, DODs overall investments are expected to 

average $150 billion a year to modernize and transition our forces.  

In addition, DOD must modernize its forces amid competing demands for 

federal funds, such as health care and homeland security.  Therefore, 

it is critical that DOD manage its acquisitions in the most cost 

efficient and effective manner possible.

DODs newest acquisition policy emphasizes the use of evolutionary, 

knowledge-based concepts that have proven to produce more effective 

and efficient weapon systems outcomes.  However, most DOD programs 

currently do not employ these practices and, as a result, experience 

cost increases, schedule delays, and poor product quality and 

reliability.

This testimony compares the best practices for developing new products 

with the experiences of the F/A-22 program.

What GAO Found:

GAOs reviews of commercial best practices have identified key enablers 

to the success of product development programs and focused on how DOD 

can better leverage its investments by shortening the time it takes to 

field new capabilities at a more predictable cost and schedule.  First, 

commercial firms use an approach that evolves a product to its ultimate 

capabilities on the basis of mature technologies and available 

resources.  This approach allows only the product features and 

capabilities achievable with available resources in the initial 

development.  Further product enhancements are planned for subsequent 

development efforts when technologies are proven to be mature and other 

resources are available.  Second, commercial firms ensure that a high 

level of knowledge exists at key junctures during a products 

development.  The knowledge-based process includes three points:

* Before a program is launched, successful programs match customer 

needs and available resourcestechnology, engineering knowledge, time, 

and funding.

* About midway through development, the ability of the products design 

is demonstrated to be stable and meet performance requirements.

* Before production begins, programs must show that a product can be 

manufactured within cost, schedule, and quality targets.

In contrast, the F/A-22 program illustrates what can happen when a 

major acquisition program is not guided by the principles of 

evolutionary, knowledge-based acquisition.  When the program was 

started, several key technologies were not mature.  Program managers 

proceeded through development without the requisite knowledge to 

effectively manage program risk and, at the start of production, key 

manufacturing processes were not under control.  The F/A-22 program has 

undergone significant cost increases.  Instead of fielding early 

capabilities to the war fighter, the development cycle has extended to 

19 years, so far, and original quantities have been significantly 

reduced, raising concerns about the capability the program will 

eventually deliver. 

DOD recognizes the need to get better weapon system outcomes, and its 

newest acquisition policy emphasizes the use of evolutionary, 

knowledge-based acquisition concepts proven to produce better outcomes 

in developing new products.  However, policy changes alone are not 

enough.  Leadership commitment and attention to putting the policy into 

practice for individual programs is needed to avoid the problems of the 

past.  DOD will have many opportunities to do so over the next several 

years with its force modernization investments.

What GAO Recommends:

GAO is not making recommendations in this testimony.  However, in a 

number of prior reports, GAO has recommended that DOD adopt policies 

with metrics for technology, design, and manufacturing maturity to 

support knowledge-based decision making.  These policies should apply 

when making decisions on individual weapons programs. 

www.gao.gov/cgi-bin/getrpt?GAO-03-645T.

To view the full report, including the scope

and methodology, click on the link above.

For more information, contact Katherine Schinasi at (202) 512-4841 or 

schinasik@gao.gov.

[End of section]

Mr. Chairman and Members of the Subcommittee:

I am pleased to be here today to participate in the Subcommittees 

hearing on how the Department of Defense (DOD) can--and must--get 

better outcomes from its weapon system investments. DOD is on the 

threshold of several major investments in acquisition programs that are 

likely to dominate budget and doctrinal debates well into the next 

decade. These programs include such systems as the Missile Defense 

Agencys suite of land, sea, air, and space defense systems; the Armys 

Future Combat Systems; and the Air Forces and Navys Joint Strike 

Fighter. Over the next 5 years, DODs overall investments are expected 

to average $150 billion a year as DOD works to keep legacy systems as 

well as modernize and transform our national defense capabilities for 

the future. Therefore, to meet these challenges, it is essential that 

sound foundations for these and other weapon system investments be laid 

now so that the resulting programs can be executed within estimates of 

available resources.

Any discussion of improvements to DODs modernization efforts must be 

set in the context of overall expected budget availability. There are 

important competing priorities. Health care costs are growing at 

double-digit rates, and spending on homeland security will likely grow 

as we seek to defeat terrorism worldwide. We face an oncoming 

demographic tidal wave, and by 2035 the number of people who are 65 or 

older will have doubled, creating much larger demands on the federal 

budget. The demand of funding for entitlement programs continues to 

grow, creating increasing pressures on discretionary funding for other 

federal priorities like education and defense. Therefore, it is 

critical that DOD manage its acquisitions in the most cost efficient 

and effective manner possible.

My testimony today is about improving the outcomes of major weapon 

system acquisitions by using best practices to capture and use the 

right product knowledge at the right time for better decision making 

during product development. As per your request, I will compare 

acquisition practices and decisions made for the F/A-22 with these best 

practices for developing new products. The divergence between F/A-22 

experiences and best product development practices, we believe, largely 

explains why the F/A-22 has been in development for over 16 years and 

its cost has grown substantially. It is also a primary contributor to 

other performance issues that are currently faced by the program. My 

testimony will also include observations on what can be done at this 

time to limit further negative outcomes in the F/A-22 program. Lastly, 

I will discuss the need for enforcing DODs newest acquisition policy, 

which on paper embraces best practices but in operation does not always 

do so, if DOD really expects to get improved outcomes in its major 

weapon system acquisitions.

Improving Major Weapon System Acquisition Outcomes:

Clearly, the acquisition process has produced superior weapons, but it 

does so at a high price. Weapon systems routinely take much longer time 

to field, cost more to buy, and require more support than investment 

plans provide for. These consequences reduce the buying power of the 

defense dollar, delay capabilities for the war fighter, and force 

unplanned--and possibly unnecessary--trade-offs in desired acquisition 

quantities and an adverse ripple effect among other weapons programs or 

defense needs. Because of the lengthy time to develop new weapons, many 

enter the field with outdated technologies and a diminished supply base 

needed for system support. Frequently, this requires upgrades to the 

capability as soon as the new system is fielded. As previously noted, 

these inefficiencies have often led to reduced quantities of new 

systems. In turn, legacy systems remain in the inventory for longer 

periods, requiring greater operations and support cost that pull funds 

from other accounts, including modernization. DOD is facing these 

problems with its tactical air force assets now. We believe DOD can 

learn lessons from the experiences with the F/A-22 program as it frames 

the acquisition environment for its many transformational investments.

DOD recognizes the need to get better weapon system outcomes, and its 

newest acquisition policy emphasizes the use of evolutionary, 

knowledge-based acquisition concepts proven to produce more effective 

and efficient outcomes in developing new products. It incorporates the 

elements of a knowledge-based acquisition model for developing new 

products, which we have recommended in our reviews of commercial best 

practices. Our body of work focuses on how DOD can better leverage its 

investments by shortening the time it takes to field new capabilities 

at a more predictable cost and schedule. However, policy changes alone 

will not guarantee success. Unless written policies are consistently 

implemented in practice through timely and informed decisions on 

individual programs, outcomes will not change. This requires sustained 

leadership and commitment and attention to the capture and use of key 

product knowledge at critical decision points to avoid the problems of 

the past.

The Case for an Evolutionary Product Development Environment:

A key enabler to the success of commercial firms is using an approach 

that evolves a product to its ultimate capabilities on the basis of 

mature technologies and available resources. This approach allows 

commercial companies to develop and produce more sophisticated products 

faster and less expensively than their predecessors. Commercial 

companies have found that trying to capture the knowledge required to 

stabilize the design of a product that requires significant amounts of 

new technical content is an unmanageable task, especially if the goal 

is to reduce development cycle times and get the product to the 

marketplace as quickly as possible. Therefore, product features and 

capabilities not achievable in the initial development are planned for 

subsequent development efforts in future generations of the product, 

but only when technologies are proven to be mature and other resources 

are available. DODs new policy embraces the idea of evolutionary 

acquisition. Figure 1 compares evolutionary and single step (big 

bang) acquisitions.

Figure 1: Comparison of Evolutionary and Big Bang Acquisition 

Approaches:

[See PDF for image]

[End of figure]

An evolutionary environment for developing and delivering new products 

reduces risks and makes cost more predictable. While the customer may 

not receive an ultimate capability initially, the product is available 

sooner, with higher quality and reliability, and at lower, more 

predictable cost. Improvements are planned for future generations of 

the product.

The Case for Knowledge-Based Product Development Process:

Leading commercial firms expect that their program managers will 

deliver high-quality products on time and within budgets. Doing 

otherwise could result in losing a customer in the short term and 

losing the company in the longer term. Thus, in addition to creating an 

evolutionary product development environment that brings risk in 

control, these firms have adopted practices that put their individual 

program managers in a good position to succeed in meeting these 

expectations on individual products. Collectively, these practices 

ensure that a high level of knowledge exists about critical facets of 

the product at key junctures during its development. Such a knowledge-

based process enables decision makers to be reasonably certain about 

critical facets of the product under development when they need to be.

The knowledge-based process followed by leading firms is shown in 

detail in table 1, but in general can be broken down into three 

knowledge points. First, a match must be made between the customers 

needs and the available resources--technology, engineering knowledge, 

time, and funding--before a program is launched. Second, a products 

design must demonstrate its ability to meet performance requirements 

and be stable about midway through development. Third, the developer 

must show that the product can be manufactured within cost, schedule, 

and quality targets and is demonstrated to be reliable before 

production begins. The following table illustrates more specifically 

what we have learned about how successful programs gather knowledge as 

they move through product development.

Table 1: Highlights of Specific Best Practices for Acquisitions:

Knowledge Point 1 (Should occur before program launch): Separate 

technology from product development.

Knowledge Point 1 (Should occur before program launch): Have clear 

measures and high standards for assessing technology maturity--

technology readiness levels.

Knowledge Point 1 (Should occur before program launch): Use a 

disciplined systems engineering process for translating and balancing 

customers desires with product developers technology, design, and 

production limitations; in other words, bring the right knowledge to 

the table when laying down a programs foundation.

Knowledge Point 1 (Should occur before program launch): Identify the 

mismatches between desired product features and the product developers 

knowledge and either (1) delay the start of the new product development 

until knowledge deficit can be made up or (2) reduce product features 

to lessen their dependence on areas where knowledge is insufficient 

(evolutionary acquisition). The main opportunities for trading off 

design features to save time and money occur here, before a program is 

started.

Knowledge Point 1 (Should occur before program launch): When do you 

know you have achieved this knowledge point? When technologies needed 

to meet essential product requirements have been demonstrated to work 

in their intended environment and the producer has completed a 

preliminary design of the product.

Knowledge Point 2 (Should occur midway between system integration and 

demonstration).: Hold a major 

decision review between system integration and system demonstration 

that determines that the product design is stable and includes specific 

criteria to move into the system demonstration phase.

Knowledge Point 2 (Should occur midway between system integration and 

demonstration): Use integrated 

engineering prototypes to demonstrate design stability and prove with 

testing that the design meets the customers requirements. It is 

important that this happen before initial manufacturing begins--a point 

when investments are increased to produce an item.

Knowledge Point 2 (Should occur midway between system integration and 

demonstration): Identify 

critical manufacturing processes and establish a plan to bring these 

under statistical control by the start of production; also establish 

reliability goals and a growth plan to achieve these by production. 

This facilitates the achievement of process control and reliability 

goals at the completion of knowledge point 3.

Knowledge Point 2 (Should occur midway between system integration and 

demonstration): When do you 

know you have achieved this knowledge point? When 90 percent of 

engineering drawings are releasable to manufacturing organizations. 

Drawings are the language used by engineers to communicate to the 

manufacturers the details of the new product--what it looks like, how 

its components interface, how to build it and the critical materials 

and processes needed to fabricate it. This makes drawings a key measure 

of whether the design is stable or not.

Knowledge Point 3 (Should occur before production): Demonstrate 

that all critical manufacturing processes are under statistical control 

and consistently producing items within the quality standards and 

tolerances for the overall product before production begins. This is 

important, since variation in one process can reverberate to others and 

result in defective parts that need to be repaired or reworked.

Knowledge Point 3 (Should occur before production): Demonstrate 

product reliability before the start of production. This requires 

testing to identify the problems, design corrections, and retest the 

new design. Commercial firms consider reliability important and its 

achievement a measure of design maturity.

Knowledge Point 3 (Should occur before production): When do you 

know you have achieved this knowledge point? When all key manufacturing 

processes have come under statistical control and product reliability 

has been demonstrated.

[End of table]

DOD programs often do not employ these practices. We found that if the 

evolutionary, knowledge-based acquisition concepts were not applied, a 

cascade of negative effects became magnified in the product development 

and production phases of an acquisition program. These led to 

acquisition outcomes that included significant cost increases and 

schedule delays, poor product quality and reliability, and delays in 

getting new capability to the war fighter. This is often the case in 

DOD programs as shown in our past work on systems like F/A-22 fighter, 

C-17 airlifter, V-22 tiltrotor aircraft, PAC-3 missile, BAT antitank 

munition, and others. We did find some DOD programs that employed best 

practice concepts and have had more successful program outcomes to 

date. These included the Global Hawk unmanned vehicle, AIM-9X missile, 

and Joint Direct Attack Munitions guided bomb. Figure 3 shows a 

notional illustration of the different paths and effects of a product 

development.

Figure 2: Different Paths That A Products Development Can Take:

[See PDF for image]

[End of figure]

It is clear that knowledge about the products technology, design, and 

processes captured at the right time can reduce development cycle times 

and deliver a more cost effective, reliable product to the customer 

sooner than programs that do not capture this knowledge.

In applying the knowledge-based approach, the most leveraged decision 

point of the three, is matching the customers needs with the 

developers resources--technology, design, timing, and funding. This 

initial decision sets the stage for the eventual outcome--desirable or 

problematic. The match is ultimately achieved in every development 

program, but in successful development programs, it occurs prior to 

program launch. In successful programs, negotiations and trade-offs 

occur before a product development is launched to ensure that a match 

exists between customer expectations and developer resources. The 

results achieved from this match are balanced and achievable 

requirements, sufficient investment to complete the development, and a 

firm commitment to deliver the product. Commercial companies we have 

visited usually limit product development cycle-time to less than 5 

years.

In DOD, this match is seldom achieved. It is not unusual for DOD to 

bypass early trade-offs and negotiations, instead planning to develop a 

product based on a rigid set of requirements that are unachievable 

within a reasonable development time frame. This results in cost and 

schedule commitments that are unrealistic. Although a program can take 

as long as 15 years in DOD, the program manager is expected to develop 

and be accountable for precise cost and schedule estimates made at the 

start of the program. Because of their short tenures, it normally takes 

several program managers to complete product development. Consequently, 

the program manager that commits to the cost and schedule estimate at 

the beginning of the program is not the same person responsible for 

achieving it. Therefore, program accountability is problematic. 

Ironically, this outcome is rational in the traditional acquisition 

environment. The pressures put on program managers to get programs 

approved encourage promising more than can be delivered for the time 

and money allotted. They are not put in a position to succeed.

The differences in the practices employed by successful commercial 

firms and DOD reflect the different demands imposed on programs by the 

environments in which they are managed. Specific practices take root 

and are sustained because they help a program succeed in its 

environment. The way success and failure are defined for commercial and 

defense product developments differs considerably, which creates a 

different set of incentives and evokes different behaviors from 

managers. Attempts at reforming weapon system acquisitions have not 

succeeded because they did not change these incentives. All of the 

participants in the acquisition process play a part in creating 

incentives. The F/A-22 program, advertised as a flagship of acquisition 

reform in its early days, failed to establish this match before program 

launch and today we are discussing the resulting outcomes to-date.

F/A-22 Did Not Employ Evolutionary or Knowledge-Based Process:

The F/A-22 provides an excellent example of what can happen when a 

major acquisition program is not guided by the principles of 

evolutionary, knowledge-based acquisition. The program failed to match 

requirements with resources and make early trade-offs and took on a 

number of new and unproven technologies. Instead of fielding early 

capability and then evolving the product to get new capabilities to the 

war fighter sooner, the Air Force chose a big bang product 

development approach that is now planned to take about 19 years. This 

created a challenging and risky acquisition environment that has 

delayed the war fighter the capabilities expected from this new 

aircraft. Program leaders did not capture the specific knowledge 

identified as key for each of the three critical knowledge points in 

product development. Instead, program managers proceeded through the F/

A-22s development without the requisite knowledge necessary for 

reducing program risk and achieving more successful program outcomes. 

Now the optimism underlying these decisions has resulted in significant 

cost increases, schedule delays, trade-offs--making do with less than 

half the number of originally desired aircraft--and concerns about the 

capability to be delivered.

F/A-22 Program Outcomes:

Since the F/A-22 acquisition program was started in October 1986, the 

F/A-22 cost and schedule estimates have grown significantly to where, 

today, the Air Force estimates the total acquisition unit cost of a 

single aircraft is $257.5 million.[Footnote 1] This represents a 74 

percent increase from the estimate at the start of development and a 

commensurate loss in the buying power of the defense dollar. Intended 

to replace the aging F-15 fighter, the F/A-22 program is now scheduled 

to reach its initial operational capability in December 2005--making 

its development cycle about 19 years. During this cycle, the planned 

buy quantity has been reduced 63 percent from 750 to 276 

aircraft[Footnote 2]. In addition, since fiscal year 2001, funding for 

F/A-22 upgrades has dramatically increased from $166 million to $3.0 

billion, most of which is to provide increased ground attack 

capability, a requirement that was added late in the development 

program.

F/A-22 Did Not Use Evolutionary Acquisition or Capture Knowledge 

Required at Key Decision Junctures:

The F/A-22 acquisition strategy from the outset was to achieve full 

capability in a big bang approach. By not using an evolutionary 

approach, the F/A-22 took on significant risk and onerous technological 

challenges. While the big bang approach may have allowed the Air Force 

to more successfully compete for early funding, it hamstrung the 

program with many new undemonstrated technologies, preventing the 

program from knowing cost and schedule ramifications throughout 

development. Cost, schedule, and performance problems resulted. The 

following table summarizes the F/A-22 programs attainment of critical 

knowledge and key decision junctures during the development program and 

the changes in development cost and cycle time at each point.

Table 2: Knowledge Attainment in the F/A-22 Program:

Best practice; Program start--1986: Attain knowledge point 1. Separate 

technology and product development, deliver mature technology, and have 

preliminary design.; Design review--1995: Attain knowledge point 2. 90 

percent of systems and structures engineering drawings releasable and 

subsystem design reviews completed.; Production start--2001: Attain 

knowledge point 3. 100% of critical manufacturing processes in 

statistical control and reliability goals demonstrated.

F/A-22 practice; Program start--1986: Knowledge point 1 not attained. 

Failed to separate technology and product development. Three critical 

technologies immature: Low-observable materials, propulsion, and 

integrated avionics. Knowledge point 1 not attained until September 

2000.; Design review--1995: Knowledge points 1 and 2 not attained. Only 

26 percent of drawings released at the critical design review in 

February 1995. Knowledge point 2 not attained until September 1998, 

after delivery of second test aircraft.; Production start--2001: 

Knowledge point 3 not attained. Less than 50 percent of critical 

manufacturing processes in control. Only 22 percent of reliability goal 

demonstrated with many outstanding deficiencies.

F/A-22s estimated development cost[A]; Program start--1986 $12.6 

billion; Design review--1995: $21.2 billion; (68 percent increase); 

Production start--2001: $28.7 billion; (128 percent increase).

Estimated development cycle time; Program start--1986: 9.4 years; 

Design review--1995: 18.1 years; (93 percent increase); Production 

start--2001: 19.2 years; (104 percent increase).

[A] The development estimate includes all F/A-22 RDT&E costs.

[End of table]:

Technology--The F/A-22 did not have mature technology at the start of 

the acquisition program. The program included new low-observable 

(stealth) materials, integrated avionics, and propulsion technology 

that were not mature at this time. The Air Force did not complete an 

evaluation of stealth technology on a full-scale model of the aircraft 

until several years into development. It was not until September 2000, 

or 9 years into development, that the integrated avionics reached a 

maturity level acceptable to begin product development. During 

development, the integrated avionics was a source of schedule delays 

and cost growth. Since 1997, avionics software development and flight-

testing have been delayed, and the cost of avionics development has 

increased by over $980 million dollars. Today, the avionics still has 

problems affecting the ability to complete developmental testing and 

begin operational testing, and the Air Force cannot predict when a 

solution will be found.

Design--The effects of immature technologies cascaded into the F/A-22 

development program, making it more difficult to achieve a stable 

design at the right time. The standard measure of design stability is 

90 percent of design drawings releasable by the critical design review. 

The F/A-22 achieved only 26 percent by this review, taking an 

additional 43 months to achieve the standard. Moving ahead in 

development, the program experienced several design and manufacturing 

problems described by the F/A-22 program office as a rolling wave 

effect throughout system integration and final assembly. These effects 

included numerous design changes, labor inefficiencies, parts 

shortages, out of sequence work, cost increases, and schedule delays.

Production--At the start of production, the F/A-22 did not have 

manufacturing processes under control and was only beginning testing 

and demonstration efforts for system reliability. Initially, the F/A-22 

had taken steps to use statistical process control data to gain control 

of critical manufacturing processes by full rate production. However, 

the program abandoned this best practice approach in 2000 with less 

than 50 percent of its critical manufacturing processes in control. In 

March 2002,[Footnote 3] we recommended that the F/A-22 program office 

monitor the status of critical manufacturing process as the program 

proceeds toward high rate production.

The reliability goal for the F/A-22 is 3 hours of flying time between 

maintenance actions. The Air Force estimated that in late 2001, when it 

entered production, it should have been able to demonstrate almost 2 

flying hours between maintenance actions. Instead, it could fly an 

average of only 0.44 hours between maintenance actions. Since then 

there has been a decrease in reliability. As of November 2002, 

development test aircraft have been completing only 0.29 hours between 

maintenance actions. Additionally, the program was slow to fix and 

correct problems that had affected reliability. At the time of our 

review in July 2002, program officials had identified about 260 

different types of failures and had identified fixes for less than 50 

percent of the failures. To achieve reliability goals will require 

additional design changes, testing, and modifications. Therefore, 

additional problems and costs can be expected if the system is fielded 

with the level of reliability achieved to date.

It Is Too Late for the F/A-22 Program to Gain Full Benefit of a 

Knowledge-Based Process:

The F/A-22 did not take advantage of evolutionary, knowledge-based 

concepts up front and now, the best it can hope for is to limit cost 

increases and performance problems by not significantly increasing its 

production until development is complete--signified by developmental 

and operational testing and reliability demonstrations. To that end, we 

have recommended that the Air Force reconsider its decision to increase 

the aircraft production rate beyond 16 aircraft per year.[Footnote 4] 

The program is nearing the end of developmental testing and plans to 

start initial operational testing in October 2003. If developmental 

testing goes as planned, which is not guaranteed, operational testing 

is expected to be completed around September 2004. By the end of this 

fiscal year, 51 F/A-22s will be on contract as low rate production 

began in 2001.

Our March 2003 report identifies various problems still outstanding 

that could have further impacts on cost, schedule, and delivered 

performance that are in addition to undemonstrated reliability goals. 

The problems identified are of particular concern, given Air Force 

plans to increase production rates and make a full rate production 

decision in 2004. The problems include:

* unexpected shutdowns (instability) of the avionics,

* excessive movement of the vertical tails,

* overheating in rear portions of the aircraft,

* separations of the horizontal tail material,

* inability to meet airlift support requirements, and:

* excessive ground maintenance actions.

:

These problems are still being addressed, and not all of them have been 

solved as yet. For example, Air Force officials stated they do not yet 

understand the problems associated with the avionics instability well 

enough to predict when they would be able to resolve them, and certain 

tests to better understand the vertical tail problem have not yet 

begun. Despite remaining testing and outstanding problems, the Air 

Force plans to continue acquiring production aircraft at increasing 

annual rates and make the full rate production decision in 2004. This 

is a very risky strategy, given outstanding issues in the test program 

and the systems less than expected reliability. The Air Force may 

encounter higher production costs as a result of acquiring significant 

quantities of aircraft before adequate testing and demonstrations are 

complete. In addition, remaining testing could identify problems that 

require costly modifications in order to achieve satisfactory 

performance.

In a February 28, 2003 report to Representative John Tierney,[Footnote 

5] we found that F/A-22 production costs are likely to increase more 

than the latest $5.4 billion cost growth recently estimated by the Air 

Force and the Office of Secretary of Defense (OSD). First, the current 

OSD production estimate does not include $1.3 billion included in the 

latest Air Force acquisition plan. Second, schedule delays in 

developmental testing could further postpone the start of the first F/

A-22 multiyear contract, which has already been delayed until fiscal 

year 2006. This could result in lower cost savings from multiyear 

procurement. Last, we found several risk factors that may increase 

future production costs, including the dependency of certain cost 

reduction plans on Air Force investments that are not being made to 

improve production processes, the availability of funding, and a 

reduction in funding for support costs. In addition, DOD has not 

informed Congress about the quantity of aircraft that can be procured 

within existing production cost limits, which we believe could be fewer 

than the 276 currently planned. Further details on F/A-22 cost growth 

and the Air Forces attempt to offset it are provided in appendix I.

Real Change in Acquisition Outcomes Requires Disciplined Enforcement of 

Acquisition Policy:

While DODs new acquisition policy is too late to influence the F/A-22 

program, it is not too late for other major acquisition programs like 

the Missile Defense Agencys suite of land, sea, air, and space defense 

systems; the Armys Future Combat Systems; and the Air Force and Navys 

Joint Strike Fighter. DODs revised acquisition policy represents 

tangible leadership action to getting better weapon system acquisition 

outcomes, but unless the policies are implemented through decisions on 

individual programs, outcomes are not likely to change. Further, unless 

pressures are alleviated in DOD to get new acquisition programs 

approved and funded on the basis of requirements that must stand out, 

programs will continue to be compromised from the outset with little to 

no chance of successful outcomes. If new policies were implemented 

properly, through decisions on individual programs, managers would face 

less pressure to promise delivery of all the ultimate capabilities of a 

weapon system in one big bang.:

Both form and substance are essential to getting desired outcomes. At a 

tactical level, we believe that the policies could be made more 

explicit in several areas to facilitate such decisions. First, the 

regulations provide little or no controls at key decision points of an 

acquisition program that force a program manager to report progress 

against knowledge-based metrics. Second, the new regulations, once 

approved, may be too general and may no longer provide mandatory 

procedures. Third, the new regulations may not provide adequate 

accountability because they may not require knowledge-based 

deliverables containing evidence of knowledge at key decision points.

At a strategic level, some cultural changes will be necessary to 

translate policy into action. At the very top level, this means DOD 

leadership will have to take control of the investment dollars and to 

say no in some circumstances if programs are inappropriately 

deviating from sound acquisition policy. In my opinion, programs should 

follow a knowledge-based acquisition policy--one that embraces best 

practices--unless there is a clear and compelling national security 

reason not to. Other cultural changes instrumental to implementing 

change include:

* Keeping key people in place long enough so that they can affect 

decisions and be held accountable.

* Providing program offices with the skilled people needed to craft 

acquisition approaches that implement policy and to effectively oversee 

the execution of programs by contractors.

* Realigning responsibilities and funding between science and 

technology organizations and acquisition organizations to enable the 

separation of technology development from product development.

* Bringing discipline to the requirements-setting process by demanding 

a match between requirements and resources.

* Requiring readiness and operating cost as key performance parameters 

prior to beginning an acquisition.

* Designing and implementing test programs that deliver knowledge when 

needed, including reliability testing early in design.

:

Ultimately, the success of the new acquisition policy will be seen in 

individual program and resource decisions. Programs that are 

implementing knowledge-based policies in their acquisition approaches 

should be supported and resourced, presuming they remain critical to 

national needs and affordable within current and projected resource 

levels. Conversely, if programs that repeat the approaches of the past 

are approved and funded, past policies--and their outcomes--will be 

reinforced with a number of adverse implications.

Conclusions:

DOD will continue to face challenges in modernizing its forces with new 

demands on the federal dollar created by changing world conditions. 

Consequently, it is incumbent upon DOD to find and adopt best product 

development practices that can allow it to manage its weapon system 

programs in the most efficient and effective way. Success over the long 

term will depend not only on policies that embrace evolutionary, 

knowledge-based acquisition practices but also on DOD leaderships 

sustaining its commitment to improving business practices and ensuring 

that those adopted are followed and enforced.

DODs new acquisition policy embraces the best practice concepts of 

knowledge-based, evolutionary acquisition and represents a good first 

step toward achieving better outcomes from major acquisition programs. 

The F/A-22 program followed a different path at its beginning, a big 

bang, high-risk approach whose outcomes so far have been increased 

cost, quality and reliability problems, growing procurement reductions, 

and delays in getting the aircraft to the war fighter. Since this 

program is nearing the end of development and already into production, 

it is too late to adopt a knowledge approach, but it can limit further 

cost increases and adverse actions by not ramping up production beyond 

current levels until developmental and operational testing are 

completed and reliability goals have been demonstrated. Regardless of 

the F/A-22s current predicament, the new policy can and should be used 

to manage all new acquisition programs and should be adapted to those 

existing programs that have not progressed too far in development to 

benefit. At a minimum, the F/A-22 should serve as a lesson learned from 

which to effect a change in the future DOD acquisition environment. The 

costs of doing otherwise are simply too high for us to tolerate.

Mr. Chairman, this concludes my prepared statement. I would be happy to 

respond to any questions that you or other members of the Subcommittee 

may have.

[End of section]

Appendix I: F/A-22 Production Cost Growth:

Over the last 6 years, DOD has identified about $18 billion in 

estimated production cost growth during the course of two DOD program 

reviews. As a result, the estimated cost of the production program 

currently exceeds the congressional cost limit. The Air Force has 

implemented cost reduction plans designed to offset a significant 

amount of this estimated cost growth. But the effectiveness of these 

cost reduction plans has varied.

During a 1997 review, the Air Force estimated cost growth of $13.1 

billion.[Footnote 6] The major contributing factors to this cost growth 

were inflation, increased estimates of labor costs and materials 

associated with the airframe and engine, and engineering changes to the 

airframe and engine. These factors made up about 75 percent of the cost 

growth identified in 1997.

In August 2001, DOD estimated an additional $5.4 billion in cost growth 

for the production of the F/A-22, bringing total estimated production 

cost to $43 billion. The major contributing factors to this cost growth 

were again due to increased labor costs and airframe and engine costs. 

These factors totaled almost 70 percent of the cost growth. According 

to program officials, major contractors and suppliers inability to 

achieve the expected reductions in labor costs throughout the building 

of the development and early production aircraft has been the primary 

reason for estimating this additional cost growth.

Mixed Success With Cost Reduction Plans:

The Air Force was able to implement cost reduction plans and offset 

cost growth by nearly $2 billion in the first four production contracts 

awarded. As shown in table 3, the total offsets for these contracts 

slightly exceeded earlier projections by about $.5 million.

Table 3: Comparison of Planned Versus Implemented Cost Reduction 

Offsets for Awarded Production Contracts:

Dollars in millions.

Fiscal year 1999 (2 aircraft); Planned offsets: $199.0; Implemented 

offsets: $200.5; Difference: $1.5.

Fiscal year 2000 (6 aircraft); Planned offsets: 329.3; Implemented 

offsets: 336.4; Difference: 7.1.

Fiscal year 2001 (10 aircraft); Planned offsets: 580.2; Implemented 

offsets: 611.1; Difference: 30.9.

Fiscal year 2002 (13 aircraft); Planned offsets: 827.2; Implemented 

offsets: 788.2; Difference: (39.0).

Total; Planned offsets: $1,935.7; Implemented offsets: $1,936.2; 

Difference: $0.5.

Source: Air Force.

[End of table]

Cost reduction plans exist but have not yet been implemented for 

subsequent production lots planned for fiscal years 2003 through 2010 

because contracts for these production lots have not yet been awarded. 

If implemented successfully, the Air Force expects these cost reduction 

plans to achieve billions of dollars in offsets to estimated cost 

growth and to allow the production program to be completed within the 

current production cost estimate of $43 billion.[Footnote 7] However, 

this amount exceeds the production cost limit of $36.8 billion.

In addition, while the Air Force has been attempting to offset costs 

through production improvement programs (PIPs), recent funding cutbacks 

for PIPs may reduce their effectiveness. PIPs focus specifically on 

improving production processes to realize savings by using an initial 

government investment. The earlier the Air Force implements PIPs, the 

greater the impact on the cost of production. Examples of PIPs 

previously implemented by the Air Force include manufacturing process 

improvements for avionics, improvements in fabrication and assembly 

processes for the airframe, and redesign of several components to 

enable lower production costs.

As shown in figure 3, the Air Force reduced the funding available for 

investment in PIPs by $61 million for lot 1 and $26 million for lot 2 

to cover cost growth in production lots 1 and 2[Footnote 8]. As a 

result, it is unlikely that PIPs covering these two lots will be able 

to offset cost growth as planned.

Figure 3: Planned Versus Actual F/A-22 Production Improvement Program 

Investment for Production Lots 1 (Fiscal Year 2001) and 2 (Fiscal Year 

2002):

[See PDF for image]

[End of figure]

Figure 4 shows the remaining planned investment in PIPs through fiscal 

year 2006 and the $3.7 billion in estimated cost growth that can 

potentially be offset through fiscal year 2010 if the Air Force invests 

as planned in these PIPs.

Figure 4: Planned Offsets to Cost Growth From Investing in and 

Implementing PIPs:

[See PDF for image]

[End of figure]

In the past, Congress has been concerned about the Air Forces practice 

of requesting fiscal year funding for these PIPs but then using part of 

that funding for F/A-22 airframe cost increases. [Footnote 9] Recently, 

Congress directed the Air Force to submit a request if it plans to use 

PIP funds for an alternate purpose.

[End of section]

Related GAO Products:

Best Practices: Setting Requirements Differently Could Reduce Weapon 

Systems Total Ownership Costs. GAO-03-57. Washington, D.C.: February 

11, 2003.

Best Practices: Capturing Design and Manufacturing Knowledge Early 

Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15, 

2002.

Defense Acquisitions: DOD Faces Challenges in Implementing Best 

Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.

Best Practices: Better Matching of Needs and Resources Will Lead to 

Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8, 

2001.

Best Practices: A More Constructive Test Approach Is Key to Better 

Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31, 

2000.

Defense Acquisitions: Employing Best Practices Can Shape Better Weapon 

System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000.

Best Practices: DOD Training Can Do More to Help Weapon System Programs 

Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August 

16, 1999.

Best Practices: Better Management of Technology Development Can Improve 

Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30, 

1999.

Best Practices: DOD Can Help Suppliers Contribute More to Weapon System 

Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998.

Best Practices: Successful Application to Weapon Acquisition Requires 

Changes in DODs Environment. GAO/NSIAD-98-56. Washington, D.C.: 

February 24, 1998.

Best Practices: Commercial Quality Assurance Practices Offer 

Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26, 

1996.

Tactical Aircraft: Status of the F/A-22 Program. GAO-03-603T. 

Washington, D.C.: April 2, 2003.

Tactical Aircraft: DOD Should Reconsider Decision to Increase F/A-22 

Production Rates While Development Risks Continue. GAO-03-431. 

Washington, D.C.: March 14, 2003.

Tactical Aircraft: DOD Needs to Better Inform Congress about 

Implications of Continuing F/A-22 Cost Growth. GAO-03-280. Washington, 

D.C.: February 28, 2003.

Tactical Aircraft: F-22 Delays Indicate Initial Production Rates Should 

Be Lower to Reduce Risks. GAO-02-298. Washington, D.C.: March 5, 2002.

Tactical Aircraft: Continuing Difficulty Keeping F-22 Production Costs 

within the Congressional Limitation. GAO-01-782. Washington, D.C.: July 

16, 2001.

Tactical Aircraft: F-22 Development and Testing Delays Indicate Need 

for Low-Rate Production. GAO-01-310. Washington, D.C.: March 15, 2001.

Defense Acquisitions: Recent F-22 Production Cost Estimates Exceeded 

Congressional Limitation. GAO/NSIAD-00-178. Washington, D.C.: August 

15, 2000.

Defense Acquisitions: Use of Cost Reduction Plans in Estimating F-22 

Total Production Costs. GAO/T-NSIAD-00-200. Washington, D.C.: June 15, 

2000.

Budget Issues: Budgetary Implications of Selected GAO Work for Fiscal 

Year 2001. GAO/OCG-00-8. Washington, D.C.: March 31, 2000.

F-22 Aircraft: Development Cost Goal Achievable If Major Problems Are 

Avoided. GAO/NSIAD-00-68. Washington, D.C.: March 14, 2000.

Defense Acquisitions: Progress in Meeting F-22 Cost and Schedule Goals. 

GAO/T-NSIAD-00-58. Washington, D.C.: December 7, 1999.

Fiscal Year 2000 Budget: DODs Procurement and RDT&E Programs. GAO/

NSIAD-99-233R. Washington D.C.: September 23, 1999.

Budget Issues: Budgetary Implications of Selected GAO Work for Fiscal 

Year 2000. GAO/OCG-99-26. Washington, D.C.: April 16, 1999.

Defense Acquisitions: Progress of the F-22 and F/A-18E/F Engineering 

and Manufacturing Development Programs. GAO/T-NSIAD-99-113. 

Washington, D.C.: March 17, 1999.

F-22 Aircraft: Issues in Achieving Engineering and Manufacturing 

Development Goals. GAO/NSIAD-99-55. Washington, D.C.: March 15, 1999.

F-22 Aircraft: Progress of the Engineering and Manufacturing 

Development Program. GAO/T-NSIAD-98-137. Washington D.C.: March 25, 

1998.

F-22 Aircraft: Progress in Achieving Engineering and Manufacturing 

Development Goals. GAO/NSIAD-98-67. Washington, D.C.: March 10, 1998.

Tactical Aircraft: Restructuring of the Air Force F-22 Fighter Program. 

GAO/NSIAD-97-156. Washington, D.C.: June 4, 1997.

Defense Aircraft Investments: Major Program Commitments Based on 

Optimistic Budget Projections. GAO/T-NSIAD-97-103. Washington, D.C.: 

March 5, 1997.

F-22 Restructuring. GAO/NSIAD-97-100BR. Washington, D.C.: February 28, 

1997.

Tactical Aircraft: Concurrency in Development and Production of F-22 

Aircraft Should Be Reduced. GAO/NSIAD-95-59. Washington, D.C.: April 

19, 1995.

Tactical Aircraft: F-15 Replacement Issues. GAO/T-NSIAD-94-176. 

Washington, D.C.: May 5, 1994.

Tactical Aircraft: F-15 Replacement Is Premature as Currently Planned. 

GAO/NSIAD-94-118. Washington, D.C.: March 25, 1994.

FOOTNOTES

[1] All references to F/A-22 costs in this testimony are in then-year 

dollars in order to maintain consistent reporting with our prior 

reports on the F/A-22. 

[2] Between 1986 and the start of engineering and manufacturing 

development in 1991, the quantity was reduced from 750 to 648 aircraft.

[3] U.S. General Accounting Office, Tactical Aircraft: F-22 Delays 

Indicate Initial Production Rates Should Be Lower to Reduce Risks, 

GAO-02-298 (Washington, D.C.: Mar. 5, 2002).

[4] U.S. General Accounting Office, Tactical Aircraft: DOD Should 

Reconsider Decision to Increase F/A-22 Production Rates While 

Development Risks Continue, GAO-03-431 (Washington, D.C.: Mar. 14, 

2003).

[5] U.S. General Accounting Office, Tactical Aircraft: DOD Needs to 

Better Inform Congress about Implications of Continuing F/A-22 Cost 

Growth, GAO-03-280 (Washington, D.C.: Feb. 28, 2003).

[6] Based on a plan to procure 438 aircraft.

[7] The F/A-22 Presidents budget for fiscal year 2004 would transfer 

$876 million in production funding to help fund estimated cost 

increases in development. As a result, the current production cost 

estimate is $42.2 billion.

[8] Production lot 1 was awarded in fiscal year 2001 and production lot 

2 was awarded in fiscal year 2002.

[9] Report 107-298, Nov. 19, 2001.