GAO

United States General Accounting Office

Report to the Chairman, Subcommittee

on National Security, Veterans’ Affairs,

and International Relations, Committee

on Government Reform, House of

Representatives

May 2000 JOINT STRIKE

FIGHTER

ACQUISITION

Development

Schedule Should Be

Changed to Reduce

Risks

GAO/NSIAD-00-74

Page 1 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

Contents

Letter 3

Appendixes Appendix I: International Participation in the Joint Strike Fighter

Program 24

Appendix II: Technology Readiness Levels and Their Definitions 25

Appendix III: Comments From the Department of Defense 26

Tables Table 1: Military Service Needs for the Joint Strike Fighter 6

Figures Figure 1: Boeing and Lockheed Martin Joint Strike Fighter Aircraft

Design Concepts 7

Figure 2: Comparison of Traditional and Joint Strike Fighter

Acquisition Cycles 8

Figure 3: Joint Strike Fighter Critical Technology Readiness Levels

at Program Start 13

Figure 4: Joint Strike Fighter Critical Technology Readiness Levels

at Program Start and Projected for Entry Into Engineering

and Manufacturing Development 14

Abbreviations

DOD Department of Defense

JSF Joint Strike Fighter

TRL technology readiness level
Page 2 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition
Page 3 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

United States General Accounting Office

Washington, D.C. 20548

National Security and

International Affairs Division

B-281196 Letr

May 9, 2000

The Honorable Christopher Shays

Chairman, Subcommittee on National Security,

Veterans’ Affairs, and International Relations

Committee on Government Reform

House of Representatives

Dear Mr. Chairman:

The Joint Strike Fighter Program is intended to produce an affordable,

next-generation aircraft to replace the Department of Defense’s (DOD)

aging aircraft inventory. The first aircraft deliveries are scheduled to begin

in 2008. As currently planned, the program will cost about $200 billion to

develop and procure over 3,000 aircraft and related support equipment for

the Air Force, the Marine Corps, the Navy, and Great Britain.

DOD has designated the Joint Strike Fighter Program as a flagship program

for acquisition. To date, the program has awarded contracts totaling over

$2 billion to Boeing and Lockheed Martin for the current concept

demonstration phase. Under these contracts, both contractors will build

the aircraft they plan to fly in the demonstration phase and also design the

aircraft they plan to build in the next phase of the development program—

engineering and manufacturing development. During engineering and

manufacturing development, the Joint Strike Fighter will be fully

developed, engineered, designed, fabricated, tested, and evaluated to

demonstrate that the production aircraft will meet stated requirements.

DOD is scheduled to award the contract for engineering and manufacturing

development to either Boeing or Lockheed Martin in April 2001.1

At your request, we reviewed the Joint Strike Fighter Program to

(1) provide information on the acquisition strategy and (2) to determine

whether the strategy is being implemented in a manner that will ensure that

the acquisition strategy objectives will be achieved. With your permission,

we discussed a draft of this report during a March 16, 2000, joint hearing by

1 At the time of this report, the Under Secretary of Defense for Acquisition, Technology, and

Logistics was reviewing competition and industrial-base implications of the Joint Strike

Fighter acquisition strategy.
B-281196

Page 4 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

the Subcommittees on Military Procurement and on Military Research and

Development, House Committee on Armed Services. At the time of the

hearing, we had not received DOD’s comments on our report. This report

contains DOD’s comments and our evaluation of them.

Results in Brief The key objective of the Joint Strike Fighter acquisition strategy is

affordability—reducing the development, production, and ownership costs

of the program relative to prior fighter aircraft programs. DOD expects the

Joint Strike Fighter acquisition strategy to save nearly $18 billion (in fiscal

year 1995 dollars) in development costs. To achieve its affordability

objective, the Joint Strike Fighter program office has incorporated various

DOD and commercial acquisition initiatives into the Joint Strike Fighter

acquisition strategy. These initiatives include modifying the traditional

weapons acquisition cycle, revising the requirements determination

process, and developing critical technologies to a level where they

represent low technical risk before the engineering and manufacturing

contract is awarded. The expectation is that incorporating these initiatives

into the Joint Strike Fighter acquisition strategy will result in a better

match between the maturity2 of key technologies and the aircraft’s

requirements. Matching the requirements and the maturity of technology

when a program enters engineering and manufacturing development is a

critical determinant of a program’s success. Once the development phase

begins, a large, fixed investment in the form of human capital, facilities, and

materials is sunk into the program and any significant changes will have a

large, rippling effect on cost and schedule. Beginning the engineering and

manufacturing development phase when critical technologies are at a low

level of maturity serves to significantly increase program risk and the

likelihood of schedule delays, which in turn result in increased program

costs.

The Joint Strike Fighter program office’s implementation of its acquisition

strategy will not ensure that the Joint Strike Fighter program will enter the

engineering and manufacturing development phase with low technical risk.

The aircraft being produced during the concept demonstration phase are

not intended to demonstrate many of the technologies considered critical

for achieving Joint Strike Fighter program cost and performance

requirements. Instead, many of these technologies—such as avionics, flight

2 A technology is considered to be mature when it has been developed to a point that it can

be readily integrated into a new product and counted on to meet product requirements.
B-281196

Page 5 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

systems, manufacturing and producibility, propulsion, supportability, and

weapons delivery system—will only be demonstrated in laboratory or

ground-testing environments. Therefore, these critical technologies will be

at low levels of technical maturity when the engineering and manufacturing

development contract is scheduled to be awarded. In addition, when the

competing contractors experienced design problems and cost overruns,

DOD restructured the program in a manner that will provide less

information than originally planned prior to selecting between the two

competing contractors. Specifically, this program restructure moves away

from best commercial practices that were evident in the original strategy,

where technology was being developed ahead of the product. Instead,

DOD’s approach moves toward the traditional practice of concurrently

developing technologies and products, which often raised cost-benefit

issues as a result of cost increases and schedule delays as problems are

encountered in technology development.

To demonstrate DOD’s commitment to acquisition reform, follow best

commercial practices, and reduce the risk of future cost growth, the

program office should focus on risk reduction efforts by maturing critical

technologies prior to entering engineering and manufacturing

development, and it should be allowed to do so without the penalty of

withdrawal of funding support. We make a recommendation that the Joint

Strike Fighter program office adjust its currently planned engineering and

manufacturing development decision date of March 2001 to allow adequate

time to mature critical technologies to acceptable maturity levels before

awarding the engineering and manufacturing development contract.

Background The Joint Strike Fighter is the centerpiece of DOD’s tactical aircraft

modernization plan, which includes the Air Force F-22 Raptor and the Navy

F/A-18 E/F Super Hornet. The program is structured to use a common

production line to produce three aircraft variants that meet conventional

flight requirements for the U.S. Air Force, short take-off and vertical

landing characteristics for the Marine Corps, and carrier operation

suitability needs for the U.S. Navy. The program will also provide aircraft to

the British Royal Navy and Air Force. Table 1 shows current service plans

for Joint Strike Fighter use.
B-281196

Page 6 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

Table 1: Military Service Needs for the Joint Strike Fighter

Source: Joint Strike Fighter program office.

DOD expects the Air Force variant of the Joint Strike Fighter to cost about

$28 million per unit; the Navy variant to be between $31 million and

$38 million; and the Marine Corps variant to cost between $30 million and

$35 million.3 Independent estimates are not so optimistic. For example, in

congressional hearings held in March 1999, the Congressional Budget

Office estimated that the unit cost of the Joint Strike Fighter could be as

much as 47 percent to 51 percent higher than expected, depending on

which variant was procured. DOD and the Congressional Budget Office

estimates vary as a result of differing estimating techniques, including

estimating the cost of incorporating stealth technologies into the Joint

Strike Fighter design. Figure 1 shows planned Joint Strike Fighter aircraft

designs by contractor.

Service Quantity Planned use

Air Force 1,763 Replacement for F-16 and A-10; complement to the F-22

Marine Corps 609 Short take-off and vertical landing aircraft to replace AV-8B

and F/A-18 C/D

Navy 480 Carrier-based, multi-role, first day of war survivable strike

fighter to complement the F/A-18 E/F

Great Britain 150 Short take-off and vertical landing replacement for the Sea

Harrier and GR.7

3 Expected costs include the cost to produce the basic aircraft, propulsion system, and

avionics. Costs are stated in fiscal year 1994 dollars.
B-281196

Page 7 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

Figure 1: Boeing and Lockheed Martin Joint Strike Fighter Aircraft Design Concepts

Source: Joint Strike Fighter program office.

Acquisition Strategy

Designed to Achieve

Affordability Goals

Through Reduced

Program Risk

The focus of the Joint Strike Fighter Program is affordability—reducing the

development, production, and ownership costs of the program relative to

prior fighter aircraft programs. To achieve this objective, the Joint Strike

Fighter program office has incorporated various DOD and commercial

acquisition initiatives into the Joint Strike Fighter acquisition strategy.

These initiatives include modifying the traditional weapons acquisition

cycle, revising the requirements determination process, and advancing the

maturity level of critical technologies so they represent low technical risk

before the engineering and manufacturing contract is awarded. The

expectation is that incorporating these initiatives into the Joint Strike

Fighter acquisition strategy will avoid cost growth, schedule slippage, and

performance shortfalls that have been experienced in other weapons

acquisition programs.

Acquisition Strategy

Modifies Traditional

Acquisition Cycle

The Joint Strike Fighter acquisition strategy modifies the cycle that

weapons acquisition programs normally follow. For example, the Joint

Strike Fighter program office was created earlier in the acquisition cycle

than it would have been in a traditional DOD weapons system program.

This enabled DOD to obtain early input from relevant stakeholders

(operators, maintenance personnel, industry representatives, government

engineers, and officials within the intelligence community) to ensure that

all aspects of cost, schedule, performance, and resource constraints are

included in decision-making. In addition, the program has encouraged

 oeing Joint Strike Fighter

 e ign oncept

 ockhee artin Joint Strike Fighter

 e ign oncept
B-281196

Page 8 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

greater involvement from the international community, which provides

both monetary and technical support (see app. I).

The traditional acquisition cycle has also been modified by combining the

first two traditional acquisition phases—Concept Exploration and Program

Definition and Risk Reduction—into one phase, known as Concept

Demonstration. Under the traditional DOD acquisition cycle, final

performance requirements are developed early in the Concept Exploration

phase (see fig. 2); in the Joint Strike Fighter program, final requirements

are determined later in the acquisition cycle. Program officials state that

this modification provides the flexibility needed to conduct cost and

performance trade-offs before requirement and design decisions become

final.

Figure 2: Comparison of Traditional and Joint Strike Fighter Acquisition Cycles

a

Joint Strike Fighter.

Source: Joint Strike Fighter program office.
B-281196

Page 9 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

Acquisition Strategy Revises

the Requirements

Determination Process

In what is known as an Operational Requirements Document, DOD defines

aircraft program requirements (such as range, speed, and acceleration)

very early in the acquisition cycle and before considering the aircraft

design necessary to meet these requirements. For the Joint Strike Fighter,

performance requirements are instead defined in five phases; in each

phase, specific aircraft design characteristics are determined as

performance requirements are set. During each phase, performance

requirements are considered in terms of the established cost targets so that

trade-offs in performance capabilities can be made as necessary.4 The most

significant trade-off to date in the Joint Strike Fighter Program was the

decision to equip the aircraft with one versus two engines. Identifying

trade-offs to balance requirements for the affordability, effectiveness, and

supportability5 of the aircraft design concept represents an ongoing effort

between the government and the two competing contractors.

Acquisition Strategy

Designed to Reduce

Technical Risk

According to the Joint Strike Fighter Single Acquisition Management Plan,

a principal objective of the program is “to demonstrate to a low level of

technical risk those critical technologies, processes, and system

characteristics necessary to produce an affordable family of strike aircraft

that meets all participants’ needs.” To achieve that objective, the Joint

Strike Fighter acquisition strategy is designed to lower technical risk

through aircraft flight demonstrations and advanced technology

development prior to awarding the engineering and manufacturing

development contract. Specifically, during the current concept

demonstration phase, DOD requires each contractor to

• demonstrate specific aircraft capabilities by designing and building

actual flying models,

• conduct demonstrations of key technologies and processes unique to

each contractor’s aircraft design, and

• submit their preferred Joint Strike Fighter design concept.

4 This process is referred to as using Cost as an Independent Variable.

5 The degree to which system design characteristics and planned logistics resources,

including manpower, meet system peacetime readiness and wartime utilization

requirements.
B-281196

Page 10 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

Each competing contractor is required to design and build two aircraft to

demonstrate the following:

• commonality/modularity to validate the contractors’ ability to produce

three aircraft variants on the same production line;

• short take-off and vertical landing, hover, and transition to forward

flight to demonstrate the aircraft’s ability to perform specific Marine

Corps and Royal Navy missions; and

• satisfactory low airspeed carrier approach flying and handling qualities

to demonstrate the aircraft’s ability to perform specific Navy missions.

Each contractor will also be required to submit a Preferred Weapon System

Concept, which outlines their preferred design concept for developing an

affordable Joint Strike Fighter aircraft to meet the goals specified in the

final requirements document. The Preferred Weapon System Concept will

include results from the flight and ground demonstrations and will

ultimately be used by DOD to select the winning aircraft design and to

award the engineering and manufacturing development contract.

Implementation of

Acquisition Strategy

Will Not Ensure That

Program Objectives

Are Achieved

Contrary to its acquisition strategy, the Joint Strike Fighter Program will

not enter the engineering and manufacturing development phase with low

technical risk. The aircraft to be used in the concept demonstration phase

are not intended to demonstrate all of the Joint Strike Fighter critical

technologies. Therefore, these technologies will be at low levels of

technical maturity when the engineering and manufacturing development

contract is scheduled to be awarded. In addition, when the competing

contractors experienced design problems and cost overruns, DOD

restructured the program in a manner that is moving away from the best

commercial practices that were evident in the original strategy and is

instead moving toward traditional practices that have caused problems on

other programs.

Critical Technologies Not

Developed to Acceptable

Levels

The aircraft being produced during the concept demonstration phase are

not intended to demonstrate many of the technologies considered critical

for achieving Joint Strike Fighter Program cost and performance

requirements, such as those for integrated avionics. Instead, many of these

technologies will be demonstrated only in laboratory or ground-testing

environments and, therefore, will be at low levels of technical maturity

when the engineering and manufacturing development contract is

scheduled to be awarded.
B-281196

Page 11 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

Matching the aircraft requirements and the maturity of technology as a

program begins is perhaps the most important determinant of a program’s

success. Once a program begins, a large, fixed investment in the form of

human capital, facilities, and materials is sunk into the program and any

significant changes will have a large, rippling effect on schedule and cost.

In the case of critical technologies, beginning an acquisition program when

the technologies are at a low level of development increases program risk

and the likelihood of schedule delays, which increases program costs.

Measuring Technology

Readiness

At our request, the Joint Strike Fighter program office identified eight

technology areas that are considered critical to meeting Joint Strike Fighter

cost and/or performance objectives. These technologies address areas such

as avionics, flight systems, manufacturing and producibility, propulsion,

supportability, and weapons delivery system.6 We requested the program

office to assign maturity levels for these critical technologies using a tool

referred to as technology readiness levels (TRLs). The TRLs were

pioneered by the National Aeronautics and Space Administration and

adopted by the Air Force Research Laboratory7 to determine the readiness

of technologies to be incorporated into a weapon or other type of system.

The Joint Advanced Strike Technology Program—from which the Joint

Strike Fighter Program evolved—used TRLs to assess early maturity levels

for many of the current Joint Strike Fighter technologies. In response to

our prior work, DOD has agreed that TRLs can be used to help guide

technology maturation and transition decisions.8 Detailed descriptions of

technology readiness levels can be found in appendix II.

In conjunction with the program office and the two competing contractors,

we determined the readiness levels of critical technologies when the Joint

Strike Fighter Program was started in 1996. That assessment showed that

when the Joint Strike Fighter Program entered the concept demonstration

phase, most of the critical technologies were well below maturity levels

6 Due to the current Joint Strike Fighter competition, the critical technologies are not

specified so as not to associate them with the respective contractors.

7 The Air Force Research Laboratory is a science and technology organization that matures

advanced technologies to the point that they can be included in weapon system programs

and be expected to perform as required. The Laboratory uses the TRLs to assess the

maturity of the technologies before they are handed off to programs.

8 Best Practices: Better Management of Technology Development Can Improve Weapon

System Outcomes (GAO/NSIAD-99-162, July 30, 1999).
B-281196

Page 12 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

considered acceptable for low risk by the Air Force Research Laboratory or

by leading commercial firms.

Readiness levels are measured on a scale of one to nine: Studies of the

basic concept have a readiness level of one; laboratory demonstrations

have a readiness level between three and six; and technologies that have

been proven through integration on the intended product have a readiness

level of nine. With this measurement tool, the program office can gauge the

likely consequences of placing various technologies at a given maturity

level into a development program and make informed choices and tradeoffs if necessary to meet program goals.

Maturity Levels of Joint Strike

Fighter Critical Technologies

The Air Force Research Laboratory considers a technology readiness level

of six an acceptable risk for a weapon system entering the program

definition stage, the point at which DOD typically begins its weapon

programs. At a lower level of technology readiness, the technology’s ability

to meet the intended product’s cost, schedule, and performance

requirements is uncertain. Reaching a maturity level of six denotes a

significant transition point for technology development—as the technology

moves from component testing in a laboratory environment to

demonstrating a model or prototype in a relevant environment. Figure 3

shows the maturity levels for the eight critical Joint Strike Fighter

technologies in November 1996, shortly after the Joint Strike Fighter

Program was started.
B-281196

Page 13 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

Figure 3: Joint Strike Fighter Critical Technology Readiness Levels at Program Start

Note: Due to the current Joint Strike Fighter competition, the critical technologies are not identified so

as not to associate them with the respective contractors.

Source: Joint Strike Fighter program office.

As shown in figure 3, none of the Joint Strike Fighter critical technologies

had achieved the desired technology readiness level of six by program

start. Instead, all of the technologies were at readiness levels of two or

three, which means that, at best, analytical and laboratory studies had been

completed or very early components had been developed.

We also obtained program office and contractor data assessing the

expected maturity levels for the Joint Strike Fighter as it enters engineering

and manufacturing development in March 2001. The Air Force Research

Laboratory considers TRL 7 as acceptable for low risk when entering the

engineering and manufacturing development stage. This maturity level

represents an advanced prototype of each of the critical technologies
B-281196

Page 14 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

demonstrated in an operational environment, such as on a flying test bed or

another aircraft similar to the Joint Strike Fighter. A prototype at this stage

would include all of the components of a critical technology in a

configuration that is very close to the size, weight, and configurations as

that expected for the Joint Strike Fighter. While the Joint Strike Fighter

Program has seen improvement in many technology areas since the

program started, maturity levels have not improved enough to indicate a

low-risk transition into the next phase. Figure 4 summarizes this data.

Figure 4: Joint Strike Fighter Critical Technology Readiness Levels at Program Start and Projected for Entry Into Engineering

and Manufacturing Development

Note: Due to the current Joint Strike Fighter competition, the critical technologies are not identified so

as not to associate them with the respective contractors.

Source: Joint Strike Fighter program office.
B-281196

Page 15 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

As shown in figure 4, all of the Joint Strike Fighter critical technologies are

expected to be at maturity levels lower than that considered acceptable for

low risk when entering engineering and manufacturing development

(TRL 7). Six of the technologies will still be below the level of maturity

(TRL 6) that is considered low risk for entering the demonstration phase

(program start), which the Joint Strike Fighter Program entered over

3 years ago.

Should any of these technologies be delayed or, worse still, not be available

for incorporation into the final Joint Strike Fighter design, the impact on

the program would be dramatic. For example, if one of the above critical

technologies needed to be replaced with its planned backup, DOD could

expect an increase of several billion dollars in production and operation

and support costs.9 The backup technology would also significantly

increase aircraft weight, which could negatively affect aircraft

performance. The currently planned technology is expected to be TRL 5 at

the beginning of the engineering and manufacturing development phase,

which indicates that substantial technology development must still occur

during that phase.

DOD Response to

Contractor Cost and

Schedule Problems Is

Inconsistent With

Acquisition Strategy Goals

Recent contractor reports indicate that both competing contractors have

cost growth and schedule concerns. In response to concerns over its ability

to meet program requirements, Boeing has redesigned its preferred design

configuration, including changing the wing shape; adding a horizontal tail,

which lengthens the fuselage; and switching from a forward sweep air

intake to a rearward sweep. According to a Lockheed Martin program

official, the company underestimated the cost of producing the two

demonstrator aircraft. In addition, Joint Strike Fighter Program documents

suggest that, due to manufacturing delays, the flight-test schedule for both

competing contractors’ Marine Corps variant could be at risk.

As a result of cost concerns, DOD restructured the Joint Strike Fighter

Program to allow each contractor leeway in correcting deficiencies. For

example, the Joint Strike Fighter flight-test program has been decreased,

which will reduce the data available for final proposal evaluation. Joint

Strike Fighter Program officials stated that with these flight-test

reductions, only the minimum acceptable flight quality demonstrations are

9 Specific details cannot be provided due to the competitive nature of the Joint Strike

Fighter Program.
B-281196

Page 16 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

expected. The number of preferred weapon system design updates has also

been reduced, which means that the contractors will provide DOD with

less information than originally planned prior to the submission of their

proposals. Finally, DOD has eliminated risk-reduction efforts and delayed

other technology demonstrations, which will increase the program’s

technical risk as it transitions into engineering and manufacturing

development (see footnote 8).

Traditional Approach to

Problems Is Underlying Cause

of Increased Risk

DOD’s traditional approach to weapon systems acquisition is to mature

technology at the same time the product is being developed. This approach

differs from best commercial practices, in which technology is developed

separately and ahead of the product. Pressures exerted on weapon system

programs create incentives for programs to include immature technologies

that may offer significant performance gains. This pressure can come from

users who demand performance improvements that necessitate the

application of unproven technologies to stay ahead of the perceived threat.

Another source of pressure is from technologists, who see a new weapon

system as an opportunity to apply a new technology. Also, the competition

for funds can encourage performance features—and requisite

technologies—that can distinguish the new weapon system from

competitors.

Once in a product development environment, external pressures to keep

the program moving (such as preserving cost and schedule estimates to

secure budget approval) become dominant. For example, DOD policies

require that a program be funded in the current year and that funds be

made available over the next 6 years in the DOD planning cycle. If a

program manager decided that an additional year was needed to reach the

desired level of technical maturity during the risk reduction/concept

demonstration phase, the planned start of the engineering and

manufacturing development phase could be delayed. This delay could

jeopardize funding for that phase, thus risking the funding support for the

entire program. Consequently, the program manager may be more likely to

accept the risk of moving forward with a lower level of technology maturity

rather than risk losing the program. That decision would raise cost-benefit

issues because cost increases and performance compromises would likely

occur.
B-281196

Page 17 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

Conclusions The Joint Strike Fighter Program’s acquisition strategy was to develop and

field an affordable aircraft that will meet each of the military services’

requirements goals. However, a key objective of the acquisition strategy—

entering into engineering and manufacturing development with low

technical risk—will not be achieved due to the manner in which the Joint

Strike Fighter program office is implementing the acquisition strategy. On

its current schedule, the program will enter the engineering and

manufacturing development phase without having reduced to an

acceptable level the technical risk of technologies that the program office

has identified as critical to meeting the program’s cost and requirement

objectives. This approach is not consistent with best commercial practices

in which technologies are more fully developed before proceeding into

product development. It is also not consistent with DOD’s planned

approach to developing the Joint Strike Fighter. Instead, the program

office’s revised approach is consistent with DOD’s traditional approach in

weapon system programs of concurrently developing technologies and

products. This traditional approach has often raised cost-benefit issues as a

result of cost increases, schedule delays, and compromised performance as

problems arose in completing technology development.

The Joint Strike Fighter Program is at an early development stage and,

therefore, DOD still has the opportunity to both demonstrate its

commitment to acquisition reform and chart a course to avoid the

problems that often befall major weapon systems. A decision to allow the

Joint Strike Fighter to proceed as planned, without mature critical

technologies, would compromise DOD’s position on acquisition reform, set

aside best commercial practices, and would perpetuate conditions that

have led to cost growth and schedule delays in many prior DOD weapon

system acquisition programs.

Recommendation To demonstrate DOD’s commitment to acquisition reform and to reduce the

risk of future cost growth, the program office should focus on risk

reduction efforts by maturing critical technologies prior to entering

engineering and manufacturing development, and it should be allowed to

do so without the penalty of withdrawal of funding support. Therefore, we

recommend that the Secretary of Defense direct the Joint Strike Fighter

program office to adjust the currently planned March 2001 engineering and

manufacturing development decision date to allow adequate time to

mature critical technologies to acceptable maturity levels, thereby closing

the gap between technology and requirements, before awarding the
B-281196

Page 18 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

engineering and manufacturing development contract. Options that the

program should consider include

• delaying the selection of a single contractor for the engineering and

manufacturing phase of the program until the program’s critical

technologies have been developed to an acceptable level or

• selecting a single contractor, but providing the time and funding for

additional risk reduction and technology maturation efforts, so that this

contractor can mature critical technologies to acceptable levels before a

decision is made to begin engineering and manufacturing development.

Matters for

Congressional

Consideration

To ensure that the Joint Strike Fighter Program enters the engineering and

manufacturing development phase with low technical risk, as envisioned

by the original acquisition strategy, Congress may wish to consider

requiring the Secretary of Defense to identify which of the eight critical

technologies discussed in this report will be incorporated on the Joint

Strike Fighter and certify that each of the identified technologies has been

demonstrated in a form that is the right size, weight, and configuration

needed for the Joint Strike Fighter aircraft. For any of the eight

technologies not initially included on the Joint Strike Fighter, the Secretary

of Defense should develop a plan showing the strategy for demonstrating

these technologies in the right size, weight, and configuration; showing the

approach for including them onto the Joint Strike Fighter; and the cost

impact if these technologies do not become available as planned. Congress

may also want to consider restricting DOD from obligating funds made

available for the engineering and manufacturing development phase of the

program until it receives this information from DOD.

Agency Comments and

Our Evaluation

In written comments on a draft of this report, the Director of Strategic and

Tactical Systems, within the Office of the Under Secretary of Defense for

Acquisition, Technology and Logistics, disagreed with our conclusions and

recommendation. DOD stated that our conclusion was based on

misinterpretation of the use of technology readiness levels to determine the

readiness of the critical technologies to enter engineering and

manufacturing development. In addition, DOD stated that (1) only the

maturity of the technology, not its integration onto the Joint Strike Fighter,

should be rated to determine its readiness to enter engineering and

manufacturing development; (2) our use of technology readiness levels

does not recognize that an evolutionary acquisition approach is being
B-281196

Page 19 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

applied to the Joint Strike Fighter Program; and (3) its own risk reduction

efforts are more meaningful than using technology readiness levels.

Contrary to DOD’s comments, there was no misinterpretation by us, the

contractors, or the program office representatives about what constituted a

readiness level of seven—the level considered necessary for acceptable

risk for entering engineering and manufacturing development. As pointed

out in this report, the readiness level definitions clearly state that for a

technology to be rated at a level seven, it must be demonstrated using

prototype hardware, such as a complete radar subsystem that is the same

size, weight, and configuration planned for the Joint Strike Fighter in an

operational environment. This definition was developed by the Air Force

Research Laboratory and was provided to the contractors prior to their

scoring of the critical technologies, and we discussed them at length during

several days of meetings when the contractors, with DOD program office

personnel present, assigned the readiness levels for the critical

technologies. During those discussions, we made it clear that

demonstrating the technology in a relevant environment would include

demonstrating the technology in a flying test bed aircraft, such as an F-16

or some other existing aircraft, and not necessarily on an actual Joint

Strike Fighter aircraft. As a result of those discussions, there was

agreement on the readiness levels assigned to each of the critical

technologies discussed. The program office then independently scored the

critical technologies. The program office scores, which are those used in

this report, were consistent with the contractors’ scores.

DOD stated that the Joint Strike Fighter Program will address the

integration risk of the critical technologies during, rather than prior to, the

engineering and manufacturing development phase. As indicated in our

report, we agree that the risk of integrating a subsystem–such as a radar–

onto the actual Joint Strike Fighter aircraft is an activity that is acceptable

for the engineering and manufacturing phase. However, we do not agree

that integrating various components of a subsystem—such as an antenna,

receiver, transmitter, and processor that make up a radar subsystem—into

a configuration that can be inserted into the Joint Strike Fighter is a task to

be left for the engineering and manufacturing development phase. In that

regard, commercial firms have told us that a key part of technology

development is getting the technology into the right size, weight, and

configuration needed for the intended product. Once this has been

demonstrated, the technology is at an acceptable level for engineering and

manufacturing development, where the emphasis should be on building the

actual Joint Strike Fighter aircraft. In separate technical comments on this
B-281196

Page 20 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

report, DOD acknowledged that the highest program risks will come from

integrating critical technologies, which it intends to do during engineering

and manufacturing development. DOD’s approach, of developing

technology during engineering and manufacturing development, represents

a fundamental difference between best commercial practices and DOD

practices and has been a leading reason for DOD’s weapon programs

incurring cost growth, schedule slippage, and technical problems.

In its comments, DOD described an evolutionary acquisition approach

being applied to the Joint Strike Fighter Program as a way to further

mitigate technology and program risk. This approach means that the Joint

Strike Fighter Program will use time-phased requirements and capabilities,

and only those technologies and capabilities that are mature and at low risk

will be installed on the first Joint Strike Fighter aircraft that are produced.

Cost and requirement trade-offs will be considered so that technologies

will not be included on the Joint Strike Fighter until they have

demonstrated an acceptable maturity level. We are concerned that since

many of the technologies assessed at low maturity levels in our report are

critical to obtaining an affordable aircraft—a primary objective for both the

Joint Strike Fighter and DOD’s overall tactical aircraft modernization

plan—their absence from the Joint Strike Fighter design could result in

much higher development, production, and support costs. The impact of

developing and producing Joint Strike Fighter aircraft without these

technologies must be considered, otherwise, DOD could find itself

committed to a program dependent on future maturation of these

technologies in order to meet program cost objectives. This would be

typical of DOD’s historical approach to developing weapon systems, and it

indicates that DOD is willing to assume greater risk during engineering and

manufacturing development than was envisioned in the Joint Strike Fighter

Program’s original acquisition strategy.

Finally, DOD stated that its own risk reduction methodology–which include

risk mitigation plans and engineering judgment–is a more meaningful

measure of risk versus the use of technology readiness levels. We do not

share DOD’s confidence in this regard. The objective of technology

readiness levels is to make decisions based on actual demonstrations that

technologies will work as needed for the intended product as opposed to

engineering judgment, which is subjective and open to interpretation. Our

prior work has shown that in place of risk mitigation plans and engineering

judgment, no matter how well intentioned, using technology readiness

levels results in a straightforward, objective, and quantifiable process for

determining a technology’s readiness for proceeding into the engineering
B-281196

Page 21 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

and manufacturing development phase of a program. DOD has recognized

the value of the technology readiness level process as indicated by its

intention to include the process in the revised acquisition guidelines—

commonly referred to as the 5000 acquisition series.

In summary, the information provided by DOD in its comments on our draft

report does not change our position that, as currently structured, the Joint

Strike Fighter Program will move into the engineering and manufacturing

development phase with unacceptable risk levels for many critical

technologies. Therefore, we have not changed the recommendation that

was in our draft report but we have added some matters for Congress to

consider. In essence, we suggest that Congress consider restricting DOD

from obligating funds for the engineering and manufacturing development

phase of the program until the Secretary of Defense certifies that the

program’s critical technologies have been demonstrated to acceptable

levels of maturity.

The full text of DOD’s comments are included in appendix III. DOD also

provided separate technical comments that we have incorporated into the

report as appropriate.

Scope and

Methodology

To determine the acquisition strategy for meeting affordability goals

through reduced risk prior to awarding the engineering and manufacturing

development contract, we reviewed program documentation on acquisition

strategy and acquisition reform initiatives and discussed these materials

with DOD and program officials. These materials include data such as the

Joint Strike Fighter Single Acquisition Management Plan, the Interim Test

and Evaluation Master Plan, Joint Interim Requirements Document, Key

Performance Parameters in the draft Joint Operational Requirements

Document, and Cost as an Independent Variable documentation.

To determine whether the program office is implementing the Joint Strike

Fighter acquisition strategy in a manner that will reduce risk and meet Joint

Strike Fighter cost, schedule, and performance goals, we collected data

and interviewed officials at various DOD locations, the Joint Strike Fighter

program office (Arlington, Virginia), and cognizant Navy, Marine Corps, and

Air Force requirements organizations. We interviewed officials and

collected contractor data from Lockheed Martin Tactical Aircraft Systems,

Fort Worth, Texas, and Palmdale, California; the Boeing Company, Seattle,

Washington, and Palmdale, California; General Electric, Cincinnati, Ohio;

and Pratt & Whitney, West Palm Beach, Florida.
B-281196

Page 22 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

To assess the maturity levels of key technologies, we used information

provided by the Joint Strike Fighter program office and contractors and

used the technology readiness level tool developed by the National

Aeronautics and Space Administration. On separate visits to the

contractors, with DOD program office personnel present, we provided

relevant technology managers the TRL scoring sheet found in appendix II

of this report. After significant discussion, and additional TRL information,

we asked these managers to score those technologies they considered

critical to enable their Joint Strike Fighter design to meet DOD

requirements for the aircraft. Upon reviewing these scores with the

program office and in order to gain an overall Joint Strike Fighter Program

perspective on technical maturity, the Joint Strike Fighter office agreed to

provide us with TRL scores for the eight technologies they considered

critical for meeting program cost and performance requirements. Those

scores are presented in this report.

We conducted our review from September 1998 through January 2000 in

accordance with generally accepted government auditing standards.

Unless you publicly announce its contents earlier, we plan no further

distribution of this report until 30 days from its issue date. At that time, we

will send copies to the congressional defense committees; the Honorable

William S. Cohen, Secretary of Defense; the Honorable F. Whitten Peters,

Secretary of the Air Force; the Honorable Richard Danzig, Secretary of the

Navy; General James L. Jones, Commandant, U.S. Marine Corps; and the

Honorable Jacob J. Lew, Director, Office of Management and Budget. We

will also make copies available to other interested parties on request.
Page 23 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

Please contact me at (202) 512-4841 if you or your staff have any questions

concerning this report. Key contributors to this assignment were Steve

Kuhta, Brian Mullins, Delores Cohen, and Matt Lea.

Sincerely yours,

Louis J. Rodrigues

Director, Defense Acquisitions Issues
Page 24 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

Appendix I

International Participation in the Joint Strike Apendxies

Fighter Program ApendxiI

Full collaborative partner

• full access to program data and structure, including representative

personnel within the program office

• ability to influence requirement definition and performance

characteristics

• Great Britain is the only participant in this category, committing

$200 million for the concept demonstration phase

Associate partner

• limited access to data and limited requirements influence

• representative personnel resident within the Joint Strike Fighter

program office

• all aspects of participation are negotiated with the Joint Strike Fighter

program office

• Denmark, the Netherlands, and Norway formed a group and committed

a total of $30 million to participate at this level

Informed customer

• limited access to program information and representation within the

program office is negotiable

• no influence on requirements

• Canada and Italy have committed $10 million each to participate at this

level

Major participants

• recently created category; also referred to as Foreign Military Sales

participation or Fee for Service

• negotiate directly with the program office for specific Joint Strike

Fighter Program information (e.g., Cost and Operational Performance

Trade processes and modeling and simulation studies)

• no representative personnel resident within the Joint Strike Fighter

program office

• Singapore, Turkey, and Israel are currently participating at this level
Page 25 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

Appendix II

Technology Readiness Levels and Their

Definitions ApendxiI

Source: Best Practices: Better Management of Technology Development Can Improve Weapon

System Outcomes (GAO/NSIAD-99-162, July 30, 1999).

Technology readiness level Description

1. Basic principles observed and reported. Lowest level of technology readiness. Scientific research begins to be translated into

applied research and development. Examples might include paper studies of a technology’s

basic properties

2. Technology concept and/or application

formulated.

Invention begins. Once basic principles are observed, practical applications can be

invented. The application is speculative and there is no proof or detailed analysis to support

the assumption. Examples are still limited to paper studies.

3. Analytical and experimental critical

function and/or characteristic proof of

concept.

Active research and development is initiated. This includes analytical studies and laboratory

studies to physically validate analytical predictions of separate elements of the technology.

Examples include components that are not yet integrated or representative.

4. Component and/or breadboard validation

in laboratory environment.

Basic technological components are integrated to establish that the pieces will work

together. This is relatively “low fidelity” compared to the eventual system. Examples include

integration of “ad hoc” hardware in a laboratory.

5. Component and/or breadboard validation

in relevant environment.

Fidelity of breadboard technology increases significantly. The basic technological

components are integrated with reasonably realistic supporting elements so that the

technology can be tested in a simulated environment. Examples include “high fidelity”

laboratory integration of components.

6. System/subsystem model or prototype

demonstration in a relevant environment.

Representative model or prototype system, which is well beyond the breadboard tested for

technology readiness level (TRL) 5, is tested in a relevant environment. Represents a major

step up in a technology’s demonstrated readiness. Examples include testing a prototype in

a high fidelity laboratory environment or in simulated operational environment.

7. System prototype demonstration in an

operational environment.

Prototype near or at planned operational system. Represents a major step up from TRL 6,

requiring the demonstration of an actual system prototype in an operational environment,

such as in an aircraft, vehicle, or space. Examples include testing the prototype in a

test bed aircraft.

8. Actual system completed and “flight

qualified” through test and demonstration.

Technology has been proven to work in its final form and under expected conditions. In

almost all cases, this TRL represents the end of true system development. Examples

include developmental test and evaluation of the system in its intended weapon systems to

determine if it meets design specifications.

9. Actual system “flight proven” through

successful mission operations.

Actual application of the technology in its final form and under mission conditions, such as

those encountered in operational test and evaluation. In almost all cases, this is the end of

the last “bug fixing” aspects of true system development. Examples include using the

system under operational mission conditions.
Page 26 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

Appendix III

Comments From the Department of Defense ApendxiI

See comment 1.

Note: GAO comments

supplementing those in the

report text appear at the

end of this appendix.
Appendix III

Comments From the Department of Defense

Page 27 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

See comment 2.
Appendix III

Comments From the Department of Defense

Page 28 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition
Appendix III

Comments From the Department of Defense

Page 29 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition
Appendix III

Comments From the Department of Defense

Page 30 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition

The following are GAO’s comments on the Department of Defense’s (DOD)

letter dated April 17, 2000.

GAO Comments 1. DOD provided reevaluated technology readiness levels to indicate that

if the risk of integrating critical technologies was not considered, then

the technology risk is expected to be at an appropriate level.

Notwithstanding our disagreement over whether integration risks

should be considered in this assessment, we believe DOD’s standards

are below minimum acceptable levels. Only half of the critical

technologies are projected to be at readiness level 7 while the other half

will still be at readiness level 6. We disagree with DOD that readiness

level 6 is acceptable for low-risk entry into engineering and

manufacturing development. Leading commercial firms typically insist

on a readiness level 8−a higher standard than that used by the Air Force

Research Laboratory−before a technology can be included on a

product.

2. DOD noted that in addition to technology readiness levels, other

considerations were necessary to decide when and where to insert new

technologies into weapon system programs. While DOD did not

elaborate on what other considerations were applicable to the Joint

Strike Fighter Program, its similar response to our prior work involving

technology readiness levels referred to such considerations as the

increasing projected life for new weapon systems, total ownership

costs, and urgency based upon threat assessments. Many of the

technologies presented in this current report directly address these

other considerations and are critical to the success of the Joint Strike

Fighter Program. Therefore, using TRLs in the Joint Strike Fighter

Program not only allows DOD to manage performance risk, but also to

address those technologies critical to meeting these other

considerations. With regard to risk assessment, leading commercial

firms who have adopted knowledge-based risk assessment approaches,

such as TRLs, have produced results such as more technically

advanced, higher quality products, in less time and at a lower cost than

their predecessors.

(707386) Letr
Ordering Information The first copy of each GAO report is free. Additional copies of

reports are $2 each. A check or money order should be made out to

the Superintendent of Documents. VISA and MasterCard credit

cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are

discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

Orders by visiting:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders by phone:

(202) 512-6000

fax: (202) 512-6061

TDD (202) 512-2537

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any list

from the past 30 days, please call (202) 512-6000 using a touchtone

phone. A recorded menu will provide information on how to obtain

these lists.

Orders by Internet:

For information on how to access GAO reports on the Internet,

send an e-mail message with “info” in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web home page at:

http://www.gao.gov

To Report Fraud,

Waste, or Abuse in

Federal Programs

Contact one:

• Web site: http://www.gao.gov/fraudnet/fraudnet.htm

• e-mail: fraudnet@gao.gov

• 1-800-424-5454 (automated answering system)


United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. GI00