U.S. GOVERNMENT PRINTING OFFICE

70–782 DTP WASHINGTON : 2001

JOINT STRIKE FIGHTER ACQUISITION REFORM:

WILL IT FLY?

HEARING

BEFORE THE

SUBCOMMITTEE ON NATIONAL SECURITY,

VETERANS AFFAIRS, AND INTERNATIONAL

RELATIONS

OF THE

COMMITTEE ON

GOVERNMENT REFORM

HOUSE OF REPRESENTATIVES

ONE HUNDRED SIXTH CONGRESS

SECOND SESSION

MAY 10, 2000

Serial No. 106–202

Printed for the use of the Committee on Government Reform

(

Available via the World Wide Web: http://www.gpo.gov/congress/house

http://www.house.gov/reform

For sale by the Superintendent of Documents, U.S. Government Printing Office

Internet: bookstore.gpo.gov Phone: (202) 512–1800 Fax: (202) 512–2250

Mail: Stop SSOP, Washington, DC 20402–0001

(II)

COMMITTEE ON GOVERNMENT REFORM

DAN BURTON, Indiana, Chairman

BENJAMIN A. GILMAN, New York

CONSTANCE A. MORELLA, Maryland

CHRISTOPHER SHAYS, Connecticut

ILEANA ROS-LEHTINEN, Florida

JOHN M. MCHUGH, New York

STEPHEN HORN, California

JOHN L. MICA, Florida

THOMAS M. DAVIS, Virginia

DAVID M. MCINTOSH, Indiana

MARK E. SOUDER, Indiana

JOE SCARBOROUGH, Florida

STEVEN C. LATOURETTE, Ohio

MARSHALL ‘‘MARK’’ SANFORD, South

Carolina

BOB BARR, Georgia

DAN MILLER, Florida

ASA HUTCHINSON, Arkansas

LEE TERRY, Nebraska

JUDY BIGGERT, Illinois

GREG WALDEN, Oregon

DOUG OSE, California

PAUL RYAN, Wisconsin

HELEN CHENOWETH-HAGE, Idaho

DAVID VITTER, Louisiana

HENRY A. WAXMAN, California

TOM LANTOS, California

ROBERT E. WISE, JR., West Virginia

MAJOR R. OWENS, New York

EDOLPHUS TOWNS, New York

PAUL E. KANJORSKI, Pennsylvania

PATSY T. MINK, Hawaii

CAROLYN B. MALONEY, New York

ELEANOR HOLMES NORTON, Washington,

DC

CHAKA FATTAH, Pennsylvania

ELIJAH E. CUMMINGS, Maryland

DENNIS J. KUCINICH, Ohio

ROD R. BLAGOJEVICH, Illinois

DANNY K. DAVIS, Illinois

JOHN F. TIERNEY, Massachusetts

JIM TURNER, Texas

THOMAS H. ALLEN, Maine

HAROLD E. FORD, JR., Tennessee

JANICE D. SCHAKOWSKY, Illinois

———

BERNARD SANDERS, Vermont

(Independent)

KEVIN BINGER, Staff Director

DANIEL R. MOLL, Deputy Staff Director

DAVID A. KASS, Deputy Counsel and Parliamentarian

LISA SMITH ARAFUNE, Chief Clerk

PHIL SCHILIRO, Minority Staff Director

SUBCOMMITTEE ON NATIONAL SECURITY, VETERANS AFFAIRS, AND INTERNATIONAL

RELATIONS

CHRISTOPHER SHAYS, Connecticut, Chairman

MARK E. SOUDER, Indiana

ILEANA ROS-LEHTINEN, Florida

JOHN M. MCHUGH, New York

JOHN L. MICA, Florida

DAVID M. MCINTOSH, Indiana

MARSHALL ‘‘MARK’’ SANFORD, South

Carolina

LEE TERRY, Nebraska

JUDY BIGGERT, Illinois

HELEN CHENOWETH-HAGE, Idaho

ROD R. BLAGOJEVICH, Illinois

TOM LANTOS, California

ROBERT E. WISE, JR., West Virginia

JOHN F. TIERNEY, Massachusetts

THOMAS H. ALLEN, Maine

EDOLPHUS TOWNS, New York

BERNARD SANDERS, Vermont

(Independent)

JANICE D. SCHAKOWSKY, Illinois

EX OFFICIO

DAN BURTON, Indiana HENRY A. WAXMAN, California

LAWRENCE J. HALLORAN, Staff Director and Counsel

J. VINCENT CHASE, Chief Investigator

JASON CHUNG, Clerk

DAVID RAPALLO, Minority Counsel

(III)

C O N T E N T S

Page

Hearing held on May 10, 2000 ............................................................................... 1

Statement of:

McNaugher, Dr. Thomas L., deputy director, Arroyo Center, Rand; Rodney Larkins, business development manager, 3M Corp.; and Dr. Wesley

Harris, professor, Department of Aeronautics and Astronautics .............. 100

Rodrigues, Louis J., Director, National Security and International Affairs

Division, U.S. General Accounting Office ................................................... 5

Soloway, Stan Z., Deputy Secretary of Defense, Acquisition Reform, Department of Defense; and Major General Raymond Huot, U.S. Air

Force, Acquisition Programs, Department of Defense ............................... 50

Letters, statements, etc., submitted for the record by:

Blagojevich, Hon. Rod R., a Representative in Congress from the State

of Illinios, prepared statement of ................................................................ 37

Harris, Dr. Wesley, professor, Department of Aeronautics and Astronautics, prepared statement of ........................................................................... 120

Huot, Major General Raymond, U.S. Air Force, Acquisition Programs,

Department of Defense:

Information concerning TRLs ................................................................... 96

Prepared statement of ............................................................................... 72

Larkins, Rodney, business development manager, 3M Corp., prepared

statement of ................................................................................................... 112

McNaugher, Dr. Thomas L., deputy director, Arroyo Center, Rand, prepared statement of ........................................................................................ 104

Rodrigues, Louis J., Director, National Security and International Affairs

Division, U.S. General Accounting Office, prepared statement of ............ 10

Shays, Hon. Christopher, a Representative in Congress from the State

of Connecticut, prepared statement of ........................................................ 3

Soloway, Stan Z., Deputy Secretary of Defense, Acquisition Reform, Department of Defense:

Information concerning TRL levels .......................................................... 98

Prepared statement of ............................................................................... 56

(1)

JOINT STRIKE FIGHTER ACQUISITION

REFORM: WILL IT FLY?

WEDNESDAY, MAY 10, 2000

HOUSE OF REPRESENTATIVES,

SUBCOMMITTEE ON NATIONAL SECURITY, VETERANS

AFFAIRS, AND INTERNATIONAL RELATIONS,

COMMITTEE ON GOVERNMENT REFORM,

Washington, DC.

The subcommittee met, pursuant to notice, at 10 a.m., in room

2247, Rayburn House Office Building, Hon. Christopher Shays

(chairman of the subcommittee) presiding.

Present: Representatives Shays, Terry, Biggert, and Blagojevich.

Staff present: Lawrence J. Halloran, staff director and counsel;

J. Vincent Chase, chief investigator; Robert Newman, professional

staff member; Jason M. Chung, clerk; David Rapallo, minority

counsel; and Earley Green, minority assistant clerk.

Mr. SHAYS. I would like to call this hearing to order, to welcome

our witnesses and our guests and again to invite anyone who wants

to, to take off their coats. It’s a hot room and we have asked it to

be cooled down, but feel free to take off your jackets.

The military procurement holiday is about to end. Defense budgets being debated today on both sides of the Capitol reflect a bicameral and bipartisan consensus on the need to modernize the

aging planes, ships, weapons and equipment used to win the cold

war. Today we discuss the need to modernize the acquisition systems the Department of Defense [DOD], will use to procure postcold war weapons systems.

Just as the weaponry of the last century won’t win the peace in

the next, the acquisition practices of the past will not efficiently or

affordably meet future defense needs. Fifteen-year development cycles enshrine old technologies now rendered obsolete in 15 months.

Massive cost overruns and schedule slippages are fueled in part by

the launch of engineering and design work before hoped-for technologies have been refined. Extraneous, often pervasive incentives,

push program officials toward artificial deadlines and premature

production commitments.

Various iterations of acquisition reform at DOD have attempted

to address these problems and reinvigorate a hidebound acquisition

culture inside and outside the Pentagon. In launching the $200 billion Joint Strike Fighter [JSF], aircraft acquisition, DOD promised

the prgram would be a model of reform driven by affordability and

the technical knowledge base, not by the disingenuous optimism

and defense budget politics that proved so costly in the past.

2

At the subcommittee’s request, the General Accounting Office

[GAO], analyzed the JSF acquisition strategy to determine if the

promise of reform is being fulfilled in practice. Their report released today finds the Joint Strike Fighter program straying from

commercial best practices and knowledge-driven benchmarks. As

the date approaches to select a prime JSF contractor and begin engineering on the final system concepts, DOD appears ready to

abandon quantitative measures of technological maturity and revert to the business as usual of concurrent technology development

and product development.

GAO recommends DOD focus on risk reduction efforts by maturing critical technologies prior entering the next phase of the JSF

program, even if that means delaying contractor selection and contract awards beyond the planned March 2001 date. The program

should be permitted to pursue the original low-risk acquisition

strategy according to GAO, without the penalty of withdrawal of

funding support.

DOD disagrees, claiming critical technologies will be mature

enough to proceed in final design and engineering next year.

As the debate unfolds, the choice should not be between a fully

funded Joint Strike Fighter and a commitment to acquisition reform. We can have both. If the program succumbs to cold war acquisition habits, costs will skyrocket, the development cycle will

stretch over the horizon, and the next generation fighter needed by

the Air Force, the Navy, and the Marines might never fly.

We welcome the testimony of all our witnesses on this important

subject.

[The prepared statement of Hon. Christopher Shays follows:]

3

4

5

Mr. SHAYS. And if I could, I would like to welcome my colleague

and ask if he has any comments to make.

Mr. TERRY. Thank you, no. I don’t have any opening statement.

I’ll submit one in a few days for the record.

I do want to compliment you on holding this hearing. When we

look through the issues concerning the future of the military and

the confidence that people have in our military and as we make a

renewed commitment as a Congress to our military, we have got

to ensure that the protocol is in place, the system is in place to ensure that we use our money wisely, that we’re looking toward the

future. And when I’m speaking at veterans organizations or just

simply people that are interested in my home of Omaha, NE and

home of Offutt Air Force Base, that’s what they want to know.

That’s why this hearing is so important today. Since it is so important, let me not continue to use the time, let’s hear from our

witnesses.

Mr. SHAYS. Thank you Mr. Terry.

Just some housekeeping, if we could do that now. I ask unanimous consent that all members of the subcommittee be permitted

to place an opening statement in the record and that the record remain open for 3 days for that purpose. Without objection, so ordered.

I ask further unanimous consent that all witnesses be permitted

to include their written statements in the record. Without objection, so ordered.

Our first panel, we have three, is Mr. Louis Rodrigues, Director,

National Security and International Affairs Division, U.S. General

Accounting Office [GAO]. In fact, I think that’s referred to more as

GAO than the full title.

Now Mr. Rodrigues, if you could stand up, we’ll swear you in, as

we swear in all our witnesses. Raise your right hand, please.

[Witness sworn.]

Mr. SHAYS. Note for the record our witness has responded in the

affirmative. What we’re going to do is we’re going to run the clock

for 5 minutes then we will flip it again for another 5. And then if

you could conclude within the 10-minute period.

STATEMENT OF LOUIS J. RODRIGUES, DIRECTOR, NATIONAL

SECURITY AND INTERNATIONAL AFFAIRS DIVISION, U.S.

GENERAL ACCOUNTING OFFICE

Mr. RODRIGUES. Thank you, Mr. Chairman. Mr. Chairman, Congressman Terry, I am pleased to be here today to discuss the application of best commercial practices to DOD weapons systems in

general, the Joint Strike Fighter in particular. Before getting into

details, I would like to emphasize the importance of the Joint

Strike Fighter decision to reforming DOD’s weapons acquisition

process.

As you know, the Department is in the process of rewriting its

directives governing systems acquisition, referred to as DOD 5000

series. At the Department’s request, we have been participating in

this effort through input to its working group. The objective of the

rewrite is to bring about better, cheaper, faster outcomes in weapons programs. It is acquisition reform. Our contributions or inputs

to this effort are based on our reports for the Senate Armed ServVerDate 11-MAY-2000 11:50 May 24, 2001 Jkt 000000 PO 00000 Frm 00009 Fmt 6633 Sfmt 6633 C:\DOCS\70782.TXT HGOVREF1 PsN: HGOVREF1
6

ices Committee on using best commercial practices to improve

weapons program outcomes.

The DOD draft rewrite embodies critical features documented in

our work. Two of these features are critical to the upcoming Joint

Strike Fighter decision. First, the technology development must be

separated from product development. That is, before entering engineering manufacturing development, we must have a match between proven technologies and requirements. And second, metrics

to accurately measure technology must be used.

In the 5000 series rewrite, they are adopting a measurement system we used in our Joint Strike Fighter assessment, referred to as

technology readiness levels. The commitment to this knowledgebased versus the current schedule- and funding-driven process is

reflected in the testimony of the Deputy Under Secretary of Defense for Acquisition Reform on March 16th before the House Government Reform Subcommittee on Government Management, Information, and Technology, ‘‘In the new systems acquisition environment, key acquisition and long-term funding commitments will not

be made until technology is mature.’’

I have a lot of respect for the people in DOD who are leading acquisition reform. Philosophically we are in agreement on the best

practices and the changes that are needed in the DOD environment

to make such practices work on weapons systems. We at GAO are

extremely encouraged by the commitment of DOD acquisition leaders to improving the weapons acquisition outcomes through the use

of a knowledge-based commercial business practices. At the same

time, however, we are concerned that the written directives and

oral commitments will have little impact if not reflected in key decisions.

In that sense, the key acquisition decision of entering engineering and manufacturing development on the Joint Strike Fighter

stands out as the flagship for weapons acquisition reform. To apply

anything less than the standards in the directives will send a clear

message that while the instructions and rhetoric are changing, it’s

business as usual.

I will now briefly describe what we have learned from our best

practices work and how we have applied that to assessing the Joint

Strike Fighter program. Our best practices work has shown that a

knowledge-based process is essential to getting better cost schedule

and performance outcomes. This means that the decisionmakers

must have virtual certainty about critical facets of the product

under the development when needed. This knowledge can be measured in three junctures that we refer to as knowledge points, as

shown in the chart to my right.

Knowledge point 1 is when a match is made between the customer’s requirements and the available technology. This occurs

prior to entering product development.

Knowledge point 2 is when the product’s design is determined to

be capable of meeting performance requirements. This occurs about

midway through the product development phase.

And knowledge point 3 is when the product is determined to be

producible within cost, schedule, and quality targets, which occurs

prior to entering production.

7

Today I’ll focus on only knowledge point 1 because it is the biggest contributor to a successful product development, achieving

subsequent knowledge points depends on it, and it is the point

where JSF should be as it enters EMD. As a technology is developed, it moves from a concept to a feasible invention to a component that must fit onto a product and function as expected. In between, there are increasing levels of demonstration that can be

measured.

In our review of best practices for including new technology and

products, we applied a scale of technology readiness levels from 1

to 9, pioneered by NASA and adapted by the Air Force Research

Laboratory. Without going into the details of each level, a level 4

equates to a laboratory demonstration of technology that is not in

a usable form.

Imagine an advanced radio technology that can be demonstrated

with components that take up a table top. When initial hand-built

versions of all the radio’s basic elements are hand wired and tested

together in a laboratory, the radio reaches a readiness level of 5.

A technology readiness level of 7 is the demonstration of the

technology that approximates its final form and occurs in an environment outside the laboratory. That same radio at level 7 would

be installed and demonstrated in a platform such as an existing

fighter aircraft. The lower the level of the technology at the time

it is included in product development, the higher the risk that it

will cause problems in product development.

According to the Air Force Research Laboratory, level 7 enables

the technology to be included on a product development with acceptable risk.

When we asked leading commercial firms to apply these standards to their own methods of assessing technology maturity, we

found that most insisted on even higher levels of readiness before

they allowed a new technology into product development.

Regarding the Joint Strike Fighter, in conjunction with the program office and the two competing contractors, we determined the

readiness levels of critical technologies. The table to my right

shows the technology readiness levels of eight critical technology

areas identified by the Joint Strike Fighter program office.

Let me try to explain this a little bit. What we had them do is

score the technologies at three points. The blue line reflects the

readiness level for each of those technology areas where they were

at program launch. The yellow line reflects where they were at the

time they did the scoring. And the red extension reflects where

they plan to be based on what they’re planning to do between now

and the down select to the engineering and manufacturing development phase. So it’s the totality of the line that shows where they

plan to be when they enter EMD.

In terms of engineering and manufacturing development, which

is reflected by the second diamond on the right, none of the critical

technology areas are projected to be at readiness level 7, which the

Air Force Research Laboratory considers acceptable for entry into

engineering and manufacturing development. Should any of these

technologies be delayed, or worse, not available for incorporation

into the final JSF design, the impact on the program would be dramatic. For example, if one of the critical technologies needed to be

8

replaced with its planned backup, DOD could expect an increase of

about 10 percent in unit costs. The backup technology would also

significantly increase aircraft weight which can negatively impact

aircraft performance. This technology is projected to be at a technology readiness level of 5 at the beginning of the engineering

manufacturing development phase, substantially below the criteria

of 7.

As noted earlier, at the policy level, DOD officials have agreed

that technology development should be kept separate from product

development and that technology readiness levels are a valid way

to assess technology maturity. However, in response to our report

on the Joint Strike Fighter, DOD balked at the use of technology

readiness levels and their implications for keeping technology development out of the fighter’s engineering and manufacturing development phase. One of the reasons DOD cited for its unwillingness to accept the technology readiness levels assessed was that

the levels were based on integration in the Joint Strike Fighter aircraft.

On the contrary, the technology readiness levels assessed by the

program office and the contractors were based on a clear understanding that a level 7 could be reached by demonstrating the technology in a relevant environment. It was further made clear that

a relevant environment would include demonstrating a technology

in an existing aircraft like an F–16, not a Joint Strike Fighter.

There is no misunderstanding.

A second reason DOD disagreed with the readiness levels assessed was that its own risk mitigation plans and judgment were

more meaningful and that they showed the technology risk to be

acceptable. Risk mitigation plans and judgment are necessary to

managing any major development effort. However, without an underpinning such as technology readiness levels that allows transparency into program decisions, these methods allow significant

technical unknowns to be judged acceptable risks because a plan

exists for resolving them in the future.

Experience on previous programs has shown that such methods

have rarely assessed technical unknowns as a high or unacceptable

risk. Consequently, they fail to guide programs to meet promised

outcomes. Technology readiness levels are based on demonstrations

of how well technology has actually performed. Their strength lies

in the fact that they characterize knowledge that exists rather than

plans to gain knowledge in the future.

In conclusion, we believe that separating technology development

from product development can create conditions for a successful

Joint Strike Fighter program. To proceed as planned, entering the

engineering manufacturing development phase of the program with

immature technologies, is to risk scheduled delays and cost growth.

Instead, the program has an opportunity to mature technologies in

a more risk-tolerant environment by making the right decisions

now.

In our report, we recommend that the Joint Strike Fighter program continue in its current program definition and risk reduction

phase, delaying the decision to move into engineering and manufacturing development until technologies are demonstrated to acceptable levels. Taking the additional time to mature the technologies

9

will then allow the program manager to focus on design and manufacturing risks during engineering and manufacturing development. It also increases the possibility of completing product development in a more timely and predictable manner. Such a delay

does not necessarily lengthen the total product development cycle.

In fact, the knowledge gained from time spent developing technologies in the beginning can often shorten the time it takes to get

the product to market.

Similarly, a delay should not be misinterpreted as a lessening of

support for the Joint Strike Fighter program. Rather, it would

demonstrate decisionmakers’ willingness to make the up-front investment necessary to mature key technologies before committing

the Joint Strike Fighter team to deliver a product. Such a commitment is more likely to put the program on a better footing to succeed than placing the burden on the engineering and manufacturing development phase.

Mr. Chairman, that concludes my statement and I would be glad

to answer any questions you or the other members may have.

[The prepared statement of Mr. Rodrigues follows:]

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

Mr. SHAYS. Thank the gentleman very much. Thank you very

much, Mr. Rodrigues. The bottom line is we’re scheduled to build

the Air Force F–22 and the F/A–18, the F Super Hornet, and then

we’re scheduled in the future to see this project go forward. You’re

not suggesting in any way that we end this program and not do

it; correct?

Mr. RODRIGUES. Correct, Mr. Chairman.

Mr. SHAYS. The bottom line to the debate we’re going to have

today is you just want them to follow their game plan.

Mr. RODRIGUES. We want them to—— Mr. SHAYS. By ‘‘they,’’ the DOD to follow the game plan.

Mr. RODRIGUES. Yes, sir. Mr. Chairman, what we’re talking

about is that the Joint Strike Fighter decision is such a major decision it is the signal of what happens in acquisition reform. It will

underscore the implementation of the changes that are being made

today in the regulations to guide this process. And if we make the

same types of decisions that we have made in the past, that is, to

allow unknowns to creep into product development, to allow technology development to be done concurrently with product development, we’ll continue to see very long development cycles like we

have seen on the F–22, like we saw on the B–2, and like we have

seen on programs historically.

And that situation undermines our modernization effort. It leads

to cost growth and schedule delays. It leads to the problems that

we have seen with the programs in the past where we end up cutting the quantities in half and never do meet our modernization

goals. In fact, it exacerbates the problems that we’re trying to resolve, one of which is aging of the fleet. It is a real issue. And to

continue to move into programs where we can’t deliver on cost and

schedule, that are fit into very fight funding wedges, puts us in a

position where we end up in the future cutting the quantities because of cost problems and ending up without the modernization

that we absolutely need.

Mr. SHAYS. The bottom line, I just want to establish the point

that GAO is not saying that we scrap this program. You want us

to follow the game plan. And so what this committee has to understand is what you’re saying and what DOD is saying and where

those disagreements occur.

I’m going to expose my ignorance a little bit here. If we could go

to the first chart, I find that’s the best way to learn. I sometimes

make my staff nervous when I do this.

Mr. RODRIGUES. What this depicts, Mr. Chairman is, in effect,

the commercial model.

Mr. SHAYS. What I need to do is just incorporate this with your

nine product requirements. How does that—this isn’t the first three

of the nine. Can you incorporate this to the—to TRLs?

Mr. RODRIGUES. TRLs are a metric. They are a metric. They’re

the metric that allows you to make the determination of the technology match to requirements. So that the first knowledge point in

any program is assuring that you have a match between the technology and the requirement.

Mr. SHAYS. Is point 3 here point 7 on TRLs?

Mr. RODRIGUES. No. The 7 on the TRL is that that match has

occurred in that technology-to-requirements match. It is the first

33

knowledge point. It is where we are approaching on the Joint

Strike Fighter.

Mr. SHAYS. And the debate, it seems to me, is whether DOD

thinks your match point at 7 is the key point. They’re willing to

move forward before they’ve reached 7. They dispute that they

haven’t reached point 7.

Mr. RODRIGUES. I’m sure that will become a confusing issue.

Mr. SHAYS. I just want to know what you think.

Mr. RODRIGUES. To me it’s absolutely clear. Applying technology

readiness levels, which is the demonstration of these technologies,

puts those technologies where they are on the second chart.

Would you put that chart back up, please?

And we’re accepting their projections of where they’re going to

be. Once again as I said, the diamond to the right is where you

should have to be. That is the level 7. That is the acceptable risk

for entry into engineering and manufacturing development.

Mr. SHAYS. What I’m asking you is do they dispute what you

have up here and are they—believe that the acceptable risk is at

point 6 and you believe the acceptable risk is at point 7? Where is

the dispute as far as you would articulate it?

Mr. RODRIGUES. They argue that there was a misunderstanding

in the application by the contractors. We did not score these. The

contractors applied the criteria. The DOD argument, as I understand it, is that there was a misunderstanding in the application.

To get to a level 7 requires the demonstration of the technology or

the hardware involved in this technology in the form fit and function. In other words, what it’s going to really look like when it has

to go onto the Joint Strike Fighter in a relevant environment.

And the Department’s position, as I understand it, is that there

was a misunderstanding and that what people were thinking when

they scored this was that we were saying it had to actually be on

the Joint Strike Fighter. Well, that obviously cannot happen before

the program enters engineering and manufacturing development

because there is no Joint Strike Fighter.

We worked very closely with the contractors and program office

people who were at those meetings to explain exactly how to do

this. I have the sheets that were provided to them. We spent days.

And we made sure that they understood that we are talking about

surrogates. There is physically no way to do the kind of demonstration they’re contending these people were thinking we were talking

about absent a Joint Strike Fighter. What we were talking about

was demonstrated on surrogates.

Mr. SHAYS. To give some life to these technologies, you’ve listed

under numbers because of proprietary issues, but—so I won’t debate which is which. But what we’re talking about is propulsion,

we’re talking about flight systems, we’re talking about weapons.

Each of these is a technology. We’re talking about structures and

materials. In avionics we’re talking about radar and the mission

systems and supportability and training and producibility. Those

are the things that we’re talking about.

Now when we started out, they were at level technology 1,

whichever one it is, was at level 3. And you’re then saying that

they have gotten to level 5 and expect to get in technology 1 to

34

level 6. And then make a commitment. They’re wanting to commit

at level 6, correct?

Mr. RODRIGUES. That’s what we’re saying. They’re not going to

agree with that.

Mr. SHAYS. But that’s what you think they’re saying.

Mr. RODRIGUES. Yes.

Mr. SHAYS. And they’re willing to commit on technology 8 at

level 6. You’re not saying they’re willing to commit on technology

4 at level 5, are you; or are you?

Mr. RODRIGUES. Right now our understanding is they plan to

move ahead—— Mr. SHAYS. And every one of the levels I mentioned—— Mr. RODRIGUES. Unless they trade these off. But right now

they’re in the program—these are critical path technologies. These

things are essential. Not technology 1 necessarily—technology

areas 2 through 7 are critical to meeting the affordability goal. And

if affordability is the primary factor driving this program, which is

my understanding of what everybody is signed up to do, an affordable next generation aircraft, then being able to launch or have a

program without these becomes really problematic.

The one example I used was if one of these areas was excluded

or you had to go to its fallback, you would add almost 10 percent

cost to the conventional variant which represents 1,700 plus of the

aircraft they’re going to build. That is a significant cost difference.

And that’s only one area.

So these all deal with affordability. As long as affordability remains paramount, the question becomes if you don’t use these,

what does it do to your cost projections? If you go do a block approach which—talking about using an iterative process going with

the block one that doesn’t have everything and moving on—I think

we have to understand what is the implications of that. What

you’re doing then is launching into the development of a product

and the production of a product that I’m not sure it even comes

close to meeting cost goals, because I don’t know what the effect

of deleting these from the first blocks are. And once you have that

production line started, if you’re betting on technology in some subsequent block in order to get the affordability into the program that

you need in the long run, I don’t think that we solved anything by

going to a block approach.

Mr. SHAYS. I would agree with that. Let me just go back to your

first chart again on the best practices model. There are three

knowledge points. And I appreciate the indulgence of the committee just to go a little further on this. Knowledge 1, this is basically

the character of best practices act for any industry in this modern

day and age.

Mr. RODRIGUES. That’s the best commercial practice.

Mr. SHAYS. The concept is matches made between the customer’s

requirements and the available technology. So whatever technology

is available, how can we meet the customer’s requirements? We try

to match those two.

Mr. RODRIGUES. Right.

Mr. SHAYS. And then the second point is when the product’s design is determined to be capable of meeting performance requirements.

35

Mr. RODRIGUES. Yes. And there’s a metric associated with that

as well. The Joint Strike Fighter right now would be in the equivalent of what’s called the technology development phase. It is in risk

reduction. There are concept demonstrations in the combined

phase. So it’s really before that point of technology match.

Mr. SHAYS. We’re not at point 2 yet.

Mr. RODRIGUES. No, the product development is the equivalent

of the engineering and manufacturing development phase. Knowledge point 2 is a standard knowledge point that occurs both in industry and in the Department. The Department just didn’t adhere

to it. They actually have the standard and they have the metrics

for measuring it. It’s something called CDR, critical design review,

and the standard is that 90 percent of the drawings are released

to manufacturing at that point. Now, unfortunately, the Department doesn’t adhere to that. The standard exists, they don’t apply

it. In commercial industry, when we were doing the best commercial practice work, they exceed the 90 percent.

Mr. SHAYS. I have two more basic questions. When we get to

knowledge point 3, the product is determined to be producible within cost schedule and quality targets. Now, is the desire on the part

of the military to move forward, in your judgment, with production

before we have reached point 2, is that because they want the product to do more than right now the technology allows?

Mr. RODRIGUES. Let me try to clarify something. This isn’t about

moving forward at this point with production. This is about entry

into product development.

Mr. SHAYS. Right.

Mr. RODRIGUES. And I don’t know—— Mr. SHAYS. But even the point of—the question still stands, and

thank you for clarifying. The bottom line, though, is I’m just trying

to understand the tension. In other words, they want this airplane

to do more than the technology presently allows; correct?

Mr. RODRIGUES. Using technology readiness levels, that’s an absolutely true statement.

Mr. SHAYS. So the issue is if they had to accept the technology

that existed today, they wouldn’t be able to have this plane do

what they want it to do. So that’s the tension. But they want the

project to keep moving forward. The message that I’m hearing from

GAO is saying, relax, we’re at the cutting edge, we’re not talking

about the cold war where we have to rush this to the marketplace.

Mr. RODRIGUES. Absolutely.

Mr. SHAYS. So we need to slow down, develop the technology, before we start to do the development.

Mr. RODRIGUES. Yes. And the other thing they’re trying to get

across in here is that when we make those decisions to accept these

unproven technologies into a product development—and maybe I

need to explain product development better. Product development

is the engineering and manufacturing development phase in this

context. In the engineering and manufacturing development phase,

we should be focusing on developing and manufacturing the final

product. In this case it’s the full-up plane with everything on it.

What we do, or what we have done historically in the Department of Defense, is we go into that phase, which is difficult in and

of itself if you’re using all proven technologies, and to try to inteVerDate 11-MAY-2000 11:50 May 24, 2001 Jkt 000000 PO 00000 Frm 00039 Fmt 6633 Sfmt 6633 C:\DOCS\70782.TXT HGOVREF1 PsN: HGOVREF1
36

grate a bunch of new technologies into a final product is still not

an easy process. What we do is we allow immature technologies

that are pacing items. These are defined, the ones we had up here,

critical technology areas. Critical means they’re in the critical path

to success. We allow that immature technology, unproven technology, to enter into the product development phase. What you end

up with then are long development cycles because now you have to

bring those technologies along. You have people having to focus on

technology development when we should be focusing on engineering

and manufacturing development of a product, not the subtechnologies that go into it. As I said, that’s a challenge by itself.

And in industry they create a job that’s doable by a program

manager. His job is to bring proven technologies together into a

form that gives you the product that meets the customer’s requirements. We expect our program managers to manage technology development and product development concurrently. And technology

development is invention. Invention cannot be scheduled. And we

set up tight schedules where these things have to fit in those

schedules. The money is all lined up. Money becomes the driver,

schedule becomes the driver. And what we have actually accomplished tends to fall by the wayside.

Mr. SHAYS. Let me recognize Mr. Blagojevich, the ranking member. I didn’t—if you have an opening statement you want to make

I’m happy to have you do it, or we can get right into questions. But

he’s your witness and you have him for as long as you need him.

Mr. BLAGOJEVICH. Great. Thank you, Mr. Chairman I’ll just dispense with the opening statement and ask a couple of questions of

Mr. Rodrigues.

[The prepared statement of Hon. Rod R. Blagojevich follows:]

37

38

39

40

Mr. BLAGOJEVICH. Mr. Rodrigues, the chairman briefly mentioned that if we slow down the effort to get into the engineering

and manufacturing phase—that’s essentially the gist of what your

argument is—that does not necessarily slow down the entire program. And it’s conceivable, is it not, that if we slow down in the

technological development phase, that in the long run the time

saved early on to get it right could actually shorten the process?

Can you speak to that.

Mr. RODRIGUES. That’s exactly the fundamentals behind everything in the practices that we’re laying out from the work that we

have done on best commercial practices: that putting the time in

to get the technology match up front makes the product development cycle doable. Instead of having programs that extend for 10

and 12 years, that are absolutely unmanageable—if I could share

with you—would you put up the chart on the 10 years? This is

what happens to you as programs stretch and grow—and this is

DOD data.

Now, I would contend that it’s very optimistic, but what their

data show is that on average you’d have an 11-year program cycle.

In 11 years you go through 4 program managers, 5 program executive officers, 8 service acquisition executives, 8 defense acquisition

executives, 5 chairmen of the Joint Chiefs of Staff, 7 Secretaries of

Defense, 3 Presidents, and 11 cycles of coming up here to get

money and going through the Department to get money. And we

wonder why programs drag on and take forever?

One of the problems we have is our cycle times becoming so long

in the product development because we’re doing technology development and you should expect problems. We see them. I mean, this

body approved entry into production on the F–22 with a single

flight hour. And we expect that to stay on schedule? And we expect

that to stay on cost? We knew virtually nothing about the capability of that plane in terms of demonstration.

And what we have are a lot of hopes pinned on judgment about

an ability to deliver those things. Judgment is not good enough in

a best commercial practice. Demonstration is what counts. If you

don’t demonstrate it, you end up with 12-year cycles and you end

up with these problems and you end up with absolutely no accountability.

We have to match cycle time to program managers’ tenures to be

able to hold people accountable. Give them a job that’s doable,

which is product development when we’re in product development.

And the only way you can do that is separating technology development from product development. And the only way you can legitimately do that is through a metric that allows you to absolutely

measure demonstration.

Mr. BLAGOJEVICH. I know that chart—three Presidents, seven

Secretaries of Defense, four program managers, the only ones that

last 11 years are generally the Members of Congress.

Mr. RODRIGUES. And, unfortunately, me.

Mr. BLAGOJEVICH. And you. Can you talk about the importance

of the continuity of keeping a program manager involved in a program like this, how it works in the commercial sector and how successful that is to have that continuity?

41

Mr. RODRIGUES. Clearly, as I look at these and try to figure out

what can you control here, the only one you could really control

and would really have a direct effect is the program manager. Program executive officers have a whole portfolio; some of the others

are political appointees, you can’t control that. But the program

manager tenure can be controlled. And if you don’t have that, what

you have is a situation we’re in now. The Joint Strike Fighter is

on its third program manager. It’s only been around for 31⁄2 years.

It’s on its third. The third one is there now. His job is to deliver

this program into engineering and manufacturing development as

scheduled. The money is there. They’re ready to go. Everything is

lined up. It’s in all their outyear budgets.

We talk about wonderful plans for modernization and the need

to be able to get a new plane. And I agree with that. We need to

get new planes out there. My problem with the way we’re going to

do this one is it won’t get there in time. It won’t get there in quantity and we won’t reach modernization. This isn’t what’s going to

get you there.

In terms of matching that program manager’s tenure—and it is

what gives you accountability, we have to give these people doable

jobs. Put them in there for the tenure of the program to deliver the

product so we can get focused on the product. Our focus now is on

the next increment of funding, the next milestone. How can people

stay focused on delivery of a product that’s 12 years away? I can’t

think of anybody. To tell you the truth I have been doing this, leading this work for over 12 years. I could not have focused 12 years

ago on where I am today. There is no way. If you told me this is

what I had to do, I would have told you you were crazy. It just

doesn’t happen. You focus in shorter increments. People don’t think

that way.

Industry has gone from basically an 18-month to a 41⁄2 month

cycle. The reason for that is to get that focus on product, give people doable jobs, get the stuff out there quickly because stuff turns

over fast in terms of technologies. And rather than have these huge

cycle times that lead to obsolescence—I mean, we’re spending hundreds of millions of dollars right now, on the F–22, to buy parts

that are already obsolete. And we haven’t even built the plane.

When you’re looking at 12- to 15-year cycle times, that’s what’s

going to happen to you. We have to shorten that.

I would submit as we did during a hearing before the Senate

Arms Services Committee, Subcommittee on Readiness and Management—that in developing a model that would deal with these

issues. One of the things that you have to do is you have to put

a strict limit on the engineering and manufacturing development or

the product development phase. And 5 years, I think Dr. Gansler

has said 67 months at this point, is their goal. Five years is a doable thing. It allows you to match tenure. It brings accountability.

It brings focus to the process. And it forces you to do trades on the

front end. If you have a limit that you’re going to be held accountable to, then you will have to do the trades of the technology and

costs on the front end that allows you to do a deliverable product

in a 5-year cycle.

Mr. BLAGOJEVICH. One final question, Mr. Rodrigues. We like the

Joint Strike Fighter, you and I. You certainly—you have said you

42

do. I certainly think it’s a good idea. We’re for this. We want to see

this get done sooner rather than later. It’s good for our military,

good for our national defense. We agree to that. Right?

Mr. RODRIGUES. Right.

Mr. BLAGOJEVICH. What do you say to the argument that if we

adopt what you’re recommending as opposed to what the Department of Defense is recommending, this can hamper the ability to

keep the support here in the Congress to be able to fund this program?

Mr. RODRIGUES. Unfortunately, I think that’s a real risk. I mean,

it requires an understanding on the part of the Members. The fact

of the matter is if we launch with immature technologies, you will

not get this program on the schedule or on the cost. We will end

up with the problems that we have had historically by going into

product development with immature technology. You cannot schedule invention. It will create problems. And if they’re critical path

items, they become the pacing item. And the cost of that is phenomenal.

Let me give you some idea of this. Right now on the Joint Strike

Fighter if you were to annualize numbers and take a look at it on

a single contractor—because that’s where we’re going to go in

EMD, down to one contractor—right now in the risk reduction

phase, we’re spending the equivalent of about $265 million a year

to do risk reduction. That includes the demo and all the other risk

reduction activities on these critical technologies. To fail at $265

million a year on something is kind of OK. When we go into EMD,

our costs for a single contractor immediately jumps to an

annualized rate of $1.2 billion and within 2 years of that we’ll be

at $4 billion a year.

Now, you have a problem with a critical path technology at that

point you are carrying a $4 billion program on your back. Not $265

million program, where you can tolerate some problems in invention. And you know the basic cost tradeoffs are for every dollar

mistake I make in the risk reduction or technology development

phase, if I correct that mistake in the engineering and manufacturing phase it’s going to cost me $10, and if I wait until the production phase to correct that same problem—which the Department is

doing on the F–22s without having not completed the development

phase—here that $1 mistake now costs you $100 to correct. So this

has to do with bringing affordability and discipline to the process

to get what we need, which is a modernized fleet. And I just don’t

see where following the practices of the past are going to get us to

where we need to be.

Mr. BLAGOJEVICH. Thank you, Mr. Rodrigues.

Mr. SHAYS. Thank you. Mr. Terry.

Mr. TERRY. This is a very interesting discussion. Before I ask a

couple of my questions I’ll say that I took a tour here a couple

weeks ago of a company in my hometown that’s one of the leading

high-tech support for our private sector industries across the Nation. I mean—and they made a comment during their presentation

when I was being wowed by their technology. They said, ‘‘Our clients expect us to be cutting edge but not bleeding edge.’’

43

And as I’ve listened to your testimony and thoughts about immature technology, that’s really what we’re talking about is getting

ahead of ourselves in the technology and perhaps slowing down.

I’ve got several questions but I’ll narrow it down to a couple here.

And one is, just so I can put it in kind of a economical or simple

terms, I see a catch–22 here where we in Congress expect out of

our Department of Defense, and I think the people of this country

expect if anyone is going to be on the bleeding edge or that’s acceptable, it would be our Department of Defense. They want that

type of technology in there.

Now, assuming that they have those type of political pressures,

how do they develop a more acceptable best commercial practice

time line? It seems to me that they never can. How do we do that?

Mr. RODRIGUES. Right.

Mr. TERRY. Taking in the desire to assure the public that we’re

using best commercial practices, but then also accepting the role

that we want them to be on the far beyond cutting edge to maybe

the bleeding edge of technology that’s always moving.

Mr. RODRIGUES. Right. I don’t see a disparity between these two

things. What we’re talking about is how you go about getting a better product. It’s really about how do you get those technologies,

those more advanced technologies in the field faster. Do you do it

through a more iterative process or do you go for everything that’s

out there. The latter takes us 20 years to get there and, by the

time you get there, you can’t support it. It’s obsolete. You have got

to buy out stuff because nobody builds this anymore. It really is

about getting down to the basics of changing the culture and the

incentives.

But the other part of it is, you can be on the bleeding edge but

be on it in technology development. Go chase those technologies.

You need parameters around what you’re going to invest in, how

you’re going to manage that technology base and how you’re going

to graduate that technology to a product, and when is the right

time to bring it into a product. I’m not saying don’t chase technologies. I’m saying don’t chase it as part of a product development.

That has cost way too much money.

Mr. TERRY. And hence, then, you’re doing your job in showing us

or talking about the best commercial practices and how to institute

that in the system. But then how do we justify, as Congressmen

or the President or Department of Defense, you know, all right, we

identify what we need in this plane, let’s manufacture it today in

our Joint Strike Fighter force. Then the technology changes 2 or 4

years after production. That makes it obsolete and we end up

spending more money in a production of the next generation, sooner than we probably should have. How do we justify that? I’m kind

of asking you a political question. That’s not your realm here.

Mr. RODRIGUES. I think the Department is moving to—and probably Mr. Soloway will talk about it—is the problem with the rapid

changes in technology. It’s a reality. You can’t change that. People

have to recognize that that’s the situation. And then what we really need to do is go to a more iterative process in how we build

things. In other words, we don’t decide today on a firm design that

we’re going to build for 20 years. I mean, that isn’t how the world

works. In fact, that isn’t how we really build planes. If you look at

44

the F–16, we built A’s and B’s and block 10 and block 20, block 30,

block 40, block 50. Now we’re at block 60. Now we’re getting ready

to build block 60-I for somebody else. I mean, in reality that’s what

you end up doing. Things change and you make those changes in

the subsystems. But that basic design that you commit to has to

last longer than just a few planes so you can get your money back

out of it.

So there are ways to do this. To me, it mostly rests with the Department. It’s what they come up here and sell. Unfortunately, we

have a system now that incentivizes everybody to come up here

and promise you the world and promise you all these wonderful

things and lay out way early in a process an exact commitment to

schedules and costs, build all that funding in. And you know when

they’re doing that? They’re doing that before they even get into

that technology development phase. They don’t have the slightest

idea of whether those things are going to work or what they’re

going to cost. But we lock into a commitment early on.

And one of the things that needs to be done is to delay the commitment until you have the technology match. We shouldn’t allow

people, to sell a product based on a technology that’s absolutely

unproven. We’re raising everyone’s expectations to levels that are

going to—— Mr. TERRY. And during your presentation when you were talking

about that and answering the chairman’s questions, I wrote a note

down: So what responsibility does Congress have? Because I think

we encourage that. So maybe we should ask you to do a GAO report on what type of congressional or political reforms we need in

house so we don’t encourage that type. I agree that in trying to

abide businesslike procedures, we don’t encourage them from our

side to do it either.

Mr. RODRIGUES. We have done some pieces on this whole issue

of what incentivizes the culture and the process at DOD, and Congress is a big player on that.

Mr. TERRY. Before we run and vote, let me ask this one simple

question of you. In your testimony you had mentioned and use kind

of colloquial language here, but they said—they’re talking about

adopting the best commercial practices, they’re talking the talk but

they aren’t walking it yet. How do we ensure that they are implementing these type of best commercial practices at the highest

level instead of just talking about it?

Mr. RODRIGUES. I think you have to do it program by program.

And the Joint Strike Fighter is the lead program and it is the one

to do it on. And in the report I believe that you’re going to release

here at the hearing, we have matters for congressional consideration. And those matters for consideration spell out what we think

you should require the Secretary of Defense to lay out for you—and

we need a metric that is understandable.

Let me talk about technology readiness levels. They are actually

very simple. I think that’s why people don’t like them because

you’ll actually—you will understand them. And to simply say what

they are, there are nine levels. What we’re talking about is progressing to the level of 7 for entry into product development or engineering and manufacturing development. The first three levels

are basically paper studies.

45

Now, I don’t know how many people want to launch a whole big

aircraft program based on a paper study but I’m not too fond of it— paper study of a new technology. Not a good idea. The next two levels, 4 and 5 are basically laboratory hardware demonstrated in a

laboratory environment. These are pretty easy things. I don’t know

how many labs you’ve been out to, but I’ve been to laser labs and

I was very disappointed. I thought I would see a laser. All I see

is these parts spread all over tables. They’re hand wired. They can

do the stuff you have to do in a lab. It’s only in that lab environment. It’s not going to be flying on a plane like we want it to, or

space aircraft.

And then to get to level 6 and 7, you actually have to get that

big thing that’s spread out in a lab down to the form and the fit

that it will have to be in in the intended product. You have to have

actually taken it out into the environment you’re going to use it in.

That doesn’t mean you put it on the plane it’s going to be on, because the plane doesn’t exist. That means you put it on a plane,

if its something that has to fly, and you actually fly it.

So now you have the form and fit that you’re going to need for

the product, you put it into the environment—we’re not talking

about operational testing. We’re saying understand the environment. That’s level 6 and 7. These things are understandable. They

are things that you can measure fairly readily. They are not subject to engineering judgment or consensus, which is what we have

tended to use or what we have used in the past that gets us into

so much trouble. And they are things that you as a board of directors need to begin to understand and to focus on and to hold the

Joint Strike Fighter to.

Mr. TERRY. What is the timeframe for the product development

timeframe?

Mr. RODRIGUES. Right now they’re planning on going into—— Mr. TERRY. That you would recommend.

Mr. RODRIGUES. Oh, I would recommend that we limit it to 5

years so we can bring accountability to the process. It would be

hard with an 8 or 10-year cycle to say we’re going to put a program

manager in there and we expect him to stay there for 10 years. I

mean, life just doesn’t happen that way.

Mr. TERRY. Thank you.

Mr. SHAYS. I’m sorry, we’re going to have to keep you here. We’ll

have just a few more questions afterwards. We have to vote. We

stand in recess. We’ll try and be back in 10 minutes.

[Recess.]

Mr. SHAYS. Mr. Rodrigues, the hearing is reopened. I’m curious,

that took me 15 minutes? So next time I’ll say 15. I’m sorry.

I want to just establish some basic points and I’m not looking for

long answers. One of the points I want to establish is to understand what I think I understand but to just have you describe it.

It’s not an uncommon practice for a building to be built before it’s

been fully designed. And the argument is you can build it faster

and you can save yourself from increased costs, inflation rate and

so on.

I make the assumption that when you’re building a building and

you haven’t designed it, you still have all the technology there. So

that kind of argument isn’t compatible where you kind of build beVerDate 11-MAY-2000 11:50 May 24, 2001 Jkt 000000 PO 00000 Frm 00049 Fmt 6633 Sfmt 6633 C:\DOCS\70782.TXT HGOVREF1 PsN: HGOVREF1
46

fore you’ve totally designed. Here we’re trying to make sure that

we know that technology exists, obviously, before production, but

even before development; is that correct?

Mr. RODRIGUES. Yes. Before product development, yes.

Mr. SHAYS. And what I want to understand as well is that with

the F–22, that basically is a new airplane, we’re not following this

practice that we’re following with Joint Strike Fighter. The F–18

which we are—which the—E and F, and F is a modification, a larger F–18, but a modification of existing technology. It’s not a new

plane, correct?

Mr. RODRIGUES. There’s very little commonality between the F–

18C/D and the F–18E/F. The Department sold that as an ungraded

F–18, but there’s nothing the same about it.

Mr. SHAYS. So I didn’t get the answer I expected. But basically

you consider it a pretty new plane.

Mr. RODRIGUES. Yeah.

Mr. SHAYS. We’re building the B–22 but we made a decision that

we would not build the AFX which was to replace the A–12 Navy

attack plane. We dropped the multi-role fighter. So we are now

going to be developing the JSF and it is going to have to—the JSF

is going to have to fulfill a lot of very different roles. It’s going to

be used by the Air Force in conventional takeoff and landing. It’s

going to be used by the Navy for shorter takeoffs on carriers, and

my understanding is it has to be a tougher plane and it’s going to

weigh a little more. And then we are going to be using it for the

Marine Corps and the U.K. Royal Navy as a vertical takeoff, like

the Harrier jet. And yet it’s all coming from one program. So there

are a lot of technologies that are in play here.

Your bottom line point is that we should not develop and produce

this plane until we follow the game plan, which is to make sure

the technology exists and that we use best commercial practices.

The first question that I want to ask in regards to best commercial

practices, what best practices would not be appropriate in the DOD

acquisition process? Can you think of any?

Mr. RODRIGUES. No I can’t think of any at all. Really what it

then comes down to—and I think it was something that came out

in an earlier question—it really comes down to how do you

incentivize the process to put these things in place. What do we do

to make it so that bringing knowledge to the table rather than

judgment, to really be focused on setting a product development up

for absolute success, how do you create the incentives to do that,

to not oversell, to not overpromise, to not overcommit, but do what

technology allows?

Mr. SHAYS. How much cheaper is it to wait now to develop the

technology than to begin to develop or produce this plane without

the technology?

Mr. RODRIGUES. As I said, the rules of thumb are if you run into

problems when you’re in technology development, $1 problem there

becomes a $10 problem when you enter into product development.

And if you actually get all the way to manufacturing or production

and you’re still doing technology and you’re having problems, that

same dollar problem now becomes a $100 problem. So——

Mr. SHAYS. So a 10-to–1 ratio and then a 100-to–1 ratio.

47

Mr. RODRIGUES. Order of magnitude increases, yes. And some estimates are much higher than that. Commercial industry, when

you’re doing huge runs of production items where you have to do

recalls, those numbers increase dramatically.

Mr. SHAYS. Now, to the best of your knowledge—you’re speculating—what would motivate, in your judgment, the military, the

DOD to move forward? And if—I always believe there are logical

reasons why they want to move forward. Do you think it’s the potential that they think Congress might withdraw if they’re not

heavily committed, and therefore they’d rather get the plane this

way, even if it costs more and they get less planes? Because if it

costs more they’re going to get less planes.

Mr. RODRIGUES. Right.

Mr. SHAYS. Or is there another reason that I’m not thinking?

Mr. RODRIGUES. No, I think that is it. It is the incentives. The

money is in place. It’s very difficult to get money in wedges and

approval and get something going. I mean, picture a program manager that comes in now. His job at this point in time on something

like the Joint Strike Fighter, is to get the program into engineering

and manufacturing development on schedule. The money is all

lined up. In fact, the program manager leaves as the down select

is made, and the process is handed over to somebody else who is

going to have to worry about this.

But let’s imagine that he takes a look at it and he says, well,

wait a second, you know, some of these technologies we really don’t

know all that much about it, so what I want to do is propose a

delay. Well, now he has to go through the process of the Department of Defense where there are all kinds of people competing for

these limited dollars waiting to find a place where they can take

it from. And the Department does it itself all the time. Then it

comes up here to the Congress and there are a lot of people looking

to fund other things and there’s only so much money to go around.

That risk is absolutely real.

Mr. SHAYS. It’s a real risk but it’s even more of a risk if the F–

22 and the F/A–18E and F are going to be much more costly than

we anticipated because they’re not following these kind of practices.

So in a sense their Joint Strike Fighter is going to be following a

process that should save money in the long run or certainly not add

to cost, competing with 2 other weapons systems that may gobble

up costs. So I have sympathy for that if that’s it.

Then I’ll just ask this last question. What is magical about a

date, these time—I mean, is there—you haven’t developed any logic

that says that a cold war enemy or a non-cold war enemy is going

to be able to beat us and supercede us. So from that standpoint we

don’t have a rush; correct?

Mr. RODRIGUES. Right. The objective was to get an affordable

family of next generation aircraft. The key was affordable. That’s

not basically threat driven at that point.

Mr. SHAYS. I’m going to just say it can be refuted by DOD. But

the bottom line is one of the potential luxuries of the cold war ending—I consider the world a more dangerous place but for other reasons—but the cold war has ended, we have some ability to develop

the technology or a lot of ability to develop the technology before

we go into development and production.

48

Mr. RODRIGUES. Absolutely. But going back, if you don’t have the

technology match, the fact that there is some threat out there you

need to deal with doesn’t justify the mismatch of technology because you’re going to run into problems when you try to build that

thing. It is going to take you longer and longer and cost you more

and more.

Mr. SHAYS. So even if the cold war hadn’t ended, you would be

arguing the same thing.

Mr. RODRIGUES. I would be arguing for some constraint in what

you do as you move forward. There are ways to deal with that.

There are some things where we don’t have capability—and I can’t

talk about those where there are real threats out there that we

can’t deal with, particularly in electronics—where we would go

ahead with a very limited scale of something to get some kind of

capability, even if we know it won’t meet the whole thing. It gives

us something in the short run to be able to deal with part of the

problem. There are cases where you can make a case for why you

would want to take those risks. I don’t see those in the Joint Strike

Fighter.

Mr. SHAYS. Thank you. Mrs. Biggert has joined us. I yield to

her—not yield to her but give her the floor.

Mrs. BIGGERT. Thank you, Mr. Chairman. I just have one question. I’m sorry I missed the testimony. But yesterday the Senate

Armed Forces Committee deleted the engineering and preliminary

manufacturing money but said that the Pentagon could get the

$424 million back if the plane proved it was ready to take the next

step. Do you think that this will be a possibility?

Mr. RODRIGUES. That they’ll take the next step?

Mrs. BIGGERT. Yes, that—— Mr. RODRIGUES. That they’ll be able to accomplish that, take the

next step within this year? Not really; no, I don’t. Not based on

where they are in technologies, provided those technologies remain

untradable from meeting the affordability goal. As long as those

technologies are there—I don’t believe you could get it done in a

year. Could they get lucky and do it? I suppose. I don’t think they

have the demonstrations laid out at this point that would get them

to the appropriate technology level to have that assurance. I just

don’t see how that would happen.

Mrs. BIGGERT. Thank you. Thank you, Mr. Chairman.

Mr. SHAYS. Thank you. Mr. Terry.

Mr. TERRY. No.

Mr. SHAYS. Is there any last—is there a question we should have

asked that you would have liked us to ask or any point that you

want to make?

Mr. RODRIGUES. There is one point that I skipped, actually two

things. One is when we talked about the differences as far as we’re

concerned in the scoring of the technologies and the Department’s

position on that. I do want to make you aware that we had a closed

hearing and we went through those technologies.

And could you put that back up again?

I want to point one thing out. One of the basic arguments was

that there was a misunderstanding that when we scored the technologies they thought we were saying in order for it to get to a

level 7, the level needed to enter EMD, that it had to be on the

49

Joint Strike Fighter aircraft. As I said, that would be an impossibility, obviously not the standard. We spent a great deal of time

working with the people who apply this at the contractor’s plants

and the program office to get this clear understanding that we

were talking about demonstrating something in close form and fit

in the environment; not on the Joint Strike Fighter, on some kind

of surrogate platform.

But if you look at this, you can see a number of those technologies are projected only to be at readiness level 4; technology

area 7, technology area 3, and technology area 2. As I said earlier,

level 4 is not form and fit, it is laboratory hardware in a laboratory

environment. So this argument about, oh, there was confusion

about scoring this because we thought you meant you had to have

it on a Joint Strike Fighter doesn’t even come into play in the scoring of those technologies.

And I can tell you we made that absolutely clear. So there is no

misunderstanding.

The other thing is what can Congress do or what should you do?

I think we should require them to be held to a level 7 of demonstration. And in those cases where they can or believe that they

should move without that, they should have to provide a very discreet—first of all, that they score it properly and own up to the

scoring. Let’s not play games. These are pretty clear things to

score. There is no confusion.

And once we score it, if the scoring comes out less than a level

7, the only way they should be able to move forward is make it

clear to you where they are on those critical technologies and then

explain to you why it is that we need to take on that additional

risk of moving forward without having demonstrated the technologies that are going to be critical to the building of that final

product, taking those technologies and having the very difficult

task of trying to integrate a whole bunch of technologies into a

final product.

Mr. SHAYS. You wanted to make that point, but it does raise a

question now. And the question it raises for me is, is this a package

deal? Do all have to be there in order to go to that next step of development or do we isolate each one of these technologies as a separate issue before we move forward?

Mr. RODRIGUES. As long as they’re on the critical path, the pacing item in any area, whatever it is—the long pole in the tent is

what you have to worry about.

Mr. SHAYS. Listen to the question I’m asking. Are they independent or do I have to take them all as a package?

Mr. RODRIGUES. No, they’re independent. You do each one.

Mr. SHAYS. Right. I anticipated that would be the answer but I

just wanted to make sure.

Are you at liberty to tell us which areas have the greatest challenge right now? Is this a proprietary issue or not? I can’t imagine

it would be. Without identifying which is one and which is two and

so on, can you tell us the area where we’re doing the best and

where we’re potentially doing the worst?

Mr. RODRIGUES. Clearly, the best—— Mr. SHAYS. Let me ask a question. Is that an uncomfortable

question?

50

Mr. RODRIGUES. I can say clearly from this, the best are the ones

that would be technology area 1 which I think you have a thing

that tells you what that is. I can’t—I don’t want to say.

Mr. SHAYS. Fair enough.

Mr. RODRIGUES. But clearly one and eight.

Mr. SHAYS. Someone will have to explain to me, and I should

have clarified before but I’m not going to push the point, why it

even matters, why we can’t have that dialog. But, fair enough. I

guess because we haven’t made a choice on who gets what.

Mr. RODRIGUES. Both contractors don’t work the same technologies.

Mr. SHAYS. That’s fair. You’ve been wonderful. You’ve put the

ball in play and you’ve given an opportunity for those that follow

to answer your points. I don’t know if you or someone else can stay

to hear the other presentations because we might seek to have your

office respond. You preferably, but someone else.

Mr. RODRIGUES. I’ll stay.

Mr. SHAYS. We’ll go to our second panel.

Is that OK?

We’ll go to panel two. And I thank you, Mr. Rodrigues.

We have Mr. Stan Soloway who is the Deputy Secretary of Defense Acquisition Reform, Department of Defense, and Major General Raymond Huot, U.S. Air Force Acquisition Program, Department of Defense. If you both would stand, we’ll swear you in. What

I think I will do is just slide you over a little bit. Mr. Soloway, if

you could just move your chair a little bit so we give Mr. Huot—

General Huot, I’m sorry General. If you raise your right hands,

please.

[Witnesses sworn.]

Mr. SHAYS. Thank you. Gentlemen, it’s great to have you here.

I appreciate your spending the time with us. And I want to make

sure that, Mr. Soloway, we have you first. Am I breaking protocol

or am I keeping protocol?

Mr. SOLOWAY. No, that’s correct.

Mr. SHAYS. I’m in charge. Well, Mr. Soloway, we’ll start with

you. Thank you for being here.

STATEMENTS OF STAN Z. SOLOWAY, DEPUTY SECRETARY OF

DEFENSE, ACQUISITION REFORM, DEPARTMENT OF DEFENSE; AND MAJOR GENERAL RAYMOND HUOT, U.S. AIR

FORCE, ACQUISITION PROGRAMS, DEPARTMENT OF DEFENSE

Mr. SOLOWAY. Thank you, Mr. Chairman and members of the

committee. It’s a pleasure for me to be here today. If I could just

divert for a moment from my prepared text and say that I do believe that there is extraordinary commonality between what the

GAO is recommending and what we are looking at in terms of a

strategy of how to deal with the issues before us, and hopefully

we’ll have a robust discussion of that as we move forward.

But I am pleased to be here to have this opportunity to discuss

with you our continued progress with acquisition reform and particularly how it relates and applies to the Department’s Joint

Strike Fighter program.

51

As you know, acquisition reform has been a top priority for the

Department for the last several years and encompasses a wide

range of initiatives and has had many real successes. Let me list

just a few. One excellent example is the Joint Direct Attack Munition [JDAM], which performed so flawlessly in Kosovo. Designated

as an acquisition pilot program, JDAM was originally expected to

cost in excess of $40,000 per unit, but through a combination of acquisition reforms and focused innovative program management we

can now purchase JDAM for less than half that amount.

Mr. SHAYS. That’s the unmanned plane.

Mr. SOLOWAY. No, JDAM is essentially a guidance, is the best

way to describe it.

General HUOT. Probably the best way to say it, it’s a strap-on kit

for a general purpose bomb. It gives it all weather capability. INS,

Inertial Navigational System, with global positioning system guidance.

Mr. SHAYS. Gotcha. I’ve seen it and I appreciate having it clarified.

Mr. SOLOWAY. Then there is the precision location GPS receiver

or PLGR. This receiver, purchased largely through the new commercial buying authorities contained in the Federal Acquisition

Streamlining Act, replaces a previous field version built to extensive military specifications that weighed over 30 pounds, required

two operators, had only one channel and cost us thousands of dollars per unit. PLGR, on the other hand, requires only one operator,

has five channels, weighs just over 2 pounds and costs less than

$1,000 per unit.

The All Ordnance Destruct System [AODS], is a flight termination system used on the rocket system launch program vehicles

at the Space and Missile System Center. The majority of these vehicles are used as targets in support of the ballistic missile defense

programs. By utilizing the commercial buying authority known as

FAR, Part 12, unit costs were reduced from the previous purchase

price of $900,000 a kit down to $55,000 a kit, but more importantly, the new kits are also technically superior.

Today, circuit cards for the avionics in the F–22 are being produced largely on a commercial line at TRW, thus saving the Department significant resources that would have had to be devoted

to unique development and production facilities. Moreover, the reliability testing on those circuit cards have demonstrated excellent

results and costs appear to be significantly lower than expected, 55

percent to 70 percent less than their military standardized counterpart.

Acquisition reform is also central to the development of the

Navy’s new Virginia Class Attack Submarine. Key to the success

of the program has been the use of integrated product and process

development teams, the use of open systems architecture, and the

insertion of commercial off-the-shelf technologies. The Navy will

benefit there with a cost avoidance of $30 million per ship but,

more importantly, the Virginia Class will operate at a 32 percent

lower total ownership cost than the comparable Seawolf.

Indeed, much has changed and for the better. Given the complexity of our business practices and the entrenched cultures we have

52

inside and outside of the government, I believe we have made substantial progress.

But clearly we must do more. The security environment we face

is unpredictable and unstable. And our success in meeting the challenges of the battlefield of the near and more distant future will

hinge in large part on our ability to access and integrate true cutting-edge technologies that provide us the dominance, speed, and

scope of information that we need.

One of the means for accomplishing these goals is through a restructuring of how we develop, manufacture, and maintain our

weapon systems. It is no secret that cycle times for new weapon

systems from concept to fielding remain unacceptably high and

that such long cycle times too often result in the fielding of already

obsolete technologies. Since some technology decisions must be

made early in a program, it is clear that our history of taking 15

or more years to field new systems is not at all consonant with the

torrid pace of technology change we see today.

There are, of course, many reasons for these long cycle times.

Key among them is often the very nature of the requirements set

forth for any individual program. Traditionally, our requirements

have been both inflexible and involved extraordinary technology

challenges that can take many years to meet. This is beginning to

change. Today both our operational and acquisition and technology

communities recognize that to optimize support for our men and

women in the field and to most responsibly steward the public’s tax

dollars, we need to institute new requirements and acquisition

strategies. Indeed, we are confident we can significantly reduce

cycle times and costs and provide items for the warfighter faster,

for a more flexible evolutionary approach.

How we do this, how we develop, manufacture, and maintain systems is based on the Department’s 5000 Series documents, our

bible for systems acquisition, which we are currently rewriting. The

DOD 5000 rewrite will drive the Department further toward evolutionary acquisition and increase our focus on flexibility and requirements. Additionally, the new 5000 will require greater technology maturity prior to entering the manufacturing phase of a

program.

In the new systems acquisition environment, key acquisition decisions and long-term funding commitments may not be made until

technology shows the required maturity and risks are better understood and mitigated than has traditionally been the case. And the

JSF program is a forerunner of this new approach. Indeed, since

its inception, the JSF program has been recognized for utilizing

and actually pioneering many acquisition reform concepts and applying them to the actual business processes and contract vehicles

being utilized, including but not limited to the critical precept of

the new 5000 series.

For instance, modeling and simulation has been proven in both

industry and government to help reduce the time resources, and ultimately risk, associated with systems development. Representations of proposed systems, basically virtual prototypes, are embedded in realistic, synthetic environments to support the various

phases of the acquisition process. The JSF program has made extensive use of M&S in the requirements development process and

53

is continuing the use of M&S with mission level virtual pilot-inthe-loop simulation to support a more thorough evaluation of required avionics capability.

In keeping with the best proven practice in the commercial technology world, virtual manufacturing and virtual maintenance are

also being pursued to facilitate planning and drive down the associated cost.

The JSF is also incorporating an evolutionary acquisition strategy. In this process, the warfighter and the buyer work side by side

to facilitate a better understanding of the requirements and decisions on tradeoffs between performance and cost. Specifying operational requirements in an incremental manner phased over time

and matching them against the projected threat and available technologies have allowed the JSF program to exercise thoughtful judgment in balancing cost, schedule and performance.

The use of evolutionary acquisition further mitigates risks by allowing technologies to be inserted as they mature.

As I mentioned before, probably the most important change in

the new 5000 rewrite is the emphasis on technology maturity before entering into system integration, or what is today known as

engineering and manufacturing development. Of course, transition

into EMD is a challenge in every program. And in the JSF program, as in others, it will be up to the design teams and the program office along with the Service Acquisition Executives and the

Defense Acquisition Executives to determine the acceptable level of

technology readiness prior to EMD decision.

Among the many factors that can help us make overall technology readiness assessment are technology readiness levels

[TRLs], which are used sometimes in the DOD. The minimum TRL

rating of 1 begins at paper studies, as Mr. Rodrigues said, of a

technology’s basic properties and rises to the maximum rating of 9

for a system in its final form, operating under mission conditions.

However, there is no hard and fast rule in DOD and NASA or

elsewhere in government as to specific threshold TRLs for any

given decision. Additionally, it should be noted that there are more

comprehensive methodologies that we do use which can and do provide more value as risk management as opposed to risk measurement tools. Indeed, the DOD 5000 rewrite does not prescribe a required technology readiness level but does recommend using TRLs

as a tool to help measure the maturity level of the technology.

What the 5000 does prescribe is that technologies be demonstrated in a relevant environment with a fallback plan at a higher maturity level. In other words, if a far-reaching, newer technology does not pass relevant testing, a lesser proven technology

could be utilized as long as it enables the system to still achieve

its critical performance requirements.

Recently the GAO provided a draft report to the Department on

the JSF that recommends extending the JSF development schedule

to allow for further maturation of technologies. Their recommendation is based on their understanding that critical technologies will

have inadequate levels of technology maturity based on TRLs at

the time the EMD contract is to be awarded and the decision made

in the spring of 2001.

54

The GAO report clearly articulates a strategy for systems development that we embrace, as evidenced by the revisions to the DOD

5000 I mentioned earlier. Indeed, for the most part, we are in violent agreement. Where we differ is on the definition and applicability of TRLs in the decisionmaking process. GAO’s position does appear to be that achieving a TRL level of 7 should be required prior

to entering EMD, again as Mr. Rodrigues pointed out. Our view,

however, is that a level 7 requires the very kind of systems integration that takes place during EMD and that it is infeasible to

produce full-scale testing of this type prior to that phase.

Moreover, as noted earlier, we, like NASA and others, see TRLs

as but one input to the decision process. In fact, as the NASA Director of Programs told us, ‘‘NASA does not formally use nor rigorously apply the definition of technology readiness level, TRL, to its

systems development. We generally commit to development of operational systems when all critical technologies have achieved TRLs

of 5 or 6.’’ In assessing a technology prior to EMD, we do seek to assess

whether the individual technology has been proven or is close to

being proven in appropriate developmental environments. We do

not agree that it is necessary or even feasible to demonstrate the

full integration of the technology prior to EMD. Indeed, the GAO

report does seem to require that all technology must be flown on

an actual or prototype JSF platform in order to demonstrate adequate maturity for EMD. However, much of the JSF avionics and

software, for example, can and should be demonstrated on the

ground at a lower cost. Technologies that must be flown for adequate demonstration will in fact be or have been flying on the concept demonstrator aircraft, commercial aircraft, the F–16, the F– 22, the F/A–18E/F and the Eurofighter.

So, in short, although the technology will not be demonstrated on

the actual JSF or a JSF prototype per se, it will be tested in a relevant environment. And the success of those demonstrations is critical to our confidence in our ability to successfully integrate the

technologies during EMD. And we are confident we will have all

critical technologies at an adequate and appropriate technology

readiness level by the time an EMD decision must be made next

year. I emphasize the decision is not to be made for another year.

GAO has also expressed concern that if we make a premature

production decision we could be locked into manufacturing processes based on an expected technology capability, thus creating the

risk that if the integration fails we will face exorbitant cost and

time in redevelopment.

We agree there is risk. There is always risk in the integration

of complex systems. And how the JSF program mitigates those

risks is the key. As I think you will hear from Major General Huot,

the risk mitigation initiatives associated with JSF have probably

been the most comprehensive and aggressive of any DOD program

ever.

Further, as I noted before, the rapid maturation of modeling and

simulation capabilities has enabled the development and testing of

a wide range of critical technologies that might not otherwise have

been possible and has played a key role in our ability to effectively

assess the risks on the JSF program. Thus, any decision to proceed

55

into EMD and production of the JSF will carefully assess all risks

against the fallback alternatives.

Finally, let me be very clear. The strategy of revised acquisition

process that will be prescribed in the DOD 5000 rewrite and which

is largely being implemented on the JSF does represent a very real

departure from our traditional approach to systems development.

In the past, we would indeed consider technology development as

part of the engineering and manufacturing phase. Now we have

technology development before we make a commitment to—proceed

into system development and demonstration, and eventually production.

Mr. Chairman, acquisition reform has been made possible by a

strong partnership between the Congress and the executive branch,

and reflects our mutual commitment to ensuring that the government operates far more efficiently and effectively than has historically been the case. I appreciate having had this opportunity to be

here today, because as we continue on the successful path we have

forged, we need the continued support and commitment of the Congress.

We also appreciate the continued support and encouragement the

GAO has provided as we continue to change the worlds’s largest

buying institution. Our disagreement on the specific role of TRLs

notwithstanding, we are in close agreement on where the DOD’s

acquisition process must go. We also believe it is important to let

programs like JSF that are demonstrating the acquisition and

management reforms of recent years to have the flexibility to manage their programs and make decisions based on weighing the risks

against costs, schedule, and technology maturity.

That concludes my statement. I would be happy to answer any

of your questions now and certainly stand ready to provide any additional information the committee believes would be helpful in fostering a clearer understanding of this or other DOD programs.

Mr. SHAYS. Thank you Mr. Soloway. That was a very helpful

statement. We did allow you to go over the 10 minutes because I

think it’s important for you and DOD to really state your case on

the record. Then we can examine your statement.

[The prepared statement of Mr. Soloway follows:]

56

57

58

59

60

61

62

63

64

65

66

67

68

Mr. SHAYS. I don’t think I’ll be as generous with the third panel.

We’ll stick with the 10 minutes.

General, now you only have 2 minutes.

Mr. SOLOWAY. That’s the way it usually works between OSD and

the services.

Mr. SHAYS. General, you have your time. And we welcome your

testimony.

General HUOT. Thank you, Mr. Chairman and members of the

committee. I appreciate the opportunity to talk about the Joint

Strike Fighter program and the significant accomplishments it has

demonstrated in the area of acquisition reform.

By way of background, for those of you on the committee who I

haven’t had the opportunity to meet, I have been in the operational

end of the warfighting business for most of my career. I flew the

F–105 Thunderchief in combat in Southeast Asia during the Vietnam conflict. Since then, I have flown several other attack aircraft,

the A–7, A–10, F–18, and almost every model of the F–16. I was

a wing commander during the Persian Gulf war. And since then,

in my headquarters experience, I’ve been the Deputy Director of

Operational Requirements and almost 2 years in this acquisition

position as the mission area director responsible for fighters, bombers and munitions, and the JSF program falls into that category.

Now, the JSF program has been a leader in the implementation

of acquisition reform since its inception as the Joint Advanced

Strike Technology Program or the JAST program. You may recall

that we combined Air Force and Navy requirements after we cancelled the multi-role fighter program and the AX, helped them

meet joint service requirements.

Now, the need to produce an affordable aircraft drove several key

decisions early in the program, including a single engine, a single

seat, and a common family of airplanes for all three services. Specific advanced technologies were selected and then prioritized based

on their contribution to not only warfighting benefits but also in

terms of life cycle costs.

We established aggressive cost goals for average unit recurring

flyaway costs and for engineering and manufacturing development.

And cost goals are being established even now for life cycle costs.

These cost goals continue to serve as baseline independent variables for requirements and technology affordability trades. And I’ll

talk about that regard in a minute because this is a new process

that we have gone through. It is a robust and highly successful cost

and operational performance trade process has been implemented

by the Joint Strike Fighter on a continuous basis in order to help

achieve those cost goals.

Participation by the three major stakeholders in this process—

our warfighters, industry, and the government has been key to the

success of this cost and operational performance trade process. The

warfighters, represented by both operators and maintainers, continue to provide a clear and unambiguous view on fighting concepts

and needs as well as expected threats in combat conditions.

And industry provides us with detailed weapons system concept

and cost information and the program office adds an understanding

of cost schedule performance supportability and resource constraints.

69

More importantly, government engineers and analysts provide

assessments on the cost and performance of contractor concepts.

The result is that every requirement on this airplane has to earn

its way onto the airplane based on cost effectiveness or cost-benefit

analysis.

Now, in contrast to some programs which are not initiated until

a formal validated requirements document existed, the Joint Strike

Fighter program was established so that specific weapons system

requirements would not be frozen until the leveraging cost and

operational performance trades had been performed and key technologies and concepts had been matured. This just-in-time approach to requirements avoided premature commitment to requirements that would be excessively costly to meet, fail to take advantage of available technology or, conversely, depend upon immature

technology. In other words, it allowed time to work down the cost

of the weapons system and insures that the requirements are consistent with the available technology.

I would like to spend just a few minutes discussing the current

phase of the JSF Program Concept Demonstration Phase we’re in.

This phase began in November 1996 and it’s scheduled to be complete in the spring of next year. The major activities during this

phase are propulsion system development, requirements analysis

and definition, the technology maturation programs and, of course,

building and flying two concept demonstrator aircraft per contractor.

In regards to the concept demonstrator aircraft, I want to emphasize a couple points to the committee. These aircraft are concept

demonstrator aircraft. They are not and were never intended to be

prototypes. These demonstrator aircraft are required in the concept

demonstration phase to accomplish three very specific objectives:

One, to demonstrate a high degree of commonality across all three

common service variants; to demonstrate short takeoff, the vertical

landing, hover and transition to and from forward flight; and then

demonstrate satisfactory low-speed carrier approach flying and

handling qualities.

In fact, there are other items to be demonstrated by the contractors on these airplanes but these are for competitive advantage

purposes. But the hard requirements are the three things I listed

above.

We are totally convinced the JSF program office and our contractor teams have done a great job in ensuring that the concept development phase of this program will demonstrate a low level of technology risk for critical enabling technology and processes prior to

entering EMD in the early part of next year on the current schedule.

This has been accomplished through a very rigorous and disciplined methodology for risk assessment, risk management, and

risk reduction. Now, Pratt & Whitney is developing engines for

both the JSF contractors’ demonstrator aircraft based on their

highly successful development program for the F–22. The result

here is a high degree of commonality not only among the JSF contractors but also between the Joint Strike Fighter and the F–22.

The F–119 engine core is essentially the same for both the F–22

and the Joint Strike Fighter. This commonality lowers JSF risk,

70

development time, life cycle cost, and accelerates that propulsion

system maturity so we can provide a safer airplane for our

warfighters.

Under the current schedule, in fact, we expect to have over

500,000 hours of operational F–22 engine time before we have that

engine at initial operational capability in the JSF. This is more

than double what we had on the F–100 engine in the F–15 when

we transitioned to the single engine F–16.

I would like to just talk a little bit about the last two concept

demonstrator—concept development goals—technology maturation

and concept demonstrator aircraft, a bit further. The JSF program

identified numerous technology maturation efforts to ensure low

risk into engineering and manufacturing development. Now these

technology maturation efforts aim to fulfill two key recommendations of the 1985–1986 Packard Commission: to apply advanced

technology to reduce costs, not just to increase performance, and to

demonstrate advanced technologies prior to the start of the EMD.

In the Single Acquisition Management Plan [SAMP], that was

signed by Dr. Kaminski in November 1996 it clearly states that the

goal of ‘‘tech mat’’ program is to evolve the most promising leadingedge technologies to a low level of risk prior to integration into the

JSF EMD program.

It’s important to emphasize that integration and its corresponding risk is, and always was, to be addressed in EMD. Exit criteria

from the concept development phase states the Joint Strike Fighter

will demonstrate to a low level of technical risk those critical enabling technologies, processes, and system characteristics. Integration on the JSF program is the focus of EMD.

Now the Joint Strike Fighter Program Office, in conjunction with

each competing contractor, identified critical technologies processes

and system characteristics required for the program tailored to

each contractor design. Robust risk management processes were established by each competing contractor and validated by the program office. The government did not specify to the contractors

which techniques must be used to track risk. The contractors selected what they felt were the best methodologies to accomplish

that task.

Now the government has been provided real-time access to those

systems, actually on a computer data base, for oversight and review during the entire phase of the program. Both contractors utilize what is known as ‘‘waterfall charts’’ using a Willoughby template. This is a common and accepted methodology in industry and

government. In fact, it’s taught at our Defense Systems Management College where risks have been identified, baselined, and

tracked to document the very specific events required to reduce the

risk of these critical technologies, processes, and system characteristics to a low level of risk prior to EMD initiation.

Implementation of that risk management strategy has not

changed since the program entered the concept development phase

in 1996. And, most significantly, all of the critical technologies have

achieved or are on track to achieve a low level of risk prior to the

start of EMD.

I want to assure the committee that DOD is convinced that our

JSF weapons system contractors are appropriately reducing the

71

risk of these technologies through an affordable mix of flight and

ground demonstrations, component demonstrations, and modeling

and simulation and analysis. Based on a request from the Subcommittee on Military Procurement of the House Armed Services

Committee, both our JSF prime contractors and Pratt & Whitney

recently reaffirmed in detailed written responses, which were also

shared with the GAO, that sufficient testing and demonstration is

in place for the Joint Strike Fighter program to enter EMD at low

risk.

In summary, clearly since the program’s inception almost 6 years

ago, the program office has been following a rigorous risk reduction

plan. The risk reduction plan is on track to reduce the risk of each

technology to low, by entry into EMD, and leave the integration of

those technologies to the EMD phase where it belongs. This risk reduction effort has been an important part in the program’s overall

goals to implement acquisition reform.

This Joint Strike Fighter is vital to the modernization of all of

our services’ air forces and many of our closest allies. Any significant delay to the program would result in increased costs and also

have serious impacts on our force structure and readiness. This

was highlighted in detail in a recent May 2 Deputy Secretary of

Defense letter to our services’ most senior leadership. That letter

was addressed to each of the service Secretaries, to the Chief of

Staff of the Air Force, the Commandant of the Marine Corps, and

the Chief of Naval Operations.

The JSF was chartered to do business differently to demonstrate

leadership in acquisition reform, and it has done this. Having embraced these concepts, it was rewarding to those who have worked

so hard on this program to be presented the DOD David Packard

Excellence in Acquisition Award in March 1997. All of this has

been accomplished under the twin goals of developing an affordable

weapons system that can meet the warfighters’ needs well into the

21st century while helping to reform the acquisition process.

So, Mr. Chairman, when you ask, ‘‘Joint Strike Fighter, acquisition reform: Will it fly?’’, my answer to you is that JSF has already

demonstrated acquisition reforms as called for by Congress, and it

will continue to write the book that future programs will follow.

Thank you.

Mr. SHAYS. Thank you.

[The prepared statement of General Huot follows:]

72

73

74

75

76

77

78

79

80

81

82

Mr. SHAYS. It’s great to have you both here and your statements

were helpful. And I would like to ask a number of questions, stating first that I think we on both sides of the table agree that JSF

is a necessary program. We’re going to see this plane built and all

its variations. And so it’s just really a question of how to proceed.

And—but I do think there are some substantive differences between GAO and your position, DOD’s position. And I’d like to investigate that a little bit. But I would also like to acknowledge, you

know, where we may agree, so then we can just really focus on

where we need to focus on.

The bottom line is that the new attack fighter, the AFX for the

Navy and the multi-role fighter, MRF, for the Air Force, is dropped

and we don’t see any hope or need to resurrect that. That’s pretty

much off the table. So it makes JSF even more important.

I think we both can agree—so you nodded your heads. I take that

as a yes. You’re going to have to speak. Actually it’s very hard for

her to take a nodding of a head. Since I didn’t address it to one,

I understand. So I’ll start with you, Mr. Soloway. And General

Huot, if you disagree, you certainly would step in. So I assume if

Mr. Soloway says—gives the answer, you are in agreement with

him and vice versa. Is that fair enough?

Mr. SOLOWAY. OK.

Mr. SHAYS. So the AFX and MRF are off the table and we agree

that JSF, Joint Strike Fighter, is our next plane, in addition to the

F–22 and the F/A–18E and F. Would you agree with GAO that the

F/A–18E and F is a significantly different plane, almost a brand

new plane, or would you contend that it is a variation on a plane

that we already have? I’ll ask both of you.

Mr. SOLOWAY. Actually, the general has the expertise.

General HUOT. Mr. Chairman, I really don’t feel qualified to provide a specific answer there. I don’t know that much about the specific differences between those two aircraft.

Mr. SHAYS. I’ve always known the nonmilitary side to be willing

to give an answer on this. So, Mr. Soloway.

Mr. SOLOWAY. But I’ve already passed to the general.

Mr. SHAYS. So that’s going to be—it’s going to stand basically,

that comment, that it is a significantly different plane by the GAO,

if it’s not refuted.

Mr. SOLOWAY. That would be my understanding, but we can certainly get you more detail on what the differences are.

General HUOT. It may be appropriate, Mr. Chairman, to go to the

Navy and answer that question.

Mr. SHAYS. OK. Fair enough. We would agree that the Joint—

JSF is intended to do three basic tasks: for the Air Force, a conventional airplane, tactical airplane; for the Navy, slow-speed structure that has to be a little more durable to take the harder landings and the shorter landings; and for the Marines, a vertical takeoff. And those are, General—I make an assumption—three very different tasks.

General HUOT. Yes, Mr. Chairman. Probably the best way to

clarify that, for the Air Force the Joint Strike Fighter will replace

F–16s and A–10’s. For the Marine Corps they will replace the AV– 8s and some of their F–18s. And for the Navy they will complement

their F–18E/F force. And in the three variants of the airplane we

83

achieve—we plan to achieve a high degree of commonality, and

right now that appears to be somewhere between 70 and 80 percent commonality between each of the three variants. There is the

conventional takeoff-and-landing variant for the Air Force. There is

the carrier-suited variant for the Navy, and the STOVL variant,

the short-takeoff-and-vertical-landing variant, that will satisfy the

needs not only for our Marine Corps, but also for some of our foreign participants, notably the United Kingdom.

Mr. SHAYS. Things like the avionics would be similar in all

three?

General HUOT. Right now the avionics are planned to be common

in all three variants.

Mr. SHAYS. So some will be the same?

General HUOT. Significant savings there in terms of commonality.

Mr. SHAYS. But admittedly we tried this in the past where we

tried to have one plane meet the needs of more than one branch— Navy, Marines, Air Force—and we haven’t always had—it hasn’t

been all that successful, correct? I know we have—— General HUOT. I would say that the F–4 is probably an example

that you might refer to where it didn’t meet everyone’s requirements.

I would point to the fact in this program and as a warfighter I

would say this has been a great process. This Cost and Operational

Performance Trades process that I talked about in my oral testimony, this is where we had what we call an ops advisory group,

an OAG, a group of warfighters that got together and worked with

the contractor and the SPO, the systems program office; and we

worked this requirements process to evolve the requirements for

the services over a 5-year period. And in fact we finally got their

final Operational Requirements Document approved by the Joint

Requirements Oversight Council, that’s a group chaired by the vice

chairman of the Joint Chiefs and with membership of the vice

chiefs of each of the services, approved on March 13th.

And as I said, what we did in there is, we went carefully through

to make sure that each of the services gets the requirements that

they need to do their individual job. So, you see, in each of these

variants there will be some unique requirements to meet unique

service requirements, but at the same time achieve the greatest degree of commonality that we can. And we think——

Mr. SHAYS. But the bottom line is—and I accept what you’re saying, but the F–4 was an example of a plane that didn’t do quite

enough for any of the branches. It’s not to say we shouldn’t do it

again to try to make it work, but this is a significant undertaking.

Because it is not just for the Navy and it’s not just for the Air

Force, it’s not just for the Marines; it’s for all three with a variation

of basically between 20 and 30 percent from one to the other.

And I would like to know if you agree or disagree with GAO

when they said that if the technology development is $1, if they

haven’t developed the technology by production development—product development, excuse me—that it’s $10, and that if they’ve got- ten to the point of production, that $1 becomes $100.

General HUOT. Mr. Chairman I have never heard those numbers

before.

84

Mr. SHAYS. So you would want us to substantiate those numbers?

General HUOT. I would have no way of commenting on those.

Mr. SHAYS. Fine. If you would speak just a little louder. But the

bottom line is you have no way. Fair enough.

Intuitively, to me, that is not an unrealistic number. In other

words I wouldn’t be surprised if that were the case, but I accept

for the time being that it hasn’t been documented. But it is now

part of the record as a statement that we need to address.

Mr. SOLOWAY. I think that we can—we will take the question for

the record also, but I am not aware of any program that we have

ever done that has had a 100 fold increase in cost from——

Mr. SHAYS. But it may just be that aspect.

Mr. SOLOWAY. Or even that aspect, I’m just not aware of it.

Mr. SHAYS. Let me clarify. Not that the whole program goes up,

but that particular failure of technology has resulted in a 10 to 1

in product development and a 100 to 1 in production. So on the

table—and your point is you’re not aware of that kind of—but intuitively we would agree that costs would go up significantly, and we

have a heck of a lot of past history to document that that’s the

case.

Now, I see a difference between the logic when you’re building

a school building your technology is there, but you haven’t designed

it. So you say, let’s do it all at once, and in many cases it’s proved

to be cheaper and you get the product sooner. You don’t wait to design everything; some things happen in process. But this is a bit

different, correct? I can’t use that same analogy, and the DOD

would be—wouldn’t use that same analogy.

Mr. SOLOWAY. What we would say, sir, on that question is that

we are in fact demonstrating the critical-path technologies prior to

going into EMD. The difference really comes down to how one defines technology readiness levels as a means of measuring where

you are in that process. We are not entering into EMD where we

have critical-path technologies that either don’t exist or we haven’t

demonstrated in what we call a relevant environment. But the difference really comes down to—and if you look, for instance, at the

way in which the technology readiness levels are defined, even in

the GAO report where at level 6 they talk in terms of system or

subsystem models or prototypes demonstration in a relevant environment, we will have done that, I believe, in every technology,

every critical-path technology prior to EMD. Level 7 speaks to a

system prototype.

Mr. SHAYS. Could someone put up that chart that GAO had. It

was figure 4 in their report. That’s the technologies 1 through 8.

Mr. SOLOWAY. My only point was going to be——

Mr. SHAYS. I know, but you were making reference to it, so let’s

just leave it. Thank you very much.

Mr. SOLOWAY. When you talk in terms of technology readiness

level 7, which Mr. Rodrigues said they believe should be the

threshold test before you go into engineering and manufacturing

development, our view on that is that definition of TRL 7 speaks

to a system prototype demonstration in an operational environment. That suggests to us that you have had to do a great deal

85

about both systems, engineering and systems integration, because

you’re dealing here with a system of systems.

Mr. SHAYS. What does it mean, though, when they say in a ‘‘test

bed aircraft.’’

Mr. SOLOWAY. A test bed aircraft can be any aircraft you’re flying

to test a given—it might not be one aircraft which—I said in my

testimony, we have the CDA, we’re using commercial aircraft, the

F–16s, F–18s, F–22, Eurofighter; various different technologies

that need to fly will fly in an aircraft——

Mr. SHAYS. But it isn’t a prototype then?

Mr. SOLOWAY. No, it’s not a prototype, though; that’s correct.

That’s the key difference.

Mr. SHAYS. The question that my counsel is asking is, who said

it was? I mean, are you putting up a strong——

Mr. SOLOWAY. I’m sorry?

Mr. SHAYS. That it was a prototype?

Mr. SOLOWAY. It says under the definition of ‘‘technology readiness levels’’ in the GAO report of last July the definition of a level

7 is a system prototype demonstration, and I think if—without trying to get——

Mr. SHAYS. Then it says, examples include testing the prototype.

Mr. SOLOWAY. Testing the prototype.

Mr. SHAYS. So we have eight basic technologies and they don’t

technically have to be tested on a prototype plane; they can be tested on another plane, correct? They could—so——

Mr. SOLOWAY. Well, you get into various discussions. For instance, on some of the—we can’t get into specific——

Mr. SHAYS. I think this is a key point.

Mr. SOLOWAY. It is a very key point, I agree.

Mr. SHAYS. And it would be wrong to suggest that somehow you

have to have a prototype. The question is, can you test it in other

ways to know it works before we move forward with development?

Mr. SOLOWAY. Sir, actually—and I’m going to give you how I

read their chart, and then General Huot may have a different sort

of interpretation.

I think there’s actually a great deal of grey area about the TRLs.

They’re not quite as clear as has been defined, and I think that’s

one of the reasons NASA and others have said you have to have

some flexibility.

Let me use the language out of the report that says, examples

include testing the prototype in a test bed aircraft. So you have to

have a prototype of the system. Now, does that mean in full form,

fit and function where, for instance, the radar array has to be the

full size, full scale that you’re going to use; or if technology allows

you to, would you be testing a variant of the radar where the basic

technology exists on a—for instance, an F–18 or some other aircraft.

Others would be in the avionics. As I noted and I believe General

Huot has talked to, the avionics don’t necessarily need to be flown

on a test bed aircraft. They can be tested in an environment which

is tremendously—has great integrity with regard to the operational

requirements.

86

So it’s not an automatic, in our view, threshold that says you

have to test everything in a prototype which involves, essentially

full scale, for each individual technology?

General HUOT. Mr. Chairman, could I make just—— Mr. SHAYS. Let me just ask this point. The laboratory goes from

one to three, correct, and then it gets out of the laboratory?

General HUOT. Those definitions become fuzzy in the sense that

in order to demonstrate some technologies on a low level, you don’t

necessarily have to fly them at all. You may be able to do that in

an environment that never involves putting that technology in an

airplane. And so what the program did—let me talk about what the

program did a little bit.

The program did not use technology readiness levels, primarily

because they didn’t include a risk management process. What they

did do is, they used these waterfall charts, or Willoughby templates

where they identified all of the key technologies that needed to be

reduced to low risk before going into EMD.

Now, these—this process identifies the technology and then lays

out over a schedule the critical events that have to happen to reduce that specific technology to a low level of readiness. In addition

to that, it provides off ramps, if you will, where if that technology

is not going to work, they would revert to an alternate technology.

This was chosen by the contractors as a way of not only assessing risk, which the TRL process does, but also managing risk and

being sure that you get to that low level of risk when you’re ready

to go into EMD. And that’s the process that they used and that’s

the process that the program office monitored each of the contractors on a—— Mr. SHAYS. Where you both agree, and then we’ll define it: You

both agree you shouldn’t go into EMD before those critical technology—before you come to low-level risk.

General HUOT. Exactly.

Mr. SHAYS. So the debate now is whether what you described as

low-level risk is low-level risk. I mean, that’s going to be—because

you buy into the point, though, that you don’t move forward if

you’re at high-level risk. And high-level risk is where you don’t

have the technology.

Mr. SOLOWAY. There’s always—as I said in my testimony, when

you’re dealing particularly in this kind of complexity of systems,

the risk of integration, which is what you do in EMD, is always

there. I mean, to say that we can whittle that risk down to a very

low level in this level of complexity is not realistic. What we are

looking to do, just to make sure we’re clear on the terminology, is

to arrive at a very low level of risk relative to the individual technologies.

Mr. SHAYS. Let me just tell you what makes me nervous. You

have described—you basically have described the technical readiness levels as tools rather than requirements. And then you’re

doing one thing more; as I hear it, you’re also in a sense redefining

low-level risk. And I thought at least we could get to the point

where we agree you don’t move forward unless you’re at low-level

risk.

You’re trying to qualify low-level risk in a way that makes me

uncomfortable.

87

Mr. SOLOWAY. I just don’t want to leave on the table a suggestion

that when you go to the integration and EMD that your risks are

automatically low. What you have done is substantially reduce

your risk by proving the individual technologies. But you still have

a high-risk element because this is a complex undertaking.

Mr. SHAYS. Well, I’m just going to tell you, that leaves me a little

uneasy because that allows you to define it in a way that allows

to you move forward, no matter, as I hear it.

Mr. SOLOWAY. In the past, we would not necessarily in all of our

systems development bring individual technologies to the level of

maturity that the JSF is doing, that we will now be requiring in

the new 5000 rewrite; that’s the individual technology levels. To

get further than that into and integrate the system is what you do

in engineering and manufacturing development. That is always— there is always risk there, as I said in my testimony.

We have reduced that risk by proving out the individual technologies. That’s the big step forward that we have taken over previous practice.

Mr. SHAYS. I feel like I’m dealing with a moving target here. I

know you’re trying to be helpful and you’re trying to be responsive.

What I want to say, and then we’ll just have our disagreements— what I want to say to you is, you view technical readiness levels

as a tool. I view them as something more than a tool. I view them

as more a requirement, and you view them more as a tool.

I view low-level risk as—obviously, when you’re dealing with

military hardware, there are risks, but I think that low-level and

high-level are pretty clear. And then there are going to be different

levels of low risk.

Now, where I may have some question is, if you’ve leveled—if so

much of one technology, say the structures and materials, you’ve

got almost everything licked, but one thing, and that one thing is

still there, but you don’t want to hold up the rest of the project because everything else is ready, I would think that you may still

have something that needs to be resolved; and you wouldn’t want

to wait another year for the whole project if you want to start to

integrate these technologies together.

So I may be a little off with that and I’ll, you know, ask GAO

to help me out on this later as well, and other witnesses. All I’m

saying to you is, I’m uncomfortable by your referring to this as a

tool and your redefining low-level risk. That’s where I’m uncomfortable.

Let me do this: I’m going to come back. I’m going to let counsel

and minority staff ask questions as well, because in the process of

their asking questions it helps to define what I want to continue

to ask.

Mr. Halloran.

Mr. HALLORAN. You want him to go first?

Mr. RAPALLO. Sure.

Mr. SHAYS. This is David Rapallo.

Mr. RAPALLO. Could you just describe what effects you mentioned

briefly in your statements—what effects a delay would have, a

delay of say 6 months?

General HUOT. I wrote down some specific notes to talk to that.

I think probably the best source of information there now—there

88

was an attachment to Mr. DeLeon’s letter to our senior service

leadership that talked about the impacts of delays.

Obviously, if you delay the program any period of time, you are

going to incur some costs of—increased costs in the program itself.

But more importantly, in this particular case, you’re looking at a

delay in development that could roll into a delay in production that

could be more than 6 months. And if you will go back and look at

the history of programs, some delays in development of a program

have led to just that.

If you were to incur any kind of a significant delay here, we

would have some significant force structure impacts and some impacts on our readiness. The DEPSECDEF letter attachment looked

at a scenario that went out as much as 3 years and said, if you

delayed the program procurement phase by 3 years, that we would

end up with about a three fighter wing equivalent force structure

shortfall that we’d have to deal with in a couple of different ways.

You could buy gap-filler aircraft, F–16s, more F–16s, or you could

do service life extensions to those aircraft if that were possible.

Now, we have done service life extensions to F–16s, so in general, we think that’s probably a ‘‘doable’’ do. How much you would

have to do would have to be determined.

In the case of the Marines, though, it appears that they would

not have that kind of an option where there really isn’t felt to be,

in my view or my understanding, a viable way to carry the AV– 8 further into the future. So they would end up with a force structure deficit in order to meet their military commitments worldwide.

And without getting into the details of what that would mean,

in terms of their ability to do their mission, that attachment in the

DEPSECDEF letter talked about some of those impacts to our

CINCs worldwide. So we think in the long run there are some very

significant impacts if you would slip this in terms of force structure

and readiness impacts.

Mr. SOLOWAY. I might add one other to that in terms of cost. The

longer we delay a decision—and I don’t want to suggest that the

decision—Mr. Chairman, you asked the question, one of the members asked the question about what is sacrosanct about a certain

timeframe for a decision.

We set out a timeframe in which we thought we could make a

decision, and we still believe that within a year we’re going to be

able to make an EMD decision. But it is driven by our requirement

of having the technology maturity of where we think it needs to be

to make that decision, not by any arbitrary line in the sand.

But presuming for a moment that we have what we believe are

adequate levels of technology maturity, you have to realize that we

would be delaying that decision for a longer period of time. Carrying both contractors, for instance, which is going to continue costs

and so forth, would have an impact on our international partners

who are participating in the program and the potential of either

pullout or other reopeners, if you will, in terms of the international

relationship.

So there are a number of sort of follow-on impacts that could

transpire as well.

Mr. SHAYS. I want Mr. Rapallo to continue to develop this line,

but I just wanted to ask you the question again: You say ‘‘adequate

89

levels of technology and maturity.’’ That’s just a different term

than ‘‘low level,’’ and ‘‘adequate’’ leaves me really—— Mr. SOLOWAY. I apologize if I seem to be obfuscating, because I

really wasn’t, sir.

When you made the comment that we would be at low risk, we

will have low risk relative to the individual technologies, yes, that

is a requirement. And that is a requirement. What is not a requirement necessarily is that TRLs be the measure of that risk.

We are recommending TRLs, by the way, in the new 5000 just

as a common language, but there are, as General Huot said, other

ways in which one assesses, measures and manages to risk mitigation and risk management other than TRLs. They themselves are

not the only.

Mr. SHAYS. I’m going to go back. You’ve made the point that the

Harrier—I call it the Harrier jet; is that the British term for it?

But the vertical takeoff jet is deteriorating significantly. It needs

to be replaced soon. The question mark is, do you speed up everything because of that? And one of the questions—I have total acceptance that if you’re in production, and then we in Congress do

what we sometimes do: We say, instead of 30 planes, it’s 20 planes;

we add costs, I understand that. I understand how we slow things

in production, but frankly sometimes the technology isn’t there or

is being revised. That’s one of the costs of slowing it down.

But I have a harder time understanding how slowing down the

decision before you go into development and production adds significantly to costs. And I’ll just have you come to that question

afterwards, but you had mentioned it and I just want to—I would

love you to continue.

Mr. RAPALLO. That would be—my only followup is, GAO has sort

of stated or described a process where if you delay the EMD process, you can develop the technologies and ultimately save money

and save time. Do you disagree with that?

Mr. SOLOWAY. No, I don’t disagree with that, but on this given

program, I think what we’re saying is, we would not go into EMD

if we did not have the technologies developed, so that the delay in

time is not necessarily going to help. We believe the technologies

will be at a level that allows us to make a responsible decision to

move to EMD.

Now, whether we agree that, based on a TRL chart, if one were

to use that measure, one would need to be at TRL 7 or 6, that’s—

that is actually the crux of the debate, and what does it require to

achieve a TRL 7 under the construct. But clearly if the technologies

are not there, you cannot—particularly the critical-path technologies that we’re talking about, then you wouldn’t make that decision.

So in this case, all we’re saying is, to say today that we ought

to delay the program, when we believe the technologies are either

there or close to there and will be there in the timeframe, there

would be no reason to delay. Otherwise, if the technologies aren’t

there, it’s correct, you would not want to go into EMD; you would

want to take advantage of further development time.

Mr. RAPALLO. Thank you.

Mr. SHAYS. If you want to come back, we will.

Mr. Halloran.

90

Mr. HALLORAN. Well, I guess the followup that occurs then is,

you believe that technology—to be ready, how will you know? If not

with some more quantitative or objective measure than TRLs, how

will you know that, say, the vertical-takeoff-and-landing technology—for example, speaking hypothetically, how will you know

whether it’s ready or not? How will you measure that?

Mr. SOLOWAY. First of all, as General Huot said, that there are

things some people call the Willoughby templates, and so there are

other methodologies that are used. And you may, I think, hear

from other witnesses in terms of probability, as well as objective

measures and so forth.

But the other reality is, and I don’t mean to be glib when I say

this, it’s either going to be working or it’s not. It’s either—the capability to do what you are talking about has either been demonstrated or not demonstrated on a concept aircraft or what have

you. The radar or whatever technology you’re talking about will

have or will not have been demonstrated on another plane. The

avionics will or will not have been demonstrated in a suitable laboratory environment.

Actually, when the chairman mentioned laboratories at levels 1

to 3, there is a category in level 6 that talks about a high-fidelity

laboratory, different than sort of the low grade. So we will actually

have demonstrated the individual technologies in a variety of different ways most relevant to what we have to prove and see.

General HUOT. I think the short answer there is that we will

continue this rigorous and disciplined approach that we have had

using these waterfalls to track each one of these technologies and

assure that it’s a low level of risk. In fact, in the input that we got

from Boeing and Lockheed and Pratt & Whitney in the inputs requested by House Armed Services, the Subcommittee on Military

Procurement, they went through, in great detail, each of the technologies describing the process that they were using and reaffirming that we would be at low level of risk prior to entry into EMD.

That is really the tool that we have been using. And it is a risk

management tool that tracks, is event based, and it also again allows you to make decisions if you need to go to an alternate technology to be at low risk prior to EMD.

Mr. HALLORAN. Stay with that for a second and describe the waterfall again. What I heard and I think what the chairman heard

is that high risk plus risk mitigation plan equals low risk.

General HUOT. No, not a plan. You have to execute the plan and

then you monitor the plan to assure you’ve gotten there. So the——

Mr. HALLORAN. That’s a quantitative way?

General HUOT. In a quantitative way.

The program office goes through, and as the contractor completes

each of the events that they’ve agreed are to track this from high

risk to medium risk to low risk, they check them off. And so you

end up essentially with a checklist that says you are at low risk

in that technology area—you have done all the things that you said

you needed to do to get there.

Mr. SOLOWAY. Let me just add one other thing, if I could, on that

point. Once you’ve done that, you also have, and General Huot

mentioned a couple of times, fallbacks. I think it’s important to understand that as you go through this process, let’s say for instance

91

that we have a technology that was demonstrated on an F–16 and

that we thought this is ready to go. For some reason, as we go forward, it doesn’t work. What do we have to have done prior to EMD

to address that potentiality? That is where the fallback is. You

have to have an assessment of what lower-level capabilities you

can access without affecting the critical performance parameters of

the plan.

Mr. HALLORAN. And cost?

Mr. SOLOWAY. Certainly.

Mr. HALLORAN. Because I was going there next anyway. I’m glad

you raised that. Because GAO made the point that in a couple of

these critical areas the fallback was quite expensive and did affect

performance or the requirements anyway, for example, and yet you

testified that the 5000 series calls for a fallback that it would be

at a higher——

Mr. SOLOWAY. The fallback itself, if it’s measured, a TRL would

be higher, but it might not be as high technology.

Mr. HALLORAN. But it involves a tradeoff of some kind.

Mr. SOLOWAY. There’s a tradeoff certainly, and you have to assess all that in this process. And you have critical performance parameters that you have to meet and you can’t go below that.

Mr. HALLORAN. Let me change the subject slightly and talk about

the definition of threshold between technology maturation and integration.

It struck me that in rescoring the original contractors, TRL scoring, the Department kind of expanded the integration to get itself

out from under some low scores here and to push things up to 6s

and 7s, to where you could define your risks more clearly, or at

least as lower. So talk about that threshold some more.

I mean, integration, there’s an example in some testimony that

comes in in a minute of a DOD system that looked fine in pieces,

but when they came together they interfered with each other and

didn’t work. And that’s clearly an integration issue, and it’s a separate set of risks, as you testify. But is getting to demonstration of

form, fit and function in a proper environment, is that integration

or is that maturization, maturity?

Mr. SOLOWAY. I would suggest that the way—if you’re taking

that comment and weighing it against the way TRLs are laid out

and what levels you hit, because it’s sort of the context of the question—that you have to do a fair amount of integration to do that.

So my answer would be, yes, we think that—let me go back to your

initial question, because what you really were asking is, did we try

to jimmy numbers to get out from under a bad score. That was the

first question.

And my answer is, and I can say this as one who was not involved in that initial process, but have gone back over the last

month or two and have met with all of the players, have had extensive discussions, services and so forth. I genuinely believe that

there was, despite what Mr. Rodrigues said, a lack of understanding of how these were being applied, because what was being done

was assessing levels relative to the risk of systems integration.

This is for the JSF. And so when you do that, you are—your relative risk is higher.

92

But when you’re weighing it against a question of, have you

proven out the individual technology, which is what we believe and

the 5000 really speaks to—and the new 5000 needs to be matured

in a way we have not traditionally done before, before you go to

EMD—when you do that, that is—the result is the rescored numbers.

Mr. HALLORAN. What was the source of the misunderstanding?

As you say, the 5000 draft had been out a while. It had reached— it had received some resistance.

Mr. SOLOWAY. This was not relative to the 5000 rewrite. This

was relative to the GAO—— Mr. HALLORAN. But the issue and the proposed role of TRLs, as

you state in your testimony, at some point push would come to

shove and a low TRL would prevent a technology from moving into

EMD; that was pretty well known.

What was the source of the misunderstanding?

General HUOT. How to grade TRLs? Our program office had

never used TRLs. When they graded them to come up with these

ratings that you see on the chart, they rated those, including the

risk of integration on the Joint Strike Fighter. Of course, that’s

going to give you a much higher risk; that’s why the lower ratings.

They did not rate the individual technology area by itself without

consideration of the risk of integration. When they went back and

did that, in fact, that was spelled out in Dr. Schneiter’s letter that

responded to the GAO report, he made it very clear that the Department position was that the Department does not agree with

the conclusion, which is based on misinterpretation of a process for

determining the readiness of technologies for incorporation in

major systems; and went on to say later that the GAO ground rules

for scoring technology rating levels included the risk of integrating

the technology onto the JSF platform.

The JSF program office used those ground rules to arrive at the

ratings contained in the draft report, and upon review and discussion with other users of technology readiness levels, that program

office determined that only the maturity of the technology, not its

integration, should be rated to determine the readiness under

EMD.

One finds that technology risk is expected to be an acceptable

level of EMD start.

Mr. HALLORAN. Let’s go back to my original question. Define ‘‘integration’’—‘‘integration’’ meaning form, fit and function down to

the size it will have to perform next to other systems, or ‘‘integration’’ meaning actually working with other systems on an airplane?

If you define it that broadly, then of course you’ll, I think, diminish the use effectiveness of TRLs as a threshold judgment to get

into EMD.

General HUOT. Let me say just a couple things here. First of all,

it was clear that some of these things have to fly on aircraft in

order to get to the low level of risk. In fact, if you would bear with

me just a moment if you look at those eight technology areas without addressing the eight technologies specifically—for example,

technology No. 4 is flying on an F–16, technology No. 1 is flying

on a concept demonstrator aircraft—in fact, a lot of these technology areas will fly on aircraft. But when the program office made

93

their initial assessment using those, the TRL scoring, they included

the risk of integration on the JSF platform to score those.

Mr. HALLORAN. Let me finally, Mr. Soloway, in your testimony,

in talking about the 5000 series, and you say—which is largely

being implemented in JSF, speaking more broadly about acquisition reform, now, what—in what way is it not largely—gives you

some room out—in what way is it not—what challenges does the

Department face in more fully applying—was this referring to this

program and others that follow it?

Mr. SOLOWAY. You’ve got two separate questions. Let me start

with the JSF specific and then move to the broader question if I

could.

On the JSF, when you—we complete the 5000 rewrite, which is

now out for comment, we expect to have it finalized in the next 30

to 60 days; it’s been a long process; it’s a fairly complex set of documents—there will be actually a new systems acquisition model, if

you will, with different kinds of entry and exit criteria based on

much of the kind of model that GAO talked about this morning.

And this is precisely where we are going in the Department, and

the kinds of criteria that you have to meet to go into various

phases of the process, and they may have different names, and it’s

really a different model.

JSF has, in effect, mirrored that model, but not precisely because

the model didn’t even exist until the last several months, so it’s not

a precise marry-up in that way. But I think JSF has very much

reflected what we are seeking to do in the 5000.

On the broader question, what DOD or the Congress needs to do,

I think there are—the list is quite long in terms of the challenges

we face. Part of it is the sort of entrenched cultures that we both

have in terms of how we view programs and systems development,

and some of the issues that you raised with Mr. Rodrigues earlier

today. And that’s both an internal, DOD problems with the Congress, and the external world. Part of it has to do with—and I’ll

give you one example.

I think one of the real challenges with the new 5000 rewrite—

and this is not relative to JSF, this is a broader question—is if you

are in this front-end process where you want to, and have to, demonstrate technologies, you want to also—and thus not make your

commitment to EMD and, really, program commitment—until

you’ve gotten to that point, you will also hopefully, as we go down

this path, be looking at more options than you might otherwise

have been looking at. And what that leads you to is having to

avoid, or wanting to avoid, a program commitment either in the

building or from the political environment before you’re ready to

say, I’ve got a set of technologies that can get me this capability,

and that’s when I’m ready to make that commitment.

You know, traditionally we’d say, we’re going to do a new jet or

we’re going to do a new ship, and then we sort of work it through.

But the ship never left the table or the plane. It was fairly rare

once you got that far down the path.

What we really want to do is both for ourselves inject a great

deal more discipline. As I said, we have tremendous agreement,

and GAO has been a partner at the table as we have built this

model, but we also have to have discipline in the process to realize

94

that we may want to be looking at different kinds of capabilities

and different kinds of mission options before you make those kinds

of commitments. And that’s traditionally been very difficult, I

think, for all of us.

So there are a number of challenges that we all face in moving

this forward. And I think that they’re very—I think they’re challenges that we can overcome. As I said, I do believe that JSF is

really the forerunner coming out and applying many of these principles.

The General was talking about the process they went through

with the various stakeholders, the different services. A really flexible requirements process driven by what the customer really needed and what we believed as we went through the process technology was going to be able to provide in a reasonable period of

time, so that we didn’t end up waiting 18 years or 20 years, as has

been the case in the past.

Mr. HALLORAN. Finally, let me ask a followup where Mr. Rapallo

began.

Would you agree that all of the impacts you cited that might flow

from a delay at the demonstration evaluation phase can potentially

be more serious, more costly and lengthier if they’re incurred in the

next or subsequent phases of the program, that, as is described— we may disagree on the $1 to $100, but there is an almost inevitable escalating effect in delaying problems in this process.

Mr. SOLOWAY. That would certainly be the conceivable output— outcome. However, I think our view at this point is that the

delay—to arbitrarily decide today to delay, to really do, in essence—and I don’t want to get into nitpicking, but in essence what

we’re talking about here when you get to level 7 is the early phases

of EMD.

That’s what we do in engineering. We focus on the manufacturing part and the production part of it. We don’t focus on the ‘‘E’’, which is the engineering. That’s a critical next step in bringing

these technologies together. We believe, and we could be proven

wrong next year and the decision would not thus be made, but we

believe the technologies will be at that proven level that enable you

now to move into that engineering of EMD phase.

So given that, the delay to us would be costly, unnecessarily costly, and would be just unnecessary on the face of it. But in a hypothetical world having nothing to do with JSF, certainly if you have

tremendous technology issues and you go into EMD and lock yourself in, you could be asking for an escalated problem. I wouldn’t

disagree with that; I just don’t think that applies to the JSF.

Mr. HALLORAN. Thank you.

Mr. SHAYS. Let me say, then, I would like David to come back

with answering a question related to what you were saying up

here.

I’ve heard you both say, you know, this is something, you know,

we have never done before. You’re very proud of the program, the

fact that you’re moving toward a best commercial practices. But

what I feel like, as I hear you, is that it’s going to be so significant

to DOD that you’ve moved in this area if you do it halfway, a

three-quarters way, this is monumental. And I might agree with

that, that it would be monumental, but I have to hear an argument

95

that says you can’t go all the way with the program. And so for you

to say you’re doing three-quarters of it—and you haven’t used those

numbers, but that’s the feeling I get—wouldn’t satisfy me unless

you say we can’t do best commercial practices here, here and here

because, and then let us evaluate it.

So I am going to stay on the table that really there is no argument against using best commercial practices to the nth degree,

unless, and I just kind of want to say that to you.

Mr. SOLOWAY. If we gave that impression, I think both of us

would regret it. How do you define doing it all the way versus the

three-quarters?

Mr. SHAYS. Well, when you start to talk about it as a tool and

not a requirement. And maybe in the private sector it’s a tool and

not a requirement, and then that’s fine.

Yes?

General HUOT. I would just say that—I would emphasize that we

use a different tool. And again the reason we did it is because we

wanted something that would allow us to manage risk, not just assess it. And our intent is not to go three-quarters of the way; we

intend to reduce every critical technology that we identified to a

low level of risk before we go into EMD—very important point.

Mr. SOLOWAY. And the requirement that is there is the technology maturation requirement. That is a requirement; that is not

a negotiable.

What is negotiable is the TRL becomes like the yardstick, so it’s

versus using a yardstick or a ruler or another method of measuring

what the outcome is going to be. But the key here, the outcome we

all seek is technology maturation. And that is not a negotiable.

Mr. SHAYS. Does the DOD 5000 rewrite follow this practice, best

commercial practices?

Mr. SOLOWAY. Yes.

Mr. SHAYS. Because when you say it, you say in your statement

on page 10, ‘‘Finally, let me be very clear. The strategy I have articulated for the revised acquisition process that will be prescribed

in the DOD 5000 rewrite, and which is largely being implemented

on the JSF, represents a real departure from our traditional approach to system development.’’ That’s the basis for my making that comment. I buy into the fact

that it represents a real departure from our traditional approach

but it’s still the word ‘‘largely,’’ you know. So just so you have a

sense of why I get the feeling I get.

Mr. Rapallo, you had a question.

Mr. RAPALLO. I just had one followup on Mr. Halloran’s questions

about the TRL levels.

You said—I’m just trying to understand. The information that

was provided to GAO was from the contractors, their evaluation, or

from the program office?

General HUOT. You know, I wasn’t involved in that process, but

I know the program office is involved with the contractors.

Mr. SHAYS. I didn’t hear the question. I would like to ask you to

ask it again.

Mr. RAPALLO. It was just a question of where the information

came from, the TRL levels, from the contractor or the joint program office, or probably some combination of both.

96

Mr. SHAYS. So what is the answer?

[The information referred to follows:]

The input for the TRLs reflects a consolidated answer from the JSF Program Office, incorporating input from the Boeing Company and Lockheed Martin Corporation.

General HUOT. I believe it is, Mr. Chairman—I wasn’t involved

in the exact process. I know the program office was involved, but

I think the contractors actually did the work either with the program office or the oversight of the program office.

Mr. RAPALLO. And your position was that the GAO asked for information based on TRL levels, asking also, including the risk of

integration?

General HUOT. Yes.

Mr. RAPALLO. That information was provided?

General HUOT. Yes.

Mr. RAPALLO. And that would be incorrect, you’re saying? Basically, I’m trying to figure out if these numbers are too low because

of that.

Mr. SOLOWAY. That’s the crux issue here, that if you measure individual technologies relative to the risk of being able to integrate,

or to integration, you come up with one answer. If you measure

them as individual technologies relative to their technology maturity, you come up with the second. And that’s what the rescoring

was.

So in our view, the rescoring actually does raise the numbers, not

to level 7, but they certainly raise it.

Mr. RAPALLO. Are these the numbers that represent the rescoring or the initial?

Mr. SOLOWAY. These numbers are the initial.

Mr. RAPALLO. So the rescore numbers would be higher than this?

The question is what are they I guess.

Mr. SOLOWAY. I think all of them would be at five and six.

Mr. SHAYS. I would like you to tell us what would they be at.

Mr. SOLOWAY. I don’t know if I have the specifics on each of

them.

Why don’t we take that for the record and give you—I’m sorry.

OK. These are our estimated ratings.

Mr. SHAYS. Based on not having integrated?

Mr. SOLOWAY. Right. These are what was referred to as ‘‘the rescored numbers.’’

Mr. SHAYS. Without the risk of integration?

Mr. SOLOWAY. Right, they were not scored. We have—and I’ll

just take them in order—7, 6, 6, 7, 6, 7, 6, 7.

Mr. SHAYS. OK.

Mr. SOLOWAY. And I have the rationale, if you just—I mean, I

can tell you what the technology—but in order of these numbers,

here again, as—I think General Huot referred to this a moment

ago. One, flying on the concept demonstrator aircraft, which is we

believe consistent, technologies already flown in aircraft helicopters

or spacecraft, another one, such systems are similar or being used

in similar commercial environments, including the Boeing 777,

Federal Express Caterpillar, etc.; another one flying on the F–16;

another technology flown on the JSF flying test bed; another one,

the same, flying test bed; another one, flown on the JSF flying test

97

bed; and another one, flying on the CDA as well as the F–22, the

F/A–18 and the Eurofighter. So in each of these cases, if you look

at are we using the technology in a relevant environment, per se,

they are all being demonstrated where—in that means.

Now, that is the rescored numbers.

Mr. SHAYS. But that’s based on your assessment, correct?

Mr. SOLOWAY. It’s a DOD assessment.

Mr. SHAYS. Right. And so in technology 7 you have a leap of two

levels just by not having to integrate. And technology 1 you leaped

up one, in technology 2 you went up two, technology 3 you went

up two. In technology 4 you went up two. In technology 6 you went

up two. In technology 7 you went up two. That’s a big jump.

General HUOT. Again, because you’re trying to mature the technology without considering the risk of the integration on the JSF

platform.

Mr. SHAYS. So we can basically accept GAO’s assessment that

they got—if we have a footnote that says it’s not integration, based

on—excuse me, based on that, and we could accept your numbers

based—excuse me, the other way around.

They’re saying it’s not integrated; therefore, that’s the score. And

you’re saying, if you don’t consider—I’m going to say it right here.

Mr. SOLOWAY. I think I understand where you’re going.

Mr. SHAYS. For my own sense of self-esteem, I need to say this

again.

The bottom line is, GAO gave you a lower score because of a failure to integrate. And you’re saying integration shouldn’t have been

a factor in your score, and your score is higher without integration.

Mr. SOLOWAY. You’re asking if we can accept this as—under that

other definition, if you will, and I think I would have to say I don’t

believe we can because there are other issues associated with it.

Mr. SHAYS. You want a little conference here?

Mr. SOLOWAY. We might want to take it for the record. But the

table itself is confusing, and I’m not sure that we would precisely

accept every level that was there.

Mr. SHAYS. I welcome sometimes a little bit of qualification by

two people in the same table.

If that’s all right, General, do you want to just say what your

sense is?

General HUOT. I think once again that you go back to where

these things came from. They were provided to the GAO, but those

scores were based on considering the risk of integration on the JSF.

When those scores were rescored again and provided to the GAO,

they were without integration of JSF, the integration on the JSF,

that’s when they go up to the appropriate level.

Mr. SHAYS. But I just want to know if you accept their score

when there’s—by—and look at the score that you have provided us,

would they accept your score, in your judgment? I mean, in your

talks with each other, was this really the big debate whether it was

integration or not and you could pretty much agree on your numbers?

OK. The answer is hard to figure out what they were saying

when they did this, right?

Mr. SOLOWAY. Let us take that for the record, sir.

[The information referred to follows:]

98

99

Mr. SHAYS. Fair enough. You all have been very responsive. And

is there a question you would have liked me to ask that you prepared all night for that you want to—I’m just going to say my staff

gave me so much to read I stayed up all night. The statements

have been very interesting on the third panel, as well as your presentation has been very, very helpful and, I think, candid.

Mr. SOLOWAY. I’d just add one point, it’s not a question, and that

is to go back to what we said earlier.

The debate here is not over whether or not we do need to demonstrate and whether or not we’re committed to demonstrating

technology maturity before we move into EMD. And that is a very

important step in a critical discipline that we are instituting in our

process with the JSF and in future programs. And I believe it’s

very important, as we do that, to recognize what it takes and to

recognize the ‘‘E’’ engineering part of EMD, which is the next step.

So I would just reiterate that point again.

Mr. SHAYS. I don’t know if I’ll be chairman next March but it will

be very interesting. One of the things we do is, we put every statement on the record; then we have to live with it. I’m just—the

problem is, when it comes to this kind of program, the same people

aren’t always the same people that have to answer for the decisions

that were made 2 years ago—I mean on the congressional side as

well as your side.

Mr. SOLOWAY. Sir, I just point out I won’t be here next March;

that’s a guarantee. I think you saw the news this morning where

both the House and Senate Armed Services Committees have included language requiring us to be able to demonstrate to the Congress that we have achieved technology maturity before we move

into EMD. And we can live with that and have no problem with

coming back to demonstrate that or document that for the Congress next year.

Mr. SHAYS. That means low-level risk.

Mr. SOLOWAY. Thank you.

Mr. SHAYS. General, any comment you’d like to make?

General HUOT. No, I think that covered it.

Mr. SHAYS. We’ll end on that note. Very nice to have you both

here. Thank you.

We’ll get to our third panel. I appreciate the patience of our third

panel. I have three panelists, if you would remain standing so I can

swear you in: Dr. Thomas McNaugher, deputy director, Arroyo

Center, RAND; Mr. Rodney Larkins, business development manager of 3M Corp.; Dr. Wesley Harris, Department of Aeronautics

and Astronautics, Massachusetts Institute of Technology.

If the three of you would—we’ll make sure we get you there. If

you would raise you right hands, please. Thank you.

[Witnesses sworn.]

Mr. SHAYS. Note for the record that all three have responded in

the affirmative, and we’ll just go down the row.

Doctor, we’ll start with you first. And actually we have two doctors here. I’m sorry.

Dr. McNaugher.

100

STATEMENTS OF DR. THOMAS L. McNAUGHER, DEPUTY DIRECTOR, ARROYO CENTER, RAND; RODNEY LARKINS, BUSINESS

DEVELOPMENT MANAGER, 3M CORP.; AND DR. WESLEY HARRIS, PROFESSOR, DEPARTMENT OF AERONAUTICS AND ASTRONAUTICS

Mr. MCNAUGHER. Thank you, Mr. Chairman. It is a pleasure indeed and an honor to be here. At the risk—— Mr. SHAYS. I will ask you to speak a little louder or move the

mic a little closer. Can you pull it a little closer?

Mr. MCNAUGHER. Somewhere between last night and this morning I have acquired a cold, so I am about an octave lower.

Mr. SHAYS. I have absolutely no sympathy for you.

Mr. MCNAUGHER. Let me start by saying I don’t know much

about the JSF program per se. I was asked to speak here because

I think I know a great deal about the weapons acquisition process,

at least as it functioned during the cold war. I wrote a book about

that. It came out in 1989 from the Brookings Institution. And in

that book, called ‘‘New Weapons, Old Politics,’’ I examined what

might be called some of the perverse patterns of cold war weapons

acquisition, one of which was the tendency to rush weapons

through development and into production.

In particular, I focused less on the move from what’s called the ‘‘demonstration-validation phase’’ and more on the move out of

what is now called EMD and into production, so-called ‘‘production

concurrency.’’ So it’s not exactly the same issue that you’re dealing

with today although I think you’ll find a lot of what I have to say

about the incentive structure and the way we measure——

Mr. SHAYS. Could you just suspend for 1 second. I’m sorry.

Thank you.

Mr. MCNAUGHER [continuing]. Production concurrency or the

tendency to substantially overlap the latter stages of development

with the early stages of a move to high-rate production was a fairly

common practice during the cold war. It was justified on two

grounds. Technically it was justified on what might be called the

‘‘theory of declining uncertainty.’’ back in the ‘‘dem-val’’ phase. In

a somewhat linear fashion, we reduce those uncertainties during

development until, by the end or nearing the end of EMD, we

should be able to move into production.

On the other hand, on strategic grounds we were confronting a

numerically superior enemy, so we justified the rush to production

as a way of getting the jump on the Soviet Union, getting ahead.

The faster you could get these new technologies out, the farther

ahead of the Soviet Union you were, everything being equal.

Now, when I started my book, I more or less agreed with those

premises. When I finally started writing, I had come to disagree,

especially with the first one, the theory of declining uncertainty. I

would substitute what I call the ‘‘J-curve theory of program uncertainty.’’ True, at the beginning of a development project you have

enormous risks, and you do, through ‘‘dem-val’’ and in the early

part of EMD, neck those down to reasonable levels. But I encountered in almost all programs a rather sharp upturn in risks as the

program actually went into production.

Three reasons for that: One was systems integration. And I like

the fact that you’re talking about systems integration this early in

101

the program. During the cold war systems integration often occurred very late in the development program, sometimes after production systems were already out in the field. And what you find,

looking back at the history, is that even simple components stuck

together in a new way produced new problems that you didn’t

know were there, these problems often force a design iteration.

The second problem that drove risks up was the move to production tooling itself. Again, even with simple systems, somehow the

move to a production model from a preproduction prototype introduced risks, some of them great, some of them small, which almost

always led to a subtle interaction between the producing engineers

and the developing engineers.

And finally, when you got this thing out into the field where real

soldiers and sailors and pilots could use it, they invariably discovered new ways to use it that it wasn’t designed to do, and it started

to break, and then you would have to go back and redesign it and

fix those. This is, in a sense, a tribute to the ingenuity of developers and also users in the military, but it could be pretty embarrassing sometimes, the kind of breakage you could get. So all three

of those then would appear very late in the process.

To the extent that production concurrency ignored those risks,

and to use the jargon of today’s hearings, to the extent that, in a

sense, it wasn’t ‘‘knowledge-based,’’ it didn’t take account of these

late-arising risks, we had chosen a fairly expensive way of developing weapons during the cold war. If you’d geared up for production,

capitalized, and hired the labor, every delay, every change, cost a

lot of money. If you already had systems in the field you were faced

with this vexing question, do we retrofit the fixes to the fielded systems, which was always expensive, or do we just let those go? This

led to what I would call the ABC approach to development.

The A model of a lot of airplanes, and other things, too, often had

serious design anomalies in it; the B model got it roughly right;

and the C model is where you really wanted to be all along, but

it took maybe even 100 or 200 production models to get there. So

it was expensive. And when you did that, the average effectiveness

of our overall force actually was lower than it would have been had

you waited and gotten all of the production run, or most of it, to

the C level.

Now, we accepted those costs ostensibly in a desire to confront

the Soviet Union. And you would think that now with the cold war

over, as you say, Congressman Shays, we could relax and we could

do this better. Let me make clear that the way you do it better is

not by having a sharp divide between development and production.

The point of my research was that the early stages of production

are part of the development process. Again, to turn to the jargon

of today, knowledge points 2 and 3 tend to fudge together and you

don’t want to have a sharp gap. Rather, you had to treat the early

stages of production in a way that with respect to late-arising uncertainties, perhaps starting very low, with a low rate of production, getting the systems out into the field, flying or shooting or

driving the dickens out of them for awhile, and taking all of that

information back and imparting it to your design before you really

went to high-rate production.

102

One of my colleagues at RAND years ago remodeled some of the

Air Force aircraft programs of the 1960’s, 1970’s and 1980’s on that

basis and concluded that you could probably get a more effective

fleet overall for a little less money if you did it that way. Now, you

added a little bit to the development cycle, which everybody

thought was too long, but with the cold war over, that shouldn’t be

much of a concern.

So we might be in a position to move now to a more relaxed approach. I’m skeptical of our ability to do this, however, because I

think that production concurrency was as much a political as a

military strategy; that is, it was rooted in the politics that we see

here today—in the politics of the acquisition process.

Surely in the development of a new system, the move into late

development and early production has to be the most vulnerable

stage. You now have prototypes of the system, you fly them, drive

them; the data almost always is going to contradict the optimism

of early assessments. Sometimes you get tragic accidents, a helicopter crashes or a plane crashes. Sometimes you get funny accidents, you know—the gun that’s supposed to shoot a helicopter

shoots the fan on a nearby latrine, as was the case with the Armys

DIVAD.

If this were purely a technical environment, everybody would

have a good laugh, and they’d go back and redesign that system

and fix it. But in the very charged political environment that can

come to surround expensive programs, you know, it’s hard to desensitize that evidence. Knowledge and information become very

dangerous. And delay—I’m thinking of your earlier discussion—you

cannot convince a program manager that a 6-month delay isn’t the

beginning of a mortal wound.

How do you handle that? Well, in a sense you stack the deck.

You raise the costs of slowing down. And you may even have production versions out there before you actually get that information.

What I’m describing, in a sense, is the dilemma of weapon acquisition in this country as a political as well as a technical process.

From a technical point of view, you really want to have a great

deal of flexibility late in the process. From a political point of view,

flexibility can be downright dangerous. So we tend to structure a

certain amount of inflexibility in there.

This is not a dilemma we have ever resolved very well in our history. If you step back from the cold war and look at the 200 years

in which this Nation has bought weapons, you would argue that

the problem wasn’t slowing the acquisition process down, it was

getting it to produce anything at all. We would test endlessly and

then not buy. Our technology generally lagged the Europeans, and

as a rule, we had trouble getting things through the juggernaut of

the political process. That ended with World War II and with the

cold war.

So while talking about political strategies may sound like I’m

being critical, we ought to understand that for 50 years we produced and systematically modernized the best force posture in the

world.? So it’s not clear to me that we can slow this thing down.

And this is why I refuse to pass judgment on the patterns of acquisition I studied in the cold war.

103

We are now in a unique period in the Nation’s history. We are

not confronting a great Soviet threat, as you’ve said, although the

world remains dangerous; moreover, we remain engaged, and there

are threats out there. So we’re sort of at the Goldilocks situation,

you know, it’s not no threat, not a big threat, but sort of right in

between. And the question is, can we come to the Goldilocks solution for weapons acquisition? That is a process that is relaxed

enough to take account of late-arising uncertainties, but not a process that goes to sleep and doesn’t produce anything.

And that returns me really to the first paragraph of your statement, Mr. Chairman, which talks about the emergence of a consensus about how we’re going to handle this.

This hearing focuses on the JSF specifically, but the JSF, as you

recognize, is the beginning of a wave of new modernizations and recapitalizations after a 10-year procurement holiday. So this hearing

is also part of the way in which we construct this consensus, and

it remains to be seen how that will work out.

Thank you.

Mr. SHAYS. Thank you very much. Your book was the reason why

you’re here and your testimony justifies your presence.

[The prepared statement of Mr. McNaugher follows:]

104

105

106

107

108

109

110

Mr. SHAYS. Mr. Larkins.

Mr. LARKINS. Good morning, or I guess I should say good afternoon.

Mr. SHAYS. Good afternoon. You’ve been wonderful to be so patient. But I learn as much from the third panel, and sometimes

more. So from my standpoint, I’m very happy you are here.

Mr. LARKINS. Thank you. As business development manager for

government research programs at 3M, I’m here to discuss 3M’s ‘‘best practices’’ associated with our new product commercialization

process and the timely integration of breakthrough technology into

‘‘change the basis of competition’’ products.

3M is a diversified manufacturing company with sales of more than $15.5 billion. We manufacture a broad range of products directed at six distinct commercial markets. 3M has grown by pioneering innovative technologies and creating new products with

these technologies, thereby creating new markets and revolutionizing existing ones.

3M has a written policy objective which states that 30 percent

of all product sales for a given year come from products that were

introduced in the 4 preceding years. Thirty-four percent of 3M’s

total sales in 1999 came from products in this category. It is these

new products which help sustain the profitable growth of our corporation. To maintain this rapid pace of technology development

and new product introduction, 3M has evolved a well-defined technology development and commercialization process.

A basic tenet to successfully introducing new products is the discipline to make the up-front investment in research and development. 3M has invested more than $1 billion per year over the past

3 years in research and development. This investment is directed

both at broadening and strengthening 3M’s existing technology

portfolio, as well as moving products rapidly through the commercialization process.

Currently, approximately 20 percent of 3M’s research and development budget is directed at technology development and enhancement. Approximately 80 percent is directed at product scale-up and

commercialization. This investment ensures the availability of critical technologies to turn into products, and the product development

resources to focus on the corporation’s priority programs.

In our quest to maintain technical and market leadership in the

markets we serve, we have evolved into a laboratory structure

which substantially segregates technology development from product development. Our experience has been, that the key technologies are used broadly across the corporation’s market groups.

By focusing on technology development in a technology center, a

critical mass of technology experts is created, and costly redundancy is eliminated.

3M has formally established 14 technology centers for the corporation’s most pervasive technologies. The responsibility of these

centers is to establish and maintain world-class capabilities in

these critical technologies. The second responsibility is to work

with product development teams in the business units to integrate

this technology into the product development programs to assure

successful and timely product introduction. This step is critical,

and a number of management tools have been established to enVerDate 11-MAY-2000 11:50 May 24, 2001 Jkt 000000 PO 00000 Frm 00114 Fmt 6633 Sfmt 6633 C:\DOCS\70782.TXT HGOVREF1 PsN: HGOVREF1
111

sure that it takes place. 3M is acutely aware that even our best

technologies, when not applied to timely commercial product development, are of no value, and we work hard to maximize that application.

Critical to the commercial success of our company is our ability

to select and to focus on programs with the best possibility for

changing the basis of competition in large market opportunities.

Despite 3M’s large investment in research and development, our

resources are finite and there are always more opportunities than

there are resources. Important steps in program initiation are opportunity identification, development of a comprehensive product or

system description, a business assessment of projected results once

success has been realized, and a technology assessment which indicates that technologies are in place to meet performance requirements. All of these elements must be in place before a program is

approved for scale-up and commercialization.

At the initiation of the product development process, a formal review is held to assure that all elements are in place for success.

Hence, success on these key programs is critical to the success of

the corporation and our goal is to put these programs in the very

best possible position to succeed.

Once under way, the development team follows a detailed new

product introduction plan which outlines all elements which must

be accomplished during the product development process. Periodic

program reviews are routinely conducted with the emphasis placed

on encouraging and rewarding candor on the part of the product

development teams and identifying and eliminating impediments to

the success of the team.

In addition, corporate review teams are employed. These review

teams bring together experts in product development from across

the corporation to identify potential roadblocks to successful commercialization and to identify resources to eliminate those roadblocks.

In summary, 3M’s approach to technology integration and product development has proven to be very successful. This process,

however, is far from being perfect. It’s a process that we will continue to evolve, and our goal, of course, is 100 percent success on

all of our priority programs in the corporation.

Thank you, Mr. Chairman. I’ll be happy to answer any questions.

[The prepared statement of Mr. Larkins follows:]

112

113

114

115

116

117

Mr. SHAYS. Thank you, Mr. Larkins. You work for, I think, a

pretty amazing company.

When I think of your company I think of it as—though very

large, it has tremendous innovation as a small company might

have. And it’s great to have you here. I would love to know how

we can seek comparables between what you do and what we need

to do in government in general, and specifically with defense.

Dr. Harris. I just want you to know, sir, I’m always awed when

I see a ‘‘doctor,’’ and I see MIT next to it.

Mr. HARRIS. Thank you. Good afternoon, Mr. Chairman. I very

much appreciate the opportunity to address this distinguished subcommittee on—— Mr. SHAYS. I would ask you to move the mic a little closer to you.

If you can turn it over the paper, as long as it doesn’t get in the

way. Does it get in the way? OK. That’s perfect.

Mr. HARRIS. I’m fine. Thank you.

Again, I very much appreciate the opportunity to address this

distinguished subcommittee on certain issues related to technology

maturity and acquisition reform. I wish to state at the outset that

I approach this important topic from the perspective of both an

academic researcher and as a former government manager, not as

an engineer practicing within the defense industry.

It is my view that our defense acquisition policy and practices

are complex. This complexity is a result of several factors, including a dynamic or shifting defense industrial base, a declining acquisition budget, a constantly evolving threat environment, a diverse

force structure, and most importantly, an increasing rate of change

of technology.

The impact of the last factor, namely the increasing rate of

change of technology, on technology maturity from an acquisition

policy perspective, is difficult to overstate. The impact is made

more profound when the global nature of much of this technology

is considered in the acquisition of new weapons systems.

I would like to add to this list the following elements that have

impact on the acquisition of defensive systems: the current focus on

greater life cycle value, the emphasis on more rapid deployment,

the emphasis on upgradability, sustainment and maintenance.

Mr. Chairman, while noting that today this subcommittee is addressing a specific program within the defense acquisition arena, I

wish to state for the record that the impact of acquisition reform

reaches beyond the procurement of defense systems and its related

technology. Through our defense industry base, acquisition reform

drives our national economy and impacts world peace, in short, our

success in developing an effective and efficient acquisition strategy

that captures mature technology, exposes the risk to control our future, to produce wealth and to continue to contribute to the advancement of humanity.

Based on my research and government experiences, I wish to

share with this subcommittee today several national successes.

First, there exist several case studies of successful acquisition of

defense systems in production. Second, there exist case studies of

successful parallel development of advanced technology to high maturity levels where the government is customer. These two successes have many things in common, and I believe are related to

118

today’s issues of acquisition reform in the environment of technology maturity.

My success or my research on economically incentivized contracts

focused on several important programs including the Sensor Fused

Weapons system, the Joint Direct Attack Munitions program, the

C–17 program, C130J, the F–414 engine, development that goes

within the F–18E/F airplane and the F–117 engine that goes in the

C–17 airplane, as well as the Boeing 757 airplane. These programs

were in production, were able to develop an economically

incentivized contract or a win-win solution for both government

and contractor.

A few comments now on the development of a critical complex

technology in advance of full system acquisition: In the early

1990’s, NASA’s Office of Aeronautics did develop and manage two

technology development programs. These were the high-speed research program and the advanced subsonic technology program.

During the same time period, NASA also worked jointly with industry and DOD to develop advanced gas turbine components within the integrated high-performance turbine engine technology program.

These three programs of technology development were successful,

and as stated in the prepared statement, over 12 reasons why they

were successful.

Mr. Chairman, the subcommittee may wish to note the very

strong commonality between the factors leading to successful development of technology parallel to full-system acquisition and the factors that enable a win-win solution for programs already in production. At the most fundamental level, the environment for favorable

development of advanced technology is very similar to the environment for acquisition of defense systems in production where technology risk is low, corresponding to technology at a high maturity

level.

These programs, both advanced technology development programs and acquisition of full defense systems, strongly suggest that

advanced technology at a high maturity level is essential to the acquisition of affordable systems with requirements for superior performance. The importance of advanced technology at a high maturity level is so great, in my opinion, that the government must

incentivize the contractor to develop advanced technology.

This means that the government must place a premium on the

development of technology to a high maturity level. The premium

must compare favorably with other awards available to the contractor. The economic realities of a high premium will, or at least

should, drive the government to a lean portfolio management condition. Advanced technologies selected for development to high maturity level must be, or should be, based on realistic projections of

need.

Mr. Chairman before concluding, I would like to add a few additional comments. I wish to note that to improve the chances of successfully developing advanced technology to a high level of maturity, one or we, those involved, should adopt common, quantitativebased language and assessment tools. Qualitative prescription of

technology readiness, such as the technology readiness levels, are

insufficient and inconsistent with realistic projections of need. The

119

desired quantitative-based language and assessment tools are recommended to be derived from probability in each technology readiness level. The TRL would be expressed or defined by probability

bands.

In conclusion, we, as a Nation, have demonstrated the capability

to produce weapons systems driven by advanced technology. My

colleagues have clearly confirmed that. To become more efficient at

the acquisition process, we must continue to develop advanced

technology in parallel with acquisition of full, complete systems.

Key elements again in this efficiency are to incentivize industry

with favorable premiums, a realistic projection of advanced technology needs, and the use of quantitative-based language and assessment tools.

The bottom line is that government and industry know enough

about each other and about advanced technology development to

make affordable acquisition of full systems and technology insertion work, to make it work efficiently. The health and wealth of our

Nation depends upon this working very efficiently.

I would entertain questions, if there are any.

[The prepared statement of Mr. Harris follows:]

120

121

122

123

124

125

126

127

Mr. SHAYS. Thank you. There will be questions. And I’ll just

throw it out to all of you.

First off, you were all here for the entire hearing, and I thank

you for that, because the value is that you can comment without

me having to repeat the questions.

Conceptually, a ‘‘best practice’’ model for acquisition that makes

sense in the private sector for private development also makes

sense for the private sector for a public customer. Would you all

agree with that?

And if not, if you would, qualify it where you would—I mean conceptually. I make the assumption nobody would disagree.

Mr. MCNAUGHER. You’re saying that the commercial best practices is transferable to the public process of——

Mr. SHAYS. Yes, and that we should seek to use commercial best

practices.

Mr. MCNAUGHER. The only caveat I would raise, Mr. Chairman,

I mean, obviously no commercial firm operates in the kind of political environment——

Mr. SHAYS. Just a little louder. I know you’re not feeling well,

and I said I wasn’t sympathetic; but I didn’t want to give you sympathy because I didn’t want it to get worse.

Mr. MCNAUGHER. There’s a lot of differences, obviously, between

buying weapons as a public good and developing a private sector

item. The only difference, though, that leads me to question the

use, the extensive use of best practice models is I don’t think in the

commercial world very often you systematically take these huge

leaps out into the unknown, as we did during the cold war and, I

think, still do; that is, not only the components of a system, but the

integration of a system really represents a dramatic jump.

Mr. SHAYS. But you don’t have any fear that we’re doing extensive use of this now. I mean, we’re not, this is the—I’m not aware

that a commercial best practices model is running rampant

through our government.

Mr. MCNAUGHER. No. What I’m saying is, trying to apply it to

the government, to this process, has limits. Most commercial firms

insofar as they’re doing marginal improvements are aware of risks

and costs at a fairly good level of certainty. You know, most of the

discussions of risks in the early phases of cold war development

programs proved in the long run to have been wrong. You don’t

know what the numbers really are. You have an idea of a cost performance curve——

Mr. SHAYS. And they were almost always understated.

Mr. MCNAUGHER. Yes. That’s not an indictment of anybody’s behavior. Technology just doesn’t give up its secrets very easily. So

we have these very precise discussions during ‘‘dem-val,’’ but, the

numbers and curves are going to shift. That’s again why you need

more flexibility down the road.

I think commercial practice may not have to deal with that kind

of uncertainty.

Mr. SHAYS. OK. We’ll touch on that a little bit more in a second.

Mr. Larkins, when I think of 3M, I would think that you use a

knowledge-based process, a best practices model for acquisition as

defined by GAO when they talked about knowledge, point one,

128

match is made between the customer’s requirements and the available technology.

Mr. LARKINS. Yes, sir.

Mr. SHAYS. Knowledge, two, when the product design is determined to be capable of meeting performance requirements. And

three, knowledge point three, when the product is determined to be

producible with cost schedule and quality targets. I would think

3M would just be right in the center of that kind of philosophy.

Mr. LARKINS. Yes, sir, that’s correct.

Mr. SHAYS. Where it might differ is that I think of 3M, with no

disrespect, but just in terms of putting it in perspective, that you

are seeing customer needs and then you’re seeing how you can

meet those needs with products that you have potentially available

in technology that you have available.

I make an assumption in most cases that technology is pretty developed, or sometimes do you have to—do you see a need and you

just start out from scratch with a technology?

Mr. LARKINS. That happens, sir.

Mr. SHAYS. What happens?

Mr. LARKINS. Where we will, in fact, start from scratch and develop the technology. A lot of it depends on how broad your window

of opportunity is in the market.

What we are finding today is that if you don’t meet a market

need quickly, then the market need is met by someone else. So

what we try to do is do a rapid technology assessment, determine

whether in fact we have technologies in house to meet that need.

If we don’t, then we will go out and make a technology partnership

or acquire a technology.

Mr. SHAYS. And admittedly the—so time is of the essence?

Mr. LARKINS. Absolutely.

Mr. SHAYS. But your technology requirements may not and certainly wouldn’t be as complex as developing an aircraft or——

Mr. LARKINS. That’s very fair to say.

Mr. SHAYS. So we need to have some empathy for the task at

hand, that applying best commercial practices is a very important

effort, but certainly we can carry the commercial to the defense;

and we may be—it may be unfair to say that you—DOD needs to

do it just like we do it in the private sector.

Mr. LARKINS. Yes, I agree that it would be unfair to say that

their technology integration problems are as easy as ours, because

they certainly are not. And—but I would like to agree with my colleague, Dr. Harris.

I think one of the reasons we are successful and are able to

achieve the type of record in product integration scale-up that we

do is because we do have a significant investment in technology.

We have a broad base of technology platforms that we can tap into,

and they’re ready to be integrated.

Mr. SHAYS. But it would be unlikely for 3M to go into development and production without the technology there to back it up?

Mr. LARKINS. We absolutely would not.

Mr. SHAYS. You would not. So the general principle that there is

significant logic in developing the technology before you went into

development and production still holds for the defense as well.

129

Mr. LARKINS. Well, all I can say is that we do not go into a product scale-up mode until we have the technologies in hand.

Mr. SHAYS. So, then, if you can’t meet the deadline, you just

don’t produce the product?

Mr. LARKINS. That’s correct.

Mr. SHAYS. Dr. Harris, do you want to comment on the questions

I have asked so far?

Mr. HARRIS. Yes, I would like to comment, sir.

The use of the words ‘‘commercial best practices’’ and our consideration of the—this environment of acquisition of defense systems,

that phrase is a very loaded one. It has many meanings and usually it differs, depending upon the speaker.

Mr. SHAYS. Is that—if I could interrupt—is that because there is

no, one, so-called ‘‘commercial best practice’’; there are—different

companies have their version of best practices?

Mr. HARRIS. Well, that’s somewhat it, sir, but I think—my statement is based primarily on the fact that there’s no commonly

agreed-upon body of knowledge as to what it really is. It’s a sort

of catchall phrase that has caught many up into some more political stance than real substance.

For example, the question of scaling along technology complexity,

the question of scaling in terms of number of products, the questions of availability of markets all would impact commercial practices. And you don’t have a serious discussion of commercial best

practices being transferable along what is in fact scalable.

So I’m somewhat personally concerned as to whether—what we

really mean when we say that DOD or the government ought to

rush immediately into commercial practices. It just doesn’t have

the substance that I think it must have in order to hold those who

practice within government accountable when we say that they

ought to use or should be using best practices.

Mr. SHAYS. So you would be more sympathetic to the view that—

that when we use TRL, technical readiness levels, that it be more

a tool than a requirement. And you heard them refer——

Mr. HARRIS. Yes, I did hear that discussion, sir. Yes, Mr. Chairman. I think we’re wrapping ourselves around an axle with the

qualitative discussions of technology readiness levels. Until we can

move to a position where we can quantify what we mean by technology readiness, we can’t hold those who practice, who acquire the

weapons systems, accountable. Because they and their contractors

will always be at each other’s throats as to what they mean when

they present an argument or position on technology readiness levels.

So I’m calling for more quantitative language and assessment

tools to enable all of us to understand precisely where we are within this arena called ‘‘technology readiness levels.’’

Mr. SHAYS. That’s spoken like a true academician. You’re fulfilling your role perfectly.

Well, one of the things is—and I’m not burdened by being a lawyer, and I don’t have any perceived—and the reason why I love this

committee and love the work as a legislator, I’m not trying to prove

a point. I’m trying to understand what we as a committee need to

recommend to the full Congress and to the executive branch.

130

But intuitively, I have accepted as a fact—and, Dr. McNaugher,

I’m going to try to have you separate a little bit the so-called politics side of this, and then we’ll get back into that. But I accept as

just very logical, without a lot of empirical knowledge, but just your

own life experiences, that if you are going to be developing technology and development, and fail to come to grips with some technology; and then you’ve got a point where the technology hasn’t

been resolved and yet you’re still moving things along, that your

costs are going to go up significantly. You’re going to begin to slow

down your development after you started, particularly in production.

And I accept the point that you made, Dr. McNaugher, that

sometimes we haven’t perfected the product until we have produced

100 planes. And that’s a real negative.

So, Dr. Harris, tell me why intuitively I shouldn’t accept the fact

that we should develop the technology before we go into development, but particularly into production.

Mr. HARRIS. Well, I can’t tell you that. I want my testimony to

read very clearly that we—— Mr. SHAYS. Don’t put words in your mouth.

Mr. HARRIS. We as a Nation have, in my opinion, been successful

in developing technology in advance of acquisition of the full system. My experiences at NASA with the high-speed civil transport,

the new 700,000-pound Mach 2.5 airplane, we developed that technology with industry as partners. The airplane was never built because it was a business decision, not because the technology was

not there, or was designed to be there, in advance of building the

airplane. So we know how to build or how to develop technology in

advance of acquisition.

Mr. SHAYS. But the question is, is it going to be cost effective?

Mr. HARRIS. Oh, I would say it definitely is cost effective. And

the effectiveness goes up, I believe, with the complexity of the final

product. If you’re talking about very complicated systems that require a real stretch in technology, then clearly the effectiveness is

going to be there, the cost effectiveness.

Mr. SHAYS. I’ll allow the other two to jump in in a second, but

where I’m not with you is, I would think that waiting to develop

the technology before you did the development, but particularly

production, would be to your advantage as long as you don’t have

the time restraint of having to get into the marketplace because

someone else is going to build the product.

Mr. HARRIS. Well, I agree, as I know the threat, at least the military threat, that this Nation currently faces, we have that time.

Mr. SHAYS. Right. OK.

Any other responses to any points that either of you, Mr. Larkins

or Dr. McNaugher, any comments you want to make?

When you heard the debate between—the dialog between the

first panel and the second panel, I want each of you to tell me your

reaction not in—as concise as you can; and tell me how you reacted

to the two different testimonies, because I think both testimonies

were quite excellent. I think GAO gave us a wonderful vehicle in

which to have this debate and dialog.

But I would love to know how you reacted as you heard it.

No criticism of the committee allowed, just the other side.

131

Mr. HARRIS. Well, I’ll go first.

Mr. MCNAUGHER. Yeah, feel free. I’m happy to defer.

Mr. HARRIS. I thought the first panel was again a very excellent

presentation. I think there was an honest mistake in interpreting

what technology readiness levels were meant. Was it to be on a

full-up system or a comparable system? And that led, I think, to

a downgrading of the so-called ‘‘technology readiness levels.’’ What I thought was missing in the first presentation was a normalization of the two possible streams to the same system, one

stream as to what we currently have, or at least what the first

panel proposes that we have, that we buy this complex weapons

system and develop the technology while we buy it. Another stream

would be to develop the critical technology off stream, then insert

it when we are ready to move forward with the full system.

Now, my question is this: What is the cost going along the first

stream and what is the total cost going along the second stream?

And what are the time lines involved? When does the clock start?

Mr. SHAYS. How would we know that if we don’t know if we can

actually develop the technology?

Mr. HARRIS. As far as I know, sir, no one has ever done a comparative analysis of that sort—unless Dr. McNaugher has done it.

Yet we get this constant bombardment of how to develop systems

and how to develop technologies that support systems off line.

But the answer is, Mr. Chairman, no one really knows.

Mr. SHAYS. What I heard, and then I’ll open it to the other two

of you, when I found myself wondering if moving forward with development represents—where you still have high risk on technology, whether that represents a significant negative. But I clearly feel that if you went into production without the technology,

then we are really opening up potential costs—I mean, that ratio.

So I found myself more tolerant of DOD kind of wanting to go to

that development stage.

I would become very concerned if they wanted to go that one step

further. In other words, I think there’s still this high risk in some

of their technology.

Mr. HARRIS. OK. I don’t know the specific technologies involved

with this weapons system, as I am not a consultant to any of the

primes. But there are two different ways of getting to the same end

goal; a highly complex system is what we don’t have a comparative

analysis to.

Mr. SHAYS. Fair enough.

Dr. McNaugher.

Mr. MCNAUGHER. First of all, I don’t know enough about the

TRL methodology to comment on it explicitly. I think it’s absolutely

appropriate to be hammering on the readiness of these technologies. The disagreement between the two prior panels—and not

everybody agreed that there was a disagreement—seemed to circle

around the question of whether you’re assessing risk in the context

of systems integration or the risk of specific technologies. I can at

least understand why you get a different number in each of those

cases, given what I said earlier.

And so the one upshot I would have, and I think I’m echoing

something Dr. Harris said, is whenever you move or this project

moves from ‘‘dem-val’’ to EMD, don’t think that just because you

132

have reached TRL 7 for components, you’re out of the woods.

There’s a lot of development work and there’s going to be a lot of

sometimes nasty surprises up ahead.

Mr. SHAYS. And I don’t think GAO was suggesting that.

Mr. MCNAUGHER. Agreed. So I can understand why there might

be a different measure, depending upon whether you’re thinking of

the specific technology or that technology in the context of an overall system, because the system does pose its own uncertainties.

Mr. SHAYS. Mr. Larkins.

Mr. LARKINS. There seemed to me to be two basic issues that

came up. I think—— Mr. SHAYS. Can you move the mic just a little closer.

Mr. LARKINS. There seem to be two basic issues of disagreement.

I think there was an awful lot of agreement, but there were two

basic issues that I got that there was disagreement. One was

that—one was the TRL as—the definition used in each individual

category. And again I have to agree with Dr. Harris here; I think

that if you’re going to use categories to evaluate a project, you

should have general agreement by everybody, the people who are

being evaluated and the evaluators, as to what those categories are

and where they fit. And there obviously wasn’t any there.

The other disagreement was that there is a substantial difference

between a fully integrated system with eight apparent critical technologies versus evaluating each of the individual critical technologies.

And frankly, at 3M, when we look at integrating several new

technologies into a program, if we consider, for example, that each

technology has a 90 percent chance of success, then you have to do

a multiple of those chances of success to arrive at a final chance

of success for your program. So if each of your programs has a 2

percent chance of failure and you have—or each of your technologies has a 2 percent chance of failure and there are four technologies, you’re looking at a 16 percent chance of failure right

there.

So I am completely sympathetic with what the Department of

Defense is saying here.

Mr. SHAYS. And yet you would be sympathetic with the fact this

integration sometimes may mean that the technology doesn’t work?

Mr. LARKINS. That’s correct. And that’s why we frankly take a

very conservative view of that approach. We—and believe me, we

do take risks, but they are calculated risks; and the greater the

risk on a commercialization program, the tighter we monitor it.

Mr. SHAYS. Do you have any questions?

Mr. Chase would like to ask questions.

Mr. CHASE. Mr. Larkins, what kind of incentives do you provide

your teams, your project teams, to ensure that a new product that

you’re trying to develop comes in on time and within budget?

Mr. LARKINS. Well, there are, in my view anyway, three different

types of incentives, peer recognition and reward—and we have programs in place which recognize individuals and teams for outstanding success in product introduction—and, of course, promotion. And

when a person is a program manager and they’re involved in a successful product introduction, there are certainly promotions that

are involved in that type of an activity. And finally, financial incenVerDate 11-MAY-2000 11:50 May 24, 2001 Jkt 000000 PO 00000 Frm 00136 Fmt 6633 Sfmt 6633 C:\DOCS\70782.TXT HGOVREF1 PsN: HGOVREF1
133

tives as well, which go along with the promotion and separate from

that.

So recognition and reward and promotion.

Mr. SHAYS. Thank you. I was going to ask that question after you

did. Because I just wonder if in the—the Federal Government,

when they are utilizing the services of a private company, whether

those companies themselves have their own internal programs to

encourage as much innovation and extra effort and so on.

Let me say this to you. I’ve really asked the questions I wanted

to ask. I’m going to allow all of you to make some closing comments

or ask yourself a question you wish I had asked, that you think I

should have asked.

I would also like to just ask if Mr. Rodrigues would mind coming

to the corner here, just to kind of share your observations.

Is there anyone from DOD that was here that would like to or—

if they hear a comment—do we have a representative from DOD

who would be?

We had let them know they would be invited if they were able

to stay. I realize they couldn’t necessarily stay. If you could, just

sit on the corner here, if you could do that.

Are there any comments that you would like to make or observations? If so, I would move the mic over to you.

This is not—the purpose of this is not a debate, but just to say

things that you would want us to focus in on.

Mr. RODRIGUES. I guess I would like to say a couple of things to

try to add a little clarification.

The work that we did on the Joint Strike Fighter and the application of the TRLs, being able to do that was the culmination of

years of work. It was not easy to define a model that reflected what

we defined as, at this point, ‘‘commercial best practices’’ that can

be applied in DOD. That was gleaned from a lot of work with a lot

of companies, and working with the Department looking at successes and failures and trying to categorize what people do to come

up with knowledge points that are critical to being able to move

forward successfully.

When we put that over at the Department of Defense for comment, the original comments that we were getting back were very,

very negative because it would require a significant change in the

way they do things.

Dr. Gansler, coming on board, he took that on and rewrote the

comments; and I can tell you I never get comments like this. At

the end, he writes a personal note, on the formal comments coming

over, ‘‘A very thoughtful and helpful report, Jack.’’ Believe me, I

never get these.

What we then did is, we said, OK, if you agree with—if you could

put that chart up, please.

Mr. SHAYS. You know, it’s interesting, just a nice little comment

like that goes a long way. We all need to do it, all of us.

Mr. RODRIGUES. That’s the only one I’ve ever gotten.

So if you look at this chart, then we said, OK, if everybody can

agree with this—and we had to come up with a model, run it back

by all the commercial companies we did, because they didn’t know

how to define what they did.

134

They don’t think about it in this term. Nobody had a model for

us to use. We had to work and find it. Find it, define it, get it understood by both parties, DOD and the commercial guys; and let’s

agree to some things.

Once we got agreement on that, we went back and said, if this

is true, let’s peel the onion. If technology development, separating

technology development from product development is so critical,

how do you do it?

Mr. SHAYS. I need to know what your bottom line is.

Mr. RODRIGUES. My bottom line is, we did a whole study to come

up with the technology readiness levels. We didn’t get there by accident. We did a whole thing focusing on knowledge point one. And

we worked with a lot of individual technologies, both commercial

and defense. What we found was that they work, that they can predict outcome, that if you meet the right technology level, you can

get success. If you don’t, you are virtually guaranteed failure.

Mr. SHAYS. You’re a brave guy to say that right next to Dr. Harris.

Mr. RODRIGUES. I’m not guaranteeing there aren’t better ways to

do it. I’m just saying they are a great indicator.

Now, where I was going with this was or trying—— Mr. SHAYS. Just don’t get too deep here.

Mr. RODRIGUES. In that report, before we applied TRLs to the

JSF Program—see, we applied it in the abstract and didn’t criticize

anything—we never said anything.

We said, here they are. Here’s a tool. Isn’t it wonderful? Can’t

you guys use this? Doesn’t it make sense? When we defined it in

there—and this went to the Department for comment, and I’ll read

you their comments. We said—we defined TRL 7. We were talking

about where does integration fit. There are two types of integration, there is subsystem integration and there is product integration.

Mr. SHAYS. OK.

Mr. RODRIGUES. We got a whole bunch of technologies that are

at subsystems. Integrating proven technologies into a product is a

real challenge, and it is something that has to be done. Matching

technology to requirement up front doesn’t stop that from happening. You still have to do it. It’s a challenge.

What we’re trying to do is isolate that, because what should be

done in product development—isolate that from the technology development which is proving the technologies in the subsystem

forms in which they have to be, so you’re not trying to do that concurrently, especially on pacing items. So we said when radio components—this is in the TRL, report—are assembled inside a case

that resembles a final radio design and then demonstrated aboard

a surrogate of the intended aircraft, the radio reads TRL 7 now

when they comment on this report, they say that the Department

agrees—this is their writing, it’s not mine, reprinted in the report—the Department agrees that TRL is an important input and

is necessary, but adds they’re not sufficient alone to decide when

and where to insert new technologies in a weapons system programs.

Military system development decisions require a total ownership

cost approach through the entire life cycle system. Now I totally

135

agree with that. We weren’t trying to imply that once you reach 7

you have to put it on a product. We were saying if you don’t reach

7, you’ve got problems and you shouldn’t put it on a product. They

went on to say in their comments—— Mr. SHAYS. I want you to bring this to a close.

Mr. RODRIGUES. This is their comment. The Department concurs

with GAO that the weapons system program manager should assure that technology is matured to TRL 7—I just read how we defined it in the report and they agreed—before insertion into a new

system. They agreed with that here.

The difference in the JSF study was we then cross-walked that,

and said, hey, you know what? When you apply this on the program you’re doing, you’re not there and you shouldn’t move ahead.

It wasn’t until we made that cross-walk in a draft report for comment that all of a sudden this issue of a misunderstanding, came

up. It came up once they saw how it was used. If they had read

this report in detail, which I had provided to them, they would

have seen how it would be used.

Mr. SHAYS. Let me just say I think they did read the report. The

advantage that GAO or Members of Congress have is that we can

look at something that someone else is building or doing, and we

can sit with hindsight and all the other things, and really be pretty

analytical about it.

I think your report was done very tastefully. I think it served a

tremendous use here and will be used by other committees because,

as you know, your report was used by the Armed Services Committee even before we released it today.

I think DOD was fairly respectful of your analysis. I think there

are—I have a sense of where our disagreements are. And a lot of

good is going to come from the report. And—but I’m excited that

this is a program being used by Defense. I want to make sure that

they’re using it to the fullest extent possible.

Let me just—I’ll let you say another comment here, but let me

just ask Dr. Harris or Dr. Larkins or Dr. McNaugher: Any question

you want to ask yourself and then answer brilliantly?

Mr. HARRIS. Mr. Chairman, no question, but I would like to conclude by simply repeating myself.

We, as a Nation, have demonstrated the capability to produce

weapons systems driven by advanced technology. To become more

efficient at the acquisition process, we must continue to develop advanced technology in parallel to acquisition of full, complete systems. Key elements in this efficiency are, No. 1, incentivize industry with favorable premiums, develop this advanced technology;

No. 2, realistic projections of advanced technology needs, meaning

a management of our DOD technology advancement portfolio;

third, use of quantitative-based language and assessment tools.

Mr. SHAYS. Thank you very much.

Mr. MCNAUGHER. Three points, the first taking off from Dr. Harris’ last remark; and it’s a subject of a different hearing, but one

that should be kept in mind.

We have never found how to make R&D per se profitable in the

defense industry. The historic tendency is, firms sink their own

money into development and get well in production. If you want to

relax this process and maybe even cancel the occasional system,

136

you probably want to have a way, a mechanism, for making R&D

profitable. We’ve never found that, and that’s the way you encourage industry. It’s a profound problem in the defense industrial

base, precisely in this area we’re moving into.

My second point has to do with the earlier discussion of the tenure of program managers. I must say I’m fascinated with the idea

of a program manager who actually has to live with the results of

his or her decisions 5 or 6 years later, I think it would be a profound change. Again, though, stepping back and looking at the full

sweep of American history, remember that before World War II the

Army had an Ordnance Department which was a bunch of full-time

program managers and technocrats, if you will. They wore uniforms, but they didn’t rotate.

I think the Army’s conclusion, as an institution coming out of

World War II, was that the Ordnance Department was not attentive enough to the user, the actual operator. And it was more fixated on the technologies than on military capability. What the

Army did in the 1950’s and 1960’s was begin to substitute line

users as program managers, and but also as the chief of R&D, for

example.

Mr. SHAYS. Wasn’t there also another danger that they developed

too cozy a relationship with the organization they were buying

from?

Mr. MCNAUGHER. Well—— Mr. SHAYS. Over time you just began to become almost their advocate, not necessarily their—— Mr. MCNAUGHER. My sense of the ordnance department, to the

extent that I understand the history, is that it had a somewhat

thorny relationship with commercial firms because it was doing its

own R&D, and it was always being besieged by Members of Congress pushing Samuel Colt’s pistol or rifle. So it was a more combative relationship.

Mr. SHAYS. So—I didn’t realize, they did the R&D; I just thought

they were——

Mr. MCNAUGHER. Springfield Arsenal. The arsenals did R&D, so

they were competing with industry to some extent. But the point

is, a program manager tenured for 15 years or 10 years may become very attentive to sort of the permanent features in the environment, of which the political structure is one, and not as attentive to the military user. So maybe there’s a length of tenure that

sort of balances those needs. I don’t think it’s 3 years either. I

think it’s longer than that.

And finally, at the risk of complicating things, I would just separate technologies into two bundles, slow-moving and fast-moving

technologies. If you look at an airplane, the engine airframe combination had its fast-moving days back in World War I and the

1950’s, and that’s slow-moving technology. You can afford to stop,

assess that technology, test it.

If you look at the electronics in the avionics—— Mr. SHAYS. Let me just interrupt you a second so we can ask a

question of the recorder.

Thank you.

Mr. MCNAUGHER. If we look at avionics in the cockpit of that airplane, which may come to 40 or 45 percent of the cost of the airVerDate 11-MAY-2000 11:50 May 24, 2001 Jkt 000000 PO 00000 Frm 00140 Fmt 6633 Sfmt 6633 C:\DOCS\70782.TXT HGOVREF1 PsN: HGOVREF1
137

plane, we’re looking at technologies where if you stop and test and

decide, you know, they’re already obsolete—I mean, they’re turning

over in a year, 18 months, 2 years—and I’ve always felt that we

really need—now that electronics technologies are seen to be the

key to the revolution in military affairs we really need to handle

these by different commercial practices. One can be much slower

and more judicious because you’re way out on the flat of the cost

performance curve; you’re pushing for that marginal improvement

in jet engines or airframes. The other is just so fast-moving that

you almost have to be turning your design and your tests over constantly. The DOD needs a different approach entirely.

Mr. SHAYS. I thank you.

Mr. Larkins, I knew that would bring you out.

Mr. LARKINS. If I could just make one.

Mr. SHAYS. Would you move the mic, please, toward you.

Mr. LARKINS. If I could just make one additional comment, I

would say that our experience in industry with the importance of

focusing on fewer larger programs, that having trained people who

are—whom we call program managers or program development

managers, whose expertise is scaling up new programs. We are beginning to see within industry a focus on that as a discipline.

3M traditionally has what we call a dual-ladder system where we

have technology people on one side of the promotion scale and management people on the other side of the professional scale. So you

have research and management moving up together, the same opportunities for advancement, the same opportunities for pay, this

kind of thing. But we are now in industry beginning to see a third

leg or a third ladder in this process, which is a focus on program

management.

Mr. SHAYS. Any other comment?

Mr. RODRIGUES. I would like to make one, because this becomes

a real area of concern; it is related to this. I have started looking

at some data really hard-looking at this whole issue, this issue of

profitability for research and development. In private industry, research and development is investment, the return is on sales. You

don’t make money in R&D. You make money building a product

and selling it to somebody. The same principles should apply in the

Department of Defense and should apply in the defense industry.

The reward for doing the right thing is you sell us something that

you make a profit on.

When I looked at this issue and was trying to study this, what

I find is when I look at tactical aircraft, from 1973 to 1991 we built

anywhere, annually, from 350 to 500 aircraft. From 1991 to today,

we’re building handsful, 40, 50 aircraft, sometimes much, much

less than that.

And then we have an industry saying, we’re not profitable; Wall

Street is beating us up. There are a lot of other factors, but one

of them is, we’re not producing a lot.

We got into a cycle where we’re heavily into R&D and so now the

issue becomes, how do we make R&D profitable? The issue becomes, we have a modernization strategy—I think this is a real

issue that we need to take a hard look at—that drives us to cycles

that make the industry unprofitable at times.

138

Is the solution to pay them for R&D, pay profit rates, increase

profit rates on something that is normally an investment account;

or is the solution to better plan on modernization so that you

equalize production, so people can make money and we can modernize?

Mr. SHAYS. I think it’s a very fine point.

I saw Dr. Harris start to waiver a little bit here. I am at overload. So whatever you guys are, I’m at overload; I see a new recorder, and we’re done.

Julie Thomas, thank you for all your work. There’s no question

you were working hard today.

And I enjoyed this hearing. I learned a lot. I think our witnesses

and all three panels were terrific. And I also appreciate the staff

for both the majority and minority for all their good work. So, everyone, have a beautiful day. This hearing is over.

[Whereupon, at 2 p.m., the subcommittee was adjourned.]

Æ