U.S. GOVERNMENT PRINTING OFFICE

WASHINGTON : 

For sale by the Superintendent of Documents, U.S. Government Printing Office

Internet: bookstore.gpo.gov Phone: toll free (866) 512–1800; DC area (202) 512–1800

Fax: (202) 512–2104 Mail: Stop IDCC, Washington, DC 20402–0001

i 

50–297 2010

[H.A.S.C. No. 111–63] 

MEASURING PERFORMANCE: 

DEVELOPING GOOD 

ACQUISITION METRICS 

HEARING 

BEFORE THE

PANEL ON DEFENSE ACQUISITION REFORM 

OF THE

COMMITTEE ON ARMED SERVICES 

HOUSE OF REPRESENTATIVES 

ONE HUNDRED ELEVENTH CONGRESS 

FIRST SESSION 

HEARING HELD 

MAY 19, 2009 
(II) 

PANEL ON DEFENSE ACQUISITION REFORM 

ROBERT ANDREWS, New Jersey, Chairman 

JIM COOPER, Tennessee 

BRAD ELLSWORTH, Indiana 

JOE SESTAK, Pennsylvania 

K. MICHAEL CONAWAY, Texas 

DUNCAN HUNTER, California 

MIKE COFFMAN, Colorado 

ANDREW HUNTER, Professional Staff Member 

JOHN WASON, Professional Staff Member 

ALICIA HALEY, Staff Assistant 
(III) 

C O N T E N T S 

CHRONOLOGICAL LIST OF HEARINGS 

2009 

Page 

HEARING: 

Tuesday, May 19, 2009, Measuring Performance: Developing Good Acquisition 

Metrics .................................................................................................................. 1 

APPENDIX: 

Tuesday, May 19, 2009 ............................................................................................ 23 

TUESDAY, MAY 19, 2009 

MEASURING PERFORMANCE: DEVELOPING GOOD ACQUISITION 

METRICS 

STATEMENTS PRESENTED BY MEMBERS OF CONGRESS 

Andrews, Hon. Robert, a Representative from New Jersey, Chairman, Panel 

on Defense Acquisition Reform ........................................................................... 1 

Coffman, Hon. Mike, a Representative from Colorado, Panel on Defense Acquisition Reform ................................................................................................... 2 

WITNESSES 

Fitch, David P., USN (Retired), Director, AT&L Leadership Learning Center 

of Excellence, Defense Acquisition University ................................................... 7 

Nussbaum, Dr. Daniel A., Visiting Professor, Department of Operations Research, Naval Postgraduate School .................................................................... 9 

Patterson, J. David, USAF (Retired), Executive Director, National Defense 

Business Institute, University of Tennessee ...................................................... 5 

APPENDIX 

PREPARED STATEMENTS: 

Andrews, Hon. Robert ...................................................................................... 27 

Coffman, Hon. Mike ......................................................................................... 28 

Dillard, John T., Senior Lecturer, Graduate School of Business and Public 

Policy, Naval Postgraduate School .............................................................. 77 

Fitch, David P. .................................................................................................. 57 

Nussbaum, Dr. Daniel A. ................................................................................. 66 

Patterson, J. David ........................................................................................... 32 

DOCUMENTS SUBMITTED FOR THE RECORD: 

[There were no Documents submitted.] 

WITNESS RESPONSES TO QUESTIONS ASKED DURING THE HEARING: 

Mr. Andrews ..................................................................................................... 83 

QUESTIONS SUBMITTED BY MEMBERS POST HEARING: 

[There were no Questions submitted post hearing.] 

(1) 

MEASURING PERFORMANCE: DEVELOPING GOOD 

ACQUISITION METRICS 

HOUSE OF REPRESENTATIVES, COMMITTEE ON ARMED SERVICES, DEFENSE ACQUISITION REFORM PANEL, Washington, DC, Tuesday, May 19, 2009. 

The panel met, pursuant to call, at 8:00 a.m., in room 2212, Rayburn House Office Building, Hon. Robert Andrews (chairman of the 

panel) presiding. 

OPENING STATEMENT OF HON. ROBERT ANDREWS, A REPRESENTATIVE FROM NEW JERSEY, CHAIRMAN, PANEL ON 

DEFENSE ACQUISITION REFORM 

Mr. ANDREWS. Good morning, ladies and gentlemen. The panel 

will come to order. Appreciate your attendance. I especially want 

to thank the witnesses for their very diligent preparation. We have 

had the opportunity to review the testimony ahead of time. It looks 

like we are going to have a very engaging and meaningful discussion this morning. 

This is the last of our series of hearings looking at the first question that we are going to be looking at in our inquiry. The members 

will recall that we are proceeding on a series of questions, the first 

of which is, ‘‘Can we design a series of metrics that accurately 

measure the difference, if any, between the price paid by the taxpayers and the value received by the taxpayers and the warfighters 

for the systems and services that we are buying?’’ 

We are going to proceed after today’s hearing with our second 

category of questions, which is really hypotheses as to what has 

gone wrong. We are going to have a series of panels talk about 

their theories and analyses of why we have a difference between 

the price paid and the value received. 

The first of our two panels on this metrics question dealt with 

the sort of measuring the orthodox algorithms that are used. In the 

major weapons system side, we had a panel on that, and on the 

services side, we had a panel on that. 

The purpose of today’s panel is to bring in some people who we 

think perhaps look at the whole question through a different prism. 

They are willing to give us some new perspectives through which 

we can analyze the difference between the price we pay and the 

value that we receive. 

We have three witnesses with a wealth and breadth of experience in the acquisition field, but also who I think benefit from what 

I would call a healthy distance from the daily responsibilities for 

that function, so they can give us a perspective that marries experience with a fresh perspective. And hopefully, we will be able to use 
2 

the testimony of these witnesses to go forward, draw some conclusions about the best metrics to use in our work, and then proceed 

with our work in analyzing the various hypotheses given for why 

we have suffered these cost overruns. 

I also want to take a moment and thank Mr. Conaway, who can’t 

be with us this morning but who is ably represented by Mr. Coffman, and all the members of the panel for their excellent contribution to the work on the Weapons Acquisition System Reform 

Through Enhancing Technical Knowledge and Oversight (WASTE 

TKO) bill, which has passed the House. We will be meeting with 

the Senate in formal conference at 4:30 this afternoon to, I believe, 

put the finishing touches on that conference report so we will be 

able to proceed to the floor in short order on that, get the bill on 

the President’s desk. Each member of the panel has made a very 

significant and welcome contribution to that effort. 

Having said that, as the chairman and Mr. McHugh have said, 

we think that that effort covers about 20 percent of the landscape. 

It looks at major weapons acquisition, which certainly needed to be 

examined. But there are so many other areas that fall outside of 

that that the panel has to pursue, as well as, frankly, reviewing 

the early stage implementation of the law that we believe the 

President will sign this week. 

So we are by no means at the conclusion of our work. We are 

really at the outset of it. And one thing we would ask the witnesses 

to do this morning is to think about the fact that, although the 

Congress is about to pass major legislation dealing with major 

weapons systems, we have not yet addressed hardware that doesn’t 

fall into the major weapons system category and the 55 to 60 percent of acquisition that is services that we have to look at on behalf 

of those who wear the uniform and on behalf of the taxpayers. 

We are glad that you are here. And at this time, I want to turn 

to Mr. Coffman for his opening statement, and we will proceed with 

the witnesses after that. 

[The prepared statement of Mr. Andrews can be found in the Appendix on page 27.] 

STATEMENT OF HON. MIKE COFFMAN, A REPRESENTATIVE 

FROM COLORADO, PANEL ON DEFENSE ACQUISITION REFORM 

Mr. COFFMAN. Good morning. Mr. Chairman, ladies and gentlemen, I would like to extend a welcome on behalf of Ranking Member Conaway, as well. He is sorry he could not be with us today. 

I would like to thank the chairman for allowing me to take a few 

introductory remarks in Representative Conaway’s place. 

Today’s hearing is an appropriate follow on to the panel’s two 

hearings which focus on how the Department of Defense (DOD) 

and the Government Accountability Office (GAO) currently assess 

performance on major weapons systems and service contracts. The 

purpose of this hearing is to think outside the box on how we 

should measure performance and about what we should be measuring and less about the how we are doing it today. 

Our previous hearings revealed that current measures of performance tend to break down if the program baseline is unrealistic. 

We want to know if the program can be corrected—if the problem 
3 

can be corrected. Furthermore, are there metrics beyond cost and 

schedule performance that are of value, such as how closely does 

the delivery system meet actual warfighter needs. Does the time of 

delivery of operational capabilities satisfy warfighter needs? How 

do we determine whether optional or tradable capabilities requested by the warfighter are affordable? 

Mr. Chairman, we have a distinguished panel of witnesses in 

front of us today. I have looked at their written testimony, and I 

look forward to their testimony. There are a couple of points I 

would like to highlight. Although Mr. Dillard is not testifying 

today, he did submit a written statement that I would like to briefly comment on. 

[The prepared statement of Mr. Dillard can be found in the Appendix on page 77.] 

Mr. COFFMAN. He states that, ‘‘The proliferation of autonomous 

fiefdoms within the department continues to increase, with each 

being a stovepipe of oversight expertise, imposing unique reporting 

requirements, assessments and reviews.’’ 

He goes on to state that, in regards to adding additional acquisition workforce professionals, that, ‘‘These new people should not be 

housed in the Pentagon, but instead where the execution of programs occur.’’ Mr. Chairman, I think this is an important point 

that I believe we need to follow closely. 

Another point of observation I would like to make is in regards 

to requirements in joint programs. We have many programs out 

there that have the word ‘‘joint’’ in front of them, but they are joint 

in name only. I understand that the current requirements generation process is called the Joint Capabilities Integration and Development System, or JCIDS. 

It seems to me that, in order to have a truly joint program that 

the requirements for that program must be borne jointly. Yet, I do 

not believe our current system, or as Mr. Dillard describes, current 

fiefdoms, foster such an environment. So I would be interested in 

hearing from our witnesses in this regard. 

I also encourage our witnesses to share their views on existing 

laws and regulations that are particularly helpful or not helpful to 

the Department’s efforts to obtain the best value and capability for 

our warfighters. We have heard from the Department in two separate hearings about how they currently measure performance and 

value on contracts. What we haven’t heard enough about is how 

they should be measuring value. Consequently, your input will be 

greatly appreciated. 

With that, I conclude. And again, thank you, fellow members, 

and thank you, Mr. Chairman. I look forward to the witnesses’ testimony. 

Mr. ANDREWS. Mr. Coffman, thank you. 

Without objection, the statement of any other member of the 

panel will be included in the record. And without objection, the 

statement of each of the three witnesses, plus the supplemental 

floor statement, will be included in the record of the hearing. 

[The prepared statement of Mr. Coffman can be found in the Appendix on page 28.] 

Mr. ANDREWS. I think each of you are veterans at this process. 

You know that we generally have a five-minute rule where we ask 
4 

for succinct summaries of your written testimony. We are going to 

be a little bit more liberal with that this morning, but then we are 

going to turn as quickly as we can to the questions from the panel 

and get into the give-and-take. 

I am going to read a brief biography of each witness, and then, 

Mr. Patterson, we are going to begin with you. 

Mr. Dave Patterson is the Executive Director of the National Defense Business Institute, which he is establishing at the University 

of Tennessee in the College of Business Administration. It is an institution inspiring business innovation for both government and industry by providing practical, sound assistance and creating economically efficient and effective defense business and acquisition 

programs. 

Prior to taking his current duties, he was the Principal Deputy 

Under Secretary of Defense Comptroller. As the Principal Deputy, 

he was directly responsible for advising and assisting the Under 

Secretary of Defense with development, execution and oversight of 

the DOD budget exceeding $515 billion, with annual supplemental 

requests of more than $160 billion. 

From August 2003 to June 2005, Mr. Patterson held duties as 

the Special Assistant to the Deputy Secretary of Defense. In that 

capacity, he was responsible for managing the Deputy Secretary of 

Defense’s personal staff, as well as providing direction and advice 

to the Office of the Secretary of Defense staff on a wide range of 

national security operations and policy subjects. 

He served in the Air Force from 1970 to 1993, retiring in the 

rank of colonel. During that time, he held responsible leadership 

and management positions, with assignments at the air wing level 

as the C5–A aircraft Commander and Deputy Operations Group 

Commander at Major Command Headquarters, U.S. Air Force, the 

Office of the Chairman, Joint Chiefs of Staff, the Office of the Secretary of Defense and Inspector General. In 1986, he was the Air 

Force Fellow at the American Enterprise Institute and served in 

Vietnam, flying 02–As as forward air controller. 

Thank you for your service to our country, Mr. Patterson. Glad 

to have you with us. 

Mr. Fitch, Mr. David P. Fitch enlisted in the Navy in 1966. He 

was commissioned in 1968 after graduation from San Jose State 

College and Aviation Officer Candidate School in Pensacola, Florida. 

In 1969, after fixed and rotary wing flight training, he was designated a naval aviator and a distinguished naval graduate. He retired from the Navy in 1998 as a captain, following a career that 

included three operational and major acquisition command assignments. 

Mr. Fitch culminated his tour in the Navy as the major program 

manager for the international and joint multi-functional information distribution system. He was a 1997 recipient of both the David 

Packard Excellence in Acquisition Award and the Department of 

Defense Value Engineering Award. 

In 2001, after nearly three years in the defense industry, Mr. 

Fitch joined the faculty of the Defense Acquisition University, 

teaching and consulting acquisition executives and program management and other acquisition disciplines. 
5 

In 2006, he led a major independent study of the Coast Guard 

Deep Water program. In July of 2008, he assumed the position of 

Director, Acquisition, Technology and Logistics (AT&L) Leadership 

Learning Center of Excellence after nearly seven years as the Dean 

of the Defense Systems Management College. 

Mr. Fitch holds degrees in business and industrial management 

from San Jose State college, an M.S. in education from the University of Southern California, and he is a graduate with highest distinction from the Naval War College in Newport, Rhode Island. 

Thank you, Mr. Fitch, for your service to our country, and we are 

glad you are with us this morning. 

Dr. Daniel Nussbaum, from 2004 to the present, has been a visiting professor at the Naval Postgraduate School operations research department. From 1999 to 2004, he was a principal at Booz 

Allen Hamilton, responsible for a broad range of costs, financial 

and economic analyses with clients across the government and 

commercial spectrum. 

From 1996 to 1999, he was the director of the Naval Center for 

Cost Analysis at the Office of the Assistant Secretary of the Navy 

for Financial Management and Comptroller here in D.C. From 

1987 to 1996, he was division head of the Naval Center for Cost 

Analysis. From 1982 to 1986, he was Deputy Director and Acting 

Director for operations research and cost analysis divisions of the 

Naval Air Systems Command in Washington, D.C. 

Again, a very distinguished servant of our country, three excellent people with experience and fresh insight. We are happy to 

have you with us. 

Mr. Patterson, we would like you to begin with your testimony. 

STATEMENT OF J. DAVID PATTERSON, USAF (RETIRED), EXECUTIVE DIRECTOR, NATIONAL DEFENSE BUSINESS INSTITUTE, UNIVERSITY OF TENNESSEE 

Mr. PATTERSON. Thank you, Mr. Chairman, members of the Defense Acquisition Reform Panel. I am very pleased to be here this 

morning to participate in a discussion of a question that has clearly 

captured the attention of the current Administration and Congress: 

How should Congress assist the Department of Defense in improving its acquisition of weapons and services so that it can meet the 

needs of the warfighter in the field while still being a good steward 

of the taxpayer’s dollars? 

The first consideration for judging the success of an acquisition 

program is whether it fielded a weapons system or information system or service in time to make a positive impact for the warfighter. 

A system or service fielded too late to need may as well not have 

been bought at all. The phrase, ‘‘too little, too late,’’ can mean lost 

lives. 

But before we look at the measures of the acquisition system 

merit, there is another consideration central to this discussion. 

When Secretary Gates made his budget announcement on April 

6th, 2009, I believe he was speaking from frustration that was as 

much about what has been the persistent problem with the acquisition system that we depend on that is simply not responsive to the 

immediate warfighter needs, as it was about winnowing bloated, 

failed and unnecessary programs. 
6 

Implicit in that expression of frustration is a clear lack of confidence in a system that actually produces programs with uncertainty and instability. The most dramatic improvement metric will 

be when the senior leadership in the Administration, Congress and 

the Department of Defense have seen such improvements, results, 

not words, that they can say they have renewed confidence in the 

stability, predictability and effectiveness of the defense acquisition 

system. 

The Defense Acquisition Performance Assessment’s Report contended that program stability and predictability were singularly 

and uniquely crucial to managing programs that were on cost, on 

schedule and performing. To that end, in the time I have, allow me 

to describe two areas of improvement for measuring program effectiveness worthy of attention. 

First, Major Defense Acquisition Programs, or MDAPs, often 

start at milestone B, the beginning of engineering and manufacturing development, with critical staff positions vacant. Percentage 

of critical staff positions filled at milestone B is an easy and important metric to be observed. It makes little difference to implement 

programs to raise the level of skills of the program staff if they are 

missing in action. 

Second, the acquisition strategy document, that is to lay out how 

the weapon system is to be acquired, the initial roadmap, if you 

will, is often flawed in that it focuses more on presenting a case 

for required capabilities and quantities than on laying out the reasoning for acquisition competition methodologies. 

For example, how the prime contractor participants in an MDAP 

competition will select subcontractors and how the winner of the 

competition will manage the subcontractors to gain improved efficiencies and effectiveness are generally given little consideration. 

Creation of the acquisition strategy document is one of, if not the 

most, important tasks the government acquisition program management can undertake. 

The strategy should establish the template for all the activities 

that will take place throughout the source selection process, engineering manufacturing development, and follow-on production and 

fielding. More important, it establishes how the program management team is thinking about the numerous events and activities 

that a program will encounter. 

The defense acquisition executive should establish a common set 

of strategy elements that all the military department service acquisition executives must include in MDAP acquisition strategy documents. Additionally, a set of standards or metrics by which the 

strategy elements can be evaluated as effective must be part of 

that process. 

In closing, I would be remiss if I didn’t acknowledge the progress 

that has been made by the Department in improving the acquisition system over the last four years. Though it is the General Accountability Office’s headline that the 96 major acquisition programs have grown in cost by $296 billion that gets attention, those 

numbers belie an equally worthy but overlooked statistic published 

in the very same report. 

The average increase in unit cost of the 28 MDAP programs with 

less than 5 years since development started is only 1 percent. Com-
7 

pared with an average of 55 percent increase in acquisition unit 

cost of 25 programs in the group with 5 to 9 years since program 

development start, there has been important improvement that 

should be recognized. 

With that, it is my privilege to be here. Thank you very much, 

and I would be happy to take any questions that you have. 

[The prepared statement of Mr. Patterson can be found in the 

Appendix on page 32.] 

Mr. ANDREWS. Thank you, Mr. Patterson. 

Mr. Fitch. 

STATEMENT OF DAVID P. FITCH, USN (RETIRED), DIRECTOR, 

AT&L LEADERSHIP LEARNING CENTER OF EXCELLENCE, 

DEFENSE ACQUISITION UNIVERSITY 

Mr. FITCH. Chairman Andrews, Congressman Coffman, members 

of the panel, thank you for the opportunity to appear here today. 

I will address the subject of acquisition performance metrics and 

your questions about how to increase the realism of program baselines, making trades between affordability and performance and 

how to assess the value of systems that do not necessarily—please 

recognize that my opinions do not necessarily reflect the views of 

the Defense Acquisition University (DAU), the Department of Defense or the Administration. 

Mr. ANDREWS. That is why we are interested. 

Mr. FITCH. I suspected as much. 

Measurement of acquisition performance must encompass both 

strategic and tactical elements. As emphasized in a recent Defense 

Science Board report titled, ‘‘Creating a DOD Strategic Acquisition 

Platform,’’ the management, execution and oversight of acquisition 

programs is moot if we aren’t spending taxpayer dollars to buy the 

right capabilities if we aren’t demonstrating strategic choice. It is 

as important to decide what capabilities we won’t buy as well as 

what we will buy. 

I believe one of the root causes of funding instability is what I 

described as too many programs chasing too few dollars. Too many 

programs chasing too few dollars is one of the root causes, I believe, of overly optimistic cost estimates. The recently implemented 

Materiel Development Decision (MDD) process provides a framework for making strategic investment decisions. 

The MDD is the formal point of inquiry into the acquisition process. MDD will increase integration of the three major acquisition 

support systems, requirements, resources, and acquisition. 

Improving requirements management, an initiative supported by 

the Congress, includes providing training to requirements writers 

and managers to ensure that they have a sufficient understanding 

of the critical elements of acquisition, such as systems engineering 

and testing. The goal is to improve collaboration between acquisition and requirements communities to exploit cost and performance 

trades and improve acquisition outcomes. 

Having a formal requirements and acquisition process is not 

unique to the DOD. We can learn important lessons from commercial industry. If you compare the DOD acquisition system with the 

process to develop electronic games, there are marked similarities 

and differences. 
8 

Notably, the process to get games on the shelves for the December holiday season starts with a precise clarity of what will be developed by when, and includes a corporate commitment to the resources required for the project. Precise clarity is a result of intense interaction between the people that divine the capabilities of 

the game and the people that will design and test the software. 

Turning to the subject of tactical acquisition metrics, the most effective tools and templates incorporate metrics, both quantitative 

and qualitative. The question was raised, ‘‘Are there metrics beyond cost and schedule performance that are of value?’’ The answer 

is yes, and an ongoing example is a probability of program success 

metrics, POPS, that are currently being developed and deployed 

across the services and other federal agencies, such as the Department of Homeland Security. 

Starting with a blank sheet of paper, a group of DAU faculty 

comprised of experienced program managers and other functional 

experts ask themselves questions, including, ‘‘What conditions facilitate the success of programs? What metrics are leading indicators of derailment?’’ 

The resulting tool, POPS, uses a structured process to continually assess and display key elements of planning, resourcing, execution and external influences that promote or negatively impact 

program success. Still evolving, POPS is being used in the Army, 

Air Force, Navy, Marine Corps and Coast Guard. Timely, accurate 

and transparent metrics integrated in a management and oversight 

process, will produce better program outcomes. 

Another question that has been asked, ‘‘Can we ensure improved 

realistic baselines.’’ Again, I believe the answer is yes, and there 

are ongoing initiatives that will yield more realistic baselines. 

These include increased emphasis on technology readiness before 

starting major development, emphasis on improved cost estimating, 

which I am sure that we will talk about today, and competitive 

prototyping, which is included in the new DOD 5000.2 and pending 

legislation. 

Prototyping increases the opportunity to identify and assess affordability and capability traits. Competitive prototyping also allows the government to observe the performance of competing industry teams before making a down-select for engineering and 

manufacturing development. No matter how thoughtfully we plan 

and discipline source selection, a paper-only source selection process is never as good as demonstrated performance. 

The ultimate assessment of whether we have delivered value and 

needed capability to the warfighter is feedback from the field from 

the warfighter. At various times, I have seen photos of messages 

written with felt-tip pens on trucks and personnel carriers that 

have been returned to the government or industry depots. One of 

those signed messages—a picture is in my written testimony— 

reads, ‘‘This truck saved my life, as well as five others, 2–April– 

2008 at 2300 Lima, Basra, Iraq.’’ Those kinds of testimonies, those 

kinds of results, are what those of us that are in the acquisition 

aim to achieve on a daily basis by our efforts and commitment. 

Before equipment is fielded, it undergoes rigorous levels of development and operational testing, and the new DOD 5000 has increased emphasis on earlier testing. 
9 

Mr. Chairman, thank you very much for the opportunity to participate in today’s discussion. As the Secretary of Defense has said, 

there is no silver bullet. It will be a combination of initiatives and 

collaboration amongst all the stakeholders to create an acquisition 

system that consistently produces successful acquisition programs. 

I look forward to your questions. 

[The prepared statement of Mr. Fitch can be found in the Appendix on page 57.] 

Mr. ANDREWS. Mr. Fitch, thank you very much for your testimony. 

Dr. Nussbaum. 

STATEMENT OF DR. DANIEL A. NUSSBAUM, VISITING PROFESSOR, DEPARTMENT OF OPERATIONS RESEARCH, NAVAL 

POSTGRADUATE SCHOOL 

Mr. NUSSBAUM. Mr. Chairman, distinguished members of the 

panel, I would like to thank you for this opportunity to discuss my 

thoughts on how to improve acquisition and cost estimating processes in the Department of Defense. These ideas are mine alone. 

As the chairman said, I am a member of the faculty at the Naval 

Postgraduate School in Monterey, California. And I have spent the 

last 30 years mainly doing, and more recently, teaching and researching in the defense acquisition management system with a 

focus on cost estimating. I was a previous director of the Naval 

Center for Cost Analysis and past president of the Society of Cost 

Estimating and Analysis. 

All my experiences in cost estimating confirm that three things 

are necessary for sound cost estimating: Acceptance of the underlying uncertainties in predicting the future, accurate and plentiful 

historical data, and professionally trained and certified personnel. 

On uncertainty, there is intrinsic uncertainty in all estimates. It 

derives from several sources, mainly that we are usually designing 

or building or operating something that is substantively different 

from what we did before. The difference could be in the product or 

in the economic conditions or the programmatic conditions. 

An estimate reflects our knowledge at a point in time when we 

freeze the problem and base the cost on the configuration and 

programmatics as they are understood at that time. From that 

baseline, many things can change that can also change the cost estimate, including labor rates, overhead rates, schedules, enhancements of the capabilities or quantities, changes when a particular 

technical solution to a problem doesn’t work as planned. And we 

need an alternative technical solution. 

On data, a hallmark, a necessary characteristic of a sound cost 

estimate, is that it is based on historical program performance from 

similar or related ongoing or past programs. Historical data is variable. Not every aircraft costs the same as every other aircraft. 

And the measurement of this variability is accomplished through 

statistical constructs, things like standard error of the estimate, 

things like confidence intervals. We assume in our community of 

cost estimating that the patterns of the past will repeat in the future. But these patterns are almost always statistically grounded 

patterns modeled with the powerful and subtle techniques known 

collectively as regression. Further, we know of no alternative ap-
10 

proach to using the past as a guide to the future if we want a scientific that is a reproducible and auditable approach. 

Not all estimating is done by government employees. There really are three sub-communities. One is the government in-house estimators. One is employees of the large vendors who also design, develop and build what we buy—Boeing, Northrop, Lockheed Martin, 

for example. And thirdly, there are support contractors or consultants to the government. Those are the three communities. 

And surely, we need to increase the capacity and the quality, the 

numbers and the training of the government estimators, but so do 

we need to enhance the professionalism of the other two communities. 

There are currently no undergraduate curricula in cost estimating, and there are only four educational institutions that I am 

aware of that teach at least one course in cost estimating, and 

those are the Naval Postgraduate School in Monterey, where I am; 

the Air Force Institute of Technology at Wright-Patterson in Ohio; 

Defense Acquisition University, diverse locations with a capital 

campus at Fort Belvoir; and the Massachusetts Institute of Technology, which offers one elective course in cost estimating within 

its engineering curriculum. 

The recent separation of the business cost estimating and financial management career field into two separate cost estimating and 

financial management tracks is a very welcome development and 

should be supported, but note that DAU support is largely limited 

to military and DOD, not the other two communities. 

The Society of Cost Estimating and Analysis—we say SCEA— 

whose membership includes approximately one-third of all cost estimators supporting DOD, is a central and indispensable player in 

the training, initial certification and periodic recertification of cost 

estimators. I note with pleasure that the executive director of 

SCEA, Mr. Elmer Clegg, is in the room. 

SCEA has collected a body of cost estimating knowledge, and it 

provides the members of the cost estimating community, provides 

training in cost estimating, has developed and offers an examination and experience-based certification program. Lockheed Martin, 

Boeing, Northrop Grumman, other vendors use SCEA’s training 

and certification as their standard. 

I appreciate very much what this committee seeks to accomplish, 

Mr. Chairman. This concludes my prepared statement, and I would 

be pleased to answer your questions. 

[The prepared statement of Dr. Nussbaum can be found in the 

Appendix on page 66.] 

Mr. ANDREWS. Well, thank you, Dr. Nussbaum, and I thank each 

of the three witnesses for their testimony. As I say, we have had 

the chance to review the written testimony. We are now going to 

get to the questioning phase. 

Mr. Patterson, I was intrigued by your reference in your written 

testimony to two kinds of requirements costs, which you express as 

customer requirements and derived requirements. What is the difference between the two? 

Mr. PATTERSON. Customer requirements are established in key 

performance parameters, which are requirements that are established for a particular weapons system that the weapons system 
11 

then must perform against. Derived requirements are those requirements that the customer did not ask for but that, as the name 

would suggest, derive as a consequence of the design process. ‘‘Oh, 

look, we could do this better if only we’’—and of course, that is 

taken to the program manager. 

The program manager will say, ‘‘Okay, we can do that. Just 

bring money.’’ And they do. 

Mr. ANDREWS. Now, I notice on page seven of your testimony 

that you say that a recent study prepared by the Monitor Company 

Group was based on selected acquisition report data, estimates that 

approximately 33 percent of the cost growth from 2000, 2007—I 

think I read this—is attributable to this second category of requirements that you are talking about, right, the derived? 

Mr. PATTERSON. That is correct. Yes, sir. 

Mr. ANDREWS. Now, how is that number reached? Where did that 

33 percent come from? 

Mr. PATTERSON. That comes from the selected acquisition reports 

(SARs). In fact, they lay that out in the SARs and explain in each 

of the SARs where the—— 

Mr. ANDREWS. Well, one of the things that we want to do is to 

make sure that we discriminate between derived changes that are 

beneficial and those that may be superfluous. How would you suggest that we might do that? In other words, I don’t want to leave 

the impression that we are saying, or the report is saying, that 

‘‘Oh, that 33 percent was waste.’’ 

Mr. PATTERSON. Oh, absolutely not. 

Mr. ANDREWS. And how do we draw the line between beneficial, 

cost-effective derived requirements and not-so-beneficial derived requirements? 

Mr. PATTERSON. Well, let me give you a real-life example. In 

1993, I came to the C–17 program. It was a troubled, problemplagued program. 

The two program managers decided that much of the problem 

was that requirements were growing in an airplane. They established a set of rules. The set of rules was very simple: If you have 

a requirement or an engineering change, which is effectively a derived requirement, then it must go to an engineering change board. 

The engineering change board will evaluate the change for its intrinsic merit. But if it doesn’t meet the safety of flight or other 

driven requirements, then it has to have a three-to-one payback in 

savings and not perturbate the schedule. 

Mr. ANDREWS. Interesting. 

Mr. PATTERSON. Real simple. 

Mr. ANDREWS. Now, in terms of the scope of that C–17 program, 

how much were the derived requirements overrun? In other words, 

how much was attributable to derived requirements? 

Mr. PATTERSON. I would only be guessing. I can certainly get 

that for the record. 

[The information referred to can be found in the Appendix on 

page 83.] 

Mr. ANDREWS. And with this method that you just described, in 

your judgment, what percent of the derived requirements were beneficial, and which failed to make that three-to-one cut and didn’t 

happen? 
12 

Mr. PATTERSON. Again, let me give you an example. 

At Edwards Air Force Base, they were testing the airplane. One 

of the requirements was that one airplane needed to start another 

airplane, but it started it with two hoses from the pneumatic system. Somebody said, ‘‘I wonder if it will start with one hose.’’ It 

didn’t. ‘‘Oh, my gosh. Well, now we have to go and figure out why 

that is the case.’’ 

And so, that took a considerable amount of test time and money, 

and as it turned out, they really wanted to have it start with one 

hose. I am sorry, that would have taken—— 

Mr. ANDREWS. Who was the someone who said that? Not the person’s name, but where were they in—— 

Mr. PATTERSON. That was the test community out on the ramp 

at Edwards. 

Mr. ANDREWS. Right. 

I am going to go back for a second round, but Mr. Fitch mentioned this probability of success metrics program. Has there been 

built into that a litmus test for a probability, if you would fall 

below it, things stop? In other words, does that program have in 

it a built-in go or no-go line? 

Mr. FITCH. It does not have a go or no-go line. It is information 

that is updated and provided—— 

Mr. ANDREWS. Do you think it should? 

Mr. FITCH. I think that there needs to be an informed review and 

decisions that look at each of the things that occur that is negative, 

because some of the things that are in the probability of success 

metrics and reporting are positive. 

So I think that it needs to have the program manager, when that 

information is put together on a monthly basis, needs to look at 

that. It needs to be reviewed by someone in his Program Executive 

Office (PEO), okay, and potentially even at higher authority when 

there are negative occurrences. 

Could I add something about—reporting? 

Mr. ANDREWS. Sure, and then we are going to go to Mr. Coffman. 

Sure. 

Mr. FITCH. Yes. 

I just wanted to explain that the derived requirement process is 

actually part of the systems engineering process so that if, for instance, the warfighter says in an aircraft, ‘‘I need a display that 

has color, I need a display that has this amount of resolution,’’ et 

cetera, the first pass, even by the warfighter or the cost estimators, 

may say, ‘‘This amount of processing may be sufficient for that.’’ 

When you get into, ‘‘Well, by the way, you are going to have 

these other software capabilities.’’ You put those together, and they 

start to build on one another, you can find all of a sudden that the 

processor that was planned may be a commercial off-the-shelf item 

with X amount of memory, throughput, et cetera, is insufficient. 

At that point is the point where it would be very useful to have 

the requirements community have a real dialogue with the acquisition community to say do I now take away some of the requirements so I can continue to use that commercial off-the-shelf processor that is less expensive, or are those requirements really important now that we have figured out what they are. So the point is 

the derived requirements is a very important process, and it is ac-
13 

tually the process where you allocate where features will go to 

hardware or software. 

Mr. ANDREWS. Thank you very much. 

Mr. Coffman. 

Mr. COFFMAN. Thank you, Mr. Chairman. 

This is to all of you. It seems like part of the problem is sometimes we are dealing with immature technologies, and I think we 

have had previous testimony to that effect, that sometimes we are 

asking the contractors to develop something. 

Should we bifurcate the process? In other words, that you are 

contracting with one entity to develop the technology, to develop 

the—I don’t know if you would go as far as a prototype, and then 

where you can go into fixed cost production with another—maybe 

that entity would be allowed to bid on it as well. But is it better 

to bifurcate the process? 

Mr. NUSSBAUM. You know, I would say we would do that now. 

If we have open competition, we certainly have a research and development (R&D) contract followed by an acquisition contract 

with—there is certainly no guarantee that the R&D contractor will 

have a follow on. 

So in some sense, we do that, but in another sense, the Department has now mandated technology readiness levels, TRLs of certain levels before a program can get beyond a milestone. They have 

narrowed that cone of uncertainty by saying you have to have a 

TRL level six. They would prefer seven, and GAO prefers seven, 

but you have to pass a TRL level of six before you go into the next 

milestone. 

Mr. FITCH. As a part of the analysis of alternatives, that process 

that immediately follows it, the technologies that would be appropriate for that system, that capability, are assessed for technology 

readiness, when you get to the point of saying who should develop 

it, I prefer the concept of the competitive prototyping, because you 

are going to get proposals from industry. 

They are not going to all have the same strengths. Some are 

going to view that I want to do this more in a hardware function. 

Others may be more software intensive. There are different technologies involved. 

Their proposal, and then watching them deliver on what they 

promise and they say is possible in competitive prototyping, is key 

to then being able to transition and make an award to that contractor who has proven that they did what they say they could do. 

So having a different contractor, if I understood your premise correctly, to develop technology and then award it to somebody differently to develop the system, I think that is not the intent of the 

competitive prototyping system. 

Mr. PATTERSON. I think, too, there is a problem that arises when 

we talk about spiral development. Originally, spiral development, 

evolutionary development, were designed to have a weapons system 

that effectively was fielded and then have block upgrades to improve the weapons system. 

In many cases, that is not how that works, because you do have 

a parallel R&D program that is working on improvements. And before the weapon is actually fielded, you have the attempt to integrate improvements. 
14 

Well, the consequence of that often is that you have a stretchedout program. Costs escalate. And in the end, performance is degraded, and it takes a while for the—much longer than had been 

anticipated for that weapons system to be fielded. So I think we 

need to have very specific and clear understood standards by which 

we will integrate or put in upgrades or technology advances into 

weapons system. It certainly can’t be during engineering, manufacturing, development. 

Mr. COFFMAN. I think it was mentioned, and it has been mentioned repeatedly, that the changes drive a lot of the cost as they 

go forward. Where did the changes emanate from? I mean, are 

some of the changes sort of broader in scope where the affected 

branch who will receive and utilize the weapons system, is it 

changes in evolving doctrine, or is it mostly at a very technical 

level with this engineering requirements? 

Mr. NUSSBAUM. You know, I think that changes come from all 

sorts of places, including changes in labor, overhead rates, schedules, requirements, quantity, absolutely everything, just like building a house. Everything that changes has the contractor saying, 

‘‘Cement has gone up. Brick has gone up. My subs have gone up’’— 

or down, but they are always changing. 

And so there is a great churn. Some of it is part of life, and some 

of it we try to control by saying, ‘‘Tell us what you are building, 

and we will cost that program.’’ But I think it is just intrinsic in 

the cost estimating process. 

Mr. COFFMAN. Is there any way to just have more—I mean, what 

would be some of the methods for having more discipline over the— 

and I take it some of it would—having a change board? 

Mr. NUSSBAUM. There is something called a CARD, a Cost Analysis Requirement Document, which is, to the chagrin of all program 

managers, we say three or six months before we are going to a 

milestone, you tell us what you are building, how many, what the 

technologies are, and we will cost that program. It may be that 

things change after that, but that is the program we are going to 

cost, because otherwise we are chasing our tail, and we don’t know 

until the very last moment what we are costing. 

And in fact, it takes time to do a cost estimate. It is a part of 

the discipline at the Office of the Secretary of Defense (OSD) is a 

Cost Analysis Requirements Document will be prepared six months 

before the milestone, and the lack of availability of that CARD results in a day-to-day slip in the milestone. So that is real discipline, 

but it has its obvious down side, too. 

Mr. PATTERSON. I think, too, that in dealing with the discipline 

and structure, that I must tell you, I submit that it requires rules. 

And the rules simply have to say that, after milestone B, you don’t 

have any more requirements. And, I mean, it is somewhat draconian, but nonetheless, if it—unless of course it is a safety of flight 

or it is an obvious design failure that needs to be corrected. 

But those kinds of things are few and far between, quite frankly. 

And then, this idea that a requirement or a change would give you 

a three to one or four to one payback in savings while not perturbing the schedule is not a bad idea, either. And as technology 

moves forward, that is entirely possible over the course of a program. 
15 

Mr. FITCH. There are times with the requirements—I recall when 

I was developing a black box that was going to go in 17 different 

platforms. I would have this platform coming to me and say, ‘‘It 

would really be easier if you changed it this way. It will make me 

less expensive for my integration, save my costs and schedule.’’ 

What I came to understand is the program manager has to be 

prepared to say no during the development phase. Now, the configuration steering boards we have in place anticipates that there 

may be compelling reasons to change a requirement, moderate a requirement during development, but that is the process of the configuration steering board, is to raise that, if you would, visibility to 

the pressures of changing requirements to have a senior level decision made about the requirement once the milestone has been approved. 

Mr. ANDREWS. Thank you. 

Mr. Cooper. 

Mr. COOPER. Thank you, Mr. Chairman. 

And I would especially like to welcome a fellow Tennessean, 

David Patterson, and congratulate him on starting Defense Business Institute at the great University of Tennessee in Knoxville. 

Thank you not only for your past service, but what you are doing 

right now. 

Can anyone provide me with enough historical perspective to 

help me understand how, during World War II, men like Henry 

Kaiser were able to produce destroyers and—I forget exactly who 

produced airplanes, but it was an amazingly productive period. And 

I don’t know how—was there less bureaucracy then? How was so 

much able to be accomplished so quickly? Anybody know? 

Mr. NUSSBAUM. My reading of history is that there were a lot of 

failures, and then we tend to remember the successes, which were 

terrific. But in fact, there were false starts. 

The P–51 was wonderfully successful, but because it had a bigger 

gas tank so it could keep up with the other aircraft, it gave us 

greater range, but we didn’t build it for that reason. Just we sort 

of lucked into that, if you will. 

I think that things were much simpler. We were able to turn 

technology generations around much faster and, therefore, absorb 

the failures. Today’s systems are very complex. They take a long 

time, and we just won’t accept failure. 

Mr. FITCH. I think that what we remember a lot is those ships 

going down the ways, okay, daily, okay? That was in the production 

phase. 

Even the P–51 that was just mentioned by Mr. Nussbaum, when 

it was first fielded, it didn’t have the engine that ended up being 

what everybody remembers about the performance of that aircraft. 

There were various modifications made after it was fielded. It was 

in the field and found that it was not exactly was needed. 

Mr. PATTERSON. I think, too, you had a tremendous industrial 

base that was able to accommodate to the level of technology that 

it was asked to accommodate to. Today, I am not sure that we 

would be able to do that again in that amount of time. 

Dr. Ron Sega did a study not long ago when he was at DDR&E, 

Director of Defense Research and Engineering, in which he looked 

at the industrial base and what it could do, and found that 62 per-
16 

cent of all of the Ph.D candidates that are enrolled in disciplines 

critical to national security have temporary visas. And that tends 

to put us at a disadvantage, because they generally don’t stay 

around and work for Lockheed or Northrop or Boeing, and Skunk 

Works, particularly, or Phantom Works. 

And if you will look at the Aerospace Industries Association data, 

you will find that in the 1990, 1991, you had 1.3 million touch 

labor workers, highly skilled workers. Today, we have something 

less than 700,000. Those are statistics that should give us pause 

to consider what we are going to do in the future. 

Mr. COOPER. Help me understand. I know that we have had failures in acquisition throughout our history, and Mr. Fitch mentioned a success when he posted the note that was on the vehicle 

that saved lives. 

But on the front page of the Washington Post recently was a 

statement by Secretary of Defense Gates when he attended the return of the remains of some of our troops. He asked how they died, 

and he was told they were in an inadequate vehicle. And he cursed 

because that was symbolic of the fact that we have had difficulty 

fielding relatively simple platforms like Mine Resistant Ambush 

Protected vehicles (MRAPs) or up-armored Humvees. 

And at least for some period of time there was only one manufacturer of up-armored Humvees in America, and yet, at the same 

time, we have automobile companies going bankrupt and looking 

for vehicles to build. So there seems to be a mismatch somehow between pretty basic demands for troops and our ability to field and 

source those even when we have able and willing automobile companies who are looking for ways to keep their plants busy. Why the 

mismatch? 

Mr. PATTERSON. Oh, I think it is a very fundamental problem in 

that you have a Department of Defense that understands fully that 

we are at war with terrorists, and you have a country that doesn’t. 

Mr. COOPER. So Chrysler or General Motors (GM) or Ford didn’t 

want to bid on the up-armored Humvee, or we couldn’t—— 

Mr. PATTERSON. Well, actually, the subsidiaries of those folks 

did, but they really were not in the business of putting out those 

kinds of vehicles. The people who were, International Harvester, 

obviously, and—but the design, fortunately or unfortunately for us, 

that was available were foreign designs of up-armored vehicles that 

were designed to sustain the kinds of things that an MRAP would 

have to sustain. 

Mr. FITCH. And the number of companies that have as a core 

competency in the technologies for armor, okay, is not the same as 

we have for the auto industry. In fact, we are doing a lot of investment in the Department of Defense today to try to find better 

armor, cheaper armor, especially lighter armor, whether that be for 

the vehicles because, when you put the armor on it, it puts a demand on the engine. It puts a demand on the drivetrain. If you 

double the weight of the vehicle, whatever the percentage is, okay, 

it has additional tolls upon the reliability of the vehicle. 

So we don’t have the same industrial base. In other words, the 

auto industry isn’t the industrial base for our Army, either. 
17 

Mr. NUSSBAUM. In a sense, the Humvee is the wrong vehicle to 

up-armor, but it is the only vehicle to up-armor. But it was built 

as a replacement for the Jeep, which didn’t go in harm’s way. 

So it was designed to optimize all the functions that it was going 

to perform, and we missed the fact that it was going to go in 

harm’s way. So it wasn’t suitable, wasn’t optimal, for up-armoring, 

but it is what we have, so we are going to up-armor it. 

Mr. COOPER. I see that my time has expired. Mr. Chairman, look 

forward to another round. 

Mr. ANDREWS. Thank you. 

With the consent of my colleagues, we are going to do another 

round, if that is okay with the panel as well, if it fits your schedule. 

Thank you. 

Mr. Patterson just made a suggestion that we might have a rule 

that says no new requirements after milestone B, and that prompted this question. Again, this piece of data that you cite, about 33 

percent of the program cost growth being attributable to these requirements changes, how many of those requirements changes happened after milestone B? Do you know? 

Mr. PATTERSON. That data, as I recall, is after milestone B. 

Mr. ANDREWS. It is all after milestone B. 

Mr. PATTERSON. That is where the majority of the growth generally takes place. 

Mr. ANDREWS. A related question: in your oral testimony, you 

talk about the fact that, if I read this correctly, the average increase in unit cost of the 28 MDAP programs of less than 5 years 

since development is only 1 percent. 

Mr. PATTERSON. That is correct. That is—— 

Mr. ANDREWS. But it is 55 percent from years 5 to 9. How many 

of that in the 55 percent category, the five to nine, got a waiver 

through milestone B, didn’t meet the requirements to get to milestone B but got waived past it? Do you know? 

Mr. PATTERSON. No, I don’t. I don’t know exactly how many. 

Mr. ANDREWS. But, I mean, would it be accurate to say it is probably most of the 55 percent cost overrun comes from that? 

Mr. PATTERSON. I would say a significant portion of it, sure. 

Mr. ANDREWS. One of the things that is in the conference report 

that we will be looking at in the WASTE TKO bill today is what 

we call intensive care, where if a program is permitted to go forward, even though it didn’t achieve the milestone B criteria, if it 

is waived past it, there is a whole set of intensive requirements 

that are imposed upon that program to try to get it back under control. 

I wanted to come back to, again, this bifurcation that you create 

between derived and customer requirements. Describe for us the 

process that you think ought to be instituted to determine whether 

a derived requirement is added to the package or not. 

Let’s say we are at a point—assuming for a moment that we accept your proposition that there are none after milestone B, which 

I assume you say there should be some exceptions, now as you 

said, for true safety or emergency purposes, but let’s assume we 

are living in a world where, except for those narrow situations, 

there are going to be no changes in requirements after we hit milestone B. 
18 

We are now in pre-milestone B, and an ‘‘Oh, by the way’’ comes 

up, as you said earlier, ‘‘Oh, by the way, this can do this.’’ Who 

should make the decision as to whether that gets added to the 

package, and by what criteria? 

Mr. PATTERSON. Well, I think that the program manager should 

have the initial cut at whether or not they are going to include that 

into the program. But program managers generally are colonels or, 

in very large programs, brigadier generals who have significant 

oversight within the Department. 

I think that if they have a set of rules that say, first of all, if 

it is not a safety of flight or if it is not some sort of safety issue, 

or if it doesn’t give me a return on my investment, then I am going 

to have a thumbs-down, initially. 

Mr. ANDREWS. How do you measure the concept of return? 

Mr. PATTERSON. Well, let’s take, for example, you have—again, 

I will turn to this C–17. 

There were parts of that airplane that were originally designed 

for aluminum lithium, for example. Well, aluminum lithium is 

strong, but it is brittle. 

So an engineering change was made to change that to a different 

alloy. We were breaking the aluminum lithium cargo floor guides 

at a regular pace, so changing that eliminated the problem of having to constantly replace it. That was a savings. 

And those are the kinds of things that I would suggest are threeto-one. And even the suppliers were given the opportunity to do 

that. 

Mr. ANDREWS. Got you. 

It strikes me that this is really the essence of the 20 percent the 

secretary talks about in his 80 percent solution, that what he really 

is aiming to get at here is to give us an adjudicatory mechanism 

that draws the line between the 80 and the 20 when you get to this 

point, and it is your suggestion the program manager should be the 

first person to weigh in on this. 

Who should evaluate his or her recommendation? 

Mr. PATTERSON. Well, then you have an engineering change 

board that would provide a corporate view of it. And if it is a particularly expensive change, then you are going to go to your service 

acquisition executive, or if it is an ACAT 1–D, an acquisition category 1–D, then you have the Defense Department acquisition executive who would have a cut at that. 

Mr. ANDREWS. And my final question, are you confident that we 

can quantify this concept of value sufficiently to hit the three or 

four? In other words, are all of the values that we want to promote—you gave a great example of saving, replacing a piece of a 

plane that is going to go wrong. But do all of the value concepts 

lend themselves to that kind of quantification that would let us 

say, ‘‘Well, this fails to meet three-to-one, so out?’’ Pretty hard? 

Mr. PATTERSON. No, it is much more difficult than that. And that 

is why it takes a lot of research and study and to set up standards 

and conditions whereby you can evaluate these. 

Mr. ANDREWS. Frankly, and I will just conclude with this, it is 

one of the reasons why we are glad we have the three of you and 

the institutions that you represent, because we really do turn to institutions like yours to assemble those data, analyze them and give 
19 

us a factual basis to draw the lines that my questions imply. 

Thank you. 

Mr. Coffman, your turn. 

Mr. NUSSBAUM. Mr. Chairman, is it—— 

Mr. ANDREWS. Sure, Doctor. 

Mr. NUSSBAUM. Is it appropriate for me to make a remark, 

or—— 

Mr. ANDREWS. Sure. It is okay. 

Mr. NUSSBAUM. I think it is not hard to measure the value of 

ideas that replace current capabilities. It is always harder to measure the value of things that represent new capabilities. They don’t 

represent a savings for the operating and support detail that you 

were going to incur. 

But if you are replacing a current capability, then it is pretty 

easy to do an estimate, but it is still an estimate, of what does it 

cost to invest to make this thing happen, and what do you save 

over time in the operating—— 

Mr. ANDREWS. I agree. The much more difficult proposition is 

where you have a new function that could be added by something 

that you discover. How do you measure that, and that requires 

trade-off analysis. It requires opportunity cost analysis. It requires 

a lot of broader inquiries. 

Mr. NUSSBAUM. And if you have that long tail, the question then 

is do you do any discounting on it, the technical issue of net 

present value and at what rates. The Office of Management and 

Budget (OMB) helps us there. 

Mr. ANDREWS. Not to be hyper-technical, but one of our problems 

is then matching up the federal credit scoring and net present 

value rules with the real-world ones, that very often a decision— 

a classic example is in energy. The Department has guidance to hit 

25 percent alternative fuels by 2020. And in order for them to do 

that, they need to do multi-year contracts. But to do multi-year 

contracts, the Congressional Budget Office (CBO) scores that as 

putting the whole net present value into one year, which makes it 

almost impossible to do, which means we don’t do much of it, which 

means we are falling backwards. So marrying the CBO criteria 

with the real-world criteria is a bit of a challenge, too. 

Mr. NUSSBAUM. Yes, sir. And so the devil really is in the details 

on this, and—— 

Mr. ANDREWS. Well, the devil is in the CBO in this case. Now, 

don’t tell Mr. Elmendorf I said that, okay? 

Mr. NUSSBAUM. And one other comment, and that is, when you 

go to the Configuration Control Board or your Service Acquisition 

Executive (SAE), you are proposing to spend investment dollars to 

make this thing happen, and the promise is that you will return 

operation and support (O&S) dollars later on. That is a nice conversation, but it doesn’t accord with the budgeting realities. 

Mr. ANDREWS. And it doesn’t score in our—nor should it. 

Mr. Fitch, do you want to add one thing, then I am going to go 

to my friend from Colorado? 

Mr. FITCH. If I could. I just wanted to say, again, that I think 

it is useful to talk about the operation requirements, the derived 

requirements, but to say again that, when we get the requirement 
20 

from the user, it is stated usually in operational terms, okay, the 

results they wanted to see. 

Mr. ANDREWS. Yes. 

Mr. FITCH. To deal with industry, and for industry to build something, those need to be translated into technical terms. We also use 

the term ‘‘derived requirements’’ for that process. So the derived requirements are really—there are a couple types we are talking 

about here, that which are a part of the normal system that you 

have to do. 

The other one I would just say is your question is how do we 

know we get value. As a program manager, I had an acquisition 

program baseline. That was my contract. The way I viewed it, it 

was my contract with my—decision authority and with the American taxpayer to produce a capability at such a cost with certain 

milestones. 

And I think that is what most of us take and go back to. And 

as we do these questions about derived requirements and everything else, I think we keep in mind that framework. 

Mr. ANDREWS. Yes. I think the panel clearly understands that 

some subset of derived requirements are quite legitimate, necessary and desirable. And I think the three of you have given us 

some interesting tools to discriminate between undesirable derived 

requirements and desirable ones, which is what we are about. 

Mr. Coffman. 

Mr. COFFMAN. Thank you, Mr. Chairman. 

We talked today about having discipline in terms of change requirements. But do we also have to have discipline when programs 

get so far out of line that they become questionable? 

And I am thinking about—and I am working if you could reflect 

on the President’s—I can’t remember the nomenclature of the follow-on helicopter. Littoral Combat Ship (LCS) is in question. I wonder if you all could reflect on when a program gets so far out of 

line. 

Mr. PATTERSON. Well, it would be helpful, I think, in the total 

scheme of things, if we never let them get out of line. But nonetheless, you are exactly right. They do happen. It does happen. 

And one of the things that I think is difficult when you take, for 

example, the VH–71, the President’s helicopter, and that is a perfect example of where you came in with one set of criteria as requirements and, over the course of the time, it changed dramatically. 

I think that we don’t have a set of standards or conditions that 

raise a budget flag that say, ‘‘Wait a second, I am sorry, you are 

red here, and you have been red three reviews in a row, and we 

are canceling your program.’’ And what we do is we put ourselves 

in a position where it is the only game in town. 

You have the President’s helicopter that is arguably old, and we 

don’t have an alternative. You chose a manufacturer and a helicopter, and there is no off-ramp. There is no plan B. And we do this 

rather consistently. 

And I am almost of the opinion that we do it by design. And I 

would offer the Marines’ Expeditionary Fighting Vehicle (EFV), for 

example. We don’t have a way to walk the dog back down the path 
21 

to get an amphibious vehicle that would replace the EFV, so we 

need to make that work. 

Mr. FITCH. I don’t have the particulars on either of the programs 

you asked about. I was the systems engineer for the VH–3D, the 

current—well, it is one of the two current presidential helicopters. 

It is a unique mission. I think there is a desire usually if the White 

House says, or whatever or whoever it is, says, ‘‘I need a capability,’’ that you find a way to do it. 

Going back to it, most of the changes that I saw, and the pressures and the surprises that I got, occur in that first—traditionally, 

the first 12 to 15 months of a program when you go towards a Preliminary Design Review (PDR), because the contractor is off doing 

a lot of things. There isn’t a lot of object deliverables to figure out 

does he get it, he doesn’t get it. 

I think that the competitive prototyping system to focus on doing 

PDRs earlier, to have effective communications with the contractor 

teams and oversight of the contractor teams during that period of 

time will do much more to get to a stable baseline at milestone B, 

which is the actual program initiation. 

Mr. NUSSBAUM. I am going to sound like a professor on one hand. 

On the other hand, because there are some historical examples of 

things which failed, failed, failed and then were terrific. Aegis was 

one, and Tomahawk was another. They just took a long time to 

bring aboard. And for some reason, we had the fortitude to stick 

with them and not say, ‘‘This is an A–12 or a Gama Goat,’’ and get 

rid of it. 

So now we come to the category of LCS. Is it a Tomahawk or is 

it a Gama Goat? We don’t know. But I am taken with my colleague’s remark that, once you get past milestone B, if you have 

three reds in a row, you have got some serious explaining to do, 

with the presumptive answer, ‘‘You are out.’’ It is a rule. 

But the problem is knowing the future, and that is always the 

problem. I don’t know whether LCS is a Tomahawk which is going 

to be absolutely terrific after a long incubation period, and the 

same for V–22. You just don’t know, so we make decisions as people. 

Mr. COFFMAN. Thank you. 

Thank you, Mr. Chairman. 

Mr. ANDREWS. Thank you. 

Well, gentlemen, thank you very much for your expertise. Your 

reward for doing such a good job is we will have to call on you 

again. As the committee goes forward, our intention is to try to 

make legislative proposals for the fiscal year 2011 authorization 

bill that will deal with the area of the problem that the WASTE 

TKO legislation that we are dealing with today does not deal with. 

And I think you have given us some very intriguing ways to 

measure the gap between what we pay and what we get. It is also 

heartening to hear what I have heard this morning, a consensus 

that this panel’s contribution to the WASTE TKO bill was essentially two concepts. 

The first was to add a whole series of reviews and a lot of scrutiny pre-milestone B, with particular emphasis on the requirements 

process. And that did make it into the conference report, and that 

will become the law this week, we think. 
22 

And second, the panel was very interested in much more rigorous 

review, what we call intensive care, of systems that pass milestone 

B by waiver, that have not met the requirements, or that fail 

Nunn-McCurdy standards and get exempted from the penalties 

there and go forward anyway. And I think if you look at the cost 

overruns, a huge majority of them fall into one of those two categories. 

So what we wanted to do was to take the best practices that you 

very ably described this morning and engage them as intensely as 

we can in the systems that, again, never met the criteria to get 

past milestone B but get past it anyway, and those that fail NunnMcCurdy but continue to live on. 

And as Mr. Patterson said a few minutes ago—I think it was Mr. 

Patterson—our ultimate goal is not to have any of those cases in 

the future by unraveling the requirements process and looking at 

it more intensely to intensify that pre-milestone B analysis of what 

is going on. 

And the other point that I would make that is more on our side 

of the table than yours, I think that the principal reason that we 

get these cost overruns is that, once something passes milestone B, 

an enormous political constituency develops around it. Now, there 

are tens of thousands of people deriving their paycheck from a 

project, hundreds or thousands of subcontractors, dozens or hundreds of congressional districts. 

And as Secretary Gates I think can attest, making changes in 

those programs is very politically difficult. If you get to these 

flawed programs earlier when their political constituencies are 

smaller and weaker, the chance to do the right thing is a lot higher. 

So not just for analytical reasons, but we think, given the dynamic of the way these decisions are made in the political world, 

the more precise we are in our measurements and the more focused 

we are in our evaluation, in the requirements phase and a little bit 

beyond that, we think the better job that we will do. 

So I would say to each of the three of you we welcome your continued participation and input. We are certainly going to call upon 

you for your feedback as we go forward in our drafting process. 

And thank you very, very much for your time and attention this 

morning. 

Members will have a period of time by contacting either majority 

or minority staff to supplement the record with written questions, 

and we would invite the witnesses to do the same thing. 

And with that, the hearing is adjourned. 

[Whereupon, at 9:09 a.m., the panel was adjourned.] 
A P P E N D I X 

MAY 19, 2009 

PREPARED STATEMENTS SUBMITTED FOR THE RECORD 

MAY 19, 2009 

(27) 
28 
29 
30 
31 
32 
33 
34 
35 
36 
37 
38 
39 
40 
41 
42 
43 
44 
45 
46 
47 
48 
49 
50 
51 
52 
53 
54 
55 
56 
57 
58 
59 
60 
61 
62 
63 
64 
65 
66 
67 
68 
69 
70 
71 
72 
73 
74 
75 
76 
77 
78 
79 
80 
WITNESS RESPONSES TO QUESTIONS ASKED DURING 

THE HEARING 

MAY 19, 2009 

(83) 

1 Report of the Defense Science Board Task Force on C–17 Review, December 1993, p 3 

2 Ibid. 

3Selected Acquisition Report As of December 1985, p 10 

4Derived requirements, on the other hand, are requirements that the customer has not specified directly as a requirement but that emerge or derive from the design decisions that are 

made. Derived requirements are not capabilities that the customer specifically has identified. 

Particularly troublesome is a subset of derived requirements that fall into the category of engineering changes—those changes that improve on ‘‘good enough’’ and that have a combined effect 

of driving up costs and missing schedule milestones. 5 Aeroflight, Aircraft of the World, Boeing C–17 Globemaster III, retrieved May 27, 2009, from 

http://www.aeroflight.co.uk/types/usa/boeing/c-17/c-17.htm 6 Battershell, A. L., The DoD C–17 versus the Boeing 777, A Comparison of Acquisition and 

Development, National Defense University, Washington, D.C. 1999, p 87–88 

RESPONSE TO QUESTION SUBMITTED BY MR. ANDREWS 

Colonel PATTERSON. To appreciate the context in which early C–17 cost overruns 

occurred, some background and explanation of a methodology for arriving at an estimate of how much of the cost overruns could be attributed to derived requirements 

is in order. In this case engineering changes were the predominant example of what 

I have described as derived requirements and that appear in the public record. 

The first manufactured part for the first C–17 was milled in November of 1987 

and the first squadron was declared to have Initial Operating Capability in January 

1995. The period between these events was one of significant turbulence for the C– 

17 program. The contract for the C–17 program was a fixed-price, incentive fee development contract, because the aircraft was to be designed using off-the-shelf technology and was determined to be ‘‘low’’ risk. 

At the time of contract award, both the Government and MDC [McDonnell Douglas Corporation] envisioned a program based on commercial practices, minimum 

Government involvement and concurrent development/production effort. Consequently, a fixed price incentive contract was used to match the perceived low 

risk of the program.1

As the program became more complex and engineering changes more frequent, 

MDC had no recourse to recoup investments in solving the engineering problems 

and meeting the cost of addressing requirements that were being added during full 

scale engineering and development than to make claims to the Government against 

the contract.2 Though it is not feasible to determine precisely how much of the total 

cost overrun was attributable to derived requirements or engineering changes, it is 

possible to understand the relationship between the unanticipated cascade of engineering changes and increased requirements and cost overruns by virtue of the 

amount of the claim MDC believed justifiable. 

COST OVERRUNS AND ENGINEERING CHANGES 

As early as 1985 engineering changes were impacting the C–17 program costs. 

The Selected Acquisition Report for December 1985 described cost increase of 

$214.5M for engineering changes ‘‘needed for a four-pallet ramp, a combat offload 

rail system, and DoD standard avionics racks.’’ 3 Though these modifications were 

not in 1985 referred to as ‘‘derived requirements,’’ they do meet the definition used 

in the study included with the oral statement provided for the record on May 19, 

2009.4 These were the typical derived requirements not in the original contract, but 

identified by the program management to be necessary during development and presented as engineering changes. These changes and the corresponding cost increase 

occurred in the very early stages of the program, In December of 1985 the Full Scale 

Development contract had been signed.5 The Government became alarmed following 

a 1992 Department of Defense Inspector General Report that identified a cost overrun of $700M on a $6.6B contract ceiling.6 McDonnell Douglas was paying for costs 

overruns on the fixed-price contract and submitted claims for approximately $300M 

for what MDC asserted were changes in scope of its initial contract agreement 

which included ‘‘costs for engineering, development, testing and production of six C– 
84 

7 Adelson, A., Company News; McDonnell May Submit Big C-17 Bill, New York Times, January 14, 1993, retrieved May 27, 2009, from http://www.nytimes.com/1993/01/14/business/company-news-mcdonnell-may-submit-big-c-17-bill.html 8 Report of the Defense Science Board Task Force on C-17 Review, December 1993, p 3 

9 Ibid. p 12 

10 Battershell, p 90 11Government Accountability Office Report, (GAO/NDIAD–94–141), Military Airlift C–17 Settlement Is Not a Good Deal, April, 1994, p 6 

17’s and three test planes.’’ 7 In the end, after significant negotiations with the Department of Defense, McDonnell Douglas agreed to spend $456M in process improvements and the Defense Department agreed to ‘‘provide an additional $438M 

for the program.’’ 

A significant portion of the cost problems centered on a cascade of engineering 

changes that typified the years and immediate several months leading up to the negotiated settlement between MDC and the Department of Defense. The magnitude 

of the engineering changes occurring during this period was described clearly in the 

December 1993 Defense Science Board’s report as it stated, ‘‘In May 1993 alone, 

over 1,000 changes were issued directly impacting production. The backlog of engineering changes at the end of May 1993 reveals 5,800 open work authorizations.’’ 8

Additionally, costly changes to the program were government-imposed. A typical example is the C–17 test program which was increased from an 80 aircraft month test 

program to a ‘‘rebaselined test program of approximately 152 aircraft months’’ at the 

recommendation of the Defense Science Board study, with cost split evenly between 

the Government and MDC.9

SOME EDUCATED CONCLUSIONS 

With the data that is immediately available from the historical record it is reasonable to conclude that the engineering changes were significant and made an impact 

on cost, schedule and performance on the C–17 program. Referring again to 

Battershell, of the $7.3B for the development costs and the first six aircraft in production Lots I and II, MDC invested approximately $1.7B of its own funds.10 This 

amounts to about 26 percent of the original contract amount of $6.6B. MDC proposed a claim against the government of $1.2B in addition to the $438M it had received in the settlement 11 intended to recover what it believed to be legitimate costs 

associated with program changes a significant number of which were engineering 

changes. Though the term ‘‘significant’’ used above is not a precise accounting regarding ‘‘how much’’ of the program was ‘‘attributable’’ to derived requirements, it 

does provide a qualitative data point that is important in evaluating opportunities 

to improve acquisition programs in the future. [See page 11.] 

Æ