U.S. GOVERNMENT PRINTING OFFICE

WASHINGTON : 

For sale by the Superintendent of Documents, U.S. Government Printing Office,

http://bookstore.gpo.gov. For more information, contact the GPO Customer Contact Center,

U.S. Government Printing Office. Phone 202–512–1800, or 866–512–1800 (toll-free). E-mail, gpo@custhelp.com.

i 

51–761 2010

[H.A.S.C. No. 111–36] 

MEASURING VALUE AND EFFICIENCY: 

HOW TO ASSESS THE PERFORMANCE 

OF THE DEFENSE ACQUISITION SYSTEM 

HEARING 

BEFORE THE

PANEL ON DEFENSE ACQUISITION REFORM 

OF THE

COMMITTEE ON ARMED SERVICES 

HOUSE OF REPRESENTATIVES 

ONE HUNDRED ELEVENTH CONGRESS 

FIRST SESSION 

HEARING HELD 

APRIL 1, 2009 
(II) 

PANEL ON DEFENSE ACQUISITION REFORM 

ROBERT ANDREWS, New Jersey, Chairman 

JIM COOPER, Tennessee 

BRAD ELLSWORTH, Indiana 

JOE SESTAK, Pennsylvania 

K. MICHAEL CONAWAY, Texas 

DUNCAN HUNTER, California 

MIKE COFFMAN, Colorado 

ANDREW HUNTER, Professional Staff Member 

JENNESS SIMLER, Professional Staff Member 

ALICIA HALEY, Staff Assistant 
(III) 

C O N T E N T S 

CHRONOLOGICAL LIST OF HEARINGS 

2009 

Page 

HEARING: 

Wednesday, April 1, 2009, Measuring Value and Efficiency: How to Assess 

the Performance of the Defense Acquisition System ........................................ 1 

APPENDIX: 

Wednesday, April 1, 2009 ....................................................................................... 29 

WEDNESDAY, APRIL 1, 2009 

MEASURING VALUE AND EFFICIENCY: HOW TO ASSESS THE 

PERFORMANCE OF THE DEFENSE ACQUISITION SYSTEM 

STATEMENTS PRESENTED BY MEMBERS OF CONGRESS 

Andrews, Hon. Robert, a Representative from New Jersey, Chairman, Panel 

on Defense Acquisition Reform ........................................................................... 1 

Conaway, Hon. K. Michael, a Representative from Texas, Ranking Member, 

Panel on Defense Acquisition Reform ................................................................ 3 

WITNESSES 

Ahern, David G., Director of Portfolio Systems Acquisition, Office of the 

Under Secretary of Defense for Acquisition, Technology and Logistics .......... 6 

Sullivan, Michael J., Director for Acquisition and Sourcing Management, 

U.S. Government Accountability Office .............................................................. 8 

APPENDIX 

PREPARED STATEMENTS: 

Ahern, David G. ................................................................................................ 38 

Andrews, Hon. Robert ...................................................................................... 33 

Conaway, Hon. K. Michael .............................................................................. 34 

Sullivan, Michael J. .......................................................................................... 51 

DOCUMENTS SUBMITTED FOR THE RECORD: 

[There were no Documents submitted.] 

WITNESS RESPONSES TO QUESTIONS ASKED DURING THE HEARING: 

Mr. Andrews ..................................................................................................... 77 

Mr. Conaway ..................................................................................................... 78 

Mr. Sestak ......................................................................................................... 80 

QUESTIONS SUBMITTED BY MEMBERS POST HEARING: 

[There were no Questions submitted post hearing.] 

(1) 

MEASURING VALUE AND EFFICIENCY: HOW TO ASSESS 

THE PERFORMANCE OF THE DEFENSE ACQUISITION 

SYSTEM 

HOUSE OF REPRESENTATIVES, COMMITTEE ON ARMED SERVICES, PANEL ON DEFENSE ACQUISITION REFORM, Washington, DC, Wednesday, April 1, 2009. 

The panel met, pursuant to call, at 7:33 a.m., in room 2212, Rayburn House Office Building, Hon. Robert Andrews (chairman of the 

panel) presiding. 

OPENING STATEMENT OF HON. ROBERT ANDREWS, A REPRESENTATIVE FROM NEW JERSEY, CHAIRMAN, PANEL ON 

DEFENSE ACQUISITION REFORM 

Mr. ANDREWS. Ladies and gentlemen, good morning. The panel 

will come to order. 

I am informed that we expected our ranking member, Mr. Conaway, will be present shortly. But because one of our minority colleagues is present, we are going to begin. 

First of all, I appreciate the indulgence of the witnesses and my 

colleagues and our staff in being here at such an early hour. I hope 

that we did not inconvenience people terribly too much. 

The reasoning behind this is that this is very substantive and 

important material. And we want the members to be able to have 

an uninterrupted block of time to really hear what the witnesses 

say, to engage in what I hope would be constructive dialogue with 

the witnesses, and not be caught up in our normal time pattern 

around here, which is the bell ringing to vote and conflicting with 

other hearings and meetings. 

We really want to give our utmost and most serious attention to 

the material, so that is the reason for this early beginning. And I 

appreciate the indulgence of the members of the panel. 

This morning we are setting out to try to answer the question, 

what is the fair measure, what is the fair way of measuring the 

difference between—the difference, if any—between the cost paid 

by the taxpayers to acquire goods and services in the Department 

of Defense (DOD) and the value we are receiving? Is there a difference between those two concepts? I think there is. And if there 

is, what is a fair way of measuring that difference? 

We have two outstanding witnesses this morning who can speak 

with great authority to that question. One, Mr. Sullivan, in his 

work at the Government Accountability Office (GAO), has frankly 

already given us a compelling measure of the answer to that question with respect to major weapons systems. He is going to talk 

this morning about the most recent work the GAO has done. 
2 

And here is essentially what it says. In 2003, we had 77 major 

weapons systems that were subject to this kind of evaluation. We 

now have 96. 

In 1977, the average cost differential—the increase in programs 

over their original base line—that is a very important concept, the 

original base line versus the adjusted one. But their cost increase 

over their original base line in 2003 was 19 percent in the aggregate. That number has gone the wrong way since 2003. It has now 

gone up to 25 percent in the most recent data that the GAO has 

presented. 

What is interesting about that 25 percent, it happens to be one 

of the triggers in the Nunn-McCurdy legislation that was passed 

quite a few years ago now, it is one of the triggers that triggers 

a very intense level of scrutiny of a major weapons system. So it 

is kind of discouraging to think that, looked at in the aggregate 

across these 96 weapons systems, in the aggregate, they now all 

trigger this kind of more intense assessment. 

Now, these data, as all good work does, really beg a different set 

of questions. Why? 

And in looking at the testimony this morning, and from hearing 

witnesses, I am sure we are going to hear that there is a lot more 

to this than meets the eye. 

The superficial response to this would be to say, ‘‘Well, geez. The 

people who are building these systems and managing them must 

be doing a really terrible job.’’ That is not necessarily so. 

And I think what we are going to hear this morning is, if you 

go deeper into the process, you find two other questions that have 

to be looked at. 

The first is how good or bad of a job we are doing at conceiving 

these systems in the first place. When there is a need identified, 

and there is a weapon system identified to fill that need, are we 

following the right process to determine what should fill the need? 

What in the jargon is called an AOA, which is an independent 

Analysis of Alternatives, how well or poorly are we doing that? Because if we do a good job with the analysis of alternatives, we presumably go down the right path to fill the need and provide the 

capabilities that the service members need. 

So, one of the questions I think lies below the disturbing data 

with which we start this morning is: How effective is that AOA system? 

And then the second goes to the question of how accurate the 

original base line is, or how inaccurate the original base line is. It 

is certainly not fair to blame those who are implementing a weapons systems, if the standards against which they are being measured were unrealistic and flawed in the first place. That may or 

may not be the case, but it is something else, again, I think that 

we are going to hear about. 

There is significant evidence to show that the huge adjustments 

from the original base line to the modified ones may not be a measure of a lack of aptitude by those implementing the systems. It 

may be a measurement of lack of accuracy by those establishing 

the original base lines. 
3 

So, the standard against which we are measured is a very important question. And frankly, it appears like we do not have the tools 

to answer that question particularly well. 

The final point that I want to make is that we also want to go 

beyond this morning’s discussion. This morning’s discussion, by necessity, focuses on major weapons systems, major weapons acquisition. 

But as we heard last week in the briefing, a significant percentage—at least half—of the procurement done under the Department 

of Defense is not major hardware systems, it is services. And we 

want to be sure that we are in a position to take a comprehensive 

look at those issues as they come along, as well. 

I am glad that my friend and copilot has arrived. And if he has 

had a chance to catch his breath, I would be happy to yield to him 

and ask him for any introductory comments. 

Good morning, Mike. 

[The prepared statement of Mr. Andrews can be found in the Appendix on page 33.] 

STATEMENT OF HON. K. MICHAEL CONAWAY, A REPRESENTATIVE FROM TEXAS, RANKING MEMBER, PANEL ON DEFENSE 

ACQUISITION REFORM 

Mr. CONAWAY. All right. Thank you, sir. My apologies. The doors 

down in front were not open, and the police had a long rollcall as 

their excuse. 

Mr. ANDREWS. So much for bipartisanship. Now we lock the 

doors on these guys. [Laughter.] 

But we locked a democrat out, too? Okay. All right. [Laughter.] 

Mr. CONAWAY. Yes, he went the other way. 

Good morning, Mr. Chairman, ladies and gentlemen. 

Mr. ANDREWS. Good morning, Mike. 

Mr. CONAWAY. I think it is appropriate for the panel’s first hearing that we have a senior member of the Defense Acquisition System and GAO’s senior acquisition management professionals sitting 

side by side. It is not that often that members of this committee 

get a chance to talk to the Department of Defense and GAO at the 

same time. 

Thank you, gentlemen, for making this possible and agreeing at 

such an early hour. 

The first question this panel identified as part of the work plan 

was whether there is a method to reasonably measure the ability 

of the defense acquisition system to deliver the goods and services 

needed by the warfighter, and to do so in a timely fashion, and to 

do so at a fair price to the taxpayer. 

Today’s hearing will likely not answer the larger philosophical 

question about how one should measure value in defense acquisition, but is an important first step for us to understand how DOD 

and GAO currently assess performance in one segment of defense 

acquisition, the major weapons systems programs, that were the 

focus of GAO’s assessment released this week. 

These programs receive a great deal of scrutiny by Congress and 

by the media for good reason. GAO’s report reveals that nearly 70 

percent of DOD’s 96 largest weapon programs were over budget 

$296 billion, or 42 percent. This is simply unacceptable. 
4 

Everyone understands why we cannot continue to tolerate these 

cost increases. There is little more to be said on that subject. 

But what we do not hear as much about is that the GAO had 

encouraging words to say about the steps the Pentagon has taken 

to improve acquisition outcomes, including early stage systems engineering, prototyping, measurable yearly plans, increasing accountability and minimizing requirements creep. 

The report states, these changes are consistent with a knowledge-based approach to weapons development that we have recommended in our work. If implemented, these changes can help 

programs to reduce risk with knowledge, thereby increasing the 

chances of developing weapons systems within cost schedule targets, while meeting user needs. 

These are encouraging signs. But to improve outcomes on the 

whole, DOD must ensure that these policy changes are consistently 

implemented and reflected in decisions on individual programs. 

I hope we hear more today about these positive improvements 

that DOD is making and what more needs to be done. Of course, 

we are likely to learn that much of what DOD does to measure performance is already statutorily required. 

I also hope our witnesses feel free to share their views on laws 

and regulations that are not assisting in their efforts to obtain the 

best value and capability for our warfighters. There is a balance to 

be struck between setting high expectations and over-regulating 

the system. 

With that I conclude, and again thank my fellow members. 

And Mr. Chairman, I look forward to the witnesses’ testimony. 

[The prepared statement of Mr. Conaway can be found in the Appendix on page 34.] 

Mr. ANDREWS. Thank you, Michael, very much. 

And just echoing my friend’s opening statement, when he talks 

about the $296 billion in overruns, it is interesting from Mr. Sullivan’s testimony we are going to hear that the weapons systems I 

made reference to have a total projected cost of $1.6 trillion. And 

half of that money is yet to be expended. 

So, to put that in some context, if the 25 percent overrun that 

the GAO now reports had not been the case, the 25 percent of that 

$800 billion is $200 billion. You know, the deficit this year is about 

$1.2 trillion. It is a sixth of the deficit that we are talking about, 

just from these systems expressed in one-year terms. So, it is a lot 

of money and is of great significance. 

Without objection, any opening statements from other panel 

members will be included in the record. 

I want to first go through the biographies of the witnesses. 

And we are going to ask the witnesses—without objection, we 

have entered your written testimony into the record of the panel— 

to summarize your testimony in about five minutes. We are not 

going to rigidly adhere to that rule this morning, but we would like 

you to try to summarize your testimony in about five minutes, and 

then we will proceed to questions from the members. 

David G. Ahern is the Director of Portfolio Systems Acquisition. 

He is responsible for providing portfolio management, technical and 

programmatic evaluation and functional oversight. His office sustains Department of Defense strategic and tactical programs in 
5 

support of the Under Secretary of Defense for Acquisition, Technology and Logistics, and the Deputy Under Secretary of Defense 

for Acquisition and Technology. 

Mr. Ahern was previously professor of program management and 

Director of the Center for Program Management at the Defense Acquisition University (DAU) at Fort Belvoir, Virginia. While at 

DAU, Mr. Ahern also served as an executive course learning team 

mentor and instructor at the Defense System Management College, 

School of Program Management. 

Mr. Ahern has also held business development, program management and business unit positions in the development of tactical information systems with General Dynamics Information Systems 

Company and the Northrop Grumman Electronic Systems sector. 

A native of Connecticut—one of the Final Four participants—Mr. 

Ahern was a career naval officer and is a graduate of the Naval 

Academy in Annapolis. He is also a graduate of the Naval Postgraduate School and Defense Systems Management College. Mr. 

Ahern’s sea duty was as a naval aviator in the RA–5C Vigilante 

during multiple deployments in the Pacific and Atlantic, and as an 

Executive and Commanding Officer of Tactical Electronic Warfare 

Squadron 33. 

Ashore, he was head, Tactical Command and Control Branch on 

the staff of the Chief of Naval Operations, project officer of the 

Navy Space Project, Class 2 Program Manager at the Joint Tactical 

Information Distribution System (JTIDS) Program Office, Program 

Manager, Navy Tactical Data Link Systems, and Deputy of the 

Program Executive, Office Space, Communications and Sensors. 

Mr. Ahern, thank you, and it is great to have you with us this 

morning. 

Mike Sullivan—no stranger to this committee—serves as Director, Acquisition and Sourcing Management at the U.S. Government 

Accountability Office. This group has responsibility for examining 

the effectiveness of agency acquisition and procurement practices 

and meeting their mission performance objectives and requirements. 

In addition to directing reviews of major weapons system acquisitions, Mr. Sullivan has developed and directs a body of work examining how the DOD can apply best commercial practices to the nation’s largest and most technically advanced weapons systems. 

This work has spanned a broad range of issues critical to the success in the delivery of systems, including quality assurance, transition to production, technology inclusion, requirement setting, design and manufacturing, reducing total ownership cost, software 

management and affordability. His team also provides the Congress with early warning on technical and management challenges 

facing these investments. 

Mr. Sullivan has been with the GAO for 23 years. He received 

a bachelor’s degree in political science from Indiana University and 

a master’s degree in public administration from the School of Public and Environmental Affairs in Indiana University. Mr. Sullivan 

is married and has two children. 

Welcome, gentlemen. We are really happy to have you with us. 

And Mr. Ahern, we will start with your testimony. 
6 

STATEMENT OF DAVID G. AHERN, DIRECTOR OF PORTFOLIO 

SYSTEMS ACQUISITION, OFFICE OF THE UNDER SECRETARY 

OF DEFENSE FOR ACQUISITION, TECHNOLOGY AND LOGISTICS 

Mr. AHERN. Thank you, and good morning, Chairman Andrews, 

Ranking Member Conaway, distinguished members of the panel. 

Thank you for the opportunity to appear before you today to discuss how the department values the acquisition programs and assesses the effectiveness of ongoing developments in procurements. 

I will be brief in order to move quickly to the panel’s questions. 

In December of 2008, the department issued a new version of the 

DOD Instruction 5000.2, Operation of the Defense Acquisition System. This instruction established policies and procedures for all of 

the department’s acquisition programs. 

It provides for a structured, disciplined process and incorporates 

many initiatives aimed at improving not only the defense acquisition system as a whole, but also execution of individual programs. 

I would like to take a few minutes to highlight a few of the initiatives. 

First, the department has established a mandatory—a mandatory—materiel development decision (MDD) review that represents 

the formal entry point into the acquisition system. Every program 

will go through that MDD. At the Materiel Development Decision, 

the preliminary concept of operations, a description of the needed 

capability, the operational risk and the basis for determining that 

a non-materiel approach will not sufficiently mitigate the capability 

gap are thoroughly discussed. 

Also discussed is study guidance for the analysis of alternatives, 

which, when completed, will have examined the full spectrum of alternatives, starting with the current capability and moving to an 

entirely new materiel solution, with a goal of balancing capability 

needs with what the department can effectively acquire and afford 

to achieve the best value proposition for our nation. 

A second major change to the DOD 5000 is a revamp technology 

development phase. In the technology development phase the department seeks to reduce technology risk, determine the mature 

technologies to be integrated into a full system and demonstrate 

critical technology elements on prototypes. 

Competitive prototypes, whether at the system or component 

level, reduce technical risk, validate designs and should improve 

cost estimates. They will also enable the evaluation of manufacturing processes and, of course, refine requirements—again, with a 

goal of ensuring the acquisition enterprise pursues the best value 

solution to meet warfighter needs. 

Taken together, the Materiel Development Decision, a materiel 

solution analysis and a technology development phase with competitive prototyping help to define the best value acquisition program to meet the warfighter needs. 

We then execute the selected alternative in the Engineering and 

Manufacturing Development (EMD) phase, using additional tools to 

keep the program on track. 

The Acquisition Program Baseline is the key document for program management. It reflects the approved program being exe-
7 

cuted. It describes the cost estimate, the schedule, performance, 

supportability and other relevant factors for the program. 

That Acquisition Program Baseline is the way that we track the 

progress of the program through the development in EMD and on 

into procurement. An acquisition strategy describes how the program manager plans to employ contract incentives to achieve required cost, schedule and performance outcomes. 

Technical reviews are another tool the department uses to assess 

program status and for decision-making purposes. There are Defense Acquisition Board reviews where the members advise the 

Under Secretary of Defense on critical acquisition decisions. 

Further, there are configuration steering boards held by the service acquisition executives that meet at least annually to review all 

requirements, changes and any significant technical and configuration changes in their programs that have the potential to result in 

cost and schedule impacts. Such changes will generally be rejected, 

deferring them to future blocks or increments. 

Program support reviews are means to inform the Milestone Decision Authority and program office of the status of technical planning and management processes by identifying cost, schedule and 

performance risk and recommendations to mitigate those risks. 

Defense acquisition executive summaries review programs 

monthly and provide an early warning report describing the actual 

program problems, warnings of potential problems and mitigation 

actions. 

In addition to the program level tools described above, the department also employs mechanisms to monitor contract-specific 

performance. Earned value management is mandatory on cost and 

fixed-price incentive contracts above a certain low threshold value. 

It is a well-known tool used by both government and industry program managers to measure contract performance against a contract 

baseline, and it provides an early warning for baseline deviations 

and a means to forecast final cost and schedule on that contract. 

I have only touched on a few elements in the new DOD 5000, 

aimed at ensuring programs are started with a solid foundation, 

are focused on disciplined execution and deliver capability to the 

warfighter within cost and schedule parameters. 

It will take time for us to fully realize the benefit of these policy 

initiatives, and we will continue to look for opportunities to further 

improve the defense acquisition system. 

I look forward to the opportunity to work with the members of 

this panel on this critical task, and I am grateful to the members 

of this committee for your support for the Defense Department. 

Thank you. 

[The prepared statement of Mr. Ahern can be found in the Appendix on page 38.] 

Mr. ANDREWS. Mr. Ahern, thank you for your service and your 

excellent work throughout your career and your contribution this 

morning. Thank you very much. 

Mr. Sullivan, welcome. Welcome back. 
8 

STATEMENT OF MICHAEL J. SULLIVAN, DIRECTOR FOR ACQUISITION AND SOURCING MANAGEMENT, U.S. GOVERNMENT ACCOUNTABILITY OFFICE 

Mr. SULLIVAN. Thank you, Mr. Chairman, Ranking Member Conaway, other members of the committee. I am pleased to be here 

this morning to discuss how best to measure the value DOD is providing to the warfighter. 

Earlier this week, we reported that a cumulative cost growth in DOD’s portfolio of 96 major defense acquisition programs was $296 

billion, and the average delay in delivering promised capabilities to 

the warfighter was 22 months. 

These outcomes mean that other critical national priorities go 

unfunded, and warfighters go without the equipment they need to 

counter ever-changing threats. This should be unacceptable. 

A single metric or set of metrics is not enough to monitor acquisitions and gain efficiencies. However, a cross-cutting set of metrics 

that can measure knowledge, processes and outcomes can be employed to improve acquisition outcomes. 

We think about metrics and their value in the following context. 

First, we use knowledge metrics to determine how well acquisition programs are managing and retiring predictable technology, 

design and manufacturing risks by gaining knowledge and retiring 

risk. These metrics are valuable, because they can predict problems, and they can identify the causes of those problems, so you 

can attack those causes. 

Second, we use cost, schedule and capability metrics that measure a program’s health. These metrics have intrinsic value as simple measurements, but they do little in the way of diagnosing cause 

and effect. This is a way for managers and decision-makers to keep 

an eye on the program. 

Third, there are certain indicators that we look for that are perhaps more important than the metrics, because they determine the 

realism of the acquisition plans from the outset, as the chairman 

was referring to in his opening statement. 

These are a set of prerequisites for any program. And without 

them, we question the value of any metric as you move forward. 

We know that the knowledge and program health metrics we use 

to measure a program’s progress and outcomes are valuable when 

used in realistic, schedule-driven product development environments. They are important indicators to decision-makers. They 

work when they are measuring realistic plans and goals that are 

supported by doable requirements, appropriate cost and schedule 

estimates, and stable funding. 

Our knowledge metrics identify potential problems that could 

lead to cost and schedule shortfalls, and their likely causes. They 

identify technology readiness levels very early, measure design stability by about midway through a development program and track 

whether critical manufacturing processes are in control at the start 

of production. 

They have predictive value. Generally, programs that do not 

measure these risks at the right junctures will encounter a cascade 

of problems beginning with design changes and continuing with 

parts shortages, changes to the manufacturing processes, labor inefficiencies on the manufacturing floor and quality problems that 
9 

will cost money. All of these things delay programs and add to 

their costs. 

Outcome metrics provide useful indicators about the health of acquisition programs, and are valuable tools to improve oversight. 

Last year, the Office of Management and Budget (OMB) tasked 

DOD to work with us to develop a comprehensive set of outcome 

metrics, to track program costs and schedule performance and 

trends. We agreed to track trends and changes across eight different cost and schedule data points, which are in my written 

statement—I will not go through them here—from each of the program’s, from their original baseline, from a five-year-out period and 

from a year ago. We do that on every program. 

These metrics give decision-makers, such as you, some visibility 

into the direction an acquisition may be heading in terms of cost 

and schedule. 

We scale these outcome metrics up from the individual programs 

to a portfolio level, to provide senior department leaders and the 

Congress with a snapshot of the cumulative impact of poor program performance on the relative health of the overall portfolio and 

which way it trends. 

For example, we know that the cost of the portfolio has doubled 

since 2000. There are 19 more major acquisitions in the portfolio. 

Development cost, as the chairman referred to, has grown by 42 

percent. And cost growth has forced the department to reduce 

quantities on many programs. Programs are getting less for their 

money, DOD’s overall buying power is reduced and less funding is 

available for other priorities. 

Metrics by themselves cannot be valuable unless the department 

does a better job ensuring that acquisitions start with realistic 

baseline estimates for cost and schedule. I think Mr. Ahern went 

through a lot of the initiatives in the new policies that we think 

are encouraging in this regard. 

We believe there is a set of prerequisites that must be a part of 

any acquisition strategy before any measurement of an acquisition’s health can be valuable. Otherwise, metrics measured unrealistic estimates will do no good. 

Quickly, these prerequisites include: number one, setting priorities by ensuring joint acquisitions more often and validating only 

candidates that are truly needed and feasible; number two, making 

a knowledge-based, evolutionary business case for the product; 

number three, separating technology development activities from 

product development activities, which we think is really key, because if you get immature technologies onto these product development programs, they cause a lot of problems; number four, limiting 

the time and the requirements for product development to a manageable level; number five, employing systems engineering discipline early to develop realistic cost and schedule estimates before 

product development starts; and committing to fully funding development programs once they are approved. 

Mr. Chairman, I will stop there and conclude my statement. I 

will be happy to answer any questions the committee may have. 

[The prepared statement of Mr. Sullivan can be found in the Appendix on page 51.] 
10 

Mr. ANDREWS. Well, thank you, gentlemen, both, for very edifying and useful testimony. We appreciate the effort. And as I say, 

your full statements have been entered into the record. 

Mr. Sullivan, the title of your testimony is ‘‘Measuring the Value 

of DOD’s Weapon Programs Requires Starting with Realistic Baselines.’’ And I think that that is a point that is made very forcefully 

in your testimony this morning and beyond that. And you highlight 

the importance of the realism of acquisition plans. 

I think it is important to note that, not only is that an important 

measurement tool, but it has everything to do with whether the 

Congress can make decisions based upon realistic assumptions. 

You know, if a system is sold to us on the basis that it is going 

to cost $1, and a realistic projection would be it is going to cost 

$1.25 or $1.40, it has a very different set of dynamics that would 

then take place in our decision-making. 

So, this is really a separation of powers issue, in a sense, that 

for us to make an intelligent, clear-eyed decision about what to do, 

we need better data on which to make that decision. 

What recommendations would you make? And I know you do this 

in your written testimony, but I would like you to elaborate. What 

recommendations would you make to improve the accuracy and 

transparency of the planning process, the standard-setting process 

that goes into these decisions? 

And secondly, to what extent do you think the new 5000.2 guidance moves us in that direction? 

Mr. SULLIVAN. Okay. First of all, I think the 5000—the new 5000 

guidance—does a lot, I think, to move us in that direction. 

Where we would, as auditors, where we would—and people that 

are interested in oversight—we have problems with what is required and what is suggested. And, you know, often times in those 

policies, there is a lot of wiggle room and a lot of encouragement 

to do things, but not a lot of requirements to do things, so we always take issue with that. 

Having said that, the principles in that new policy—and Mr. 

Ahern went through some of that—— 

Mr. ANDREWS. Right. 

Mr. SULLIVAN [continuing]. Address a lot of the things that we 

think have to be in place to improve these weapon system—— 

Mr. ANDREWS. Tell us what some of them are. 

Mr. SULLIVAN. For example, at the beginning, the initial decision—it escapes me for the moment—— 

Mr. AHERN. MDD. 

Mr. SULLIVAN [continuing]. The MDD decision is much more of 

a joint decision today. I think they have tried to bring in—you 

know, there are three big processes. There is a requirement-setting 

process, the funding process and then the acquisition execution 

process. And they have trouble speaking to each other a lot of 

times. 

I think there has been a real push to try to get them together, 

so that there is agreement, a joint agreement on how to move forward with a program. That is one way. 

The Analysis of Alternatives that you spoke of, I think the policy 

is trying to tweak the way that they analyze alternatives and try 

to—and is trying to bring a little more jointness into that, for ex-
11 

ample, so that you do not have—in the past, an Analysis of Alternatives was done by perhaps one of the services. 

And if it was the Army, they were looking at something that had 

wheels. And if it was the Navy, it was something that floated. And 

if it was the Air Force, it flew. And so, there were probably a lot 

of options that did not get full consideration in that. 

As you move to more jointness, I think you get a better idea—— 

Mr. ANDREWS. Do you think it is also characteristic in that AOA 

that there was not adequate consideration, simply revamping an 

existing system as opposed to starting all over again? 

Mr. SULLIVAN. Probably that was the case. And so now, the guidance does state that the number for—you know, when you have a 

new threat or something that you have to counter, the first thing 

you do is look at doctrine. You look at training, you look at a different way, or modifying existing systems. 

You know, that is a tough one to answer. But I think a lot of programs do get started that probably should not. You know, they 

probably could have found an alternative way to do things. 

Mr. ANDREWS. Are you satisfied that there is enough guidance on 

looking to the commercial world as a place to look for solutions to 

these needs that are identified, in the AOA process? 

Mr. SULLIVAN. I am not satisfied that they do enough of that. 

You know, usually when there is a—when they are thinking— 

well, I will give you an example of kind of commercial, off-the-shelf 

programs that the department has tried to start in the past. I 

think it is fair to say that programs like Warfighter Information 

Network–Tactical (WIN–T), which is a communications program, 

was designed to kind of take commercial, off-the-shelf items and 

modify them a bit, and make them available to the warfighter. And 

that did not work very well. 

There has been a lot of talk about the Presidential Helicopter 

and how that has gotten way out of control, because they were 

thinking that would be a commercial, off-the-shelf item. And it 

came in, often times—— 

Mr. ANDREWS. Who has been talking about that? I do not know. 

Mr. CONAWAY. First I have heard of it. 

Mr. SULLIVAN. The requirement-setting process has an impact on 

all that stuff. 

Mr. ANDREWS. Right. 

Mr. SULLIVAN. Once you have a feasible idea to do something 

commercially, and the users kind of start looking at it, all of a sudden requirements start getting piled on that, and it becomes something much different. 

Mr. ANDREWS. Well, thank you, gentlemen. 

I am going to turn to Mr. Conaway for his questions. 

Mr. CONAWAY. Well, thanks, gentlemen. I appreciate you both 

being here. 

You know, it is pretty presumptuous of us to think that we are 

going to ever know as much about acquisitions as the panel, or 

probably a bunch of you all sitting in the room. 

I guess our role, though, is to try to elicit from you the solutions, 

because I have got to believe that with the vast background that 

both of you have, and everybody else across the system has, nobody 

wants us in the position that we are in today. And so, having us 
12 

ask the right questions, I think is the best job that we can do in 

this circumstance. 

On the mandatory development decision, I guess, MDD, is there 

enough rigor there to make sure that the folks making the estimates are not simply—and this is a bit crass—but simply low-balling the estimates in order to get the program started? Because 

once something is started and the initial inertia is overcome, it 

moves, whatever happens. 

And so, is there enough, you know, auditing or somebody checking the guys doing these assessments in that initial phase to say, 

you know, these estimates are not realistic? 

Mr. AHERN. I think you are talking to me, sir, congressman. 

The MDD process, we really do not get too hard into money. But 

as was described in my—into the funding that is going to be required—that really is a transition period between the Joint Capabilities Integration and Development System (JCIDS) process, as 

Mr. Sullivan mentioned, and ‘‘little a.’’ 

Let me elaborate for a minute what I expect in the MDD. And 

I am absolutely a proponent of it. I think it is really critical. 

As Mr. Sullivan said, it is the place where we get both the resource sponsors, the senior people from the Joint Chief of Staff 

(JCS), as well as ourselves, in a room and talk through what it is 

we are trying to do. And that is predicated on a good job over in 

the JCS arena of going through the Concept of Operations 

(CONOPS) for the system, whatever they are talking about, going 

through an Analysis of Alternatives themselves, a smaller Analysis 

of Alternatives. 

Is a materiel solution needed, or can we change training and doctrine and get the job done, and that capability gap in that way? 

If a materiel solution is needed, then they neck it down. And 

they have an Initial Capability Document (ICD) that is actually approved by the Joint Requirements Oversight Council (JROC), of the 

very senior group in the Joint Chiefs of Staff (JCS). 

So, coming into the MDD, the JCS has said, this is a capability 

gap that we need to fill, and it needs a materiel solution. 

In the MDD itself, with that as a starting point, and the JCS 

briefs that ICD what it is they want from a materiel solution, then 

we talk through what the Analysis of Alternatives needs to be. And 

that is where we kick off and start the Analysis of Alternatives. 

We have all had an opportunity to look at the guidance for the 

Analysis of Alternatives. And it is going to start, Chairman Andrews, with, can we modify the current system? That is absolutely 

already there. 

And then it will gradually go up in a ladder step, if you will, all 

the way to pressing technology to a really—got to choose my words 

carefully here, gentlemen—but you can have a number of alternatives, one of which, the last of them could be we have got to push 

technology to get what it is. 

But then, in that Analysis of Alternatives, you have got to look 

at measures of effectiveness. How is it supposed to perform? What 

reliability are you expecting of it? What suitability are you expecting of it? What survivability are you expecting of it? 

And then you need to look at the costs. And so, what we will do 

in the MDD is, we will talk through that acquisition, the Analysis 
13 

of Alternatives plan. Is it rich enough? Is it robust enough? Is it 

considering the alternatives that you are talking about? 

And I have done one of these. I am not just making this up as 

I go along. We have been through one. 

I structured very carefully to be sure that we had that dialogue, 

because we have the senior stakeholders in the room, and we want 

to get the benefit of their advice and counsel as we are going forward. And then we will talk through it. 

Of course, a very significant part of the Analysis of Alternatives 

is, in fact, the cost analysis versus the various alternatives that we 

are talking about. If you are going to modify a system, it should 

be less expensive than pushing technology. And that needs to be 

compared. 

What are you going to get in terms of suitability and effectiveness versus cost? That is part of the outcome. And do you understand the environment that it is going to be operated in? So, we 

talk through the Analysis of Alternatives plan. 

And finally in that MDD, we give the program manager—and 

there is a designated program manager for all of them—an opportunity to tell us what he thinks the whole program will look like. 

Now, that is early to need. 

But there will be funding associated, because, as you all know, 

we do a Program Objective Memorandum (POM) over a number of 

years. And there will be a rudimentary schedule—not prejudicing 

the AOA, because remember, the program has not started yet at 

the MDD. 

Following the MDD, we go to a milestone A, which is, as Mr. Sullivan recognized, the beginning of the technology development 

phase. And following that technology development phase, then we 

will get into the product development phase. That is where we 

snap the chalk line and put down the basis of the program, of the 

cost estimate. 

But we do want to have the program manager talk to us in general terms about how they are looking at this program, how long 

it is going to take to get that capability to the warfighter. 

So, I hope that in a brief—— 

Mr. CONAWAY. Well, I would be—— 

Mr. AHERN [continuing]. Brief, why it is the MDD is important 

to us, and what we do in that MDD and how we go forward with 

it. 

Mr. CONAWAY. A classic example of not answering my question. 

I was more focused on the cost estimate and a rigorous review 

of that, wherever it fell in the system. 

Mr. AHERN. Okay. 

Mr. CONAWAY. I am not sure it was before we decided to go forward we decide how much it is going to cost, or after we are into 

it and we decide it is going to cost. 

Mr. AHERN. Can I answer that? 

Mr. CONAWAY. But—— 

Mr. AHERN. I am sorry. 

Mr. CONAWAY. Yes, but I want to make sure that whoever is responsible for doing the cost estimate—I come from an oil and gas 

background. And when you get a geologist who has put together a 

new prospect, they get in love with it. And they want it sold, they 
14 

want it done. The problem is, they have got to go sell it to a third 

party, who takes a different look at it than the guy putting it together. 

So, I do not want our folks so in love with their prospect—and 

in this sense, it was an acquisition system—that they lose sight 

of—lose objectivity. 

Is there in that, somewhere we snap the chalk line, an independent—not necessarily the GAO—but an independent review of 

this whole system to that point, so we make sure we do not have 

folks who have fallen in love with a system and are no longer objective on the costs and all these other things you mentioned? 

Mr. AHERN. Yes, sir, there is, absolutely. There is an independent cost analysis done on all the major systems by the CAIG, 

the Cost Analysis Improvement Group, on every system. 

So, when we come to snapping the chalk line at what we call 

milestone B, the beginning of EMD, we will have two separate estimates to look at, one from the service and one an independent cost 

estimate—— 

Mr. CONAWAY. Okay. One quick follow up. 

Mr. AHERN [continuing]. Done totally differently. 

Mr. CONAWAY. The 5000.2 regulation from last August, are you 

going back and applying those to all existing systems? Or is that 

just for new systems going forward? 

Mr. AHERN. It will apply to existing systems, depending upon 

where they are, sir. 

One of the programs that is already in production, it probably 

will not see too many changes, based on the new 5000.2. 

Mr. CONAWAY. Even if—— 

Mr. AHERN. But if a program has just started, we will absolutely 

start it in the last six months or so. 

Mr. CONAWAY. But if changes could rein in some cost overruns, 

though, you would do that, wouldn’t you? 

Mr. AHERN. No, sir. We would not force anything in. We have not 

fundamentally changed the sequence of events that a program goes 

through. What we have tried to do is increase the discipline in following that sequence of events. 

As Mr. Sullivan remarked, there is room to tailor—or as he said, 

‘‘wiggle’’—inside the 5000.2. What we are trying to do is reduce the 

wiggle and ensure that we have a disciplined process that we follow for each and every of the acquisitions. 

So, I do not think we would—we are conscious of what you are 

saying, sir, and we would not drive cost into a program to adhere 

to the 5000. 

Mr. CONAWAY. You would drop cost out of it, though? 

Mr. AHERN. I think we will. I am really keen on the idea of doing 

that technology development phase after we do the Analysis of Alternatives, to go to that phase where we will look at competitive 

prototyping—and we have already done a couple of programs along 

those lines—where we have two or three competitors putting together either an entire prototype or key elements of the system, 

and demonstrating it. 

And that, as I said in my remarks this morning, that helps us 

to understand, is the technology available. And frankly, in putting 

together a prototype, it can give us a real leg up on cost estimating. 
15 

So, I think that this new 5000.2, with its emphasis on that 

phase, should, in fact, help us to drive costs out of programs. It will 

tell us about technology maturity, and it will tell us what is doable 

within a period of time. 

Mr. CONAWAY. Thank you, Mr. Chairman. 

Mr. ANDREWS. Thank you, Mr. Conaway. 

Mr. Cooper. 

Mr. COOPER. Thank you, Mr. Chairman. 

Thanks to the witnesses. 

It seems to me that, if there has ever been an alphabet soup of 

bureaucratic quagmire, this is it, with 130-some attempts to reform 

the system since World War II. And I am not sure that anyone can 

even understand whether any of these attempts or reforms worked 

or not. So, I guess our first job is to not make the problem worse. 

When the Secretary of Defense told us at breakfast last week 

that there was something like 50,000 private sector contractors 

whose only job is to oversee other private sector contractors on 

things like contract performance and things like that, I think it 

gives us and the public an idea of how monstrously complex this 

whole process has become. 

It seems to me that—and forgive me, because you gentlemen 

have spent decades studying this, and we are largely new to the 

complexities of this topic—that some of these problems are self-inflicted wounds. 

Mr. Assad talked to us recently and said basically that it is the 

CAIG that comes up with much more realistic cost estimates. But 

sometimes politicians and contractors refuse to listen to those, and 

we prefer the lower numbers, however unrealistic they are. 

So, when it comes to setting and enforcing baselines, well, we 

sometimes enjoy self-delusion, because, guess what, the numbers 

usually, almost always turn out to be higher. 

I wonder about things like the rapid acquisition process, if it is 

essential. And maybe this is just for smaller systems. But if we can 

somehow bypass our own bureaucracy when we need to, why don’t 

we do it more often? 

I also wonder about if there is any good news in here? Is there 

a pony here somewhere? Are there certain systems that are so astonishingly reliable or productive or necessary for the warfighter? 

You know, have any contractors ever been rewarded for those? 

In my area, we fly still a lot of C–130s, most of which were built 

before I was born. And they are still going, and they are still the 

warhorse. They are still reliable. 

So, I would like to leaven the bad news with the good, if there 

is any good news. And I still want to be reminded that we spend 

more on our defense than almost every other nation on Earth combined. 

So, we are the policemen of the world. We are the warfighters 

of the world. You know, the value for the taxpayer is increasingly 

essential as our taxpayers are losing patience with lots of different 

things that we are undertaking. 

So, forgive me for the general sort of take on this. But when gentlemen like you come to us and say there are strategic and tactical 

failures in the procurement of our essential weapons systems, that 
16 

is from top to bottom. That is the military. That is the Secretary 

of Defense (SECDEF). That is us, the White House. 

So, this may be a task far larger than a simple panel can undertake, but we appreciate your guidance. And if, in the time remaining, if either of you would care to reflect, I would appreciate it. 

Mr. SULLIVAN. Well, you know, I would like to go back to, after 

hearing you speak there, the chairman asked the question to me, 

what are the things that should change. And I do not know that 

I answered it real well. 

It is those prerequisites, though, that I talked about earlier. And 

when you look at what you just described, we kind of describe that 

as—I do not want to use the word ‘‘failures’’—but a lack of success, 

if you will, at a strategic level, and then down into the execution 

phases. 

And at a strategic level, if you want something good to happen, 

probably the first thing, the first prerequisite is to have less programs vying for this money that is available. When you have the 

number of systems—and one of the reasons there are 96 systems 

that are in the portfolio now, is because it is a relatively servicecentric kind of a system, so you have a lot of parochialism. 

And I think sometimes where you could have a joint solution, or 

you could have a solution that does not have to become an acquisition program, the services kind of compete with each other to get 

programs started. 

So, at a top level, Office of the Secretary of Defense (OSD), the 

Secretary of Defense can do a better job of getting a handle on that, 

trying to control the service-centric aspects of this and try to reduce the number of programs that are really competing in 

unhealthy ways for that limited dollar that is out there. 

Mr. COOPER. The average tenure of a SECDEF has been 16 

months over the last 40 years. 

Mr. SULLIVAN. And in fact, the average tenure—we looked it 

up—of the Under Secretary of defense for acquisitions is around 16 

to 18 months. So, there is no real good continuity. That is an excellent point. 

I do not know what the answer to that is, unless there is an undersecretary position that can somehow have a time term, or something like that, that would be able to stay in place longer. 

But the turnover really affects a lot of this. You cannot prioritize 

properly. 

Once you cannot kind of keep control of the number of programs 

that are beginning, when you get too many programs into this portfolio, you get unrealistic baselines as a result. And I would say, you 

know, that is the other thing, that you need to have requirements 

that are analyzed a lot more. 

I think the key thing that the new 5000 policy does is the preliminary design review that they are calling for now. They want to 

do that right around the milestone B, which is where they snap the 

chalk line, as Mr. Ahern—that is when they start spending the big 

money. The earlier they do that, the more realistic estimates they 

will get, if they are doing that properly. It is a lot of systems engineering that has to be done early. 
17 

And that kind of sorts out, you know, risky technologies. And unrealistic requirements are going to drive unrealistic cost and schedule estimates. 

If you do not understand the requirements that the user is coming up with, and you do not have discipline there to say we cannot 

do all that right now—you know, there are technologies that you 

have to develop; we can get that in the next generation, but not 

now—these programs will be hard to control. 

Mr. COOPER. Thank you, Mr. Chairman. 

Mr. ANDREWS. Thank the gentleman from Tennessee. 

The gentleman from Colorado, Mr. Coffman, is recognized for 5 

minutes. 

Mr. COFFMAN. Thank you, Mr. Chairman. 

Mr. Sullivan, can you give me an example—I think you mentioned immature technologies—can you give me an example of 

that? 

Mr. SULLIVAN. Well, for example, programs. You know, I hate to 

single out programs, but I will take one that is almost done, I 

guess. The F–22, for example, had technologies that were part of 

the key performance parameters of the aircraft that were very immature when they started milestone B and started spending the 

big acquisition dollars. 

And they fused avionics on that aircraft. They did not understand those technologies well at all. In fact, some people would say 

they were not invented when they opened up the factories to build 

that, some of the propulsion technologies. Some of the stealth technologies they were not real sure about on that one. 

I do not want to single out the F–22. You can pick almost any 

major—even the C–17, which was a relative—you know, it was a 

big cargo aircraft with relatively mundane requirements, not for a 

cargo aircraft, but in general. And they had some technology issues 

on that with some of the material technology they were using that 

caused them a lot of problems. Very immature technologies on that. 

You can name almost any really made—the Future Combat Systems, as, you know, we have looked at that and found that—I do 

not know the exact numbers, but it is maybe 50 or 60 different key 

technologies that are supposed to drive that system. Probably the 

majority of those are too immature to be in product development. 

And the way we look at that, that we have something called 

technology readiness levels that, actually, we recommended that 

the department begin using years ago. And the department has 

started using those. 

And, in fact, Director of Defense Research & Engineering 

(DDR&E) does these technology readiness assessments that Mr. 

Ahern referred to, so now they are doing that on every major weapon system acquisition before it begins. They go in and look at those 

and assign technology readiness levels to those programs. 

Some are still beginning with technologies that are too immature. But I would say they are getting better at that. 

Mr. COFFMAN. And you said words to the effect that there are 

some programs that are not good candidates, that the system does 

not necessarily ferret out programs that are the best candidates. 

Mr. SULLIVAN. You would like examples of those? 

Mr. COFFMAN. Could you give one example of that? 
18 

Mr. SULLIVAN. Well, that was a kind of a general—more of a general statement. 

I think that it would be—if you want to consider a good candidate to start product development, what we think the best practice for that would be is a candidate where you understand the requirements, you have looked at the requirements and determined 

that there are things you can do and things you cannot do, and you 

have gotten ridden of some of the requirements that are not doable. 

You have looked at your funding, and this program fits into a 

funding profile that the department can count on, and you have 

looked at technologies and things like that. And all of these things 

fit. 

I would say that hardly any of the major weapon system acquisitions are good candidates, according to that criteria. Lately there 

have been a couple, I think, that we are looking at now that we 

think the department is doing good with. 

I do not know. Small diameter bomb is an example of that, where 

they have really looked at those requirements. 

Mr. COFFMAN. Okay. Let me ask a question of both of you. 

In 1992, I was with the Marine Corps, and came up here on Capitol Hill and had a meeting with the officer in charge of the Marine 

Corps liaison program. He was a brigadier general whose name escapes me right now, but he said something that I have never forgotten. And I would like if you would both respond to this. 

He said that, kind of almost—I will paraphrase it. We get weapons systems that we neither want nor need, but are based on a congressional—but are manufactured in a congressional district whose 

congressman sees it as a jobs program for their district. 

Can you respond to that? 

And that was his statement in 1992. First of all, did you think 

that that statement was reflective of the environment in 1992? And 

is reflective of the environment today? 

Mr. AHERN. No, sir. 

Mr. COFFMAN. Okay. 

Mr. AHERN. I mean, let me say—I mean, as you read my biography, I was program manager and Program Executive Office 

(PEO) at that period of time. Obviously, I am aware that there 

are—congressmen have industry in their district. 

But I have never felt—never, ever felt—that we were pressured, 

encouraged or in any way directed to do anything that entered into 

that. I honestly think that we made mistakes, that each time we 

started down the road toward a product development, we have 

done it as well as we can. 

We appreciate your interest and support, but I have never felt 

that we have been pressured into doing something for that kind of 

a reason. 

Mr. COFFMAN. Okay. 

Mr. ANDREWS. If the gentleman would yield just a moment, I 

would like to piggyback on this question. 

Mr. Ahern, could you say the same thing about a situation where 

there has been a cost overrun in an existing program and there has 

been an effort to limit it or eliminate it, there has not been congressional pressure to resist that elimination or cutback? 
19 

Mr. AHERN. Well, as you know from my portfolio, I have been 

through a number of cost growths, because it is a difficult job. And 

again, I can say, yes, sir, never has happened. Never has happened 

to me personally, and I have had some fairly senior jobs in the Pentagon and in the Navy and in the Air Force. 

No, sir. It has absolutely never happened to me. And to my 

knowledge, the work that we have done in the Pentagon, the reviews, the Nunn-McCurdy reviews, I have not heard that at all, sir. 

No, sir. 

Mr. ANDREWS. Mr. Coffman, thank you. Mr. Ahern, thank you. 

Mr. COFFMAN. Mr. Chairman, I am wondering if Mr. Sullivan 

might be able to respond. 

Mr. SULLIVAN. Yes, I think the first part of your question I can 

deal with, and that is a warfighter saying that they did not get 

what they need. The reasons are many-fold. 

But the one that I look at most is, a lot of times a warfighter 

does not get their urgent needs, because we are busy working on 

very highly complex, single-step-to-big-bang capability systems that 

are draining a lot of funds and time and energy, like the F–22 or 

the Joint Strike Fighter or the Future Combat System. 

Or you can go across the board. A lot of these big, you know, 

kind of unachievable requirements programs take so much time 

and so much money, that I think a lot of times the warfighters— 

there are bill-payers out there for these, right? We all understand 

that. 

So, there are smaller programs that have to pay the price when 

an F–22 program begins with an unrealistic cost and schedule estimate and every year needs to be plused up in order to get the development through. And so, people pay for that. 

And I think that is something that needs to be looked at, as I 

think the warfighter suffers that way, because these big programs 

are taking the dollars. 

The other thing I would say is, if you look at the Mine Resistant 

Ambush Protected (MRAP) vehicle acquisition, the MRAP acquisition actually was pretty good, you know. Once they focused on the 

fact that they needed to counter this threat, and it was an urgent 

threat, they moved very quickly to get MRAPs to the field. 

Before that, I think that this acquisition process hindered the 

ability to do that, because there were other acquisition programs— 

you know, the Army, for example, had other programs that it needed to fund, and things like that. I think it took a long time for people to accept the fact that the MRAP had—we had to put money 

into this, we had to go with requirements that were doable right 

now, and we had to meet an urgent need. 

Once those decisions were made, that went pretty well. 

Mr. ANDREWS. If I may, just also, I think one of the reasons it 

received such a high level of congressional scrutiny on an almost 

daily basis, Chairman Hunter at the time, Mr. Skelton watched the 

process like a hawk, as did several other members. And I think 

there is some subjective value—some suggestive value in that. 

Mr. Ellsworth is recognized. 

Mr. ELLSWORTH. Thank you, Mr. Chairman. And I think my comments to be more general. 
20 

First, in fairness, I want to say that we brought up U. Conn.’s 

basketball program. I thought I had better bring up Mr. Sullivan’s 

alma mater—— 

Mr. SULLIVAN. Maybe not right now. [Laughter.] 

Mr. ELLSWORTH. They have been known to throw a few balls 

through the hoop over the years, so hopefully they will get back to 

that. 

Mr. SULLIVAN. Thank you, sir. 

Mr. ELLSWORTH. When I first got put on this panel and you distributed some paperwork, one of the charts that jumped off the 

screen was the actual chart that showed the acquisition process. It 

reminded me of a Dr. Seuss configuration. And that was funny in 

the Dr. Seuss books, but in military acquisitions, I did not find 

much humor in that, and thought, you know, how in the world can 

anybody navigate their way? 

And as I still try to learn that system, there are just things that 

jump off the page. You would almost have to think it was planned 

confusion. And I hope that is not the case. 

But it would seem to breed the shirking of responsibility, how we 

go through, and how many points we could push off the responsibility on these decisions. 

And another thing comes to mind, and I think that Mr. Coffman 

said it well. And I heard the president—I think Mr. Cooper and I 

were at a meeting at the White House a few weeks ago where the 

president said, ‘‘I have to make decisions on national defense really 

based on national defense, not on a congressman’s district and 

what they make there.’’ And so, that gives me great hope that 

those decisions will be made on that. 

One of the things we also have to be—and maybe you could talk 

to this—is fairness to the contractor. You know, when we are building the hull of a ship and signing that contract, and then it is designed as we go, as we build up from the basic platform up, not 

unlike the Congressional Visitors Center, where it started off at 

$300 million and kept changing and changing and changing, and 

then went to $600 million. I think it is the same with the helicopter, if we keep adding things as we go. 

We have also got to be able to give these—nobody is going to go 

out and just build a ship and hope that the United States government buys that ship, or get ready and build a factory that is ready 

to go, whether it is two submarines a year, one submarine a year 

and say, hey, I hope they up that someday. 

We have to give the contractor some kind of vision of what we 

are looking at. I know that is tough in two-year cycles and six-year 

cycles and every four years. But I think we have to give them 

something to look at when they are putting employees together and 

buying equipment, and how we address that along with rapid acquisition. 

Where is the middle ground there between the normal acquisition process and rapid? Is there something in between there that 

works better, simplifying it, making the steps down? 

So I guess, if I had direct questions, it would be, you know, 

maybe we can look at the percentage of programs pulled just because of cost. The warfighter needed it, but it was just too expensive. 
21 

Or the percent pulled because we thought we needed it. It sounded good. It would be nice to have. But then we realize halfway in, 

maybe we do not. It was a great wish, but we did not really want 

it. 

And I am just talking in general. I know that this is our first 

meeting, Mr. Chairman, and I appreciate it, as we try to get our 

arms around it. But maybe you could address the fairness to the 

contractor and how we look to improve that system also. 

Mr. SULLIVAN. The way I look at that is—that is a really good 

question, because obviously, we are in a—the defense industry is 

an industry that does not necessarily build things in volume, because—you were kind of iterating to that. 

There are not real commercial markets for these things. These 

are going to be—you know, you build a couple of submarines and 

you are done. You build a number of fighter aircraft, and there is 

no more market for that. There is no re-sale. There is nothing like 

that. 

So, you are starting out with an industry that—this is very capital-intensive, very labor-intensive and low volume. And in addition 

to that, it is cutting edge. 

And I do not think any of us will ever get to a time where it is 

not risky to build defense acquisitions, these big weapon systems, 

because we want them. We want the best in the world. The risk 

could be a lot less than that. 

So, what the department usually does with these big contractors, 

the way they deal with that, I think, is with these cost-plus contracts that are a necessary part of doing business, because of what 

you referred to. 

No company is going to, on their own nickel, begin to invest in 

the facility and the tooling that they need to take care of the government, because that is the only game in town. So, if they do not 

get it, they have wasted their money. 

So, the government pays for all of that. So, that is kind of the 

fairness to the contractor, I guess, is we kind of take on their risk. 

On the other hand, we look at the funding on an annual basis, 

and never really give contractors on some programs the stability 

that they need, the security they need in receiving those funds on 

a year-to-year basis to be able to do their job better. 

And when you combine that with the requirements that we have 

already discussed, when you have a contractor working on a weapon system that has capabilities that they really are not able to 

build right away, and they have this cost-plus contract, and they 

have the President and the Congress every year looking at their 

budget, it does create a really unstable environment for them to operate. So, those are some things. 

Requirements are unachievable. Cost-plus contracting comes up 

a lot. I think a lot of people talk about getting rid of it. I do not 

think necessarily reasonable people think that is a good idea. But 

if you can get to a position where you build—where you have requirements to build things that are doable, the cost-plus contracts 

make a lot more sense. 

Mr. ELLSWORTH. Thank you both very much. 

Thank you, Mr. Chairman. 

Mr. ANDREWS. Thank you, Mr. Ellsworth. 
22 

Mr. Hunter is recognized for 5 minutes. 

Mr. HUNTER. Thank you, Mr. Chairman. 

And in the interest of time here, we will start off with my first 

question. In 2003 to 2008, we have had kind of a permissive environment. I think you would agree to that. 

Who was responsible, if you had to lay it out in terms of who was 

most responsible to least responsible for the permissive environment, that maybe the services and Congress and contractors took 

advantage of and are now getting reined in? Who is responsible? 

When it comes to DOD, Congress, contractors, who would you lay 

the blame on the most from 2003 to now? 

Mr. AHERN. I started here in 2006. And, of course, I was an instructor before that. 

I would not go to blame. I would say that the discipline in looking at the system, at the elements of a development going forward, 

were not as rigorous then as they are now. But I am not sure that 

I could say that the blame is there. The system was always there. 

The system has always been there. 

And there have been some good programs started, and they are 

continuing to execute through that period of time. I think of the P– 

8, the Navy’s replacement for the P–3, which I think was started 

in that period of time. 

So, I am unable to ascribe blame. I think a number of the programs that are in the portfolio Mr. Sullivan mentions are, in fact, 

executing over the last 5 years quite well. I believe that that is in 

his report. 

So, I am not sure that there was a permissive environment in 

that period of time. I think that what we have done recently, or 

in the new 5000, is to add—and in other things that we have 

done—to add additional expectations. 

But I would not agree that there was a permissive environment 

during that period of time. And I think, as I said, there are a number of programs that have not shown cost growth, that were started 

in that period of time. 

Mr. HUNTER. Almost every service acquisition officer who I have 

talked to in the last—I have only been here for about 2.5 months. 

But everyone that I have talked to said there was indeed a permissive environment, and that contractors took advantage of it—not 

necessarily at the big level programs, but down at the lower levels, 

middle levels, and that that is being reined in now. 

And industry is suffering to some extent. The services are having 

to be more responsible, and DOD is having to be more responsible, 

too. 

Mr. SULLIVAN. One of the comments I would make is that, you 

know, one of the things that started around, in the 2000 timeframe 

is when we started having these acquisition programs like Missile 

Defense Agency (MDA), Future Combat Systems, Joint Strike 

Fighter. Some of these very, very complex what they call systems 

of systems, as opposed to just building a weapon system. You had 

the Future Combat System, which is a—ostensibly, there are 19 

programs kind of working within that that somebody is trying to 

coordinate. 

Joint Strike Fighter is three separate fighter aircraft variants 

that Lockheed Martin is trying to do at one time. There are a lot 
23 

of programs out there like that. So, I do not think the government 

particularly did that very well. 

The MDA is an acquisition program. It is an agency. And it 

has—I do not know the number now, but it is in the teens of highly 

complex programs that are supposed to work together. 

So, I think one of the things that happened is that it got much 

more complex, so complex that we cannot understand it too well. 

And I think the permissiveness, a lot of that might have come 

from what happened. You know, there was an attempt at acquisition reform in the mid 1990s. 

Mr. HUNTER. Well, as complexity goes up, costs go up and everything else goes up. 

Mr. SULLIVAN. I think absolutely, yes. 

Mr. HUNTER. Let me jump in here, because we have to run. 

How do you—and this kind of goes to those mid-level warfighter 

programs that guys really need, programs like Joint Improvised 

Explosive Defeat Organization (JIEDDO), Task Force (Odin) and 

things that we funded, because of immediate current threats that 

the warfighter has begged for, but that Congress almost has to 

force on the different services. And these programs start off agile 

and responsive, and they end up slow, unresponsive, bureaucratic. 

And these are all programs where the ongoing operation is just 

as important as the initial acquisition of them. 

And it seems like everything works well when it first starts off. 

It is lean and mean, and then it kind of gets out of control, once 

they get their billions and they hire 300 bureaucrats. Everything 

slows down and they become unresponsive, and they do not necessarily do what they were intended to do in the first place. 

But that is more of an ongoing thing, but it still has to do with 

the initial acquisition and programs not doing what they are supposed to do after the initial acquisition. 

How do you fix that? 

Mr. AHERN. Well, there are a couple of things that occur to me 

immediately, congressman. One is the MDA that Mr. Sullivan mentioned a couple of minutes ago. That was initiated, I think, in 2002 

timeframe, and is one that had a very near-term expectation that 

they would develop a capability by—develop and field a capability—by 2004. 

And then, as time went—and then continue on. And I think the 

MDA has a number of elements in development moving toward a 

block capability that will enable them to expand upon that initial 

capability. 

And what we have done with the MDA is to ensure that they 

continue to have that Research, Development, Test & Evaluation 

(RDT&E), or that technology development focus, and move toward 

transitioning the capabilities that they are developing to the services to operate as rapidly as we can. It requires focus in that area. 

Another one that comes to my mind that we are working on now 

is the Intelligence Surveillance and Reconnaissance (ISR) task 

force that Secretary Gates started. And that is one that I support 

from an acquisitions standpoint. And it is absolutely to push ISR 

resources into the operating areas as quickly as we can. 
24 

I expect it to operate for a period of time. I expect to continue 

to push on it. And the same kind of thing is happening with the 

MRAP. 

And it really is, as—it is up to the people, the discipline that I 

mentioned earlier on, to have, continue to have that interest in getting the job done to support the warfighter. And I cannot say it any 

better than that. 

We have to do our jobs responsibly to ensure that we do maintain the focus on the urgent operational needs, the support for the 

warfighters, as I think we are doing in the ISR task force, as we 

did with the MRAP and continue to support the MRAP, and as we 

are doing with the MDA. Those are three examples that occur to 

me. 

Mr. ANDREWS. Thank you, Mr. Hunter. 

The chair recognizes Mr. Sestak for 5 minutes. 

Mr. SESTAK. Thanks, Mr. Chairman. 

I honestly believe that the requirement side, particularly since 

General Cartwright used to have J8, really has come a long way 

from the Cost Benefit Analysis (CBAs), and the modeling he has 

done down there. So, I want to focus more on the acquisition side. 

But could I ask you, sir, if you could get back to us? My limited 

understanding is that under Instruction 5000, all the modeling that 

is used on the requirement side has to go through a validation, 

verification and analysis by an independent somebody to make sure 

the models used are well. My understanding is, probably only 5 

percent have. 

Could you get back to us on that—— 

[The information referred to can be found in the Appendix on 

page 80.] 

Mr. AHERN. Yes, sir. 

Mr. SESTAK [continuing]. Because we are using models in the requirement side that have never been validated. How good are they? 

So, I think that is what I would really like to know. My understanding is, even though there is a requirement to in 5000, hardly 

any of them have. And I think that is something that does not bode 

for credibility on the modeling being used. 

The other side of the acquisition side, I was quite taken by your 

comments about the Earned Value Management (EVM) program, 

where we are supposed to forecast final cost. Also, sir, with you, 

I was taken with how you want to try to establish prerequisite indicators to try to get realistic costs, to kind of break this tyranny 

of optimism we have. 

The other day, when the Littoral Combat Ship (LCS) came forward, initially it was supposed to be $250 million. Now it is $460 

million. The internal DOD figures only have a 50 percent confidence factor that that is going to be the actual cost. 

I guess my point is, we have Nunn-McCurdy. Thirty programs 

the last three years have come over to us, breaking, breeching 

Nunn-McCurdy. Nice monitoring system, but no enforcement, no 

teeth on it. 

You mentioned the CAIG, sir. CAIG estimates are not mandatory, yet they turn out to be, by and large, much more right than 

the services. Should we, A, make the CAIG estimates mandatory? 
25 

Number two, shouldn’t the Congress have exposure to those confidence factors that you have internally to the services, for example? The aircraft carrier that is going to be built here in a few 

years is going to cost, according to some estimates, $13.5 billion. 

But the internal confidence factor of costing on that is 37 percent. 

We are building a lot of Virginia class submarines. But the internal confidence factors on the costing for the Virginia class submarine—two of them being built here in the next years—is less 

than 50 percent. 

Should Congress here have access to that data, so before we commit to something like an LCS, we know at that time maybe that 

20 percent confidence factor before we get too far down the road? 

Sir. 

Mr. AHERN. Sir, taking it in terms of the CAIG being mandatory, 

I think we need to maintain that balance. The experience that I 

have is, we tend to use the CAIG estimate for the estimate for the 

program. But we need to have that dialogue between the CAIG and 

the service estimate, the program office estimate, because they are 

different. They are done differently. 

And I think that that dialogue is necessary. And if we made the 

CAIG mandatory, I think that the service cost estimating, which 

enables that dialogue, would—— 

Mr. SESTAK. Do you think we should have access to both cost—— 

Mr. AHERN. Well, I think you should have access to the cost estimates. 

Mr. SESTAK. I mean, if they are not an—— 

Mr. AHERN. I am not sure—— 

Mr. SESTAK [continuing]. Independent one, like you said, 

shouldn’t Congress know before we dole the national treasure out, 

what CAIG’s estimate is versus the service’s, or at least the confidence factor that is coming forward? 

Mr. AHERN. I think we do send over the—I mean, the results of 

that estimating is done in the Acquisition Program Baseline (APB) 

that is reported in the Selected Acquisition Report (SAR) reports 

for the programs. 

So, I think that the results of that work is absolutely reported 

to you in the SARs that come over on all the major programs when 

they initiate them at a program milestone. I really do think it is 

there. 

Mr. SESTAK. All right. I probably missed it. 

May I ask another question? I think my time—oh, I have time 

for one more. 

My question is on the more jointness. I was always quite taken 

with what Representative Skelton and many did with the JCS, you 

know, with having—back in the Goldwater-Nichols day, when the 

chairman walked in after that was passed, everybody else stood up. 

Before that, the JCS would not stand up when he walked in the 

room, because he was one of equals. 

To some degree, do you think, as we have a wonderful system, 

including having OSD involved in the JCIDS process now, which 

it was not before—I mean, a few years ago they were brought in— 

do you think that we need to structure the JROCs, but not so much 

as what we did to the JCS, so it is not everybody is equal there? 
26 

You may get the least common denominator. It is my program 

versus your program. 

But do we need a Goldwater II, in a sense, to say there should 

be one final requirements guy before it goes any further, rather 

than least common denominator. Any comments? 

Mr. SULLIVAN. Well, my comment on that is that, you know, 

Goldwater-Nichols was to make jointness. And it did it on the operations side, I think. You know, if we see the way—— 

Mr. SESTAK. But not the procurement side. 

Mr. SULLIVAN. Not the procurement side. So, there, I think you 

raise a really excellent point. 

These programs have to be acquired jointly, and they are not 

right now. There is still too much service-centricity, even in the 

JCIDS process, which, when you read the JCIDS policy, it is really 

pretty good policy. 

Mr. SESTAK. It is. 

Mr. SULLIVAN. It establishes functional capability boards and a 

lot of joint matrix processes that are supposed to look at requirements and weigh in in a purple-suited way, if you will. 

Those things are not in effect right now. They have not staffed 

them properly. 

I think you still—I do not think that the policy has been implemented well. 

Mr. SESTAK. I am out of time. But my only thing is, when it finally gets to the JROC, it is wonderful work up till there. But that 

final decision, I just do not know if you need to change it like the 

JCS—— 

Mr. SULLIVAN. Yes, if I could just comment briefly on one other 

thing you were talking about, the CAIG estimates. 

We have done a lot of work in that area. And we found that the 

CAIG usually has a little bit better accurate estimate, but still, far 

from what the outcomes usually tend to be. And the department 

does not always accept those estimates. 

And I think that an independent CAIG would be something that 

might be helpful to the Congress, something similar to how they 

do Director of Operational Test & Evaluation (DOT&E). They established an independent director. I think that is something that 

might be helpful. 

Mr. AHERN. I would like to go back and comment on Mr. Sullivan’s last comment. I think that the more we emphasize that need 

to do the work prior to beginning a program, the product development, we will have a far better informed, independent as a well as 

a service estimate. 

So, I do not think the answer, sir, is structurally to set up an 

independent CAIG or to do something along those lines. I think it 

is to ensure that we have more information on the products before 

we get into the product development. 

The competitive prototyping, the insistence upon technology maturity are the kinds of information that we need to have. Otherwise 

we are using old information, parametrics. Well, we did it this way 

a couple of years ago, so we will use that kind of way. Or circuit 

boards cost $1.98 in 2005, so assume they cost $1, you know, $2.05. 

We need to have better information than that. 
27 

That is why I think, yes, the CAIG estimate, the service estimates are critically important. But to improve them, what we need 

to do is improve the information that goes into them. 

Mr. SULLIVAN. But in order to improve that information, you 

might need an independent assessor overseeing all that—— 

Mr. SESTAK. You could be joined and do both. 

Mr. SULLIVAN. This could be a chicken and egg thing. I do not 

know. 

Mr. ANDREWS. I would like to thank Mr. Sestak, particularly for 

the observation about the dichotomy between the jointness in operations—— 

Mr. SULLIVAN. Yes. 

Mr. ANDREWS [continuing]. But the lack of jointness in procurement. I think it is a very critical point—— 

Mr. SULLIVAN. Yes, very—— 

Mr. ANDREWS [continuing]. That goes to a lot of what Mr. Ahern 

and Mr. Sullivan said. Thank you. 

I am going to ask if Mr. Conaway has any concluding remarks. 

Let me also mention that, if any member of the panel would like 

further analysis or questions, please submit them in writing. I am 

sure the witnesses would comply with a written answer. 

Mr. Conaway. 

Mr. CONAWAY. Well, witnesses, thank you very much for coming 

this morning at an unusual hour for a hearing like this. I appreciate that. 

I would like to follow up with you with respect to these CAIG estimates and the ones that are actually used, and what requirement 

there is for reconciling the differences between the two on the front 

end. 

Mr. CONAWAY. And then, Mr. Sullivan, you may have some historical data about tracking reasonable estimates, CAIG estimates 

and reality. 

Mr. SULLIVAN. Yes, I do. 

[The information referred to can be found in the Appendix beginning on page 78.] 

Mr. CONAWAY. It might be helpful for us to look at it. 

But in closing, it is actually the panel’s responsibility to find legislative issues that need to be addressed on a go-forward basis. So, 

if there is legislation that you see is needed to help what we are 

trying to do and trying to accomplish, which I think all of us have 

the exact same goal, please point that out to us, as well, because 

that is really the goal of what we are trying to get done. 

Chairman, thank you. 

Mr. ANDREWS. Thank you. 

I would like to thank the witnesses for their outstanding preparation. We are going to call on you as the process goes forward, I 

am sure. 

There are two items I would ask you to supplement the record 

with. 

Mr. Ahern, I am interested in your views on how we might eliminate the wiggle room that has been identified in the 5000 guidance. 

I know you are working very avidly on that. We would be interested in your direction to us on how that process can be sharpened 
28 

and improved. I thought your comments this morning were very 

edifying. 

[The information referred to can be found in the Appendix on 

page 77.] 

Mr. Sullivan, I would like your views on the question of the 

SARs. The selective acquisition reports that come over are an excellent tool for the Congress to evaluate problems. As I understand 

it—and correct me if I am wrong—I think all the SARs come postmilestone B. Am I right about that? 

Mr. SULLIVAN. Yes. 

Mr. ANDREWS. Yes. I am interested in whether there is a similar 

analytical tool that could be created by legislation pre-milestone B, 

so that we could get an early warning signal that there is something wrong in the design phase. 

Now, I understand it is a whole different set of questions, but I 

think that a SAR-type tool pre-milestone B would be very useful 

for us. I would be interested in your thoughts. 

And you, as well, Mr. Ahern. 

[The information referred to can be found in the Appendix beginning on page 77.] 

Mr. AHERN. All right. 

Mr. ANDREWS. The committee is going to—the panel is going to 

proceed after we return from the recess with another hearing. We 

will be consulting with the minority as to what the topic ought to 

be with respect to that. 

I am confident that we are going to broaden the issue of defining 

what I began this morning talking about, the delta, the gap between what we are paying for and what we are getting, beyond the 

issue of the major weapons systems, as well. I think we have had 

an excellent discussion of that subject, but we want to go beyond 

that to the budget, the huge—the majority of the procurement 

budget that is not major weapons systems. 

You have given us some very sobering news this morning, nearly $300 billion in overruns. And I think the good news we hear is that 

there are tools in place for us to understand the causes of these 

problems, which then gives us the ability to find solutions. 

What is disturbing, of course, is that the trending is in the wrong 

direction. If you look at the difference between 2003 and 2008, I 

think your data are accurate. The problem is getting worse and not 

better. And I think that goes to Mr. Conaway’s point, and Mr. Cooper’s point earlier. You know, the ream upon ream of assessment 

of this has gotten us nowhere—worse than nowhere. 

So, we are interested in trying to put teeth into the decisionmaking process in a way that makes this work. 

Again, the record will be open for any member to submit more 

questions to the panel. 

We thank both panelists for excellent presentations this morning, 

and we stand adjourned. 

[Whereupon, at 8:54 a.m., the panel was adjourned.] 
A P P E N D I X 

APRIL 1, 2009 

PREPARED STATEMENTS SUBMITTED FOR THE RECORD 

APRIL 1, 2009 

(33) 
34 
35 
36 
37 
38 
39 
40 
41 
42 
43 
44 
45 
46 
47 
48 
49 
50 
51 
52 
53 
54 
55 
56 
57 
58 
59 
60 
61 
62 
63 
64 
65 
66 
67 
68 
69 
70 
71 
72 
73 

WITNESS RESPONSES TO QUESTIONS ASKED DURING 

THE HEARING 

APRIL 1, 2009 

(77) 

RESPONSES TO QUESTIONS SUBMITTED BY MR. ANDREWS 

Mr. AHERN. The version of DoD Instruction 5000.2 just issued in December 2008 

eliminates much of what others refer to as ‘‘wiggle room’’ compared to the 2003 

version. I view the Department’s task now as one of the Components properly executing their acquisition programs in accordance with DoDI 5000.2 and the OSD staff 

ensuring we undertake disciplined, thorough program and milestone reviews before 

allowing programs to proceed to the next acquisition phase. We will monitor how 

the new guidance is being applied and, if necessary, will issue policy changes to 

sharpen and improve the process. If any of our changes require new legislation, the 

Department will submit a legislative provision for your consideration. [See page 28.] 

Mr. AHERN. What you suggest is a SAR-like submission during the Technology 

Development phase of the acquisition process. Technology Development is a continuous technology discovery and development process reflecting close collaboration between the S&T community, the user, and the system developer. It includes significant competitive prototyping that will inform us on the realism of requirements and 

the maturity of technology. It will also significantly improve our cost estimates for 

the Engineering and Manufacturing Development and the Production and Deployment phases. However, at this point in the acquisition process there is no clearly 

defined program to report on. As such, an annual SAR-like submission that purports 

to provide SAR-quality information for the technology development effort’s life cycle 

would have limited credibility or utility. There is an existing certification requirement (10 U.S.C. 2366a) that does establish expectations for system cost during technology development. Under that provision, if the projected cost of the system, at any 

time prior to Milestone B approval, exceeds the cost estimate for the system submitted at the time of the certification by at least 25 percent, the Milestone Decision 

Authority, in consultation with the Joint Requirements Oversight Council, shall determine whether the level of resources required to develop and procure the system 

remains consistent with the priority level assigned by the Joint Requirements Oversight Council. The Milestone Decision Authority may then withdraw the certification concerned or rescind Milestone A approval if the Milestone Decision Authority determines that such action is in the interest of national defense. This is consistent with the iterative nature of the Technology Development phase in assessing 

the viability of technologies while simultaneously refining user requirements allowing the Milestone Decision Authority to make an informed judgment whether the 

priority warrants committing to a higher-cost program. Once the program has been 

initiated and receives Milestone B approval, the Department establishes an Acquisition Program Baseline, holds the program manager accountable for execution to it, 

and provides Congress the SAR. [See page 28.] 

Mr. SULLIVAN. A pre-Milestone B SAR-like report could be a valuable tool for assessing whether a program is on track to have a solid understanding of requirements, technology, and cost before formally becoming an acquisition program. It 

could serve as the basis for illuminating early trades in all areas (cost, schedule, 

requirements, and technology) and as an early warning mechanism to identify programs that are proceeding without the requisite knowledge in those areas. However, 

given that the pre-Milestone B technology development phase is a time when cost, 

schedule, and performance trades should be encouraged, we would not recommend 

using this reporting tool as a baseline control mechanism to apply Nunn-McCurdylike standards to technology development costs. Annual reporting could begin at 

Milestone A. Much like the current SAR, it could provide basic information about 

the mission need the program fulfills the acquisition and technology development 

strategies, the program’s activities to date, and contract performance. It could also 

provide information on the programs’ schedule, cost, performance, and knowledge 

that is tailored to the early stages of the acquisition process. These data could include the following: 

Capability Need: A description of the capability need that justifies the program, 

including the following: 

• capability gap that needs to be filled 

• priority level assigned by JROC to this capability need 
78 

• timeframe in which the overall capability is required 

• type of materiel solution preferred—information system solution, evolutionary 

development of an existing capability, or a transformational approach 

Analysis of Alternatives (AOA): A description of the assessment and results, 

including the following: 

• the scope of alternatives considered in the AOA 

• the recommended solution derived from the AOA 

• the technical, operational, and programmatic risks identified with the recommended solution 

Schedule: A baseline estimate set at Milestone A and current estimates for the 

completion of the following: 

• Systems engineering reviews: System Functional Review, System Requirements 

Review, Software Specification Review, Preliminary Design Review 

• Technology development: Technology Readiness Assessment, Prototype Demonstration (start and completion) 

• Requirements: Capability Development Document 

• Development cycle: Estimated cycle time in months (Milestone A to B), Estimated cycle time in months (Milestone B to C) by increment of capability (if 

applicable) 

Cost: A baseline estimate set at Milestone A and current estimates in base year 

and then year dollars for the following: 

• Cost estimate for Milestone B through completion reported as a range of likely 

costs 

Performance: Prioritized list of Key Performance Parameters that includes: 

• Proposed performance baseline at Milestone A 

• Current estimate of performance 

• Level of performance that will be demonstrated in the Technology Development 

Phase 

• Level of performance that has been demonstrated in the Technology Development Phase 

• Critical technologies that are enablers for each Key Performance Parameter 

• Description of requirements that were added or removed during the Technology 

Demonstration Phase 

Technology knowledge: List of the program’s critical technologies that includes: 

• Milestone A, current, and projected Milestone B technology readiness levels 

• Most current test environment (lab, relevant, operational) 

• Most current physical status (breadboard, functional prototype, full-up prototype) 

• Description of trades available if technology does not mature as planned (use 

an existing technology, reduce or defer requirements, etc) 

• Schedule for maturing technologies to TRL level 7 (i.e. demonstrated in a realistic environment) 

Design knowledge: Current estimates of the following: 

• Total and projected number of drawings released by the Preliminary Design Review 

• Estimated size of the software development effort (in lines of codes) 

[See page 28.] 

RESPONSE TO QUESTION SUBMITTED BY MR. CONAWAY 

Mr. SULLIVAN. DOD policy requires the CAIG to prepare an independent life cycle 

cost estimate for a major defense acquisition program Milestone B decision. The policy states that the Milestone Decision Authority shall consider the CAIG estimate 

before approving the program to start system development. It does not require a reconciliation of the CAIG estimate with other service or program office estimates. 

In a July 2008 report, GAO found that program cost estimates are often significantly understated—a finding consistent with cost growth patterns reported by 

RAND, the Institute for Defense Analysis (IDA), and other organizations that con-
79 

1GAO, Defense Acquisitions: A Knowledge-Based Funding Approach Could Improve Major 

Weapon System Program Outcomes, GAO–08–619 (Washington. D.C.: July 2, 2008); Assessment 

Panel of the Defense Acquisition Performance Assessment Project for the Deputy Secretary of 

Defense, Defense Acquisition Performance Assessment Report (Washington, D.C.: Jan. 2006): Defense Science Board, Defense Science Board Summer Study on Transformation: A Progress As- sessment (Washington, D.C.: Feb. 2006); RAND, Historical Cost Growth of Completed Weapon 

System Programs (Santa Monica: 2006); and Institute for Defense Analysis, Costs Growth in 

Major Weapon Procurement Programs, Presentation to the 38th Annual DoD Cost Analysis Symposium (Williamsburg: Feb. 2005). 2GAO. Defense Acquisitions: Assessments of Selected Weapon Programs. GAO–09–326SP 

(Washington, D.C.: March 30, 2009). 

duct defense analyses.1 In that report, GAO analyzed the cost of 20 major defense 

acquisition programs through December 2007. While the CAIG estimates generally 

underestimated costs by a smaller amount than program office and service estimates, the CAIG estimates could underestimate a program’s costs by billions of dollars (see table 1). For example, the initial service estimate for the development of 

the Marines’ Expeditionary Fighting Vehicle was about $1.1 billion. The CAIG estimated the development cost of the program to be $1.4 billion, but the expected development costs for the program had grown to close to $3.6 billion. In the case of 

the Future Combat System (FCS), the Army’s initial estimate for the development 

cost was about $20 billion, while the CAIG’s estimate was $28 billion. DOD began 

the program using the Army’s estimate of $20 billion, but development costs for FCS 

had grown to an estimated $28 billion. Many programs are also approved to start 

development based on the service or program office cost estimate rather than the 

CAIG estimate. Less than a quarter of the 48 programs in GAO’s 2009 assessment 

of weapon system programs that provided data used the estimate made by the CAIG 

as a basis for the program’s baseline, while almost 70 percent of the programs used 

the program office or service cost estimate.2 [See page 27.] 
80 

RESPONSE TO QUESTION SUBMITTED BY MR. SESTAK 

Mr. AHERN. The Department addresses Modeling and Simulation (M&S) 

Verification, Validation, and Accreditation (VV&A) both in policy and in guidance. 

On the policy side, DoD Instruction 5000.2, ‘‘Operation of the Defense Acquisition 

System,’’ addresses M&S as part of an integrated test and evaluation (T&E) continuum that includes developmental, operational, and live fire T&E; family-of-systems interoperability testing; and information assurance testing. The Test and Evaluation Strategy requires empirical data to validate models and simulations and expects reconciliation of pre-test predictions with post-test results. The policy also provides for the use of accredited models in support of developmental T&E, initial operational T&E, and live fire T&E. DoD Instruction 5000.59, ‘‘DoD Modeling and Simulation (M&S) Management,’’ establishes the M&S Steering Committee that oversees the development of VV&A policies, plans, and procedures. DoD Instruction 

5000.61, DoD ‘‘Modeling & Simulation Verification, Validation and Accreditation,’’ 

establishes common-sense guidelines and requires that models and simulations used 

to support major DoD decision-making organizations and processes (e.g., the Defense Acquisition Board; Joint Requirements Oversight Council; and Planning, Programming, Budgeting, and Execution System) ‘‘shall be accredited for that specific 

purpose by the M&S application sponsor’’. DoD Instruction 5000.61 also requires 

that VV&A be documented. In terms of guidance, the Department has taken a number of steps. The Defense Acquisition Program Support Methodology used in program support reviews includes strong criteria for evaluating a program’s VV&A efforts. We have an on-line ‘‘VV&A Recommended Practices Guide.’’ A new military 

standard, MIL–STD 3022, ‘‘Documentation of VV&A for M&S’’ was approved last 

year and is already in use for acquisition purposes across the Department. A DoD 

VV&A Documentation Tool automates production of the MIL–STD 3022 VV&A document set and became operational this year. We are also developing risk-based 

VV&A guidelines and pursuing routine examination of VV&A when M&S informs 

major acquisition decisions. The Department does not keep central records of VV&A 

and no studies have been performed to assess VV&A documentation. So, without a 

data call to the DoD Components, it is not practical to provide a quantitative assessment of overall DoD VV&A performance. The Department recognizes that VV&A is 

important so that we have confidence in our models and simulations. While VV&A 

is covered both in policy and guidance, we also know that we need to continue working with the Components to ensure models used in our decision-making processes 

are properly accredited. [See page 24.] 

Æ