i

DEFENSE SYSTEMS MANAGEMENT COLLEGE

SIMULATION BASED

ACQUISITION:

A NEW APPROACH

Report of the

Military Research Fellows

DSMC 1997-1998

Lieutenant Colonel Michael V. R. Johnson, Sr., USA

Lieutenant Colonel Mark F. McKeon, USMC

Lieutenant Colonel Terence R. Szanto, USAF

December 1998

PUBLISHED BY THE

DEFENSE SYSTEMS MANAGEMENT COLLEGE PRESS

FORT BELVOIR, VIRGINIA 22060-5565
ii

DISCLAIMER

This book was produced in the Department of Defense (DoD) school

environment in the interest of academic freedom and the advancement

of national defense-related concepts. The views expressed in this book

are those of the authors and do not reflect the official position or policy

of the DoD or those of the United States Government.
iii

ACKNOWLEDGMENTS

This Report is the result of an 11-months Military Research Fellowship

program sponsored by the Defense Systems Management College.
iv

NOTICE

Copies of this report may be obtained by writing or faxing:

DEFENSE SYS MGMT COLLEGE

ATTN: AS PR

9820 BELVOIR RD, STE G3

FT BELVOIR, VA 22060-5565

Telephone: (703) 805-4366

DSN: 655-4366

Fax: (703) 805-3726
v

TABLE OF CONTENTS

Page

Preface........................................................................................................................................ vii

Objective of Study .............................................................................................................. vii

Methodology....................................................................................................................... viii

Special Thanks.................................................................................................................... ix

Chapter 1 – Introduction

Better Performance............................................................................................................ 1-1

Faster Schedule .................................................................................................................. 1-2

Cheaper Cost ...................................................................................................................... 1-3

Guide to this Report .......................................................................................................... 1-4

Chapter 2 – Definitions and Terminology

M&S Definitions ................................................................................................................ 2-1

Hierarchies and Classes of Models and Simulations ...................................................... 2-2

Virtual and Synthetic ......................................................................................................... 2-5

Lifecycle Cost and Total Ownership Cost ........................................................................ 2-5

SBA Vision Statement ....................................................................................................... 2-6

SBA Definition ................................................................................................................... 2-7

Chapter 3 – Background

Guidance ............................................................................................................................. 3-1

Previous and On-Going Efforts ........................................................................................ 3-2

Common Operating Digital Environment (CODE)....................................................... 3-5

Defense Systems Affordability Council (DSAC) ............................................................ 3-5

Joint SBA Task Force ........................................................................................................ 3-5

Assumptions, Trends, and SBA Enablers ........................................................................ 3-6

Chapter 4 – Essential Aspects of SBA

Balancing Requirements ................................................................................................... 4-1

Numerous Design Alternatives ......................................................................................... 4-2

The Iterative Design Process ............................................................................................ 4-5

Converging on an Optimal Solution................................................................................. 4-5

Test as an Integral Part of Design ..................................................................................... 4-7

Decision Risk Analysis ...................................................................................................... 4-9
vi

Chapter 5 – Expanding the SBA Envelope

Factors Influencing the Use of SBA: Domain, Infrastructure, and Need .................... 5-1

SBA Push and SBA Pull .................................................................................................... 5-5

SBA Push ............................................................................................................................ 5-5

SBA Pull.............................................................................................................................. 5-9

Chapter 6 – A Future State of SBA

Needs Generation Phase ................................................................................................... 6-3

Iterative Design and Development Phase ....................................................................... 6-4

Production Phase ............................................................................................................... 6-5

Operations and Sustainment Phase.................................................................................. 6-6

Net Result ........................................................................................................................... 6-6

Chapter 7 – Challenges to Implementing SBA

Cultural Challenges ........................................................................................................... 7-1

Technical Challenges.......................................................................................................... 7-2

Process Challenges............................................................................................................. 7-6

Chapter 8 – Conclusions and Recommendations

Conclusions......................................................................................................................... 8-1

Recommendations ............................................................................................................. 8-2

APPENDICES

Appendix A – Acronyms and Glossary ............................................................................. A-1

Appendix B – Interviews and Personal Contacts............................................................. B-1

Appendix C – About the Authors ..................................................................................... C-1

LIST OF FIGURES

Figure 2-1. Hierarchies of Models and Simulations ..................................................... 2-2

Figure 2-2. Matrix of Classes of Models and Simulations............................................ 2-3

Figure 2-3. Hierarchies and Classes of Models and Simulations................................. 2-4

Figure 2-4. Three Principal Interacting Communities.................................................. 2-8

Figure 2-5. Application of M&S Across Three Dimensions ........................................ 2-9

Figure 3-1. Growth in Computing Power....................................................................... 3-6

Figure 4-1. Affordability: Capability vs. Cost ................................................................ 4-3

Figure 4-2. SBA Process .................................................................................................. 4-4

Figure 4-3. Simulate-Emulate-Stimulate Process, Top-Down View............................ 4-6

Figure 4-4. Simulate-Emulate-Stimulate Process, Side View ...................................... 4-6
vii

PREFACE

This study presents the combined efforts of three military Research Fellows, participating in an

11-month Defense Systems Management College Research Fellowship program, sponsored by

the Under Secretary of Defense for Acquisition and Technology. In keeping with its role as the

center for systems management education in the Department of Defense (DoD), the Defense

Systems Management College (DSMC) conducts this annual fellowship program to research a

subject of vital interest to the U.S. defense acquisition community.

To achieve program objectives, program managers have long been applying modeling and simulation (M&S) tools to efforts within the various stages of their programs. Recently, however,

declining defense budgets have increased the pressure on the acquisition community to find

cheaper ways to develop and field systems. Additionally, the rapid pace of changing world events

demands that these material solutions get into the hands of the warfighter faster. To meet the

increased challenge of budget and time constraints, many programs have radically changed the

way they conduct business. These programs recognize the powerful increases in productivity

and decreases in cost brought by M&S tools. Within these programs, program management

looks to weave M&S applications across program phases and seeks to leverage the strengths of

external M&S applications to efforts within their program. This new way of doing business,

coupling rapid advances in simulation technology with process change, is fueling a new approach

to how we acquire defense systems. This new approach is being termed Simulation Based

Acquisition, or SBA.

Objective of Study

The objective of this book is to convince program managers that SBA is a smarter way of doing

business. We will do this by defining SBA, explaining the strengths of SBA, and describing the

forces that will encourage its use. Where possible, we highlight best practices and useful

implementation guidance.

Within the DoD, there are a staggering number of variables that an acquisition program office

must evaluate and analyze. There is an almost infinite number of possible applications of SBA

activities within DoD acquisition programs. Where they apply, we present examples of commercial applications of SBA. Most, however, are narrowly focused and are of questionable

general use to acquisition program offices. This is partly because acquisition programs within

the DoD are unique in their complexity compared with many commercial enterprises. Systems

that the DoD produces are usually composed of many varied sub-components that push the

boundaries of technology. When these complex sub-components are brought together in the

aggregate at the system level, the complexity of the program is compounded. We hope that this

“round down range” will stimulate discussion and provide the mark from which to “adjust fire.”
viii

Methodology

We conducted our research in four areas. First, we embarked on basic research and information

collection while attending the 12-week residential Program for Management Development

(PMD) course at the Harvard University Graduate School of Business. Second, we conducted

an extensive search of the applicable literature. Third, we conducted interviews and attended

briefings and conferences on Simulation Based Acquisition. Finally, we held two in-process

reviews with students attending the DSMC Advanced Program Managersment Course (APMC),

and with the DSMC faculty.

Our initial research began while we were at the Harvard Business School. There, we presented

and discussed issues concerning SBA with faculty members and our fellow classmates. As our

classmates (157 students from 38 countries) represented both U.S. and international companies, we gained truly global insights into a few of the topic areas. We were also fortunate to have

a few classmates working for U.S. defense contractors, who provided valuable perspectives into

government and industry interrelationships and model sharing. Our discussions centered on

applicable business practices and answers to numerous questions raised by our research topic.

For example, should program managers be provided incentives to design and develop models

and simulations that allow for reuse and/or integration into other programs? Are there applicable business practices, or measures of success/metrics that could evaluate the effective use of

a government program’s modeling and simulation efforts? Where did they see technology

going in the near future?

Our second area of research was a comprehensive literature review and Internet search covering the topical areas of modeling and simulation and Simulation Based Acquisition. A particularly useful area was the SBA Special Interest Group on the Defense Modeling and Simulation

Office’s World Wide Web home page (www.dmso.mil), which contains a lot of historical and

current information on the subject, as well as up-to-date links to modeling and simulation

organizations and groups which are active on the web.

Our third area of research, conducting interviews and attending briefings and conferences,

provided most of our information. We broke our areas of emphasis into three categories: government, defense industry and commercial industry. To gain insight into government programmatic issues, we visited Service Acquisition Offices, acquisition and test organizations, newly

formed program offices, and established program offices. We obtained an understanding of the

defense industry’s support to government programs through visits to corporation headquarters

and contractor facilities. We visited commercial firms that have been making significant

investments in simulation technology. In all, we conducted over 85 interviews (see Appendix B). Some interviews were as short as thirty minutes, while some lasted over the course

of three days. Most, however, were three or four hours in duration. The level of the interviewees

varied greatly, from Senior Acquisition Officials and Program Managers to individuals tasked

with constructing physical wooden mock-ups (used in the verification of virtual models). Though

the sources varied, there was a great deal of commonality in the views expressed. We also
ix

participated in and attended several SBA conferences and workshops. Each site visited provided

unique insights into the collage that is the Simulation Based Acquisition picture.

Our final area of data collection was through peer and faculty review at DSMC. Through frank

discussions conducted in our office spaces and the use of the DSMC’s Management Deliberation Center (a Group Decision Support System), many of the APMC students provided us with

an excellent sounding board on the direction and progress of our research. These in-process

reviews helped ensure we were addressing the issues most important to the acquisition community concerning Simulation Based Acquisition.

Special Thanks

The Research Fellows extend a special note of appreciation to Ms. Joan Sable, DSMC Military

Research Fellowship Coordinator. Ms. Sable ensured that our administrative and logistical

requirements were met at DSMC and Harvard, and her support enabled us to concentrate our

attention and energies on the research and writing of this report.

For all of their guidance throughout our research project we pay special thanks to Colonel

Kenneth “Crash” Konwin, Director Defense Modeling and Simulation Office; Ms. Robin Frost,

Office of the Secretary of Defense (Director of Test, Systems Engineering, and Evaluation);

and Mr. Steve Olson and the other members of the National Defense Industrial Association’s

SBA Industry Steering Group.

We appreciate the efforts of the DSMC Press staff for their many hours working on this report

to ensure its highest quality. Thanks to the Visual Arts and Press staff for their work on the

graphs, charts and cover page as well as their many hours in the layout of this report. Finally, we

extend a special thank you to Air Force Academy Cadets First Class Paul Ferguson and Nathan

Atherley for their research assistance during their summer internship at DSMC.

There are others, too numerous to mention individually, who deserve recognition. The three

Research Fellows would like to thank all of those interviewed. As a token of our appreciation,

we dedicate this effort to you. May our report be as helpful to you as you were to us.
x
1-1

1

INTRODUCTION

acquisition processes (indeed the very solutions

themselves) are usually complicated.

To be successful, a program office must balance the complex relationships that exist

between “better”, “faster”, and “cheaper.”

The degree of balance is not usually measured

directly, but it can be measured in terms of

the risk in meeting objectives. Risk is a measure of the inability to achieve a program’s

defined performance, schedule, and cost

objectives. It has two components: the probability of failing to achieve particular performance, schedule, or cost objectives and the

consequences of failing to achieve those

objectives.1

 SBA can address the first component of risk by increasing the likelihood of producing systems that have “better” performance,

“faster” schedule, and “cheaper” cost.

Better Performance

No one would advocate providing the

warfighter with something that is “almost as

good as.” The material solutions that the U.S.

defense acquisition community produces are

the tools of victory on future battlefields. Finding and producing “better” solutions starts

with the warfighter’s ability to correctly articulate the requirement. This frames all of the

acquisition activities that follow.

The Services and DoD are committed to provide superior weapon systems and materials

to the warfighter faster, and at less cost. To

achieve this, the acquisition community is

charged with providing “better, faster, and

cheaper” material solutions. Our research

shows that the practices and processes of simulation based acquisition (SBA) will help overcome many of the hurdles associated with

acquiring “better, faster, and cheaper”

solutions.

The phrase “better, faster, and cheaper” is

deceptively simple, because the execution of

this task across the Services and DoD is a complex undertaking. As many program offices

will attest, this statement rings true for three

reasons. First, the current Government acquisition process, with its oversight requirements

and the nature of its funding, is not the most

streamlined or efficient of processes. Second,

many large Government acquisition programs

develop complex systems, which push the limits of technology. And third, program complexity is further magnified since most of the material solutions produced are not stand-alone

products. That is, they are “force multipliers”

that must interface with and enhance the combat performance of other systems that are either fielded or in development. Unlike many

commercial programs, therefore, the current
1-2

Concepts that satisfy stated requirements must

then be identified and evaluated. The military

has a much harder problem to solve than does

the commercial automotive or aviation industries, since they usually have a good starting

point. And while they produce some very complicated products, the basic concepts for commercial products do not deviate greatly from

one generation to the next. Follow-on systems

within the DoD, on the other hand, are usually radical departures from their predecessors. Take as an example, air superiority fighters; the F-22 can hardly be viewed as a variant

of the F-15. The increased performance

demanded of follow-on DoD systems brings

with it the increased likelihood that these

performance objectives will not be met.

Through the use of modeling and simulation,

an SBA process offsets this increased performance risk in three ways. First, it enables the

warfighter to become a member of the design

team and to influence the design much earlier

than the current process allows. Simulation

provides the tools for the warfighter to visualize and interact with the system to perform

operational analyses and assessments. The

design team rapidly incorporates necessary

changes into the design based upon this expert

input and the result is a better solution. Second, the SBA process provides rapid feedback

to the design team by enabling them to perform “what if” analyses, or iterations, on

hundreds of point designs. Rather than building a physical prototype to validate a single

point design, designers can use a virtual prototype to look at potentially hundreds of

design variations. The rapid feedback and

learning from these iterations will enable the

program office to produce a better system.

Third, not only are we able to conduct more

iterations, but we are also able to converge on

more optimal designs because we can test

many of our assumptions to see if they are true.

Furthermore, as the tools for looking at the

entire life cycle of the system improve, design

teams will be able to move from form and fit

issues to the more complex issues of function.

This, too, will contribute to better solutions.

Faster Schedule

The military goes to war with what is fielded,

not with what is on the drawing boards or in

the acquisition pipeline. As Lieutenant General Paul J. Kern, Director of the Army’s

Acquisition Corps stated, “The current acquisition process was good for producing systems

in the Cold War environment, where we had a

predictable enemy with known lead times.

Now, many of our foreseeable potential

enemies are different; they are not constrained

by a rigid, inflexible acquisition process. They

can purchase weapon systems and/or sub-components in an open-air market environment,

like a global off-the-shelf system. Through

mixing and matching various weapon systems

and subsystems, they can rapidly generate some

very lethal systems. We lose if they can purchase

and bring together their systems faster than

we can develop ours because of long cycle

times.”2

By enabling the acquisition process to get

inside the adversary’s decision cycle, an SBA

process increases the likelihood of developing and producing systems “faster”. If we can

model what we need answers to, we can get

rapid feedback through the power of simulation. The program gets the required information much sooner in the process than it does

under the current system, translating into

faster cycle times. Additionally, we can conduct many more processes concurrently

because more people can access the information they need at the same time. Concurrent
1-3

engineering activities within an SBA process

will compress the cycle time of weapons acquisition. Working with virtual prototypes and digital product descriptions allows program managers to exhibit agility in designing and evaluating concepts. For example, using computational fluid dynamics, the test community can

help the design team look at weapons release

issues much earlier and at a fraction of the cost

of running a wind tunnel test. Virtual manufacturing allows the manufacturing community

to look at procedures such as reducing the

number of parts in the design as well as rehearsing production processes. Using threedimensional solid modeling, shop-floor mechanics can assess the maintainability of the

design and make design inputs alongside the

engineers, much earlier in the process.

Cheaper Cost

“Cheaper” can be broken down into two areas.

The first area involves reducing the initial

acquisition cost of systems. This point was

illustrated by Norm Augustine two decades

ago in one of his famous Augustine’s Laws:

“In the year 2054, the entire defense budget

will purchase just one tactical aircraft. This aircraft will have to be shared between the Air

Force and Navy 3 days each per week.”3

Clearly there’s a need to reverse and stabilize

the trend in rising initial acquisition costs. Second, “cheaper” also includes reducing the annual costs of operating and sustaining systems

until and including disposal. The foreseeable

trend in system life span is that DoD will be

living with major systems for longer periods

of time. As systems get older, the sustainment

costs usually rise. To address concerns within

these two areas, the Defense Systems

Affordability Council, chaired by the Under

Secretary of Defense for Acquisition and Technology, recently directed the establishment of

ambitious top level goals for reducing total

ownership cost.

Regarding the first area of reducing initial

acquisition costs, an SBA process increases the

likelihood that a program will stay within its

cost objectives during the acquisition phase.

The synthetic environment reduces the reliance on costly physical prototypes and tests

for making programmatic decisions. Physical

tests are primarily done to validate models and

generate confidence in the use of the models.

These validated models can then be reused to

perform numerous design simulations, at a

fraction of the cost of one physical test. Simulations also help to focus the test effort on the

critical evaluation areas, thereby avoiding unnecessary physical tests. As noted by Air Force

Major General Leslie Kenne, Program Manager for the Joint Strike Fighter aircraft, “We

haven’t begun to tap all the benefits that modeling and simulation has to offer to reduce our

testing requirements.”4

Regarding the second area of reducing sustainment costs, an SBA process increases the

likelihood that a program can reduce the

annual costs of operating and sustaining systems until and including disposal. Although

it’s difficult to find a definitive source for the

data, it is frequently mentioned that approximately 70 percent of the total costs of a system are determined by the time a new program receives approval to start (referred to as

“Milestone I”), and 85 percent are determined

by the time a program’s design is selected

(referred to as “Milestone II”).5

 These early

decisions, which directly affect total ownership

cost, are currently being made with limited

knowledge of system cost, schedule and performance implications. It is speculative at best

to determine how much of a system’s total

ownership cost can be influenced by better
1-4

design decisions. Certainly, however, as the

ability to simulate the entire system’s life cycle

continues to improve, so too will the ability to

make intelligent tradeoffs on how much to

spend now in improvements in manufacturability, reliability, maintainability, and supportability in order to save later. A significant

value of SBA is that it allows program offices

to begin evaluating the long-term cost impacts

of design decisions as part of the design

process, rather than relying on engineering

change proposals and modifications to fix the

problem after the design is frozen.

Guide To This Report

Chapter Two, Definitions and Terminology,

lays the foundation for the SBA Vision

Statement and Definition that follow.

Chapter Three, Background, presents current

policy and guidance, previous studies and conclusions, and some assumptions and trends

upon which SBA depends upon.

Chapter Four, Essential Aspects of SBA,

introduces the theoretical practices that

support an SBA process.

Chapter Five, Expanding the SBA Envelope,

identifies the forces that will generate progress towards implementing SBA. These

include the forces internal to a program that

will push SBA along, as well as the external pull from the warfighting and resource

allocation communities.

Chapter Six, A Future State of SBA, develops

a model and vision for what SBA can hope to

achieve in the future, by identifying notional

SBA components and their interactions.

Chapter Seven, Challenges to Implementing

SBA, describes some of the challenges that a

program may face as it tries to implement an

SBA approach.

Chapter Eight, Conclusions and Recommendations, summarizes our conclusions about the

prospects for implementing SBA, and provides

our recommendations for making the transition to this new way of acquiring defense

systems.

What we present throughout this report we have

heard many times from many people. It is our

hope that we have presented our research in

such a way as their message comes across loud

and clear and that you are convinced that SBA

is a better way of doing business.
1-5

ENDNOTES

1. Defense Systems Management College,

Acquisition Strategy Guide, Third Edition, (Fort

Belvoir, VA: Defense Systems Management

College Press, January 1998).

2. LTG Paul J. Kern, Military Deputy to Assistant

Secretary, Army RD&A, interview with authors,

7 May 1998.

3. Norman R. Augustine, “Augustine’s Laws and

Major System Development Programs,” Defense

Systems Management Review, Volume 2,

Number 2, p. 64.

4. Major General Leslie J. Kenne, Program Manager, Joint Strike Fighter Program, Remarks at

the National Defense Industrial Association SBA

Conference, 17 March 1998.

5. John Krouse. Computer Aided Engineering,

Onward Press, June 1993, p. 7.
1-6
2-1

2

DEFINITIONS AND

TERMINOLOGY

logical representation of a system, entity, phenomenon, or process.”1

 (Entities are “a distinguishable person, place, unit, thing, event,

or concept about which information is kept.”2

)

It is important to note that a model can exist

without a single piece of software.3

 It can be a

hardware mockup or a simple equation on a

piece of paper. A simulation is “a method for

implementing a model over time.”4

 Thus, to

tie the two together, simulations are the software that implements models over time, within

the context of a scenario.5

A great deal of confusion often results from

the common practice of using the terms “modeling” and “simulation” interchangeably, as

well as that of using the term “M&S” to stand

for both models and simulations and modeling

and simulation.6

 In this report we use the term

“M&S” to stand for modeling and simulation,

which is an analytical problem-solving approach.7

 Modeling and simulation, as defined

in the DoD M&S Glossary, is “the use of models, including emulators, prototypes, simulators, and stimulators, either statically or over

time, to develop data as a basis for making

managerial or technical decisions.”8

Many within the DoD, the Armed Forces, and

industry realize the enormous potential of a

Simulation Based Acquisition process, and

SBA has recently been the topic of several conferences and workshops. There has been some

disagreement and confusion, however, in getting to a common understanding of exactly

what SBA is. Certainly SBA is a new way of

doing business for acquiring DoD weapon systems, which implies a need to change our processes. And it has been described by the policy

and cultural changes necessary to bring about

this changed process, the favorable environment necessary to speed it along, as well as

the technical impediments to its swift enactment. But what exactly is SBA?

M&S Definitions

First we need to clarify some terminology used

throughout this book. (We highly recommend

referring to DoD 5000.59-M, DoD Modeling

and Simulation (M&S) Glossary, for current

definitions in this fast changing area. It is available on-line on the DMSO World Wide Web

Home page, mentioned previously in the Preface.) According to DoD 5000.59-M, a model

is “a physical, mathematical, or otherwise
2-2

Hierarchies and Classes of Models

and Simulations

As shown in Figure 2-1, models and simulations have been classified hierarchically

according to their level of aggregation. Aggregation is the ability to group entities, while

preserving the effects of entity behavior and

interaction when grouped.9

 The four general

classifications are Engineering, Engagement,

Mission/Battle, and Theater/Campaign.10

Engineering models and simulations are at the

least level of aggregation, whereas Theater/

Campaign models and simulations are at the

highest. Within these classifications, models

and simulations can vary in their level of detail

or representation.

• Engineering level models and simulations

provide measures of performance (MOP)

concerning such issues as design, cost,

manufacturing, and supportability. They

can include aerodynamics, fluid flow,

acoustics, and fatigue, as well as physicsbased models of components, subsystems,

and systems.11

• Engagement level models and simulations

are used for evaluating the effectiveness of

an individual system against another system in one-on-one, few-on-few, and manyon-many scenarios. They provide measures

of effectiveness (MOE) at the system-onsystem level.12

• Mission/Battle level models and simulations are used for evaluating the effectiveness of a force package, or multiple platforms performing a specific mission. They

provide MOE at the force-on-force level.13

• Theater/Campaign level models and simulations are used to evaluate the outcomes

of joint and combined forces in a theater

Figure 2-1. Hierarchies of Models and Simulations

Mission/ 

Battle

Engagement

Engineering

Theater/

Campaign
2-3

Real

Real

Simulated

Virtual Constructive

Live Smart

Simulated

People

Equipment

or campaign level conflict. They provide

measures of value (sometimes referred to

as measures of outcome-MOO) at the

highest levels of conflict.14

This classification of models and simulations

is significant to acquisition programs, because

the type of information required determines

the level of aggregation to be used. Solutions

to broad issues, such as mission need statements, can be explored with highly aggregated

models and simulations. More focused issues,

such as the maturing of the design, dictate that

the models and simulations move towards and

into the engineering category. Programs must,

therefore, move up and down this ladder of

abstraction to tailor the models and simulations to their needs. For example, highly aggregated simulations can explore the battlefield effects of increasing an aircraft’s combat

radius. If the results are promising, the design

team could then use engineering level models

and simulations to address the associated

detailed design issues, such as increasing the

aircraft’s internal fuel capacity. There are also

additional considerations for tailoring the

level of aggregation, such as unnecessary

computational burden and the complexity of

information management.

Models and simulations have long been classified into the three classes of Live, Virtual,

and Constructive, which attempt to delineate

the degrees of human and equipment realism,

as shown in the matrix in Figure 2.2.15 Live

simulations denotes real people operating real

systems; virtual simulations denotes real

people operating simulated systems; and constructive models and simulations denote simulated people operating simulated systems.

Smart simulations denotes simulated people

operating real equipment. The live, virtual and

smart classes are applied to simulations, but

not models since live models would be humans

Figure 2-2. Matrix of Classes of Models and Simulations
2-4

Battle

Theater/

Validation

Mission/ Live

Engagement

Engineering

Campaign

Constructive

Virtual

Smart

and actual equipment. The term virtual model

would be misleading since virtual simulations

inject humans-in-the-loop to exercise motor

control, decision making, or communications

skills, and the human element of a virtual simulation is not modeled (the simulated systems

in virtual simulations would be made up of

constructive models). On the other hand, in

constructive simulations, real people stimulate

(make inputs) to the constructive models, but

the people are not involved in determining the

outcomes.17 Hence it is appropriate to have

both constructive models and constructive

simulations.

As noted by the DoD M&S Glossary, the classification of live, virtual, and constructive can

be somewhat problematic for two reasons.

First, there is no clear delineation between

the categories, because the degrees of

human involvement and equipment realism

are infinitely variable. The second reason it is

a problematic classification is highlighted by

the fact that the bottom right quadrant of

Figure 2.2 has not been previously named

to indicate the class of simulated people

operating real equipment.18 We have named

this class of simulations as smart simulations.

There is value, however, in marrying the hierarchies and classes of models and simulations

together, as shown in Figure 2-3, to show the

range of possibilities when discussing M&S

support to acquisition. Constructive models

and simulations can range from highly aggregated, theater level models and simulations,

to physics-based, engineering level models and

simulations. This range of aggregation can be

applied to virtual simulations, and we can

envision the same for the obverse of virtual

simulations (the “smart” classification). The

possible combinations are wide-ranging, including such hybrid combinations as hardware/

Figure 2-3. Hierarchies and Classes of Models and Simulations
2-5

software in the loop (HW/SWIL), human in

the loop (HITL), and wargaming. It is not

necessary to package every model or simulation into a particular category—the utility is

in exploring the numerous possible hybrid

combinations available and tailoring them to

the requirement.

While the classification of live simulation was

useful for the training community, the concept

of real people and real equipment takes on a

different purpose for the acquisition environment. In the training vernacular of “all but war

is simulation,” real people and real equipment

are still simulations of the real event of war.

For the acquisition of systems, there are two

roles for the classification of real people and

real equipment. The first role is testing that

which cannot be adequately depicted in the

synthetic environment, because of insufficient

knowledge or technology. The data obtained

from these tests could be used for future model

development. The second role is to validate

the models and resultant simulations. The

“live” events in these two roles can range from

component level tests to large-scale physical

prototypes, but in each case their purpose is

to validate. The first role validates either a concept or design, whereas the second role validates a model or simulation.

Virtual and Synthetic

When discussing M&S and SBA, it is important to have a common understanding of the

word “virtual.” In the most general sense, virtual “refers to the essence or effect of something, not the fact.”19 In practice, the word virtual is usually paired with another word, as in

virtual battlespace, virtual reality, and virtual

prototype. Virtual battlespace is defined as

“the illusion resulting from simulating the

actual battlespace.”20 Virtual reality is “the

effect created by generating an environment

that does not exist in the real world. Usually,

[virtual reality is] a stereoscopic display and

computer-generated three-dimensional environment which has the effect of immersing the

user in that environment. This is called the

immersion effect. The environment is interactive, allowing the participant to look and

navigate about the environment, enhancing

the immersion effect. Virtual environment and

virtual world are synonyms for virtual reality.”21 Synthetic environments, on the other

hand, are defined as “... simulations that represent activities at a high level of realism, from

simulations of theaters of war to factories and

manufacturing processes. These environments

may be created within a single computer or a

vast distributed network connected by local

and wide area networks and augmented by

super-realistic special effects and accurate

behavioral models. They allow visualization of

and immersion into the environment being

simulated.”22 Finally, virtual prototypes are “a

model or simulation of a system placed in a

synthetic environment, and are used to investigate and evaluate requirements, concepts,

system design, testing, production, and sustainment of the system throughout its life cycle.”23

Life Cycle Cost and Total Ownership Cost

The concepts of life cycle cost (LCC) and total

ownership cost (TOC) also figure prominently

when discussing SBA. The term TOC often

appears to be replacing the term LCC. TOC

is the totality of costs associated with the

Department of Defense, including the costs related to weapon systems, whereas LCC is the

totality of costs over time related to developing, acquiring, operating, supporting and

disposing of weapon systems.

24 The Secretary

of Defense, William S. Cohen, states in his

Annual Report to the President and the Congress
2-6

that “Total ownership cost is the sum of all

financial resources necessary to organize,

equip, and sustain military forces sufficient to

meet national goals in compliance with all

laws; all policies applicable to DoD; all

standards in effect for readiness, safety, and

quality of life; and all other official measures

of performance for DoD and its components.”25 When viewed this way, LCC inherently refers to that subset of TOC that has to

do with weapons systems.

In practice, however, TOC is often used in a

weapon system context, in which case TOC

and LCC appear to be synonymous terms, as

they both cover all costs to research, develop,

acquire, own, operate and dispose of a weapon

system for a specific (or assumed) number of

years. However, TOC as applied to weapon

systems seems to imply a greater effort to capture more of the costs covered by both definitions than has heretofore been practiced

routinely in DoD under the banner of LCC.26

For example, if a new system or concept will

require additional security forces as part of its

operation, then the life cycle cost of this support would most likely be factored in to the

TOC of the system or concept, but would probably not have been included under LCC. There

are many on-going debates on this subject,

which we don’t intend to resolve here. In this

book we use the more-encompassing term TOC.

SBA Vision Statement

For many years, the training community has

leveraged the strengths of M&S to augment

live training. As noted by General Richard

Hawley, Commander of Air Combat Command, “...[M&S] has been a key part of our

training for many years, but...we’ve never fully

exploited the contributions that modeling and

simulation can make to our readiness

programs.”27 Similarly, although the acquisition community has also made M&S a key part

of the systems acquisition process for many

years, it too is beginning to realize that the

benefits of M&S haven’t been fully exploited.

In his 16 March 1998 Memorandum, the Honorable Jacques S. Gansler, Under Secretary

of Defense (Acquisition and Technology),

stated, “As we undergo changes in our defense

acquisition infrastructure, let me take this

opportunity to firmly state my commitment to

the use of M&S in the acquisition of our weapons systems.... Therefore, it is essential that

we plan for the use of M&S in our acquisition

strategies. I expect programs to make the

upfront investment in M&S application and

technology and will be looking for evidence

of that investment in program planning and

execution.”28 Dr. Gansler goes on to note additional steps he is endorsing for the SBA

initiative to capitalize on the current efforts

in M&S.

The DoD, in collaboration with industry,

developed a vision statement for SBA that

envisions an acquisition process in which DoD

and industry are enabled by robust, collaborative use of simulation technology that is

integrated across acquisition phases and

programs. The goals of SBA are to:

1. Substantially reduce the time, resources,

and risk associated with the entire acquisition process;

2. Increase the quality, military worth, and

supportability of fielded systems, while

reducing total ownership costs throughout

the total life cycle; and

3. Enable Integrated Product and Process

Development (IPPD) across the entire

acquisition life cycle.29
2-7

SBA Definition

This vision statement promotes discussions about SBA, and most people basically

agree with its intent. However, an expanded

definition of SBA is instructive:

Simulation Based Acquisition is an

iterative, integrated product and process approach to acquisition, using

modeling and simulation, that enables

the warfighting, resource allocation,

and acquisition communities to fulfill

the warfighter’s materiel needs, while

maintaining Cost As an Independent

Variable (CAIV) over the system’s

entire life cycle and within the DoD’s

system of systems.

A discussion of each of the definition’s critical

elements follows.

“Simulation Based Acquisition is an iterative,

integrated product and process approach to

acquisition”: SBA enables Integrated Product

and Process Development (IPPD) teams to

converge on optimal solutions by balancing

requirements through an iterative design process. DoD and contractor organizations work

internally and with each other as an integrated

team effort.30 IPPD used to mean that we put

a logistician on the design team to translate

his maintenance and support knowledge into

meaningful design guidelines. Now IPPD

means arming the hands-on experts with the

tools to rapidly see the results of their design

inputs, thereby making them a part of the

design team. For example, using three-dimensional “solid” models, the F-22 aircraft program enabled two mechanics to inject maintainability changes into the design tradeoffs,

because they were able to visualize the system

much earlier in the process.

“...through modeling and simulation”: M&S

activities make SBA possible. Stepping into

the synthetic environment enables exercising

the power of simulation. Within the same time

frame, many more analytical excursions can

be made with virtual designs than would be

possible with physical prototypes. The

increased level of user involvement, coupled

with compressed time and space feedback,

lead to better learning and problem solving.

This is far superior to the traditional approach,

which is driven by real experiences using physical prototypes.31 M&S facilitates the team and

increases communication, making team

members more effective.

“...the warfighting, resource allocation, and

acquisition communities”: SBA is more than

just acquisition. SBA helps to link the DoD’s

three principal decision support systems: the

Requirements Generation System, the Planning Programming and Budgeting System, and

the Acquisition Management System.32 Figure

2-4 shows the relationship of these three communities. The name of the community representing the Requirements Generation System

is changed to the more expansive term of

warfighting community, which includes: requirements personnel, operators, maintainers/

sustainers, and trainers. The resource allocation community is also an integral part of the

acquisition process, as it allocates the program

budgets (subject to approval by OSD and Congress), and has the burden of balancing the

Services’ and DoD’s budget(s). The acquisition community, as used here, includes both

government and industry agencies involved in

developing and fielding military systems.

“...to fulfill the warfighter’s materiel needs

while maintaining Cost As an Independent

Variable (CAIV)”: indicates the use of a strategy that balances mission needs with projected
2-8

out-year resources. We are now saying that we

will trade performance in order to achieve our

cost objective. By linking improved cost

models with our computer-aided engineering

tools, we’ll be able to better predict the costs

of different alternatives so as to make better

informed tradeoff analyses.

“...over the system’s entire life cycle” means

to look both within and across all phases of

the program, as early as possible during the

acquisition of the system. It encompasses the

collaborative use of simulation beyond the

traditional performance issues to address the

system’s entire life cycle cost issues during the

design, to include manufacturability, supportability, lethality, sustainability, mobility,

survivability, flexibility, interoperability,

reliability, and affordability.

“...and within the DoD’s system of systems.”

This signifies to fully explore the system’s

interaction within and impact upon the DoD’s

system of systems, to capture the desire for

effective total systems integration, as well as

the collaborative use of M&S across programs.

As the United States participates in greater

numbers of combined operations, this aspect

of the definition will have to be expanded to

include a “system of systems” look across allied

systems and programs as well.

Figure 2-5 graphically illustrates the three

dimensions that SBA attempts to integrate.

First, the vertical dimension is where M&S has

been traditionally applied within each program

phase, without much regard to reuse later in

the program. The second dimension SBA

attempts to integrate is the horizontal application of M&S across the phases of the

program. More than just sticking with and

Figure 2-4. Three Principal Interacting Communities

Acquisition

Warfighting Resource

Allocation
2-9

Programs

Across

Within Phase

Within Program and Across Phases

MS O

MS I

MS II

MS III

growing models over the life of a program, this

means addressing the entire life cycle’s issues

as early as possible during the design. The acquisition community must stay focused on the

real goal, which is to produce superior weapon

systems, not superior virtual prototypes. The

key to success is to get the design right before

building the system, after which, in effect, the

design becomes frozen. This horizontal application across phases of the program indicates

an attempt to explore the cost drivers across

the entire life cycle of the system, and when it

makes sense, to address those issues in the

design. For example, more reliable systems can

translate into better combat effectiveness, as

well as potential manpower decreases for

maintenance, decreased mobility footprint,

and reduced sparing levels. Acquiring good

cost figures for these types of operations and

support costs across the expected life of the

system will help determine how much should

be spent on reliability during system design.

After production, as we continue to refine our

system models with field data and operator

input, the simulations will help us decide if we

need to modify or build a new system, as well

as consider non-materiel solutions. Looking

at the furthest point in the life cycle, there may

even be disposal costs that could be mitigated

by changes in the design. While the importance

of looking at disposal costs may seem somewhat far-fetched, it is interesting to note that

the Army spends about $100 million a year

demilitarizing ammunition, of which $13

million alone is for stocks from World War I.33

If we don’t start looking at disposal costs during the design, we may find that in the future,

we won’t be able to afford to buy the nextgeneration system, because we couldn’t afford

to dispose of the current one.

Figure 2-5. Application of M&S Across Three Dimensions
2-10

Finally, the third dimension SBA attempts to

integrate is across programs. If we continue

to maximize each system alone, the overall system of systems will be less than optimal. Programs need to recognize the value of the

interaction of their system within the overall

system of systems. Individual programs cannot afford to build everything themselves; they

need to rely on each other for their similar

needs. There are few design tradeoff analyses

being conducted within the services between

their own programs, much less between

different services’ programs. Currently, the

Joint Requirements Oversight Council

(JROC) reviews all Mission Need Statements

(MNS) for joint applicability, and follows up

on major programs before milestone reviews.

To do these tradeoffs effectively, however, they

have to be made at a level much lower than

the JROC. The capability to do these tradeoff

analyses will begin to extend beyond the

traditional interface control procedure

method, which allows each individual system

to be optimized, but which results in the overall system of systems’ performance being suboptimized. Failure to look at the big picture

can also result in over-designing systems as

well as unnecessary duplication of effort, both

of which waste resources.

When we begin to look outside a single system, we see even fewer interactions of these

issues in the context between systems, as for

example the maintenance and logistics considerations between the next generation

amphibious assault vehicle and the ships that

will transport it. We see this happening even

today within large, integrated weapon systems

such as submarines and aircraft, where the

major sub-components (sonars, torpedoes,

missiles, and munitions, etc.) are designed and

developed independently and integrated later

after each has been built. The design interface is controlled by an interface control document (ICD), with little to no concurrent design

tradeoffs possible across the ICD.
2-11

ENDNOTES

1. DoD 5000.59-M, DoD Modeling and Simulation

(M&S) Glossary, December 1997.

2. Ibid.

3. Defense Modeling and Simulation Office

(DMSO), Verification Validation & Accreditation (VV&A) Recommended Practices Guide

(November 1996), Chapter 1 Overview, pp. 1-4.

4. DoD 5000.59-M, DoD Modeling and Simulation

(M&S) Glossary, December 1997.

5. DMSO, VV&A Recommended Practices Guide.

6. Ibid.

7. Ibid.

8. DoD 5000.59-M, DoD Modeling and Simulation

(M&S) Glossary, December 1997.

9. Aggregation definition from DoD 5000.59-M,

DoD Modeling and Simulation (M&S) Glossary,

December 1997.

10. L.K. Piplani, J.G. Mercer, and R.O. Roop, Systems Acquisition Manager’s Guide for the Use of

Models and Simulations, Report of the Defense

Systems Management College 1993-1994 Military

Research Fellows, DSMC Press, September 1994.

11. Piplani et al, ibid.

12. Ibid.

13. Ibid.

14. Ibid.

15. DoD 5000.59-M, DoD Modeling and Simulation

(M&S) Glossary, December 1997.

16. Ibid.

17. Ibid.

18. Ibid.

19. Ibid.

20. Ibid.

21. Ibid.

22. Ibid. The authors have removed the word

“internetted” from the beginning of this definition as it is not necessary to be connected to the

internet in order to have a synthetic environment.

23. Ibid.

24. Dan Fink; Deputy, Acquisition Logistics Integration; Deputy Chief of Naval Operations (Logistics), OPNAV N4; interview by authors 8 July

1998.

25. William S. Cohen, Secretary of Defense, Annual

Report to the President and the Congress, 1998.

26. Fink, ibid.

27. General Richard E. Hawley, Commander Air

Combat Command, Keynote Address at 19th

Interservice & Industry Training, Simulation and

Education Conference, 2 December 1997,

Orlando, FL.

28. Dr. Jacques S. Gansler, Under Secretary of

Defense (Acquisition & Technology), “M&S in

Defense Acquisition,” memorandum, 16 March

1998.

29. DoD Acquisition Council of the Executive Council on Modeling and Simulation, 5 December

1997.

30. DoD Guide to Integrated Product and Process

Development; Office of the Under Secretary of

Defense (Acquisition and Technology), February

5, 1996.
2-12

31. Stefan Thomke, “Simulation, Learning and R&D

Performance: Evidence from Automotive Development,” Research Policy 27 (1) (May 1998),

55-74.

32. CJCSI 3170.01, Requirements Generation

System (Formerly MOP 77)), 13 June 1997.

33. Kern, ibid.
3-1

3

BACKGROUND

Guidance

DoD 5000.1, “Defense Acquisition”

This directive encourages the use of M&S by

stating that “models and simulations shall be

used to reduce the time, resources, and risks

of the acquisition process and to increase the

quality of the systems being acquired. Representations of proposed systems (virtual prototypes) shall be embedded in realistic,

synthetic environments to support the various

phases of the acquisition process, from

requirements determination and initial

concept exploration to the manufacturing and

testing of new systems, and related training.”3

DoD 5000.2-R, “Mandatory Procedures for

MDAPs and MAIS Acquisition Programs”

This directive mandates that “accredited modeling and simulation shall be applied, as

appropriate, throughout the system life cycle

in support of the various acquisition activities:

requirements definition; program management; design and engineering; efficient test

planning; result prediction; and to supplement

actual test and evaluation; manufacturing; and

logistics support. [Program Managers] shall

integrate the use of modeling and simulation

There are numerous studies, papers and program examples that highlight the benefits

M&S can provide in technical risk reduction,

time savings, direct cost savings and/or cost

avoidance, and management risk reduction.

Since many of these documents are continually being refined (particularly in the dynamic

growth area of M&S), it is important to view

them first-hand for possible changes.1

 For

example, the Army recently coined the phrase

Simulation and Modeling for Acquisition,

Requirements and Training, or SMART, to

describe their initiative for SBA.2

This chapter presents current policy and guidance, previous studies and conclusions, and

some assumptions and trends upon which SBA

depends. Thus far, the only reference to Simulation Based Acquisition that appears in the

Defense Acquisition Deskbook is Dr.

Gansler’s 16 March 1998 memorandum previously mentioned in Chapter Two. We expect

this will soon change, however, as the acquisition community continues to implement this

“smarter” way of doing business. The following synopses are a quick walk-through of

recent information relevant to SBA.
3-2

within program planning activities, plan for life

cycle application, support, and reuse models

and simulations, and integrate modeling and

simulation across the functional disciplines.”4

Regarding test and evaluation, it further states

that “modeling and simulation shall be an

integral part of test and evaluation planning...Test and evaluation programs shall be

structured to integrate all developmental test

and evaluation (DT&E), operational test and

evaluation (OT&E), live-fire test and evaluation (LFT&E), and modeling and simulation

activities conducted by different agencies as

an efficient continuum. All such activities shall

be part of a strategy to provide information

regarding risk and risk mitigation, to provide

empirical data to validate models and simulations, to permit an assessment of the attainment

of technical performance specifications and system maturity, and to determine whether systems

are operationally effective, suitable, and

survivable for intended use.”

For operational test and evaluation, it states

that “The use of modeling and simulation shall

be considered during test planning. Whenever

possible, an operational assessment shall draw

upon test results with the actual system, or subsystem, or key components thereof, or with

operationally meaningful surrogates. When

actual testing is not possible to support an

operational assessment, such assessments may

rely upon computer modeling, simulations

(preferably with real operators in the loop),

or an analysis of information contained in key

program documents. However, as a condition

for proceeding beyond LRIP, initial operational test and evaluation shall not comprise

an operational assessment based exclusively

on computer modeling; simulation; or, an

analysis of system requirements, engineering

proposals, design specifications, or any other

information contained in program documents

(10 USC2399). The extent of modeling and

simulation usage in conjunction with operational and test evaluation shall be explained

in the Test and Evaluation Master Plan....”7

For Live Fire Test and Evaluation on other

than ACAT 1A programs, it states: “...Alternatively, in the case of a covered system (or

covered product improvement program for a

covered system), the USD(A&T) or the CAE

[Component Acquisition Executive] may

waive the application of the required survivability and lethality tests and instead allow testing of a system or program by firing munitions

likely to be encountered in combat at components, subsystems, and subassemblies, together

with performing design analyses, modeling and

simulation, and analysis of combat data in lieu

of testing the complete system configured for

combat.”8

Regarding the four key systems engineering

tasks of requirements analysis, functional

analysis/allocation, design synthesis and verification, and system analysis and control, it states

that “The verification of the design shall include a cost-effective combination of design

analysis, design modeling and simulation, and

demonstration and testing....”9

DoD Directive 5000.59, “DoD Modeling and

Simulation (M&S) Management”

The purpose of this directive is threefold. First,

it establishes DoD policy, assigns responsibilities, and prescribes procedures for the management of M&S. Second, it establishes the

DoD Executive Council for Modeling and

Simulations (EXCIMS). Third, it establishes

the Defense Modeling and Simulation Office

(DMSO).
3-3

It directs that it is DoD policy that “Investments shall promote the enhancements of

DoD M&S technologies in support of operational needs and the acquisition process;

develop common tools, methodologies, and

databases; and establish standards and protocols promoting the internetting data exchange,

open system architecture, and software reusability of M&S applications. Those standards

shall be consistent with current national, Federal, DoD-wide and, where practicable, international standards for open systems... The

DoD Components shall establish verification,

validation, and accreditation (VVA) policies

and procedures for M&S applications managed by the DoD Component. The ‘DoD

M&S Executive Agent’ shall establish VVA

procedures for that application.... M&S

applications used to support the major DoD

decision making organizations and processes

(such as the Defense Planning and Resources

Board; the Defense Acquisition Board; the

Joint Requirements Oversight Council; and

the DoD Planning, Programming, and Budgeting system)... shall be accredited for that

use by the DoD Component for its own forces

and capabilities. Each DoD Component shall

be the final authority for validating and accrediting representations of its own forces and

capabilities in joint and common use M&S.

Each Component shall be responsive to the

other Components to ensure that its forces and

capabilities are appropriately represented in

the development of joint and common use

M&S.”10

DoD 5000.59-P, “Modeling and Simulation

(M&S) Master Plan”

This plan establishes numerous M&S activities. First, it establishes the DoD vision for

DoD M&S and outlines a strategy for achieving future DoD M&S-based capabilities.

Second, it assigns M&S implementation responsibilities and provides guidelines for the

development, cooperation, and coordination

of DoD M&S efforts. Third, it establishes

DoD M&S objectives, identifies action, and,

where possible, assigns responsibilities for accomplishing them. Fourth, it provides a basis

for developing supporting plans and programs,

including the DoD Modeling and Simulation

Investment Plan (MSIP), and the DoD

Component’s M&S master and investment

plans. Fifth, it provides justification for resource allocations to M&S within DoD Component programming and budgeting processes

and fosters the integration of the defense and

civilian M&S bases into a unified national and

international base using common standards,

processes and methods.11

DoD Manual 5000.59-M, “DoD Modeling and

Simulation (M&S) Glossary”12

This manual prescribes a uniform glossary of

M&S terminology for use throughout the Department of Defense. In addition to the main

glossary of terms, it includes a list of M&Srelated abbreviations, acronyms, and initials

commonly used within the DoD.

DoD Instruction 5000.61, “DoD Modeling and

Simulation (M&S) Verification, Validation and

Accreditation (VV&A)”

This Instruction implements policy, assigns

responsibilities, and, prescribes procedures

for the VV&A of DoD M&S. It designates

the Defense Modeling and Simulation

Office (DMSO) as the DoD VV&A focal

point, to be the central source of DoD VV&A

data and information and to assist DoD and

non-DoD organizations, as requested, in resolving VV&A issues or obtaining information on DoD VV&A practices. It specifies that
3-4

information and data on VV&A activities will

be readily available through the DoD M&S

Resource Repository (MSRR) system including, as a minimum, DoD Component VV&A

policies and procedures, V&V results, and

accreditation documentation. This instruction

also specifies minimum documentation

requirements for verification and validation

information, and accreditation results.13

“Department of Defense Verification, Validation and Accreditation (VV&A) Recommended

Practices Guide”

This guide provides background and information on recommended principles, processes,

and techniques for use in DoD VV&A efforts

that support the analysis, acquisition, and

training communities.14

“Simulation, Test, and Evaluation Process

(STEP) Guidelines”

Provides a set of guidelines for the program

manager to refer to in implementing the

DoD’s October 3, 1995 direction to make the

Simulation, Test and Evaluation Process an

integral part of Test and Evaluation Master

Plans. STEP proposes a Model-Simulate-FixTest-Iterate approach, and is defined as “...an

iterative process that integrates simulation and

test for the purpose of interactively evaluating improving the design, performance, joint

military worth, survivability, suitability, and

effectiveness of systems to be acquired and

improving how those systems are used.” STEP

is described as “a test and evaluation answer

to the DoD challenges of implementing IPPD

and SBA.”15

Previous and On-Going Efforts

“Study on the Effectiveness of Modeling and

Simulation in the Weapon System Acquisition

Process,” Science Applications International

Corporation, October 1996

The purpose of this report was to cite documented contributions to the total acquisition process. It concluded that “There is

consistent evidence of M&S being used effectively in the acquisition process but not in an

integrated manner across programs or functions within the acquisition process. Substantial evidence has been collected from individual

success stories, though the benefits are not

readily quantifiable into a general standard. The

key is in focusing on the integration of M&S

applications, across acquisition programs and

throughout the process, not in exploring the

applications themselves....”16

In attempting to quantify metrics, it noted

“cost savings are especially difficult to quantify” and are often “more correctly classified

as ‘cost avoidance’ and are measures of significant additional work or results that were

obtained using M&S tools which would have

cost the reported ’savings’ if they had been

obtained by more traditional methods.” This

study grouped the challenges that preclude the

seamless use of M&S in the acquisition process into technical, cultural, and managerial

challenges, as noted below:

Technical Challenges:

• Interoperability of M&S Tools;

• Availability of Data Descriptions;

• Security/Sensitivity of Data;

• Physics-based M&S;

• Hardware and Software Limitations;

• Variable Resolution.
3-5

Cultural Challenges:

• Acquisition Processes;

• Incentives for M&S Use;

• M&S Workforce;

• Acceptance of M&S.

Managerial Challenges:

• OSD and Service Guidance;

• Ownership of Data;

• VV&A Requirements;

• Funding Process;

• Use of System Models.

This report defined SBA as a “term to

characterize the general approach of significantly increased use of M&S tools and the new

processes which they enable in a new, more

integrated approach to program development” (italics emphasis added). It further

characterized SBA in terms of a systems

engineering process by saying “The positive

results of simulation efforts in systems engineering have become evident in what we refer

to as Simulation Based Acquisition.”

Finally, Appendix C of the October 1996 study

contains a useful summary of previous studies

and recommendations dating from 1989 to 1995.

Common Operating Digital Environment

(CODE)

CODE, a DoD initiative related to SBA, is

being co-sponsored by the Assistant Deputy

Undersecretary of Defense for Logistics

Reinvention and Modernization (ADUSD/

LR&M), and the National Defense Industrial

Association (NDIA). A joint industry- and

government-working group is developing a

framework for a “large-grained interoperability” technical environment to enable digital

weapon system life cycle information management by 2002. The vision is for government

and industry partners to create, exchange, and

sustain information solely in a digital environment. The CODE will provide access to digital data across the virtual extended enterprise,

and the functionality to support business processes and decision making. The initial report

is cognizant of SBA and notes that the Joint

Technical Architecture and its subsystem High

Level Architecture need to be reviewed with

other stovepipe architectures to ensure an

optimum life cycle environment.17

Defense Systems Affordability Council

(DSAC)

In December 1997, the DSAC identified three

fundamental areas contributing to the

affordability of acquisition programs: 1) reduction of total ownership costs; 2) 50 percent

reduction of acquisition program cycle time

for new systems; and 3) realistic programming

and program stability (zero percent program

cost growth) enabled by a broad range of

potential improvements in requirements setting, funding management, and acquisition

practices.18 As noted in the previous chapter,

Simulation Based Acquisition contributes to

all three of these objectives.

Joint SBA Task Force

The Acquisition Council of the Executive

Council on Modeling and Simulation commissioned a six month effort to develop a road

map in which DoD and Industry are enabled

by robust, collaborative use of simulation technology that is integrated across acquisition

phases and programs. The Terms of Reference

for this effort called for six major items to be

included in the road map:

1. near- and long-term DoD actions needed

to accelerate the SBA concept;
3-6

2. industry actions needed to accelerate

SBA;

3. notional representations of systems architectures and a conceptual framework to

identify key requirements for seamless

data transfer and interaction;

4. opportunities for reuse and commonality

across programs;

5. primary ownership of each module in the

systems architecture; and

6. estimated government and industry

investments needed to implement the proposed road map, and methods to

determine the return on investment.19

The SBA Task Force is ongoing during the

publication of this research effort, and their

results will be presented at a conference in

November 1998.

Assumptions, Trends, and SBA Enablers

Brigadier General Robert Armbruster,

Deputy for Systems Acquisition at the U.S.

Army Aviation and Missile Command, notes

that three change factors now make SBA possible: technology, advocacy, and Life Cycle

Management.20

Technology has advanced to the point where

it is often less expensive to simulate processes,

components, and systems, than it is to physically create or run them. Computing resources

Figure 3-1. Growth in Computing Power

Reprinted from Research Policy, Volume 27, Thomke, Stefan. “Simulation, Learning and R&D Performance:

Evidence from Automotive Development,” pp. 55-74, 1998, with permission from Elsevier Science.

105

1955

1955

1960

1960

1965

1965

1970

1970

1975

1975

1980

1980

1985

1985

1990

1990

1995

1995

2000

2000

104

103

102

10

1

0.1 $30K

DEC PDP8

Intel 4004 IBM 704 Altair 8800

IBM 7090

DEC PDP1

(PDP6)

(DEC VAX 780)

TMC CM-1

Cray T3D

Cray T3E

$12M

(Structural Biology)

Intel iPSC

(Higgs Boson Mass)

(72-Hour Weather)

(Pharmaceutical Design)

(DEC PDP11)

Illiac IV

(IBM 360)

IBM Stretch

Apple II

IBM 7094

CDC 6600

Pentium PC

Cray X-MP

Cray Y-MP

Cyber 205

Cray C90

Cray T90

Cray 2

Cray I Pentium Pro PC

$5K

Apple WS

IBM PC

Sun WS

Macintosh PC

Functional/Affordable

IBM 360/90

CDC 7600

(2D Plasma)

(48 Hour Weather)

(3D Plasma)

(Airfoil Design)

Leading Edge

Year

Peak Speed 

(Millions of Operations Per Second)
3-7

that were worth $30K in 1960 only cost about

10 cents today.21 In 1965, Gordon Moore (who

co-founded Intel Corporation in 1968) made

the observation that computing power was

doubling every 18-24 months. Now known as

Moore’s Law, his theory has proved to be remarkably accurate: in 26 years, the number of

transistors on a chip has increased more than

3200 times, from 2300 on the 4004 processor

in 1971, to 7.5 million on the Pentium II processor.22 This exponential growth in computing power is shown graphically in Figure 3-1.23

In 1960, many companies began using simulation tools, but they were often inadequate.24

Recent increases in computing power and

simulation tools, however, have greatly

increased the capabilities of M&S efforts to a

program. While physical models and prototypes do not represent reality completely,

programs have grown accustomed to using

them and understand their limitations. Now,

programs are finding that in many instances

the computer models are more correct than

the physical models. The following example

highlights this point. The experiment, to be run

in both the physical and virtual, was to test the

ability of a ship’s compartment to withstand

an explosion. During the validation testing of

the ship’s model, the test engineers could not

get the computer simulation results to agree

with the actual physical test results. The test

design for the compartment called for a circumference weld, which had been correctly

included in the finite element computer

model. However, as the physical prototype had

been incorrectly spot welded, the results of the

physical test did not match the results from

the simulation. When the error was realized,

it was easier (and more cost-effective) to

change the computer model to a spot weld,

and re-run the simulation, at which time, the

results matched.25 Another example is from

the automotive industry. A simulation analysis showed that the vehicle skin would rip

below a certain thickness. Not believing the

simulation, the engineers overrode the analysis of the simulation and made the vehicle skin

thinner. During testing the skin ripped as the

simulation had predicted. It was at this point

that the engineers became believers in the

results of simulations.

This leads to the second change factor of

“advocacy.” Leadership is essential to effective change in an organization and today

change within an organization must be led and

not just managed.26 Within OSD and the Services senior leadership within the acquisition

community is taking an active role towards the

implementation of SBA. Advocacy for SBA,

however, is not solely limited to senior leadership but can be found at all levels of the

acquisition community. At the Electronic

Proving Ground at Fort Huachuca, Arizona,

they are finding that simulations can be better than physical tests. Security issues and

civilian communication interference concerns

limit what can be physically tested on some

systems. Testing in the virtual environment

removes such barriers. Thus some simulations

can be more realistic than what can actually

be tested in the physical.27 Success stories, such

as this and the examples in the preceding paragraph, are beginning to make believers out of

the most hardened skeptics and are helping

to promote this technology.

And the third change factor contributing to

make SBA possible is the desire to be able to

perform Life Cycle Management—simulation

enables us to do LCM.28 Simulation makes it

possible to really get cost on the table and on

equal footing with the performance requirements. This forces the requirements community to make the hard tradeoffs as part of the
3-8

design process. As noted by BG Bergantz, Program Manager for the Army’s Comanchehelicopter program, SBA enables us to “put

the intellectual before the physical.”29
3-9

ENDNOTES

1. The DoD Acquisition Deskbook has a good synopsis of M&S guidance in Information Structure,

Topics, Section 2.9.1 M&S in Acquisition: The

Process, General Information, and Background

(31 March 1998).

2. Kern and Purdy, “Simulation Based Acquisition:

A Good Thing, But How Do We Get There?”

Army RD&A Magazine, July-August 1998, p. 4.

3. DoD 5000.1, Defense Acquisition, 15 March 1996,

(Section D Policy, Paragraph 2.f. Modeling and

Simulation) 23 March 1998.

4. DoD 5000.2-R, Mandatory Procedures for Major

Defense Acquisition Programs (MDAPS) and

Major Automated Information System (MAIS)

Acquisition Programs, 23 March 1998 (Paragraph

3.4.4, Modeling and Simulation).

5. DoD 5000.2-R, Mandatory Procedures for MDAPS

and MAIS Acquisition Programs, 23 March 1998

(Paragraph 3.4.1, Test and Evaluation Strategy).

6. DoD 5000.2-R, Mandatory Procedures for MDAPS

and MAIS Acquisition Programs, 23 March 1998

(Paragraph 3.4, Test and Evaluation).

7. DoD 5000.2-R, Mandatory Procedures for MDAPS

and MAIS Acquisition Programs, 23 March 1998

(Paragraph 3.4.5, Operational Test and

Evaluation).

8. DoD 5000.2-R, Mandatory Procedures for MDAPS

and MAIS Acquisition Programs, 23 March 1998

(Paragraph 3.4.9, Live Fire Test and Evaluation)

(Not applicable to ACAT IA programs).

9. DoD 5000.2-R, Mandatory Procedures for MDAPS

and MAIS Acquisition Programs, 23 March 1998

(Paragraph 4.3, Systems Engineering).

10. DoD Directive 5000.59, Modeling and Simulation

(M&S) Management, Section D, Policy, 4 January

1994.

11. DoD 5000.50-P, DoD M&S Master Plan (MSMP),

October 1995.

12. DoD 5000.59-M, DoD Modeling and Simulation

(M&S) Glossary, December 1997.

13. DoDI 5000.61, DoD Modeling and Simulation

(M&S) Verification, Validation and Accreditation

(VV&A), 29 April 1996.

14. DMSO, VV & A Recommended Practices Guide

(November 1996), Chapter 1 Overview.

15. Dr. Philip Coyle and Dr. Patricia Sanders, Simulation, Test, and Evaluation Process (STEP) Guidelines, 4 December 1997.

16. Anne Patenaude, et al., Science Applications

International Corporation, Study on the Effectiveness of Modeling and Simulation in the Weapon

System Acquisition Process, Final Report, October 1996.

17. Common Operating Digital Environment

(CODE) Top Level Policy Guidance Document,

http://www.acq.osd.mil/cals/ accessed on 1 July

98.

18. Joint Simulation Based Acquisition Task Force

Terms of Reference, April 1998

19. Ibid.

20. Brigadier General Robert E. Armbruster, Deputy

for Systems Acquisition, U.S. Army Aviation and

Missile Command, remarks during presentation

at Army SBA Conference, Orlando, FL, 21-22

January 1998.

21. David Chang; Director, Vehicle Synthesis, Analysis & Simulation Process Center; General Motors, interview by authors, 31 March 1998.

22. www.intel.com/intel/museum/25anniv/Hof/

moore.htm accessed on 5 May 1998.
3-10

23. Stefan Thomke, “Simulation, Learning and R&D

Performance: Evidence from Automotive Development,” Research Policy 27 (1) (May 1998), 55-

74.

24. Michael Carty, Gene Coffman, and Hwa-Sung

Na, Ford Motor Company, interview by authors,

27 February 1998.

25. Myles Hurwitz, Naval Surface Warfare Center,

Carderock Division, interview by authors, 18 February 1998.

26. Lieutenant Colonel Charles L. Beck, Jr., USAF,

et al, A Model for Leading Change: Making Acquisition Reform Work, Report of the 1996-1997

DSMC Military Research Fellows.

27. John Haug, Directorate for Technical Mission,

U.S. Army Test and Evaluation Command, interview by authors, 11 February 1998.

28. Armbruster, ibid.

29. Brigadier General Joseph L. Bergantz, Program

Manager RAH-66 Comanche helicopter program, remarks during presentation at Army SBA

Conference, Orlando, FL, 21-22 January 1998.
4-1

4

ESSENTIAL ASPECTS OF SBA

Balancing Requirements

Early user involvement is essential in defining, refining, and balancing requirements. The

following quotations emphasize this point:

• “The best way to define a program is

through M&S—plunk the user down into

the virtual world, have him play with the

concept, and work out the tactics, training,

and procedures—take the user out of the

abstract and into the virtual combat

world.”1

• “SBA and M&S enable you to play optometrist with the user—do you like it this way,

or that?”2

• “We need more user input during development of the system. Simulation brings

the warfighters and engineers together so

that each has a better appreciation of the

future system and its interaction in the

future battlefield.”3

• “Spiral development concentrates on soldier feedback. It involves the soldier who

uses the equipment directly in the development process, brings all the parties

together—from the user to the developer

to industry.”4

This chapter presents five essential features

of a Simulation Based Acquisition process

which, when implemented, will present the acquisition community with capabilities not currently imbedded in the current process. First,

SBA will, through early and continuous

involvement, allow the user to define, refine,

and balance requirements. This is the critical

first step in producing better, faster, and

cheaper material solutions. Second, an SBA

process supported by the synthetic environment, allows design teams to concurrently explore greater numbers of possible material

solutions than is possible within the current

acquisition process. Third, the iterative nature

of the SBA design process will enable IPPD

teams to converge systematically on optimal

solutions more efficiently than is currently

possible. Fourth, SBA will support changing

the role of testing into that of being an integral part of the design process. Finally, SBA

supports informed tradeoff analyses through

a Decision Risk Analysis process. This chapter will explore each aspect separately. Though

presented individually, when the five essential aspects of SBA are combined, their synergistic effect towards assisting the acquisition

process is greater than the sum of the

individual parts.
4-2

• “The warfighter only has a general notion

of his requirements up front, and is unable

to give a detailed description early on. M&S

enables the user and the developer to

walk up the spiral development ladder

together.”5

The traditional acquisition process assumes

the user completely understands the requirements and the impact each has on the program. To be more specific, it has been assumed

that the user completely understands the overall cost associated with each requirement.

According to Lieutenant General George

Muellner, Air Force Principal Deputy for

Acquisition, former Director of Requirements

for Air Combat Command, and former Director of the Joint Advanced Strike Technology

Program: “The old way of doing business was

for the user to throw his requirement over the

fence to the acquisition community, who

would then spend five years working the margins, when 90% of the performance was

already locked up. Now we’re getting where

we can keep the trade space open, don’t lock

up the requirements, and force the requirements people to deal early on with these

issues.”6

Balancing requirements in the defense acquisition business means bringing into balance

what the user wants with what can be

affordably accomplished, or as stated by Lt.

Gen. Muellner, converting the warfighter’s

wants into affordable needs.7

 According to Col

Cuff, U.S. Army Training and Doctrine Command (TRADOC) Systems Manager (TSM)

for the Crusader field artillery program: “With

simulation, the user has a better understanding of the cause and effect relationship between requirements, cost, and schedule. The

TSM gets to see the operational impact of

design and requirements tradeoffs. The user

always wants everything, but previously did not

have an appreciation for the cost and schedule impacts that requirements had on program

execution.”8

 An SBA process supports the user

through visual representation of the various

design alternatives early in the design phase.

The user can then compare alternatives and

provide input to the design team on the value

of competing designs with relationship to his

requirements. Visualization also makes identification and resolution of many issues easier

and faster. The entire IPPD team can see

where the issues are and then focus on how to

solve the problem.

System requirements are really a combination

of many things: the user’s operational needs,

logistics concerns, lessons learned, environmental issues, etc. These system requirements

must then be balanced in terms of cost and

force effectiveness. The user plays a key role

in balancing requirements, and the applied use

of M&S across the phases of a program can

significantly affect the outcome of the final

system.

Figure 4-1 shows the JSF program’s depiction

of this balancing act between cost and capability in determining the most affordable

solution. The JSF sets cost objectives to balance mission needs with projected out-year

resources, taking into account anticipated

process improvements in both DoD and the

defense industries.9

Numerous Design Alternatives

Concurrent systems engineering can be viewed

as a process of balancing requirements, or

making compromises between competing

requirements. Its primary function is to ensure

the product meets the customer’s cost, schedule, and performance needs encompassing the
4-3

total product life cycle.10 This systems engineering approach is not new; what is new is

the ability to conduct many parts of this process in the synthetic environment using M&S.

The benefit of doing this process in the synthetic environment is that designers and developers can try things without fear of failure.

A recent report on virtual prototyping noted

that during the design cycle, a hesitancy for

revisions exists once a marginally working system has been achieved, as a result of the increased risk associated with making changes.11

This increased risk is a result of the time lost

and the associated cost of building alternative

hardware, even though the alternative may be

a superior design.

Because making design changes in a synthetic

environment can be faster and cheaper (and

therefore less risky) than making changes to a

physical prototype, an IPPD team is able to

explore more design options. Harvard

University’s Stefan Thomke, in his research

with the automobile industry, cites a powerful

example. An automotive company found that

an assumption regarding a component

involved in side-impact crashes was in fact

false. This assumption had never been tested

because it seemed obvious that making a component in a car stronger would improve crashworthiness. Physical prototypes and crashes

were reserved for high payoff issues. Engineers

were not willing to use costly physical testing

to test assumptions they were confident about.

Because it was quick and inexpensive to check

out the assumption using simulation, one

engineer insisted on it. Surprisingly, the team

discovered that strengthening the component

actually decreased crashworthiness, by unexpectedly propagating the load to another part

of the vehicle. The solution was to reduce the

stiffness of the component, which went against

the conventional wisdom. This discovery led

to a reevaluation of other reinforced areas and

has improved the crashworthiness of all cars

under development. These and other design

changes improved crashworthiness by 30 percent—a significant increase. Interestingly, the

Figure 4-1. Affordability: Capability vs. Cost production

O&S

Lethality

Survivability

Supportability

Capability

Affordability

Cost R&D
4-4

time and cost to build physical vehicle prototypes for the two verification experiments

at the completion of the project exceeded

the time and cost of the entire advanced

development M&S project.12

Once a physical prototype exists it is difficult

for people to envision a new concept that is

substantially different. It is very compelling for

the design team to get to a point design, and

then consider possible excursions or engineering changes from that point design, rather than

examining the entire design space. Designing

in the synthetic environment, however,

decreases the cost and time of looking at

alternative designs.

Thomke notes that the advantages of substituting real physical objects with virtual experimentation can be very substantial— once

set up, virtual tests can be run at very little

additional cost per run.13 In addition, he

notes several other benefits of virtual

experimentation. These include:

• Better depth and quality of the analyses

because it is possible to slow the test events

down and zoom in on minute areas.

• Information from problem-solving cycles

becomes available to other development

tasks earlier in the development process,

therefore other tasks can proceed more

quickly, and higher degrees of design

concurrence are feasible.

• With faster problem-solving cycles, designers will be able to push (or “front-load”)

problem-discovery to earlier development

phases and thus make problem-related

engineering changes earlier and at a lower

cost.

Figure 4-2. SBA Process

QFD ANALYSIS PHYSICAL TEST

DELPHI

PROCESS Mission 

Engagement

Engineering

Campaign

CONSTRUCTIVE

SIMULATION

Simulation Based Acquisition Process

VIRTUAL

ENVIRONMENT

INTERACTIVE

DIGITAL

SIMULATION

DISTRIBUTED INTERACTIVE ENVIRONMENT
4-5

• Reduced cost and time to get experimental feedback encourages the design

team to conduct more diverse “what if”

experiments which, in turn, may lead to

the discovery of more effective (and

unanticipated) design changes.14

The Iterative Design Process

The Joint Strike Fighter (JSF) program portrays the process of iteratively designing a

system using the SBA process as shown in

Figure 4-2.15 The JSF process is a building

block approach, with each step building upon

the previous steps as well as iterating within

the steps and between steps. It begins with the

Delphi Process, which is a method devised by

the RAND Corporation to obtain expert judgment from multiple experts without many of

the biases associated with interpersonal pressures.16 These results are then fed into a Quality Functional Deployment (QFD) analysis,

which refines the requirements into ways to

accomplish those requirements. QFD provides

a way of tracking and tracing tradeoff analyses through various levels, from requirements

through design decisions to production and

support processes, while preserving traceability to user needs.17 These results are then used

to build the appropriate level of constructive

simulation. At first, these simulations will be

at an aggregated campaign level, but will continue to be iterated and refined back through

the previous two steps and down the constructive simulation hierarchy to mission/battle,

engagement, and finally the engineering level.

As more is learned, the simulation

progresses to interactive digital simulation

and interaction within a virtual environment, which introduces the human-in-theloop. Again, there can be much iteration

within each step, as well as feedback to earlier steps to capture the knowledge gained.

As the program matures into flight test,

these results are also fed back into the program to capture the knowledge, and to help

challenge and possibly revise the inputs to the

process.

Once a program reaches the constructive

simulation step, it has the option to begin

interacting with other systems in a distributed

interactive environment. This can be a powerful tool for the IPPD team because it enables

them to receive feedback on their system well

before the design is locked in. Virtual systems

can participate in exercises and experiments

to determine their effect on the battlefield, and

those results and warfighter feedback are captured early enough in the program to help the

design team conduct more informed tradeoff

analyses. In effect, the warfighter becomes an

integral part of the design team.

Converging on an Optimal Design

Another view of this systems engineering and

simulation process was developed by United

Defense Limited Partnership and is being used

on the Crusader field artillery system. Their

Simulate, Emulate, Stimulate process is shown

in Figure 4-3 as a top-down view, and in Figure

4-4 from a side view.18

The Simulate, Emulate, Stimulate process

starts with a low fidelity model, and moves

through the four steps of analyze, design,

evaluate, and revise, successively converging

until reaching a high fidelity model of the final

system. Early models of the system are used

in a series of simulations to evaluate concepts

of potentially high-risk areas to identify and

resolve deficiencies. As the system design

matures, some subsystem simulations are

replaced with emulations, which are models

that accept the same inputs and produce the
4-6

Simulate - Emulate - Stimulate Process

Stimulate Stimulate

Emulate Emulate

Simulate

High Fidelity

Low Fidelity w Fidelity

Figure 4-3. Stimulate-Emulate-Stimulate Process, Top-Down View

Figure 4-4. Stimulate-Emulate-Stimulate Process, Side View

High Fidelity

Simulate

Stimulate

Emulate

Low Fidelity
4-7

same outputs as a given system.19 These emulations have a higher level of fidelity and translate to a more detailed evaluation of the system performance. Further design maturation

allows the model to stimulate actual system

hardware, leading to the final stage of bench

top integration and then system test.20

SBA is about using simulation to explore the

design space, to validate designs, and to verify

that the proposed design will meet the enduser’s expectations and is manufacturable,

supportable, and affordable. Simulation

enables the team to see the results of their

design decisions not only in terms of performance, but also to project the cost of those

decisions across the system’s life cycle. The

rapid feedback from the design iterations helps

the team converge on an optimal solution—

otherwise they’ll continue to work on the problem until they run out of time. It’s impossible

to know if a single design is the best possible

solution; however it is possible to evaluate one

design against another. The confidence in

reaching an optimal design should increase as

the number of designs explored increases, particularly if there is significant feedback and

learning taking place between the design

iterations.

Data and personnel interfaces need to be

seamless and integrated to facilitate rapid

design iterations. This requires an integrated

data environment to work effectively. A Product Information Management (PIM) tool is

extremely important to manage across the virtual enterprise and can facilitate widely separated teams. It controls and provides easy

access to the large quantities of engineering

and product-related data that are generated

during concurrent engineering, while tracking

the numerous rapid changes from different

sources in the organization that often occur

at the same time.21 Concurrent engineering

demands a great deal of work-in-process

information. The PIM links the engineering

tools with the other tools necessary to manage

a fully integrated environment, such as office

automation, systems engineering, business

management, configuration management,

reference library, and particularly web and

Internet connectivity for cost-effective

subcontractor access.

The integrated data environment is changing

the entire design process. For example, on the

Crusader program, critical information no

longer exists in specifications and drawings.

It’s now in very large databases that need to

be managed and linked. Requirements for the

Crusader now exist as a highly structured,

hierarchical, linked data set of over 7,000

requirements. Using their integrated data

environment management system, the Crusader design team can quickly identify the collateral impact of a requirements change.22 The

team can trace the top-level warfighting tasks

all the way down to the detailed design and

back up to the final product. During design

iterations, if the final product falls short of

expectations in some areas (for example in

initial acquisition cost, interoperability, or

maintainability), this traceability from requirements to design will be key to enabling the

program office, in conjunction with the

warfighter, to make smart tradeoffs.

Test as an Integral Part of Design

The traditional “build – test – fix” or “test –

fix – test” process is giving way to a new process in which test is becoming an integral part

of design. As mentioned before in Chapter

Three, the Simulation, Test, and Evaluation

Process (STEP) Guidelines calls it a “model –

simulate – fix – test – iterate” approach,
4-8

wherein there are many iterative loops

possible in this process.24

Dr. Henry Dubin, Technical Director of the

Army’s Operational Test and Evaluation Command (OPTEC), emphasizes this changing

process by saying that we no longer “conduct

test and evaluation for the purpose of determining pass or fail of critical threshold

parameters...[we now] focus testing on what

the acquisition team needs to learn—test to

learn, and evaluate to understand the [operational] impact of what we have learned on

mission accomplishment.”25

When viewing the test process, it’s important

to keep in mind that both physical tests and

virtual tests have limitations. Dr. John Foulkes,

Director of the Army’s Test and Evaluation

Management Agency, notes that the “test and

evaluation community should view anything

short of actual war as being simulation, including live testing. Program Managers should

ensure the right mix of constructive (digital)

simulation, virtual simulation, and live simulation (testing), and continually re-evaluate

why and how we test. Constructive and virtual

simulation should reduce the burden on the

amount of live simulation required, thereby

reducing the time and cost of testing.”26

Models, by definition, do not represent reality completely.27 This applies to both physical

and synthetic models. Models are, therefore,

an attempt at replicating reality. Many reasons

may prevent a model from accurately representing reality or being a high fidelity model.

Inaccuracy of the model can be the result of

the inability to capture all the attributes of the

real situation, possibly because of human

error, ignorance, time limits, or economics.28

Many times we use M&S to overcome physical limitations of live testing, including safety

concerns, interoperability issues, interference

with civilian activities, non-availability of the

threat, and affordability.29 In part M&S is done

to reduce the cost of representing aspects of

the real that are irrelevant to the experiment,

as well as to control out some aspects of the

real to simplify the analysis of the results.30

Dr. Dubin proposes a test strategy of using

M&S to design and develop the system, to

mature the models with the system, and to use

the test data for calibrating and maturing the

system model. By truly integrating testing early

on we should gain synergies to greatly enhance

the overall value of the program. He emphasized his point by stating the converse: “Building a simulation solely for the purpose of

reducing your Milestone III testing requirements is almost always a waste of money.”31 Dr.

Phil Coyle, Director of Operational Test and

Evaluation, Office of the Secretary of Defense,

noted the same importance of early test involvement: “M&S and testing are intertwined; when

they are not, neither is effective.”32

The benefits of M&S to the test and evaluation community are numerous, among them

the ability to:

• conduct full and continuous evaluation of

the system;

• evaluate system performance where live

simulation is neither feasible or practical;

• identify and concentrate the live simulation

resources in the high risk areas;

• stress the system at less than system level;

• conduct excursions for the development of

test and evaluation plans and to identify live

simulation scenarios.33
4-9

Involving the tester early in a program may

also allow the tester to influence the design to

minimize what must be tested and to facilitate the testing that must be conducted. The

knowledge gained through M&S of a system

will enable the test community to design and

conduct more effective and efficient testing.

The Crusader program noted that the biggest

challenge is to combine testing and M&S to

leverage savings in time and resources, while

satisfying the decisionmakers that the requirements are met.34 The test community needs to

be on board early as part of the design solution.

This way they gain credibility in the models as

they’re being used early in the program, and

build in confidence in the models through

incremental testing.

The intent of modeling and testing is also

beginning to change. For example, the role of

the Navy’s David Taylor Model Basin has

become more of validating models rather than

designing models. The purpose of testing is

evolving into validating the model so that the

analytical calculations derived from the model

are believable.

A mix of analytical and physical testing is often

the right answer, depending upon the credibility and confidence in the tools. New program starts may require physical prototypes

to prove out the models, since there may not

be known models from which to build. But the

goal remains the same—to conduct a combination of M&S and testing in order to get a

better product. As one tester stated: “The old

paradigm was to test a company of tanks. The

new paradigm will be to test a platoon, but to

simulate a battalion.”35

Decision Risk Analysis

Though modeling and simulation costs are

coming down, they still can be very expensive

to apply. Programs probably cannot afford to

use M&S on every issue and need the ability

to prioritize their efforts. One way to prioritize efforts is through a methodology called

Decision Risk Analysis (DRA), which quantifies and ties together cost, schedule, performance, producibility, risk, and quality to

permit informed tradeoff analyses.36 A decision risk analysis tool quantitatively provides

the program manager with a means of assessing a program’s probability of achieving

program success, while employing a CAIV

strategy. If the assessed risk is low and well

known, combinations of many program activities may be acceptable. If assessed risk is substantial and/or unknown, a detailed break out

of activities accompanied with detailed time,

cost, and performance estimates for completion of those activities from assessment in a

Delphi-type environment may be required.

One such tool is the Venture Evaluation and

Review Technique (VERT), which is a government-tool designed to do DRA.37 It is a Monte

Carlo simulation-based tool that aids program

managers in measuring risk in an unbiased

manner. VERT uses the same approach as a

Gannt chart and it adds probability to make it

a risk measurement tool. To build a more

accurate Gannt chart you interview IPT members to get a more accurate assessment of how

long activities will take. You end up with a

range of time an event could take, along with

a corresponding probability. You can then run

a series of “what ifs” to determine various outcomes as parameters are changed. The Gannt

chart will allow you to identify the program’s

critical path for cost, schedule, and performance. You can then focus simulation in those
4-10

high-risk areas to reduce programmatic risk.

If the risk is low, no modeling may be required,

or a low fidelity, kinematics-based CAD/CAM

model may be all that is required for proof of

principle. On the other hand if the risk is

high, a higher fidelity simulation or physical test may be required for proof of concept.

Implementing a program like VERT can take

a lot of manpower to collect and maintain the

data; however, it can pay big dividends by helping determine the high risk areas. A DRA

approach helps the manager determine where

to employ modeling and simulation efforts and

to what level of fidelity.
4-11

ENDNOTES

11. Tim Thornton, Coleman Research Company

(CRC), Collaborative Virtual Prototyping System

(CVPS), 9 January 1997.

12. Stefan Thomke, “Simulation, Learning and R&D

Performance: Evidence from Automotive

Development,” Research Policy 27 (1) (May

1998), 55-74.

13. Ibid.

14. Ibid.

15. Colonel Phil Faye, USAF, Joint Strike Fighter

Program briefing, 30 March 1998.

16. For more information on Delphi, see Defense

Acquisition Deskbook, Information Structure

1.2.4 Establish Risk Management Strategy,

AFMC—Identify the Ranking of Risk, Submitted by Major Paul Loughnane, USAF, AFMC/

ENPI (December 1997)

17. For more information on QFD, see the DoD

Guide to Integrated Product and Process Development, Office of the Undersecretary of Defense

(Acquisition and Technology), 5 February 1996.

18. Bruce Van Wyk, United Defense Limited Partnership, Crusader Program briefing, 2 April 1998.

19. DoD 5000.59-M, DoD Modeling and Simulation

(M&S) Glossary, December 1997.

20. Bruce Van Wyk, United Defense Limited Partnership, Crusader Program briefing, 2 April 1998.

21. John Krouse, “Manager’s Guide to Computer

Aided Engineering,” pp. 156-157.

22. Bruce Van Wyk, United Defense Limited Partnership, Crusader Program briefing, 2 April 1998.

1. Will Brooks, Chief, Simulation Branch, Army

Materiel Systems Analysis Activity, interview by

authors, 11 February 1998.

2. Colonel Lynn Carroll, USAF, Warfighter Training Research Division Chief, Air Force Research

Laboratory, briefing to the Joint Strike Fighter

Mission Level Strategy session, 30 March 1998.

3. Rob Beadling, John J. McMullen Associates, Inc.,

interview by authors, 6 March 1998.

4. LTG Kern, USA, “SBA: Fielding a Capability

Based Force,” presentation and remarks at Army

SBA Conference, Orlando, FL, 21-22 January

1998.

5. Colonel Picanso, USAF, Electronic Systems Center, comments during presentation at Joint Strike

Fighter Mission Level Strategy session, 30 March

1998.

6. LTG George Muellner, USAF, Air Force Principal Deputy for Acquisition, interview by authors,

30 March 1998.

7. Colonel Kenneth Konwin, USAF, Director of

Defense Modeling and Simulation Office,

interview by authors, 31 July 1998.

8. Colonel Michael Cuff, U.S. Army, TRADOC

Systems Manager for Cannon Systems, interview

by authors, 2 April 1998.

9. Major General Leslie J. Kenne, Program Manager, Joint Strike Fighter Program, “Joint Strike

Fighter—The Affordable Solution” 30 April 1998

briefing, on-line reference www.jast.mil/html/

progbriefs.htm on 22 April 1998

10. Defense Acquisition Deskbook, Information

Structure 2.6.1 Systems Engineering, Frontline

Wisdom and Advise, AFMC - The Engineering

Process From Industry Point of View, Submitted

by Mr. Kammer, HQ AFMC/ENPP (09 June

1998).
4-12

23. Dr. John B. Foulkes; Director, U.S. Army Test

and Evaluation Management Agency; Presentation and remarks at Army SBA Conference, 21-

22 January 1998.

24. Dr. Philip E. Coyle and Dr. Patricia Sanders,

Simulation, Test, and Evaluation Process: STEP

Guidelines, 4 December 1997, pp. 1-3.

25. Dr. Hank Dubin; Technical Director, U.S. Army

Operational Test and Evaluation Command; Presentation and remarks at Army SBA Conference,

21-22 January 1998.

26. Foulkes, ibid.

27. Thomke, ibid.

28. Ibid.

29. Dubin, ibid.

30. Thomke, ibid.

31. Dubin, ibid.

32. Dr. Philip E. Coyle; Director, Operational Test

& Evaluation, Office of the Secretary of Defense;

Presentation to 1997 Defense Modeling and

Simulation Office Industry Days, 22 May 1997.

33. Foulkes, ibid.

34. Colonel William Sheaves, U.S. Army; Project

Manager, Crusader field artillery program; Presentation and remarks at Army SBA Conference

(21-22 January 1998).

35. John Haug; Director, Technical Mission, Test and

Evaluation Command, U.S. Army, interview by

authors, 11 February 1998.

36. Dr. Stuart W. Olson, STRICOM, AMSTI-E,

Briefing to MG John E. Longhouser, Commander, US Army Test and Evaluation Command and BG John P. Geis, Commander, US

Army Simulation, Training and Instrumentation

Command, 7 February 1997.

37. The U.S. Army Logistics Management College,

Fort Lee, VA teach courses on VERT. For more

information see on-line http://www.almc.army.mil/
5-1

5

EXPANDING

THE SBA ENVELOPE

destination. How much SBA can be done in a

program largely depends on three factors:

1. the domain the program is in;

2. how well developed the infrastructure is;

and

3. how compelling a need there is to change

the current way of doing business.

Regarding the first factor, how much SBA can

be done will partly depend on the maturity of

the program’s domain. The DoD M&S Glossary defines a domain as “The physical or abstract space in which entities and processes

operate. A domain can be land, sea, air, space,

undersea, a combination of any of the above,

or an abstract domain, such as an n-dimensional mathematics space, or economic or psychological domains.”1

 If the domain is fairly

mature with well-developed models, then a

program has a much better chance of leveraging others’ investments to their benefit. On the

other hand, a program’s knowledgeable

industry partners may still be using blueprints

and fax machines. These programs will probably have to set their sights a little lower in

how far they will be able to reap the benefits

In this chapter we describe how Simulation

Based Acquisition practices within the Department of Defense may be expanded. To set the

stage, we present the factors influencing the

extent to which a program can implement

SBA. Next, we present the areas that programs

can “push” while implementating SBA activities within their program. Finally, we discuss

the “pull” programs will receive from the

warfighting and programming communities

once they and other programs begin

implementing SBA .

Factors Influencing the Use of SBA –

Domain, Infrastructure, and Need

Many programs are doing smart things with

M&S, and many of these programs say they

are already implementing SBA. Others

counter that those programs are only beginning to scratch the surface of what is possible using a simulation-based approach to

acquisition. They are just gaining benefits

from having a common description of the

product, so they cannot claim to be implementing SBA. SBA is a new approach that

has a sliding scale—programs can do a little,

or they can do a lot. As someone said at a

recent conference, SBA is a journey, not a
5-2

of moving to an acquisition strategy based

upon the heavy use of M&S.

The automotive and aerospace industries are

two of the most advanced domains because of

their lengthy involvement with M&S, and have

had good success in developing their enterprises by integrating with their suppliers. (An

enterprise is defined as “an arbitrarily-defined

functional and administrative entity that exists

to perform a specific, integrated set of missions

and achieve associated goals and objectives,

encompassing all of the primary functions

necessary to perform those missions.”2

) Conversely, the U.S. shipbuilding industry has many

domestic suppliers that do not have any expertise in the synthetic environment, and are

unable, therefore, to provide CAD drawings

(much less digital models) of their products.

This impedes the benefits the prime contractors in the shipbuilding industry can accrue by

using M&S, since they have the additional

expense of reverse engineering the models of

their suppliers’ products, with all of the attendant

worries about the validity of the models.

It’s important to note that in spite of this limitation, the shipbuilding industry has gained

significant benefits from using M&S. For

example, General Dynamics Electric Boat

Corporation was able to reduce the number

of pipe hangars on the New Attack Submarine (NSSN) from 40,000 to about 18,000.

Since hangars are a major cost item on a submarine, this equates to a major cost savings

for the program. General Dynamics more than

recouped their implementation costs because

the cost reduction on the first boat alone

exceeded the entire cost of modeling it, and

they are building three more.3

The second factor influencing the transition

to SBA will be how much of the infrastructure

is in place to facilitate a program’s entry into

a simulation-based acquisition process. The

Defense Modeling and Simulation Office

(DMSO) is building many parts of this infrastructure. For example, DMSO has created the

Modeling and Simulation Resource Repository (MSRR), which is a collection of models,

simulations, object models, Conceptual

Models of the Mission Space (CMMS), algorithms, instance databases, data sets, data standardization and administration products,

documents, tools, and utilities. The MSRR is

a collection of resources hosted on a distributed system of resource servers. These servers

are interconnected through the worldwide web

using the internet for the unclassified MSRR

and through the Secret Internet Protocol

Routing Network (SIPRNET) for the classified MSRR. The MSRR provides a layer of

services that includes the registration of

resources and users, description and quality

information of resources, and specialized

search capabilities.4

 DMSO has also created

the Modeling and Simulation Operational

Support Activity (MSOSA), which assists DoD

activities in meeting their M&S needs by providing operational advice and facilitating

access to M&S information and assets. The

MSOSA is a contractor-staffed activity operating under the direction of the DMSO Director of Operations.5

 Programs needing M&S

help or assistance should contact the MSOSA.

Contact information is available through the

DMSO website (mentioned previously in the

preface).

DMSO is also leading a DoD-wide effort to

establish a common technical framework to

facilitate the interoperability of all types of

models and simulations, among themselves

and with command, control, communications, computers and intelligence (C4I) systems, as well as to facilitate the reuse of
5-3

M&S components. This Common Technical

Framework (CTF) consists of three pieces: the

CMMS, Data Standards (DS), and the High

Level Architecture (HLA).6

The mission of the CMMS is to develop a

conceptual model of the mission space for

each DoD mission area to provide a common

basis for development of consistent and

authoritative M&S representations. Its purpose is to provide designers with an evolvable

and accessible framework of tools and

resources for conceptual analysis. The mission

space structure, tools, and resources will provide both an overarching framework and

access to the necessary data and detail to permit development of consistent, interoperable,

and authoritative representations of the environment, systems, and human behavior in

DoD simulation. Using this framework,

designers will be able to develop a clear picture

of what they wish to represent in order to produce a workable model or simulation for any

application. This picture will be multi-dimensional and must include a depiction of the

entities, actions, and interactions that must be

represented. There will be several CMMS corresponding to broad mission areas (such as

conventional combat operations, other military operations, training, acquisition, and

analysis).7

The second piece of the CTF is the M&S Data

Standards Program. The mission of the M&S

DS Program is to enable data suppliers to provide the M&S community with cost-effective,

timely, and certified data to promote reuse and

sharing of data; interoperability of models and

simulations within themselves and with the

warfighter’s C4I systems; and improved credibility of M&S results. The strategic objective

is to establish, promulgate, and oversee

policies, procedures and methodologies for

M&S data requirements; data standards; data

verification, validation, and certification;

authoritative data sources (ADS) and data

security to provide quality data as common

representations of the natural environment,

systems, and human behavior. The Data

Program is organized into four areas: Data

Engineering (including Data Interface

Format (DIF) and tool development),

Authoritative Data Sources, Data Quality,

and Data Security.8

Within the Data Engineering area, the Synthetic Environment Data Representation and

Interchange Specification (SEDRIS) effort is

significant for facilitating SBA. Currently,

there is no uniform and effective standard

mechanism for interchanging synthetic

environments between M&S applications.

SEDRIS will support the unambiguous interchange of data between database generation

systems by using a standard application

programmer’s interface that will allow the data

to be captured from the producer’s native format with accompanying explanatory metadata.

Although SEDRIS by itself will not solve all

interoperability problems, it will provide the

technology to enable solutions.9

The third piece of the CTF is the High Level

Architecture (HLA) program. The HLA is a

com-posable approach to constructing simulations that recognizes that no single, monolithic simulation can satisfy the needs of all

users; all uses of simulations and useful ways

of combining them cannot be anticipated in

advance; and future technological capabilities

and a variety of operating configurations must

be accommodated.10 The HLA provides a

common framework within which specific system architectures, or federations, are defined

by three components. These are:
5-4

1. Ten rules that define the relationships

among the federation components;

2. An object model template that specifies

the form in which simulation elements are

described;

3. An interface specification that describes

the way simulations interact during

operation.11

A federate is a member of a HLA federation,

which may include federation managers; data

collectors; real world (“live”) systems such as

instrumented ranges, sensors, or C4I systems;

simulations; passive viewers; and other

utilities.12

The MSRR, MSOSA, and CTF are all pieces

of the SBA infrastructure that facilitate the

entry of programs into SBA. (We’ll have more

to say about standards and interoperability in

Chapter 7, Challenges to Implementing SBA.)

At this stage, many of the tools needed to do

enterprise-wide simulation are at an immature, “micro” level, and are not ready to handle

“macro” jobs.

The big three auto makers have been solving

their enterprise data interoperability needs by

dictating that their first-tier suppliers must use

their computer-aided design packages for development and submission of their products.

(Chrysler uses Dassault Systemes’ CATIA,

Ford uses SDRC’s Ideas, and General Motors

uses Unigraphics Solutions, Inc.’s package).

The commercial side of The Boeing Company

has also standardized with CATIA. Others are

choosing to solve their enterprise needs by

using standards for the universal exchange of

product information. For example, the Standard for the Exchange of Product Data

(STEP) is an emerging international standard

for representing data about products that provides a set of standard definitions for the data

throughout the product life cycle.13 Using

STEP, The Boeing Company (military side) is

able to successfully exchange design information between its St. Louis facility (which uses

Unigraphics to design the Joint Strike

Fighter’s fore body) with its Seattle facility

(which uses CATIA in its role as systems integrator).14 General Dynamics Land Systems

(GDLS) uses both Pro-E and Computervision

for their design software, while their vendors

mostly use Autocad, Unigraphics, and Pro-E.

GDLS maintains interoperablity by focusing

their efforts on ensuring the accuracy of the

data.15 Likewise, United Defense Limited

Partnership is using various commercially

available tools (e.g. Pro-E for solid modeling

and Corypheous for visualization) and linking them together to get the job done.16 The

effort a program must devote to ensuring

seamless interoperability between the products of M&S tools will depend on factors such

as the number of different M&S tools used,

their degree of compatibility and the level

which they interface. Translation efforts in the

examples cited above, range from being a nuisance to being major problems that require

full time support.

Regardless of the method used, programs

need a way to communicate freely in their integrated data environments. HLA compliance

for simulations has been mandated by OSD

for 30 September 2000 and non-compliant

development and modification of simulations

must cease by 30 September 1998; however,

“HLA is not an interoperability ‘magic wand.’

That is, HLA will not automatically make

every simulation suitable for federating with

every other simulation, nor guarantee a valid,

meaningful exchange of information across the

federation ... but the HLA does provide the
5-5

critical technical foundation for the interoperability of simulations among themselves, and

with live systems.”17 Said another way, HLA

will provide the common “plug” within a

federation, but not guarantee that models

from different federations will “play” together in simulation. When examining the

possibilities, the real answer may often be that

applications need to be “compatible but not

necessarily common.”18

A final factor influencing the transition to SBA

will be how compelling a need there is to

change the current way of doing business. Crisis is often the reason for change and has been

a force driving many programs to turn to

increasing and more innovative uses of M&S.

Certainly this is the case for the Army’s

Comanche helicopter program. The program

office used M&S extensively to conduct a competitive fly off in which two competing teams

used man-in-the-loop simulation instead of

building costly prototype aircraft. In addition,

M&S activities allowed program management

to work within budget constraints and maintain a viable program throughout developmental test and evaluation, with only one flying

prototype.19 In the automotive industry this

was most apparent with Chrysler, which was

on the verge of bankruptcy in the early 90s with

many employees working half days. In

response to this crisis, Chrysler built a new

technology and design center and made a significant switch-over to using computer-aided

engineering. A workforce that saw and supported the need for change made this possible.

Similarly, within the DoD there is a compelling need to find a better and more affordable

way of acquiring systems. It is unwise to continue business as usual in light of the rapidly

evolving M&S technology that will support

SBA as a better alternative.

SBA Push and SBA Pull

The domain of the program, the maturity of

the infrastructure, and the degree of need will

all influence how much SBA can be implemented in a program. But the key point is that

any program can start moving down the path

to implementing SBA. This “SBA Push” is

what programs are doing within their enterprise to implement SBA, given the existing

state of the domain, infrastructure, and need.

Likewise, the “SBA Pull” is the expectation

and demand for SBA from outside agencies

(for example, the warfighting and resource

allocation communities). Dollar for dollar,

programs that are implementing SBA will

be perceived as better programs, because of

the increased visibility, superior insight, and

subsequent “better, faster, and cheaper”

products.

SBA Push

We’ll start first by looking at how programs

can expand the envelope by moving down the

path to implementing SBA. Regardless of

where a program is in its life cycle, it can benefit by initiating SBA practices. It is usually

not cost-effective for a program to wait until

everything is completely available and in place

before starting to use SBA. It’s more important to get started, achieve some successes,

learn, and continue to build. A good analogy

is that of building a housing subdivision. Once

the overall plan is in place, there’s a tradeoff

between the cost-efficiencies of scale and volume and the investment required. Usually this

means building a few houses, selling them to

generate cash flow to finance additional

houses and infrastructure, and continuing to

expand throughout the subdivision. An added

benefit to this approach is the learning that
5-6

occurs as each house is built. The last house

should be significantly better, take less time,

and cost less to build than the first.

Defense acquisition programs are normally

inventing or reinventing a new product and

inventions do not have reliability—reliability

is achieved through history, use, and time. It’s

important, therefore, to get started in SBA in

order to continually and incrementally

improve the processes, rather than waiting

until everything is in place. The movement

towards SBA will be a progression. It has been

stated that “the concept is revolutionary, but the

implementation will be evolutionary.”20

An important consideration for the program

manager is to not oversell SBA. As noted by

one industry manager, the rate of change

towards SBA is a concern. The community

must accurately portray the direction, the

speed of advance, and the capabilities of the

final end state of SBA so that expectations are

in line with reality. There will be incremental

changes and payoffs. Creating unrealistic

expectations can negate any potential benefits.

In particular, there may be many things that

cannot be modeled well enough to get the

information needed.

Does it require an up-front investment for programs to adopt SBA? Certainly there are initial investments required in terms of the computer and software purchases, as well as

workforce training.21 There are other costs, including application software changes requiring

the updating and revalidation of simulations.

Much has been said about The Boeing

Company’s failure to recoup its approximately

$1.5B to $2B investment for the full CAD/

CAM system used on the 777 airplane. However, their automated, simulation-based

design and build process enabled them to deliver a completely reliable, operational aircraft

on day one, a first-of-its-kind in the commercial aircraft industry. In the past it usually took

a year after delivery to work out all the software bugs, maintenance issues, and parts distribution for a new aircraft. Boeing’s instant

success with the first aircraft has stimulated

more sales and has further benefits that extend

across improved operations for the entire

Boeing commercial aircraft workforce and

procedures. The company continues to remain

well positioned to outstrip their competition

with faster aircraft upgrades.22 With better

customer satisfaction, lower operating costs,

and lower maintenance costs, Boeing expects

to sell more 777 aircraft. They are also able to

use this technology on other aircraft for no

development costs. Finally, if Boeing hadn’t

got the aircraft to market as quickly, Airbus

might have taken some of the sales. Benefits

like these are difficult to factor into the

investment decision with hard data because

there is no way to predict the outcome had

traditional processes been used. This “cost

avoidance” or opportunity cost issue makes it

difficult to determine the return on investment

for switching to the SBA approach. Programs

can show cost avoidance figures resulting from

the building of fewer prototypes, or more efficient tests, or the need for only one flying

testbed. But it is very difficult, if not impossible, to put dollar values on risk reduction or

a greatly improved product.

In any event, most programs probably will not

receive extra funding to change over to this

new SBA approach. Reality is a matter of

reallocation of the existing budget to obtain

the best value, and determining when that

value will be received. Again, this will require a tradeoff analysis to balance specific

needs with the available resources. The
5-7

Army dictates the use of a Simulation Support Plan (SSP) to effectively manage and integrate the use of M&S within acquisition programs.23 The Air Force handles this tradeoff

analysis by requiring programs to reflect their

M&S strategy and requirements in the appropriate acquisition documentation, such as Operational Requirements Documents, Test and

Evaluation Master Plans, and Single Acquisition Management Plans.24 The objective of

these requirements is the same: to determine

the high-leverage areas that simulation can

enhance, while at the same time planning to

build from these successes.

High-leverage areas can vary greatly from one

program to another. Factors that can influence

a program’s high-leverage areas are such

things as the program’s developmental time

line and the program’s strategies and goals.

For example, rather than baselining a system

first and then creating a trainer, programs are

creating early-on test-beds that provide the

opportunity to iterate and immerse the user,

which can ultimately grow into a trainer.25

Through the use of the virtual environment

the Crusader program, with a simulation budget of $9M, was able to reduce the program’s

requirement for physical prototypes by six. At

a price tag of $50M each, the program realized a total cost avoidance of $300M.26 Strategies and goals also help identify programmatic

high-leverage areas. Chrysler has reduced

their cost of doing business by $800M by taking eight months out of the cycle time through

the use of the virtual environment during design development. Their prototypes are now

as good as the first production cars were when

they were using the traditional production

method.27 General Motors does 80 percent of

its manufacturing in-house (compared with

Chrysler’s 20 percent and Ford’s 50 percent). GM’s costs, therefore, are associated

with tooling, and they must concentrating on

increasing their confidence level before committing to tooling.28 This goal of increasing the

confidence level in the design is supported

through the use of SBA practices. Ford, on

the other hand, concentrates on a global, distributed engineering capability, which emphasizes the importance of product information

management.29 Designing in the virtual environment supports this strategy of distributed

engineering and information management.

Many of today’s programs focus primarily on

the design development portion and not the

full life cycle of a system and indeed, this is an

important first step in achieving the full benefits of SBA. Within systems, we’re just

beginning to simulate beyond the traditional

performance issues to address the entire life

cycle’s cost issues during the design. These

include manufacturability, supportability,

lethality, sustainability, mobility, survivability,

flexibility, interoperability, reliability, and

affordability. One of the nearest term capabilities that will benefit many programs is “virtual manufacturing”—the ability to look at the

manufacturing issues during design.

The F-22 program is looking at supportability

issues by using a 3-D model that has enabled

two mechanics to influence the design by making critical inputs long before any hands-on

prototypes were available.30 The JSF program

is developing the Joint Interim Mission Model

(JIMM) to merge two legacy models, enabling

test and training inputs to their early modeling efforts until JMASS is available.31 The Crusader program, together with the combat

development community, has jointly conducted a series of simulated warfighting

experiments called Concept Evaluation Programs (CEPs), which provided early warfighter

insights into the changing tactics, techniques,
5-8

and procedures associated with the new

weapon system.32 During the redesign of the

Boeing 737 aircraft, Boeing found that

digitizing older systems where the design was

only on blueprints can still pay big dividends.

They did a full Digital Product Assembly and

Digital Product Definition for only those parts

that were new (the wing, empennage, and

main landing gear). Although the digitization

of the old parts was tedious, Boeing found it

to be worthwhile.33 The JSF created cost curves

for each alternative during its Analysis of

Alternatives, which allowed operational

tradeoffs using CAIV.34

Using the CAIV concept, it is critical for

a program manager to know how design

decisions will impact on overall program costs.

Some programs have developed cost models

based on historical costs of similar type

systems. A good example is the CVN-77 aircraft carrier program that looked at previous

ships to determine what the cost drivers were

throughout the life cycle of a ship.35 This identified where the CVN-77 program should

target attention to achieve the most cost

reduction.

The Crusader program has pretty high confidence in its cost models and has established

an Ownership Cost Working Group consisting of representatives from the contractor,

user community, and the program office to

identify life cycle cost drivers and methods to

eliminate or minimize them. During the early

phase of the Crusader’s development, an independent study group was commissioned to

analyze the program’s requirements and design in order to provide recommendations

balancing weight, cost, and performance. To

facilitate this process, a model was developed.

The model was based on interviews with design engineers and subject matter experts. The

model showed the overall change in force effectiveness based on incremental increases or

decreases in weight, cost ,and performance.36

The program manager and user representative are able to use this information

collaboratively in making tradeoff decisions.

An SBA approach enables early identification

of the key drivers (such as cost, schedule, and

performance) throughout the system’s life

cycle, by simulating systems and environments

external to the system.37 The Longbow Hellfire

missile program eliminated its fly-to-buy criteria by the close coupling with their contractor enabled by using the Simulation/Test Acceptance Facility (STAF) to finalize the acceptance test procedures long before the production phase.38

The High Mobility Artillery Rocket System

(HIMARS) program is an example of how

valuable M&S development and reuse is

becoming to program managers. The

HIMARS is a multiple launch rocket system

that uses a five-ton truck as its mobility platform. The HIMARS program manager

decided that using M&S was the best approach

for development of the HIMARS rocket

launcher. No model of a five-ton truck existed,

but a model was needed for integration efforts

if M&S was to be used to develop the rocket

launcher. The HIMARS program spent $500K

of a $300M budget to build a 5-ton truck model

from the Family of Tactical Vehicles (FMTV)

program, and is willingly providing this model

to other programs that ask for it. Other programs in the Tank and Automotive Command

can now build off of this initial model to

develop models of the other 24 variants of 2.5-

ton and five-ton vehicles that have 85 percent

commonality39 at much less expense than if

each program were to create a new model.

Later on, the Military Traffic Management
5-9

Command (MTMC) used the five-ton truck

model to conduct rail impact analysis during

transportability testing. MTMC was also able

to use the model to conduct air and ship

impact analysis.40 Users outside the Tank and

Automotive community will also be able to

benefit when this model is added to DMSO’s

MSRR, and is searchable and readily available. What started out as a requirement to

model a single vehicle for one program has

grown into the potential for influencing many

programs and aiding the requirements of other

organizations.

Many companies and programs are using a

visualization room to bring team members

together for the free exchange of ideas

between disciplines. Although much of the

work in an integrated data environment can

be distributed and decentralized, a visualization room visually presents the digital data in

forms that a team can use as a group. Concepts and work-in-progress can be displayed,

analyzed, and debated by integrated product

teams, even if they are geographically dispersed. Work can be imported electronically

to conduct design reviews or to immerse a

customer for feedback. The Marine Corps’

Advanced Amphibious Assault Vehicle

(AAAV) program has an excellent visualization room. It was built as an integral part of

the entire AAAV facility, which includes government and contractor personnel. The

visualization room provides the IPPD team a

location to discuss issues while viewing 3-D

representations of the vehicle or any component. If the team needs further clarification

of an issue they can move to the adjacent highbay area to look at a mockup or the physical

prototype. This capability is invaluable for

quickly and effectively resolving program

issues.

In building the 777 commercial airliner,

Boeing used an application called FlyThru that

provides a three-dimensional view of the

airplane as it is being designed. This allows

errors to be detected and fixed prior to committing to expensive manufacturing processes.

The fit of parts and systems on the 777 airplane was 20 times better than what had been

normally achieved in the past. According to

Boeing’s manager of Visualization Tools, “Being able to digitally build the entire plane and

see the parts of the plane before building it,

was the biggest money saver for the 777.”41

SBA Pull

SBA Pull (as we described at the beginning of

the chapter) is the expectation and demand from

others once they see the advantages of SBA

From program inception, the warfighter community will become an active participant in

program design and development. Long

before any physical prototypes are available,

virtual prototypes of system concepts will be

able to participate in exercises that demonstrate the battlefield effects. Logisticians will

be able to quantify the cost effectiveness of

making the system more supportable and work

with the design team to find the most costeffective mix of performance and support

parameters. Using visualization tools such as

three-dimensional solid modeling, people with

real field experience can also provide realtime user feedback on design iterations,

because they will be able to “see” and interact

with the design as it matures.

This user feedback will begin to extend beyond

the traditional service boundaries of the sponsoring organization. Programs will begin to

look at how their system will interact with

other systems on the battlefield by interfacing
5-10

those system models on a virtual battlefield.

The user will be able to begin using the new

system in the expected environment and

assessing any potential incompatibilities or

unforeseen circumstances that could be

averted in the design. Programs will be able

to use each others’ models to get the design

information they need. The combatant commands can also be immersed at various intervals as the design matures and provide their

assessment and feedback on how well the system is meeting their expectations. There will

be better dialogue with the resource allocation community since the program will have

much improved tools to keep the cost of the

system affordable. In exploring the cost drivers over the entire system’s life, they will also

have much more accurate projections of the

system’s cost. Regardless of who is interfacing with the program, these outside agencies

will have a much enhanced ability to see what

the program is. Indeed, many programs noted

that a significant side benefit was the great

marketing tool this approach provides for visitors and oversight personnel, because program

outsiders could instantly grasp the import of

what they saw.

The Crusader field artillery program started

a force effectiveness analysis even before the

request for proposal was issued. The contractor conducted trade studies to optimize the system within the overall system, using multiple

scenarios. They looked at the problems

encountered (e.g. thermal analysis, ammunition capacity, reliability, maintainability, availability, etc.) in terms of cost per force effectiveness. They discovered that the Crusader

could do things the current system (with upgrades) cannot. For example, the current Paladin system can not keep up with the Bradley

armored personnel carrier or the M1 tank. In

addition, the Crusader frees up the Multiple

Launch Rocket System to hit deeper targets.

In short, the Crusader increased the operational tempo of the battlefield. Some of these

insights were intuitive, and some were not. The

value of much of this information was difficult to explain, and the contractor was caught

between satisfying the requirements stated in

the request for proposal and making tradeoffs

whose value to the user he could only guess

at. For example, if he could decrease the size

of the crew from five people down to three,

what was the change in total ownership cost?

Were the cost savings significant enough to

justify spending money now to decrease the

crew size? What were the operational tempo

impacts on the Bradley and would it be able

to get the necessary information quickly

enough to the Crusader, which is 40 kilometers away? These are the types of issues that

the requirement community needs to deal with

early. The earlier in the design phase we find

the answers to these questions, the better—in

the past we’ve often not discovered these

issues until operational testing, or even worse

not until they’re deployed to the first unit. By

then we have already produced and fielded the

system, when if the problem had been discovered earlier the cost to fix it would have been

considerably less.

While SBA provides the capability to integrate

back into the requirements generation process, it can also reach forward into the resource

allocation process (the Planning, Programming, and Budgeting System), by providing

total ownership cost information for the out

year budgets. For example, there is an obvious extra cost associated with putting simulators on board an aircraft carrier, but it may be

worth that cost to counteract deployed training atrophy.44 SBA provides a way to explore

those costs.
5-11

Even low cost demonstrations are finding it

cost-effective to use a simulation-based

approach. The Predator unmanned aerial

vehicle advanced concept technology

demonstration creatively used M&S to predict

operational effectiveness and assess alternative force structure options, and thereby

determined an optimum system configuration.45

Many programs are expanding the envelope

on what’s possible in SBA, and they are doing

it cost-effectively. And although the tools are

still limited, these programs are beginning to

address the bigger implications of systems of

systems issues, where in the past we did not

have the ability to address them at all. Larry

Winslow, Director of Technology for Boeing’s

Phantom Works, summed it up by saying “Now

we’re doing Program SBA. In the future we’ll

be doing Systems of Systems SBA.”46
5-12

ENDNOTES

14. Larry Winslow, Director of Technology, Boeing

Company Phantom Works, interview by authors,

29 April 1998.

15. Robert Lentz, General Dynamics Land Systems,

interview by authors, 24 February 1998.

16. Richard Staiert, United Defense Limited Partnership, interview by authors.

17. High-Level Architecture (HLA) Transition Report, March 1998, p. 2, on line http://www.dmso.

mil/hla/policy/rept611.html accessed on 29 May

1998.

18. Dr. Stuart Starr, The MITRE Corporation,

interview by authors, 4 March 1998.

19. Bergantz, ibid.

20. Dr. David Chang, General Motors, interview by

authors, 31 March 1998.

21. For more information on implementing computer aided engineering and concurrent engineering within a company, see “Manager’s Guide

to Computer Aided Engineering,” John Krouse,

OnWord Press, 1993.

22. John Roundhill, Vice President, The Boeing

Company, remarks at INCOSE Annual Symposium in Vancouver, B.C., on 30 July 1998.

23. Army Regulation AR 5-11, “Management of

Army Models and Simulations,” 10 July 1997.

24. Air Force Acquisition Policy 97A-004, “Modeling and Simulation in Support of the Air Force

Acquisition Process,” 28 November 1997.

25. Colonel Lynn Carroll, USAF, Warfighter Training Research Division Chief, Air Force Research

Laboratory, briefing to the Joint Strike Fighter

Mission Level Strategy session, 30 March 1998.

1. “A Taxonomy for Warfare Simulation

(SIMTAX),” Military Operations Research Society (MORS) Report, 27 October 1989, on-line

http://www.dmso.mil/docslib/mspolicy/glossary/

accessed 24 June 1998.

2. DoD 5000.59-M, DoD Modeling and Simulation

(M&S) Glossary, December 1997.

3. Jim Boudreaux, Electric Boat Corporation, interview by authors. 24 February 1998.

4. MSRR home page, on-line http://www.msrr.

dmso.mil/ accessed on 25 June 1998.

5. MSOSA home page, on-line http://www.msosa.

mil.inter.net/Default.htm accessed on 25 June

1998.

6. Judith Dahmann, “High Level Architecture,”

briefing (1998). See http://www.dmso.mil/ for a

more in depth discussion of the CTF and its three

components: CMMS, Data Standards, and HLA.

7. DMSO CMMS home page, on-line http://

www.dmso.mil/projects/cmms/ accessed on 26

June 1998.

8. DMSO Data Standardization home page, on-line

http://www.dmso.mil/projects/ds/ accessed on 26

June 1998.

9. http://www.sedris.org/faq_trpl.htm

10. HLA home page, on-line http://hla.dmso.mil/

accessed on 25 June 1998.

11. HLA home page, on-line http://hla.dmso.mil/

accessed on 25 June 1998.

12. DoD 5000.59-M, DoD Modeling and Simulation

(M&S) Glossary, December 1997.

13. PDES, Inc. home page, on line http://

pdesinc.scra.org/ accessed on 5 May 1998.
5-13

26. Armbruster, ibid.

27. Greg Avesian; Chrysler Corporation; Manager,

Solutions Marketing; interview by authors, 26

February 1998.

28. Chang, ibid.

29. Richard Riff, Manager, C3P Project Office, Ford

Motor Company, interview by authors, 23 April

1998.

30. Larry Winslow, Director of Technology, Boeing

Company Phantom Works, interview by authors,

29 April 1998.

31. Major General Leslie J. Kenne, Program Manager, Joint Strike Fighter Program, Remarks at

the National Defense Industrial Association SBA

Conference, 17 March 1998.

32. Sheaves, ibid.

33. Michael Mecham, “Aerospace Chases The Software Boom,” Aviation Week & Space Technology,

6 October 1997, 46-49.

34. Kenne, ibid.

35. Dan Selfridge, Newport News Shipbuilding,

“Lifecycle Cost and Manning Cost Analysis”

briefing given to NAVSEA PMS 392 Submarine

TOC IPT Meeting on 2 April 1998.

36. Jim Shields, Chief of Systems Engineering, Crusader Program Office, interview by authors, 1 July

1998.

37. Armbruster, ibid.

38. Ibid.

39. http://www.army-technology.com/contractors/

transport/stewart_stevenson/index.html accessed 11 Aug 1998.

40. Brock Birdsong, Missile Research, Development

and Engineering Center, US Army Aviation and

Missile Command, phone interview with author,

29 July 1998.

41. Marilyn Morgan, “Boeing Leads Way in Virtual

Mock-up”, Boeing News, April 3,1998.

42. Peter Cherry, Vice President, Vector Research,

Inc., interview by authors, 27 April 1998.

43. CJCSI 3170.01, Requirements Generation

System, (Formerly MOP 77), 13 June 1997.

44. Colonel Lynn Carroll, U.S. Air Force, Warfighter

Training Research Division Chief, Air Force Research Laboratory, briefing to the Joint Strike

Fighter Mission Level Strategy Session, 30 March

1998.

45. MG Kenneth R. Israel, Director, Defense Airborne Reconnaissance Office, “Modeling and

Simulation Employed in the Predator Unmanned

Aerial Vehicle Program,” on-line http://

www.acq.osd.mil/daro/homepage/predms.htm

accessed 11 December 1997.

46. Larry Winslow, Director of Technology, Boeing

Company Phantom Works, interview by authors,

29 April 1998.
5-14
6-1

6

A FUTURE STATE OF SBA

“Future commanders must be able to visualize and create the “best

fit” of available forces needed to produce the immediate effects and

achieve the desired results.”

Joint Vision 2010

As the Joint Vision 2010 quotation above

states, commanders must be able to project

and select the best fit of systems available to

accomplish their mission. This applies to not

only the existing systems available today, but

also to looking ahead at the deficiencies and

opportunities in accomplishing future missions. SBA will provide this capability to

warfighting commanders by giving them a synthetic view of future battlefields, where conceptual systems can be integrated and their

operational effectiveness assessed.

In describing this future process a key

assumption is that the systems acquisition process will continue to be centered on programs.

The purpose of the acquisition process will be,

as it is now, to produce superior systems in

response to mission needs. An SBA approach

will not change this product focus, but instead

will enhance DoD’s ability to acquire the “best

fit” of new systems to meet the warfighter’s

needs. Some argue that in order to achieve the

full potential of SBA it is necessary to move

This chapter portrays the implications of an

SBA approach for future defense systems

acquisition, by projecting SBA capabilities and

trends. While it is difficult to predict exactly

how change will be implemented or to

describe the resultant organizational structure,

it is possible to envision what an SBA

approach holds for the future. Within this

acquisition process of the future, systems will

be thoroughly modeled and simulated prior

to “bending metal.” Beginning with initial user

needs, requirement definition, through design,

testing, manufacturing, logistics, training,

operational usage, and disposal, the synthetic

environment and integrated teams will play a

major role. Ideally, the acquisition process

becomes a continuum without a definable

beginning or end, continually assessing projected threats and needs against the ability to

meet those needs. One senior acquisition

official put it best when he said, “If the models are done correctly, the simulations will tell

us when we need to start a new program.”1
6-2

from a “product-centric” focus of systems

acquisition, to a “mission-centric” view.2

 The

argument is that during the earliest stages of

turning a mission need into a material solution, it is necessary to have the broadest possible view in order to optimize the tradeoffs

across service boundaries. This broad view

could only be achieved by merging similar mission areas across the services, and divesting

the services of control of the funds for these

mission areas. The managers of these mission

areas would have budget and decision

authority to determine where best to invest

funding to meet user requirements.

But just because the technology will enable the

possibility of doing business this way does not

necessarily make it desirable or a necessary

pre-condition for implementing SBA. In fact

it would be a major barrier to implementing

SBA, as the services have no intention of giving up their ability to determine the

requirements for the material solutions to

their mission needs. This cuts to the very foundation of how the services interpret their roles

and missions. There is nothing inherent about

this new way of doing business that requires

the services to give up their right to determine

their future, so we reject the argument that

implementing SBA requires a “purple,” or

multi-service, acquisition corps. Instead, our

assumption is that acquisition will remain in

the province of each service and be programcentric. There are good reasons for maintaining separate services, and they are continuing

to learn how to interoperate and function

as a joint team. So too can the acquisition

system.

Another assumption is that a primary objective

of SBA is to get the design right before building a system, when in effect the design

becomes frozen. Any changes required after

the start of production result in costly

engineering changes and/or system modifications. If the design is kept mostly in the synthetic environment prior to production, the

cost and time required to make the change is

significantly reduced. In an SBA process, the

defining point is when the program moves out

of the synthetic environment and begins

production.

The new ability to bring systems together while

they are still in design will be key to ensuring

that systems will work together when they are

fielded, and that they are not over- or underdesigned. Competing new systems can be

evaluated side-by-side on equal terms to determine the best alternative. Duplication of

capability within and across Services will be

more readily apparent, resulting in significant

cost avoidance. According to Dr. Peter Cherry,

Vice President of Vector Research, Inc., “The

tools we’re using are getting better, but now

we need a richer context in which to make decisions, otherwise we’ll continue to only make

acquisition decisions on the margin. We make

marginal decisions now—the challenge is to

provide a system of systems context so we

can do better than decisions on the margin.

Previously we only made incremental changes

program by program. Today, systems of systems require the total integration of systems.

We need to be able to walk the system’s impact

on the battlefield all the way from the platoon,

to the company, to the battalion level. We have

a lack of the full understanding of system supportability issues. We still talk about systems

from the cockpit perspective—we need to

elevate up to a higher level, and talk to Congress about battlefield effectiveness, not how

much faster or how stealthy a system is. For

example, we need to be able to show the value

of the JSTARS [Joint Surveillance and Target

Attack Radar System aircraft] to the artillery,
6-3

not how many targets it can track and it’s mean

time between failure.”42

System of systems issues are a primary concern

of the Service Chiefs and the Joint Requirements Oversight Council (JROC); however,

these concerns are now primarily handled

at a very high level through the use of a

Capstone Requirements Document (CRD).

A CRD is used to identify overarching requirements for a system, or several programs that form a system of systems. It contains performance-based requirements to

facilitate development of individual program

requirements.43

Programs implementing SBA will be able to

give the users the necessary insight to balance

these needs and deficiencies in time to influence the design, rather than waiting for the

Services and JROC to validate ever-increasing

point design solutions during Milestone

reviews. At this high level it’s too late to impact

the design cost-effectively, as all of the assumptions and tradeoff decisions have already been

made in getting ready for the review. What

are being presented are the results of the

program optimized against the static requirements dictated in the Operational Requirements Document (ORD), which assumed the

user fully understood the impact of those

requirements.

With this in mind, we will start by looking at

the beginning of a program. This chapter

discusses a future state of SBA by looking

at four major phases over the life cycle of a

program, beginning with:

1. generating a mission need;

2. iterative design and development activities;

3. production; and

4. operations and sustainment.

Needs Generation Phase

SBA will encompass more than what we today

call acquisition. SBA will provide a common

synthetic environment that the warfighting

community can use to experiment with nonmaterial solutions to meet deficiencies, and to

help them determine affordable requirements.

As stated in the Chairman of the Joint Chiefs

of Staff Special Instruction for the Requirements Generation System, the warfighter

determines the need for a material or a nonmaterial solution by conducting a mission area

analysis of the current and projected capabilities to accomplish assigned missions. The Services’ first try to satisfy mission needs through

nonmateriel solutions, such as changes in doctrine or tactics. The Services will be able to

use updated models and simulations provided

by the SBA community as part of their mission area analysis. If a nonmateriel solution is

deemed not feasible, the need is translated

into a Mission Need Statement (MNS) which

is expressed in broad operational terms.3

 As

the MNS is documented, validated and

approved, the SBA process will bring the

warfighter and acquisition communities

together from the very beginning of program

definition. Members of the acquisition community will have a better understanding of the

warfighter’s need because they participated in

the MNS development. Information gained

from non-material experiments can provide

the acquisition community insight into the

warfighter’s needs, and any models developed

may be of use in searching for a material

solution.
6-4

The generation of an MNS marks the end of

the needs generation phase in the SBA process. The warfighting and acquisition communities will determine the members of the IPT

who will collaborate to generate and define

the operational requirements during the next

phase of iterative design and development.

Iterative Design and

Development Phase

The second major phase of a program in this

future state of SBA is iterative design and

development. As discussed in Chapter One,

the early decisions in a program are critical

because to a large extent they will determine

system effectiveness and what the system will

cost throughout its life cycle. The iterative

process of determining requirements sets the

stage for the TOC of the program. SBA

enables the acquisition community to support

the warfighters in determining affordable

requirements, and provides a vehicle to get

industry involved during early concept development. Rather than locking in requirements before understanding them fully, the

requirements group in a program office

develops simulations to conduct engineering

trade studies that quantify capability versus

cost. Flexibility in design is maintained as long

as possible by keeping the requirements fluid

until the cost impacts of each are well understood. The warfighters, industry, and program

office personnel are an integrated team that

makes informed decisions regarding capability versus cost issues across the entire life cycle

of the system.

During this phase, models and simulations

representing new and revolutionary concepts

and systems are evaluated in a series of

iterative “what if” trade off analyses. New concepts are evaluated in a virtual environment

to determine their operational impact and

system effectiveness. Cost models provide the

TOC impacts of competing concepts and

designs.

New conceptual models are evaluated to

determine if they provide significant operational benefit, and promising concepts are

developed further. As confidence in the

models grows, many other issues can be

addressed concurrently. Initial cost models of

a design provide a rough order of magnitude

(ROM) estimate of the TOC. As the design

matures these ROMs are refined and become

more accurate. Using an integrated visual representation of the system, functional experts

on the IPPD team make their requirements

and concerns known to all IPPD members, and

work directly with each other in determining

the optimum design. Training requirements

are addressed and decisions made whether to

build a stand-alone trainer or to have training

embedded in the system. Logisticians and

testers address supportability and testability

issues and compare results between competing conceptual systems. The transportation

community evaluates transportability of the

system design. Manufacturing issues are

addressed by designing a virtual factory. Iterations continue until a high level of confidence

is reached in the design, at which time the

system is submitted for a production decision.

If approved, all models, simulations and data

are passed on for use in producing the system.

There should be little chance of a system being

disapproved during a production decision,

because all stakeholders have frequent opportunities for visibility and feedback during system design. Periodic reviews of the system’s

progress will be possible by immersing the

audience using tools such as visualization

rooms. Problem areas can be resolved quickly
6-5

by bringing in all stakeholders to buy off on

critical affordability decisions as part of the

design tradeoffs.

Oversight activities are necessary to ensure

programs are producing the best possible

weapons systems. Although a program will be

considering systems of systems issues as part

of their iterative development, there will still

be macro-level issues that need to be resolved

at levels higher than any one program. This

oversight function will be able to consider both

joint and combined systems of systems issues,

for those systems that will operate with other

services and allied nations.

The primary activity of the oversight function

is assessing and prioritizing efforts from a

system of systems perspective. It assesses the

progress of programs’ iterative developments

and the expected production time to ensure

that the system will be available when required.

Programs will stay in iterative development

and continue to refine the design in a synthetic

environment, continually assessing new

technologies as well as input from the field. A

decision to begin production is made in sufficient time to allow for production, fielding,

and training to occur prior to the need date.

Oversight reviews will be periodic instead of

activity-based, but can also be unscheduled in

response to unforeseen changes in the need.

These reviews will replace the current milestone reviews, because there will be minimal

need for successive and increasing levels of

dollar commitment as the program design

matures because the dollar commitment will

be significantly reduced. Instead, the ability

to conduct most of the design in the synthetic

environment will have the effect of fusing

the activities leading up to today’s present

Milestone III production decision.

Production Phase

The third major phase of a program in this

future state of SBA is production, which begins

after a production decision is made at the end

of the design and development phase. All

models and simulations previously developed

for a program are available for production.

These models and simulations include tooling and factory design. Existing models and

simulations are developed to a greater level

of detail as necessary for full-scale production.

As a result of previous rigorous development

and validation of the models and simulations

there should be:

• Minimum Engineering Change Proposals

because the design coming into the Production Phase was determined acceptable to

all stakeholders;

• Reduced tooling time because tooling

requirements were already identified;

• Less decisions to be made because there

should be few surprises and therefore less

tradeoff decisions required—all known

tradeoffs were evaluated and decided on;

• Better quality of the end item because reliability, maintainability, supportability,

manufacturability, producibility, and other

support issues were all considered and

incorporated into the design;

• An affordable design because evaluation

of cost models with Cost As an Independent Variable (CAIV) forced early tradeoff decisions between performance and

affordability.
6-6

Operations and Sustainment Phase

The fourth and final major phase of a program

in this future state of SBA is operations and

sustainment, which ends upon system disposal.

Actual operations and support (O&S) data are

used to update the system’s models and simulations. There is continuous analysis of this

O&S feedback to determine future deficiencies. A deficiency may only require a system

upgrade or it may require development of a

totally new concept. Proposed changes to the

existing models and simulations or new models and simulations to meet the deficiency are

again evaluated in a synthetic environment and

the warfighters can use this data to assess nonmaterial alternatives, based upon the acquisition community’s updated models. Operational units and commands could use these

models and simulations to better predict their

budget requirements. Cost models that reflect

actual costs incurred for operations and

support of the system could be used to accurately project quarterly and annual training

and operational requirements.

Net Result

This proactive approach to assessing program

status is driven by mission need and the date a

system is required, rather than the excessive

length of the acquisition cycle. Programs can

be much more flexible to respond to changing

mission needs, because there is not as much

emphasis on locking down the requirements.

The entire process is more agile because programs are able to stay in iterative development

until production lead times coincide with need

dates.

The ultimate effect of this new SBA process

will be the ability to conduct “what if”

scenarios across the entire spectrum of the

DoD. Program teams will be charged with

evaluating the interactions of their system(s)

with other systems on the battlefield. The oversight function will be armed with the knowledge to make better informed decisions on

which systems will best meet the military’s

needs. Program teams will have a much closer

link to the battlefield upon which the systems

operate, because they will be able to constantly

see the battlefield effects and implications of

their design decisions. All will help keep the

team focused on the real goal of producing

superior systems.
6-7

ENDNOTES

1. LTG George Muellner, Air Force Principal

Deputy for Acquisition, interview by authors, 30

March 1998.

2. For a more complete description of this alternative view, see “Functional Description Document

– 980209” developed by the NDIA’s Industry

Steering Group, available through the SBA Special Interest Area on-line http://www.msosa.mil.

inter.net/sia-sba/sba_sia_documents.asp?

process=view_archive

3. CJCSI 3170.01, Requirements Generation

System (Formerly MOP 77)), 13 June 1997.
6-8
7-1

7

CHALLENGES TO

IMPLEMENTING SBA

significant change in the way DoD does

business. There is a natural resistance to

changing the old ways if they appear to be

working. There is significant resistance to

believing in virtual prototypes, as people are

more apt to believe one physical test versus

thousands of virtual tests. For example, the

technology for simulating the crashworthiness

of a vehicle far outpaces the acceptance of the

results, to the point where their accuracy is

better than the variability between different

physical crashes.1

 However, despite this capability, all automobile manufacturers continue

to conduct expensive crash tests (albeit they

are conducting much fewer tests than in the

past). A culture change is required to view

computer modeling as analogous to physical

modeling. Even so, acceptance of M&S is

occurring at an increasing rate as M&S is

successfully implemented. The engineers

frequently resist using M&S at first, but soon

discover that it can be a very valuable design

tool, and that they often do not need hardware

to find problems.

An important corollary to this acceptance

problem is to “not believe everything you

see.” Simulations create compelling visual

Many programs are making great progress in

implementing SBA; however, even the best

programs are a long way from implementing

the ideal process outlined in the previous chapter. This chapter outlines the cultural, technical, and procedural challenges to implementing this future state of SBA. Some of these

challenges will be overcome more quickly than

others depending to a large extent on how

much emphasis they receive. Cultural challenges result from people’s views, beliefs, and

management actions; technical challenges

result from M&S limitations in technology;

and process challenges are caused by the way

the DoD is organized and operates.

Cultural Challenges

Many people maintain that cultural barriers

are the biggest challenge to overcome in moving to an SBA process. These challenges

include acceptance of M&S, not believing

everything seen, working as teams in a

distributed environment, and becoming

proactive in determining an affordable design.

Acceptance of the results of M&S is probably

the single biggest challenge, as it involves a
7-2

arguments that can drive complacency in questioning the underlying models. The mere fact

that something can be modeled and simulated

does not mean that it can be built. A rigorous

process of verifying, validating, and accrediting (VV&A) models and simulations for their

intended purpose can counteract this problem,

and can increase the level of confidence that

the models and simulations are representative

of what was intended and accurately represent

reality.2

Another cultural challenge is learning how to

work as teams in distributed environments.

Western culture and training places an emphasis on individual performance, but systems are

designed and built by teams. An SBA process

can accentuate this long-standing problem

because even more emphasis is placed on

working together as a team, and the teams may

change frequently and not be co-located.

Many companies are experimenting with ways

to create just-in-time teams and reward team

performance.

A final cultural challenge is learning to become

proactive in designing an affordable system.

This involves continually refining the requirements with the user to find the optimum mix

of performance and affordability, and looking at the costs across the entire life of the

system. The range of options to explore can

be intimidating, and the tendency will be to

move on with a design that appears good

enough, rather than continuing to explore

design options to reduce the TOC of the

system.

Technical Challenges

Technical barriers are perhaps the easiest to

envision. Technical challenges include

interoperability concerns, and limitations in

what can be effectively and affordably

modeled.

Interoperability challenges result from the

desired ability to seamlessly share models and

simulations between programs and across

services to analyze tradeoffs in a system of

systems synthetic environment. As discussed

previously, the DMSO is pursuing various

initiatives in the Common Technical Framework to facilitate interoperability and reuse.

Certainly more technical improvements will

continue emerging, such as computer-aided

tools, which will facilitate the development and

deployment of reuse standards and policies

such as the High Level Architecture.

Programs can experience interoperability challenges within their own program, as they

attempt to move information up and down the

hierarchy of models and simulations (for

example, from campaign to mission to

engagement to engineering, and then back

up). There are interoperability challenges in

making different tools communicate with each

other, particularly those that were developed

for use in a different context from the one for

which they are now intended. For example, a

logistics model that helps determine the

optimum stockage level for spare parts may

not be compatible with another tool that

predicts reliability, or even worse, gives

conflicting results.

The MSRR will provide increasing access to

programs to assist in their finding what models

and simulations are available for their systems

of systems analyses, which will facilitate maximum reuse. This is particularly evident with

items that have been designed for reuse, such

as threat and environment data. However,

designing for reuse costs approximately three

times as much as designing for a specific
7-3

program,3

 and is only feasible when all of the

programs intending to use the models and

simulations are known in advance. HLA begins to solve this problem by helping groups

that choose to interact with each other define

specific system architectures called federations. This creates a problem when a program

wants to use another program’s model or simulation that was developed as part of another

federation. An extreme but probable example

would be attempting to use models and simulations from several other federations to get a

true picture of the battlefield.

In an ideal SBA process, models and simulations can be reused as easily as using a library

or repository. However, historically this type

of ad hoc reuse has only resulted in about 20

percent savings in software code reuse.4

 A lot

of time is often spent analyzing the code to

determine what is applicable for reuse, and

often the most valuable results are the insights

gained into how to functionally design the new

code, rather than the reuse of the code itself.

A SBA process may not actually reuse models

and simulations in this traditional software

sense, but it does point out the considerable

difficulty experienced in a similar process

which is much simpler than SBA envisions.

Some of the easiest areas of reuse to envision

are for common areas that many programs will

need, such as models of the environment in

which systems will operate, and of the threats

these systems will face. Efforts are already

ongoing in some of these areas, such as the

SEDRIS effort mentioned earlier, and the Virtual Proving Ground under development at

the Army’s Test and Evaluation Command.

The DoD has a vested interest in reducing

duplication of efforts in these areas, and also

in having a common depiction of them across

all systems. Programs will be able to use the

DoD’s environment to evaluate many aspects

of their system. If a program has a requirement that is not satisfied by the common

environment, it can be expanded to meet the

needs, and it can be subsequently brought back

into the aggregate common environment for

reuse by other programs. The common environment gets expanded as a result. Having a

common environment will also provide credibility and help to minimize issues that could

arise if each contractor developed his own

environment and threat models, especially if

these systems were in a competitive selection

process. Other items the DoD should consider

providing are the analytical tools that will be

used to evaluate contractors’ models. For

example, if the DoD planned to use a constructive model such as the Army’s CASTFOREM

(Combined Arms and Support Task Force

Evaluation Model) to evaluate a contractor’s

proposed model, then the program office

should consider providing CASTFOREM to

the contractor early in the program.

There are also technical limitations in what

can be effectively and affordably modeled,

including reliable cost figures, failure and

reliability prediction, and human behavior.

Cost models that accurately project TOC need

to be available as early as possible to enable

informed tradeoff analyses that all communities and decision makers can believe. Discrepancies will otherwise persist between cost

estimates prepared by differing agencies—for

example, the Government Accounting Office

might predict higher cost estimates than the

program’s estimates. These discrepancies

could effectively cripple the program’s ability

to design a more affordable system, because

the credibility of the system’s total ownership costs that are being used to make design tradeoffs is called into question. Cost

curves for each alternative considered allow
7-4

quantification of alternatives and operational

tradeoffs with cost as an independent variable.

The objective is to use modeling and simulation to help determine the “bend in the knee”

when looking at performance versus cost. This

bend in the knee refers to the point where the

cost of providing greater performance begins

to exceed the benefit it provides, or its point

of marginal return. By showing the user where

the bend in the knee starts to occur on a given

performance characteristic then we can ask if

it is worth the extra cost to achieve the

incrementally less capability per dollar spent.

The earlier we are able to show the user the

cost implications of his requirements, the

sooner we can agree to the necessary tradeoffs

to result in an affordable design. This forces

the user to make tough calls early in the

system’s development and avoids pursuing

unaffordable solutions.

We need to be able to treat CAIV at the earliest stages, by incorporating cost considerations

into the computer-aided design and

engineering tools. It is otherwise impossible

to get cost on the table as a trade mechanism.

We are still using cost models that use dollars

per pound as the basis of estimating, as

opposed to conducting detailed cost tradeoffs.

Just as computer-numerically-controlled

(CNC) machines are able to manufacture

items directly from the digital design data, cost

models would be able to directly use the digital

data to project manufacturing costs. Even

more sophisticated cost models would be able

to address costs across the life cycle of the

system—operations costs, upgrade costs,

maintenance costs, and disposal costs, for

example.

Today’s state-of-the-art in modeling and simulation does not provide the ability to fully and

accurately predict failures to the point where

we can replace physical reliability testing with

reliability testing using modeling and simulation. Durability algorithms are in short supply, because of the complexity of durability

models. The ability to predict failure varies

from one product area to another. Physics of

failure is farthest along in electronics.5

 For

example, transistor manufacturers are able to

predict with high confidence when transistors

will fail. In chips, we know exactly how they’ll

function and how they’ll behave, but the same

does not hold for the mechanical world. Currently, we do not have good models for predicting failure and determining reliability of

mechanical systems or their components. The

ability to predict the failure of a new system

requires an understanding of the internal physics of the materials within the system. The

ability to model this level of fidelity within

components and within a system is in its

infancy, but some fields of study are making

strides by applying recent increases in computing power. Meteorological experts are

beginning to model weather systems at

increasing levels of fidelity to achieve a better

understanding of how thunderstorms, hurricanes, and tornadoes behave. Pharmaceutical

companies are beginning to model new drugs

at the atomic level and then immerse themselves into the model to help them create the

drug. We can expect similar advances in other

fields as the technology matures. We need to

develop physics of failure and cost models to

help predict reliability through simulation. If,

in simulation, we are able to test a program to

the point of failure, then we can improve overall system reliability by improving the area that

failed. We can also use these failure data to

predict the required maintenance, manpower,

spares, and life cycle cost for a system given

the number of operating hours.
7-5

The limitations to our ability to build high

fidelity physics of failure models result from

such things as the limited understanding of

how materials fail, variations in the manufacturing processes of materials, and variations

in the properties of materials. One of the most

significant limitations is the cost and time

required to capture all the stress loads that a

system or component will experience during

operations. Each test conducted on a system

captures data for only one set of operational

conditions. Program schedule and cost constraints limit the number of samples that can

be taken, so assumptions are necessary to

extrapolate from this limited sample size to

make conclusions for all possible conditions

that may be experienced. If we could accurately model the physics of the system, we

could conduct many more tests in the synthetic

environment than are cost-effective with

today’s approach.

A program manager must demonstrate that

the system he is developing will meet the reliability required by the user. Durability testing

is the process that provides the evidence of

whether a system is reliable enough to proceed toward production. Durability testing is

very demanding of program resources,

requiring the PM to schedule and pay for hundreds, if not thousands of hours of system

operations and testing. Confidence in the

reliability of the systems that we field requires

that we have an acceptable level of understanding about why and how often a system

will fail, which can be a significant cost driver

of a program.6

 The challenge is to reduce the

number of reliability testing hours required.

If we can reduce the number of reliability testing hours required of an end item, we can

reduce both the schedule and budget necessary for a program during this phase. As an

example, the M1A2 tank program dedicated

four physical prototype vehicles just to support

durability testing.7

Today, when we develop a new system it has

no known reliability.8 Only through use and

time do we achieve reliability. The only true

way to achieve this knowledge is to run the

physical system. When we do this we gain confidence in the reliability of that one system we

tested and then we infer reliability to any system that is built to the same specifications.

Therefore, the two critical factors are first, that

the prototype we test be in accordance with

the system’s specifications; and second, that

we are able to accurately repeat builds of subsequent systems. The weakness of this approach is that the physical prototype is likely

to have flaws and inconsistencies compared

with the final production version of the system because of human error in building the

prototype, and configuration changes as we

gain more knowledge about the system and as

technology continues to develop.

Modeling and simulation actually enables us

to improve the confidence in the ability to

accurately build and test a prototype that is

truly representative of a production item.

Using M&S we are able to keep the design

open to changes longer because we have

greater confidence that the final design will

meet the required need and that the production run will be successful. This means we need

less time allocated for last minute changes and

gives us more time to learn about the system

as it matures, to incorporate lessons learned

back into the system, and to integrate the latest

technology if desired. The changes made using

M&S early in the design can be much less

expensive than if we were making changes to

a physical prototype, which would not be

possible until much later in the program.
7-6

A final technical limitation is human behavior modeling. Conceptually, SBA could

include modeling and simulating the acquisition process itself to help produce better

acquisition strategies. Human behavior is difficult to model, however, because our

assumptions cannot be correct for all individuals. Some factors that influence how an individual will react in a given situation include

experience, education, values, self-confidence,

and fatigue. The variability in reactions is so

great that accurately predicting behavior is not

possible and any process, such as acquisition,

that is heavily dependent on human interface

and decisions is difficult to predict and model.

Probably the best we’ll be able to achieve in

this area will be through the use of wargaming

techniques. Wargaming allows for shortfalls in

models and data, and provides a way for program managers to simulate interactions with

entities that cannot be directly controlled. The

insights gained by understanding the

implications of decisions should enable

managers to develop more effective

strategies.9

Process Challenges

Process challenges result from the way DoD

operates and is organized, and include the

change in the test process, security of classified

and proprietary data, and metrics.

A major concern of program managers is the

cost and time required to test a system. A

significant portion of a program’s schedule

and budget is usually allocated for testing,

and an SBA approach has the potential of

reducing both. As mentioned earlier, the

role of testing is no longer to validate point

designs; rather the role of testing is to validate models so that there is a high confidence

in the resultant simulations from those models.

Test personnel should be members of the IPT

during the earliest stages, and should work

with the developer to consider and resolve test

issues. A good example of addressing test

issues early is the Follow On To TOW (TubeLaunched, Optically Tracked, Wire Guided)

hand-held anti-tank missile, or FOTT. In the

FOTT program the specifications for the Virtual Proving Ground were included in the

Request for Proposal, and test personnel were

co-located with the program developers.10

These test personnel grew up with the models

and matured them with tests as required to

gain confidence in their use. They also helped

develop the acceptance test procedures (ATP),

so that many iterations and excursions could

occur quickly and inexpensively and the results

could be checked against the ATP after each

iteration. Involving developmental test and

operational test personnel early in the program and using M&S to gain higher confidence in the first missile allowed the FOTT

PM to reduce missile requirements for engineering and manufacturing development

(EMD). The Javelin program (a similar fireand-forget missile) required 190 missiles for

EMD, while FOTT required only 40 missiles.11

Within the test community there needs to be

a shift in focus from testing hardware to testing simulations of hardware. By testing simulations of future hardware systems, IPT members can resolve many problems before expensive prototypes are built. Test simulations help

to identify the problem areas where efforts

should be focused to increase the probability

that live testing is successful. The knowledge

gained from conducting these simulations will

lead to better quality testing as well as more

efficient and less testing of the physical hardware. As an example, during the Longbow

helicopter Hellfire missile development there

were more than 500 simulated firings and only
7-7

20 live shots. This resulted in the Longbow

Simulation/Test Acceptance Facility (STAF)

paying for itself in one year, with several years

left in the program.12 In addition, other

programs will be able to “reuse” the facility,

further amplifying the cost avoidance for DoD.

The purpose of conducting live testing is

changing from validation of the physical system to validation of the model. General

Motors is using tests to validate model design

assumptions and not to find fixes. The goal is

to get the design right as early in the life cycle

as possible.13 As confidence in the results of

models goes up the number of live tests

required will come down. Any questions that

simulations cannot answer may require a

physical prototype of the subsystem in question to reduce risk to an acceptable level.

Information gained from building and testing

any physical prototype of the system should

be used to update the system’s model.

A big challenge is to have an authoritative

DoD source for a common synthetic test

environment. If we intend to compare competing system designs, we need to ensure we

evaluate those systems within a common

synthetic environment that is also available to

the system developers. This common

environment is required to provide a level

playing field for evaluating conceptual designs

of a system. These relationships are critical in

ensuring that the run-time databases, derived

from the synthetic environment database, will

be correlated so that all “views” of the environment are the same. An important “view”

is that of computer-generated forces that do

not “see” the battlefield but must use the data

representations to correctly interpret the

environmental conditions.14 This means that

the computer controlling the actions of military forces in a simulation needs information

about the terrain and objects in the environment represented in a way it can interpret. For

example, the visual representation of an object

that a soldier-in-the-loop needs during a simulation is no good to the computer. The computer needs information about the same terrain and objects, but numerically represents

the essential information, such as position,

height, weight, etc. Unfortunately, synthetic

environment data interchange today is cumbersome, expensive, and often unreliable.

A second major process concern among

government and industry is ensuring the

security of classified and proprietary data. If

we expect agencies and companies to provide

data through repositories, we must be able to

provide a high level of confidence in the security of those data. We want data to be readily

available and shared with those who need

the information and at the same time, we

must keep data from those who do not have

a need to know. Today’s technology allows

us to share data easily and there are many

ways that data can get into the wrong hands.

As an example, an automobile manufacturer

inadvertently left some data of a “next generation” vehicle on a data tape that was

reused and given to one of its suppliers. The

supplier inadvertently passed the data on to

a competitor. This security breach of corporate knowledge allowed the competition

to make up a year-and-a-half in development

time.15

Program offices are also very careful about

who uses the models and simulations of their

program. They want to know exactly how

and for what purpose the data will be used.

This is understandable because a competing program could use the detailed information provided by the models and simulations of another program to show how its
7-8

system is superior or where the other program is deficient. The information contained

in a model could also be a valuable source to

determine vulnerabilities of the new system.

A final process challenge will be developing

metrics, which are needed for an overall

assessment of progress in the execution of a

program. Metrics are measures of success that

serve as a powerful management tool for

evaluating effectiveness in accomplishing

project objectives and in achieving and

improving customer satisfaction.16 They allow

program managers to manage on the basis of

facts and data. Metrics solely focused on

individual process results do not give a picture

of overall success in implementation. Metrics,

therefore, should also be structured to identify

the overall effects of SBA implementation.

Measures that could be used to gauge success include schedule, responsiveness and

timeliness, and communications. The measurement of variances between planned and

actual schedules, consumption of resources,

productivity, customer satisfaction, cycle time,

and completion of tasks could be particularly

useful in determining if a program is capturing the expected benefits and cost avoidance.

In a collaborative, concurrent, and distributed

engineering environment, communications

takes on an increasingly important role.

Organizations must be able to easily and

quickly move large amounts of data. They

must track how well information is moving

through the organization, such as whether the

Product Information Manager is facilitating

concurrent engineering, or whether the engineers are experiencing unacceptable delay

times.

There are difficult cultural, technical, and procedural challenges to implementing the ideal

SBA process of the future. But we don’t have

to wait until all the challenges are solved before we start implementing those parts where

we can extract value. The process is beginning

to change, and those who adapt will find the

biggest rewards.
7-9

ENDNOTES

1. Stefan Thomke, “Simulation, Learning and R&D

Performance: Evidence from Automotive Development,” Research Policy 27 (1) (May 1998).

2. See “DoD VV&A Recommended Practices

Guide,” Defense Modeling and Simulation Office, November 1996, for detailed information on

VV&A.

3. James Shiflett; Vice President, SAIC; interview

by authors, 20 March 1998.

4. LTC Gene Glasser, U.S. Army; Director, Army

Software Reuse Center, interview by authors, 7

May 1998.

5. Jerry Chapin, U.S. Army Tank and Automotive

Command, interview by authors, 25 February

1998.

6. Art Adlam, U.S. Army Tank and Automotive

Command, interview by authors, 25 February

1998.

7. Robert Lentz, General Dynamics Land Systems,

interview by authors, 24 February 1998.

8. David Chang and Prakash Shrivastaka, General

Motors, interview by authors, 31 March 1998.

9. For an innovative approach to using wargaming

in acquisition, see on-line http://www.npt.nuwc.

navy.mil/divnewport/Business-War-Games/

main.html.

10. John Haug, Director, Technical Mission,

TECOM, interview by authors, 11 February 1998.

11. John Bier, Technical Management, Close Combat Anti-Armor Weapon System program office,

phone interview, 8 September 1998.

12. Brigadier General Robert E. Armbruster, Deputy

for Systems Acquisition, U.S. Army Aviation and

Missile Command, remarks during presentation

at Army SBA Conference, Orlando, FL, 21-22

January 1998.

13. David Chang and Prakash Shrivastaka, General

Motors, interview by authors, 31 March 1998.

14. http://www.sedris.org/doc_trpl.htm

15. Unnamed source due to confidentiality.

16. Many of the ideas expressed here were inspired

by the “DoD Guide to Integrated Product and

Process Development,” Office of the Under Secretary of Defense (Acquisition and Technology),

5 February 1996.
7-10
8-1

8

CONCLUSIONS AND

RECOMMENDATIONS

decision makers throughout the life cycle of a

program. Risks can be identified, minimized,

mitigated, or even eliminated to increase the

probability of program success. Products

developed when implementing SBA can serve

as excellent communication and marketing

tools. Being able to visualize a system as it

evolves from a low fidelity concept into a high

fidelity system is very useful in resolving conflicts throughout development. Simulation and

visualization are powerful tools for IPTs and

decision makers which allow them to see the

results of each decision and evaluate the ramifications before having to make a costly final

decision.

Using SBA, programs are making decisions

based not only on more information, but also

better information, which is leading to a better

product. These programs are being “pushed”

into using SBA for a variety of reasons, but

once there, they are finding many benefits.

Both industry and government programs are

turning to SBA as a means of remaining competitive and making the most of limited funds.

Companies that have implemented SBA on

one program have started to apply SBA to

other programs and are looking for ways to

leverage efforts across programs. Many

This chapter provides our conclusions regarding SBA, along with some recommendations

on where to proceed. Our conclusions are that

SBA is a smarter way of doing business, but

there are significant challenges ahead to realizing its full potential. Our recommendations

are:

• educate the workforce on this new concept;

• encourage a single SBA process for DoD;

• identify SBA efforts within program

documents;

• continue developing the tools and infrastructure to make it happen;

• fund SBA as well as M&S development;

and

• conduct a follow-on study that results in an

SBA Recommended Practices Guide.

Conclusions

SBA is a smarter way of doing business. It

creates an environment where complex issues

can be integrated and evaluated to support
8-2

programs are discovering that using M&S is

the only way to accomplish certain tasks because of cost, safety, time, or scale considerations. As decision makers and users begin to

see programs implementing SBA, there will

be a “pull,” or an expectation that SBA will

be used in other programs. As they are shown

the cost impacts, warfighters are beginning to

see an impact and the value of delaying the

finalizing of their requirements. Programs using SBA will be perceived as better and in fact

will be better, than programs not using SBA.

SBA will enable programs to quickly determine what the issues are and to focus on them.

Not only will the program benefit from implementing an SBA approach, but outside organizations will benefit also.

The roles of industry and DoD are unclear for

implementing SBA, however. There is

uncertainty about who should make the first

move toward institutionalizing SBA. In the

spirit of the Single Process Initiative to follow

best commercial practices, DoD is reluctant

to dictate to industry how to implement SBA,

believing that it will occur naturally in the competitive commercial world. Industry’s attitude,

on the other hand, is that if DoD wants a standard system, it will have to put some money

behind development.1

 Industry is particularly

concerned that each service will come up with

different standards and ways to implement

SBA.

As discussed in the last chapter, there are technical, process, and cultural challenges to

implementing SBA. The technical challenges

are impeding the full potential of SBA, but

are continuing to improve at various paces

among the domains. SBA will continue to pick

up speed and move toward a critical mass as

these barriers continue to fall. The process

challenges require further analysis and study.

SBA provides a significantly different

approach to identifying and meeting military

requirements from the current process. The

implications of a collaborative, concurrent

development approach versus the present

sequential, increasing buy-in approach are not

well understood, and may require us to reevaluate our current process. Activities within

the process may need to occur more often or

at a different time in the development of a

system. The cultural challenges require

education and experience—education to

spread the word on this new perspective on using M&S, and experience that comes by getting

started and learning with a new process. Many

programs are doing smart things and there is a

great need for educating all SBA stakeholders

about what SBA is and how it is different from

just using M&S in a program. Acceptance of

SBA will continue to grow as more people

learn its concept, capabilities, and benefits.

Gaining experience with SBA is one of the

best ways of overcoming cultural barriers.

Recommendations

Our first recommendation is to educate the

workforce on the new SBA concept by incorporating classes on SBA into the curriculum

at the Defense Systems Management College

that define what SBA is, what tools are available, what SBA efforts are ongoing, and what

the vision of SBA is. These classes should also

include lessons learned by programs that successfully implemented SBA as well as from

programs that failed to implement SBA. As

noted earlier, there is only a single reference

to SBA in the Defense Acquisition Deskbook.

Some of the areas that are now devoted to

M&S should be reviewed with the goal of

incorporating the broader viewpoint of using

an SBA approach in acquisition, rather than

just the use of M&S. Once this happens, the
8-3

field should be encouraged to provide input

on best practices, and wisdom and advice. To

get the dialogue started, we recommend that

this book be included in the Deskbook as a

reference source for SBA.

Our second recommendation is to encourage

all stakeholders to define and develop a single

DoD SBA process. The Services need to work

together to develop an agreed upon single

process for SBA. A single process will reduce

the burden on contractors when dealing with

multiple Services. The result should be a

reduction in the overall cost to DoD across

programs. Certainly we don’t want a situation

that replicates the one in some industries that

dictate specific software packages for their

suppliers.

Our third recommendation is to identify SBA

efforts within existing program documents,

such as the Simulation Support Plan, the Test

and Evaluation Master Plan, and the Single

Acquisition Management Plan. The intent is

to move beyond just specifying how M&S will

be used in a program, to outlining how M&S

will be applied to change the process to an

SBA approach.

Our fourth recommendation is to continue

developing the tools and infrastructure to

enable programs to conduct systems of systems

analyses. The DoD should continue to look

for investments in common SBA tools that will

support analysis of programs within a system

of systems synthetic environment. DoD

needs to find the resources to develop common portions of the SBA infrastructure, as

this issue is too big and too important to

leave to a piecemeal approach by individual

programs, which cannot afford to fund largescale M&S efforts to benefit all programs. Programs do only what is required to successfully

field a system. A program may develop a

capability that all may benefit from, but this

is secondary and not the focus of the program. Common tools developed by a program

or by DoD need to be available to other

programs.

Our fifth recommendation is to fund SBA

development in addition to M&S development

activities. Efforts need to focus on achieving

the holistic view of SBA and not on individual

and unrelated M&S activities. The bigger picture of SBA must be emphasized and efforts

coordinated to optimize their contribution to

achieving SBA. When screening priorities for

funding, use an SBA filter as well as an M&S

filter.

Our sixth and final recommendation is to conduct a follow-on study to analyze the impacts

of SBA on cost, schedule, and performance,

and from that develop an SBA Recommended

Practices Guide. While we concentrated our

efforts on the success stories and programs

that touted prowess in implementing SBA,

there are probably failures and lessons learned

the hard way. Some of these may be the result

of technology limitations, while others are

errors in implementation, but they are all valuable lessons. This effort should focus on the

hard evidence to show why SBA is indeed a

better, faster, and cheaper way of doing business, and thereby ferret out the key criteria

programs should use in determining how to

implement SBA in their program.
8-4

ENDNOTES

1. Bill Gregory, “Simulation’s Latest Promise,”

Armed Forces Journal International, June 1998,

Volume 135, Number 11, p.34.
A-1

APPENDIX A

ACRONYMS AND GLOSSARY
A-2
A-3

APPENDIX A

ACRONYMS AND GLOSSARY

ACRONYMS

Acronym Term

AAAV Advanced Amphibious Assault Vehicle

ACAT Acquisition Category

ACTD Advanced Concept Technology Demonstration

ADS Authoritative Data Sources

AI Artificial Intelligence

APMC Advanced Program Managers Course

C4ISR Command, Control, Communications, Computers, Intelligence, Surveillance,

and Reconnaissance

CAD Computer Aided Design

CAE #1 Component Acquisition Executive

CAE #2 Computer Aided Engineering

CAIV Cost as an Independent Variable

CAM Computer Aided Manufacturing

CASTFOREM Combined Arms and Support Task Force Evaluation Model

CEP Concept Evaluation Program

CFD Computational Fluid Dynamics

C4I Command, Control, Communications, Computers, and Intelligence

CMMS Conceptual Model of the Mission Space

CNC Computer-Numerically-Controlled

CODE Common Operating Digital Environment

CRD Capstone Requirements Document

CTF Common Technical Framework

DIF Data Interface Format

DMSO Defense Modeling and Simulation Office

DoD Department of Defense

DRA Decision Risk Analysis

DS Data Standards

DSAC Defense Systems Affordability Council
A-4

DSMC Defense Systems Management College

DT&E Developmental Test and Evaluation

EMD Engineering and Manufacturing Development

EXCIMS Executive Council for Modeling and Simulation

FMTV Family of Tactical Vehicles

FOTT Follow-On To TOW

HIMARS High Mobility Artillery Rocket System

HITL Human-in-the-Loop

HLA High Level Architecture

HWIL Hardware in the Loop

ICD Interface Control Document

IPPD Integrated Product and Process Development

IPT Integrated Product Team

JIMM Joint Interim Mission Model

JMASS Joint Modeling and Simulation System

JROC Joint Requirements Oversight Council

JSF Joint Strike Fighter

JSTARS Joint Surveillance and Target Attack Radar System

LCC Life Cycle Cost

LCM Life Cycle Management

LFT&E Live Fire Test and Evaluation

LRIP Low Rate Initial Production

MAIS Major Automated Information System

MDAP Major Defense Acquisition Program

MNS Mission Need Statement

MOE Measures of Effectiveness

MOO Measures of Outcome

MOP Measures of Performance

M&S Modeling and Simulation

MSIP Modeling and Simulation Investment Plan

MSOSA Modeling and Simulation Operational Support Activity

MSRR Modeling & Simulation Resource Repository

MTMC Military Traffic Management Command

NDIA National Defense Industrial Association
A-5

O&S Operations and Support

ORD Operational Requirements Document

OSD Office of the Secretary of Defense

OT&E Operational Test and Evaluation

PIM Product Information Management

PM Program Manager

PPBS Planning, Programming, and Budgeting System

QFD Quality Functional Deployment

ROM Rough Order of Magnitude

SBA Simulation Based Acquisition

SEDRIS Synthetic Environment Data Representation and Interchange Specification

SES Simulate, Emulate, Stimulate

SIPRNET Secret Internet Protocol Routing Network

SISO Simulation, Interoperability, and Standards Organization

SMART Simulation and Modeling for Acquisition, Requirements and Training

STAF Simulation Test Acceptance Facility

STEP #1 Simulation, Test, and Evaluation Process

STEP #2 Standard for the Exchange of Product Data

SWIL Software in the Loop

TCO Total Cost of Ownership

TOC Total Ownership Cost

TOW Tube-Launched, Optically-Tracked, Wire-Guided

VERT Venture Evaluation Review Technique

VV&A Verification, Validation, and Accreditation
A-6

GLOSSARY

Accreditation

The official certification that a model or simulation is acceptable for use for a specific purpose.

(reference DoD M&S Glossary)

Advanced Concept Technology Demonstration

Technology demonstrations that are tightly focused on specific military concepts and that

provide the incorporation of technology that is still at an informal stage into a warfighting

system. The ACTDs have three objectives: a) to have the user gain an understanding of and

to evaluate the military utility of concepts before committing to acquisition; b) to develop

corresponding concepts of operation and doctrine that make best use of the new capability;

and c) to provide the residual operational capability to the forces. ACTDs are of militarily

significant scope and of a size sufficient to establish utility.(reference DoD M&S Glossary)

Aggregation

The ability to group entities while preserving the effects of entity behavior and interaction

while grouped. (reference DoD M&S Glossary)

Algorithm

A prescribed set of well defined, unambiguous rules or processes for the solution of a problem

in a finite number of steps. (reference DoD M&S Glossary)

Artificial Intelligence

The effort to automate those human skills that illustrate our intelligence e.g., understanding visual images, understanding speech and written text, problem solving and medical

diagnosis. (reference DoD M&S Glossary)

Battlespace

Refers both to the physical environment in which the simulated warfare will take place and

the forces that will conduct the simulated warfare. All elements that support the front line

forces (e.g., logistics, intelligence) are included in this definition of battlespace. (reference

DoD M&S Glossary)

Benchmark

The activity of comparing the results of a model or simulation with an accepted representation of the process being modeled. (reference DoD M&S Glossary)

‘bend in the knee’

Refers to the point where incremental increases in performance are obtained at ever

increasing costs, or the point of diminishing returns.
A-7

Common-Use M&S

M&S applications, services, or materials provided by a DoD Component to two or more

DoD Components. (reference DoD M&S Glossary)

Component

(See “DoD Component”)

Computational Fluid Dynamics

The accurate numerical solution of the equations describing fluid and gas motion and the

related use of digital computers in fluid dynamics research. (reference DoD Mission Success

from High Performance Computing, DoD HPCMO Report, Office of the Secretary of Defense, DDRE, March 1995)

Computer Simulation

A dynamic representation of a model, often involving some combination of executing code,

control/display interface hardware, and interfaces to real-world equipment. (reference DoD

M&S Glossary)

Conceptual Model of the Mission Space

First abstractions of the real world that serve as a frame of reference for simulation development by capturing the basic information about important entities involved in any mission

and their key actions and interactions. They are simulation-neutral views of those entities,

actions, and interactions occurring in the real world. (reference DoD M&S Glossary)

Constructive Model or Simulation

Models and simulations that involve simulated people operating simulated systems. Real

people stimulate (make inputs) to such simulations, but are not involved in determining the

outcomes. Constructive simulations are often referred to as war games. (Note: Also see

additional information under “Live, Virtual, and Constructive Simulation.”) (reference DoD

M&S Glossary)

Cost as an Independent Variable

Methodologies used to acquire and operate affordable DoD systems by setting aggressive,

achievable lifecycle cost objectives, and managing achievement of these objectives by trading off performance and schedule, as necessary. Cost objectives balance mission needs with

projected out-year resources, taking into account anticipated process improvements in both

DoD and industry. CAIV has brought attention to the government’s responsibilities for

setting and adjusting lifecycle cost objectives and for evaluating requirements in terms of

overall cost consequences. (reference DSMC Glossary of Defense Acquisition Acronyms and

Terms, 8th Edition)

Decision Risk Analysis

A methodology that quantifies and ties together cost, schedule, performance, producibility,

risk, and quality to permit tradeoffs.
A-8

DoD Component

One of the four services within the Department of Defense, i.e. Army, Navy, Air Force, or

Marine Corps.

Domain

The physical or abstract space in which the entities and processes operate. The domain can

be land, sea, air, space, undersea, a combination of any of the above, or an abstract domain,

such as an n-dimensional mathematics space, or economic or psychological domains.

(reference DoD M&S Glossary)

Emulate

To represent a system by a model that accepts the same inputs and produces the same

outputs as the system represented. For example, to emulate an 8-bit computer with a 32-bit

computer. (reference DoD M&S Glossary)

Emulation

The imitation of a computer system, performed by a combination of hardware and software, that allows programs to run on systems that would normally be incompatible. (reference

http://www.wcom.com/cgi-bin/dictQuery.cgi?key=emulation)

Entity

A distinguishable person, place, unit, thing, event, or concept about which information is

kept. (reference DoD M&S Glossary.)

Enterprise

An arbitrarily-defined functional and administrative entity that exists to perform a specific,

integrated set of missions and achieve associated goals and objectives, encompassing all of

the primary functions necessary to perform those missions. An intranet, for example, is a

good example of an enterprise computing system. (reference http://www.pcwebopaedia.com/

TERM/e/enterprise.html)

Environment

The texture or detail of the natural domain, that is terrain relief, weather, day, night, terrain

cultural features (such as cities or farmland), sea states, etc.; and the external objects, conditions, and processes that influence the behavior of a system (such as terrain relief, weather,

day/night, terrain cultural features, etc.). (reference DoD M&S Glossary)

Executive Council for Modeling and Simulation

An organization established by the USD(A&T) and responsible for providing advice and

assistance on DoD M&S issues. Membership is determined by the USD(A&T) and is at the

Senior Executive Service, flag, and general officer level. (reference DoD M&S Glossary)
A-9

Federate

A member of a High Level Architecture Federation. All applications participating in a

Federation are called Federates. This may include federation managers, data collectors,

real world (“live”) systems (e.g., C4I systems, instrumented ranges, sensors), simulations,

passive viewers and other utilities. (reference DoD M&S Glossary)

Federation

A named set of interacting federates, a common federation object model, and supporting

Runtime Infrastructure, that are used as a whole to achieve some specific objective. (reference

DoD M&S Glossary)

Fidelity

The accuracy of the representation when compared to the real world. (reference DoD M&S

Glossary)

Hierarchy

A ranking or ordering of abstractions. (reference DoD M&S Glossary)

High Level Architecture

Major functional elements, interfaces, and design rules, pertaining as feasible to all DoD

simulation applications, and providing a common framework within which specific system

architectures can be defined. (reference DoD M&S Glossary)

Human Factors

The discipline or science of studying man-machine relationships and interactions. The term

covers all biomedical and psychological considerations; it includes, but is not limited to,

principles and applications in the areas of human engineering, personnel selection, training, life support, job performance aids, and human performance evaluation. (reference DoD

M&S Glossary)

Human-in-the-Loop

A model that requires human interaction. (reference DoD M&S Glossary)

Infrastructure

An underlying base or foundation; the basic facilities, equipment, and installations needed

for the functioning of a system. (reference DoD M&S Glossary)

Integrated Product and Process Development

An approach to systems acquisition that brings together all of the functional disciplines

required to develop, design, test, produce and field a system. This is essentially the same as

Concurrent Engineering. (reference DoD M&S Glossary)
A-10

Integrated Product Team

Integrated Product Teams are a means to achieve concurrent engineering or Integrated

Product and Process Development. They are multi-disciplinary teams consisting of representatives from all disciplines involved in the system acquisition process, from requirements

development through disposal. Having the participation of all the appropriate disciplines,

Integrated Product Teams are often empowered to make decisions to achieve successful

development of their particular product. (reference DoD M&S Glossary)

Interoperability

See: M&S Interoperability.

Life Cycle Cost

The total cost to the government of acquisition and ownership of that system over its useful

life. It includes the cost of development, acquisition, operations and support (to include

manpower), and where applicable, disposal. (reference DSMC Glossary of Defense Acquisition

Acronyms and Terms, 8th Edition)

Live Simulation

A simulation involving real people operating real systems. (Note: Also see additional information under “Live, Virtual, and Constructive Simulation.”) (reference DoD M&S Glossary)

Live, Virtual, and Constructive Simulation

The categorization of simulation into live, virtual, and constructive is problematic, because

there is no clear division between these categories. The degree of human participation in

the simulation is infinitely variable, as is the degree of equipment realism. This categorization of simulations also suffers by excluding a category for simulated people working real

equipment (e.g., smart vehicles). (Note: also see each term separately, e.g., live simulation)

(reference DoD M&S Glossary)

M&S Infrastructure

M&S systems and applications, communications, networks, architectures, standards and

protocols, and information resource repositories. (reference DoD M&S Glossary)

M&S Interoperability

The ability of a model or simulation to provide services to and accept services from other

models and simulations, and to use the services so exchanged to enable them to operate

effectively together. (reference DoD M&S Glossary)

Maintainability

The ability of an item to be retained in, or restored to, a specified skill level, using prescribed procedures and resources, at each prescribed level of maintenance and repair.

(reference DSMC Glossary of Defense Acquisition Acronyms and Terms, 8th Edition)
A-11

Mathematical Model

A symbolic model whose properties are expressed in mathematical symbols and relationships; for example, a model of a nation’s economy expressed as a set of equations. Contrast

with: graphical model; narrative model; software model; tabular model. (reference DoD

M&S Glossary)

Measures of Effectiveness (MOE)

A measure of operational success that must be closely related to the objective of the mission or operation being evaluated. A meaningful MOE must be quantifiable and measure

to what degree the real objective is achieved. (reference DSMC Glossary of Defense

Acquisition Acronyms and Terms, 8th Edition)

Measures of Outcome (MOO)

Metrics that define how operational requirements contribute to end results at higher levels,

such as campaign or national strategic outcomes. (reference M&S Report, DSMC 1994 Research Fellows Report)

Measures of Performance (MOP)

Measures of the lowest level of performance representing subsets of measures of effectiveness (MOEs). Examples are speed, payload, range, time on station, frequency, or other

distinctly quantifiable performance features. (reference DSMC Glossary of Defense

Acquisition Acronyms and Terms, 8th Edition)

Metadata

Information describing the characteristics of data; data or information about data; descriptive information about an organization’s data, data activities, systems, and holdings.

(reference DoD M&S Glossary)

Metric

A measure of the extent or degree to which a product possesses and exhibits a certain quality,

property, or attribute. (reference DoD M&S Glossary)

Metric(s)

A process or algorithm that may involve statistical sampling, mathematical computations,

and rule-based inferencing. Metrics provide the capability to detect and report defects within

a sample. (reference DoD M&S Glossary)

Mission Space

The environment of entities, actions, and interactions comprising the set of interrelated

processes used by individuals and/or organizations to accomplish assigned tasks. (reference

DoD M&S Glossary)
A-12

Model

A physical, mathematical, or otherwise logical representation of a system, entity,

phenomenon, or process. (reference DoD M&S Glossary)

Modeling

Application of a standard, rigorous, structured methodology to create and validate a physical, mathematical, or otherwise logical representation of a system, entity, phenomenon, or

process. (reference DoD M&S Glossary)

Modeling & Simulation Resource Repository

The MSRR is a collection of Modeling and Simulation (M&S) resources. MSRR Resources

include models, simulations, object models, Conceptual Models of the Mission Space

(CMMS), algorithms, instance databases, data sets, data standardization and administration products, documents, tools and utilities. (reference www.msrr.dmso.mil/)

Modeling and Simulation

The use of models, including emulators, prototypes, simulators, and stimulators, either statically or over time, to develop data as a basis for making managerial or technical decisions.

The terms “modeling” and “simulation” are often used interchangeably. (reference DoD

M&S Glossary)

Modeling and Simulation Accreditation

See: Accreditation.

Model-Test-Model

An integrated approach to using models and simulations in support of pre-test analysis and

planning; conducting the actual test and collecting data; and post-test analysis of test results

along with further validation of the models using the test data. (reference DoD M&S Glossary)

Physical Prototype

A model whose physical characteristics resemble the physical characteristics of the system

being modeled; for example, a plastic or wooden replica of an airplane. A mock-up. (reference

DoD M&S Glossary)
A-13

Planning, Programming, and Budgeting System (PPBS)

The primary resource allocation process of DoD. One of three major decision making support

systems for defense acquisition (the other two are the Requirements Generation System

and the Acquisition Management System). It is a formal, systematic structure for making

decisions on policy, strategy, and the development of forces and capabilities to accomplish

anticipated missions. PPBS is a cyclic process containing three distinct, but interrelated

phases: planning, which produces Defense Planning Guidance (DPG); programming, which

produces approved program objectives memorandum (POM) for the military departments

and defense agencies; and budgeting, which produces the DoD portion of the President’s

national budget. (reference DSMC Glossary of Defense Acquisition Acronyms and Terms, 8th

Edition)

Producibility

The relative ease of manufacturing an item or system. This relative ease is governed by the

characteristics and features of a design that enables economical fabrication, assembly,

inspection and testing using available manufacturing techniques. (reference DSMC Glossary

of Defense Acquisition Acronyms and Terms, 8th Edition)

Reliability

The ability of a system and its parts to perform its mission without failure, degradation, or

demand on the support system. (reference DSMC Glossary of Defense Acquisition Acronyms

and Terms, 8th Edition)

Reliability Model

A model used to estimate, measure, or predict the reliability of a system; for example, a

model of a computer system, used to estimate the total down time that will be experienced.

(reference DoD M&S Glossary)

Risk

Risk is a measure of the inability to achieve a program’s defined performance, schedule,

and cost objectives, and has two components: 1) The probability of failing to achieve particular performance, schedule, or cost objectives; and 2) The consequence of failing to achieve

those objectives. (reference DSMC Acquisition Strategy Guide, Third Edition)

SBA Pull

Those forces (attitudes, expectations, incentives, directives, policies, etc.), external to a

program office, that influence a program office to adopt an SBA approach.
A-14

SBA Push

Those forces (attitudes, expectations, incentives, directives, policies, etc.), internal to a

program office, that influence a program office to adopt an SBA approach.

Simulation Based Acquisition

An iterative, integrated product and process approach to acquisition, using modeling and

simulation, that enables the warfighting, resource allocation, and acquisition communities

to fulfill the warfighter’s materiel needs, while maintaining Cost As an Independent Variable

(CAIV) over the system’s entire lifecycle and within the DoD’s system of systems.

Solid Modeling

A digital representation of the surface characteristics of an object.

Stimulate

To provide input to a system in order to observe or evaluate the system’s response. (reference

DoD M&S Glossary)

Stimulation

The use of simulations to provide an external stimulus to a system or subsystem. An example

is the use of a simulation representing the radar return from a target to drive (stimulate) the

radar of a missile system within a hardware/software-in-the-loop simulation. (reference DoD

M&S Glossary)

Supportability

The degree of ease to which system design characteristics and planned logistics for resources,

including the logistic support elements, allows for the meeting of system availability and

wartime utilization requirements. (reference DSMC Glossary of Defense Acquisition Acronyms

and Terms, 8th Edition)

Synthetic Environment

Simulations that represent activities at a high level of realism, from simulations of theaters

of war to factories and manufacturing processes. These environments may be created within

a single computer or a vast distributed network connected by local and wide area networks

and augmented by super-realistic special effects and accurate behavioral models. They allow

visualization of and immersion into the environment being simulated. (reference DoD M&S

Glossary)
A-15

System of Systems

Seamless connectivity of Command, Control, Communications, Computers, Intelligence,

Surveillance, and Reconnaissance (C4ISR) across the sea-land-space interface in a joint

warfighting environment, and assimilation...into a coherent tactical picture...to develop a

multi-dimensional netted architecture...while providing rapid sensor-to-shooter connectivity. (reference Admiral Jay Johnson, Chief of Naval Operations, during a visit to the

Joint Strike Fighter program, 1995). The authors use the term System of Systems in a context broader than C4ISR, to encompass interactions between sub-systems of the same system, multiple system integration, user tradeoff analysis and optimization, and warfighting

scenario optimization. One of the first places for the term System of Systems to show up in

print was in the U.S. Naval Institute Proceedings May 1995 article “The Emerging System

of Systems,” by Admiral William A. Owens, US Navy. To the authors’ knowledge, there is

no commonly accepted definition of this term as yet.

Total Cost of Ownership

See: Total Ownership Cost (TOC).

Total Ownership Cost

The sum of all financial resources necessary to organize, equip, and sustain military forces

sufficient to meet national goals in compliance with all laws; all policies applicable to DoD;

all standards in effect for readiness, safety, and quality of life; and all other official measures of performance for DoD and its components. (reference 1998 Annual Report to the

President and the Congress, Office of the Secretary of Defense)

Validation

The process of determining the degree to which a model or simulation is an accurate

representation of the real world from the perspective of the intended uses of the model or

simulation. (reference DoD M&S Glossary)

Verification

The process of determining that a model or simulation implementation accurately represents

the developer’s conceptual description and specification. Verification also evaluates the

extent to which the model or simulation has been developed using sound and established

software-engineering techniques. (reference DoD M&S Glossary)

Verification, Validation, and Accreditation

See each separate term.

Virtual

The essence or effect of something, not the fact. (reference DoD M&S Glossary)
A-16

Virtual Battlespace

The illusion resulting from simulating the actual battlespace. (reference DoD M&S Glossary)

Virtual Manufacturing

A model or simulation of the processes required for making an item.

Virtual Prototype

A model or simulation of a system placed in a synthetic environment, and used to investigate and evaluate requirements, concepts, system design, testing, production, and sustainment

of the system throughout its lifecycle. (reference DoD M&S Glossary)

Virtual Reality

The effect created by generating an environment that does not exist in the real world. Usually,

virtual reality is a stereoscopic display and computer-generated three-dimensional

environment which has the effect of immersing the user in that environment. This is called

the immersion effect. The environment is interactive, allowing the participant to look and

navigate about the environment, enhancing the immersion effect. (reference DoD M&S

Glossary)

Virtual Simulation

A simulation involving real people operating simulated systems. Virtual simulations inject

human-in-the-loop in a central role by exercising motor control skills (e.g., flying an airplane), decision skills (e.g., committing fire control resources to action), or communication

skills (e.g., as members of a C4I team). (Note: Also see additional information under “Live,

Virtual, and Constructive Simulation.”) (reference DoD M&S Glossary)

Virtual World

See synthetic environment.

Visualization

The formation of an artificial image that cannot be seen otherwise. Typically, abstract data

that would normally appear as text and numbers is graphically displayed as an image. The

image can be animated to display time varying data. (reference DoD M&S Glossary)

War Game

A simulation game in which participants seek to achieve a specified military objective given

pre-established resources and constraints; for example, a simulation in which participants

make battlefield decisions and a computer determines the results of those decisions. Also

used synonymously with constructive simulation. (reference DoD M&S Glossary)
A-17

Wargaming

The act of conducting a war game.

What-if Analysis

Analysis conducted to determine the impact of changing a variable during the design of a

weapon system, as in “What if we change the stiffness of the beam, what will be the impact

on the cost, schedule, and performance of the system?”
A-18
B-1

APPENDIX B

INTERVIEWS AND PERSONAL CONTACTS
B-3

APPENDIX B

INTERVIEWS AND PERSONAL CONTACTS

* SBA Industry Steering Group Member 1

** Joint SBA Task Force Member2

COMPANY CONTACT

ADVANCED AMPHIBIOUS ASSAULT Richard Bayard

VEHICLE (AAAV) PROGRAM OFFICE Assistant Program Manager

AAAV Technology Center rbayard@notes.hqi.usmc.mil

USMC 991 Annapolis Way 703-492-3344

Woodbridge, VA 22191-1215

Mark Routson

General Dynamics

routsonm@gdls.com

703-492-3222

AERONAUTICAL SYSTEMS CENTER (ASC) Dr. Jerry Arnett

Distributed Mission Training Office arnettjb@asc-yw.wpafb.af.mil

Wright Patterson Air Force Base, OH

AIR FORCE MATERIAL COMMAND Lt Col Joseph M. Terry**

Modeling and Simulation Integration Office AFMC/MSIO-OL Liaison Officer

Operating Location terry.joe@afams.af.mil

12350 Research Parkway 407-208-5762

Orlando, FL 32826

AIR FORCE RESEARCH LABORATORY Col Lynn Carroll, Chief

6001 S. Power Road, Bldg. 558 Warfighter Training Research Division

Mesa, AZ 85206-0904

ANSER Dr. Aron Pinker

1215 Jefferson Davis Highway pinkera@anser.org

 Arlington, VA 22202 703-416-3439

APPLIED SYSTEMS INTELLIGENCE INC. Norm Geddes

10882 Crabapple Road, Suite 2 President

Roswell, Georgia 30075 norm@asinc.com
B-4

COMPANY CONTACT

ARMY SOFTWARE REUSE CENTER LTC Gene Glasser

ASQB-IRC (Stop C-2) Director, ASRC

6000 6th St, Suite S122A glassere@issc.belvoir.army.mil

Ft Belvoir, VA 22060-5576 806-3860/4300

ASSISTANT SECRETARY OF THE ARMY Lieutenant General Paul J. Kern

(RESEARCH, DEVELOPMENT AND Military Deputy to the ASA(RDA)

ACQUISITION), ASA(RDA) 703-697-0356

Attn: SARD-DO

103 Army Pentagon Ellen Purdy**

Washington, D.C. 20310 Senior Operations Research Analyst

purdye@sarda.army.mil

703-604-7006

COL Michael Lavine

lavinem@sarda.army.mil

703-604-7111

Dr. Herbert Fallin

Director for Assessment and Evaluation

fallinh@sarda.army.mil

703-697-2653

Mike Truelove**

Acquisition Analyst

Science Applications International

 Corporation (SAIC)

truelovm@sarda.army.mil

703-604-7013

Paul Amos

Acquisition Analyst

Science Applications International

 Corporation (SAIC)

amosp@sarda.army.mil

703-604-7003

ATACMS-BAT PROJECT OFFICE COL John Holly

Army TACMS-BAT hollyj@redstone.army.mil

ATTN: SFAE-MSL-AB 205-876-1141

Redstone Arsenal, AL 35898-5650
B-5

COMPANY CONTACT

BALL AEROSPACE Larry Jobson

8381 Old Courthouse Road ljobson@ball.com

Vienna, VA 22182 703-917-9125

BARRONCAST, INC. Bruce Barron

215 W. Oakwood Rd. Vice President & General Manager

P.O. Box 138 Bruce@barroncast.com

Oxford, Michigan 48371 248-628-4300

Lee Bennett

Controller

lee@barroncast.com

248-628-4300

Merlin Warner

Sales Engineer

Sales@barroncast.com

BOEING COMPANY Larry Winslow. Director

Seattle, WA 98124 Technology for Phantom Works

Michael W. Johnson*

Defense & Space Group

Vice-President Modeling & Simulation

michael.w.johnson@boeing.com

253-773-3055

CACI Dr. Klaus Dannenberg

1600 Wilson Blvd Senior Vice President

Arlington, VA 22209-2515 kdannenberg@hq.caci.com

703-558-0255

CARDEROCK NAVAL SURFACE Myles Hurwitz

WARFARE CENTER Head, Computer M&S Department

9500 MacArthur Blvd West NSWC/Carderock

Bethesda, MD 20817-5700 mhurwitz@oasys.dt.navy.mil

(301)227-1927

CENTER FOR NAVAL ANALYSIS Dr. Henry Eskew

4400 Ford Avenue 703-824-2254

Alexandria, VA 22302
B-6

COMPANY CONTACT

CHRYSLER CORPORATION Arthur Anderson

800 Chrysler Drive Manager, Vehicle Engineering &

Auburn Hills, MI 48326-2757 Dimension Control

Large Car Platform

Greg Avesian

Manager, Solutions Marketing

gja1@chrysler.com

248-576-6676

CLOSE COMBAT ANTI-TANK WEAPON John Bier

SYSTEM (CCAWS) PROJECT OFFICE Technical Manager

SFAE-MSL-CC-TM Follow on to TOW Program

Redstone Arsenal, AL 35898 john.bier@msl2.redstone.army.mil

256-842-0494

Allen Zumbach

256-876-3200

COLEMAN RESEARCH CORPORATION Tim Thornton

6820 Moquin Drive tthornton@hsv.crc.com

Huntsville, AL 35806 205-922-6000

COMANCHE PROGRAM OFFICE Dan Hollenbaugh

Redstone Arsenal, AL 35898 Program Test Engineer

256-313-4464

COMPUTER SCIENCES CORPORATION Dennis Laack

China Lake, CA Joint Accreditation Support Activity

 (JASA)

dlaack@csc.com

805-987-9641 x130
B-7

COMPANY CONTACT

CRUSADER PROGRAM OFFICE Col Bill Sheaves

SFAE-GCSS-CRE, Building 171 Program Manager

Picatinny Arsenal, NJ 07806-5000 wsheaves@pica.army.mil

973-724-4588

Kevin Fahey

Deputy Program Manager

kfahey@pica.army.mil

973-724-4588

Jim Shields

Chief of Systems Engineering

jshields@pica.army.mil

973-724-7608

DEFENSE ADVANCED RESEARCH Gary Jones, Director

PROJECTS AGENCY (DARPA) Advanced Technology Integration

5201 Leesburg Pike, Suite 503 gjones@darpa.mil

Falls Church, VA 22041 703-681-1484

DEFENSE INTELLIGENCE AGENCY Richard Bernstein**

Bolling Air Force Base Intelligence Officer

Washington, D.C. 20340-5100 bernstr@acq.osd.mil

703-820-1623
B-8

COMPANY CONTACT

DEFENSE MODELING AND Bruce Bailey

SIMULATION OFFICE DoD M&S Working Group

1901 N. Beauregard St bailey@dmso.mil

Alexandria, VA 22311 703-824-3420

Col Kenneth “Crash” Konwin

konwin@dmso.mil

703-998-0660

Dr. Judith Dahmann

jdahmann@msis.dmso.mil

703-998-0660

Jona McKee

jmckee@msosa.dmso.mil

703-998-1637

John Gray

Senior M&S Consultant

jgray@msosa.dmso.mil

703-998-1623

Heikki Joonsar**

hjoonsar@msis.dmso.mil

703-575-2452

DEFENSE SYSTEMS MANAGEMENT Shane Donahue

COLLEGE donahue_patrick@dsmc.dsm.mil

9820 Belvoir Rd 703-805-5411

Fort Belvoir, VA 22060

Randy Zittel

Professor Systems Engineering

Zittelr@dsmc.dsm.mil

703-805-5267

DEPUTY CHIEF OF NAVAL OPERATIONS Dan Fink, Deputy

(LOGISTICS), OPNAV N4 Acquisition Logistics Integration

Navy Pentagon 2000 fink.dan@hq.navy.mil

Washington, DC 20350 703-601-1679
B-9

COMPANY CONTACT

DEPUTY UNDERSECRETARY FOR John D. Binford**

DEFENSE, (INDUSTRIAL AFFAIRS & General Engineer

INSTALLATIONS), INDUSTRIAL jbinford@acq.osd.mil

CAPABILITIES AND ASSESSMENT 703-681-5472

5203 Leesburg Pike, Suite 1403

Falls Church, VA 22041-3466

DRAPER LABORATORY, INC. Dr James Negro

The Charles Stark jnegro@draper.com

555 Technology Square 617-258-3860

Cambridge, MA 02139-3563

Moshe Cohen

moshe@draper.com

617-258-4425

ELECTRIC BOAT CORP Gregory Angelini*

75 Eastern Point Road Software Technology Team Leader,

Groton, CN 06340-4989 Computer Systems Technology

John Porter

CVX Principal Engr.

jporter@ebmail.gdeb.com

860-433-7342

Jim Boudreaux

jboudrea@ebmail.gdeb.com

860-433-6378

ELECTRONIC SYSTEMS COMMAND COL Hoot Gibson

5 Eglin St Director, Modeling, Simulation and

Hanscom AFB, MA 01731-2100 Training PAD

gibsonh@wg.hanscom.af.mil

781-377-6554

EVANS AND SUTHERLAND Paul Hinote

12443 Research Parkway, Suite 303 phinote@es.com

Orlando, FL 32826 407-658-1802
B-10

COMPANY CONTACT

FOCUS: HOPE Joe Petrosky

1400 Oakman Blvd General Manager

Detroit, MI 48238 Center for Advanced Technologies

petrosky@exchange1.focushope.edu

www.focushope.edu

313-494-4274

FORD MOTOR COMPANY Gene Coffman

24500 Glendale Ave Staff Technical Specialist

Detroit, MI 48239 gcoffman@ford.com

313-592-2079

Hwa-Sung Na

Principal Engineering Specialist

hna@ford.com

313-592-2707

FORD MOTOR COMPANY Michael Carty

2000 Rotunda Drive Product Development Center

Dearborn, MI 48121-2053 Mcarty@ford.com

313-317-9213

Richard Riff

Advanced Engineering Center Manager

C3P Project Office

Rriff@ford.com

313-337-9670

GENERAL MOTORS David Chang, Director

30200 Mound Road National Automotive Organization

Warren, MI 48090-9010 Vehicle Synthesis, Analysis &

 Simulation Process Center

LNUSTC1.FZ9MR7@gmeds.com

810-986-6880

Prakash Shrivastava

LNUSTC1.LZ1NYJ@gmeds.com

810-986-2206
B-11

COMPANY CONTACT

HARVARD BUSINESS SCHOOL Professor Dorothy Leonard

Soldiers Field Road 617-495-6637

Morgan Hall Boston, MA 0216

Stefan Thomke

Assistant Professor

sthomke@hbs.harvard.edu

617-495-6569

Ron Fox

Professor Emeritus

rfox@hbs.edu

315-655-4694

HEADQUARTERS DEFENSE Richard Bernstein

INTELLIGENCE AGENCY rbernste@MSIS.dmso.mil

Bolling AFB, MD 202-231-8934

HIGH MOBILITY ARTILLERY Mr. Brock Birdsong

ROCKET SYSTEM (HIMARS) Mechanical Engineer, Missile RDEC

1411 Westview Drive birdsong-cb@redstone.army.mil

Coralville, IA 52241 319-339-8755

IDA Hans Mair

1801 North Beauregard Street Operational Evaluation Division

Alexandria, VA 22311-1772 mairh@asme.org

703-845-2034

INSTITUTE FOR SIMULATION Dr. Ronald Hofer, Ph.D.

TECHNOLOGY Associate Director

3280 Progress Drive rhofer@ist.ucf.edu

Orlando, FL 32826 407-658-5576

Ernie Smart

Director

Governmental & Industrial Partner-

 ships

esmart@ist.ucf.edu

407-658-5014
B-12

COMPANY CONTACT

INTEGRATION & TECHNOLOGY Bob Dodd

BATTLE LAB Military Analyst

CDR, USATRADOC doddr@monroe.army.mil

Attn: ATCD-B 757-727-5715

Ft Monroe, VA 23651-5000

J.T. PARKER ASSOCIATES Ted Parker*

3061 Mimon Road President; ADM (Ret)

Annapolis, MD 21403-1317 jparker@tecnet1.jcte.jcs.mil

301-261-1216

JOHN J. MCMULLEN ASSOCIATES, INC Rob Beadling*

2341 Jefferson Davis Hwy Program Manager

Arlington, VA 22202 CVX Advanced Simulation

rbeadling@jjma.com

703-418-4273

JOHNS HOPKINS UNIVERSITY Dennis Lester

801 University Boulevard, SE dennis.lester@jhuapl.edu

Albuquerque, NM 87106 505-853-1975

JOHNS HOPKINS UNIVERSITY Dr. Mark Moulding

Applied Physics Laboratory Physicist

11100 Johns Hopkins Road mark.moulding@jhuapl.edu

Laurel, MD 20723-6099 240-228-4979

Glenn Gealy

glenn.gealy@jhuapl.edu 2

40-228-6855

Jim Coolahan**

SBA Task Force Technical Support

James.coolahan@jhuapl.edu

240-228-5155

Bob Lutz**

SBA Task Force Technical Support

Robert.lutz@jhuapl.edu

240-953-5000
B-13

COMPANY CONTACT

JOINT SIMULATION SYSTEM (JSIMS) Dave Walker

Orlando, FL Analysis and Integration Manager

david_walker@jsims.com

407-384-2909, ext. 738

JOINT STRIKE FIGHTER COL Phil Faye

PROGRAM OFFICE Director, Requirements (RQ)

1745 Jeff Davis Hwy, Suite 307 fayepa@jast.mil

Arlington, VA 22202 703-602-7390 x6649

KROUSE ASSOCIATES John Krouse

7310 Holly Park Dr. jkrouse@compuserve.com

Concord, OH 44060 440-354-5334

LOCKHEED MARTIN A.J. (Beau) Beauregard, P.E.*

Washington Operations Director, Advanced Programs

1725 Jefferson Davis Highway Aeronautics Sector

Arlington, VA 22202-4127 beau.beauregard@lmco.com

703-413-5711

LOCKHEED MARTIN Linda Poole

PO Box 748 Program Manager

Fort Worth, TX 76101 Virtual Product Development Initiative

 (VPDI)

817-763-2096

LOCKHEED MARTIN AERONAUTICAL Margaret Herald*

SYSTEMS margaret.s.herald@lmco.com

86 South Cobb Drive 770-494-9833

Marietta, GA 30063

LORAL/LOCKHEED MARTIN LTG (Ret) Dan Schroeder

FEDERAL SYSTEMS Consultant

546 Lob Lolly Dr Frigate120@aol.com

Woodlake, NC 910-245-3136

MICROSOFT Mark Walker

One Microsoft Corporation markwalk@microsoft.com

Redmond, WA 98052-6399 425-703-3181
B-14

COMPANY CONTACT

MITRE Marvin Hammond

11493 Sunset Hills Rd mhammond@mitre.org

Reston, VA 20190 703-883-5867

Stuart Starr

starr@mitre.org

703-883-5494

MULTIGEN, INC. Bill MacKrell

6701 Democracy Blvd N.E.Regional Sales Representative

Bethesda, MD 20817 bmackrell@multigen.com

301-571-2466

MULTIGEN, INC. Ray Homan

550 S. Winchester Blvd Exec. V. President and Gen. Manager

San Jose, CA 95128 rhoman@multigen.com

408-261-4100

NAVAL COASTAL SYSTEMS STATION Roger Leete

Panama City, FL leete_roger@ccmail.ncsc.navy.mil

850-234-4265

NAVAL RESEARCH LABORATORY Bill Sandberg

4555 Overlook Avenue, SW Deputy Director, Laboratory for

Washington, D.C. 20375 Computational Physics and

 Fluid Dynamics

sandberg@lcp.nrl.navy.mil

202-767-0526

NAVAL SEA SYSTEMS COMMAND (NAVSEA) Mike Rice

2541 Jefferson Davis Hwy O3KQ

Alexandria, VA 22242-5610 rice_mike_l@hq.navsea.navy.mil

703-602-0887, ext. 327

Stuart Marcus**

05K11

marcus_stuart_j@hq.navsea.navy.mil

703-602-7242, ext. 107
B-15

COMPANY CONTACT

NAVAL SURFACE WARFARE CENTER Joe Stelling**

Port Hueneme Division SBA Task Force Leader

Building 1380 805-228-8719

4363 Missile Way

Port Hueneme, CA 93043-4307

NAVAL UNDERSEA WARFARE CENTER Jeff Feaster

1176 Howell St Engineering ,T&E Dept

Newport, RI 02841 feasterjt@npri70.npt.nuwc.navy.mil

401-841-6892 x22314

Debra Jones

Electronics Engineer

Capital Purchase Program Manager

401-841-2012 x 23351

Tom Kowalczyk

kowalczyktw@code80.npt.nuwc.navy.mil

401-841-6892

NAVY M&S MANAGEMENT OFFICE CAPT Jay Kistler

Crystal City Presidential Towers kistler.jay@hq.navy.mil

Arlington, VA 22202 703 601-1482

NAVY MODELING AND SIMULATION George Phillips

MANAGEMENT OFFICE Navy M&S Point of Contact

N6M1 phillips.george@hq.navy.mil

703-685-6995

NEWPORT NEWS SHIPBUILDING Wayne Evans

4101 Washington Ave Manager, Lifecycle Engineering

Newport News, VA 23607-2770 evans_wa@nns.com

757-380-4876

Dan Selfridge

Lifecycle Engineering

selfridge_dj@nns.com

757-688-6335
B-16

COMPANY CONTACT

NORTHROP GRUMMAN Rahim Munshi

8900 East Washington Blvd Systems Engineer

Pico Rivera, CA 90660 rahim_munshi@email.northgrum.com

562-948-8824

NSSC, SC-21 PROGRAM Janet Jaensch

2531 Jefferson Davis Hwy Director, SC-21 Modeling & Simulation

Arlington, VA 22242 jaensch_janet@hq.navsea.navy.mil

703-602-6453, x 105

Thien Ngo

Asst. Director, Modeling and Simulation

ngo_thien_c@hq.navsea.navy.mil

703-602-6453, x 122

OCI, INCORPORATED Donald Hirsch

1235 Jefferson Davis Highway dhirsch@oc-inc.com

Arlington, VA 22202 703-416-0002

OPTIMETRICS, INC. (OMI) Frank Wysoki*

1 Newport Drive wysoki@omi.com

Forest Hill, MD 21050 703-791-2286

ORIGINAL SIM Christian Rochefort

5524 St. Patrick Regional Sales Manager

Montreal, Canada H4E 1A8 crochefort@originalsim.com

514-766-8868

PLYNETICS EXPRESS Rick Hannan

1067 Centre Road Project Manager

Auburn Hills, MI 48326 Rhannan@plynex.com

248-373-4400

Steven Willis

Program Manager

swillis@plynex.com
B-17

COMPANY CONTACT

PONTIAC-GENERAL MOTORS CORP. Ronald Strayhorn

100 Renaissance Center Vice President, Sales and Service

Detroit, MI 48243-7301 rstrayhorn@pmd72.hbs.edu

313-667-4132

PROGRAM EXECUTIVE OFFICER Lorraine Shea

THEATER AIR DEFENSE (TAD) Director M&S, PEO TAD, NAVSEA

PEO TAD, NAVSEA Shea_Lorraine@hq.navsea.navy.mil

Crystal City

Arlington, VA

RAYTHEON ELECTRONIC SYSTEMS William Stewich

1001 Boston Post Rd Mgr. Business Dev. C4I Programs

Marlboro, MA 01752 william_stewich@ccmail.res.ray.com

508-490-3228

RAYTHEON SYSTEMS COMPANY Randy Boys*

Plano, TX 75086 rboys@rtis.ray.com

972-575-6128

RAYTHEON SYSTEMS COMPANY Steve Olson*

7700 Arlington Blvd steve_olson@fallschurch.esys.com

Falls Church, VA 22204 703-876-1942

RL ENGWALL & ASSOCIATES Dick Engwall*

560 Choptank Cove Court Management Consultant

Annapolis, MD 21401 rlengwall@aol.com

410-571-8623

SCIENCE APPLICATIONS INTERNATIONAL Annie Patenaude

CORPORATION (SAIC) anne.m.patenaude@cpmx.saic.com

8301 Greensboro Dr., Suite 290 703-749-5109

McLean, VA 22102
B-18

COMPANY CONTACT

SCIENCE APPLICATIONS INTERNATIONAL Tom Brown

CORPORATION (SAIC) f.thomas.brown@cpmx.saic.com

2511 Jefferson Davis Highway

Arlington, VA 22202-3911 Dave Thomen

thomend@mail.etas.com

703-414-0189

SCIENCE APPLICATIONS INTERNATIONAL Col (Ret) James Shiflett

CORPORATION (SAIC) Vice President

3045 Technology Parkway 407-207-2725

Orlando, FL 32826-3299

W. San Horton

Senior Technical Manager

407-282-6700

SECRETARY OF THE AIR FORCE Lieutenant General George Muellner

1060 Air Force Pentagon Principal Deputy (Acquisition)

Washington, DC 20301 703-695-7311

MAJ Gary Hopper

SAF/AQPS

hopperg@af.pentagon.mil

703-588-7171

MAJ Gerald S. Smither, Jr.**

SAF/AQIK

smitherg@af.pentagon.mil

703-588-7191

Lt. Col Paul W. Coutee**

SAF/AQRE

couteep@af.pentagon.mil

703-588-5757
B-19

COMPANY CONTACT

SIMULATION TRAINING AND Stan Goodman

INSTRUMENTATION COMMAND goodmans@stricom.army.mil

(STRICOM)

12350 Research Parkway Donald Jones

Orlando, FL 32826 Deputy Product Manager,

Air & Combat Training Systems

jones1@stricom.army.mil

407-384-5142

Les Curless

les_curless@stricom.army.mil

407-384-3860

Dr. Stuart Olson

stuart_olson@stricom.army.mil

407-384-5132

Admiral Piper

pipera@stricom.army.mil

407-384-3935

Phillip Sprinkle

Deputy PM Training Devices

sprinklp@stricom.army.mil

407-384-5202

Ed Trier

triere@stricom.army.mil

407-384-3800

Bill Blanding

blandinb@stricom.army.mil

407-384-5118

SPACE & WARFARE SYSTEMS Phillip Hornick

COMMAND (SPAWAR) Deputy Program Manager M&S

4301 Pacific Hwy C60 Topside hornickp@spawar.navy.mil

San Diego, CA 92110-3127 619-537-0145
B-20

COMPANY CONTACT

SPACE AND MISSILE SYSTEMS CENTER Mark Fagan

180 Skynet Way Chief, Modeling & Simulation Division

Los Angeles Air Force Base, CA 90245 Directorate of Developmental Planning

mark.fagan@losangeles.af.mil

310-363-2461

STANFORD RESEARCH INSTITUTE (SRI) Ralph Toms

INTERNATIONAL Senior Technical Advisor

333 Ravenswood Avenue ralph_toms@sdd.sri.com

Menlo Park, CA 94025 650-859-2852

STANFORD RESEARCH INSTITUTE (SRI) Greg Wilcox*

INTERNATIONAL wilcox@sri.com

1611 North Kent Street 703-247-8467

Arlington, VA 22153

TECHNICAL AUTOMATION SERVICES Murray Cantor

CORPORATION (TASC) Program Management

55 Walkers Brook Dr mrcantor@tasc.com

Reading, MA 01867 617-942-2000

Hal Jones*

Technical Automation Services

 Corporation (TASC)

55 Walkers Brook Drive

Reading, MA 01867-3297

hljones@tasc.com

617-942-2000 x 2207

THE UNIVERSITY OF MICHIGAN Panos Papalambros

College of Engineering Professor and Chair

2236 G.G. Brown Building Mechanical Eng. & Applied Mechanics

Ann Arbor , MI 48109-2125 http://www-

 personal.engin.umich.edu/~pyp/

734-764-8464
B-21

COMPANY CONTACT

TRAINING AND DOCTRINE COMMAND COL Michael Cuff

Systems Manager for Cannon Systems TRADOC Sys. Mgr for Cannon Systems

Commandant U.S. Army Field cuffm@usafas.army.mil

Artillery School (USAFAS) 580-442-6902

Ft Sill, OK 73503-5600

TRIDENT SYSTEMS INC. Nick Karangelen*

10201 Lee Highway President

Suite 300 nkarang@tridsys.com

Fairfax, VA 22030 703-691-7765

U.S. ARMY DIRECTOR OF INFORMATION COL James Deal

SYSTEMS FOR COMMAND, CONTROL, Executive Officer

COMMUNICATIONS & COMPUTERS (DISC4) dealjc@hqda.army.mil

Pentagon 3E458 703-697-5503

Washington, DC 20301

U.S. ARMY LOGISTICS Dr. George Huntley

MANAGEMENT COLLEGE HUNTLEYG@LEE-DNS1.ARMY.MIL

Attn: ATSZ SAM PQMD 804-765-4265

Bldg. 12500

2401 Quarters Rd

Fort Lee, VA 23801-1705

U.S. ARMY MATERIAL SYSTEMS Patrick O’Neill

ANALYSIS ACTIVITY Chief, C4I Branch

AMXSY-CA

392 Hopkins Road Will Brooks

Aberdeen Proving Grounds, MD 21005-5071 A2ATD Program Office

Chief, Simulation Branch

Wbrooks@arl.mil

410-278 – 4946

U.S. ARMY MATERIAL SYSTEMS Jerry Moeller

ANALYSIS ACTIVITY (AMSAA) Gmoell@ria-emh2.army.mil

Attn: AMXSY-A 309-782-7823

Rock Island, IL 61299-7260
B-22

COMPANY CONTACT

U.S. ARMY MODELING AND Linda Stone

SIMULATION OFFICE Stonell@hqda.army.mil

DAMO-ZS 703-601-0010, x 33

1111 Jefferson Davis Hwy

Crystal Gateway North William Dunn

Arlington, VA 22202 Dunnwih@hqda.army.mil

703-601-0011 x 25

U.S. ARMY TANK AND Arthur Adlam

AUTOMOTIVE COMMAND Attn: AMSTA-TR-VP

USATACOM TARDEC Adlama@cc.tacom.army.mil

Warren, MI 48397 – 5000 810-574-8882

Tim Bailey

Attn: AMSTA-TR-N/002

Baileyt@tardec.tacom.army.mil

810-574-5074

Jerry Chapin

Chapinj@cc.tacom.army.mil

810-574-6144

David Holm

Operations Research Analyst

holmd@cc.tacom.army.mil

810-574-6537

Paul Skalny

National Automotive Center

skalnyp@cc.tacom.army.mil

810-574-5436
B-23

COMPANY CONTACT

U.S. ARMY TEST & EVALUATION Dave Brown

COMMAND cdbrown@tec1.apg.army.mil

Aberdeen Proving Ground, MD 21005 410-278-1473

John Haug

jhaug@tec1.apg.army.mil

410-278-1275

LTC Skip Paul

wpaul@tec1.apg.army.mil

410-279-1480

U.S. ATLANTIC COMMAND Lt. Col Robert Strini

Joint Training, Analysis and Chief, Synthetic Theater of War

Simulation Center (JTASC) (STOW) Branch

116 Lakeview Pkwy strini@acom.mil

Suffolk, VA 23435-2699 757-686-7525

UNDER SECRETARY OF DEFENSE Honorable Jacques S. Gansler

ACQUISITION TECHNOLOGY, USD(A&T) 703-695-2381

Pentagon Washington, DC 20301

Dave Oliver

Principal Deputy

oliverdr@acq.osd.mil

703-697-7017

UNITED DEFENSE LIMITED Richard Staiert

PARTNERSHIP (CRUSADER PROGRAM) Project Manager

4800 East River Road Advanced Systems Development

Minneapolis, MN 55421-1498 dick_staiert@udlp.com

612-572-4815

Bob Stratton

Business Development Manager

robert_stratton@udlp.com

612-572-6257

Steven Untz

Crusader Information Mgmt. Manager

steve_untz@udlp.com

612-572-7946
B-24

COMPANY CONTACT

UNITED DEFENSE LIMITED David Wallestad

PARTNERSHIP (CRUSADER PROGRAM) Crusader Program Director

(continued) dave_wallestad@udlp.com

4800 East River Road 612-572-4799

Minneapolis, MN 55421-1498

UNITED DEFENSE LIMITED Warren Richeson

PARTNERSHIP Director

12443 Research Pkwy 407-380-5500

Orlando, FL

Bob Hatton

Manager, Orlando Ops

Training Systems Group

bob_hatton@udlp-orl.com

407-380-5500

UNIVERSITY OF ILLINOIS AT Wayne J. Davis

URBANA-CHAMPAIGN Professor

Urbana, IL 61801 General/Mechanical & Industrial Eng.

w-davis@uiuc.edu

217-333-0531

UNIVERSITY OF IOWA Dr. Edward Haug

The University of Iowa Dept of haug@nads-sc.uiowa.edu

 Mechanical Engineering 319-335-5726

Iowa City, IA 52241

VECTOR RESEARCH, INC Peter Cherry

P.O. Box 1506 Cherryw@vrinet.com

Ann Arbor, MI 48106 734-973-9210
B-25

ENDNOTES

1. The SBA Industry Steering Group (ISG) is

industry’s primary interface with DoD regarding

SBA. The ISG is endorsed by the National

Defense Industrial Association (NDIA), the

International Council on Systems Engineering

(INCOSE), the National Training Systems

Association, the Aerospace Industries Association (AIA), the Institute of Electrical and Electronics Engineers (IEEE) and the Electronic

Industries Association (EIA). Members below are

identified with a *.

2. The Joint SBA Task Force is discussed in Chapter

3, Background. Members below are identified

with a **.
B-26
C-1

APPENDIX C

ABOUT THE AUTHORS
C-2
C-3

APPENDIX C

ABOUT THE AUTHORS

Operations Officer, Marine Aircraft Group31 at MCAS Beaufort, SC. LtCol McKeon

began his acquisition career as the Program

Manager for the USMC Land and Training

Area Requirement System at the Defense

Training and Performance Data Center in

Orlando, FL. Subsequent acquisition assignments include Project Manager for USMC/

USN KC-130 Flight Simulators and as Program Manager for USMC and DoD Virtual

Reality R&D programs at the Naval Air Warfare Center Training Systems Division,

Orlando, FL. LtCol McKeon graduated from

the U.S. Naval Academy in 1978 with a B.S. in

Political Science. He received his M.S. in Computer Systems Management/Information

Technology in 1991 from the Naval Postgraduate School, Monterey, CA. He is a graduate

of the Harvard Graduate School of Business’s

Program for Management Development, the

USMC Amphibious Warfare School, and the

Marine Command and Staff Course.

Lt Col Terence R. Szanto, US Air Force, has

13 years of acquisition experience at the program office, Air Staff, joint, and international

levels. He was most recently assigned to Headquarters Air Force, providing recommendations for $24 billion of the $450 billion Air

Force Future Years Defense Plan. In a joint

acquisition assignment in the Republic of

Korea, he coordinated military technology

cooperation programs. At the Electronic

Systems Center, Hanscom AFB, he upgraded

the B-1 and B-52 aircraft mission planning

systems for smart and conventional weapons

capabilities, and maintained 238 tactical aircraft mission planning systems deployed at 150

LTC Michael V.R. Johnson, Sr., US Army, has

over 20 years of operational and acquisition

experience. His most recent assignment was

with Headquarters, Army Special Operations

Command as Acquisition Branch Chief. Other

acquisition assignments include: Maintenance

Engineer with Communications and Electronics Command; Training with Industry at

Martin Marietta; Project Director at Simulation Training and Instrumentation Command

(STRICOM); and the first Project Director for

the Distributed Interactive Simulation Office.

LTC Johnson’s operational assignments include: aviation assignments in Fulda, Germany

with the 11th Armored Cavalry Regiment, at

Fort Carson, Colorado with the 4th Infantry

Division as a Company Commander in the

Aviation Brigade, Brigade Plans Officer for a

Mechanized Infantry Brigade; and as the

Operational Aviation Officer for the 4th Division, and at Fort Knox as a Cavalry Platoon

Leader with the 2/6 Cavalry Brigade. LTC

Johnson has a Masters Degree in Contract

Acquisition Management and is a graduate of

the U.S. Military Academy, the Harvard

Graduate School of Business’s Program for

Management Development, the Army’s Command and General Staff College, and the

Defense Systems Management College’s

Advanced Program Manager’s Course. He is

rated in the OH-58, AH-1 and UH-1

helicopters. He is also Airborne and Ranger

qualified.

LtCol Mac McKeon, US Marine Corps, has

20 years of aviation service, flying the F-4

Phantom, A-6 Intruder, and the F/A-18

Hornet. His most recent assignment was as the
C-4

locations worldwide. His first acquisition

assignment was at Space Division, Los Angeles AFS, CA. Lt Col Szanto spent a career broadening assignment in F4-E aircraft maintenance

at George AFB, CA. He graduated from the

U.S. Air Force Academy in 1981 with a B.S.

in Engineering Mechanics, and received an

MBA from Pepperdine University. Lt Col

Szanto is a graduate of the Harvard Graduate School of Business’s Program for Management Development, is a distinguished

graduate of Squadron Officers School and Air

Command and Staff College, and will attend

the Industrial College of the Armed Forces

this fall.
Back to Home Page

✺