Defense Science Board 

Task Force 

on 

MISSILE DEFENSE

PHASE III 

MODELING AND SIMULATION

MARCH 2004 

Office of the Under Secretary of Defense 

For Acquisition, Technology, and Logistics 

Washington, D.C. 20301-3140 
 

This report is a product of the Defense Science Board (DSB). 

The DSB is a Federal Advisory Committee established to provide 

independent advice to the Secretary of Defense. Statements, opinions, 

conclusions and recommendations in this report do not necessarily 

represent the official position of the Department of Defense. 

This report is UNCLASSIFIED. 

 

 

This page intentionally left blank


____________________________________________________________ TABLE OF CONTENTS

MISSILE DEFENSE PHASE III ____________________________________________________ iii

TABLE OF CONTENTS

Executive Summary.................................................................................................v

Chapter 1. Scope of the Modeling and Simulation Effort ................................. 1

Chapter 2. Communication and Interface Standards ...................................... 17

Chapter 3. Fidelity ................................................................................................ 29

Chapter 4. Ensuring the Validity of Simulations ............................................. 45

Chapter 5. Additional Opportunities for M&S Contribution......................... 63

Appendix A. Terms of Reference ........................................................................ 73

Appendix B. Task Force Membership................................................................ 75

Appendix C. Glossary of Acronyms and Abbreviations ................................. 77
TABLE OF CONTENTS ____________________________________________________________ 

iv __________________________________________________________ DSB TASK FORCE ON
____________________________________________________________EXECUTIVE SUMMARY 

MISSILE DEFENSE PHASE III ____________________________________________________ v

EXECUTIVE SUMMARY 

Terms of Reference

 Phase I – Reported in May 2002 - Assess the program for 

mid-course discrimination

 Phase II – Reported in October 2002 - examine:

– Capability need (evolution of ballistic missile threats)

– International cooperation

– Boost phase technology

– Battle management, command and control, and communications

 Phase III – Modeling and Simulation – assess:

– The scope of the modeling and simulation effort

– The appropriateness of the level of fidelity

– The impact of communication on the end-to-end models

– The approaches to the validity of simulations

– Additional opportunities for M&S contribution

Terms of Reference

 Phase I – Reported in May 2002 - Assess the program for 

mid-course discrimination

 Phase II – Reported in October 2002 - examine:

– Capability need (evolution of ballistic missile threats)

– International cooperation

– Boost phase technology

– Battle management, command and control, and communications

 Phase III – Modeling and Simulation – assess:

– The scope of the modeling and simulation effort

– The appropriateness of the level of fidelity

– The impact of communication on the end-to-end models

– The approaches to the validity of simulations

– Additional opportunities for M&S contribution

This report covers Phase III of the work of the Defense Science 

Board (DSB) Task Force on Missile Defense. Phase I examined midcourse discrimination. Phase II, a 2002 DSB Summer Study, examined 

the four issues shown in the chart above. Following completion of the 

Phase II report, the Director, Missile Defense Agency (MDA) asked 

the Task Force to examine modeling and simulation (M&S). The 

Under Secretary Defense for Acquisition, Technology and Logistics 

(USD[AT&L]) concurred. The request was further refined to the five 

areas shown. After a brief introduction, this report will cover each of 

the five issues in turn. 
EXECUTIVE SUMMARY ___________________________________________________________ 

vi __________________________________________________________ DSB TASK FORCE ON

An Integrated Layered Defense Against 

Missiles of All Ranges

ALTITUDE (km)

Medium 

Range 

(MRBM) 

Intermediate Range 

(IRBM)

Short

Range 

(SRBM) 

1,200

1,000

800

600

400

200

0 0 3,000 0 5,500 >5,500

0

0 1,000

Midcourse

Boost

Terminal

Ascent

PreBoost

Consequence Management

Intercontinental

(ICBM) 

Burnout

Deployment

Re-entry

RANGE (km)

Capability-based requirement: Defeat missiles of all ranges in 

all phases of flight

Development approach: Spiral development to field 

incremental improvements towards the capability goal

This chart is a reminder of the scope of the effort that modeling 

and simulation must serve. The end capability required is a capability 

to defeat missiles of all ranges in all phases of flight. This capability is 

to be achieved with spiral development—fielding increments of 

capability that build toward the end objective. 
____________________________________________________________EXECUTIVE SUMMARY 

MISSILE DEFENSE PHASE III ____________________________________________________ vii

Bottom Lines

 Credible modeling and simulation (M&S) is essential to 

development, testing and deployment. It provides the only 

way/place that can:

– Account for the how & why of full program spending

– Integrate all the parts

– Solve element problems at the architecture level

– Test at the architecture/integrated system level

– Assess Ballistic Missile Defense (BMD) integrated system quality

and reliability

– Explore the full operating envelope

– Look beyond current plans and programs

– Demonstrate to the user/operator how the system is expected to 

perform in potential scenarios

– Train and operate at the system level

Bottom Lines

 Credible modeling and simulation (M&S) is essential to 

development, testing and deployment. It provides the only 

way/place that can:

– Account for the how & why of full program spending

– Integrate all the parts

– Solve element problems at the architecture level

– Test at the architecture/integrated system level

– Assess Ballistic Missile Defense (BMD) integrated system quality

and reliability

– Explore the full operating envelope

– Look beyond current plans and programs

– Demonstrate to the user/operator how the system is expected to 

perform in potential scenarios

– Train and operate at the system level

This chart and the five that follow offer the bottom line findings 

and recommendations of the Task Force. Subsequent sections will 

provide the detail underwriting these bottom lines. This first chart 

provides a broad picture of the demands on modeling and simulation 

to support MDA development, acquisition, and operation. Given the 

nature of the missile defense program, M&S plays an unusually 

central role. As indicated, it is the only way to account for all the 

pieces of the program, the resources that go into all those pieces, and 

their integration into elements (e.g., ground-based missile defense, 

kinetic boost phase, Aegis, and PAC-3) and into the system of 

systems. It is the only way to assess overall performance and quality. 

M&S is also the only way to train and operate at the system level. The 

remaining bottom lines charts are self-explanatory. 
EXECUTIVE SUMMARY ___________________________________________________________ 

viii _________________________________________________________ DSB TASK FORCE ON

Bottom Lines (cont.) 

 To provide needed analytical underpinnings for system design, 

development, and operations, MDA models and simulations need to 

be more adequately linked and integrated with appropriate fidelity 

– The specific purposes for the various levels of modeling and simulation 

(from component to campaign) need to be explicitly and authoritatively 

defined

– The key MDA models and simulations are legacy models, developed 

largely as stand-alone models and hence are not well designed to fit 

together into the needed modeling and simulation system architectures

– Key models in the family of MDA models and simulations need to be 

vertically integrated – that is validated, accredited component and 

element models need to logically feed system level models that, in turn, 

feed campaign and theater models.

– Element models need to be horizontally and seamlessly linked and

validated for their intended purpose to provide usable system level 

models 

– The fidelity and credibility of the element and component representations 

in the system level models are not adequate to provide confidence in 

model outputs

Bottom Lines (cont.) 

 To provide needed analytical underpinnings for system design, 

development, and operations, MDA models and simulations need to 

be more adequately linked and integrated with appropriate fidelity 

– The specific purposes for the various levels of modeling and simulation 

(from component to campaign) need to be explicitly and authoritatively 

defined

– The key MDA models and simulations are legacy models, developed 

largely as stand-alone models and hence are not well designed to fit 

together into the needed modeling and simulation system architectures

– Key models in the family of MDA models and simulations need to be 

vertically integrated – that is validated, accredited component and 

element models need to logically feed system level models that, in turn, 

feed campaign and theater models.

– Element models need to be horizontally and seamlessly linked and

validated for their intended purpose to provide usable system level 

models 

– The fidelity and credibility of the element and component representations 

in the system level models are not adequate to provide confidence in 

model outputs

Bottom Lines (cont.)

 MDA development, deployment and testing must be supported by 

adequate system level models

– The national teams (Battle Management Command, Control and 

Communications [NTB] and Systems Engineering & Integration [NTS]) 

are developing separate system level models with little communication to 

determine whether a common model would be a more effective solution

– “Systems’’ and command and control, battle management, and 

communications (C2/BM/C) models need to be developed in close 

collaboration to exploit multi-sensor data

– Integrated discrimination, designation emphasis is needed to ensure that 

the timelines, latency, and message content provide for needed data 

fusion in the system design and system simulations

– The MDA should specify the system level model to be used for system 

effectiveness assessment and require, contractually, that all elements 

provide interfacing representations. The Task Force found MDWAR 

promising though not yet adequate to the task.

– The M&S focus needs to shift from continuing to expand currently largely 

adequate element models to filling some glaring gaps (e.g., the 

discrimination function) and on integrated system level M&S

– The simulation architectures do not allow the models and simulations to 

interact as needed to provide the desired insights

Bottom Lines (cont.)

 MDA development, deployment and testing must be supported by 

adequate system level models

– The national teams (Battle Management Command, Control and 

Communications [NTB] and Systems Engineering & Integration [NTS]) 

are developing separate system level models with little communication to 

determine whether a common model would be a more effective solution

– “Systems’’ and command and control, battle management, and 

communications (C2/BM/C) models need to be developed in close 

collaboration to exploit multi-sensor data

– Integrated discrimination, designation emphasis is needed to ensure that 

the timelines, latency, and message content provide for needed data 

fusion in the system design and system simulations

– The MDA should specify the system level model to be used for system 

effectiveness assessment and require, contractually, that all elements 

provide interfacing representations. The Task Force found MDWAR 

promising though not yet adequate to the task.

– The M&S focus needs to shift from continuing to expand currently largely 

adequate element models to filling some glaring gaps (e.g., the 

discrimination function) and on integrated system level M&S

– The simulation architectures do not allow the models and simulations to 

interact as needed to provide the desired insights
____________________________________________________________EXECUTIVE SUMMARY 

MISSILE DEFENSE PHASE III ____________________________________________________ ix

Bottom Lines (cont.)

 Support of future development 

– There needs to be a significant M&S program specifically oriented 

to defined and future conceptual development blocks

– A system level (Missile Defense Warfare Assessment and 

Research Simulation [MDWAR]?) M&S deliverable should be 

developed for each block with appropriate configuration control to 

assure the user/operator that the representation is valid and 

credible for envisioned scenarios

– The system level modeling requirements need to also serve the 

needs of United States Strategic Command (USSTRATCOM) in 

filling their global integrated missile defense responsibilities

Bottom Lines (cont.)

 Validation and accreditation

– Given the complexity of the integrated missile defense task and 

the limitations on validation through flight testing, hardware-in-theloop (HWIL) simulations become key to confidence in M&S validity

– A master “Accreditation Tree” is needed with clearly delineated 

buyoff dates by M&S users, fully linked to the system development 

and deployment schedule

– There is a need for an authoritative library of accredited models at 

levels from phenomenology and physics to integrated system level

models. Accreditation documentation should be specific as to the

purposes for which accredited

Bottom Lines (cont.)

 Validation and accreditation

– Given the complexity of the integrated missile defense task and 

the limitations on validation through flight testing, hardware-in-theloop (HWIL) simulations become key to confidence in M&S validity

– A master “Accreditation Tree” is needed with clearly delineated 

buyoff dates by M&S users, fully linked to the system development 

and deployment schedule

– There is a need for an authoritative library of accredited models at 

levels from phenomenology and physics to integrated system level

models. Accreditation documentation should be specific as to the

purposes for which accredited
EXECUTIVE SUMMARY ___________________________________________________________ 

x___________________________________________________________ DSB TASK FORCE ON

Bottom Lines (cont.)

 A Modeling and Simulation Executive in MDA, reporting to the 

Director, is needed to provide adequate management attention to the 

critical role of Modeling and Simulation

– Responsible for the funding, acceptance, delivery, and maintenance of 

the M&S software for each component, element, and block

– Establish MDA ownership of Systems and C2/BM/C M&S development at 

senior level with responsibility, authority, and budget to provide coherent 

product for the user/operator

 Establish (separate) MDA ownership of the independent verification, 

validation, and accreditation (VV&A) process at a senior level with 

sufficient resources to engage external, rather than just internal, 

expertise to provide MDA and the user/operator a confident 

representation of Block system performance (through assuring that 

each element of the M&S representation is assessed).

– Establish a VV&A Quality Assessment/Audit Team with a balance of

experienced missile defense element algorithm, C2/BM/C, software

experts, and operators that reports to the M&S Executive 

Bottom Lines (cont.)

 A Modeling and Simulation Executive in MDA, reporting to the 

Director, is needed to provide adequate management attention to the 

critical role of Modeling and Simulation

– Responsible for the funding, acceptance, delivery, and maintenance of 

the M&S software for each component, element, and block

– Establish MDA ownership of Systems and C2/BM/C M&S development at 

senior level with responsibility, authority, and budget to provide coherent 

product for the user/operator

 Establish (separate) MDA ownership of the independent verification, 

validation, and accreditation (VV&A) process at a senior level with 

sufficient resources to engage external, rather than just internal, 

expertise to provide MDA and the user/operator a confident 

representation of Block system performance (through assuring that 

each element of the M&S representation is assessed).

– Establish a VV&A Quality Assessment/Audit Team with a balance of

experienced missile defense element algorithm, C2/BM/C, software

experts, and operators that reports to the M&S Executive 
___________________________________________________________________ CHAPTER 1 

MISSILE DEFENSE PHASE III ____________________________________________________ 1

CHAPTER 1. SCOPE OF THE MODELING AND SIMULATION 

 EFFORT

This chapter discusses the scope of the DSB study. The Task Force 

considered overall adequacy to be a part of scope and thus includes 

findings and recommendations on the overall adequacy and 

suitability of current M&S and development plans. 

Scope of Simulation Needs 

 Support program decision points, design trades

 Predict & assesses system performance

 Help develop C2/BM/C capabilities specification

 Help define BMDS spiral and technology development needs 

 Support detailed engineering development

 Explore concepts and system trade-offs

 Guide and augment flight testing

 Support exercises, wargames, training

 Identify block system capabilities against ballistic missiles of all 

ranges in all phases of flight for the user/operator

 Support operational decisions in execution

It is critically important to define the purpose of 

each level of simulation to guide models and 

simulation development

Scope of Simulation Needs 

 Support program decision points, design trades

 Predict & assesses system performance

 Help develop C2/BM/C capabilities specification

 Help define BMDS spiral and technology development needs 

 Support detailed engineering development

 Explore concepts and system trade-offs

 Guide and augment flight testing

 Support exercises, wargames, training

 Identify block system capabilities against ballistic missiles of all 

ranges in all phases of flight for the user/operator

 Support operational decisions in execution

It is critically important to define the purpose of 

each level of simulation to guide models and 

simulation development

The scope required for the Ballistic Missile Defense System 

(BMDS) development blocks is determined by the backgrounds, 

targets, elements, activities, and systems that M&S must describe 

adequately to assess overall performance and contributions of the 

major elements. 

Overall issues include: 

 The range of threats and defensive systems that can 

be analyzed 
SCOPE OF THE MODELING

AND SIMULATION EFFORT_________________________________________________________ 

2___________________________________________________________ DSB TASK FORCE ON

 Ability to model relevant threats and elements for 

initial defensive operations (IDO) and subsequent 

blocks 

 Adequacy of the scope to treat IDO and subsequent 

blocks; complicated in that the systems are still to be 

defined 

 The scope of M&S needed for adequate analyses 

 The more specific issues listed in the chart above are 

taken from the summary MDA briefing at the Joint 

National Integration Center (JNIC), which defined 

the specific elements that implicitly define a 

program of adequate scope 
___________________________________________________________________ CHAPTER 1 

MISSILE DEFENSE PHASE III ____________________________________________________ 3

Pyramid Approach to the Scope of BMDS 

Models and Simulations (M&S)

Core Phenomenology Models

Verification – Validation – Accreditation 

BMDS Element M&S

System Level

Models (End-to-End)

Campaign

& Theater

Models

Validity Requires

Vertical Integration

Decreasing Fidelity

Acceptable

Component Models

Pyramid Approach to the Scope of BMDS 

Models and Simulations (M&S)

Core Phenomenology Models

Verification – Validation – Accreditation 

BMDS Element M&S

System Level

Models (End-to-End)

Campaign

& Theater

Models

Validity Requires

Vertical Integration

Decreasing Fidelity

Acceptable

Component Models

The MDA M&S community makes frequent reference to a 

pyramid of models of increasing scope and reducing fidelity. In 

actual practice, the models are not integrated in horizontal layers and 

there is little linkage to models at higher level of aggregation from 

those below in the pyramid. These shortcomings and their 

consequences are discussed further in following charts and sections. 
SCOPE OF THE MODELING

AND SIMULATION EFFORT_________________________________________________________ 

4___________________________________________________________ DSB TASK FORCE ON

An MDA Notional Horizontal Integration 

to the System Level 

System Level Simulation – MDWAR/EADSIM/EADTB/MDSE

SM-3 SPY-1 Cobra

Dane

UEWR SBIRS GBR XBR EKV PAC

2/3

Sensor THADD

Aegis Ground-Based Missile Defense Patriot THAAD

C2/Battle Management/Communications

 The Task Force found that the integration at the C2/Battle 

Management level, needed to provide the desired system 

integration, is not being simulated and there is no plan to do so

 Simulation is an essential approach to identifying the 

consequences of inadequate system horizontal integration 

An MDA Notional Horizontal Integration 

to the System Level 

System Level Simulation – MDWAR/EADSIM/EADTB/MDSE

SM-3 SPY-1 Cobra

Dane

UEWR SBIRS GBR XBR EKV PAC

2/3

Sensor THADD

Aegis Ground-Based Missile Defense Patriot THAAD

C2/Battle Management/Communications

 The Task Force found that the integration at the C2/Battle 

Management level, needed to provide the desired system 

integration, is not being simulated and there is no plan to do so

 Simulation is an essential approach to identifying the 

consequences of inadequate system horizontal integration 

This chart depicts the MDA notional relationship between the 

system and element models and between models within the vertical 

layers. However, as noted earlier, the current core models have little 

or no horizontal integration. Thus, the evaluation of the elements’ 

separate and integrated benefits is uncertain. The system level 

models do not adequately address the C2/BM/C interface and data 

flow. 
___________________________________________________________________ CHAPTER 1 

MISSILE DEFENSE PHASE III ____________________________________________________ 5

The Heritage and Challenge

 Most current MDA M&S activities are legacy items

– Developments, in some cases, more than a decade old

– Many activities originally dominated by needs and interests of the 

MDAPs

– Developed under different organizations, in isolation, with different 

objectives 

 The MDA objective to treat M&S as a coordinated effort is 

only a few years old

– Focus is primarily at the phenomena and integrated element level

with other hierarchical levels mostly being developed 

autonomously

– A major part of the strategy and effort and a major challenge is

salvaging/converting/migrating/extending legacy items

– A major issue is the appropriate allocation of resources in 

achieving the required capability across the M&S classes and 

elements

The Heritage and Challenge

 Most current MDA M&S activities are legacy items

– Developments, in some cases, more than a decade old

– Many activities originally dominated by needs and interests of the 

MDAPs

– Developed under different organizations, in isolation, with different 

objectives 

 The MDA objective to treat M&S as a coordinated effort is 

only a few years old

– Focus is primarily at the phenomena and integrated element level

with other hierarchical levels mostly being developed 

autonomously

– A major part of the strategy and effort and a major challenge is

salvaging/converting/migrating/extending legacy items

– A major issue is the appropriate allocation of resources in 

achieving the required capability across the M&S classes and 

elements

 There has been a great deal of activity in the Department of 

Defense (DoD) attempting to integrate M&S into programs and 

focusing on the issue of fidelity for performance assessment, 

acquisition support, and training and war games. Most current 

MDA-specific M&S activities are legacy items, sometimes a decade or 

more old, and originally developed with different objectives than 

those now required by MDA. These models are being updated, 

although plans to use the modernized versions are not clear. 

Moreover, there appear to be many redundant codes that are 

maintained for a variety of non-technical reasons. An organized 

attempt to coordinate MDA M&S activity and resources has only 

been in place for a few years and has focused primarily on managing 

the phenomena-, core-, or integration-class simulations. 
SCOPE OF THE MODELING

AND SIMULATION EFFORT_________________________________________________________ 

6___________________________________________________________ DSB TASK FORCE ON

Element, Activity & Integration Model Shortfalls 

 Campaign and theater models - CAPS & JWARS models

– Not integrated 

– Not linked to the models below them in the M&S hierarchy

– Little contribution to increased scope

 System level models

– MDWAR is a widely used MDA system model 

 BMDS model focus is largely on C2/BM/C 

 Available element models appear appropriate, but decoupled

– Models essential for later phases not available (boost phase, 

discrimination)

 Important candidate future elements are not available (e.g., Space 

Tracking and Surveillance System [STSS], space-based interceptor, 

alternative Exoatmospheric Kill Vehicle [EKV])

 Threat, phenomenology models lack essential data (e.g.,hypervelocity 

impact)

 MDA needs to ensure that core phenomenology models are managed 

and maintained to ensure availability of valid inputs for the BMDS 

M&S community

As noted earlier, campaign- Commanders Analysis Planning and 

Simulation (CAPS) and theater-level Joint Theater Warfighting 

System (JTWAR) models are currently decoupled from those below 

them in the M&S pyramid. They do not explicitly guide or draw on 

the results of MDWAR level simulations. They use independent, 

shorthand versions of engagements that are fast and convenient to 

run, but that do not provide confidence in the validity of the insights. 

Force-on-force, mission, and system integration level models are 

redundant, do not treat distinct levels of aggregation, and are not 

closely coupled to element level models. They contain similar but 

independent aggregated models of elements and communication, 

which have not been extensively compared. 

MDWAR appears capable of adequate scope and extension to the 

level of sensors, interceptors, communications, and command control 

(C2) needed for near-term system level studies. Even so, it currently 

has significant limitations discussed on the next chart. 

Ballistic missile defense activity-level models are immature and 

decoupled. Only the battle management and communications (BMC) 

has significant coupling, activity, and potential impact; but it is 
___________________________________________________________________ CHAPTER 1 

MISSILE DEFENSE PHASE III ____________________________________________________ 7

incomplete. Progress is hindered by continuing redefinition of its 

scope. 

Element level models seem individually adequate but are 

decoupled from one another. Important models needed for later 

phases—such as surface- and space-based boost phase intercept 

(BPI)—are not now available, although some are being built. 

Threat, phenomenology, and lethality models are flexible, but 

they lack data of the requisite fidelity in the regimes of interest for 

IDO and later phases. Some data such as hypervelocity impact in 

midcourse and boost cannot be measured in the laboratory or field 

and cannot be credibly deduced from system-level tests. Planned 

improvements will make data more accessible, but not necessarily 

more credible. 
SCOPE OF THE MODELING

AND SIMULATION EFFORT_________________________________________________________ 

8___________________________________________________________ DSB TASK FORCE ON

MDWAR – A Candidate System Level 

(End-to-End) Model

 Modern, object-oriented, event driven model architecture

– Variable fidelity, messages, N-1 element simulations

– Range of variable fidelity models

– Causal for correct system delays

– Insert hardware, models thru gateway—interfaced w/ MDSE 

 Sensor models tested against other systems models 

– Not against detailed element models with some exceptions

– Element program offices reluctant to provide representations for

MDWAR or to support comparison of outputs

 Configuration control via complicated library

 Need to examine, in detail, how and when to incorporate 

needed extensions

MDWAR – A Candidate System Level 

(End-to-End) Model

 Modern, object-oriented, event driven model architecture

– Variable fidelity, messages, N-1 element simulations

– Range of variable fidelity models

– Causal for correct system delays

– Insert hardware, models thru gateway—interfaced w/ MDSE 

 Sensor models tested against other systems models 

– Not against detailed element models with some exceptions

– Element program offices reluctant to provide representations for

MDWAR or to support comparison of outputs

 Configuration control via complicated library

 Need to examine, in detail, how and when to incorporate 

needed extensions

MDWAR is a flexible, event-driven model with modern, objectoriented programming, in which events are modeled in a causal 

manner if the relevant system delays are specified correctly by the 

user. At present, MDWAR has important strengths and weaknesses. 

It has a range of models of sensors, interceptors, and communications 

of variable fidelity that seem adequate to provide element 

simulations of the components. Its suite of simplified radar and 

infrared sensor models have been tested against those in Extended 

Air Defense Simulation (EADSIM) and other systems models of 

similar fidelity and have produced reasonable agreement on 

detection, track, commit, and intercept times. However, these models 

have not been tested against detailed element models. 

MDWAR explicitly treats actual tactical and strategic message 

formats. It treats their transmission, collisions, and delays 

stochastically. It allows for the federation of other communication 

models and hardware thru a flexible JRE-type gateway to federate 

with Missile Defense System Exerciser (MDSE) to support HWIL 

simulations. 

MDWAR uses Ground-based Midcourse Defense (GMD)-

supplied, pre-competitive fly-out tables for the ground-based 
___________________________________________________________________ CHAPTER 1 

MISSILE DEFENSE PHASE III ____________________________________________________ 9

intercept (GBI) and exoatmospheric kill vehicle (EKV) whose 

grounding in and comparison with data is unclear. The treatment of 

communication, interceptors, and C2 is deficient. 

Code versions and results are configuration controlled through a 

detailed library of code, input, and output conditions. This library 

could be used to transmit appropriate versions of MDWAR to users; 

but the current version is complicated, requiring about 800,000 

inputs. 

MDWAR uses the Jet Propulsion Laboratory (JPL)-developed 

SPEEDES operating system for distributed processing to achieve 

running times that are typically several times faster than real time to 

support man-in-the-loop simulations. SPEEDES is a flexible system 

that is appropriate for distributing processing over 10s of 

intermediate-level work stations. There are other ways of distributing 

such simulations over many and faster CPUs, some of them are 

compatible with JAVA and dynamic allocation. SPEEDES is not. 
SCOPE OF THE MODELING

AND SIMULATION EFFORT_________________________________________________________ 

10__________________________________________________________ DSB TASK FORCE ON

Some Needed Contributions to System Engineering

 Specification compliance issue

– Adequacy of the baseline architecture to satisfy the system 

capability

 Sensitivity and breakpoint analysis

 Cost effectiveness trades

– Tradeoffs between operational and system investments to 

minimize cost

– The most cost effective flowdown and allocation of specifications

– The point of diminishing returns for BMDS investment

Some Needed Contributions to System Engineering

 Specification compliance issue

– Adequacy of the baseline architecture to satisfy the system 

capability

 Sensitivity and breakpoint analysis

 Cost effectiveness trades

– Tradeoffs between operational and system investments to 

minimize cost

– The most cost effective flowdown and allocation of specifications

– The point of diminishing returns for BMDS investment

MDA has identified the main areas where M&S needs to 

contribute more to systems engineering. The systems-level models 

now in development, particularly the extension of MDWAR to a 

C2/BM/C model, could address some of these needs, although 

missing models and lack of calibration currently undercuts their 

potential contribution. The models are not sufficiently tested to 

support sensitivity or breakpoint analyses or cost-effectiveness 

trades. The lack of key models for boost phase and discrimination 

limits their ability to guide investment planning or acquisition. 
___________________________________________________________________ CHAPTER 1 

MISSILE DEFENSE PHASE III ____________________________________________________ 11

Findings 

 Campaign & theater level models – CAPS and JWARS

– Not linked to the models that need to provide the inputs 

– Of limited value for such purposes as offense/defense assessment

and trade-offs 

 There is a family of system level models capable of 

operating at various levels of fidelity 

– Models are redundant & decoupled

– Not linked to element models 

– Element representations in these models are not provided by the 

element program offices

– Validity and accreditation is less than robust

– No evidence that M&S used in coherent fashion to support needs

Findings 

 Campaign & theater level models – CAPS and JWARS

– Not linked to the models that need to provide the inputs 

– Of limited value for such purposes as offense/defense assessment

and trade-offs 

 There is a family of system level models capable of 

operating at various levels of fidelity 

– Models are redundant & decoupled

– Not linked to element models 

– Element representations in these models are not provided by the 

element program offices

– Validity and accreditation is less than robust

– No evidence that M&S used in coherent fashion to support needs

Theater- (JWARS) and campaign-level (CAPS) models are over 

simplified and decoupled from system level and element models. 

CAPS and JWARS do not incorporate concepts of operations 

(CONOPS) which limits their utility for offense-defense and related 

policy tradeoffs. 

The number of systems, mission, and force-on-force models 

appear redundant. Further, they are largely decoupled from element 

models. They are not fed properly from lower levels nor are they 

integrated between systems, mission, and force-on-force levels. They 

do not use lower level models provided or approved by the system 

program offices (SPOs). 

As presently configured, M&S has little calibrated predictive 

capability. Thus, the activity serves more as an engineering adjunct 

than as a full M&S program. The Task Force found little evidence that 

M&S is being guided by or used to support either BMC or systems 

engineering priority programs. 
SCOPE OF THE MODELING

AND SIMULATION EFFORT_________________________________________________________ 

12__________________________________________________________ DSB TASK FORCE ON

Findings (cont.) 

 MDWAR system level model

– Adequate for NTB studies, training, & possibly system engineering

– Lack of comparison w/ high fidelity models reduces confidence

– Treats effects of potentially critical CEC data rate limitations with 

low fidelity lumped model approximation

• Prevents analysis of CEC bandwidth limitation effects on other 

systems elements

– Predictive capability limited by lack of models, calibration

– Has not been provided to the user as a model of BMDS block 

performance

 Activity level effort is primarily in C2 and battle 

management

– Incomplete and delayed by frequent changes in scope and 

priorities

– Work on C2/BMC appears to be in conflict with analyses needed 

for confidence in performance of IDO

Findings (cont.) 

 MDWAR system level model

– Adequate for NTB studies, training, & possibly system engineering

– Lack of comparison w/ high fidelity models reduces confidence

– Treats effects of potentially critical CEC data rate limitations with 

low fidelity lumped model approximation

• Prevents analysis of CEC bandwidth limitation effects on other 

systems elements

– Predictive capability limited by lack of models, calibration

– Has not been provided to the user as a model of BMDS block 

performance

 Activity level effort is primarily in C2 and battle 

management

– Incomplete and delayed by frequent changes in scope and 

priorities

– Work on C2/BMC appears to be in conflict with analyses needed 

for confidence in performance of IDO

MDWAR seems adequate for the NTB studies and training 

simulations for which it was designed and possibly for some system 

engineering simulation use. Its flexible treatment of man-in-loop 

issues is useful in studying C2 for global Integrated Missile Defense 

(IMD). However, the lack of comparison of high fidelity element 

models with those in MDWAR for validation and accreditation 

reduces confidence in its results and utility. 

MDWAR cannot treat cooperative engagement capability (CEC) 

rates other than via an effects (lumped) model and treats effects of 

potentially critical CEC data rate limitations with low fidelity lumped 

model approximation. It operates on a ~1 second time step. Hence, it 

cannot treat the kHz rates of CEC and advanced discrimination in a 

fundamental manner. Nor can it capture the impact of CEC and 

advanced discrimination bandwidth requirements. 

MDWAR lacks comparisons and calibration for its current models 

and lacks fundamental models for BPI, forward observation, and 

advanced discrimination, which limits its predictive capability. 

However, with modest extension, it could adequately treat airbreathing threats, U.S. missiles, and offense-defense integration. 
___________________________________________________________________ CHAPTER 1 

MISSILE DEFENSE PHASE III ____________________________________________________ 13

At the activity level, there is significant effort only in improved 

BMC, which is incomplete, delayed by shifting and frequent changes in

scope and priorities, and has conflicting objectives for IDO. 
SCOPE OF THE MODELING

AND SIMULATION EFFORT_________________________________________________________ 

14__________________________________________________________ DSB TASK FORCE ON

Recommendations

 Modeling & simulation development and confident 

utilization need to be improved

– Identify a system level deliverable which captures the operational 

effectiveness of all elements in a given Block and deliver it to

USSTRATCOM to use in support of defense of the Regional 

Combatant Commands (RCCs)

– Provide a truly independent (and consistent) source of verification, 

validation and (separately) accreditation for all models which feed 

into the systems level model

– Integrate the now separate C2/BM/C and systems level 

approaches for model development into one model that assures a 

common architecture (and common M&S representation)

Recommendations

 Modeling & simulation development and confident 

utilization need to be improved

– Identify a system level deliverable which captures the operational 

effectiveness of all elements in a given Block and deliver it to

USSTRATCOM to use in support of defense of the Regional 

Combatant Commands (RCCs)

– Provide a truly independent (and consistent) source of verification, 

validation and (separately) accreditation for all models which feed 

into the systems level model

– Integrate the now separate C2/BM/C and systems level 

approaches for model development into one model that assures a 

common architecture (and common M&S representation)

M&S development and product confidence need to be improved. 

The first step is to identify a system level deliverable that captures the 

operational effectiveness of all elements in a given block and 

provides it to USSTRATCOM to use in support of the relevant 

regional combatant commands (RCCs). A well-documented version 

of MDWAR that has carefully calibrated models for each of the 

elements to be delivered in each block could serve this need. 

Careful calibration requires the development and use of a 

consistent and truly independent source of verification, validation, 

and accreditation for all models which are integrated into the 

systems-level models. 

To have the maximum impact, the C2/BM/C and systems-level 

approaches need to be integrated in a common architecture and a 

common M&S representation. These efforts are currently almost 

completely separate. 
___________________________________________________________________ CHAPTER 1 

MISSILE DEFENSE PHASE III ____________________________________________________ 15

Recommendations (cont.)

 Explore establishing a single systems-level model as the 

standard for integrated system performance assessment 

and trade-offs – MDWAR a likely choice

– Require PMs to provide models or representations of their 

elements and components 

• With standard interface definitions, specifications, and controls

• Accredited for specific, well defined purposes

– Sort out the redundancies in systems-level models in use and 

development

– Establish USSTRATCOM as a customer for the modeling and 

simulation (representing the RCCs) for the block system 

performance across the envelope

Recommendations (cont.)

 Explore establishing a single systems-level model as the 

standard for integrated system performance assessment 

and trade-offs – MDWAR a likely choice

– Require PMs to provide models or representations of their 

elements and components 

• With standard interface definitions, specifications, and controls

• Accredited for specific, well defined purposes

– Sort out the redundancies in systems-level models in use and 

development

– Establish USSTRATCOM as a customer for the modeling and 

simulation (representing the RCCs) for the block system 

performance across the envelope

To realize the full potential of M&S—and to provide the forwardlooking guidance for decisions the MDA needs—MDA should 

specify a systems-level model as the standard for integrated system 

performance assessment and trade-offs and enforce its use in decision 

making. MDWAR is presently the most likely choice. The goal should 

be to produce fixed software builds to specify Emergency Defensive 

Operations (EDO), IDO, and IMD blocks and provide them to the 

combatant commanders for assessment and training. The builds 

should then be updated with each hardware block and modification, 

specifying the capabilities of the current hardware. The foundation of 

this effort should be the development and use of a standard set of 

interface definitions, specifications, and controls. 

MDA should require in contracts that SPOs provide models or 

representations of their elements and components accredited for 

specified purposes, after sorting out the redundancy issue in systemslevel models in use and in development. It should then provide those 

models to USSTRATCOM (representing the RCCs) as a description of 

the block system performance across its expected engagement 

envelope. 
SCOPE OF THE MODELING

AND SIMULATION EFFORT_________________________________________________________ 

16__________________________________________________________ DSB TASK FORCE ON

Recommendations (cont.)

 MDA should

– Assume responsibility for supporting core phenomenology and 

physics models

– Add credible, higher fidelity representations of GBI discrimination 

and kill probability to the GMD system model

– Incorporate boost phase intercept models (and spaced based 

elements, as they evolve) into the integrated systems simulation

– Scope impact of high bandwidth CEC and advanced 

discrimination needs on simulations and predictions of system 

performance

Recommendations (cont.)

 MDA should

– Assume responsibility for supporting core phenomenology and 

physics models

– Add credible, higher fidelity representations of GBI discrimination 

and kill probability to the GMD system model

– Incorporate boost phase intercept models (and spaced based 

elements, as they evolve) into the integrated systems simulation

– Scope impact of high bandwidth CEC and advanced 

discrimination needs on simulations and predictions of system 

performance

To assure that M&S can satisfy both its diagnostic and predictive 

functions, MDA must assume direct responsibility for the needed 

supporting core phenomenology and physics models. 

For M&S to be useful in guiding advanced data fusion and 

discrimination efforts, MDA must add credible, higher fidelity 

representations of ground-based intercept discrimination and kill 

probability to ground-based midcourse defense (GMD) system 

models. 

To guide advanced technology efforts, MDA must incorporate 

boost phase intercept models (and spaced-based elements, as they 

evolve) into integrated systems simulations. The lack of this 

capability is impeding current attempts to evaluate the relative merits 

of various basing modes, define the initial capability levels desired, 

and assess their synergism with current midcourse systems. 

In order to prevent the fragmentation of efforts at advanced fusion 

and discrimination, MDA must scope the impact of high-bandwidth 

CEC and advanced discrimination needs on simulations and 

determine whether they can be adequately modeled in current 

systems simulations such as MDWAR or whether they will require 

fundamentally new approaches. 
___________________________________________________________________ CHAPTER 2 

MISSILE DEFENSE PHASE III ____________________________________________________ 17

CHAPTER 2. COMMUNICATION AND INTERFACE STANDARDS 

Communications Challenges in BMDS

Modeling and Simulation

Two Main Topical Areas

 Area 1: Modeling of communications networks and information exchange

– Incorporating communications realism (finite bandwidth, latency, corruption, 

redundancy, disruption) to establish system level impacts of practical problems

• “Facts of life” effects (weather, EMI, network node availability)

• Inconsistent or erroneous implementation of standards

• Defense suppression (e.g., denial of service attacks)

– Key modeling and simulation issue for area 1: The implications of communications 

modeling on simulation architecture and run time

 Area 2: Federating specialized sims into an end-to-end engineering 

simulation

– Making models of BMDS functions “communicate” with each other and operate 

causally to represent the detection-to-intercept engagement timeline

• Matching “output pins” on one simulation to “input pins” on another simulation

• Developing the ability to derive/validate timing & error budgets for BMDS 

engagement sequences

– Key modeling and simulation issue for area 2: MDA simulations and simulation 

architectures capability to support end-to-end engineering analyses/verification

Communications Challenges in BMDS

Modeling and Simulation

Two Main Topical Areas

 Area 1: Modeling of communications networks and information exchange

– Incorporating communications realism (finite bandwidth, latency, corruption, 

redundancy, disruption) to establish system level impacts of practical problems

• “Facts of life” effects (weather, EMI, network node availability)

• Inconsistent or erroneous implementation of standards

• Defense suppression (e.g., denial of service attacks)

– Key modeling and simulation issue for area 1: The implications of communications 

modeling on simulation architecture and run time

 Area 2: Federating specialized sims into an end-to-end engineering 

simulation

– Making models of BMDS functions “communicate” with each other and operate 

causally to represent the detection-to-intercept engagement timeline

• Matching “output pins” on one simulation to “input pins” on another simulation

• Developing the ability to derive/validate timing & error budgets for BMDS 

engagement sequences

– Key modeling and simulation issue for area 2: MDA simulations and simulation 

architectures capability to support end-to-end engineering analyses/verification

Communications – the physical transmission of information 

between nodes in a simulation and the more abstract relationship 

between models & simulations in an end-to-end federation 

(simulation of simulations) – plays an important role in the 

engineering of the Ballistic Missile Defense System and in the reliable 

assessment of BMDS capabilities. Communications plays a vital role 

in at least two ways. 

The first is in the timeline realism that is introduced by modeling 

the system communications paths, attendant message flows, and 

message service delays. Getting the message traffic flow right – as 

well as the physics-based and human-in-the-loop events that trigger 

that flow – is a large step towards understanding the timeline 

impacts of what is often assumed away (or more positively, assumed 

to be working) in top-level architecture performance assessments. 

The main issue in this arena is the ability to incorporate 
COMMUNICATION AND 

INTERFACE STANDARDS __________________________________________________________ 

18__________________________________________________________ DSB TASK FORCE ON

communications modeling ex post facto without having to pull the 

host simulation apart and without seriously impacting run time. 

The second role of communications is as the linkage between 

models that might make up a federated end-to-end system 

engineering simulation. Such an end-to-end simulation would be 

potentially valuable for engineering design and verification of BMDS 

engagement sequences. Communications in this context applies to 

the ability of element (including C2/BM/C), component, and 

auxiliary models (such as, sensor energy management, data 

association, fusion, and tracking tools; common environmental and 

signature codes; test bed implementations of discrimination and 

decision algorithms) to signal each other and to act upon information 

passed with standardized message formats across standardized 

interfaces. 

Though some models and simulations developed either at MDA 

or at the element program offices were conceived to communicate 

with a centralized executive over Distributed Interactive Simulation 

(DIS) or High Level Architecture (HLA) interfaces, others were not. 

Further, most were not designed to work with each other or to act on 

messages that reflect a network-centric fire-control approach. The 

main issue in this arena is the practicality of constructing engineering 

and verification tools that leverage significant investments at MDA in 

modeling and simulation. Clearly, an unattractive alternative would 

be to architect and build the required end-to-end simulation 

capability at MDA from the ground up. 
___________________________________________________________________ CHAPTER 2 

MISSILE DEFENSE PHASE III ____________________________________________________ 19

Area 1: Findings

 There is a clear need for realistic communications modeling before 

the workability of message-intensive BMDS capabilities can be fully 

understood

– Understand bandwidth constraints on performance of

• Project Hercules Decision Architecture

• Integrated Fire Control (Launch of Remote, Engage of Remote)

• Target Object Map Transmission and Feature Aided Track 

Association

• Plot Fusion (CEC-Like) and Network Based Discrimination

 Message-by-message modeling with queue management and nodeby-node network modeling will throttle simulations designed for real 

time or faster than real time operation (e.g. wargaming applications)

– Given current computing power at the JNIC 

 Non-real time BMD system-level engineering simulations can handle 

almost arbitrary communications fidelity and should in the interest of 

grasping the difference between BMDS concepts and implementation

Area 1: Findings

 There is a clear need for realistic communications modeling before 

the workability of message-intensive BMDS capabilities can be fully 

understood

– Understand bandwidth constraints on performance of

• Project Hercules Decision Architecture

• Integrated Fire Control (Launch of Remote, Engage of Remote)

• Target Object Map Transmission and Feature Aided Track 

Association

• Plot Fusion (CEC-Like) and Network Based Discrimination

 Message-by-message modeling with queue management and nodeby-node network modeling will throttle simulations designed for real 

time or faster than real time operation (e.g. wargaming applications)

– Given current computing power at the JNIC 

 Non-real time BMD system-level engineering simulations can handle 

almost arbitrary communications fidelity and should in the interest of 

grasping the difference between BMDS concepts and implementation

Transformational communications promises to eliminate network 

throughput and message service delays as BMDS issues. In the 

meantime, finite bandwidth communication pipes will be both an 

enabler of advanced BMDS concepts and a potentially powerful 

constraint on performance. Faithful modeling of communications is 

essential to understanding the difference between principles and real 

world practice. 

Message intensive concepts like integrated fire control and data 

fusion are likely to preclude the explicit modeling of communications 

traffic in real time simulations used for CONOPS testing and operator 

training. On the other hand, design and verification simulations can

and should attack the transmission and handling of information down 

to the bit level so that MDA engineers can get a handle on how 

communications—normally assumed to be what it must be in 

architecture-level analyses—modifies ideal BMDS performance. 
COMMUNICATION AND 

INTERFACE STANDARDS __________________________________________________________ 

20__________________________________________________________ DSB TASK FORCE ON

Area 1: Recommendations

 Develop fast running effects models for real time 

simulations which still capture the impact of finite 

communications resources on BMDS operational flexibility 

and performance

 Federate or integrate realistic communications models 

with high fidelity system-level simulations used for 

engineering and verification analysis

– Federate where possible with parts of existing codes

• e.g., use EADTB for definitive Link-16 model

– Initiate new communications modeling efforts to support detailed

analyses of the new message intensive operational concepts 

Area 1: Recommendations

 Develop fast running effects models for real time 

simulations which still capture the impact of finite 

communications resources on BMDS operational flexibility 

and performance

 Federate or integrate realistic communications models 

with high fidelity system-level simulations used for 

engineering and verification analysis

– Federate where possible with parts of existing codes

• e.g., use EADTB for definitive Link-16 model

– Initiate new communications modeling efforts to support detailed

analyses of the new message intensive operational concepts 

War gamers on consoles at the JNIC are not overly troubled by the 

details of networks and communications. On the other hand, they are

interested in real time workloads and the relationship between the 

decisions they make and the BMDS behavior they get. 

Representation of communications throughput, network processing, 

and message latency are important to both cases. Real time 

functioning of war gaming simulations will require a fast running 

“effects model” of the communication system that distills the details 

into simple quality metrics and delays that still permit bona fide

operational problems to be identified. Engineering design and 

verification simulations will require far more detail. In this case, real 

time operation is not the overriding concern. Federation with 

community standard communication models (e.g., Extended Air 

Defense Test Bed [EADTB] for Link-16) should be the first choice, 

with new models being built only as necessary. 
___________________________________________________________________ CHAPTER 2 

MISSILE DEFENSE PHASE III ____________________________________________________ 21

Area 2: End-to-End Capabilities Sought

 A means to determine if all BMDS functions between detection and

intercept can in truth be executed in the engagement time available

 A means of identifying all execution bottlenecks and failure modes 

that are not the result of accidents or poor quality control

 An end-to-end simulation capability that is functionally complete

– Captures all the key factors in probability of engagement success (PES) 

chain

 A federated architecture that maximizes reuse of MDA core models

and element simulations and that can accommodate new models over

time

– Element models and communications models 

– BMD Benchmark for tracking in multiple object environments

– Hercules Test Bed for radar beam and energy management

– Hercules Fusion Toolbox

Area 2: End-to-End Capabilities Sought

 A means to determine if all BMDS functions between detection and

intercept can in truth be executed in the engagement time available

 A means of identifying all execution bottlenecks and failure modes 

that are not the result of accidents or poor quality control

 An end-to-end simulation capability that is functionally complete

– Captures all the key factors in probability of engagement success (PES) 

chain

 A federated architecture that maximizes reuse of MDA core models

and element simulations and that can accommodate new models over

time

– Element models and communications models 

– BMD Benchmark for tracking in multiple object environments

– Hercules Test Bed for radar beam and energy management

– Hercules Fusion Toolbox

MDA’s need for an end-to-end engineering simulation capability 

is closely analogous to computer chip design and manufacture. A 

manufacturer wants to develop a new computer chip that will clock 

at speed X. The design team lays out the connections and logic gates 

and looks for some way to validate the logic before handing the 

blueprint over to the assembly line and committing to the expense of 

full-scale production. Will the chip really clock at speed X or is that 

just a drawing board fantasy? 

The NTS needs to answer similar questions in the system 

engineering job, preferably in the design phase when mistakes are 

cheapest to fix and well before surprising behaviors are caught by 

big-ticket HWIL and live-fire demonstrations. The increasing 

complexity of BMDS interactions over time, and the genuine 

potential for unpredicted and unwelcome emergent behaviors, 

dictates the requirement for an exploratory simulation tool that is 

long overdue in its architecting and development. That same tool will 

also prove highly useful in the verification phase. 

Emphasis on M&S federation and enforcement of 

communications standards recognizes that many of the piece-parts of 

a useful end-to-end system engineering capability already exist, 
COMMUNICATION AND 

INTERFACE STANDARDS __________________________________________________________ 

22__________________________________________________________ DSB TASK FORCE ON

though they have not been developed from a master plan under 

centralized management. There should be no need to rebuild those 

tools. Likewise, new models must find an easy way to link into the 

end-to-end simulation framework as the BMDS evolves. 
___________________________________________________________________ CHAPTER 2 

MISSILE DEFENSE PHASE III ____________________________________________________ 23

Area 2: Federation Issues With Ready Solutions 

Given A Well Thought Out Simulation Architecture

 Common bed down

– Maintaining identical views of the battlefield between simulators

– All element positions in one simulation must be the same in all other 

simulators

 Assigning executive authority

– Accepting start and stop commands from alien simulations

 Execution speed

– Real time to multiples (and sub-multiples) of real time

 Time synchronization

– Measurement and control of time differences between simulators

 Environmental data exchange

– Enforcing common threat and environment

– Ensuring that when one sim affects the environment, all other sims 

know it 

 Perceived data exchange, including tactical messaging system

– Passing digital data to and from tactical battlefield messaging systems

Area 2: Federation Issues With Ready Solutions 

Given A Well Thought Out Simulation Architecture

 Common bed down

– Maintaining identical views of the battlefield between simulators

– All element positions in one simulation must be the same in all other 

simulators

 Assigning executive authority

– Accepting start and stop commands from alien simulations

 Execution speed

– Real time to multiples (and sub-multiples) of real time

 Time synchronization

– Measurement and control of time differences between simulators

 Environmental data exchange

– Enforcing common threat and environment

– Ensuring that when one sim affects the environment, all other sims 

know it 

 Perceived data exchange, including tactical messaging system

– Passing digital data to and from tactical battlefield messaging systems

Though conceptually powerful, M&S federation will require 

careful forethought. Enforcement of functional causality, time 

synchronization, common environments, messaging standards, and 

decision making protocols are clear challenges. 
COMMUNICATION AND 

INTERFACE STANDARDS __________________________________________________________ 

24__________________________________________________________ DSB TASK FORCE ON

Area 2: Findings

 Enforcement of DIS and HLA interface standards means that any 

simulation could interact with another through a master simulation 

executive – in principle

 On the other hand, most missile defense models are not designed to 

communicate except in this most formal way – bits flow both ways, 

what about information?

– Models for the most part not functionally specialized

– Models not designed to plug into some overall intent

 Element models currently limited to tightly integrated representations 

of autonomous performance

– Often, component (radar, C2/BM/C, interceptor) representations cannot 

be extracted

– Messages may be passable across an interface from BMDS C2/BM/C, 

but wholly unusable

 Move towards integrated battle management and fire control (network 

centric warfare) will call for component models that can be individually 

signaled and manipulated

– Models currently reflect the challenge of designing the actual BMDS

Area 2: Findings

 Enforcement of DIS and HLA interface standards means that any 

simulation could interact with another through a master simulation 

executive – in principle

 On the other hand, most missile defense models are not designed to 

communicate except in this most formal way – bits flow both ways, 

what about information?

– Models for the most part not functionally specialized

– Models not designed to plug into some overall intent

 Element models currently limited to tightly integrated representations 

of autonomous performance

– Often, component (radar, C2/BM/C, interceptor) representations cannot 

be extracted

– Messages may be passable across an interface from BMDS C2/BM/C, 

but wholly unusable

 Move towards integrated battle management and fire control (network 

centric warfare) will call for component models that can be individually 

signaled and manipulated

– Models currently reflect the challenge of designing the actual BMDS

Though DIS and HLA standards are not mutually compatible, 

simulations written to either standard should be capable of plugging 

into a master simulation executive. For example, MDWAR has the 

capability to connect to any other simulation that can pass and 

receive messages over a standard communications interface. The 

ability to federate models through a central controller offers the 

potential for development of very powerful system engineering and 

verification tools that make full use of program office investments in 

high fidelity element representations. 

However, the practical difficulty of constructing end-to-end 

simulations of the BMDS mirrors the difficulties of integrating and 

orchestrating missile defense elements in the real world. Models of 

legacy elements tend to be tightly integrated representations of 

autonomous performance and may be unresponsive, by design, to 

coordinating messages from C2/BM/C. Furthermore, as the BMDS 

moves towards more network centric fire control implementations 

such as launch on remote and engage on remote, there will a be a real 

appetite for sensor and weapon models that can be independently 

signaled and manipulated. Unfortunately, such stand-alone modules 

cannot be extracted from many of today’s element models which 
___________________________________________________________________ CHAPTER 2 

MISSILE DEFENSE PHASE III ____________________________________________________ 25

were designed to simulate integrated element performance and not 

the detailed behavior of individual components. 
COMMUNICATION AND 

INTERFACE STANDARDS __________________________________________________________ 

26__________________________________________________________ DSB TASK FORCE ON

Area 2: Findings (cont.)

 C2/BM/C and system engineering national teams both see need for 

end-to-end engineering design/verification tools and the need for a 

federation of interacting models

– However, each are taking their own approach with little-to-no crosstalk

– Performance certifications not likely to be in a common M&S frame of 

reference

 No end-to-end system engineering and verification tool exists today

– Difficult to understand how BMDS could be put on alert without such a 

tool 

– Also not clear to everyone what the problem would be if that tool did not 

exist

 BMDS simulation design driven by a combination of past practice, 

service priorities, computer science, consensus, and happenstance

– No engineering analysis plan against which M&S completeness could be 

judged

– A framework of engagement sequences and event traces exists today 

that could drive modeling and simulation requirements and could 

establish relationships between models, but it is not being used that way 

yet to any effect

Area 2: Findings (cont.)

 C2/BM/C and system engineering national teams both see need for 

end-to-end engineering design/verification tools and the need for a 

federation of interacting models

– However, each are taking their own approach with little-to-no crosstalk

– Performance certifications not likely to be in a common M&S frame of 

reference

 No end-to-end system engineering and verification tool exists today

– Difficult to understand how BMDS could be put on alert without such a 

tool 

– Also not clear to everyone what the problem would be if that tool did not 

exist

 BMDS simulation design driven by a combination of past practice, 

service priorities, computer science, consensus, and happenstance

– No engineering analysis plan against which M&S completeness could be 

judged

– A framework of engagement sequences and event traces exists today 

that could drive modeling and simulation requirements and could 

establish relationships between models, but it is not being used that way 

yet to any effect

Both the NTS and the NTB are pursuing end-to-end simulation 

capabilities for their respective purposes, but they are not in 

meaningful communication with each other. This has resulted in 

parallel development paths and potentially serious disconnects. 

There is presently no end-to-end engineering design and 

verification tool available, nor is there a clear recognition of the need 

for an associated development roadmap. Element model autonomy is 

an obstacle to the end-to-end simulation of BMDS capabilities. One 

could ask the question “How far beyond its design space could 

element X be pushed if interfaced to an integrated BMDS fire control 

and discrimination network?” However, the answer is presently 

outside the ability of MDA system engineers to address rigorously. 

Beyond that, stand-alone element models might be expected to accept 

network messages over an interface, but might also be expected to 

“drop on the floor” all messages it was not programmed to process. 
___________________________________________________________________ CHAPTER 2 

MISSILE DEFENSE PHASE III ____________________________________________________ 27

Area 2: Findings (cont.)

 Communications between models is an ideal not 

implemented in practice

– More than just the enforcement of communications standards

– True model-to-model communications means the ability of each 

model to operate on information from the outside, implement 

decisions, and pass information back across an interface 

Area 2: Findings (cont.)

 Communications between models is an ideal not 

implemented in practice

– More than just the enforcement of communications standards

– True model-to-model communications means the ability of each 

model to operate on information from the outside, implement 

decisions, and pass information back across an interface 

The problem of federating models and simulations which were 

not designed to be interlocked, and which were constructed to no 

formal standards of fidelity, is a formidable challenge, though one 

worth taking on. 
COMMUNICATION AND 

INTERFACE STANDARDS __________________________________________________________ 

28__________________________________________________________ DSB TASK FORCE ON

Area 2: Recommendations

 Develop functionally complete, end-to-end system engineering tools 

for detailed evaluation of event trace timing, validity, and error 

budgets

 Investigate move to a federated simulation architecture for high fidelity 

end-to-end engineering analysis

– Maximize use of sunk MDA investments in specialized M&S 

• BMD Benchmark, Hercules Test Bed, BM/C3 Element Support Task 

(BEST), MDWAR, EADTB, element and component models

– Accommodate upgrades without major recoding

 Add interoperability requirements to all existing system engineering 

contracts for development of BMD component-level models and 

include similar language in all future contracts

– Define and enforce model communications standards

• Move towards network centric warfare will drive emphasis towards

signaling and manipulation of components (launchers, radars), not 

just elements

Area 2: Recommendations

 Develop functionally complete, end-to-end system engineering tools 

for detailed evaluation of event trace timing, validity, and error 

budgets

 Investigate move to a federated simulation architecture for high fidelity 

end-to-end engineering analysis

– Maximize use of sunk MDA investments in specialized M&S 

• BMD Benchmark, Hercules Test Bed, BM/C3 Element Support Task 

(BEST), MDWAR, EADTB, element and component models

– Accommodate upgrades without major recoding

 Add interoperability requirements to all existing system engineering 

contracts for development of BMD component-level models and 

include similar language in all future contracts

– Define and enforce model communications standards

• Move towards network centric warfare will drive emphasis towards

signaling and manipulation of components (launchers, radars), not 

just elements

The recommendations are straightforward: 

1. Develop an end-to-end system engineering tool for 

validation of BMDS engagement sequences and event traces. 

Such a tool is essential for understanding and allocating 

timing and error budgets end-to-end and for understanding, 

during the BMDS design phase, that concepts will really 

work as advertised. 

2. So as to maximize the sunk cost investment in M&S at MDA, 

a federated approach should be investigated. 

3. All future M&S development should be initiated with 

interoperability in mind to keep pace with evolving 

architectural concepts that stress true integration rather than 

the orchestration of elements originally designed for standalone operation. 
___________________________________________________________________ CHAPTER 3 

MISSILE DEFENSE PHASE III ____________________________________________________ 29

CHAPTER 3. FIDELITY 

Fidelity Needs and Issues

 The classes of MDA M&S activities examined range from 

phenomena to campaign - from constructive to live

 “Appropriate fidelity” is the minimum fidelity needed to 

answer the question being addressed

 The detail of parameter inputs should match the fidelity 

and application of model outputs

– Detailed models can include hundreds of parameters

– Parameters that drive models can be difficult to identify 

– Unknown sensitivity of simulation outputs to these parameters 

exists

– A poorly estimated parameter makes it difficult to determine that 

the simulated results may be questionable

Fidelity Needs and Issues

 The classes of MDA M&S activities examined range from 

phenomena to campaign - from constructive to live

 “Appropriate fidelity” is the minimum fidelity needed to 

answer the question being addressed

 The detail of parameter inputs should match the fidelity 

and application of model outputs

– Detailed models can include hundreds of parameters

– Parameters that drive models can be difficult to identify 

– Unknown sensitivity of simulation outputs to these parameters 

exists

– A poorly estimated parameter makes it difficult to determine that 

the simulated results may be questionable

Global Fidelity Concerns by M&S Class 

 Threat, engineering, phenomenology, and lethality models

– e.g., Strategic Scene Generation Model (SSGM), BEST, Parametric 

Endo-Ex-lethality Simulation (PEELS), PEGEM....

– Detailed simulations often founded upon uncertain or unavailable data

– Nuclear environment not significantly addressed

 Element level models

– e.g., THAAD, PAC3, GBM, Aegis, ABL, KE....

– Class most likely to benefit from improved HWIL activity

– Element contractors a key source of M&S data and resources

• Level of involvement and coordination inadequate

– BMDS HWIL evaluation models need to be well anchored via flight test

 Integrated system level MDA models

– e.g., EADSIM, EADTB, MDSE, MDWAR....

– Primarily computer/software and/or operator in the loop based

– Currently focused on demonstrating connectivity – need to also be 

focused on developing architecture and systems

– Element program involvement often uncoordinated

Global Fidelity Concerns by M&S Class 

 Threat, engineering, phenomenology, and lethality models

– e.g., Strategic Scene Generation Model (SSGM), BEST, Parametric 

Endo-Ex-lethality Simulation (PEELS), PEGEM....

– Detailed simulations often founded upon uncertain or unavailable data

– Nuclear environment not significantly addressed

 Element level models

– e.g., THAAD, PAC3, GBM, Aegis, ABL, KE....

– Class most likely to benefit from improved HWIL activity

– Element contractors a key source of M&S data and resources

• Level of involvement and coordination inadequate

– BMDS HWIL evaluation models need to be well anchored via flight test

 Integrated system level MDA models

– e.g., EADSIM, EADTB, MDSE, MDWAR....

– Primarily computer/software and/or operator in the loop based

– Currently focused on demonstrating connectivity – need to also be 

focused on developing architecture and systems

– Element program involvement often uncoordinated
FIDELITY ____________________________________________________________________ 

30__________________________________________________________ DSB TASK FORCE ON

Observations with respect to M&S fidelity have been segregated 

into three general classes: phenomena modeling, element modeling, 

and integrated element or core modeling. 

SSGM, BEST, PEELS, and PEGEM are representative of the 

phenomena class of M&S. This class encompasses underlying threat 

characteristics, environment, and lethality models used to develop 

the BMDS. The primary overall concern with respect to this class is 

the appropriateness, value, and utility of the underlying database of 

measurements used to anchor the models. Additionally no clear 

position has been taken with respect to including a nuclear 

environment into the BMDS. 

THAAD, PAC-3, GMD, ABL, and Kinetic Interceptors (KI) 

modeling activities are representative of the element class of M&S. 

The primary concern with respect to this class of M&S is the role and 

the level of hardware inclusiveness that HWIL simulators are playing 

in the BMDS development. Additionally, roles and responsibilities 

among the various participants in this area are not always clear. 

EADSIM, EADTB, MDSE, and MDWAR are representative of the 

integrated element or core class of M&S. The relative importance and 

the roles of commander-in-the-loop, operator-in-the-loop, and 

hardware-in-the-loop activities and how these activities are balanced 

is an area of concern. Additionally, the purpose for which these M&S 

activities have been developed and how they are employed needs to 

be balanced between competing evaluation and architectural design 

objectives. 
___________________________________________________________________ CHAPTER 3 

MISSILE DEFENSE PHASE III ____________________________________________________ 31

Some Continuing Systemic M&S Fidelity 

and Associated VV&A Challenges

 Technical

– Demands of hard-to-kill (HTK) end game require a higher level of fidelity 

in the ground simulations (both digital and hardware-in-the-loop (HWIL)) 

(Independent Review Team (IRT) 98)

 Programmatic

– Accelerate the upgrade of a HWIL facility adequate for credible testing of 

the EKV in a variety of endgame geometries against a variety of threats 

(IRT 99)

– Vital information needed for development and decisions will come from 

simulations / HWIL anchored with flight test data (IRT 1999)

– VV&A as presently practiced with respect to M&S techniques is not 

sufficiently disciplined to inspire confidence in their use in T&E (DSB 

1999)

– The task force determined that the majority of problems associated with 

DoD software development are a result of undisciplined execution (DSB 

2000a)

 Fiscal

– More investment needed in conceptual modeling, phenomenology, and 

experimentation to gather realistic input data in order to provide decision 

makers with confidence in M&S results (DSB 1999)

Some Continuing Systemic M&S Fidelity 

and Associated VV&A Challenges

 Technical

– Demands of hard-to-kill (HTK) end game require a higher level of fidelity 

in the ground simulations (both digital and hardware-in-the-loop (HWIL)) 

(Independent Review Team (IRT) 98)

 Programmatic

– Accelerate the upgrade of a HWIL facility adequate for credible testing of 

the EKV in a variety of endgame geometries against a variety of threats 

(IRT 99)

– Vital information needed for development and decisions will come from 

simulations / HWIL anchored with flight test data (IRT 1999)

– VV&A as presently practiced with respect to M&S techniques is not 

sufficiently disciplined to inspire confidence in their use in T&E (DSB 

1999)

– The task force determined that the majority of problems associated with 

DoD software development are a result of undisciplined execution (DSB 

2000a)

 Fiscal

– More investment needed in conceptual modeling, phenomenology, and 

experimentation to gather realistic input data in order to provide decision 

makers with confidence in M&S results (DSB 1999)

Many of the current fidelity concerns have been observed and 

identified during prior M&S program reviews and remain relevant 

today. In part it was the existence of these systemic issues that 

became the rational for establishing a coordinated M&S program. 

Such legacy factors continue to limit the M&S program in achieving 

its full potential and value. 
FIDELITY ____________________________________________________________________ 

32__________________________________________________________ DSB TASK FORCE ON

M&S Fidelity Metrics

 Miss distance was traditionally the standard metric for MDA M&S 

fidelity

 HTK as a concept has been verified by flight experiment

– There will never be sufficient resources for flight testing alone to provide 

confidence in the BMDS across the full envelope of operations

– M&S at any level will use less resources than live flight testing

– M&S is the only viable means to explore additional HTK capability space

 Conventional wisdom

– HTK thought to require high fidelity modeling of every nuance in order to 

faithfully evaluate miss distance

 Less often appreciated

– An interceptor sensitive to every nuance is poorly designed 

– The impact of uncertainty and the buildup of consequences

– The threat will never be understood to a comparable degree of accuracy

 The most important factor in HTK capability evaluation at the intercept 

envelope edges will be HWIL testing of acceptable fidelity

M&S Fidelity Metrics

 Miss distance was traditionally the standard metric for MDA M&S 

fidelity

 HTK as a concept has been verified by flight experiment

– There will never be sufficient resources for flight testing alone to provide 

confidence in the BMDS across the full envelope of operations

– M&S at any level will use less resources than live flight testing

– M&S is the only viable means to explore additional HTK capability space

 Conventional wisdom

– HTK thought to require high fidelity modeling of every nuance in order to 

faithfully evaluate miss distance

 Less often appreciated

– An interceptor sensitive to every nuance is poorly designed 

– The impact of uncertainty and the buildup of consequences

– The threat will never be understood to a comparable degree of accuracy

 The most important factor in HTK capability evaluation at the intercept 

envelope edges will be HWIL testing of acceptable fidelity

The traditional metric for missile defense has been its ability to 

faithfully model “hit-to-kill” (HTK) engagements. This metric can 

serve as a basis for defining the issues as well as for providing 

motivation when addressing missile defense M&S fidelity. Although 

HTK has been verified as a concept, the expense associated with 

flight tests are such that there will never be sufficient resources to 

either obtain quality statistical data about any single flight construct 

or to address performance at the edge of the capability envelope. 

Both an understanding of the threat and of interceptor 

performance is critical for a successful HTK. Yet, in the real world, 

the threat will never be understood to the degree preferred. A HTK 

paradox is that a well-understood threat, or assuming a wellunderstood threat, can result in a fragile design whereas a poorly 

understood threat can result in a robust design. 

The principal mechanism available to understand overall missile 

defense performance, within realistic economic constraints, is 

through the use of “hardware-in-the-loop” testing. 
___________________________________________________________________ CHAPTER 3 

MISSILE DEFENSE PHASE III ____________________________________________________ 33

Hardware-in-the-Loop Simulation

 Current Kill Vehicle (KV) HWIL fidelity demand concentrates on the 

seeker

– HWIL seeker with synthetic imaging

– Other factors: HWIL needs to address missile body, IMU, controls, valves, 

etc.

– Subtle integration errors potentially missed

 Expensive to obtain high fidelity in an integrated system but cost is 

still lower than flight testing

– Test limitations may make flight testing less valid at the system level than 

the right level of HWIL testing

 Current HWIL testing tends to involve hardware that is either 

convenient or pushing the state-of-the-art rather than what is required 

or appropriate

 Relative roles and responsibilities with respect to Element Programs, 

Service’s and MDA were not clearly defined

Hardware-in-the-Loop Simulation

 Current Kill Vehicle (KV) HWIL fidelity demand concentrates on the 

seeker

– HWIL seeker with synthetic imaging

– Other factors: HWIL needs to address missile body, IMU, controls, valves, 

etc.

– Subtle integration errors potentially missed

 Expensive to obtain high fidelity in an integrated system but cost is 

still lower than flight testing

– Test limitations may make flight testing less valid at the system level than 

the right level of HWIL testing

 Current HWIL testing tends to involve hardware that is either 

convenient or pushing the state-of-the-art rather than what is required 

or appropriate

 Relative roles and responsibilities with respect to Element Programs, 

Service’s and MDA were not clearly defined

While HWIL testing can be expensive, its costs are likely to be 

modest compared to flight testing. The Task Force found that some 

tested HWIL assemblies are not as inclusive as they might be with 

respect to the actual flight hardware. 

Most HWIL interceptor simulations are focused on evaluation of 

the seeker, associated processors, and related algorithms. This is not 

surprising as the seeker is a critical component. Equally important 

however is the remainder of the interceptor including its more basic 

components such as inertial measurement unit (IMU), controls, 

valves, and functional responses such as missile body flexure 

dynamics. Typically these other factors, even when included, are not 

fully incorporated into the HWIL. 

Some hardware often is only bench mounted and not subjected to 

the dynamic environment necessary to be realistically exercised and, 

as a result, subtle integration errors can be missed. There have been 

exceptions to this HWIL testing observation, such as hover tests, or 

combined synthetic target generation full up dynamic table tests of 

more complete versions of the test article. However, such higher 

quality HWIL testing remains the exception rather than the rule. At 

the element level the critical source of HWIL data, hardware, and 
FIDELITY ____________________________________________________________________ 

34__________________________________________________________ DSB TASK FORCE ON

testing capability resides with the major defense acquisition 

programs (MDAP’s). However the relative roles and responsibilities 

between the MDAPs, the Services, and MDA remain unclear. 
___________________________________________________________________ CHAPTER 3 

MISSILE DEFENSE PHASE III ____________________________________________________ 35

The Impact of Uncertainty

 Uncertainty in M&S fidelity is the difference between “idealized behavior” and 

reality considering the element model, the threat, and the required 

performance

 Uncertainty in fidelity is addressed through sensitivity analysis across the full 

range of the uncertainty

– Identifies what has to be modeled at what level of fidelity

 Earlier DSB report recommendations remain relevant

– Traditionally what is well understood (i.e., the environment) is modeled at high 

levels of fidelity regardless of the sensitivity of the result to the phenomena 

– Conversely what is not well understood is modeled simplistically regardless of its 

importance

 BMDS system design

– Assuming a well defined threat can lead to a brittle system design

– Recognizing the inherent uncertainty leads to a robust system design

 Significant MD threat uncertainty exists due to

– Lack of intelligence data

– Manufacturing realities

– Threat responses

The Impact of Uncertainty

 Uncertainty in M&S fidelity is the difference between “idealized behavior” and 

reality considering the element model, the threat, and the required 

performance

 Uncertainty in fidelity is addressed through sensitivity analysis across the full 

range of the uncertainty

– Identifies what has to be modeled at what level of fidelity

 Earlier DSB report recommendations remain relevant

– Traditionally what is well understood (i.e., the environment) is modeled at high 

levels of fidelity regardless of the sensitivity of the result to the phenomena 

– Conversely what is not well understood is modeled simplistically regardless of its 

importance

 BMDS system design

– Assuming a well defined threat can lead to a brittle system design

– Recognizing the inherent uncertainty leads to a robust system design

 Significant MD threat uncertainty exists due to

– Lack of intelligence data

– Manufacturing realities

– Threat responses

Uncertainty, that is the difference between reality and the model, 

plays a critical role in understanding M&S fidelity. One objective of 

sensitivity analysis is to ensure that those parameters that have the 

most impact on the simulation performance are accounted for during 

model development in proportion to their importance. An earlier 

DSB report found that often what is well understood (for example the 

environment) is modeled more extensively (consequently employing 

more resources) than that which is not well understood, regardless of 

its relative importance to overall simulation performance. This is an 

example of parameter uncertainty and sensitivity analysis being 

improperly applied to missile defense M&S. 

Of singular importance to the current M&S fidelity issue is the 

observation that there will always be uncertainty associated with the 

threat, either because of a lack of intelligence data, because of 

manufacturing uncertainties, or because of deliberate uncertainties 

introduced by the potential enemy. This reality establishes a limit to 

the value of certain modeling parameters. 
FIDELITY ____________________________________________________________________ 

36__________________________________________________________ DSB TASK FORCE ON

Threat Modeling

 BEST is the principal high fidelity M&S activity associated with threat signature 

development

– Brings code languages up to date

– Replicates existing physics code (OSC, XPATCH, etc.)

 The level of fidelity has probably reached the point of diminishing returns 

– Further fidelity of threat signature modeling unlikely to benefit BMDS 

• If assumed representative may be counterproductive 

• An example is early conclusions reached about midcourse discrimination capability via EO/IR observations

 Lethality modeling fidelity suffers from a similar deficiency

– An exception is hypervelocity phenomena - unavailable via ground test

 Sensitivity to input assumptions is a critical modeling result 

Threat

response

Uncertainty in 

threat

Specific 

threat

MD Element

Model

Performance

metric

(i.e., miss distance, etc.)

Threat Uncertainty Space

Threat Modeling

 BEST is the principal high fidelity M&S activity associated with threat signature 

development

– Brings code languages up to date

– Replicates existing physics code (OSC, XPATCH, etc.)

 The level of fidelity has probably reached the point of diminishing returns 

– Further fidelity of threat signature modeling unlikely to benefit BMDS 

• If assumed representative may be counterproductive 

• An example is early conclusions reached about midcourse discrimination capability via EO/IR observations

 Lethality modeling fidelity suffers from a similar deficiency

– An exception is hypervelocity phenomena - unavailable via ground test

 Sensitivity to input assumptions is a critical modeling result 

Threat

response

Uncertainty in 

threat

Specific 

threat

MD Element

Model

Performance

metric

(i.e., miss distance, etc.)

Threat Uncertainty Space

BEST is the acronym for current M&S activity associated with 

phenomena development involving the electro-optic (EO) and radio 

frequency (RF) characteristics of the environment and threat 

signature. BEST brings existing code such as OSC and XPATCH up 

to date and incorporates this code into a common framework. With 

respect to midcourse, at least, this activity has probably reached the 

point of diminishing returns. That is, the level of modeling detail has 

reached that point where it exceeds the expected threat uncertainty. 

This situation is illustrated in the accompanying figure which 

represents how the threat characteristics are mapped by the element 

model to the desired response. Significant activity has been 

expended in characterizing the threat to a level of detail that will 

provide the desired performance; for example, an acceptable miss 

distance. However the uncertainty associated with the threat 

generally exceeds the actual modeled detail signature of the threat. 

The most robust missile defense design is one that will provide 

acceptable performance, given the more realistic uncertainties 

expected. Modeling the system with tighter threat parameters can 

produce overconfidence and may be counterproductive. 
___________________________________________________________________ CHAPTER 3 

MISSILE DEFENSE PHASE III ____________________________________________________ 37

A similar, but less serious, situation exists in the lethality area 

where uncertainty in the threat becomes a basis for modeling and for 

drawing conclusions about performance. However, in this case, there 

is little reliable data on hypervelocity impacts and the associated 

models are poorly anchored in that regard. Activity to anchor such 

models should continue as, and if, such data becomes available. 
FIDELITY ____________________________________________________________________ 

38__________________________________________________________ DSB TASK FORCE ON

BMDS System Level Modeling

 Fidelity at any particular level of the BMDS hierarchy will depend upon 

the problem to be addressed at each level, not on the level itself

 Higher level (i.e., cookie cutter, etc.) solutions generally assume the 

answer will be decoupled from other program inputs

– Not clear decoupling assumptions are valid in current models

 Focus of higher level modeling fidelity has been on interoperability, 

communications, and CONOPS development

 There appears to be no current M&S tool of appropriate fidelity for 

future block architecture development

– Of equal importance to current objectives

– Needs to be considered and designed into developing models

– Unclear that NTB and NTS M&S objectives are in agreement

BMDS System Level Modeling

 Fidelity at any particular level of the BMDS hierarchy will depend upon 

the problem to be addressed at each level, not on the level itself

 Higher level (i.e., cookie cutter, etc.) solutions generally assume the 

answer will be decoupled from other program inputs

– Not clear decoupling assumptions are valid in current models

 Focus of higher level modeling fidelity has been on interoperability, 

communications, and CONOPS development

 There appears to be no current M&S tool of appropriate fidelity for 

future block architecture development

– Of equal importance to current objectives

– Needs to be considered and designed into developing models

– Unclear that NTB and NTS M&S objectives are in agreement

It is often implied that M&S fidelity increases as model detail is 

added to those simulation levels closer to the individual hardware 

components and decreases as modeling is aggregated to higher 

levels. While the accuracy may increase with model detail, fidelity 

might not. Aggregation to higher missile defense levels takes many 

forms and the actual level of fidelity achieved will depend on the 

question being asked and the purpose for which the model was 

developed. 

Higher level system simulations generally assume that the answer 

will be decoupled from other program inputs, yet it is not clear that 

decoupling is valid in current models. These effects are discovered 

only by applying end-to-end engagement simulations, where 

statistical performance is assessed by Monte Carlo iterations of the 

entire set of coupled processes. 

Examining the higher level, or core missile defense models, it is 

clear that the primary program focus in this area has been on the 

interoperability, communications, and CONOPS functions. Although 

several models were available that could contribute to developing 

and evaluating future block architectures, no higher level M&S tool 
___________________________________________________________________ CHAPTER 3 

MISSILE DEFENSE PHASE III ____________________________________________________ 39

was found that was available to uniquely perform this critical 

function. 
FIDELITY ____________________________________________________________________ 

40__________________________________________________________ DSB TASK FORCE ON

Flight Testing and Model Anchoring

 Flight test flexibility will always be limited

– Cost and schedule constraints

– Need to demonstrate progress

– Safety and range capabilities

 Objectives of flight testing

– To provide proof of concept

– To support the acquisition process

– To verify the BMDS model

 The objective determines those flight constructs selected and fidelity 

of the model anchoring 

– Examining system capabilities at the edge of the performance envelope 

requires extending flight test data into new regimes, with associated risks

– Interpolation vs extrapolation will be a factor in construct selection

 Focus should be on operational system risk management not flight

test risk management

– If flight testing is dominated by need for success, then we will not find out 

what we don’t know

– Structure flight test program to support BMDS modeling objectives 

Flight Testing and Model Anchoring

 Flight test flexibility will always be limited

– Cost and schedule constraints

– Need to demonstrate progress

– Safety and range capabilities

 Objectives of flight testing

– To provide proof of concept

– To support the acquisition process

– To verify the BMDS model

 The objective determines those flight constructs selected and fidelity 

of the model anchoring 

– Examining system capabilities at the edge of the performance envelope 

requires extending flight test data into new regimes, with associated risks

– Interpolation vs extrapolation will be a factor in construct selection

 Focus should be on operational system risk management not flight

test risk management

– If flight testing is dominated by need for success, then we will not find out 

what we don’t know

– Structure flight test program to support BMDS modeling objectives 

Linking simulations and models to real world performance is 

critical for fidelity. The ultimate linkage is by way of a full up flight 

test program to anchor the BMDS system model. Such testing is a 

complicated process that will always be limited by a variety of 

constraints. An important question to address is the objective of such 

a flight test program. One purpose of a flight test program is to 

provide a means to manage operational system risk. 

Since there will never be sufficient flight tests to fully characterize 

the BMDS within its operational space, and certainly not at the 

performance envelope edges, flight tests should be constructed with 

emphasis on model verification as a goal, as well as to verify 

operational issues. Focusing on BMDS model verification and 

selecting flight test constructs with this as one objective can increase 

risk. However, data resulting from a broader test spectrum will 

enhance confidence in the M&S program to predict performance for 

the BMDS across a broader spectrum of capabilities. 
___________________________________________________________________ CHAPTER 3 

MISSILE DEFENSE PHASE III ____________________________________________________ 41

Findings

 The fidelity of the threat signature models for midcourse 

are probably at the point of diminishing returns with the 

exception of: 

– Lethality anchoring data (e.g., hypervelocity KKV data) 

– “Advanced Discrimination” and “Forward Based Sensing” 

approaches that do not rely on detailed “a priori” data

 Element HWIL models with appropriate fidelity important 

to explore capabilities not amenable to flight test

– HWIL can and should be exercised at a higher level of integration 

than just the seeker and associated algorithms

– Consider structuring flight tests to also support model validation 

into areas of new capability 

– Manage operational system risk rather than flight test risk

Findings

 The fidelity of the threat signature models for midcourse 

are probably at the point of diminishing returns with the 

exception of: 

– Lethality anchoring data (e.g., hypervelocity KKV data) 

– “Advanced Discrimination” and “Forward Based Sensing” 

approaches that do not rely on detailed “a priori” data

 Element HWIL models with appropriate fidelity important 

to explore capabilities not amenable to flight test

– HWIL can and should be exercised at a higher level of integration 

than just the seeker and associated algorithms

– Consider structuring flight tests to also support model validation 

into areas of new capability 

– Manage operational system risk rather than flight test risk

There are three primary conclusions that can be drawn from the 

previous analysis of fidelity, one for each class of M&S: phenomena, 

element level, and core or integrated element modeling. 

With the possible exception of hypervelocity lethality, threat 

signature, and phenomena modeling of exoatmospheric 

engagements, the fidelity of the threat signature models for 

midcourse is close to the point of diminishing returns and resources 

might be adjusted accordingly. 

Increasing the level of HWIL M&S activity along with an increase 

in the associated degree of hardware involvement for the element 

level programs is important to explore system capabilities at the 

envelope edges. The flight test program should be structured with 

model verification as one primary objective, in addition to its other 

program objectives. 

Integrated element or core model development should address 

architecture and block change development as well as 

interconnectivity. There is no dedicated BMDS architecture model 

capable of evaluating block change alternatives, including a common 

set of metrics for such decision making purposes. 
FIDELITY ____________________________________________________________________ 

42__________________________________________________________ DSB TASK FORCE ON

Recommendations – Threat Phenomena and 

Environmental Fidelity

 Threat phenomena modeling fidelity

– Consider limiting further expenditure of resources in those areas 

of M&S associated with detailed payload threat signature 

development for midcourse

– Conserve related resources associated with the gathering, 

exploitation, testing, measurement, and verification of such 

detailed payload threat signatures

– Continue to obtain data for M&S purposes via intelligence and 

surrogate means as related to the threat and associated delivery

capabilities

– Exploit prior signature development programs by fully reducing 

already available data (e.g., MSX)

 Environmental modeling fidelity

– Model environmental details at a level of fidelity consistent with 

their expected marginal contribution to the BMDS effectiveness

– Establish a corporate position with respect to including nuclear

effects in the BMDS modeling and simulation

Recommendations – Threat Phenomena and 

Environmental Fidelity

 Threat phenomena modeling fidelity

– Consider limiting further expenditure of resources in those areas 

of M&S associated with detailed payload threat signature 

development for midcourse

– Conserve related resources associated with the gathering, 

exploitation, testing, measurement, and verification of such 

detailed payload threat signatures

– Continue to obtain data for M&S purposes via intelligence and 

surrogate means as related to the threat and associated delivery

capabilities

– Exploit prior signature development programs by fully reducing 

already available data (e.g., MSX)

 Environmental modeling fidelity

– Model environmental details at a level of fidelity consistent with 

their expected marginal contribution to the BMDS effectiveness

– Establish a corporate position with respect to including nuclear

effects in the BMDS modeling and simulation

With respect to the threat and phenomena areas, a primary 

recommendation is that MDA consider limiting additional resource 

expenditures associated with highly detailed threat signature 

development normally associated with midcourse. Data associated 

with advanced discrimination techniques which are not “a priori” 

dependent should continue to be obtained. Such a recommendation 

would also be applicable to those supporting functions associated 

with the gathering, measurement, and verification of such detailed 

exoatmospheric data. Threat capability data should continue to be 

obtained, modeled, and existing data sources such as Midcourse 

Space Experiment (MSX) should be exploited. 

MDA should establish a corporate position with respect to the 

inclusion of nuclear effects globally into the BMDS program. The 

importance of sensitivity analysis and its role with respect to 

environmental phenomena code development needs to be 

emphasized. 
___________________________________________________________________ CHAPTER 3 

MISSILE DEFENSE PHASE III ____________________________________________________ 43

Recommendations – Lethality Fidelity

 The fidelity of PEELS has probably reached the point of 

diminishing returns

– Consider limiting further resource expenditures for lethality M&S 

activities which are highly dependent upon threat payload design

details

– Utilize hypervelocity flight test data, when it becomes available, to 

anchor exoatmospheric lethality models

– Continue M&S development activities with respect to the impact of 

the threat intercept/detonation effects

Recommendations – Lethality Fidelity

 The fidelity of PEELS has probably reached the point of 

diminishing returns

– Consider limiting further resource expenditures for lethality M&S 

activities which are highly dependent upon threat payload design

details

– Utilize hypervelocity flight test data, when it becomes available, to 

anchor exoatmospheric lethality models

– Continue M&S development activities with respect to the impact of 

the threat intercept/detonation effects

Also, consider limiting additional resources associated with 

lethality modeling effects that may be unrealistically dependent upon 

threat design details, with the exception of anchoring hypervelocity 

impact models as, and if, relevant data becomes available. 
FIDELITY ____________________________________________________________________ 

44__________________________________________________________ DSB TASK FORCE ON

Recommendations – Element Fidelity

 Improve the general level of fidelity for “hardware-in-the-loop” testing 

of all BMDS elements 

– Expand the role of HWIL within the BMDS development and its elements

– Incorporate feasible hardware components and operational software into 

HWIL simulations

 Harmonize flight test activities, exercises, wargames, and M&S 

program objectives to enhance confidence in BMDS model fidelity

– Jointly develop flight test constructs that support both the demonstration 

of the BMDS effectiveness as well as the extrapolation of its performance 

via M&S

– Consider developing flight test scenarios based upon their value in 

calibrating the M&S toolset at the design envelope edges

– Expand flight performance statistical database and examine higher risk 

“off nominal” operations using enhanced HWIL capabilities

Recommendations – Element Fidelity

 Improve the general level of fidelity for “hardware-in-the-loop” testing 

of all BMDS elements 

– Expand the role of HWIL within the BMDS development and its elements

– Incorporate feasible hardware components and operational software into 

HWIL simulations

 Harmonize flight test activities, exercises, wargames, and M&S 

program objectives to enhance confidence in BMDS model fidelity

– Jointly develop flight test constructs that support both the demonstration 

of the BMDS effectiveness as well as the extrapolation of its performance 

via M&S

– Consider developing flight test scenarios based upon their value in 

calibrating the M&S toolset at the design envelope edges

– Expand flight performance statistical database and examine higher risk 

“off nominal” operations using enhanced HWIL capabilities

With respect to fidelity at the element level, the general role of 

HWIL and its associated degree of hardware inclusion should be 

elevated to provide for greater confidence in the corresponding 

statistical database generated as well as for performance excursions 

into flight regions not addressable in the flight test program. HWIL 

components should be as close to the flight configuration as possible, 

including realistic dynamics as applicable. Most current HWIL 

simulations are primarily centered on the seeker and its associated 

algorithms. 

Developing flight test constructs should be a joint activity 

between the element program office, test and evaluation, and the 

M&S program office. M&S should not be a passive participant in the 

flight test program but should have an active role in the development 

of flight test scenarios, structured specifically to enhance the fidelity 

and ability of the BMDS simulations to extend into unexplored 

regions with confidence. 
___________________________________________________________________ CHAPTER 4 

MISSILE DEFENSE PHASE III ____________________________________________________ 45

CHAPTER 4. ENSURING THE VALIDITY OF SIMULATIONS 

Selected Definitions 

 INDEPENDENT – A knowledgeable and experienced capability not 

involved in the design or development of a model

 VERIFICATION – The process of determining that a model 

implementation accurately represents the developer’s conceptual 

description and specifications. Verifications also evaluates the extent 

to which the model or simulation has been developed using sound 

and established software engineering techniques.

 VALIDATION – The process of determining the degree to which a 

model is an accurate representation of the real-world from the 

perspective of the intended uses of the model.

 ACCREDITATION – The official determination that a model or 

simulation is acceptable for use for a specific purpose.

 BASIS OF CONFIDENCE (BOC) Document – A document that 

identifies the functionality and limitations of a model and provides a 

summary and reference of VV&A activities. The BOC is typically used 

to support accreditation or use decisions. 

Selected Definitions 

 INDEPENDENT – A knowledgeable and experienced capability not 

involved in the design or development of a model

 VERIFICATION – The process of determining that a model 

implementation accurately represents the developer’s conceptual 

description and specifications. Verifications also evaluates the extent 

to which the model or simulation has been developed using sound 

and established software engineering techniques.

 VALIDATION – The process of determining the degree to which a 

model is an accurate representation of the real-world from the 

perspective of the intended uses of the model.

 ACCREDITATION – The official determination that a model or 

simulation is acceptable for use for a specific purpose.

 BASIS OF CONFIDENCE (BOC) Document – A document that 

identifies the functionality and limitations of a model and provides a 

summary and reference of VV&A activities. The BOC is typically used 

to support accreditation or use decisions. 

As noted on this chart, independent verification is the process of 

determining that a model implementation accurately represents the 

developer’s conceptual description and specifications. Verification 

also evaluates the extent to which the model or simulation has been 

developed using sound and established software engineering 

techniques. Validation refers to the process of determining the 

degree to which a model is an accurate representation of the real 

world from the perspective of the intended uses of the model. 

Accreditation is the official certification that the model or simulation 

is acceptable for use for a specific purpose. The Basis of Confidence 

documents are used to record the functionality and limitations of 

models and will be discussed later in this section. 
ENSURING THE

VALIDITY OF SIMULATIONS________________________________________________________ 

46__________________________________________________________ DSB TASK FORCE ON

Key Independent VV&A Needs 

 Area 1 – General scope and content quality of VV&A

– A VV&A effort across element, program, and core models that 

adequately tests the model “physics/algorithms” for intended uses 

and over “envelope corners” to ensure that M&S predictions when 

needed for blocks, spirals and phases of BMD capability are valid 

 Area 2 – Process and scheduling rigor of VV&A

– A VV&A process that uses “best practices” learned throughout 

DoD and elsewhere 

– An adequate schedule definition and discipline in VV&A to tie it to 

the complex blocks, spirals and phases design/deployment 

schedule 

– A talent and funding level for VV&A adequate to meet the above 

needs

Key Independent VV&A Needs 

 Area 1 – General scope and content quality of VV&A

– A VV&A effort across element, program, and core models that 

adequately tests the model “physics/algorithms” for intended uses 

and over “envelope corners” to ensure that M&S predictions when 

needed for blocks, spirals and phases of BMD capability are valid 

 Area 2 – Process and scheduling rigor of VV&A

– A VV&A process that uses “best practices” learned throughout 

DoD and elsewhere 

– An adequate schedule definition and discipline in VV&A to tie it to 

the complex blocks, spirals and phases design/deployment 

schedule 

– A talent and funding level for VV&A adequate to meet the above 

needs

The DSB examination addressed two inter-related fundamentals 

of VV&A. The first is ensuring adequacy and completeness of the 

technical aspects of the models. Are the underlying physics, 

algorithms, and engineering representations adequately checked? 

The second area is the maintenance of uniform process and 

documentation. Does VV&A employ best practices, have adequate 

definition and discipline, and involve acceptance by those 

responsible for the performance of the elements and systems being 

modeled? Is the process adequately synchronized to the blocks, 

spirals, and phases of the MDA development and deployment 

schedule? For both of these basic areas, a corollary consideration 

relates to levels of talent and funding available to implement the 

VV&A process. Another corollary involves “buy-in” to the VV&A 

process by all element program managers, Service operators, war 

gamers, and other stakeholders. 

During the study, much VV&A information from MDA was 

received in the (archived) presentations to plenary sessions. One 

additional splinter session was held on May 19, 2003, to specifically 

discuss VV&A activity with MDA M&S managers. This session was 
___________________________________________________________________ CHAPTER 4 

MISSILE DEFENSE PHASE III ____________________________________________________ 47

particularly beneficial for examining and discussing some of the 

documentation which time would not have otherwise permitted. 

The overall summary observation reported to the full Task Force 

following this splinter meeting, and reflected in what follows in this 

section, was that further confidence was gained in the existence of an 

aggressive MDA process to establish and maintain VV&A rigor, to 

include filling in past gaps. The hard, detailed effort to fully 

synchronize VV&A to the various blocks, spirals, and phases remains 

to be completed. 
ENSURING THE

VALIDITY OF SIMULATIONS________________________________________________________ 

48__________________________________________________________ DSB TASK FORCE ON

VV&A Repository

 Need: A comprehensive web-based VV&A repository with 

secure access to VV&A documentation and track VV&A 

activities

 Recommended approach: improve current system with 

collaboration tools

 Current status

– Hosted at the JNIC / IDC 

– Researched site enhancement requirements and identified 

possible software solutions

– Evaluating alternative hosting sites 

 Issues

– Information assurance, security, and need to know

– JNIC / IDC need to accelerate updating site

VV&A Repository

 Need: A comprehensive web-based VV&A repository with 

secure access to VV&A documentation and track VV&A 

activities

 Recommended approach: improve current system with 

collaboration tools

 Current status

– Hosted at the JNIC / IDC 

– Researched site enhancement requirements and identified 

possible software solutions

– Evaluating alternative hosting sites 

 Issues

– Information assurance, security, and need to know

– JNIC / IDC need to accelerate updating site

This chart summarizes an approach to establish a comprehensive 

and accessible archive for all model VV&A documentation. The Task 

Force strongly favors this effort. 
___________________________________________________________________ CHAPTER 4 

MISSILE DEFENSE PHASE III ____________________________________________________ 49

Findings 

 The credibility of models is mixed 

– There is no consistent verification, validation and accreditation 

process across MDA M&S development

– The Aegis, PAC-3, and THAAD models are generally carefully 

validated for the purposes for which they are accredited

– The GMD model treats some critical parameters such as 

discrimination and overall probability of kill too simplistically for 

adequate credibility 

• Sensitivity to uncertain parameters remains a key issue 

requiring systematic treatment and reporting

– The ABL model departs from the use of variable fidelity 

representations to federate the ABL battle manager directly

– There is no usable model for boost phase intercept systems

– No useable models for space based kinetic interceptors or 

sensors nor space based DEW elements were identified

Findings 

 The credibility of models is mixed 

– There is no consistent verification, validation and accreditation 

process across MDA M&S development

– The Aegis, PAC-3, and THAAD models are generally carefully 

validated for the purposes for which they are accredited

– The GMD model treats some critical parameters such as 

discrimination and overall probability of kill too simplistically for 

adequate credibility 

• Sensitivity to uncertain parameters remains a key issue 

requiring systematic treatment and reporting

– The ABL model departs from the use of variable fidelity 

representations to federate the ABL battle manager directly

– There is no usable model for boost phase intercept systems

– No useable models for space based kinetic interceptors or 

sensors nor space based DEW elements were identified

PAC-3, THAAD, GMD, Aegis, Airborne Laser (ABL) models seem 

adequate for their intended uses, for which they have been carefully 

accredited, but they are decoupled from each other and from the 

system & activity layers of the M&S pyramid. GMD model 

treatments of discrimination and kill probability are oversimplified 

and do not provide confidence in sensitivity assessments. The ABL 

model just federates its battle manager. There are no useful models 

for boost phase kinetic or directed energy intercept from the surface 

or space. 

Current background, target, and lethality models are being 

integrated into a modern package, which will make the data more 

readily available to users. However, the impact of this effort is likely 

to be limited because the current element models embed those data 

packages. The new products appear to be useful primarily to kinetic 

interceptors (KI), for which no system level model is being built. A 

fundamental limit is the lack of data at appropriate hypervelocity 

closing velocities with large masses, which cannot be addressed 

through laboratory or ground tests. Thus, ongoing efforts will make 

the data more accessible, but not more credible. 
ENSURING THE

VALIDITY OF SIMULATIONS________________________________________________________ 

50__________________________________________________________ DSB TASK FORCE ON

Findings (cont.)

 IV&V (and A) needs to go well beyond “business as usual”

– The credibility of the BMD system performance will be uniquely 

dependent on the credibility of the M&S insight

– Need assurance that projections inferred represent the best 

insight that can be offered

 Peer review by “outsiders” is required to assure the fidelity 

of the scientific and engineering representations 

– Not currently accomplished consistently across the program

 The Nuclear Quality Assurance (NQA) approach may be 

required to document all scientific and engineering M&S 

representations and to enable traceable configuration 

control

Findings (cont.)

 IV&V (and A) needs to go well beyond “business as usual”

– The credibility of the BMD system performance will be uniquely 

dependent on the credibility of the M&S insight

– Need assurance that projections inferred represent the best 

insight that can be offered

 Peer review by “outsiders” is required to assure the fidelity 

of the scientific and engineering representations 

– Not currently accomplished consistently across the program

 The Nuclear Quality Assurance (NQA) approach may be 

required to document all scientific and engineering M&S 

representations and to enable traceable configuration 

control

A key element of configuration control is independent verification 

and validation (IV & V) (and accreditation [A]). The Capability 

Maturity Model (CMM) level designation provides a measure of the 

maturity and discipline inherent in a software development team. 

The nuclear reactor community (submarine and civilian) has a rigid 

nuclear quality assurance NQA regimen. All calculations are 

documented in scientific notebooks; all are checked. Use of 

unchecked calculations or codes is simply not permitted. This 

regimen, foreign to MDA, would introduce immediate configuration 

control for spiral development and block change. In addition, it 

would facilitate IV&V and accreditation processes. 
___________________________________________________________________ CHAPTER 4 

MISSILE DEFENSE PHASE III ____________________________________________________ 51

Findings (cont.)

 Area 1 – general scope and content quality of VV&A

– The MDA VV&A process has trended over several years (since SDI) from 

“head to head” comparison of parallel models toward use of fewer

models, each checked by a relatively smaller “associated but 

independent” VV&A effort

– The impact of this trend will increase beyond Block 04 as fewer of the key 

models have benefited from parallel comparison

 Area 2 – process and scheduling rigor of VV&A

– MDA has substantially increased emphasis on M&S VV&A processes

• Importance of incorporating and maintaining established software

practices including VV&A reflected by management personnel 

qualifications, M&S directives and VV&A program content

– However, diversity of VV&A process description details by the element, 

program and core VV&A presentations raises concern

– Proprietary M&S codes must not be permitted to impede IVV&A

 Overall, VV&A comprises ~10-15% of total M&S funding

Findings (cont.)

 Area 1 – general scope and content quality of VV&A

– The MDA VV&A process has trended over several years (since SDI) from 

“head to head” comparison of parallel models toward use of fewer

models, each checked by a relatively smaller “associated but 

independent” VV&A effort

– The impact of this trend will increase beyond Block 04 as fewer of the key 

models have benefited from parallel comparison

 Area 2 – process and scheduling rigor of VV&A

– MDA has substantially increased emphasis on M&S VV&A processes

• Importance of incorporating and maintaining established software

practices including VV&A reflected by management personnel 

qualifications, M&S directives and VV&A program content

– However, diversity of VV&A process description details by the element, 

program and core VV&A presentations raises concern

– Proprietary M&S codes must not be permitted to impede IVV&A

 Overall, VV&A comprises ~10-15% of total M&S funding

Regarding the general method by which missile defense VV&A 

seeks to ensure M&S physics and algorithm quality, area one in the 

discussion above, the Task Force observes the following trend. In 

earlier years it was common practice to develop and compare 

multiple and largely independent models. Such “parallel” models 

may have initially incorporated their own sets of implicit and explicit 

assumptions, but final results were vetted by give and take among 

multiple M&S development teams expert in the phenomenology, 

physics, and engineering areas involved. This practice has generally 

given way to one in which significantly fewer core and element 

models are relied upon for predicting performance parameters. 

VV&A is now generally performed by a number of comparatively 

small groups, one being assigned to each major model. Whether or 

not this situation is sufficient to ensure M&S content quality as 

overall BMDS complexity increases beyond Block 04 will be further 

discussed below. 

With regard to VV&A process specification, configuration control, 

and scheduling rigor, area two discussed above, the Task Force 

observes a substantial increase in emphasis by MDA over the past 

two years, particularly following release of the Weyant M&S report. 

The positive steps taken include addition of management personnel 
ENSURING THE

VALIDITY OF SIMULATIONS________________________________________________________ 

52__________________________________________________________ DSB TASK FORCE ON

well versed in software processes including VV&A. Also 

encouraging is the VV&A direction contained recently revised M&S 

policy. However, from the briefings received it is also observed that 

improvements in policy have not yet fully translated to uniform 

VV&A process description and practices across all the element and 

core models. Accreditation status and currency is also not yet 

uniform; within a single simulation it is common to find some models 

accredited and some not. The fact that some key models are held as 

proprietary to their developers was noted, although it is not clear 

whether or not this creates any impediments to the VV&A process. 

Lastly, it was observed that the appropriate level of resources 

currently being allocated to VV&A represents 10-15 percent of the 

amount dedicated to M&S overall. 
___________________________________________________________________ CHAPTER 4 

MISSILE DEFENSE PHASE III ____________________________________________________ 53

Findings – Area 1

General Scope and Content Quality of VV&A

 For Block 04, non-C2/BM/C VV&A is generally sufficient, largely 

owing to long maturation of key models, although some aspects, e.g., 

track redundancy and cuing, warrant particularly close continued

VV&A attention

 For Block 04, VV&A for any required new C2/BM/C M&S efforts are 

particularly hard pressed by the deployment schedule

 For blocks beyond 04, current VV&A of “physics/algorithms” M&S 

critical to prediction of future element and associated architecture 

performance may be insufficient, e.g., KI and/or SSTS

 Level of VV&A for blocks beyond 04 should reflect increasing BMDS 

and associated M&S complexity, particularly in the C2/BM/C area;

IVV&A level of support may be insufficient to do so

Findings – Area 1

General Scope and Content Quality of VV&A

 For Block 04, non-C2/BM/C VV&A is generally sufficient, largely 

owing to long maturation of key models, although some aspects, e.g., 

track redundancy and cuing, warrant particularly close continued

VV&A attention

 For Block 04, VV&A for any required new C2/BM/C M&S efforts are 

particularly hard pressed by the deployment schedule

 For blocks beyond 04, current VV&A of “physics/algorithms” M&S 

critical to prediction of future element and associated architecture 

performance may be insufficient, e.g., KI and/or SSTS

 Level of VV&A for blocks beyond 04 should reflect increasing BMDS 

and associated M&S complexity, particularly in the C2/BM/C area;

IVV&A level of support may be insufficient to do so

The IVV&A-related conclusions may be categorized into the same 

areas discussed above. For area one, dealing with the M&S technical 

quality, the Task Force concludes that Block 04 models, with the 

notable exceptions of those for C2/BM/C and perhaps multi-element 

functions such as track correlation and weapon cuing, have 

effectively been verified and validated for their intended Block 04 

uses. These models benefit from much prior use and a generally 

greater degree of cross checking with parallel models – in some cases 

going back for decades. 

This general endorsement does not apply to the C2/BM/C area, 

however. In that area, there is a concern that time is very short to 

perform IVV&A for new models such as MDWAR sufficient to 

reliably predict system capabilities and limitations. (Although the 

Block 04 element models are generally sufficient to address a wide 

variety of questions, it is impossible to anticipate in advance all 

element interactions; because of this, it is important to review 

carefully the core models for each use in making program decisions.) 

Beyond Block 04, there are additional difficulties. Many future 

system elements—such as KI and SSTS—are still at immature levels 

of development, use, and VV&A. This conclusion extends to a 
ENSURING THE

VALIDITY OF SIMULATIONS________________________________________________________ 

54__________________________________________________________ DSB TASK FORCE ON

particularly serious concern in the case of C2/BM/C models for 

future blocks, where increasingly numerous systems interactions 

imply increasingly complex VV&A. Reliable C2/BM/C modeling is 

required sooner, not later, to optimize future block architectures and 

designs taking full account of threat and countermeasure spectra. 
___________________________________________________________________ CHAPTER 4 

MISSILE DEFENSE PHASE III ____________________________________________________ 55

Findings – Area 2 

Process and Scheduling of VV&A

 Additional M&S policy and directives (post 2000), along with strong 

management/staff M&S procedural competency, is making positive 

impact on VV&A process/control rigor and uniformity

 No master “accreditation tree” with clearly delineated buyoff (and 

buyoff recommendation) dates by M&S users, including the two 

national teams, for blocks/ spirals/ phases, is currently approved 

(although aspects are in work). Such a document would help tie 

VV&A to the complex BMDS design/deployment process and prevent 

M&S “escapes”

Overall, given the architectural, technological and operational 

complexities of BMDS, a VV&A level of effort of ~10-15% of 

total M&S (characteristic of M&S for much less complex 

systems) should be closely examined for sufficiency. 

Findings – Area 2 

Process and Scheduling of VV&A

 Additional M&S policy and directives (post 2000), along with strong 

management/staff M&S procedural competency, is making positive 

impact on VV&A process/control rigor and uniformity

 No master “accreditation tree” with clearly delineated buyoff (and 

buyoff recommendation) dates by M&S users, including the two 

national teams, for blocks/ spirals/ phases, is currently approved 

(although aspects are in work). Such a document would help tie 

VV&A to the complex BMDS design/deployment process and prevent 

M&S “escapes”

Overall, given the architectural, technological and operational 

complexities of BMDS, a VV&A level of effort of ~10-15% of 

total M&S (characteristic of M&S for much less complex 

systems) should be closely examined for sufficiency. 

Turning to area two, VV&A process management, the Task Force 

first concludes that M&S policy and directives issued after 2000, 

along with added staff M&S procedural competency, is making a 

positive impact on VV&A process and control rigor and uniformity. 

(For example, the basis of confidence documents now being 

developed for the element and core simulations provide good 

documented support for the approval of models; their use is 

encouraged.) However, accreditation (participation, uniformity, 

tracking, and synchronization to development/deployment 

schedules) is found to be an area for improvement. An accreditation 

tree incorporating more detailed planning, wider distribution for 

feedback, and regular updating is thus recommended below. 

It is important that the system element material developers 

assume a stake in the models used to represent the system elements 

developed under their authority. The material developers, the PMs, 

their staff, and their contractors, are most familiar with the details of 

their system. They are most familiar with the key engineering 

aspects of their systems, and they will know the sources of error and 

imperfection. They will be also able to provide informed judgments 

regarding degree of abstraction uncertainty and are likely to know 

about the idiosyncrasies of their system. Additional M&S policies 
ENSURING THE

VALIDITY OF SIMULATIONS________________________________________________________ 

56__________________________________________________________ DSB TASK FORCE ON

and managerial directives are anticipated to make a positive impact 

on needed participation in accreditation. 

A final conclusion that overarches both the technical quality and 

the process management areas is that, given the architectural, 

technological, and operational complexities of BMDS, a VV&A level 

of effort of ~10-15 percent of total M&S (characteristic of M&S for 

much less complex systems) should be closely examined for 

sufficiency. (This statement may apply to VV&A of operational 

codes, but these were not examined by the Task Force.) 

 
___________________________________________________________________ CHAPTER 4 

MISSILE DEFENSE PHASE III ____________________________________________________ 57

Recommendations

 Form an IVV&A Quality Assessment/Audit Team with a 

balance of experienced MD element algorithm, C2/BM/C 

and software experts, as well as representatives from the 

M&S user and MD operator communities 

– Report directly to MDA M&S management to identify any escapes 

in IVV&A uniform technical quality and standard process 

compliance 

– Critique IVV&A milestone dates in view of block development 

schedules

– Make recommendation on all accreditations

– Coordinated with, but smaller than and independent of current 

M&S IPT

Recommendations

 Form an IVV&A Quality Assessment/Audit Team with a 

balance of experienced MD element algorithm, C2/BM/C 

and software experts, as well as representatives from the 

M&S user and MD operator communities 

– Report directly to MDA M&S management to identify any escapes 

in IVV&A uniform technical quality and standard process 

compliance 

– Critique IVV&A milestone dates in view of block development 

schedules

– Make recommendation on all accreditations

– Coordinated with, but smaller than and independent of current 

M&S IPT

Based on the above considerations, findings, and conclusions, the 

DSB Task Force has formulated four recommendations in the area of 

VV&A. The first two are aimed at ensuring the technical quality of 

the VV&A process; the latter two deal with process and funding 

concerns. 

A relatively small standing assessment/audit team should report 

directly to senior MDA M&S management to identify any escapes in 

VV&A uniform technical quality and process compliance, and would 

augment the strong process orientation of the current management 

team. This team should critique VV&A milestone dates in view of 

the block development schedules and help to harmonize the technical 

status of the models with the need dates. It is further suggested that 

this team make recommendations on future accreditations. The team 

would coordinate with and not take the place of the larger M&S 

Integrated Product Team (IPT) currently in place. 
ENSURING THE

VALIDITY OF SIMULATIONS________________________________________________________ 

58__________________________________________________________ DSB TASK FORCE ON

Recommendations (cont.)

 Further refine and document IVV&A strategy to ensure 

M&S validity to the many “envelope corners” of global 

multi-tiered BMD 

– Pay attention to what is not included in the models, to what is 

assumed to work perfectly, and to the sensitivity of the results to 

uncertain input parameters

– “Break the codes” in as many ways as possible (a few examples -

ROEs, shot correlation, penaids, track branching, ....) 

– Expand “threads” strategy to check all functionality of all models

– IVV&A the data catalogs for all models to ensure that key 

variances and other statistics are recorded and tracked

Recommendations (cont.)

 Further refine and document IVV&A strategy to ensure 

M&S validity to the many “envelope corners” of global 

multi-tiered BMD 

– Pay attention to what is not included in the models, to what is 

assumed to work perfectly, and to the sensitivity of the results to 

uncertain input parameters

– “Break the codes” in as many ways as possible (a few examples -

ROEs, shot correlation, penaids, track branching, ....) 

– Expand “threads” strategy to check all functionality of all models

– IVV&A the data catalogs for all models to ensure that key 

variances and other statistics are recorded and tracked

This recommendation is to help ensure that M&S can stand as a 

principal source of confidence in the missile defense systems 

performance over the entire range of expected operating conditions. 

VV&A must identify what gaps exist in the models as well as what is 

being erroneously assumed to work perfectly. The strategy should be 

to “break the codes” in as many ways as possible. 

Examples of this strategy include testing the models under 

multiple threat-defense geometry and saturation scenarios and under 

various countermeasure, shot-to-shot correlation, and track 

branching conditions. One relatively standard tool that the strategy 

should consider is the use of carefully designed code testing 

“threads” to stress all code functionality. In addition, it is important 

that the VV&A process examine the parametric data used in all 

models with regard to accuracy and completeness, as well as include 

all variances necessary for purposes such as Monte Carlo runs. The 

mechanisms for updating and tracking parameter and variance 

values based on hardware testing should also be included. 
___________________________________________________________________ CHAPTER 4 

MISSILE DEFENSE PHASE III ____________________________________________________ 59

Recommendations (cont.)

 Develop, or accelerate development of, accreditation tree 

for all models, but especially for higher level models, that 

addresses previously stated concerns. Approvals should 

probably include that of NTS, and perhaps NTB, but 

should definitely culminate at a higher government level

 MDA should review IVV&A total level of effort to ensure 

that it supports the complexity level and criticality of the 

M&S effort (to address previously stated observation and 

conclusion points)

Recommendations (cont.)

 Develop, or accelerate development of, accreditation tree 

for all models, but especially for higher level models, that 

addresses previously stated concerns. Approvals should 

probably include that of NTS, and perhaps NTB, but 

should definitely culminate at a higher government level

 MDA should review IVV&A total level of effort to ensure 

that it supports the complexity level and criticality of the 

M&S effort (to address previously stated observation and 

conclusion points)

Accreditation sign-offs would likely include that of NTS and 

perhaps NTB, but should definitely culminate at a higher 

government level. This process recommendation is aimed at 

establishing better uniformity and tracking of all M&S verification 

and validity responsibilities and current/projected status. While the 

VV&A material presented in MDA’s briefings during this Task Force 

study did indicate that there was an accreditation process extant for 

each model, these separate accreditation processes did not appear to 

be extremely uniform with clear-cut responsibilities, sequencing, and 

stakeholder participation. The Task Force views this 

recommendation as a way of extending further the existing positive 

trend toward highest quality VV&A process and metrics. 

While the Task Force did not review VV&A funding in depth, this 

recommendation stems mainly from the previously discussed 

observation that MDA VV&A levels, compared to the levels of the 

supported M&S, appear to be comparable to VV&A levels for many 

less complex and less critical (than missile defense) DoD systems. 

Finally, while not a formal recommendation, it is suggested that 

MDA discuss VV&A strategy and tactics with both government and 

integration contractor VV&A personnel associated with the Navy 
ENSURING THE

VALIDITY OF SIMULATIONS________________________________________________________ 

60__________________________________________________________ DSB TASK FORCE ON

Aegis program. Aegis is a highly complex system with many M&S 

parallels to ballistic missile defense. A brief teleconference with M&S 

experts at the integration contractor indicated VV&A for the many 

Aegis models was headed by the Navy (at Dahlgren) for U.S. 

operational ships, and by the integration contractor (at Moorestown) 

for foreign sales ships. Both structures seem to work well, but more 

importantly there appear to be lessons learned of use to MDA in both 

locations. 
___________________________________________________________________ CHAPTER 4 

MISSILE DEFENSE PHASE III ____________________________________________________ 61

Recommendations (cont.)

 MDA should assess the peer review approach used by 

the DOE Labs as a basis for establishing a model for the 

FFRDCs to make a value added contribution as a source 

of IV&V and accreditation

– Depending upon the results of the assessment (conducted by an 

independent review team), MDA should then select and implement 

a FORMAL configuration control and IVV&A approach

Recommendations (cont.)

 MDA should assess the peer review approach used by 

the DOE Labs as a basis for establishing a model for the 

FFRDCs to make a value added contribution as a source 

of IV&V and accreditation

– Depending upon the results of the assessment (conducted by an 

independent review team), MDA should then select and implement 

a FORMAL configuration control and IVV&A approach

Independent review is an important aspect of adequate VV&A. 

Peer review is used effectively for this purpose in the Department of 

Energy (DOE) laboratory system. While there is no complete analog 

available to MDA, it may be that one or more Federally Funded 

Research and Development Centers (FFRDCs) could fill that role. 

MDA should evaluate the potential value of such an approach. 
ENSURING THE

VALIDITY OF SIMULATIONS________________________________________________________ 

62__________________________________________________________ DSB TASK FORCE ON
 

___________________________________________________________________ CHAPTER 5 

MISSILE DEFENSE PHASE III ____________________________________________________ 63

CHAPTER 5. ADDITIONAL OPPORTUNITIES FOR M&S 

 CONTRIBUTION 

The Task Force recognized that additional opportunities exist for 

modeling and simulation in the development, deployment, and 

preparation to operate a ballistic missile defense system. As a 

consequence, issues beyond the scope, breadth, fidelity, element 

“communication,” and verification were considered. 

Findings – Scope and Integration

 Opportunity exists to provide M&S support to:

– Develop the defined Block of capability and to evaluate and assess its 

component, element, and integrated system performance

– Explore trade-offs for evolution towards the end capability – an integrated 

missile defense to defeat missiles of all ranges in all phases of flight

 Current M&S directed at supporting 1, but:

– Current system level architecture approaches rely primarily on kinematic 

analyses at the element level

– Integrated system level M&S needs to offer the capability to assess the 

merits of integrating land, sea, and space based resources

– The need to assess advantages of true system-level integration or true 

fusion of resources has not been adequately served by separate, and 

often uncoordinated, systems level and C2/BM/C M&S efforts

Findings – Scope and Integration

 Opportunity exists to provide M&S support to:

– Develop the defined Block of capability and to evaluate and assess its 

component, element, and integrated system performance

– Explore trade-offs for evolution towards the end capability – an integrated 

missile defense to defeat missiles of all ranges in all phases of flight

 Current M&S directed at supporting 1, but:

– Current system level architecture approaches rely primarily on kinematic 

analyses at the element level

– Integrated system level M&S needs to offer the capability to assess the 

merits of integrating land, sea, and space based resources

– The need to assess advantages of true system-level integration or true 

fusion of resources has not been adequately served by separate, and 

often uncoordinated, systems level and C2/BM/C M&S efforts

MDA tends to rely on modeling and simulation as a “tool” and 

views M&S from a variety of perspectives as part of the cost of doing 

business. The opportunity exists to provide coherence and rigor to 

the M&S process across MDA and to now use credible, 

independently verified and accredited M&S techniques to: 

1. Evaluate the capabilities (needed) for each element of a given 

block 

2. Provide a basis for determining and defining a path ahead 

based upon well understood tradeoffs of performance vice 

capability evolution, schedule, and costs 
 

ADDITIONAL OPPORTUNITIES 

FOR M&S CONTRIBUTION ________________________________________________________ 

64__________________________________________________________ DSB TASK FORCE ON

Because of “grass roots” M&S development at the component, 

subsystems, and even element level and the adoption of a 

“capabilities-based” architecture perspective, there has been 

reluctance to use M&S to “define” systems-level (including 

C2/BM/C) evolution paths. 
 

___________________________________________________________________ CHAPTER 5 

MISSILE DEFENSE PHASE III ____________________________________________________ 65

Findings – Scope and Integration (cont.)

 Need to also focus a significant M&S effort on exploring tradeoffs for 

formulation of future (Block 6 and beyond) architectures

– There does not appear to be a coherent vision for what M&S capabilities 

are needed so that future system evolution paths can be assessed

– Increased attention is needed for M&S to also support developing and 

implementing a long term vision for global missile defense 

– BMDS cannot be flight tested as a system against any significant portion 

of the “operating envelope”

– M&S validated by IV&V and systems integration “tests” will represent THE 

SYSTEM to the user

– The users - STRATCOM and the RCCs - do not have the resident 

capability to develop models of BMDS system performance 

• They are charged with planning and execution with assurance to the 

NCA of how the system and its elements will perform

• BMDS M&S needs to include support for this warfighting function 

Findings – Scope and Integration (cont.)

 Need to also focus a significant M&S effort on exploring tradeoffs for 

formulation of future (Block 6 and beyond) architectures

– There does not appear to be a coherent vision for what M&S capabilities 

are needed so that future system evolution paths can be assessed

– Increased attention is needed for M&S to also support developing and 

implementing a long term vision for global missile defense 

– BMDS cannot be flight tested as a system against any significant portion 

of the “operating envelope”

– M&S validated by IV&V and systems integration “tests” will represent THE 

SYSTEM to the user

– The users - STRATCOM and the RCCs - do not have the resident 

capability to develop models of BMDS system performance 

• They are charged with planning and execution with assurance to the 

NCA of how the system and its elements will perform

• BMDS M&S needs to include support for this warfighting function 

Modeling and simulation should be elevated in status within 

MDA so that it is viewed as the deliverable representation of “the 

system” both to MDA and, importantly, to the user. 

A credible M&S representation of BMDS, as verified in limited 

test bed demonstrations, will provide the user and National 

Command Authority emulation of system-level performance over all 

corners of the envelope against adversary threats yet to be developed. 

M&S thus needs to be the basis for the formulation of Block 6 and 

subsequent architectures. Importantly, if used with rigor across both 

systems and C2/BM/C, M&S can assure that evolution paths are not 

precluded for future BMDS development. 
 

ADDITIONAL OPPORTUNITIES 

FOR M&S CONTRIBUTION ________________________________________________________ 

66__________________________________________________________ DSB TASK FORCE ON

Findings – Scope and Integration (cont.)

 Integrated system effectiveness – achieving the intended end 

capability – is likely to also depend significantly upon offense/defense 

integration 

– The need for USSTRATCOM and/or RCCs to defend areas of 

responsibility (AORs) effectively against potential threats require 

preemptive strikes against some launch areas 

• While not an assigned MDA mission, the M&S program needs to account for it

• MDA-developed systems will be the primary source of precision data for attack

– STRATCOM, to improve the RCC capability to defend their AORs, must 

offer a global missile defense perspective for the benefit of all and provide 

the command and control services necessary to support global missile 

defense

– M&S of the BMDS operation will be the basis for STRATCOM and the

RCCs to:

• Develop contingency plans for the AORs and global missile defense

• Characterize the integration of offensive and defensive forces

• Assess the likely effectiveness of the deployed system against evolving 

threats

– The BMDS M&S “deliverable” must be credibly integrated into force 

planning and execution tools

Findings – Scope and Integration (cont.)

 Integrated system effectiveness – achieving the intended end 

capability – is likely to also depend significantly upon offense/defense 

integration 

– The need for USSTRATCOM and/or RCCs to defend areas of 

responsibility (AORs) effectively against potential threats require 

preemptive strikes against some launch areas 

• While not an assigned MDA mission, the M&S program needs to account for it

• MDA-developed systems will be the primary source of precision data for attack

– STRATCOM, to improve the RCC capability to defend their AORs, must 

offer a global missile defense perspective for the benefit of all and provide 

the command and control services necessary to support global missile 

defense

– M&S of the BMDS operation will be the basis for STRATCOM and the

RCCs to:

• Develop contingency plans for the AORs and global missile defense

• Characterize the integration of offensive and defensive forces

• Assess the likely effectiveness of the deployed system against evolving 

threats

– The BMDS M&S “deliverable” must be credibly integrated into force 

planning and execution tools

Change 2 to the Unified Command Plan assigns global missile 

defense responsibilities to USSTRATCOM. To fill this role, 

USSTRATCOM and the regional combatant commands will need a 

verified capability to model and simulate the system’s behavior. 

Measures of effectiveness of missile defense will include area 

protected (in the United States, or regions of the world with U.S. 

deployed forces, friends, and allies) and launch area denial. The 

latter, a measure of the threat posed from regions of the globe, may 

well only be countered through precision strike of launchers in the 

area. MDA-developed systems will be the primary source of 

precision data for attack. 

 This consideration alone is rationale for strong coupling of 

offense and defense. In addition, USSTRATCOM—with global strike, 

missile defense, and global intelligence, surveillance, and 

reconnaissance (ISR) integration responsibilities—has the need to be 

able to use elements of the system efficiently and effectively. 

Modeling and simulation of the BMDS should account for these 

integration objectives. 
 

___________________________________________________________________ CHAPTER 5 

MISSILE DEFENSE PHASE III ____________________________________________________ 67

Effective modeling and simulation of the global BMDS concept of 

operations will provide USSTRATCOM and the regional combatant 

commanders with the capability to develop architecture evolution 

plans, understand the performance of the system in spiral 

development, and plan (and train) for scenarios of the future. 
 

ADDITIONAL OPPORTUNITIES 

FOR M&S CONTRIBUTION ________________________________________________________ 

68__________________________________________________________ DSB TASK FORCE ON

Findings – Scope and Integration (cont.)

 Effective M&S support of block evolution/system 

integration phasing within spiral development approach 

demands effective M&S configuration control 

– In an environment of day-to-day changes, “phased” system 

integration as a result of test bed evaluations, spiral development, 

and block changes to architecture (capability) options

 Air defense is not part of the current MDA M&S evaluation 

task 

– Several of the elements (e.g., PAC-3, Aegis) have air defense 

responsibilities and obvious self defense priorities 

– The systems level BMDS M&S must include representation of the 

resource allocation and C2/BM/C issues resident in multi-mission 

elements to address systems-level performance

Findings – Scope and Integration (cont.)

 Effective M&S support of block evolution/system 

integration phasing within spiral development approach 

demands effective M&S configuration control 

– In an environment of day-to-day changes, “phased” system 

integration as a result of test bed evaluations, spiral development, 

and block changes to architecture (capability) options

 Air defense is not part of the current MDA M&S evaluation 

task 

– Several of the elements (e.g., PAC-3, Aegis) have air defense 

responsibilities and obvious self defense priorities 

– The systems level BMDS M&S must include representation of the 

resource allocation and C2/BM/C issues resident in multi-mission 

elements to address systems-level performance

The entrepreneurship of code developers, the evolution of 

components, subsystems, and elements of the system through spiral 

development, and the lack of systems engineering driven 

requirements modus operandi, makes “configuration control” of 

modeling and simulation produce a formidable set of challenges. 

Modeling and simulation for elements of the system and for 

future BMDS evolution do not reflect the capabilities inherent or 

essential for operation in many of the elements. For example, Aegis 

and Patriot self-defense objectives against cruise missiles are not 

accounted for in BMDS models. Thus, the resource allocation models 

viewed to be available for ballistic missile defense applications are in 

question. 
 

___________________________________________________________________ CHAPTER 5 

MISSILE DEFENSE PHASE III ____________________________________________________ 69

Findings

 Modeling & simulation evolution towards a systems level 

product suitable for assessing capabilities and making 

systems trade-offs for future spirals appears diffused, 

complicated, and slowed by the split between systems 

level and C2/BM/C M&S developments 

 BMDS M&S activities need to fully account for net-centric 

and GIG-BE evolution

– These activities need to provide for rapid, collaborative planning, 

command and control support for the engagement environments 

facing STRATCOM and the RCCs

– Fusion of ISR and weapon system sensor data for BMDS 

implementation needs to be explicitly included in system level 

M&S

Findings

 Modeling & simulation evolution towards a systems level 

product suitable for assessing capabilities and making 

systems trade-offs for future spirals appears diffused, 

complicated, and slowed by the split between systems 

level and C2/BM/C M&S developments 

 BMDS M&S activities need to fully account for net-centric 

and GIG-BE evolution

– These activities need to provide for rapid, collaborative planning, 

command and control support for the engagement environments 

facing STRATCOM and the RCCs

– Fusion of ISR and weapon system sensor data for BMDS 

implementation needs to be explicitly included in system level 

M&S

The current division of M&S efforts between systems and 

C2/BM/C seems detrimental to developing an effective integrated, 

global BMDS (one that uses multi sensor discrimination). 

Communications initiatives (NCES and GIG-BE) have been largely 

ignored. Thus, data fusion and global interoperability are not 

modeled (or defined). 

BMDS “global system level” models must account for integrated 

elements and C2/BM/C representing a system-of-systems in order to 

capture how the system “works” and how it can be used against 

potential threat scenarios. 
 

ADDITIONAL OPPORTUNITIES 

FOR M&S CONTRIBUTION ________________________________________________________ 

70__________________________________________________________ DSB TASK FORCE ON

Findings (cont.)

 Configuration control will be critical in developing and 

using the models and simulations for future spirals

– Definition (or identification) of a standard architecture is required 

for each block so that the M&S software is compatible

– The “handover” (model update) for subsequent blocks must be 

managed through configuration control

 IV&V and accreditation is currently performed on certain 

(not all) BMDS M&S models internally

 Two relevant models are the DOE Weapons Lab “peer 

review” process and the NQA software development 

standard – the current MDA approach does not appear to 

meet this standard 

 Internal BMDS M&S IV&V and accreditation does not 

appear to have the organizational stature nor resources 

for the task(s) at hand

Findings (cont.)

 Configuration control will be critical in developing and 

using the models and simulations for future spirals

– Definition (or identification) of a standard architecture is required 

for each block so that the M&S software is compatible

– The “handover” (model update) for subsequent blocks must be 

managed through configuration control

 IV&V and accreditation is currently performed on certain 

(not all) BMDS M&S models internally

 Two relevant models are the DOE Weapons Lab “peer 

review” process and the NQA software development 

standard – the current MDA approach does not appear to 

meet this standard 

 Internal BMDS M&S IV&V and accreditation does not 

appear to have the organizational stature nor resources 

for the task(s) at hand

Modeling and simulation does not appear to have the “status” 

within MDA to achieve the needed capability. Budget, 

independence, and direct reporting responsibilities to MDA 

leadership are essential. 

MDA needs to determine a standard technical architecture for the 

interface of the M&S block components. This decision should be 

made within 6 months in order to give industry clear direction. The 

high level architecture (HLA) is currently designated by DoD, but 

recent commercial-of-the-shelf options such as the subscriber/ 

publish software should be considered as an alternative. A small 

team composed of people with relevant software experience from 

government, contractors, and FFRDC’s should be tasked to make 

recommendations to MDA. After making a recommendation, MDA 

should then require all contractors to adhere to the standard, starting 

with Block 2008 capabilities. 
 

___________________________________________________________________ CHAPTER 5 

MISSILE DEFENSE PHASE III ____________________________________________________ 71

Recommendations

 View systems level M&S to be a deliverable to USSTRATCOM

– On behalf of STRATCOM and the RCCs, STRATCOM should offer flag 

level assignment to (at) MDA responsible for “buying” (accepting) the 

systems level BMDS M&S capability for the user

 STRATCOM should use the combined expertise resident at the JNIC 

and STRATCOM to develop an integrated offense/defense M&S 

capability based upon the MDA deliverable

 MDA should assign a senior individual responsibility for BMDS M&S 

for each block 

– That individual must have the authority, budget, and staff to integrate 

systems and C2/BM/C M&S efforts into a credible “BMDS M&S 

deliverable” for each block to include responsibility for configuration 

control and utility of the product for the user

 MDA should assign a senior individual to lead IV&V and accreditation

– To report independently and directly to Director, MDA

– With the authority and budget to execute the desired independent

program

Recommendations

 View systems level M&S to be a deliverable to USSTRATCOM

– On behalf of STRATCOM and the RCCs, STRATCOM should offer flag 

level assignment to (at) MDA responsible for “buying” (accepting) the 

systems level BMDS M&S capability for the user

 STRATCOM should use the combined expertise resident at the JNIC 

and STRATCOM to develop an integrated offense/defense M&S 

capability based upon the MDA deliverable

 MDA should assign a senior individual responsibility for BMDS M&S 

for each block 

– That individual must have the authority, budget, and staff to integrate 

systems and C2/BM/C M&S efforts into a credible “BMDS M&S 

deliverable” for each block to include responsibility for configuration 

control and utility of the product for the user

 MDA should assign a senior individual to lead IV&V and accreditation

– To report independently and directly to Director, MDA

– With the authority and budget to execute the desired independent

program

Recommendations on this and the next two charts are self 

explanatory. 

Recommendations (cont.)

 Top level BMDS M&S tools should be used to establish 

tradeoffs for future architectures based upon well 

understood (common fidelity) systems and C2/BM/C 

representations

 BMDS M&S should account for the other operational 

responsibilities resident in the elements (e.g., air defense, 

cruise missile defense) so that the operational 

effectiveness of the system as a whole can be reliably 

assessed

 MDA should assess utilization of the NQA approach to 

force configuration control, regimen, and appropriate peer 

review of the systems level BMDS M&S capability

Recommendations (cont.)

 Top level BMDS M&S tools should be used to establish 

tradeoffs for future architectures based upon well 

understood (common fidelity) systems and C2/BM/C 

representations

 BMDS M&S should account for the other operational 

responsibilities resident in the elements (e.g., air defense, 

cruise missile defense) so that the operational 

effectiveness of the system as a whole can be reliably 

assessed

 MDA should assess utilization of the NQA approach to 

force configuration control, regimen, and appropriate peer 

review of the systems level BMDS M&S capability
 

ADDITIONAL OPPORTUNITIES 

FOR M&S CONTRIBUTION ________________________________________________________ 

72__________________________________________________________ DSB TASK FORCE ON

Recommendations (cont.)

 MDA should immediately assign a small team of 

government staff supported by contractors to consider 

various architecture standards (e.g., HLA) with the 

objective of recommending a BMDS M&S standard within 

six months. MDA should promulgate the standard and 

require all contractors to adhere, starting with Block 2008

 MDA should integrate the systems and C2/BM/C M&S 

approaches so that a useable M&S characterization of 

BMDS can be efficiently and effectively produced

Recommendations (cont.)

 MDA should immediately assign a small team of 

government staff supported by contractors to consider 

various architecture standards (e.g., HLA) with the 

objective of recommending a BMDS M&S standard within 

six months. MDA should promulgate the standard and 

require all contractors to adhere, starting with Block 2008

 MDA should integrate the systems and C2/BM/C M&S 

approaches so that a useable M&S characterization of 

BMDS can be efficiently and effectively produced
 

__________________________________________________________________ APPENDIX A 

MISSILE DEFENSE PHASE III 

MODELING AND SIMULATION ____________________________________________________ 73

APPENDIX A. TERMS OF REFERENCE
 

TERMS OF REFERENCE ___________________________________________________________ 

74__________________________________________________________ DSB TASK FORCE ON
THE UNDER SECRETARY OF DEFENSE

30 10 DEFENSE PENTAGON

WASHINGTON, DC 20301-3010

ACQUISITION,

TECHNOLOGY

ANO LOGISTICS 3 0 JAN 2003

MEMORANDUM FOR CHAIRMAN, DEFENSE SCIENCE BOARD

SUBJECT: Tenns ofReference--Defense Science Board Task Force on Ballistic

Missile Defense - Phase III (Modeling and Simulation)

In coordination with guidance from the Director, Missile Defense Agency, you are

requested to form a Defense Science Board (DSB) Task Force on Ballistic Missile

Defense -- Phase III (Modeling and Simulation).

Modeling and simulation (M&S) plays a key role in both developing the Ballistic

Missile Defense System and assessing its likely effectiveness. In this context, modeling

and simulation covers the gamut from detailed engineering simulation of components to

systems of systems simulations.

Specific purposes of this phase of the task force are to assess:

) the scope of the modeling and simulation effort

2) the appropriateness of the level of fidelity of classes of simulations

3) The impact of communications in the end-to-end models

4' the approaches to ensuring the validity of simulations for all uses, including

exercises and wargaming done for training and operations concept

development

5) additional opportunities for M&S contribution to BMDS development and

evaluation

The Task Force is asked to address these issues in a system of systems context with

particular emphasis on battle management systems, command and control systems, and

the global sensor system. The Task Force is also asked to provide advice on the state of

modeling and simulation for use in assessing overall performance of segments of the

BMDS, e.g., ground-based midcourse intercept system, space-based interceptor system.
This Task Force will be co-sponsored by me as the Under Secretary of Defense

(Acquisition, Technology, and Logistics) and the Director, Missile Defense Agency.

Dr. William R. Graham and General Larry D. Welch, USAF (Ret) will serve as Task

Force Co-Chairmen. Mr. Paul Hoffwill serve as the Executive Secretary; and LtCol

Roger W. BasI, USAF, will serve as the DSB Representative.

The Task Force will be operated in accordance with the provisions ofP .L. 92-463,

the "Federal Advisory Committee Act," and DoD Directive 5104.5, "DaD Federal

Advisory Committee Management Program." It is not anticipated that this Task Force

will need to go into any "particular matters" within the meaning of Section 208 of Title 18,

United States Code, nor will it cause any member to be placed in the position of acting as

procurement official.
 

__________________________________________________________________ APPENDIX B 

MISSILE DEFENSE PHASE III 

MODELING AND SIMULATION ____________________________________________________ 75

APPENDIX B. TASK FORCE MEMBERSHIP

Chairmen 

Dr. William Graham National Security Research 

Gen Larry Welch, USAF (Ret) Institute for Defense Analyses 

Members 

LtGen Bruce Brown, USAF (Ret) Institute for Defense Analyses 

Mr. Douglas Bryant Teledyne Brown Engineering 

Dr. Greg Canavan Los Alamos National Laboratory 

Dr. Sean Collins SPARTA, Inc. 

Dr. Julian Davidson Davidson Technologies, Inc. 

Mr. John Elwell The Charles Stark Draper Laboratories 

Dr. Delores Etter U.S. Naval Academy 

Dr. Eric Evans MIT Lincoln Laboratories 

Dr. John Foster Northrop Grumman Space Technology 

Dr. Jack Hammond Lockheed Martin 

RADM Wayne Meyer, USN (Ret) W.E. Meyer Corporation 

Dr. Robert Strickler Private Consultant 

Executive Secretary 

Mr. Paul Hoff Missile Defense Agency 

Defense Science Board Representative 

LtCol Roger Basl, USAF 
 

TASK FORCE MEMBERSHIP ________________________________________________________ 

DSB TASK FORCE ___________________________________________________________ 76

Government Advisors 

MAJ Tom Cook, USA Missile Defense Agency 

COL Kevin Greaney, USA Missile Defense Agency 

Mr. Keith Englander Missile Defense Agency 

Staff 

Mr. Frank Brown MDA SETA Support 

Ms. Barbara Bicksler Strategic Analysis 

Ms. Julie Evans Strategic Analysis 

Mr. Matthew Hawes MDA/DA (CSC) 
 

__________________________________________________________________ APPENDIX C 

MISSILE DEFENSE PHASE III 

MODELING AND SIMULATION ____________________________________________________ 77

APPENDIX C. GLOSSARY OF ACRONYMS AND 

ABBREVIATIONS

ABL Airborne Lasers 

AOR Area of Responsibility 

 

BMC Battle Management and Communications 

BMDS Ballistic Missile Defense System 

BOC Basis of Confidence 

BPI Boost Phase Intercept 

 

C2 Command and control 

C2/BM/C Command and Control/Battle Management/Communications 

CAPS Commanders Analysis Planning and Simulation 

CEC Cooperative Engagement Capability 

CMM Capability Maturity Model 

CONOPS Concepts of Operations 

 

DIS Distributed Interactive Simulation 

DoD Department of Defense 

DOE Department of Energy 

DSB Defense Science Board 

 

EADSIM Extended Air Defense Simulation 

EADTB Extended Air Defense Test Bed 

EDO Emergency Defensive Operations 

EKV Exoatmospheric Kill Vehicle 

EO Electro-Optic 

 

FFRDCs Federally Funded Research and Development Centers 

 

GBI Ground-Based Intercept 

GBR Ground Based Radar 

GMD Ground-Based Midcourse Defense 

 

HLA High Level Architecture 

HTK Hit-to-Kill 

HWIL Hardware-in-the-Loop 
 

GLOSSARY ___________________________________________________________________ 

DSB TASK FORCE ___________________________________________________________ 78

 

ICBM Intercontinental Ballistic Missile 

IDO Initial Defensive Operations 

IMD Integrated Missile Defense 

IMU Inertial Measurement Unit 

IRBM Intermediate Range Ballistic Missile 

IRT Independent Review Team 

ISR Intelligence, Surveillance, and Reconnaissance 

IV&V Independent Verification and Validation 

IVV&A Independent Verification, Validation, and Accreditation 

 

JNIC Joint National Integration Center 

JPL Jet Propulsion Laboratory 

JWARS Joint Warfighting System 

JTWARS Joint Theater Warfighting System 

 

KI Kinetic Interceptors 

KV Kill Vehicle 

 

M&S Modeling and Simulation 

MDA Missile Defense Agency 

MDAP Major Defense Acquisition Program 

MDSE Missile Defense System Exerciser 

MDWAR Missile Defense Warfare Assessment and Research Simulation 

MRBM Medium Range Ballistic Missile 

MSX Midcourse Space Experiment 

 

NQA Nuclear Quality Assurance 

 

PAC 2/3 PATRIOT Advanced Capability, level2/level 3 

PEELS Parametric Endo-Exo Lethality Simulation 

PES Probability of Engagement Success 

 

RCC Regional Combatant Commands 

RF Radio Frequency 

 

SBIRS Space Based Infrared System 

SM-3 Standard Missile-3 

SPO System Program Office 

SPY-1 AEGIS Radar 

SRBM Short Range Ballistic Missile 

SSGM Strategic Scene Generation Model 
 

__________________________________________________________________ APPENDIX C 

MISSILE DEFENSE PHASE III 

MODELING AND SIMULATION ____________________________________________________ 79

STSS Space Tracking and Surveillance System 

 

T&E Test & Evaluation 

THAAD Theater High Altitude Area Defense System 

 

UEWR Upgrade Early Warning Radar 

USD(AT&L) Under Secretary of Defense for Acquisition, Technology and 

Logistics 

USSTRATCOM United States Strategic Command 

 

VV&A Verification, Validation, and Accreditation 

 

XBR X-Band Radar