Distributed Interactive Simulation of Combat

September 1995

OTA-BP-ISS-151

GPO stock #052-003-01428-9
Recommended Citation: U.S. Congress, Office of Technology Assessment, Distributed

Interactive Simulation of Combat, OTA-BP-ISS-151 (Washington, DC: U.S. Government

Printing Office, September 1995).
iii

oreword

istributed interactive simulation (DIS) is the linking of aircraft, tank,

and other military simulators in diverse locations so that the crew of

one simulator can “see,” operate with, “shoot” at, or be “destroyed” by

the other vehicles being simulated. Command structures can also be

simulated. This allows forces to practice and train in situations too costly or

risky to practice with real weapons.

The simulators are technological descendants of the Link Trainer, the famous flight simulator introduced in 1929. Modern vehicle simulators use electronic digital computers to calculate how instrument indications, visual displays, and sounds should change in response to a user’s handling of controls. A

prominent trend, over the last decade, has been the increase in the detail and

apparent realism with which increasingly affordable computing power can

generate and display a scene that a helicopter pilot might see through his

canopy or a tank crewman might see through his periscope.

The linking of the simulators’ computers into a network, using the technologies and standard communications procedures used in the Internet, as well as

others, allows each simulator crew to practice teamwork with other crews

This background paper is the third publication of the Office of Technology

Assessment’s (OTA’s) assessment of combat modeling and simulation

technologies. It complements OTA’s background paper, Virtual Reality and

Technologies of Combat Simulation, which focuses on the human-computer

interface technologies used in simulations. The assessment was requested by

the House Committee on Armed Services (now the House Committee on National Security); the Senate Committee on Armed Services; and its Subcommittee on Defense Technology, Acquisition, and Industrial Base (now the Subcommittee on Acquisition and Technology).

In undertaking this assessment, OTA sought the contributions of a wide

spectrum of knowledgeable individuals and organizations. OTA gratefully acknowledges their contributions of time and intellectual effort. OTA also appreciates the help and cooperation of officials of the Department of Defense

and the Department of Energy. As with all OTA publications, the content of

this background paper is the sole responsibility of the Office of Technology Assessment and does not necessarily represent the views of our advisors or reviewers.

ROGER C. HERDMAN

Director
iv

dvisory Panel

George Rathjens, Chair

Massachusetts Institute of

Technology

Donald Blumenthal

Lawrence Livermore National

Laboratory

Jerome Bracken

Yale University

Edward C. Brady

Strategic Perspectives, Inc.

David R. Cheriton

Stanford University

Paul K. Davis

The RAND Corp.

Col. Trevor N. Dupuy1

The Dupuy Institute

John Englund

Analytic Services, Inc.

Joseph P. Fearey

Jet Propulsion Laboratory

Amoretta M. Hoeber

AMH Consulting

John D. Kettelle

Frank Lanza

Loral Corp.

Creve Maples

Sandia National Laboratory

Jed Marti

Sarcos Research, Inc.

Duncan Miller

MIT Lincoln Laboratory

Stuart H. Starr

The MITRE Corp.

Larry Stone

Metron, Inc.

Jack Thorpe

Science Applications International

Corp.

Verena S. Vomastic

Electrospace Systems, Inc.

Jordan Weisman

Virtual World Entertainment

________________

1 Deceased June 5, 1995

Note: OTA appreciates and is grateful for the valuable assistance and thoughtful critiques provided by the advisory panel members.

The panel does not, however, necessarily approve, disapprove, or endorse this report. OTA assumes full responsibility for the report

and the accuracy of its contents.
v

roject Staff

Peter Blair

Assistant Director

OTA Industry, Commerce, and

International Security Division

Alan Shaw

Program Director

International Security and

Space Program

PRINCIPAL STAFF

Brian McCue

Project Director

Michael Callaham

Senior Analyst

Darian Unger

Intern (summer 1994)

ADMINISTRATIVE STAFF

Jacqueline Robinson-Boykin

Office Administrator

Ellis Lewis

Administrative Secretary

Don Gallagher

Secretary

CONTRACTORS

Christopher Lawrence

Elizabeth Sheley

Linda Voss
vi

Introduction: Preparing For the Next War 1

Recent Developments 5

Simulators 8

Types of Simulations 9

The Evolution of Distributed Interactive Simulation 15

SIMNET 15

Fidelity, Fog, and Friction 32

Verification, Validation, and Accreditation 35

Standards for DIS 48

Scalability and Scaling 52

Funding for DIS Systems 53

ontents
istributed

Interactive

Simulation of

Combat

INTRODUCTION: PREPARING FOR THE NEXT WAR

[The Romans] do not begin to use their weapons first in time

of war, nor do they then put their hands first into motion, while

they avoided so to do in times of peace; but as if their weapons did

always cling to them, they have never any truce from warlike exercises; . . . nor would he be mistaken that should call those their

exercises unbloody battles, and their battles bloody exercises.—

Flavius Josephus, De Bello Judaico [The Jewish War] (79)

The military is proverbially accused of always training for the

last war. To avoid this predicament, the U.S. military makes extensive use of simulators, simulations, and exercises, designed to

emulate present or projected conditions. Models and simulations

are used for several important purposes: training (to maintain

readiness), analysis (of the effects of proposed tactics or system

acquisitions), planning and rehearsal of operations, and demonstration of new technologies. Simulators, such as the Link Trainer, have been used primarily for training and mission rehearsal.

More abstract simulations and models have been used for analysis and operations planning.

The simulation entry on the Critical Technologies List issued

by the Director of Defense Research and Engineering (DDR&E)

always stood as an area of undisputed superiority over the former

Soviet Union. In today’s changed world, modeling and simulation technology is one of 20 ‘‘technology areas’’ selected by the

Department of Defense (DoD) for research and development

funding emphasis. DoD’s Defense Modeling and Simulation Office (DMSO), founded in 1991, has the lead in structuring DoD’s

approach to modeling and simulation, especially the high-tech

forms.

| 1
2 Distributed lnteractive Simulation of Combat

SOURCE: Perceptronics, Inc., 1995.

Advances in the technologies of microelectronics, computer networking, and computer graphics

have led to supercomputers, the Internet, and synthetic environments for virtual reality (VR). They

have also made possible a new kind of military

training, pioneered by the Defense Advanced Research Projects Agency (DARPA—now ARPA1

)

in its Simulation Network (SIMNET) program,

which began in 1983.

SIMNET began as a Tank Team Gunnery

Trainer to replace existing tank simulators.

Through adroit use of increasingly capable and

economical computer equipment, SIMNET’s developers expanded their system from a tank simulator to a tank battle simulator for company-sized

units. Multiple interconnected tank trainers maneuvered on the same imaginary battlefield and

cooperated to engage a common enemy. Their

crews sat inside tanklike boxes (see figure 1) and

viewed the imaginary battlefield through television screens mounted where the vision ports

would normally be.

DARPA’s SIMNET program ended in 1989,

but the simulators, communications protocols,

and network developed by the program are still

being used and upgraded by ARPA’s Advanced

Distributed Simulation (ADS) program, the

Army’s Battlefield Distributed Simulation-Developmental (BDS-D) program, and other Department of Defense (DoD) programs (see box 1).

The name SIMNET still occurs in the “SIMNET/

Close Combat Tactical Trainer” line item in the

DoD Comptroller’s 1995 Procurement Programs

(P-1) report to Congress (189). The term is still

applied to simulators, networks, and simulations

that still use the SIMNET communications protocols, which have been superseded by the Distributed Interactive Simulation (DIS) protocols.

A distributed interactive simulation is “distributed” in the sense that simulator computers at

1 The Advanced Research Projects Agency was established in 1958, renamed the Defense Advanced Research Projects Agency in 1972,

and renamed the Advanced Research Projects Agency in 1993.
Driver's compartment of a SIMNET M1 tank simulator. The

driver stem the tank with handlebars and throttles it by

rotating one of the handgrips, much like driving a

motorcycle, while viewing the virtual world outside through

three vision blocks-displays of computer-generated

imagery where periscopes would be on a real Ml. When he

drives overstimulated rough terrain, electromechanical

actuators move his seat so that he feels bumps. The driver

communicates with other crew members over an intercom

system and hears simulated engine noise, firing of the tank’s

gun, and other sounds of combat.

multiple training sites nationwide are connected

by means of a local-area network (LAN; see figure 2), which in turn maybe connected to a widearea network (WAN) such as the the Defense

Simulation Internet (DSI) (see figure 3). Trainees

can enter disparate tank, helicopter, or other simulators and all participate in the same simulated

combat. The simulation is “interactive” in the

sense that humans interact with the computers to

influence the simulation. The battle does not follow a set script, and trainees win or lose on the basis of their performance. The trainees fight a

similarly networked opposing force, which may

be a combination of virtual forces-real people

operating simulators-and semiautomated forces

(SAF)--vehicles and weapons simulated by computers with humans providing operational superDistributed Interactive Simulation of Combat 3

Crew compartment of a SIMNET M1 tank simulator. The

loader’s position is closest to the camera, the gunner's

position beyond and to the Ieft, the commander stool

beyond and to the right. The Ioader has one vision block;

the gunner has one that shows finer detail, and the

commander has several vision blocks, but cannot open the

hatch for an outside view.

vision. In both the virtual forces and the

semiautomated forces, human behavior is simulated by humans, albeit humans not affectedly the

many stresses of real combat.

These simulations hold promise beyond their

capability to train equipment operators to work as

a team, Ever larger numbers of networked tanks,

airplanes, and other platforms allow higher echelon commanders to plan operations and conduct

them in simulation before conducting them in

combat. Proposed weapon systems can also be

simulated in order to evaluate and, if necessary,
4 Distributed Interactive Simulation of Combat

I

1979

1980

1981

1982

1983

1984

1985

1986

1987

1988

1989

1990

1991

1992

1993

1994

KEY:

I I

I TGT/TTGT

DARPA funding

I

SIMNET-T SIMNET-D ADS

I

DARPA & Army funding I

I

I

I

I

Army funding

I

I ENCATT 

A2ATD: Anti-Armor Advanced Technology Demonstration DARPA: Defense Advanced Research Projects Agency

ADCATT: Air Defense Combined Arms Tactical Trainer DIS: Distributed Interactive Simulation [Project]

ADS: Advanced Distributed Simulation ENCATT: Engineering Combined Arms Tactical Trainer

AVCATT: Aviation Combined Arms Tactical Trainer FSCATT: Combined Arms Tactical Trainer

ARPA: Advanced Research Projects Agency STOW-E: Synthetic Theater of War—Europe

BDS-D: Battlefield Distributed Simulation-Developmental TGT: Tank Gunnery Trainer

CATT: Combined Arms Tactical Trainer TTGT: Tank Team Gunnery Trainer

CCTT: Close Combat Tactical Trainer

redesign them—before the first unit is built. This platform navigation and weapon guidance, opentechnique is known as virtual prototyping.2,3 ing up many possibilities to improve all applicaCreating a shared virtual battlefield requires tions of modeling and simulation and to tie them

more than computers and display equipment. It re- more directly to actual combat.

quires collection of detailed weapon, terrain, and High-tech simulation has critics as well as

climatic data. Some simulators can use data col- boosters. Some critics have seized on the possibillected and archived for other purposes, including ity of “negative learning”: trainees who have re2ARPA’S Simulation-Based Design (SBD) ( 157,158,159, 160) and TransTech ( 161 ) programs and the Army’s current Louisiana Maneuvers

(LAM) program (138, 199) are using virtual prototyping of proposed military equipment to reduce acquisition cost and time and to increase

operability and supportability.

3 There are many nondefense applications of virtual prototyping. Well-known examples include Boeing Aircraft Company’s virtual prototyping of its new 777 commercial transport aircraft, and NASA’s use of Lockheed’s Preview (for “Prevent Real Errors by Virtual Investigation

of Engineering Worlds” (5)) software to visualize the first Hubble Space Telescope servicing mission, which identified a need to redesign the

Corrective Optics Space Telescope Axial Replacement (Costar) (35,5 1,86, 137).
Distributed Interactive Simulation of Combat 15

DARPA’s SIMNET project began in 1983 to exploit technologies and concepts developed in DARPA’s Tank Gunnery Trainer project.1 The Army became a cosponsor in 1985 and began using the

technology for training (SIMNET-T) and research and development (SIMNET-D). Meanwhile, DARPA

founded the Advanced Distributed Simulation program to continue technology development on its own.

In 1989 DARPA stopped funding the SIMNET-T and SIMNET-D projects, which two years later became

the Army’s Combined Arms Tactical Trainer (CATT) project and Battlefield Distributed Simulation-Developmental project, respectively. The CATT project is a collection of several projects to develop interoperable distributed simulation equipment for several combat arms. Its Close Combat Tactical Trainer project, for training armored units, most closely resembles the early SIMNET project; other CATT projects

are for training fire support, aviation, air defense, and engineering units. DARPA—now called ARPA—

jointly fund the Anti-Armor Advanced Technology Demonstration program. ARPA’s Advanced Distributed Simulation program manages and partially funds the Synthetic Theater of War series of technology

demonstrations. The Army’s Distributed Interactive Simulation project coordinates the development distributed interactive simulation protocols, standards, and technologies for DoD-wide use.

A 1992 study by the Army Science Board expressed concern that the Army’s follow-on programs

to SIMNET were fragmented—in particular, that “CATT and BDS-D are being pursued as separate, independent efforts rather than as a single, integrated program. ”2

1 E.A. AIIuisi, “The Development of Technology for Collective Training: SIMNET, a Case History” Human Factors,

33(3)343-362, 1991

2 U.S. Army, Army Science Board, 1991 Summer Study Final Report, Army Simulation Strategy (Washington, DC, December

1991), p. 2.

SOURCE: Office of Technology Assessment, 1995.

relearned to rely on artifacts of the simulations may

err in real combat. Other critics have questioned

the process by which the high-tech simulations

have been deemed valid models of combat, saying

that the formalized verification, validation, and

accreditation (VV&A) process needed to certify

mathematical models of combat has been skipped

in the case of the high-tech simulations. DoD is

just beginning to use DIS for cost and operational

effectiveness analyses (COEAs) to justify budget

requests to Congress (and within DoD) for system

acquisition (156,138,199). When DIS is used for

such predictive uses, validation should describe

and quantify the uncertainties in the predictions.4

RECENT DEVELOPMENTS

The following DIS-related developments have occurred since the Senate Committee on Armed Services held hearings on simulation in 1992 (145).

They are discussed further in either this background paper or a previous one (147). For brevity,

this list focuses on, but is not limited to, official

activities; there have also been related developments in industry and academia.

1992 Live exercises were conducted at the Wurtsmith Air Force Base Weapon Storage Area

to validate the Air Force/Defense Nuclear

4

For lists of questions or factors to consider in assessing validity, see reference number, p. 281, table I; reference number(135), p. 129;

and reference number (197), p. 19, table 2.1.
6 Distributed lnteractive Simulation of Combat

Vehicle Simulator

,

II—— / Computer Operator Controls

Integrated Unit Soldier System

Small Unit (Company, Platoon) Commander

n

v Local Area Network v Local-Mea Network (LAN) )

I Observer/Analyst

‘Flying Carpet Simulator” or "Stealth Node”

Gateway

v

Gateway

(Computer) ( )

Wide-Area Network (WAN)

KEY: CIG = Computer Image generator; LAN = Local-area network; WAN = wide-ares network.

SOURCE: Office of Technology Assessment, 1995.

1993

1994

Agency Security Exercise Evaluation System (SEES) (41,141);5

Army Pamphlet 5-11, Verification, Validation, and Accreditation of Army Models and

Simulations, was published;

DoD Directive 5000.59, “DoD Modeling

Defense Simulation Internet experimental integration area at

the Network Engineering and Assessment Facility Laboratory

of the Defense information Systems Agency's Center for

Systems Engineering in Reston, Virginia.

and Simulation (M&S) Management,” was

issued;

The first Modeling and Simulation Technology Area Plan (TAP) was prepared, the first

Technology Area Review (TAR) was conducted, and the first Technology Area Assessment (TAA) was prepared;

5

SEES version 2.0 is a multi-user, interactive,entity-level constructive simulation, not a distributed interactive virtual simulation. However,

the method used to validate SEES version 2.0 could be used with distributed interactive virtual simulations.
Distributed Interactive Simulation of Combat 7
8 Distributed interactive Simulation of Combat

1995

The first of six planned Anti-Armor

Advanced Technology Demonstration

(A2ATD) experiments was conducted;

The SIMVAL’94 Simulation Validation

symposium was conducted by the Military

Operations Research Society (MORS) to address the question, “How Much Verification

and Validation Are Enough?”

The Synthetic Theater of War-Europe

(STOW-E) exercise was conducted;

The final draft of the DoD Modeling and

Simulation Master Plan (MSMP) was-completed (195);

A handbook, aimed at DIS exercise managers, on methodology for VV&A of DIS, is

drafted (175);

Proposed revisions of the DIS protocol are

discussed at the 12th DIS Standards Workshop; and

An Architecture Management Group

(AMG) is formed to steer development of

the High-Level Architecture (HLA) called

for by the draft MSMP.

SIMULATORS

A simulator is an instrumented mock-up, such as

the famous Link Trainer for pilots. The trainee operates the simulator in much the same way that he

or she would operate the vehicle (or other equipment) that it simulates. An instructor may program the simulator to present the trainee with

particular tasks or problems as the session is underway. Simulators are useful because they are

The first Link Trainer, built in 1929, used compressed air to

make the cockpit move in response to manipulation of the

controls.

cheaper and safer to operate than the equipment

that they simulate.

The variety and sophistication of simulators

have increased since the Link Trainer was

introduced in 1929, as have the realism and costs

of top-of-the-line simulators. The best modem

simulators use immense computing capacity and

exotic display equipment to synthesize or reconstruct digitally stored scenes, sounds, and vibrations, giving trainees an experience so vivid and

realistic that some call it “virtual reality”

(182,147). Still, simulators cost less than would

the use of real platforms and weapons for training.

They are also safer and provide more frequent and

varied training opportunities than actual combat

does.

Less expensive models lack a motion base (a

cab or cockpit platform moved by pistons or electromechanical actuators to provide a kinesthetic

sense of being there), but are nevertheless ade-
Distributed Interactive Simulation of Combat | 9

quate for most training goals.6,7 For example, the

Air Force, following the lead of the Air National

Guard, is buying current block F-16C simulators

for $650,000 each for use as unit (or unit-level)

training tevices (UTDs).8 They have only a 60-degree vertical by 78-degree horizontal field of view

and no motion base, but they cost only $10,000

each per year to maintain—a thirtieth the cost of

the most recent simulator—and the Air Force has

concluded that they can meet all critical training

tasks except 1) post-merge ‘‘fur balls’’9 in air-toair engagements, 2) multiship tasks, and 3) maneuvers such as ‘‘the pop’’10 in air-to-ground

missions (168).

 It appears that DoD purchases a very small

fraction of the output of the domestic simulator industry.11 It is likely that most of DoD’s purchases

for combat modeling and simulation are not

counted in simulator sales.

TYPES OF SIMULATIONS

A simulation is a dynamic information construct

reflecting some larger reality or possible reality.

Its parts and processes correspond to parts and

processes of the larger reality, albeit with less than

perfect fidelity. In some cases the correspondence

involves considerable abstraction. DoD defines a

model as ‘‘a physical, mathematical, or otherwise

logical representation of a system, entity, phenomenon, or process,’’ and a simulation as ‘‘a

method for implementing a model over time’’

(153).12 DoD also distinguishes three major cate6 A meta-analysis of 26 experiments on flight simulator training effectiveness concluded in 1992 that ‘‘for jet training, motion cuing was

found to add nothing to the simulator training effectiveness,’’ but cautioned that ‘‘the positive effects of motion for any one task may have been

masked by the negative effects of motion for another task’’ (60). Similarly, a 1993 study by the U.S. Army Research Institute concluded that no

data on training transfer (i.e., effectiveness) supported the use of motion-based simulators, possibly because motion may not help in some cases,

or possibly because of insufficient experimentation or shortcomings in experimental technique (8).

7 The Federal Aviation Administration (FAA) requires a flight simulator to have a motion base capable of at least three degrees of freedom

(e.g., pitch, roll, and yaw). The FAA classifies an otherwise similar system lacking a motion base and/or a visual system (e.g., a computer image

generator (CIG)) as a flight training device. An FAA Level 7 flight training device, which must faithfully represent the instruments, controls, and

responses of a particular aircraft, can be used for all training for which an FAA Level C flight simulator can be used, less visual and motion

aspects. An FAA Level C flight simulator must have a motion base capable of six degrees of freedom (pitch, roll, yaw, heave, surge, and sway). A

Level D flight simulator must have, in addition, sound simulation, daylight scene generation with at least 400 edges or 1,000 polygons, and

nighttime scene generation with at least 4,000 points of light. Experienced crews can receive training credits in an FAA Level D flight simulator

to become type-rated (i.e. qualified) to fly a new but similar type of aircraft with zero flight time. The U.K. Civil Aviation Authority has similar

regulations (136).

8 F-15C UTDs cost $750,000 each.

9 ‘‘Fur ball’’ is fighter-pilot jargon for a phase of an engagement of several friendly aircraft with several opposing aircraft in which the opposing groups of aircraft are merged (mixed together) and sorting (the assignment of targets) is especially difficult.

10 The pop, or pop-up is a maneuver in which a strike aircraft that has been approaching its target at low altitude in order to avoid detection

suddenly climbs in order to acquire (see and identify) and engage the target.

11 The U.S. Bureau of the Census reported that 1993 sales of electronic teaching machines, teaching aids, and trainers and simulators (product code 36991) totaled $1.1 billion (149). Most of these sales—about $1 billion—were sales of electronic trainers and simulators (product code

36991 81). Electronic teaching machines, teaching aids, trainers and simulators are one of four products produced by the ‘‘electrical equipment

and supplies, not elsewhere classified’’ industry, to which the Office of Management and Budget has assigned Standard Industrial Classification

code 3699, which corresponds to the 1987 Input-Output Account 580500 defined by the U.S. Bureau of Economic Analysis (BEA). OTA’s

round-by-round analysis (148) of the BEA’s unpublished six-digit annual Input-Output tables for 1987 (the last year for which detailed BEA I-O

tables were available to OTA) indicated that the federal government puchased, for national defense, less than 1 percent of the industry’s output

(about $3.5 billion; cf. (45)) directly and less than 2 percent indirectly (double-counting value added in different tiers).

12 Paul Davis, Corporate Research Manager at the RAND Corporation, considers the terms model and simulation to be ‘‘essentially synonymous,’’ both meaning ‘‘almost any kind of representation of reality.’’ He notes that ‘‘few terms are used consistently across communities or

even from day to day within a given community’’ (24).
10 Distributed interactive Simulation of Combat

A modem Link trainer: the Apache (AI--64) Combat Mission

Simulator.

gories of models and simulations (M&S)---live,

virtual, and constructive (195):

Live simulation [is] simulation involving

real people operating real systems. . . .

Virtual simulation [is] simulation involving

real people operating simulated systems. Vial

simulations inject human-in-the-loop (HITL) in

a central role by exercising motor control skills

(e.g., flying an airplane), decision skills (e.g.,

committing fire control resources to action), or

communication skills (e.g., as members of a C4I

team) .6’

Constructive . . . models and simulations . . .

involve simulated people operating simulated

systems.

DoD acknowledged that there is no clear division between these categories (195):

The categorization of simulation into live,

virtual, and constructive is problematic, because

there is no clear division between these categories. The degree of human participation in the

simulation is infinitely variable, as is the degree

of equipment realism. This categorization also

suffers by excluding a category for simulated

people working real equipment (e.g., smart vehicles).

Live simulation-humans using real equipment---is the oldest

form of combat simulation. Using a rifle to fire live ammunition

in a field training exercise would be a pure example--and

very dangerous. This photo shows a soldier firing simulated

bullets-coded invisible infrared pulses emitted by the

eye-safe Iaser mounted on his rifle, which also fires blank

ammunition to simulate the sound of firing. This is a mixture of

live simulation and virtual simulation--humans using

simulated equipment. The soldier's helmet and suspenders

are equipped with sensors to detect coded pulses from lasers

on other soldiers’ rifles and on other types of weapons. If the

sensors detect a simulated bullet, audible and visual signals

indicate to the soldier, and to referrees, that the soldier has

been hit. The system of lasers and detectors is called the

Multiple Integrated Laser Engagement System (MILES).

❚ Live Simulation

Simulations have been used for combat training

and rehearsal since time immemorial and for operations research and planning for at least a century. The most ancient and familiar type of combat

simulation involves real soldiers, sailors, or, in

this century, aircrews operating real equipment—

but not in combat. DoD now calls such exercises

live simulations.

13 C4I (or C4

I) stands for Command, Control, Communications, Computers, and Intelligence.
Distributed Interactive Simulation of Combat | 11

The Army’s Louisiana Maneuvers of the early

1940s (44), the U.S. Navy’s mock attack on Pearl

Harbor just before the real one (100), and portions

of recent REFORGER,14 Team Spirit, and Atlantic Resolve exercises are all examples of live simulation. Most field exercises are scripted in

advance.

Though usually a means of training, exercises

can be used for research. For example, the Louisiana Maneuvers that preceeded U.S. involvement

in World War II taught the U.S. Army important

lessons about how to handle large armored formations.15

The Army’s National Training Center (NTC)

offers the most realistic land-warfare exercise environment available today. Instrumented American fighting men and machines, including some

playing the role of opposing forces (OPFOR), do

battle on a desert battlefield the size of Rhode Island using weapons that fire coded laser pulses

instead of bullets. The pulses specify the type of

shot simulated, so that a tank receiving a rifle shot

knows that it is unhurt, while a tank hit with a

TOW16 antitank guided missile knows that the

time has come to simulate its own demise by shutting off its systems and emitting smoke. Training

at NTC was given major credit for the success of

U.S. forces in the Persian Gulf War.

Live simulation is used by all the services, as

well as by the U.S. Unified Combatant Commands and other organizations. Air warfare, naval

warfare, amphibious warfare, and joint (i.e., multiservice) and theater warfare are simulated. The

U.S. Navy’s air combat simulation conducted at

Naval Air Station Monterrey, California, was publicized by the motion picture Top Gun. Less well

known, perhaps, are the Red Flag and Green Flag

exercises conducted at the Air Force Fighter

Weapons School at Nellis Air Force Base, Nevada, which simulate air-to-air, strike, and surfaceto-air combat, as well as electronic warfare. There

are many other periodic and ad hoc live simulations.

Military exercises, especially those conducted

abroad or on the high seas, serve diplomatic and

other purposes beyond their utility to the military.

REFORGER and Team Spirit are prime examples

in this regard: these periodic exercises help maintain close U.S. ties with the Federal Republic of

Germany and the Republic of Korea, respectively.

❚ Constructive Simulation

Prominent historical examples of constructive

simulations include exercises on maps and sand

tables,17,18 and games with model ships played at

the U.S. Naval War College19. Computer-assisted

exercises (CAXs) have largely superseded sandtable and map exercises. In some cases they pit a

commander (often with staff and subordinate

headquarters) against a similarly-equipped human playing the role of an opposing commander.

In other cases, there are no humans playing military roles. Instead, an analyst enters data about a

14 REFORGER stands for Return of Forces to Germany. REFORGER exercises have been superseded, in the post-Cold War era, by Atlantic

Resolve exercises.

15 Some years earlier German forces conducted major exercises in the course of developing the Blitzkrieg approach to warfare. Interestingly, these exercises began before tanks were available: crews operated cars fitted with tanklike bodies (17).

16 A tube-launched, optically tracked, wire-command link guided missile.

17 These older methods are still used. For example, to visualize and rehearse their part in Operation Desert Storm, one U.S. Army tank battalion commander and his staff maneuvered military models, which they had purchased in garrison in Germany, on Saudi Arabian sand—sans

table (206). The Army history of the Persian Gulf War shows another example (123).

18 The Army may use ‘‘virtual sandtables’’ in the future. The Army’s Communications and Electronics Command (CECOM) is funding the

development of an architecture for a ‘‘VR-based dispersed command post. . . , where each commander’s virtual environment integrates a virtual

conference room containing virtual video walls, battlemap projections, interactive 3D sandtable models, and visual representations of remote

conference participants’’ (47).

19 See, for example, (105), especially pp. 73ff; (100); and (110) .
12 Distributed Interactive Simulation of Combat

File Map Scale Map Features Show As Local Force Special Privilege —

1:63,712 

Zoom: click middle to zoom in around point; click right to zoom out around point; click and drag middle to set screen area

Select an item to edit

This map display shows part of a tank battle being simulated by an early version of the Modular Semi—Automated Forces (ModSAF) software A

constructive simulation, it simulates movement and combat of individual entities (vehicles and soldiers) and displays the Iocation and status of

individual vehicles, as shown here, or small units, such as tank platoons and companies, An operator may be privileged to give orders (e g , move to

contact, or move and wait) to various units. Use of ModSAF IS a computer-assisted exercise and maybe a command post exercise (CPX) for the

benefit of unit commanders and their staffs. A computer running ModSAF can participate in a distributed interactive simulation by sending specially

formatted messages reporting the Iocation and status of each entity that it simulates, and by processing similar messages from other computers (or

manned simulators or Instrumented vehicles) to determine whether the entities or events that they simulate should be displayed or should Influence

the entities simulated locally

SOURCE: The RAND Corp., 1995,

real or hypothetical situation into a computer and ercise only commanders, their staffs, and the

then instructs the computer to predict, step by communications system. The forces that they prestep, what might happen, in some cases with reso- tend to command appear only as marks on a map

lution (details) of individual vehicles. This, too, is or chart display. Procedural CPXs are often heavia constructive simulation (see figure 4). ly scripted to insure that certain procedures are reCommand post exercises (CPXs) and their na- hearsed or tested; scripts may branch at certain

val equivalents, Fleet Exercises (FLEETEXs) ex- points according to command decisions made by
Distributed Interactive Simulation of Combat 13

Echelon Notation Commander’s ranka

Army Group

Army

Corps

Division

Brigade

Group/Regiment b

Battalionc

Company d

Platoon

Squad

Section

Individual

xxx

xx

x

●●●

●●

●

General

General

Lieutenant General

Major General

Brigadier General

Colonel

Lieutenant Colonel

Captain

2d or 1st Lieutenant

Sergeant

Corporal

Private or Private First Class

a Typical.

b Regiment, if a cavalry unit.

c Squadron, if a cavalry unit.

d Troop, if a cavalry unit; battery, if an artillery unit.

SOURCES: J.F. Dunnigan, How to Make War--A Comprehensive Guide to Modem Warfare, completely revised edition (New York, NY: William

Morrow, 1988) ; U.S. Army, Operations, FM 100-5, 1986; U.S. Department of Defense, Common Warfighting Symbology Version 1, MILSTD-2525, Sept. 30, 1994

the participants. CPXs are both live and constructive simulations. The participants in one command post (CP) use some of the command and

control equipment that they would use in war:

field telephones, computer terminals, maps or

map displays, and sometimes radios. In this respect a CPX is a live simulation. The equipment

connects the CP not only to others in the CPX but

also to computers (if a CAX) or referrees who perform constructive simulation of the forces that

those in the CP command. In a different interpretation, the command and control system is considered to be a simulator, not just a means of

communicating with one. In this sense a CPX is

a virtual simulation. Because it is problematic, if

not impossible, to classify a CPX (and other simulations) as live, constructive, or virtual, Paul Davis has proposed a generalized classification

scheme, which includes the hybrid classes

constructive/virtual (CV), virtual/live (VL),

constructive/live (CL), and constructive/virtual/

live (CVL) (24).

Simulations of all these types differ from one

another in scope, focus, abstraction, and aggregation. One training task will require a particular activity (its focus) to be simulated in detail but can

tolerate greater abstraction in the simulation of a

related activity, which might be the focus of a different simulation. Concreteness and detail are

necessary to make a tank simulator seem realistic

to the crew using it, but the simulator need not

show forces more than 20 km away or correctly

predict which side will win if the only objective of

the simulation is to improve the combat skills of

participating tank crews. In contrast, both in exercises and war, commanders and their staffs (see

table 1) deal, with aggregated forces. They do so

through abstraction: an armored battalion, consisting of about a thousand armored vehicles and

about 16,000 soldiers at full strength (see table 2),

is often represented on paper maps and computer

displays at higher echelon headquarters as a rectangle (any unit) enclosing an oval (representing

tank tracks, hence armor) with two “x”s on top
14 Distributed Interactive Simulation of Combat

Unit Tanks Soldiers Subordinate Units

Transport

Soldiers

Tanks

Armored personnel

carriers

Artillery

Anti-tank guided

munitions

Machine guns

Heavy mortars

Anti-aircraft guns

Self-propelled SAMs

Light SAMs

Anti-tank rocket

launchers

Armored

Division

3,500

16,300

348

652

143

523

1,360

66

24

24

60

900

Mechanized

Infantry

Division

3,500

16,600

290

727

143

660

1,616

66

24

24

60

1,000

KEY: SAM = surface-to-air missile.

a This is a simplified table. Terms for vehicles and weapons are not

necessarily official U.S. Army terms. Numbers of personnel, vehicles, and weapons approximate those of units at full strength.

SOURCE: J.F. Dunnigan, How to Make War--A Comprehensive

Guide to Modern Warfare, completely revised edition (New York, NY:

William Morrow, 1988)

(representing the division level of aggregation).

An armored battalion, consisting of tens of vehicles and hundreds of soldiers (see table 3),

would be represented by the same symbol, except

with two vertical bars (instead of two “x”s) on top

of the rectangle. A mechanized infantry battalion

with about 700 soldiers (see table 4) would be represented by the same symbol, with an “x” (representing crossed rifles) criss-crossing the rectangle

enclosing the oval. Messages are equally abstract:

“US 1AB” would represent the U.S. Army’s First

Armored Battalion.

Whatever else they do, combat models model

aspects of individual and collective human behavior, such as operational decision making. This

modeling has always been controversial, because

human behavior cannot be reduced to a set of principles as simple and reproducibly verifiable as

those of, say, aerodynamics or solid mechanics,

Battalion 33-60 182-540 3 tank companies,

1 headquarters,

other units.

Company 10-22 40-88 3-4 tank platoons,

1 headquarters

Platoon 3-5 12-20 3-5 tanks

SOURCES: J.F. Dunnigan, How to Make War—A Comprehensive

Guide to Modern Warfare, completely revised edition (New York, NY:

William Morrow, 1988); U.S. Army, Operations, FM 100-5, 1986

Unit Soldiers Subordinate Units

Battalion 696 6 companies: 4 infantry, 1

antitank, 1 headquarters

Company 111 3 infantry platoons, 1

headquarters section

Platoon 33 3 infantry squads, 1

headquarters

Squad 9 9 individuals

SOURCES: J.F. Dunnigan, How to Make War—A Comprehensive

Guide to Modern Warfare, completely revised edition (New York, NY:

William Morrow, 1988); U.S. Army, Operations, FM 100-5, 1986.

which are used to model and simulate the response

of an airplane to a pilot’s control actions. Combat

is a much more complex and less predictable process, the result of actions by many humans. Of

necessity, modeling them requires many simplifying assumptions, often without any scientific justification.

The uncertainty and unpredictability in combat

is further complicated today with the shift from a

defined superpower confrontation to more complicated and ambiguous tasks for the military.

Stuart Starr, Director of Plans at the MITRE Corporation, has observed that because of geopolitical changes in the last 5 years, “models of

strategic and high intensity conventional conflicts” are of less interest. In their place, “the

M&S community is being confronted with the

need to develop credible models of potential

crises, humanitarian assistance, and conflicts with
I

I

Distributed Interactive Simulation of Combat 15

This computer-generatad scene is analogous to scenes presented to pilots in flight simulatore. The scene actually illustrates(although not precisely)

an episode of Iive simulation in the November 1994 Synthetic Theater of War Europe exercise, in which U.S. Air Force F-15C Eagles and US. Navy

F/A-18 Hornets practiced joining procedures. Their positions were trackad by the Tactical Aircrew Combat Training System at the Marine Corps Air

Station at Cherry Point, North Carolina, end relayed to participating vehicle and radar simulators and constructive simulations.

SOURCE: Office of Technology Assessment, 1995.

and among third world nations. These demands

will stress the technology communities’ abilities

to model human behavior and to represent a broad

spectrum of environmental conditions” (131).

❚ Virtual Simulation

Some DIS proponents argue that constructive

simulations often do not simulate human behavior

well, and that the solution to this problem is to put

human soldiers (but not their vehicles and weap-

-ons) back into simulation. Partly for this reason

(validity)--but mainly because it was designed to

train small units--SIMNET put human soldiers,

by the company, into a computer simulation and

tried to evoke realistic individual and organizational behavior by presenting them with a realistic, threatening tactical environment, using sights

and sounds to raise the tension level as much as

conveniently possible (see box 2). The sight-andsound presentation created by SIMNET is

counted by many as “virtual reality” or a “synthetic environment” (see figure 5) (182,147).

Participants in a DIS exercise can get very immersed in the experience. During an early SIMNET exercise, a tank commander dove out of his

tank to take a compass reading, as he had been

trained and had grown accustomed to get out of a

real steel tank in order to get an accurate magnetic

compass reading, unaffected by the magnetic effect (permeability) of the tank. When he realized

where he was and what he had done, he went back

into the simulator (95).

THE EVOLUTION OF DISTRIBUTED

INTERACTIVE Simulation

Computer-mediated distributed interactive simulation can be traced to the Air Force’s Semi-Automatic [sic] Ground Environment air defense

system, which was developed during the 1950s

(147). Virtual simulation can be traced to the first

Link Trainer, demonstrated in 1929 (147), although rehearsing or playing at war with props for

weapons-a form of virtual simulation-is

doubtless prehistoric. However, modem distributed interactive simulation began with DARPA’s

SIMNET program

SIMNET

SIMNET, a billion-dollar (194) development program of DARPA (now ARPA) and the Army, took

existing simulation technology and applied microprocessing and networking technology to

achieve an interactive simulation. SIMNET simulates the combined arms battlefield, allowing
16 Distributed Interactive Simulation of Combat

SIMNET marked a shift in simulator design from an engineering or physical reproduction approach in the tradition of the Link Trainers and flight simulators to manipulating “brain state” and relying

more on human factors analysis. Bringing the psychological realm of human behavior into simulation

enabled designers to understand just which sensory cues to give the trainee in order to move his mental focus past the physical mechanism he occupied to the virtual battlefield. “The design did not concentrate on the armored vehicle per se; rather... design became behaviorally driven and began by identifying cues that crew members needed to learn their specific duties and tasks. ”1

The entertainment industry has long been expert at using just those sounds and sights that engage the imagination of the audience while eliminating any distraction that would break the “suspension

of disbelief. ”2 Some of these techniques were used in the SIMNET simulators. For example, the sound

of the tank firing was enhanced with overtones from a tiger’s growl3 to add an adrenaline rush to enhance the realism of the simulation.

The difference between the Link-style approach and the SIMNET-style approach is reflected in

two different concepts of fidelity. Fidelity in the simulation world means much the same thing as it does

in the audio world, A “high fidelity” stereo system duplicates the real sound so that it is hard to tell the

difference. The effectiveness of simulators has traditionally been measured the same way—by how

close the experience felt to the real thing. In SIMNET, however, it was found that fidelity or “accuracy in

details” 4 was not necessarily important to achieve immersion that was necessary for teaching skills and

sometimes not as effective as specific cueing, a concept the SIMNET team called selective fidelity.

Instead of starting with a complete cockpit and subtracting the extraneous components, the SIMNET team started with nothing and built a simulated tank interior based on what they felt they needed to

cue a desired behavior. The beginning of the design process was a task list of what skills they wanted

to teach. Nothing was added to the design that did not contribute to eliciting some specific, desired

training behavior.5 This approach resulted in a lower fidelity and much cheaper simulator, which made it

possible to produce large number of simulators to network together.

1 E.A. Alluisi, “The Development of Technology for Collective Training: SIMNET, A Case History,” Human Factors, 33(3):

343-362, 1991.

2 J. Isdale, “What is Virtual Reality? A Homebrew Introduction,” March 1993, available Online at <URL:ftp://sunsite.unc.edu/

pub/academic/computer-science/virtual-reality/papers/whatisvr.txt>.

3C. Portman, Hollywood soundman, interview with L. Voss at Perceptronics, Inc., Woodland Hills, CA, August 1992,

4 Webster's New Collegiate Dictionary (Springfield, MA: G. & C. Merriam Company, 1974).

5 J. McDonough, interview with L. Voss at Illusion Engineering Inc., Westlake Village, CA, August 1992.

forces as large as battalions to fight opposing connections, as shown in figure 2. The image genforces of similar size; it provides an alternative to

combined arms field exercises. The network can

link hundreds of simulators at widely dispersed

sites in real time. By the end of the project almost

300 simulators—Ml Abrams tanks, M2 and M3

Bradley fighting vehicles, aircraft, and command

posts (2)—had been produced.

The hardware for a SIMNET-type simulator includes a computer (called a simulation host), an

image generator, a sound system, and network

erator is a special-purpose computer that processes the simulation data and generates computer

images of the action. The simulation hosts are

connected via local- and wide-area networks to

other simulators engaged in the same battle. When

the tank simulator is networked into a battlefield

simulation, the crew see the action of the battlefield in computer-generated scenes through windows or portals at their stations. The actions by

their tank simulator—fire and maneuver—are re4
Distributed Interactive Simulation of Combat 17

What the SIMNET designers discovered was that faithful reproduction of the physical details of

the equipment and even the battlefield visuals were less important than the action itself in engaging the

battlefield behavior. Encountering other combatants in a battlefield environment elicited a combative

response.

A psychologically different interaction results when trainees become aware that they are working

with others on a battlefield, whether cooperatively with a wing man or in opposition with an enemy platoon. The importance is in projecting one’s conscious thinking process onto the battlefield.6

In SIMNET the battlefield radio communication, group skills, tactics, and plans for engaging the

enemy are implemented in real time in an environment that approximates the experience of the battlefield under controlled circumstances. The purpose is to train advanced skills. SIMNET trains soldiers

who are already proficient in individual skills (such as driving and gunnery) the next level of skills, which

is working together as a unit. At this advanced level, the fidelity of system and environmental representations is less important than the fidelity of the human interaction. Interactive simulation enables soldiers

to learn group fighting skills and enables commanders to improve tactical and strategic decision making.

A SIMNET simulator has several video slots, each portraying that part of a battlefield a war fighter

would see looking through a corresponding window of a cockpit. Early in the development of SIMNET

instructors noted that shortly after the beginning of a simulation session, an experienced war fighter

ceases to notice gaps or seams in the displayed scene, because he easily projects out into the battlefield and is absorbed in the experience and the interaction with other people. An inexperienced war

fighter, on the other hand, does not recognize the pattern of battle unfolding as easily and remains distracted by the imperfections of the simulation.7

It remains a tenet of successor programs, such as the Army’s Close Combat Tactical Trainer project, that the important aspect of battlefield fidelity is the percentage of desired behavior learned. This

tenet shifts the focus from the machinery to the human interaction.

6 H. Crooks, interview with L. Voss at Illusion Engineering Inc., Westlake Village, CA, August 1992.

7 Ibid.

SOURCE: Office of Technology Assessment, 1995.

fleeted in the other simulators, too, and replicated

in sound and sight. If one tank is hit by another, the

damage is calculated by the tank that was hit and

reported back through the network. The image

generator generates scenes of the rough geographical features of the battlefield-hills, blue or

cloudy skies, trees, houses—and the features of

battle---enemy tanks, perhaps hiding behind trees

in the distance; smoke and fire; helicopters overhead; and the boom of mortar fire.

SIMNET evolved from a 1979 DARPA project, the tank gunnery trainer (TGT). The TGT was

designed to use an Apple II computer to overlay

computer-generated imagery (CGI) on background imagery played back from a videodisc.

This innovative approach, which was demonstrated in late 1979, promised to provide gunnery

practice at a much lower cost than range gunnery

practice. The TGTs were expected to cost less than

$10,000 in quantity.

In 1981, Jack Thorpe, an Air Force officer, became manager of the TGT program. He proposed

networking five TGTs to provide a low-cost platoon-level training device, a tank team gunnery

trainer (TTGT), in which five trainees would be

shown the same scene and would compete to be

the first to sight and fire at an enemy tank. He also

proposed a different type of networking that
18 | Distributed Interactive Simulation of Combat

would permit large numbers of simulators to be

networked so that they could interact—e.g., so

that the crew in one simulator would see an image

of tanks operated by the crews of other simulators

moving on the displayed virtual battlefield.20 In

1982 DARPA awarded contracts directed toward

this goal and also approved the SIMNET project,

an extension of it focused on Army collective

training needs (2). 

The first SIMNET contracts were awarded in

1983. At that time the network architecture concept still envisioned generating imagery for each

simulator centrally and transmitting it over the

network to the simulator. It was soon discovered

that this would be too costly, but it also appeared,

at least to Thorpe, that microprocessors capable of

rendering a simulator’s imagery would soon be

available and affordable enough to put a computer

image generator (CIG) in each simulator. The

links of the planned simulator network would then

not need the high information-carrying capacity

required to transmit imagery; instead, each simulator would only need to broadcast a short message (a packet21 or datagram22) on the network

every second or so to notify other simulators of its

position and certain other attributes (heading, velocity, turret azimuth, gun tube elevation, etc.).

The computer in each other simulator would examine each datagram to decide whether the datagram was from a tank within, entering, or leaving

its simulated field of view (2). If so, the simulator’s CIG would make the appropriate change in

the scene that it displayed.

A breadboard23 version of the SIMNET CIG

was demonstrated late in 1984. In January 1985 a

prototype M1 tank simulator, made of plywood,

was demonstrated to the Secretary of the Army,

the Chief of Staff, Army, and other Army officials.

Army interest and funding continued. Later that

year two developmental M1 simulators were networked together at the annual convention of the

Association of the U.S. Army. By May 1986, two

production-model M1 simulators were networked, and by October 1986, eight M1 simulators—two platoons—at Fort Knox, Kentucky,

had been networked with computers that represented artillery fire support elements, close air

support aircraft, a tactical operations center, and

an administrative and logistical operations center.

In November 1987, two attack aircraft simulators

were added to the net. On simulator displays,

these looked like hybrids of A-10 and F-16 aircraft, so exercise participants and observers jokingly dubbed the simulated aircraft ‘‘A-13s,’’

because 13 is the average of 10 and 16.

By the end of 1987, the first battalion-level

training exercises had been conducted with five

air defense simulators and two helicopter simulators added, bringing the total number of simulators at Fort Knox up to 56 fully interactive ground

vehicle and aircraft simulators (2). By February

1988, the SIMNET facility contained a total of 71

interactive simulators. Battalion-level force-onforce operational exercises with Forward Area Air

Defense elements were conducted in April. A test

20 In 1978, while working at the Air Force Office of Scientific Research, Thorpe had proposed a similar networking of flight simulators (2).

21 A packet is a physical (e.g., electrical or optical) signal composed of a sequence of pulses, representing 1s and 0s, in a certain order: first a

special sequence of pulses (a flag) marking the beginning of the packet; then a header describing the packet type or format, the address of the

computer or computers for which it is intended, and in some cases a number used by the receiving computer to determine whether the packet was

garbled during transmission; then the information field containing the data, message, or message fragment (e.g., a virtual tank’s position), which

the rest of the packet merely transports; and finally a flag marking the end of the packet.

22 A frame is the information represented by a packet. A frame is a logical object; a packet is a physical (if intangible) object. A datagram is a

frame used in certain types of networking protocols (standard communication procedures), such as the Transmission Control Protocol (TCP)

and Internet Protocol (IP).

23 In the development of electronic equipment, a breadboard version refers to a prototype that performs the electronic functions required but

is not constructed mechanically like the intended product. In the early days of radio development, experimental or hobbyist radio equipment

was often constructed by mounting components on a breadboard borrowed from the experimenter’s kitchen (or on a wooden board of similar

shape), instead of a more costly steel or aluminum chassis of the type used for commercial and military equipment.
Distributed Interactive Simulation of Combat | 19

of long-haul networking the next month was successful. By the fall of 1989, six SIMNET sites or

nodes had been established at Fort Knox; Fort

Rucker, Alabama; Fort Benning, Georgia; and European sites (2).

At these sites, four-man tank crews took their

stations inside simulators built to look like tanks

on the inside (as shown in figure 1-1). The crew

members operated switches, knobs, levers, and

other controls replicating the controls in a real

tank of the type being simulated. Through the simulated periscopes and gunsight, they saw scenery,

including simulated vehicles, pass by as they

drove or traversed the turret.

Each simulator simulated the dynamics of a

tank’s engine, transmission, drive train, and suspension: the scenery would pass more slowly

when the simulator ascended a hill (because

tracked vehicles slow down on a grade) or traversed a muddy field (because tracked vehicles

slip or bog down in deep mud). In addition, sometimes the gun or engine would fail to work, because unreliability was simulated, as was damage

to electrical, hydraulic, weapons, and other systems. The virtual vehicles broke down, consumed

fuel, and expended ammunition at rates calculated

from experience with real vehicles.

❚ Computer Image Generation

SIMNET gambled on, and successfully exploited,

breakthroughs in computer image generation. In

the 1960s, DARPA sponsored head-mounted display research that would find application in future

flight simulators (147). In the 1970s high-resolution CIGs in flight simulators projected wideangle views that followed the movement of the

pilot’s eyes on a dome above the pilot’s head.

Computer-generated imagery was used in the Advanced Simulator for Pilot Training (ASPT) in

1976; in the Visual Technology Research Simulator at the Naval Training Systems Center (NTSC)

in Orlando, Florida (now the Naval Air Warfare

Center Training Systems Division), and in the

Unit Conduct of Fire Trainer (UCOFT), a fullmission tank simulator (2).

The TGTs used in the TTGT were simulators

with a handle with triggers on the front and an eyepiece through which to view scenes played back

from a video recording. The five trainees in the

TTGT would be shown the same video-disc-generated scene and compete to be the first to fire on

an enemy tank (18). Random access video, in

which the trainee could change direction and get a

different view (provided it had been prerecorded),

was the next step. But even random access video

disc technology, in its infancy in 1979 (18), was

not flexible enough for interactive training. All

the possible images could not be recorded in advance on video. They had to be generated as needed by the computer. CIG seemed integral to

accomplishing interactive simulation.

However, during the late 1970s CIG cost $2

million to $3 million per channel,24 and CIGbased trainers ranged from $5 million for UCOFT

to $30 million for ASPT (2). For SIMNET simulators, the concept was to have the simulator’s microprocessors project the appropriate moving

image for each member of a tank crew. This required eight different views or CIG channels for

each SIMNET tank simulator. Lower cost became

imperative.

By the end of the SIMNET project, the cost of

each visual display subsystem was less than

$100,000 (2). The 20- to 30-fold cost reduction

from the cost per CIG channel in the early 1970s is

partly attributable to the decline in cost and the increase in scale (i.e., number of transistors) of integrated circuits during that period. It is also partly

attributable to the fact that each display system

(periscope or gunsight simulator, called a vision

24 ‘‘Channel’’ or ‘‘visual channel’’ when used with respect to [computer-generated imagery] systems, implies one computing entity, often

part of several, which processes image data for that part of the simulation presented visually through a display system. Such parts may be one of

several elements of the outside visual scene. . . the greater the number of. . . channels, the larger the potential. . . [field of view] provided by the

display system’’ (136).
20 | Distributed Interactive Simulation of Combat

block) for which a SIMNET CIG generated imagery displayed fewer pixels25 than did a typical

flight simulator display. This could be justifed because a tank crewman looking through a periscope

has a smaller field of view than does a fighter pilot

looking through a canopy. (A SIMNET tank simulator did not provide an open-hatch view for the

commander.)

Today, as competition continues to drive the

costs of computer image generation down, the

compromise on resolution is being reversed: newer simulations are using increasingly higher resolution graphics to serve other uses besides

training. A 1992 Defense Science Board summer

study of simulation, readiness, and prototyping

concluded that CIG technology advances would

be driven by the commercial sector in the near future (191). Performance and resolution of highend graphics work stations are increasing, and

cost is decreasing. Advances by the consumer

electronics industry and video game producers are

creating widespread benefits for the visual display

market, including better and cheaper image projection systems and liquid crystal displays, which

are used in head-mounted displays for flight simulation, surgery, and automotive repair, and in infrared scopes such as those used in the Persian

Gulf War (33).

The fidelity of simulated scenes depends not

only on the types of phenomena modeled but also

on scene complexity and display resolution. Most

image generators represent surfaces (e.g., terrain)

as connected polygons (e.g., triangles), so the

number of polygons in a scene is a measure of

scene complexity. Rendering a more complex

scene takes longer, so a more useful performance

metric is polygons per second, calculated by multiplying the scene complexity in polygons by the

frame rate in Hz (147).

The image generator used in the developmental

SIMNET M1 tank simulator could render scenes

at 120 thousand polygons per second26—thus it

was named the 120T CIG (98). Its frame rate was

15 Hz (2), so it rendered scenes composed of

8,000 polygons. This composite was divided

among eight channels—one for each vision block.

There were three vision blocks for the driver, three

for the commander, and one for the loader, each

displaying an image 320 pixels wide by 128 pixel

high, and a higher-resolution vision block for the

gunner (98).

Production models of SIMNET simulators

used GT100-series CIGs produced by BBN Delta

Graphics, as was the 120T. The GT100 series included the GT101, which was equivalent to the

120T; the GT102, which was equivalent to two

120Ts and was usable for 16 channels or for eight

higher-resolution channels; the GT110, which

was equivalent to 10 GT101s; and the GT120,

which was equivalent to two GT110s (98).

Each CIG had to store an object representation—information about the shape, size, texture,

and pigmentation—of each type of simulated vehicle participating in the simulation; it also had to

store representations of the terrain, vegetation,

and cultural features (buildings, roads, and

bridges, etc.). Terrain elevation information was

organized into load modules each representing a

500-mi2 patch of terrain as a surface composed of

32 triangles (which are polygons). Each CIG

could store 196 load modules at one time in

memory—enough to represent a square 3.5 km on

a side. Each simulator was required to display ob25 Pixel (or pel) is an abbreviation for picture element—a dot on a display screen. For comparison, a Super VGA (SVGA) computer monitor

can display an image 1,024 pixels wide by 768 pixels high.

26 For comparison, the current state of the art in commoditized systems is illustrated by the capabilities of the fully scaled-up Silicon Graphics, Inc., Onyx Reality Engine 2 (over 2 million triangles per second (129)) and of the Division, Inc., Pixel Planes 6 system (4 million photo-textured (147) or 5 million Gouraud-shaded (147) triangles per second (28)). Considerable scaling will be required to meet the rendering requirements that ARPA has specified for its Simulation-Based Design (SBD) project: 30 million polygons at 30 frames per second, or 300 polygons

per second (147).
Distributed Interactive Simulation of Combat | 21

jects and terrain out to a range of 3 km, so when a

simulated vehicle moved 500 m in one direction,

it would load into memory the load modules representing the next row of 500 mi2 terrain patches

beyond those already represented ahead of it,

overwriting the load modules representing the

row of 500 mi2 square terrain patches that it was

leaving behind.

Typically the heights of the vertices of the triangular patches of terrain were elevations at 125-m

intervals. Representing coastlines and rivers in a

recognizable form required using elevations at

smaller intervals, which the CIG could accommodate as long as each load module contained no

more than 32 triangles. A terrain patch that included a coastline might use most of its 32

triangles to represent the coastline and would need

few to represent the ocean.

❚ Synthetic Environments

Terrain Database Construction

Battlefield interaction takes place on complex terrain, and the virtual terrain of a synthetic environment should reflect its important features. A tank

commander’s view of the battlefield is very different from that of a pilot, and tank commander’s requirements for detail in terrain representation are

different. Having a full field of view is critical for

pilots; providing it requires a terrain database and

a means of rendering and displaying the scene that

the pilot would see, whatever the direction of his

gaze. A tank driver, on the other hand, needs only

the restricted field of view provided by his vision

block; however, his tank simulator needs to know

soil properties that affect traction. Ideally, the behavior of the soil should also be simulated. For example, if wet or sandy, the soil should develop ruts

if many tracked vehicles traverse it. 

A significant indirect cost of the use of computer image generation is the digitization of terrain

data for use as terrain databases. Those databases

contain all the information about a piece of land

necessary for the computer to project hills, trees,

and other features combatants would need to see.

A terrain database describing Fort Hunter Liggett,

California, has been widely used for DIS demonstrations because of its availability and detail.

Rapid construction of terrain databases is needed for mission rehearsal on a contingency basis for

two DoD programs in particular: the Advanced

Research Projects Agency’s Warbreaker program

(see box 3) and the Air Force’s Special Operational Forces Aircrew Training System. The Air Force

requires the capability to rehearse, in simulation,

for a mission anywhere in the world with only 48

hours advance notice, using a site-specific (not

just typical) terrain database representing 500,000

nmi2 with photographic texture in multiple levels

of detail (89). DoD’s draft MSMP contains a number of goals for rapid terrain database generation,

including a requirement to ‘‘demonstrate an initial

capability in FY 1996 to produce, within one

week, standard terrain data to meet M&S functional area requirements contained within a nominal 2,500 km2 area,’’ and a requirement to

‘‘demonstrate in FY 1997 the capability to produce standard terrain data to meet modeling and

simulation functional area requirements contained within a nominal 2,500 km2 area (with

three-dimensional terrain, including three-dimensional man-made features, reasonably attributed),

within 72 hours’’ (195).

The Undersecretary of Defence for Acquisition

and Technology has recently appointed the Defense Mapping Agency’s Terrain Modeling Project Office to be the modeling and simulation

Executive Agent for terrain modeling, to be responsible for meeting the requirements for terrain

database generation set forth in the MSMP.

Cartography, the depiction of the three-dimensional Earth in two-dimensional media, has been

revolutionized by the use of airborne and spacebased remote sensing imagery. Remotely sensed

data, unlike ground survey data, provides multidimensional views of terrain using multispectral

scanners, infrared cameras, imaging radars

(36,198), and photography. Another revolution

has been in the direct use of digital terrain. The ultimate product of cartographic data need not necessarily be a paper map. The raw electronic digital

cartographic information can be directly used and
22 Distributed Interactive Simulation of Combat

The Advanced Research Projects Agency’s Warbreaker program focuses on finding and destroying time-critical targets, including tactical ballistic missile launchers, such as those that launched Scud

missiles during the Persian Gulf war. The program is a part of DoD’s Precision Strike Initiative, Like the

Air Force’s Semi-Automatic Ground Environment air defense system1 and some subsequent projects,2

Warbreaker uses DIS to simulate a complex system of systems: the weapons, sensor, intelligence, and

vehicle systems that must work together to destroy critical mobile targets, The first step of the program

was to recreate in simulation the systems and vehicles involved in targeting Scud missiles, modeling

their limitations as well as their capabilities. The next step is to improve the process for a more successful outcome.

The objective of the program is to change system design, acquisition, and training in this particular area by evaluating a mix of existing and potential systems to determine how they can operate together more affordable and efficiently and to identify new technologies needed to accomplish the task.

The technologies that will be developed to accomplish the mobile targeting include end-to-end integration of data collection, targeting technologies, and allocation of resources for strike.3 Technologies that

will be exploited include those of data fusion, battle management, and intelligence collection, processing and distributing. Connecting national assets to theater-level targeting systems that support mobile

and fixed target acquisition is a part of the design.

The program is part of a Distributed Defense Simulation testbed4 that serves several larger objectives as well. ARPA hopes to use the Warbreaker infrastructure as a procurement tool that brings

together the technologist, the developer, and the user for increased effectiveness in weapon systems

development and procurement, Warbreaker will also establish a platform for integrating and networking

existing systems more effectively. Ultimately, the goal is to establish a DoD-wide multilevel secure DIS

infrastructure in which diverse simulations and simulators can be networked into a scenario.

1 J.F. Jacobs, The SAGE Air Defense Systems: A Personal History (Bedford, MA: MITRE Corp., 1986)

2 An example is the simulated interoperation of the U.S. Air Force’s Tactical Air Control System and U.S. tactical air defense

systems with air defense systems of the North Atlantic Treaty Organization. See J.E. Freedman and S.H. Starr, “Use of Simulation in the

Evaluation of the IFFN Process, ” in North Atlantic Treaty Organization Advisory Group for Aerospace Research and Development

Conference Proceedings No. 268 (AGARD-CP-268), presented at a Meeting of the Avionics Panel, Paris, France, 15-19 October

1979.

3 U.S. Department of Defense, Defense Advanced Research Projects Agency, “Warbreaker (Critical Mobile Targets) System

Engineering and Integration Research and Development,” preproposal conference presentation, Arlington, VA, Apr. 6, 1992).

4 U.S. Department of Defense, Defense Advanced Research Projects Agency, Advanced Systems Technology Office, “Distributed Defense Simulation for System Engineering in the Acquisition Process: The SERCES and the Warbreaker Programs,” industry

briefing paper (Arlington, VA: 1992).

SOURCE: Office of Technology Assessment, 1995,

imbedded within other systems (96). Direct digi- for military operations including telephone poles,

tal data are used for navigation and routing, but for

many uses a map is more informative than any picture could be.

Three-dimensional terrain databases have been

generated from photographs in two weeks using

four weeks of manpower. Overhead imagery can

be turned into three-dimensional scenes showing

key buildings, topography, and important features

power lines, and roads (37). Digital information

on troubled areas is increasing, and techniques for

direct use of remotely sensed data are improving

(97). The final step will be to learn to feed that information directly into targeting weapon systems.

Challenges for terrain database generation include the need to combine data collected by a variety of sensors and stored on various media in a
Distributed Interactive Simulation of Combat | 23

variety of formats and security classifications

(146). Timely collection of terrain data is also a

challenge. One of many technologies that might

be useful is airborne or spaceborne interferometric

synthetic-aperture radar, which can collect elevation data even at night and through cloud cover

(36,198). A different type of challenge is updating

terrain database in real time to represent changes

in the terrain, such as cratering, that should occur

as a consequence of simulated actions, such as

barrage or bombardment (147).

Although technological advance is increasing

scene rendering detail and speed, it does not yet

satisfy the most demanding requirements for terrain detail and extent, such as for strike mission

simulation and simulation-based design of complex systems such as ships (147). This is a critical

issue in real-time interactive simulation as is terrain database correlation (agreement of the databases, describing the same real or synthetic

terrain) among different simulators. Terrain database correlation problems are as old as DIS (in the

generic sense of the term), having occurred in the

development of the Air Force’s Semi-Automatic

Ground Environment air defense system (76) and

in numerous subsequent cases.

Other Synthetic Environments

The draft MSMP includes action items intended

to fulfill the objective of providing timely and authoritative representations of the natural environment—including the oceans, the atmosphere, and

space, as well as terrain. Modeling these environments was not essential in the early days of SIMNET, when it was sufficient to fight simulated

tank battles at high noon on clear days. Expansion

of DIS to accomodate joint exercises involving a

greater variety of platforms and sensors (acoustic,

radio, infrared, and optical) set in various climates

and times of day has made it imperative to model

many aspects of the environment—not just terrain. The Undersecretary of Defense for Acquisition and Technology has recently appointed

Executive Agents for modeling the oceans, the

lower atmosphere, the upper atmosphere, and outer space, to be responsible for meeting the requirements for terrain database generation set forth in

the MSMP.

❚ Semiautomated Forces

Realistic collective training of an Army unit requires simulation of some situations in which the

unit is greatly outnumbered by opposing forces. It

would be expensive if all the opposing forces were

virtual forces—soldiers, perhaps experts in foreign force doctrine, operating simulators. To avert

this cost, DoD has developed, and is improving,

automated computer-generated forces (CGF) and

semiautomated forces. Using automated or semiautomated opposing forces allows a simulation to

populate the virtual battlefield with a larger number of enemy troops (and perhaps neutrals and

noncombatants) and to test proposed doctrine and

tactics on a larger scale. SAF were required to simulate large-scale battle exercises for command

training, such as REFORGER (see box 4). However, the effectiveness of using an automated

force, is limited by the realism of its behavior. The

original SAF simulation in SIMNET was rudimentary, but the technology is being improved by

efforts such as the Advanced Research Projects

Agency’s Intelligent, Imaginative, Innovative,

Interactive What If Simulation System for

Advanced Research and Development (I4

WISSARD) program.

Current problems and limitations in semiautomated forces include the number of control personnel required to manage SAF in an exercise and

the realism of SAF behavior. The goal is SAF that

are so capable that the operator of a simulator will

be unable to tell the difference between SAF and

forces controlled by other manned simulators.

Different types of entities (tanks, fighter planes,

and dismounted soldiers, for example) pose different challenges for simulation.

Efforts towards this goal are resulting in a shift

from SAF that are algorithm-based (as in SIMNET) to SAF that are based on AI (artificial intelligence). In the original algorithm-based SAF, the

computer would move each vehicle step by step

based on its next objective. Each tank stays in

formation with other tanks and avoids obstacles.
24 Distributed Interactive Simulation of Combat

Milestones in distributed simulation were achieved during the 1992 REFORGER exercise. In previous years the annual exercise had deployed thousands of troops in Europe. In 1992 much of the action was simulated by networking and interoperating the Air Force’s Air Warfare Simulation and the

Army’s Corps Battle Simulation using a new translation protocol, the Aggregate Level Simulation Protocol. The Aggregate Level Simulation Protocol synchronized the timing of events in the two constructive

simulations and relayed event and status messages from each to the other, so that the results of one

could influence the other, ’

Perhaps the most significant feature of the exercise, however, was the saving of $34 million over

REFORGER ’88. REFORGER ’88 had deployed 17,500 people; REFORGER ’92 deployed only 6,500

people, but simulated the actions of 175,000 people, including the semiautomated forces controlled by

200 experts at Fort Leavenworth, Kansas. The post-exercise review concluded that the training accomplished in REFORGER ’88 was negative for lower-level troops,2 who mostly sat at roadsides, and minimal for command staff. REFORGER ’92 had fully documented, free-play interaction resulting in positive

training at all levels. However, because REFORGER ’92 was live at the command post level but

constructive at the soldier level, fewer soldiers were trained than in 1988, This is an improvement only if

one accepts the proposition that the negative learning by soldiers in REFORGER ’88 outweighed the

value of any positive learning at that echelon. Another way to summarize the results: REFORGER ’92

demonstrated the potential of constructive simulation to save money and other resources in providing

realistic command and staff training in the context of a large-scale scenario, but it was not designed to

provide realistic training opportunities for individual soldiers.

1 Earlier in 1992, in the annual Ulchi Focus Lens exercise, the Aggregate Level Simulation Protocol had linked computers

running the Corps Battle Simulation in Korea with computers running the Air Warfare Simulation and the Navy’s Research, Evaluation

and Systems Analysis model at the Warner Preparation Center in Germany. Later, a constructive Marine Corps simulation was added

to the confederation of simulations that could be linked by the Aggregate Level Simulation Protocol. See MITRE Corporation, Aggregate Level Simulation Protocol (ALSP) Program Status and History (McLean, VA: March 1993); available online at <URL:http://alsp.arpa.mil/alsp/89-92-history/89-92-history.html>.

2 U.S. Department of Defense, Office of the Undersecretary of Defense for Acqusition, Defense Science Board Task Force on

Simulation, Readiness, and Prototyping, Impact of Advanced Distributed Simulation on Readiness, Training and Prototyping, condensed (Washington, DC: January 1993).

If the lead tank sees an obstacle and changes direc- scenario can be added to the code for a more comtion, the next tank may slowdown to avoid a collision. As the lead tank moves around the obstacle

and gets back on course, the second tank has to

readjust to get back in formation. Multiply these

decisions by 16 tanks and the result is sometimes

erratic behavior.

This control base has been improved with a

concurrent control base that takes the same goals

and arbitrates them using weighted distributions

for each goal in an ongoing balancing of the directive for the tank. Using this “fuzzy logic” method, also, additional objectives for a mission

plex response.

The next step up in complexity is knowledge

acquisition by the SAF using techniques of artificial intelligence, including expert systems and

case-based reasoning. Expert systems are developed by knowledge engineering: interviewing an

expert (e.g., a pilot) to discover the rules of thumb

and information he or she uses to solve a problem,

then putting that information into the software in a

form that a computer can use to direct action. This

is a laborious process. An alternative is case-based

reasoning: designing SAF software that can learn
Distributed Interactive Simulation of Combat 25

REFORGER ‘88 REFORGER ’92

Deployed troops 17,487 6,500

Engaged troops more than 97,000 more than 20,000

Engaged tracked vehicles 7,000 135

Engaged tanks 1,080 0

Simulated troops more than 97,000 175,000

Simulated tracked vehicles 7,000 12,810

Simulated tanks 1,080 1,950

Maneuver HQ 35 41

Exercise scale corps vs. corps army group vs. front

Style umpire free play, intelligent opposing force

Intelligence play scripted sensor level feedback

Training effect (battalion and below) negative (on roads) positive (at home station)

After-action review anecdotal analytic data

cost $53.9 million $19.5 million

SOURCE: Office of Technology Assessment, 1995.

from its own successes and failures in simulated would turn out differently. This could provide a

combat. For the computer to “learn” from experience, it must be able to acquire knowledge from

new experiences in real time and add it to its

knowledge base (i.e., expertise) in such as way as

to be able to use it later. The computer’s knowledge base consists of many cases that the computer uses in real time to match the scenario it is

dealing with and respond accordingly. If the computer sees something new, it is added to the database. Case-based reasoning is the basis for

machine learning in 14 WISSARD (142).

Currently, such added case-based learning is

coded in by hand after an after-action review.

Ideally, a computer could do such a review automatically in realtime. A pilot could train the computer by dogfighting with it. If the pilot performed

a maneuver that the machine had never seen before, the computer might be killed but would add

that trick to its knowledge base. If the pilot tried to

repeat his success, he would find that the computer does not respond in the same way. The scenario

richer training experience for the pilot and make

the simulator training useful longer.

❚ Simulation Architecture

The DoD’s draft Modeling and Simulation Master

Plan defines architecture as “the structure of components in a program [or] system, their interrelationships, and principles and guidelines

governing their design and evolution over time”

(195). It calls for the development of a comprehensive, abstract high-level architecture for DoD

simulation: “the major functional elements, interfaces, and design rules, pertaining as feasible to all

DoD simulation applications, and providing a

common framework within which specific system

architectures can be defined” (75,167).81 The current, still evolving DIS architecture evolved from

that of SIMNET.

When the SIMNET program began, most simulation data processing was done by a central

27 Examples of specific system architectures are the proposed Communications Architecture for Distributed Interactive Simulation and the

object and event architectures embodied in the Distributed Interactive Simulation protocols.
26 | Distributed Interactive Simulation of Combat

computer.28 Initially SIMNET was envisioned to

use a similar architecture: control inputs from several simulators would be transmitted to a central

computer, which would process the data and send

video signals back to display systems—television

receivers—in all the simulators. Each simulator

would require several television channels—one

per simulated periscope or gunsight—and it was

soon realized that a cable from the central computer would only be able to serve a few simulators.

The architecture was not scalable—that is, the

number of networked simulators could be increased eonomically.

SIMNET’s program manager, Jack Thorpe, invented a different, more scalable architecture.

Each simulator would be designed to do its own

simulation processing and image generation.

Each simulator was to be a self-contained unit,

with its own host microprocessor, terrain database, display and sound systems, cockpit controls,

and network interface. Each would generate all the

sights and sounds necessary to train its crew. Each

tank crew member would see a part of the virtual

world created by the graphics generator using the

terrain data base and information arriving via the

net regarding the movements and status of other

simulated vehicles and battle effects—the precise

part being defined by the crew member’s line of

sight: forward for the tank driver or from any of

three viewing ports in a rotatable turret for the tank

commander.29

With the shift from centralized processing to

autonomous, distributed processing, the concept

of data communication changed. Each simulator

was to communicate with every other simulator

instead of with a central mainframe computer.30 In

addition, each simulator is responsible for what it

represents on its displays. Each simulator’s microprocessor carries a model for all of the objects

on the network, including itself, that describes

that object, its state, and activity and is responsible

for updating that model when something changes.

In order to overcome the problem of massive

amounts of data exchange, the SIMNET team decided that constant updates were not necessary,

only significant changes needed to be communicated. In between, the simulator projects other objects, based on its internal model and assuming

there is no change, according to its last information: if a tank is traveling a particular direction at a

certain speed, a SIMNET simulator projects the

vehicle continuing on in that direction at that

speed until told otherwise. This way, the action

continues between update information and the

speed of update merely influences the smoothness

of the transition to a new state. A more sophisticated version of this ‘‘dead reckoning’’ idea is

‘‘remote entity approximation,’’ which incorporates not only first order approximations like velocity, but also second order approximations like

acceleration. The key idea behind both dead reckoning and remote entity approximation is the substitution of local data processing, which is easy,

for data transmission to each simulator over the

network, which is expensive and constrained.

The basic design of the network for SIMNET

was set by 1985. Simulators were linked through

local area networks onto a larger wide area net, the

long-haul network. With that design in mind, the

project was divided into three different areas of

contractor effort: networking technology, simulator design and manufacture, and computer generated graphics for the simulators.

The wide area network (WAN) that DARPA

used for SIMNET was based on a network that had

already been developed for them by BBN called

the Terrestrial Wide Band Network (TWBnet).

DARPA expanded it into a network, called the De28 However, in 1978 the Naval Training Systems Center had converted its mainframe-based simulation processing to a multi-minicomputer

architecture, a precursor to distributed processing (2).

29 J. Thorpe, 1987, cited in (2).

30 In recent years, as the number of entities in large simulations has increased, this sort of broadcasting has become a bottleneck and has been

replaced by a technique called multicasting. See the section Scalability And Scaling, below.
Distributed Interactive Simulation of Combat | 27

fense Simulation Internet (DSI), that could be

used by the Defense Department for simulation.

(It is also used for video teleconferencing.) Development of the DSI is now managed by the Advanced Information Technology Services Joint Program Office (AITS-JPO), a partnership between

ARPA and the Defense Information Systems

Agency (DISA) established in 1994. ARPA continues to develop the technology; and DISA will

have responsibility for operating the DSI after it

becomes operational. In February 1995, the DSI

Program Manager (PM), Stuart Milner of ARPA,

handed over the program management responsibility to Commander Walt Corliss of DISA, recognizing that the network communications

infrastructure had become more stable and reliable. The concentration of the DSI PM has shifted

to processes, procedures, and exercise support.

ARPA’s Real-time Information and Technology

Networking (RITN) research and development

program will continue to feed new technology

into the DSI (69).

The DSI’s lines are being upgraded for better

communications and continually extended to include more ‘‘nodes,’’ or locations where one or

more simulators networked together on a local

area network (LAN). A LAN connects the simulators at a particular site to one another and to the

WAN (the DSI). In late 1994 the DSI had about

100 nodes. Once operational, the DSI will become

part of the Defense Information Systems Network

(DISNet), which will ‘‘migrate’’ to become part of

the Defense Information Infrastructure (DII) and

will be interfaced with the Global Command and

Control System (GCCS), so that commanders can

train ‘‘at home’’ using their wartime command

and control systems. DoD’s draft Modeling and

Simulation Master Plan calls for greater exploitation of wireless technology, including satellite

communications, so that forces may participate in

simulation even when deployed overseas, and to

link live forces into distributed simulations, as

was demonstrated in the 1994 STOW-E exercise

(see box 5). 

The decision to use distributed data processing

and the network concept required a new architecture for data communication over the network that

would allow information to be passed back and

forth between hundreds of simulators fast enough

to convey all the necessary interaction in real time.

The DIS architecture that evolved is, even more

than the computer-generated visuals or the selective fidelity simulators, the great legacy of the

SIMNET program.

The DIS architecture is an open architecture

based on the International Standards Organization’s Basic Reference Model for Open Systems

Interconnection (OSI): ISO 7498-1984. The ISO

OSI networking standard provides a seven-layer

structure that specifies standards and protocols

covering everything from the physical network to

the data interchange between simulators. There

are specific standards for communication over the

long-distance network, for electronic communication for each particular type of local area network, and for the particular application, in this

case the SIMNET simulation. In terms of this

seven-layer standard, the SIMNET protocols are

the application layer (the top layer, layer seven)

that uses the services provided by the network and

data link layers without necessarily requiring services from the intermediate layers (113). The standard provides common rules by which

independent users can play on the net.

An important recent development is the establishment, by the Under Secretary of Defense for

Acquisition and Technology, of the DMSO Architecture Management Group (AMG) to oversee development of the High-Level Architecture (HLA)

called for by the DoD Modeling and Simulation

Master Plan draft of January 1995. The AMG’s

membership was approved by the DoD Executive

Council on Modeling and Simulation (EXCIMS)

on Mar. 22, 1995, and the AMG held its first meeting on Mar. 31, 1995. Over the course of the next

year, the AMG will review and evaluate a series of

prototypes conducted to test and further define the

architecture. Based on the results of these prototypes and other analyses as necessary, the AMG

will prepare a baseline definition of the HLA,

which is due in June 1996. This architecture will

be recommended to the EXCIMS, who, after appropriate review, will submit it to the Under Secretary of Defense (Acquisition and Technology)
28 Distributed Interactive Simulation of Combat

Synthetic Theater of War (STOW) is a program managed by the Advanced Research Projects

Agency (ARPA). It aims to demonstrate a capability for large-scale (ultimately theater-level) battlefield

simulation using advanced distributed simulation (ADS) to combine virtual, constructive, and live simulated combat onto one, interactive electronic battlefield. The program has several objectives: 1) improve the quality of simulations through entity level resolution of combat and environmental representations in simulation, 2) improve simulation training effectiveness and flexible interfaces with operational

command, control, communications, computers, and intelligence systems, 3) integrate distributed live,

virtual and constructive simulation models, 4) reduce the overhead costs of simulation knowledge

based, semiautomated forces, 5) provide faster databases with improved information transfer, 6) improve after action and analysis tools, and 7) provide improved simulation-driven crises rehearsal capabilities to warfighters.1

STOW was originally recommended by General Paul F. German and the Defense Science Board

in 1992 as one of 12 experiments to prove the usefulness of advanced DIS technology to a variety of

DoD organizations responsible for weapon system acquisition and joint warfighting. The project was at

first called the Southwest Experiment, because all the military services had instrumented ranges in the

southwestern United States that could be linked together to create a joint (i. e., multiservice) synthetic

battlefield. 2 The objective was to link several instrumented training and test ranges with simulator sites

to test proposed weapons and joint warfare doctrine using live, constructive, and virtual simulation.

In August 1992, Director of Defense Research and Engineering Victor Reis signed a Memorandum of Agreement with the Joint Chiefs of Staff (JCS) to “formulate and pursue” the recommended

technology demonstrations. The agreement assured the emphasis on joint war fighting.

Operators reviewed the original experiments, and eventually six categories of demonstrations

were funded: the synthetic battlefield, force generation, F-1 6/Joint Electronic Combat Electronic Warfare

Simulation, commercial battle games, Special Operational Forces/Joint Task Force integration, and the

Warbreaker program, which included demonstrations of technology for Theater Area Missile Defense

and for finding and attacking Critical Mobile Targets. The JCS allocated $11.5 million through 1995 for a

synthetic battlefield demonstration that would involve all the Commanders in Chief (CINCs) of combatant commands and all the military services.

The Joint Staff Operational Plans and Interoperability Directorate (J-7) joined with ARPA to use the

demonstrations to “find practical ways in which ADS can improve joint doctrine, plans, operations, training, and education. ”3 The alliance combines the resources and operational requirements pull of the

CINCs with development of a technology that promises to help satisfy the new initiatives in joint war

fighting exercises mandated by Congress with the new East Coast simulation center and the new mission training responsibilities of the U.S. Atlantic Command (USACOM) specified in the Unified Command Plan,

1 Department of Defense, Advanced Research Projects Agency, Advanced Systems Technology Office, “Synthetic Theater

Of War, ” as of May 12, 1995, <URL:http://www.arpa.mil/asto/stow.html>.

2 Preference for center location later shifted toward South Carolina. USACOM, influenced by the Navy and Marine Corps,

preferred South Carolina, where the Marine Corps has a warfare center and the Navy’s Battle Force Tactical Trainer and naval bases

are located.

3 Maj. Gen. A. Rogers, Joint Staff, J-7, presentation, National Training Systems Association conference, Alexandria, VA, March

25, 1993.
Distributed Interactive Simulation of Combat 29

Three major technology demonstrations have been scheduled for STOW, each a milestone marking the end of a phase of the ADS current technology demonstration plan. The first, Synthetic Theater of

War-Europe (STOW-E), was conducted from November 4 through November 8, 1994, during, and as

part of, the annual U.S. Army Forces Europe JTF exercise Atlantic Resolve ’94 (the successor to REFORGER). It demonstrated the linking and interaction of live, virtual, and constructive simulations, and

marked the end of phase I of the ADS current technology demonstration plan. About 3,500 entities—aircraft, ships, armored vehicles, etc.—participated. The maximum number of entities participating simultaneously—the “high-water mark’’—was over 2,000. The first technical papers reviewing achievements

and problems are beginning to appear as this background paper goes to press.4

STOW-E stemmed from a request by General Maddox, Commander in Chief of U.S. Army Forces

Europe, who, having SIMNET simulators and the live range of the Combat Maneuver Training Center

(CMTC) in Germany available, wanted to be able to use the capabilities of simulated training to tie together a SIMNET battalion, constructive forces, and live forces on the instrumented range to do largescale exercises. In Germany today for political, economic, and ecological reasons it is not possible to

maneuver live units larger than a battalion.

In STOW-E, constructive simulations, such as the Brigade/Battalion Battle Simulation (BBS), were

linked to the simulation network via a translation unit (called an Advanced Interface Unit or Advanced

Translator System), which translated constructive simulation results into DIS-compatible messages. The

translators were designed to disaggregate a platoon (for example) that is simulated as an aggregate

unit by the BBS into four tanks visible to SIMNET simulators whenever a SIMNET simulator approached

the agregated platoon close enough to see one of its tanks. STOW-E also used ModSAF, a DIS-compatible constructive simulation. Translators were also used by the participating field instrumentation systems, such as the CMTC Instrumentation System, to report the locations and status of real vehicles to

the simulation network so that representations of them could be seen by simulator crews. Several bandwidth reduction techniques-designed to reduce network congestion and permit more entitles to be

simulated—were demonstrated in STOW-E.

The next major technology demonstration planned for STOW is Synthetic Theater of War-1997

(STOW-97), to be sponsored by USACOM, focused at the Joint Task Force level, and conducted in

1997, In late 1994, the projected scale of STOW-97 was revised downward from 100,000 entities to

50,000 entities. The third and final major demonstration planned for STOW is Synthetic Theater of

War-2000 (STOW-2000), to be sponsored by USACOM and conducted in 2000. It will mark the end of

phase Ill of the technology demonstration plan, i.e., of technology insertion and transition to operational

systems.

4 University of Central Florida, Institute for Simulation and Training, 12th DIS Standards Workshop Proceedings, July 14, 1995,

available online at <URL:ftp://ftp.sc.ist.ucf.edu/public/STDS/workshop/12th/proceedings/> . See also P. Marion, “STOW-E-First Impressions Report,” Nov. 28, 1994, available online at <URL:gopher://triton.dmso.mil.:70/00/twg/dsi/dsi-docs/stow-e.txt>

SOURCE: Office of Technology Assessment, 1995.

for approval. During the course of its work, the ❚ Uses and Limitations of SIMNET

AMG will brief DoD’s Modeling and Simulation SIMNET matured as a company-level simulation

Working Group, which supports the EXCIMS, on and now easily handles battalion-level actions.

its activities (165,166,1 12,180).
30 | Distributed Interactive Simulation of Combat

The primary controversy surrounding SIMNET is

the degree to which it is possible, or even desirable, to create a large battle by connecting SIMNET nodes (and their other-service equivalents,

as these develop) together (see e.g., box 6). While

intuitively one might think that the whole is the

sum of the parts and that therefore with enough

SIMNET simulators one could simulate a large

battle, other considerations may supervene.

For example, SIMNET focused on tanks and

other vehicles, and DIS does not yet handle the individual soldier well. Yet the humble foot soldier,

modernly equipped, retains his key role in warfare. The considerable antitank potential of

today’s infantry makes the foot soldier an important player on the tanks’ battlefield and the key to

the tanks’ survival should they venture into woods

or urban areas. The Army’s Integrated Unit Simulation System (IUSS) project and Individual Port

(I-Port) project have demonstrated that individual

dismounted infantry troops can participate in synthetic combat (147), and the Army recently

awarded a $1 million contract to Avatar Partners,

Inc.,31 of Boulder Creek, California, for the development of a wireless unencumbered virtual reality

system—called Dismounted Infantry Virtual Environment (DIVE)—for infantry training applications. (121) However, it may not be feasible,

affordable, or necessary (for some training tasks)

to populate a synthetic battlefield with as many

dismounted troops as there would be on a real

battlefield.

Another problem is that most participants in a

battle spend much more time waiting than fighting. Moreover, battle is a stochastic (i.e., random)

process—actually, the result of many stochastic

processes. To deduce valid results from a simulation, it would be necessary to simulate the stochastic processes32 and to do so many times, so

that typical outcomes and the range of outcomes

can be discerned.33 Considering the hours of inactivity that most trainees would endure in an accurate simulation, it is questionable whether the

troops’ training time (and simulator time) would

be well spent in repeating the same scenario many

times in order to estimate the expected outcome

(e.g., losses) and the range of outcomes. Perhaps,

if the goal were a simulation of a large battle, all of

the forces should be semiautomated, not just one

side’s. Indeed, computerized combat models featuring large forces, in some cases resolved down

to the vehicle level, have been available for some

time. These include Corps Battle Simulation,

Concepts Evaluation Model, Janus, Joint Integrated Contingency Model, TACWAR, etc.34

SIMNET’s graphical interface will be wasted on a

higher echelon commander, whose personal point

of view will be the inside of a command post regardless of whether the battle is fought inside a

computer, in SIMNET, at NTC, or in the real

world.

SIMNET’s use for large battles has been proposed for two different purposes. One is exercise,

31 Avatar Partners, Inc., is now called DIVE Laboratories, Inc. DIVE Laboratories, Inc., will also assist General Reality Company in an

effort to ‘‘develop a VR-based dispersed command post architecture for military commanders,’’ for which General Reality Company was

awarded a Phase 1 Small Business Innovation and Research (SBIR) contract from US Army CECOM (47) .

32 In SIMNET, this is done by letting humans control some processes, such as aiming and firing weapons. Other stochastic processes, such

as wind velocity, are simulated by computers using Monte Carlo methods—in effect, throwing dice electronically. The result of a roll of a pair of

electronic dice is adjusted to give the right average speed and gustiness—i.e., range or standard deviation of speed. For example, the average

value rolled by a pair of dice is 7; values range from 2 to 12, with a standard deviation of about 1.71. In order to model a westerly wind with an

average speed of 14 knots, gusting to 17.5 knots, one might double the value rolled to get the speed in knots.

33 The result of a single simulation might be valid in the sense that it could occur in reality, but it might be so unrepresentative of outcomes of

similar conflicts that one would be foolish to assume that it is a reliable prediction of what will happen, without some knowledge of the range of

outcomes that occur in several runs of the simulation and without some evidence that outcomes of comparable real-world situations usually lie

within the range of simulation outcomes.

34 Some of these models were developed at Federally Funded Research and Development Centers (FFRDCs), formerly called Federally

Chartered Research Corporations (FCRCs).
Distributed Interactive Simulation of Combat 31

At the time SIMNET was being developed, the Navy was working to develop a naval gunnery

training system, also, called Battle Force In-port Training (BFIT), later renamed Battle Force Tactical

Training (BFTT). The Navy was working on networking ships in port. The Defense Advanced Research

Projects Agency (DARPA) combined SIMNET networking technology and simulators with BFIT to demonstrate component interoperability (a first step towards joint warfighting training) the following month in

April 1990.

The BFIT demonstration networked a Navy ship, the U.S.S. Wasp, in port at Norfolk, Virginia, with

Marine Corps SIMNET simulated helicopters at Fort Rucker, Alabama, simulated Army tanks, Navy staff

command centers, and an observation node at the Institute for Defense Analyses. The demonstration

included a Marine Corps pilot and Army tank commander calling for Navy fire of a Tomahawk missile

over simulated terrain (using the Fort Hunter-Liggett, California, terrain database). The simulated Tomahawk mission required command, control, and communication between Navy, Army, and Marine Corps

elements to coordinate missile flight through friendly forces to its target. The simulated Tomahawk missile showed up on the Navy radar as did the pilots when they flew over the U.S.S. Wasp symbol on their

screens. It was the first time the Marine Corps, Navy, and Army played on the same network using their

own systems. 1 The joint operation was not completely smooth, but improved with practice.

The demonstration not only illustrated the DIS technology’s joint component interoperability, but

also illustrated some technical accomplishments and features of the technology. Two valuable features

of DIS technology are its modularity and its flexibility. Ultimately, any type of computer system that can

do the processing and display can be brought onto the network and added to the interaction (within the

limits of the number of sites supported by the networking technology). In the DARPA/Navy BFIT proofof-principle demonstration the computer consoles were real consoles in real ships, turning real equipment into simulators. This type of simulation, with outputs from simulation interactions used as inputs for

the visual displays of real equipment such as sonar and radar screens, is called stimulation. So not only

were geographically dispersed simulators networked in real time, but simulators designed for interactivity were also networked with real equipment not specifically designed for interactive training. This was

demonstrated on a larger scale in the 1994 STOW-E exercise. A milestone yet to come is the routine

incorporation of equipment for interacting in simulation (called field instrumentation in the DIS community) into platforms such as tanks and airplanes.

1 BFIT II Battle Force In-Port Training 1992 (videocassette), VO 92030, U.S. Navy, Naval Ocean Systems Center, CA, 1992.

SOURCE: Office of Technology Assessment, 1995.

the other is simulation. At present, the large com- duced, so that its utility could be evaluated prior to

puterized combat models are used in these ways.

In command post exercises CPXs they determine the combat outcomes resulting from the

commanders’ decisions. As simulations, they are

used to develop answers to questions such as force

sizing and deployment. In the latter vein, one possible use of SIMNET is “virtual prototyping, ” in

which a candidate weapon would take its place on

the simulated battlefield before it is even prothe effort and expense of production.

SIMNET proponents argue that the older combat models lack the humans necessary to simulate

human behavior. SIMNET is constructed to present its trainees with a realistically threatening environment, using sights and sounds to raise the

tension level as much as conveniently possible.

The sight-and-sound presentation created by

SIMNET is counted by many as “virtual reality”
32 | Distributed Interactive Simulation of Combat

or a ‘‘synthetic environment.’’ Therefore, it is argued, SIMNET forces will display behavior more

human than the behavior programmed into the

forces that inhabit pure computer models.

This point begs the question of whether the

SIMNET players, regardless of how tense,

aggressive, or scared SIMNET can make them,

will really behave as they would if they were in

mortal danger. A related question, which is important in simulations conducted for research (rather

than training), is whether the participants are as

qualified, trained, and experienced, as will be

those who would fight a real war. This issue is not

peculiar to DIS; it has been raised earlier in the

context of CPXs, other constructive simulations,

FLEETEXs, and crisis simulation games, many of

which (for various reasons, including training) are

played by participants much less experienced or

distinguished than those they attempt to emulate.

The argument that human participation is necessary for realism seems difficult to reconcile with

acceptance (by proponents of that argument) of

the use of CGF. If the CGF don’t behave realistically, the battle is only half-realistic in this regard,

and if they do then CGF could be used as friendly

forces, in simulations conducted for research or

the provide realism and context in simulations

conducted for training. This apparent contradiction is not necessarily a real contradiction in simulations conducted for training, in which changing

the behavior of the trainees under certain conditions is the objective, or if semiautomated forces

(rather than CGF) are used.

FIDELITY, FOG, and FRICTION

Computerized combat models are often received

with skepticism—how can a computer program

possibly capture such a complex activity as combat? The quality of SIMNET’s synthetic environment, like that of a stereo, is judged in terms of

fidelity. Music lovers sometimes have an opportunity to compare the output of their stereos to the

real thing (as they recall it) just by listening, and

a stereo manufacturer or broadcast or recording

engineer can assess fidelity by measuring the

sound content of real music and of his product’s

output. SIMNET designers were under no illusion

that they could fool trainees into actually believing they were in tanks in battle, and they recognized the need to put their efforts where they

would do the most good. Thus they decided to

measure their success in terms of the degree to

which the trainees’ behavior resembled that of actual tank crews, and to adopt a selective fidelity approach in which only the details that proved to be

important in shaping behavior would be replicated. Cost considerations received their due, and

thus the inside of the SIMNET tank simulator

could well be termed a minimalist representation

of a tank’s interior, while the sounds presented by

SIMNET are quite realistic: sound effects are

cheap, and people react strongly to them. Especially evocative of reality are the low-frequency

‘‘sounds’’ delivered directly to the seat of the driver’s pants: these verge on being tactile stimuli

rather than sounds, and convincingly recreate the

sense of driving over uneven terrain.

Users of SIMNET and other virtual reality systems almost universally report an initial dismay at

the apparent lack of fidelity, followed by acceptance and a report that ‘‘it seemed so real’’ when

they start to interact with the program and get a

sense of involvement.35 This feeling can be replicated at any video game arcade.

One can easily carp at particular gaps in realism, and any such observations should be preceded by consideration of the selective fidelity

concept and the definition of fidelity as percentage

of behavior learned. For example, the German use

of cars dressed up as tanks in their prewar maneuvers was a shortcoming in realism but did not stop

the German tank crews from deriving considerable training benefit and the German commanders

35 SIMNET workers feel that the ultimate accolade, sometimes given after a live field exercise, is a soldier’s statement that ‘‘That felt just

like in the simulator!’’
Distributed Interactive Simulation of Combat | 33

from honing their Blitzkrieg theory of warfare

(17).

Yet the gaps in fidelity are there. Even the National Training Center, which affords the trainees

real tanks, real terrain, real weather, and human

opponents, has been criticized on the basis of residual artificialities such as the inability of its laser

‘‘bullets’’ to penetrate smoke and dust. Such criticisms of NTC, SIMNET, or other high-tech simulations often evoke the response that the low-tech

alternatives, such as shooting at white paper targets, are even less realistic. Some skeptics reject

even this argument, maintaining in effect that ‘‘a

little fidelity is a dangerous thing’’ and that the

convincing realism of high-tech simulation could

or does perniciously foster negative learning in

which the trainees adapt to the simulation, artificialities and all, to the detriment of actual field

performance (see box 7).

Accuracy and fidelity must be considered in

context. SIMNET’s sketchy presentation of the

interior of the tank is accepted because the purpose of SIMNET is not to familiarize tank crews

with the interiors of their tanks, but to give them

experience in using a tank that they already know

how to drive. Indeed, for some purposes one could

validly dispense with all the high-tech equipment

and revert to model tanks on a sand table. An

American unit commander in Desert Storm did

just that to coordinate the movement of his vehicles and avoid a traffic jam.

Use of SIMNET to create large simulated

battles could decrease fidelity by departing from

what SIMNET does best—intense armored combat at the company or battalion level. The larger

battlefield requires that the soldier perform many

other tasks, such as repair and maintenance of his

equipment and of himself (including sleeping),

ensuring the security of rear areas, and just plain

living outdoors. SIMNET does not allow for the

simulation of these tasks.

If the goal is to provide higher echelon officers

with training and exercise of their command functions, use of SIMNET is not necessary: a CPX,

with combat resolution provided by some other

means, would afford a realistic training environment for the officers. If, on the other hand, the goal

is to fight large battles in SIMNET to find out how

large battles work (a laudable goal—for all the effort expended by humanity on war, land battles remain poorly understood) or to experiment with

new formations, equipment, or force structure,

then SIMNET or something of its ilk may be the

right tool, if one can be sure that results are accurate.

Clausewitz cited the great difficulty of obtaining correct information on the battlefield and

dubbed it ‘‘the fog of war’’ (16).36 Traditional

combat models have often been faulted for being

idealized to the point of omitting the fog of war.

SIMNET and other DIS, through their use of multiple people and portrayal of some communications processes, promise to replicate the fog of war

better than have previous models. In one SIMNET

run witnessed by OTA observers, three M1

Abrams tanks covered the retreat of four M-2

Bradley fighting vehicles that had been deployed

far forward as scouts. The four Bradleys safely

reached the main U.S. position, a ridge line occupied by hull-down friendly vehicles. The returning M1s were not so lucky: one was destroyed by

enemy fire, one was mistaken for an advancing enemy and became an instance of fratricide, and the

third became disoriented and lined up on the

wrong side of the ridge. Ordered to proceed to the

rear, it turned around and proceeded forward and

was promptly demolished by the advancing enemy. A traditional computer model would have

been hard pressed to produce such a set of occurrences. However, the use of semiautomated forces

threatens to remove the human element and reintroduce unrealistic clarity.

36 Clausewitz’s classic work, Vom Kriege [On War], was first published posthumously by his wife in 1832.
34 Distributed Interactive Simulation of Combat

Providing and manipulating the virtual environments of trainees is done to promote the learning of

behavior that will be useful on the battlefield and to promote the retention and transfer of those skills to

the real battlefield. However, the effectiveness of the training is limited by the understanding of the designers of the virtual environment. Error or inadequate fidelity can cause negative learning transfer: the

learning, in simulation, of behavior that is rewarded with success on the simulated battlefield but is inappropriate on a real battlefield. This possibility is of particular concern to designers and users of combat aircraft mission simulators. Split-second timing is critical in some phases of strike missions and defensive counter-air missions. Where the motor skills coordination must be precise, the simulator must be

equally precise. 1

SIMNET designers encountered what they considered a negative learning transfer when they studied a SIMNET predecessor, the Unit Conduct of Fire Trainer. One trainee thought he was greatly improving his gunnery until his trainer kept quiet. Without the external help directing his attention to targets, the gunner’s success rate returned to his initial low percentage.2 In SIMNET, concern has been

expressed about negative learning transfer because the hatches in SIMNET tank simulators are “buttoned up, ” After bumping their heads a couple of times, commanders might learn not to look out their

hatches.

Artificialities in the live simulation conducted at the National Training Center may also cause negative learning. Because weapons’ fire is simulated with lasers, smoke and dust not only provide good

visual cover, they actually make the participant impervious to bullets in the sense that a hit will not register since the laser beam will not penetrate the obscurants. A training result might be an overreliance on

dust and smoke for cover and overconfidence under such conditions.

How serious a problem is negative skill transfer? Opinions vary, but it is generally agreed that the

benefits of simulation training and the advantages it provides for combat readiness far outweigh any

negative transfer.

Negative learning can even happen in real combat situations. During World War II, the British

engaged the Italians in North Africa and learned how to fight the Italians successfully. When the Germans arrived to reinforce the Italians, the British had all the wrong fighting skills.3 Fighting skills involve

a constant learning process, some of it corrective. With technology changes and upgrades to equipment, old skills have to be unlearned and new skills applied. Dealing with corrections to skill sets is a

fact of life with or without simulation, Adaptability in combat is, itself, an essential skill. However, the

nature of simulation in training is such that small design mistakes can handicap large numbers of

people and any potential negative skill transfer is something that requires, therefore, vigilant watching

for and correction as part of the verification, validation, and accreditation process.

1 Imprecise coordination of computer-generated imagery with simulator base motion (if any) can also cause simulator sickness, which is closely related to motion sickness (seasickness, airsickness, etc.).

2 J. McDonough, interview with L. Voss at Illusion Engineering Inc., Westlake Village, CA, August 1992.

3 M. Carver, Dilemmas of the Desert War (Bloomington, IN: University of Indiana Press, 1986), p. 16.

SOURCE: Office of Technology Assessment, 1995.
Distributed Interactive Simulation of Combat | 35

Clausewitz separately identified ‘‘friction’’ as a

major force on the battlefield. SIMNET can replicate some sources of friction, such as vehicle

breakdowns, but many believe the real world of

the battlefield to harbor a myriad of misadventures not provided for in SIMNET or other models.

VERIFICATION, VALIDATION, AND

ACCREDITATION

Predictions generated by computerized combat

models are often received with skepticism—how

can a computer program possibly capture such a

complex activity as combat? The issue here is the

validity of models and simulations and, more fundamentally, the process—validation—by which

validity is assessed. The Department of Defense

defines validation as ‘‘the process of determining

the degree to which a model or simulation is an accurate representation of the real world from the

perspective of the intended uses of the model or

simulation’’ (153,155,185).

A variety of techniques have been proposed

and used to validate computer models of combat;

they include critical analysis of the data and computational procedures used in the models and

comparison of the outputs (predictions) of models

and simulations to actual outcomes—of historical

battles (e.g., see references 34 and 15) or exercises

(e.g., see references 41 and 141)—or to the predictions of other models and simulations.

Critical analysis of data for correctness is an

element of data verification, validation, and certification (VV&C). Critical analysis of the computational procedures used in models is called

structural validation, which ‘‘includes examination of [model and simulation] assumptions and

review of the [model and simulation] architecture

and algorithms in the context of the intended use’’

(174). Comparison of model and simulation outputs to outcomes of historical battles is one type of

output validation, which “answers questions on

how well the M&S results compare with the perceived real world’’ (174).

Two related steps in the acceptance process are

often considered along with validation: verification and accreditation. DoD defines verification

as ‘‘the process of determining that a model or

simulation implementation accurately represents

the developer’s conceptual description and specifications’’ (185). It establishes that the computer

code is free of technical ‘‘bugs’’ (i.e., errors37).

DoD defines accreditation as ‘‘an official determination that a model or simulation is acceptable

for a specific purpose’’ (185).

To date, validation of SIMNET has rested

largely on users’ assertions that it ‘‘feels right.’’

Such declarations are evidence of face validity—

i.e., that on the face of it the experience is not so

unrealistic as to preclude the intended use. However, such declarations do not guarantee that the

effectiveness of the vehicle, crew, or unit will be in

combat what it was in simulation. In fact, some

observers worry that SIMNET may feel too

right—that it may be ‘‘seductive’’ (32).

For example, a tank commander must discern

an enemy tank at a distance of a thousand yards or

more, despite the camouflaged appearance of the

enemy and his likely exploitation of concealment

offered by vegetation or terrain. This task pushes

human vision to its limits. Although target acquisition feels the same in SIMNET as in real life

(each requiring great alertness and a quick reaction to the first sign of the barely-detectable enemy), real-life target acquisition entails the

interpretation of visual details far too subtle to be

presented on a SIMNET screen, creating a discrepancy in the average distance at which targets

are acquired. Such a discrepancy can be discovered only through a systematic validation effort,

such as one that was undertaken to validate target

acquisition models (205). Other discrepancies,

such as the tank commander’s limited field of

view in an M1 simulator, are more obvious and are

37 Reference number (78) describes the origin of this use of the word ‘‘bug.’’
36 Distributed Interactive Simulation of Combat

This illustration depicts a virtual battlefield scene such as might be displayed in a Close Combat Tactical Trainer simulator--except the armored

vehicles and aircraft are cut away to show cut-away views of the simulators in which crews are controlling the operation of their simulated vehicles on

the virtual battlefield. Also depicted is the cabling that connects the simulators. The vehicles shown include Abrams tanks (foreground and center), a

Bradley armored fighting vehicle (Ieft, with infantry troops dismounting), an Apache helicopter (above left), a U.S. Air Force A-10 Warthog close air

support aircraft (above Ieft center and above right), dismounted infantry troops (center, beyond Abrams tank), and various enemy vehicles (above

right). Close Combat Tactical Trainer simulators have not yet been built.

SOURCE: U.S. Army, Combined Arms Tactical Trainer Project Manager, 1995.

being addressed in the Army’s Close Combat Tac- about the Battle of 73 Easting (a tank battle in the

tical Trainer (CCTT) project—the follow-on to Persian Gulf War). DARPA compared the data to

SIMNET-T (see figure 6). the results of SIMNET simulations to see how

Various approaches to validating SIMNET closely the simulated battles compared to the real

through comparison of its results to presumed one (15). Similar comparisons have been perfacts 38 can be imagined. One validation project formed with constructive models such as CASTconducted by DARPA collected detailed data FOREM (50), the Concepts Evaluation Model

38 It is sometimes difficult to identify the real world to which a model’s or simulation’s output should be compared: “The basic point about

‘reality’ is the non-existence of general agreement as to what it is.. . . As in science generally, the dichotomy of’ fact’ and ‘theory’ is an oversimplification” (140). See also reference number (174).
Distributed Interactive Simulation of Combat | 37

(204), Janus(A) (107, 109), the Oak Ridge

Spreadsheet Battle Model (56), various Lanchestrian models (34,52,53,54,58,108), the Quantified Judgement Model (30), and other models

(94), applied to battles too numerous to list here

exhaustively39 but including the Battle of Iwo

Jima (34), the Battle of the Bulge (Ardennes 1944)

(204), the Inchon-Seoul Campaign (52), the SixDay War of 1967 (30), the 1973 War of Atonement

(30), the 1982 War of the Cherry Blossoms in the

Bekaa Valley of Lebanon (30), the Battle of 73

Easting in the Persian Gulf War (50,107), and the

World War II U-boat campaign in the Bay of

Biscay. In current DoD terminolgy, these comparisons would be described as output-validation activities intended to validate (or to invalidate) the

models for predictive uses by comparing their predictions to historical outcomes.

This approach to validation is not quite as easy

at it sounds. One problem is obtaining good data

for such an undertaking; this may be impossible

(e.g., if one is interested in validating a model of

strategic nuclear war between superpowers) or

could require more time and money than did creating the model in the first place. Those who attempt

it may be subjected to the unenlightened charge

that it is waste of money to predict the past. Another possible pitfall is that if modelers use all applicable historical cases to create the model (56),

comparing its predictions to those outcomes may

show what looks like a good fit to the layman, but

it would have no goodness of fit in a statistical

(i.e., scientific) sense, and the model’s predictions

would be of uncertain accuracy and reliability. A

statement such as ‘‘we have calibrated our model

to the results of the 1973 Arab-Israeli war, and it

can reproduce them perfectly’’ (62, p. 145) implies that the model has descriptive validity—i.e.,

it explains the outcome of the combat (22)—but

the statement provides no confidence in the reliability with which the model will predict the outcome of a similar future war, and other models

may be able to be calibrated to explain the outcome equally well.

It should also be recognized that the historical

outcome may have been a freak occurrence. For

example, naval warfare modelers would be hardpressed to reproduce the defeat of the Bismarck,

which entailed a lucky torpedo hit on the ship’s

rudder. Some have argued that the Battle of 73

Easting had many unique features that make it unrepresentative of likely future battles. Even if it

were perfectly representative, it is only one case,

and if other conflicts began the same way, they

might not end the same way. The late military historian Trevor DuPuy said in 1991 that:

I would be most reluctant to take 73 Easting

as the truth of history. It is true, but it is not necessarily consistent with the best truth. We must

have many, many more before we can say what

the lessons are that have been learned from this

(32).

A related approach would be to load SIMNET

with a battle fought at the NTC and see whether

the SIMNET results match NTC results. The

events of the Battle of Seventy-Three Easting,

while extraordinarily well-documented, are far

from perfectly known. For example, it is not

known whether the crews of some or all of the

Iraqi tanks were hiding in bunkers near their vehicles when the U.S. forces opened fire and could

not return fire immediately. It is also not known

whether some of the Iraqi tanks had been destroyed by U.S. bombing before the battle. Finally, the outcome of 73 Easting may have been one

of several possible outcomes. Use of NTC battles

for validation would be preferable, at least in this

regard, because the validators could pick an NTC

battle that has been fought many times and for

which there is a resulting strong sense of how it

usually turns out and of the variation in outcomes.

On the other hand, use of NTC battles to validate SIMNET for predictive uses would beg the

39 See references (30) and (56) for other examples.
38 | Distributed Interactive Simulation of Combat

question of whether NTC exercises are themselves valid live simulations. There are doubts:

I’m always concerned about NTC data, because it isn’t real war, despite the stress and

strains and drama of the exercise. People know

that they’ll be going home that evening and getting a warm, good meal and sleeping in a relatively warm bed (32).

More fundamentally, model-to-model and simulation-to-simulation comparisons are often dismissed with the argument that two simulations

could agree, and both could be wrong. However,

such comparisons are useful, because if it turns

out that two simulations disagree in an important

respect, then they cannot both be valid,40 and discovering this can be important. Moreover, if one

of the simulations was previously validated and

accredited, it could be used as a proxy for reality,

for purposes of output validation.

Another, and very different, approach to validating SIMNET (or any similar system) would entail detailed examination of its inner workings.

The obvious starting point for this structural validation is the set of assumptions embodied by the

model. Some assumptions are data. Some engineering data, such as armor thicknesses and vehicle road speeds, are so well known as to be

unlikely to pose any problem, but closely related

performance parameters, such as armor penetration and vehicle speeds over various types of sand

and soil, may be based on test-range trials whose

conditions may or may not make them suitable for

use.41 Assumptions that are not engineering data,

or perhaps not data at all, are harder to validate.

False assumptions are not the only threat: missing

assumptions are much harder to detect and their

consequences are often much more severe. One

can satisfy oneself that all of a model’s assumptions are true and still have room for concern that

the model’s output might be untrue.

Having examined the model’s data, a structural

validation effort would continue by examining the

equations and other relationships that make the

model function as it does. Some of these, such as

those governing the flight of projectiles, would be

straightforward and incontestable physics or engineering equations. Others, such as those relating

fires to smoke and smoke to obscuration, could

prove at least as contentious as the data used to

drive them.

In the case of simulations such as SIMNET that

present a person with sensory input, a final step in

validity checking is needed: that which establishes that the sights and sounds presented to the

trainee are as they should be. If some submodel

determines that a given burning tank should create

smoke that restricts visibility to 20 meters, does

the model’s synthetic environment in fact present

objects out to 20 meters and no farther when the

user drives into the smoke from this burning tank?

It would at first appear that virtual simulation

could not induce the perception of being in mortal

danger. If this were true, it might add an irresolvable obstacle to the assessment of validity of virtual simulation, which is not to say that it would

make inaccurate predictions. However, in some

cases it may be possible to measure the physiological stress of simulator users and compare it to the

stress of actual combatants. There are clinical precedents: in two recent experiments acrophobic patients were transported virtually (by means of a

head-mounted display) to a high place in a synthetic environment and their physiological signs

of stress were measured (63,82,134). The stress

was presumably a symptom of fear (phobia).

However, the only way to assess the fearfulness,

per se, of a subject is to ask the subject whether he

or she is fearful, as S.L.A. Marshall did for Men

Against Fire (91).

40 Possibly the simulations consisted of using models to generate predictions based on erroneous, invalid data. In this sentence, we use

simulation in a broad sense, to include data.

41 More likely, and much more threatening to validity, the performance parameters are as much the results of running other models as they

are of testing equipment in the field. For a detailed and seminal treatment of this problem, see (135).
Distributed Interactive Simulation of Combat | 39

The same could be done with users of simulators of real or proposed weapon systems. The difficult (but not impossible) step in such a

comparison would be making corresponding

measurements on and assessments of actual combatants. Some—perhaps most—people would

consider this a waste, believing it obvous that real

combat is more stressful and fear evoking than

simulated combat, for most soldiers. Cases of

simulator users losing the insight that ‘‘it’s not

real’’ are so rare and startling—even amusing—

that they are recounted as anecdotes, even in Congess, by proponents of virtual simulation.42

However, even if simulators do not (or are not allowed to) evoke the same stress and fear that combat does, the effect on a virtual simulation’s

predictive accuracy and reliability is unclear, because some individuals perform better when

stressed and fearful, while others perform worse.

Unit leaders endeavor to channel their soldiers’ responses to fear into constructive action rather than

panic.

Another way in which simulator operators may

behave unlike real combatants is that they may

participate in the same scenario so many times

that they learn. This is desirable for training but is

usually undesirable in research, in which the objective is often to predict how effectively human

operators will behave the first time they see real

combat. Accruing statistical confidence in such

predictions requires many repetitions (11,41,

141), but preventing the operators from learning

might require using a fresh crew for each repetition. An alternative approach that has been used in

the validation of a small-unit entity-level

constructive simulation (41,141) is randomization of scenarios, to estimate and account for

learning.43

The degree to which a model’s output should be

believed also depends in part on what the output

is. Models are to a considerable degree idealizations, and as such generally understate the deleterious effects of the fog of war, and portray military

activity as proceeding more smoothly than it really does. Similarly, even if a virtual prototype

works well in simulation, a physical system built

from the design may fail because of some detail or

phenomenon that was not modeled. On the other

hand, if a virtual prototype functions poorly, it

seems doubtful that unmodeled phenomena

would make it work better in the real world.

Accreditation constitutes official certification

that a computer model is acceptable for a particular purpose. It provides assurance to users, particularly contractors, that their work will not later be

dismissed on the grounds that it was performed

using a questionable computer model. It has been

suggested that accreditation for a particular purpose be extended not simply to models but to

model-user combinations (21).

A problem underlying much of the difficulty of

creating valid military simulation models is the

lack of necessary data, and in general the lack in

the U.S. system of what is termed military science

abroad (20). There is no consensus as to how one

validates a model and what constitutes a valid

reality against which to check it, though there is no

shortage of practical biases and philosophies

about what standards modeling and simulation

42 Dave Evans of Evans and Sutherland, a pioneer in computer image generation for cockpit simulators as well as head-mounted displays

(HMDs), has said that one pilot flying a Boeing 747 airplane simulator fainted when his virtual airplane hit a ditch on landing. Professor Frederick Brooks, quoting him, said that nothing so dramatic had happened to a trainee wearing an HMD (10).

43 For example, if a simulation experiment were conducted to assess whether tank units would be more effective, in a certain scenario, if their

tanks were equipped with a new radio system, the crews of 10 units might participate in 10 exercises each—5 exercises using the simulated

inventory radios and 5 exercises using the simulated proposed radios. The order in which each unit uses inventory radios and proposed radios

should be chosen randomly, so that, for example, crews do not all use the inventory radios 5 times while warming up and then the proposed

radios 5 times after having improved their proficiency—an improvement that careless analysis might attribute solely to advantages of the proposed radios. More sophisticated experiments and analysis could estimate the rate of learning and extrapolate performance with each radio back

to estimate cold (i.e., unrehearsed) performance with each radio or forward to estimate performance with each radio after much practice and

rehearsal.
40 | Distributed Interactive Simulation of Combat

should meet. Lack of standards for models inhibits their verification and use. A program manager

trying to select an appropriate model to use to support his program ‘‘finds it very difficult to locate

unbiased help in deciding what model to choose,

what data [are] acceptable, and what models have

been successful in the past. Each program manager is, to a great extent, left on his own to make

these decisions and then defend them against a

host of organizations which are chartered to challenge such decisions’’ (101, p. 36). Models tend to

be biased toward fulfilling the needs of a particular military service or defense agency. They will

represent with high fidelity the details of interest

to a particular service but not those of interest to

other services. Different communities have contradictory requirements for models. Models developed by the acquisitions community, for

example, are traditionally not accepted by test and

evaluation agencies. Current practice for test and

evaluation personnel is to assume that acquisition

models, which are developed by the contractors of

the system, are inherently biased (101, p. 2). In the

training community, the validity of SIMNET exercises is questioned, since they do not meet traditional training measures (201, pp. 14-15).

One interesting contrast between the field of

defense modeling and other similar fields, e.g.

econometrics,44 is the lack of a tradition of validating models routinely (21, p. 30). Peer review

would seem to present an ideal solution to the difficulties posed by the field of defense modeling,

but peer-reviewed journals cater to an academic

side of the field that is more that usually detached

from the real-world version. (This split is doubtless exacerbated by security considerations.) Articles appearing in these journals often have a

more judgmental character than articles in scientific journals. The field of defense modeling combines empirical, historical, mathematical, and

judgment in a mix likely to sow disesteem among

peers: the more subjective aspects create plenty of

room for disagreement, while the ability to point

to a few facts allows the disputants to feel that

their opponents are not dealing with reality.

It is in this context, then, that one must consider

short-term proposals to accomplish validation in

part, or entirely, by peer review of the work or by

the certification or licensing of the workers. Workers fear that such review, certification, or licensing

by peers would at best institutionalize back-stabbing and at worst foster the creation and self-perpetuation of a one-viewpoint orthodoxy.

In the long run, peer review could perhaps be

made to work if a healthier combat modeling culture develops. At present such a culture is hard to

identify because of the security classification surrounding a considerable portion of the work, the

disparate backgrounds of workers in the field, the

near-total lack of a full-fledged academic curriculum related to combat modeling,45 and the distaste

with which some view the organized study of how

to kill people in large numbers. These barriers

seem difficult to surmount, and lead to the discussion of standards in the following section. It is important to note, however, that one form of peer

review exists at present and appears to work—internal peer review. Through a conscientious program of internal peer review, some organizations

devoted to defense modeling and simulation do

manage, according to their own view, to enhance

the quality of their products.

A number of recent efforts to improve validation of M&S, including some used for DIS, are

noted in the following subsections.

44 Even in econometrics, the fitting of a model to data is more common than is the statistical testing of the resulting model on the basis of

subsequent data, accounting for degrees of freedom (i.e., the number of ways in which the model can be made consistent with the peculiarities of

the data to which it was fit, rather than to basic trends in the data). Complex models developed by academic economists are seldom validated

(83,84).

45 OTA recognizes that several first-rate universities offer courses of study and research in defense policy analysis, but these programs are

small and cast a wide net. They often combine political science and defense modeling in a single course.
Distributed Interactive Simulation of Combat | 41

❚ Army Pamphlet 5-11

The Army’s pamphlet, Verification, Validation,

and Accreditation of Army Models and Simulations, issued in 1993, (174) is the most detailed official plan for VV&A yet released by a DoD

component. More recently, the Marine Corps issued a VV&A plan, and the other services, and

DoD, have drafted plans.

❚ SEES 2.0 Accreditation Exercises

From 1993 (41) through 1995 (141) Lawrence

Livermore National Laboratory’s Conflict Simulation Laboratory (LLNL CSL) has documented

a series of force-on-force (FOF) exercises (live

simulations). They were conducted in October

1992 at the Wurtsmith Air Force Base Weapon

Storage Area to ‘‘accredit’’ (as well as validate)

the Security Exercise Evaluation System version

2.0 (SEES 2.0). SEES 2.0 is a multiuser, interactive, entity-level constructive simulation, that can

simulate the behavior of individuals in small units

such as infantry squads, security police patrols,

special forces units, or terrorist groups, under the

supervision of squad or team leaders. Although

SEES 2.0 is neither a DIS or a virtual simulation,

the method used by CSL to inform accreditation

could be used to validate distributed interactive

virtual simulations. We briefly summarize the

SEES accredition study here. 

The LLNL CSL developed SEES 2.0 for the

Air Force Security Police and the Defense Nuclear

Agency for use in evaluating the physical security

of weapon storage areas and other facilities. It was

developed from the Urban Combat Computer Assisted Training System and later evolved into the

Joint Tactical Simulator.

The accreditation study defined two quantitative measures of performance that were predicted

by the SEES, relevant to the application, and measurable in live simulations. One measure was the

friendly-to-hostile force ratio, which varied as the

live or constructive simulation progressed. The

other measure was the time (after the beginning of

the simulation) at which key events occurred, such

as the transport of a stolen weapon through the

weapon storage area security perimeter. For each

of several conditions (scenarios), several attacks

were simulated. In the terminology of experimental design, this provided for blocking and replication.

SEES 2.0 is a stochastic, Monte Carlo simulation. In different SEES simulations (as in live simulations) starting from the same condition,

different things happen in general and to the measures of performance in particular. Several SEES

simulations and several live simulations were

conducted for each scenario, to increase the statistical confidence with which one could conclude

that SEES was a valid model of the force-on-force

exercises, or to reduce (i.e., improve) the statistical significance with which one could conclude

that performance measures of SEES simulations

were statistically different from those of the FOF

exercises. The SEES and FOF results were

compared using a two-sample Kolmogorov-Smirnov test. (See box 8.) At most times during the

scenario, one could not distinguish the SEES

force ratios from the FOF force ratios with less

than a 10-percent chance of error. However, there

was one time at which the SEES force ratios could

be distinguished from the FOF force ratios with

less than a 10-percent chance of error, implying

that SEES did not accurately predict what the live

forces did at that time in the live simulation. Analysis of simulation data indicated that SEES predicted implausible individual and unit behavior in

a certain ambush situation. This conclusion led to

subsequent adjustment (tuning) of the SEES model. 

The accreditation study report acknowledged

that several issues of validity could be raised: one

is whether the live forces, who 1) used MILES

equipment instead of actual weapons, 2) were

obliged to observe Occupational Health and Safety Administration regulations, and, 3) because of

instrumentation limitations, were required to operate in daylight (contrary to doctrine and training) in most replications, could be considered

proxies for real forces operating under operational

conditions. The scaling of scenario time to

compare the pace of action in the live exercises

with the faster pace in SEES runs, and the use of
42 Distributed Interactive Simulation of Combat

The two-sample Kolmogorov-Smirnov (K-S) test1 is a statistical procedure that can be used to

decide whether two sets of data (for example, one set from a constructive simulation, the other from a

live simulation or from history) are statistically equivalent,

Specifically, the two-sample K-S test is a statistical procedure for deciding whether two groups

(samples) of numbers, each selected randomly (and independently, with replacement) from a larger

group (a population) of numbers, were selected from the same population or, alternatively, from different

populations.

For example, in one of the K-S tests performed in the SEES V2.0 accreditation project, one sample (called the simulation sample) was a set of 40 Red/Blue (i.e., hostile-to-friendly) surviving force ratios at two minutes after the commencement of hostilities, from 40 simulations of a scenario (called Scenario 3) using SEES V2.0, a constructive simulation with human participation. The other sample (called

the experimental sample) was a set of 20 Red/Blue surviving force ratios at two minutes after the commencement of hostilities, from 20 force-on-force (FOF) exercises with the same scenario. On the basis

of a two-sample K-S test, the authors of the project’s final report2 concluded, with a statistical significance of about 0.0281, that the two samples of force ratios were drawn from different populations—i.e.,

that the surviving force ratios predicted by SEES V2.0 were not the same, statistically, as the surviving

force ratios from the live simulation, The phrase “with a statistical significance of about 0.0281” means

“with a probability of error no greater than about 0,0281” (i.e., about 3 percent). It was later determined3

that this discrepancy arose in part because of “the inadequate modeling in SEES of human factors for

ambush situations. ”2

The qualification, “. . . about ., .,” refers, in part, to the fact that the authors used an approximate

version of the K-S test that is accurate for comparing very large samples (i. e., samples containing data

from many replications of a simulation or experiment) but is not necessarily accurate for comparing

samples of 20 or 40 numbers. An assessment of the accuracy for the sample sizes used in the accreditation project was planned but was never conducted because of resource limitations.

OTA also uses the qualification, “... about.., ” to refer to uncertainties introduced by other practices. For example, as noted above, the K-S test may only be properly applied to samples selected

the asymptotic (large-sample size) version of the lation capabilities are not adequate for evaluations

Kolmogorov-Smirnov test for the comparisons

were also flagged for discussion.

❚ A2ATD Validation Experiments

In 1994 the Army conducted the first of six experiments planned to validate BDS-D simulators,

SAF (called SAFOR), and the BDS-D simulation

(simulators and SAF operating together). The

VV&A effort is funded by the Army’s BDS-D

program and is conducted as part of the Army- and

ARPA-funded Anti-Armor Advanced Technology Development program. The VV&A effort is intended to address a deficiency of particular

interest to Congress: “The current BDS-D simuto support acquisition decisions. . . A2ATD. . .

was initiated with the goal of maturing DIS as a

credible evaluation tool to support acquisition decisions” (11). Box 9 describes the A2ATD in

greater detail.

❚ Joint Staff Instruction on VV&A

On January 12,1995, the Vice Director of the Joint

Staff issued Joint Staff Instruction 8104.01, Verification, Validation, and Accreditation of Joint

Models and Simulations (183). Modeled on Army

Pamphlet 5-11, it specifies general VV&A procedures for use by the Joint Staff.
Distributed Interactive Simulation of Combat 43

randomly and independently, with replacement, from a population. In the case of the simulation sample

or the experimental sample, the population refers to all the force ratios that would occur if the exercise

were repeated an infinite number of times, in each case under the same conditions (including participant experience). Random and independent selection, with replacement, implies that each possible

force ratio should have the same probability of occurring regardless of the replication number—e.g., it

should have the same probability of occurring in the first, second, and twentieth replication. This implies

that there should be no learning by Red or Blue forces, over the course of those successive simulations, in any way that would have an influence on the probabilities of force ratios at two minutes after the

commencement of hostilities. In order “to reduce [the] learning curve” in the force-on-force exercises,

the SEES accreditation project allowed the Red and Blue forces 38 “free plays, ” in addition to the 40

“record events”3 (20 for each of 2 scenarios), in the force-on-force exercises. Free play was also allowed in the later SEES V2.0 simulations. In spite of this, the SEES participants, “without coaching,

learned the same successful tactics as [did the] FOF [exercise] participants. ”3

The experimental and simulation samples for other times after the commencement of hostilities

(at least until eight minutes) were in better agreement. For times at 30-second intervals from the start

until seven minutes later (except for two minutes after start), the experimental and simulation samples

were not so different statistically that the K-S test could reject the hypothesis that they were drawn from

the same population with a statistical significance better (i.e., less) than 0.10. In fact, at 4.5 minutes

after the commencement of hostilities the experimental and simulation samples were so similar that the

hypothesis that they were drawn from the same population could not be rejected with statistical significance better than 0.9853,

1 D.A. Darling, “The Kolmogorov-Smirnov, Cramer-von Mises Tests, ” Annals of Mathematical Statistics, 28:823-838, 1957.

2 G. Friedman and R. Toms, Security Exercise Evacuation System (SEES) V2.0, Accreditation Project Final Report, February

1993, FOR OFFICIAL USE ONLY,

3R.M. Toms, “Results of a Verification, Validation, and Accreditation Excercise,” briefing, presented at the Office of Technology

Assessment Workshop on Issues in Combat Model Validation, Washington, DC, Feb. 8, 1995.

SOURCE: Office of Technology Assessent, 1995,

❚ TRADOC Research on DIS Validation

The Army’s Training and Doctrine Command

(TRADOC) has been sponsoring conceptual work

on DIS validation. One study, by the RAND Corporation, which was to consider alternative approaches to the problem, is nearly complete (26).

It elaborates ideas expressed in earlier RAND

work on validation (21,22,64), focusing on DIS

and emphasizing that there are a number of nonpredictive uses of DIS for which comparison of

model predictions—or DIS outcomes, construed

as predictions—to historical or future real-world

outcomes is neither necessary nor appropriate for

validation. For example, if a system of DIS equipment, software, and databases is to be used for

training, then validation should seek to assess the

degree to which trainees have learned the desired

conditional behaviors as a result of the training. 46

Predictive uses are classified into various subcategories of weakly predictive uses and strongly

predictive uses, the former denoting predictions

with considerable uncertainty, such as theater

combat simulations, and the latter denoting pre46 This begs the question: why are those conditional behaviors believed to be good or optimal’?
44 Distributed Interactive Simulation of Combat

The Anti-Armor Advanced Technology Development (A2ATD) program was undertaken by the

Army and DoD as a whole, with the goal of improving DIS applications enough to be credible as an

evaluation tool to support acquisition decisions. The purpose of the A2ATD is to develop and demonstrate a verified, validated, and accredited DIS capability to support anti-armor weapon system virtual

prototyping, concept formulation, requirements definition, effectiveness evaluation, and mission area

analysis on a combined arms battlefield at the battalion task force or brigade level.

The A2ATD program has four technical objectives:

demonstrate DIS as an evaluation tool and verify, validate, and accredit simulators used in A2ATD

experiments, semiautomated forces, and the Battlefield Distributed Simulation-Developmental

(BDS-D) simulation;

develop, demonstrate, and document techniques and analytical tools to evaluate the causes of

simulation outcomes;

link one or more constructive models such as Janus to DIS; and

demonstrate upgraded virtual prototypes of M1A2 Abrams tanks, M2A3 and M3A3 Bradley armored fighting vehicles, non-line-of-sight (NLOS) and line-of-sight antitank (LOSAT) weapons, and

virtual prototypes to be developed—of the Apache and Comanche helicopters, the Enhanced Fiber Optic Guided Missile (EFOGM), AGS, JAVELIN, and Hunter.

In FY 1993 VV&A of simulated and semiautomated forces and development of analytical tools to

support the evaluation of causes of simulation outcomes began, with the goal of providing the foundation for a series of six experiments in FY 1994-1996. The first experiment, completed in September

1994, replicated two M1A2 Initial Operational Test and Evaluation (lOT&E) vignettes to compare the results of virtual simulation with those of live simulation and to compare constructive simulation with live

and virtual simulation, Four experiments planned for FY 1995 will use BDS-D simulation of the effects of

heavy force modernization in a Southwest Asia scenario and demonstrate the linkage of Janus to BDSD, BDS-D and ModSAF VV&A will be systematically expanded by addition of the following simulators:

Apache, Comanche, M1A2, M1A2 firing STAFF, LOSAT, and NLOS. One experiment will be conducted

in FY 1996, a Rapid Fire Projection Initiative scenario with AGS, LOSAT, EFOGM, JAVELIN, and Hunter

simulators. Experiments 1 ,2, and 4 will be local-area network experiments conducted at the Mounted

Warfare Test Bed (MWTB). Experiments 3, 5 and 6 will be wide-area network experiments. Experiment 3

will link the MWTB, Land Warner Test Bed (LWTB), and the BDS-D site at TACOM. Experiment 5 will link

the MWTB, LWTB, and the Aviation Test Bed (AVTB). Experiment 6 will link the MWTB, LWTB, and BDSD site at MICOM.

The A2ATD experiment analysis cycle will proceed as follows. The scenario vignette and performance data will feed into BDS-D and the Combat Arms Task Force Engagement Model (CASTFOREM).

BDS-D simulations will be run with verified, validated, and accredited ModSAF and simulators (level 2

CIGs) in a level 2 environment with level 2 DIS standards. BDS-D simulation runs will be made with and

dictions with little uncertainty, such as some engi- ❚ MORS SIMVAL ’94 Symposium

neering simulations. The draft report concludes In September 1994 the Military Operations Rethat use of subjective judgement may be necessary - -

search Society held SIMVAL ’94, the fourth in a

for weakly predictive uses, and it notes the impor- series of mini-symposia and workshops designed

tance of describing and quantifying the uncertain- ty address the problem of simulation validation.

ties in the predictions.
Distributed Interactive Simulation of Combat 45

without simulators to provide the basis for comparing SAF and simulator behaviors and simulation outcomes. ModSAF behavior algorithms will be changed as appropriate for subsequent experiments.

BDS-D simulation outcomes will also be compared to CASTFOREM outcomes to determine differencesand reasons for differences. CASTFOREM algorithm changes and runs may be made to try to bring

the outcomes into better agreement with BDS-D simulation outcomes. CASTFOREM runs will be made

for the full scenario with the version of CASTFOREM that produces the best agreement with BDS-D simulation results. Results of the BDS-D simulation and CASTFOREM runs will be analyzed to evaluate antiarmor weapons’ contribution to force effectiveness and to provide the basis for BDS-D simulation VV&A.

ModSAF physical algorithms for vulnerability target acquisition, delivery accuracy, mobility, and

rate of fire were validated as conforming to Army standards, and system performance was consistent

with the M1A2 simulator. Behavioral algorithms were validated as being able to represent armor tactics

and doctrine. ModSAF was benchmarked for Experiment 1 against IOT&E and CASTFOREM.

DIS analytical tools were developed to support simulator and ModSAF VV&A and to evaluate

causes of simulation outcomes. A quick-look analysis capability was developed to permit review of experiment outcomes within 30 minutes of trial completion. Protocol data units were added and post-processors were developed to analyze the causes of simulation outcomes for the spectrum of measures of

effectiveness, performance, and behavior specified in the evaluation plan. Each element of the network

was integrated to insure that it could send and receive protocol data units. The network was also exercised to insure that it had sufficient throughput to render real time images for the scenario.

Detailed evaluation and test plans were prepared and troops were trained before the experiment.

Pilot tests were run to insure that the experiment could be executed and data could be collected and

analyzed using DIS analytical tools. CASTFOREM was run prior to the experiment to fine tune the scenario and benchmark ModSAF.

Forty-eight trials were run over a 12-day period. Twenty-four trials were run for each vignette (12

trials with manned simulators and 12 trials with ModSAF only). The two vignettes run were a hasty attack (14 M1A2 tanks attacking 4 tanks and 3 BMPs1

) and a hasty defense (14 M1A2 tanks defending

against 26 tanks and 1 BMP). Four M1A2 simulators were used in the manned simulator trials. To minimize crew learning effects the scenario and platoon location were randomized.

The first A2ATD Experiment developed and demonstrated the first integrated DIS capability using

2.0.3+ protocols, high-fidelity environment, next-generation image generators, validated simulators and

ModSAF, and DIS analytical tools to support VV&A and analysis of experiment outcomes. As a by-product of the first experiment, the MWTB was modernized and W&A tools and procedures are being offered as DIS VV&A guidelines.

1 A BMP IS a type of armored vehicle originally produced in the Union of Soviet Socialist Republics

SOURCE: W.J. Brooks, “Anti-Armor Advanced Technology Demonstration (A2ATD),” Phalanx, December 1994; available online at

<URL:gopher//triton.dmso.mil.:70/11/twg/mors/phalanx/a2atd.txt>

The objective of SIMAL ’94 was to address the units for V&V” (207). The implication is that the

question: “How much validation is enough?” The MORS group could not find a way to determine

group agreed that “there were no quantifiable
46 | Distributed Interactive Simulation of Combat

how much validation there is, or how much is

enough.47

Afterwards, the SIMVAL Senior Advisory

Group met and decided to use the output of the

workshop to draft a set of VV&A templates. The

draft guidelines will be posted on an electronic

bulletin board system for review, comment, and

revision. When it appears that a consensus has

been reached, the revised guidelines will be published. No additional SIMVAL workshops will be

planned until the M&S community has had a

chance to use the draft guidelines in case studies

(207).

❚ DoD Modeling and Simulation

Master Plan

In January 1995 the Under Secretary of Defense

for Acquisition and Technology began formal

coordination of the Final Draft of the DoD Modeling and Simulation Master Plan (195). A complex

and important document, it lays out a vision,

goals, and responsibilities for VV&A and other

tasks required (153) for management of Defense

models and simulations. Specifically, it requires

a DoD Instruction on VV&A to be prepared—and

one has been drafted (154). It echoes the question,

‘‘How much validation is enough?’’ as an unresolved issue.

❚ Methodology Handbook

for VV&A of DIS

DMSO has sponsored the drafting of a methodology handbook for VV&A of DIS (175). It is aimed

at DIS exercise managers. A companion document, the DIS VV&A Implementation Guide, is

being developed to describe in detail the actions

required to accomplish each step identified in this

VV&A handbook. If done well, the DIS VV&A

Implementation Guide could serve as the much

needed key to quantifying validation so the requirements could specify how much is enough.

❚ OTA Workshop on Validation

OTA hosted a workshop on issues of validating

combat models on February 8, 1995. OTA’s purpose was to hear views and collect information for

possible inclusion in the report of this assessment.

Some of the material presented is summarized in

this background paper—e.g., in the section

“SEES 2.0 Accreditation Exercises” above and in

the following section.

❚ Validation of CCTT for Training

The Project Manager (PM CATT) of the Army’s

Combined Arms Tactical Trainer project (CATT)

has been conducting a vigorous effort to validate

the Close Combat Tactical Trainer (CCTT), which

is the first of several DIS-based training systems

to be developed under the CATT funding and

management umbrella. PM CATT recognizes that

because CCTT is to be used for training, CCTT

validation should seek to assess the degree to

which trainees learn desired conditional behaviors

(i.e., different behaviors in various situations) as

a result of CCTT training. Documenting the behaviors desired in various tactical situations has

been a major effort, involving detailed research

into hundreds of Army training manuals, field

manuals on doctrine, and other documents. However, this process is a necessary step in the measurement of the effectiveness of training and of

training transfer—i.e., the transfer of behaviors

learned in a simulator to a real vehicle. 

OTA notes that validation of CCTT (or any other program) for particular training uses does not

validate it for predictive uses, such as predicting

the force multiplication that some proposed command and control system would provide to a tank

battalion under specified circumstances. Such a

NO TAG OTA partly disagrees: the SEES accreditation study illustrates how specific measures of performance can be defined and how statistical tests can determine the statistical significance at which one cannot distinguish simulation outputs from actual outcomes in history, future

history, or some set of live simulations considered to be a proxy for future history. Of course, details of implementing such a comparison will

vary from case to case, and how little statistical significance to require (‘‘how much validation is enough?’’) remains a subjective matter, generally implying a value judgment.
Distributed Interactive Simulation of Combat 47

Modeling and Simulation

Define Resource Repository \ Components’ review

requirements )

- Review resources of identified models

- Extract model choices

r 1 I

Domain expert reviews

with appropriate TWGs

and FWGs

I I

I I I

Optional consultation

[TWGs, FWG, others)

/ \ /

MSWG Select model suite

Endorsement of simulation

Optional Execution of results DAB simulation plan simulation

consultation by EXCIMS ‘ plan process

feedback

SOURCE: DoD Executive Council for Modeling and Simulation

use would be a misappropriation (4) of CCTT

models and simulations, unless they are validated

for the predictive use.

❚ Cost/Training Effectiveness Analysis

A recent report by DoD’s Office of the Under Secretary of Defense for Personnel and Readiness

noted that an early evaluation of CCTT “Quickstart” modules will be the first major DoD study

to quantify the training benefits of advanced distributed simulation (192). A project to improve

Cost/Training Effectiveness Analysis is now in

progress (in Phase 2) at the University of Central

Florida’s Institute for Simulation and Training; it

is sponsored by the Office of the Assistant Secretary of Defense (Force Management and Personnel).

❚ Simulation Endorsement Process

At the March 22, 1995, meeting of DoD’s Executive Council for Modeling and Simulation (EXCIMS), Captain James Hollenbach, USN,

Director of DMSO, briefed the EXCIMS on a proposed simulation endorsement process that would

involve the EXCIMS in the joint acquisition process as a simulation endorsing agency. Following

discussion, the EXCIMS agreed to test the process

in the context of the Joint Advanced Strike

Technology (JAST) Program, which volunteered

to test the process. Based on the results of this test,

the EXCIMS will decide whether it will serve as

a simulation endorsing agency (180). The process

(see figure 7 (67)) is, in essence, a process that the

EXCIMS could use to assure itself, and to try to

assure the Defense Acquisition Board (DAB), that
48 Distributed Interactive Simulation of Combat

The DIS protocol specifies types and formats of Protocol Data Units (PDU)s, which are packets

(physically speaking) or datagrams (logically speaking) of simulation data transmitted from one simulator to any or all other simulators in a distributed simulation. Various kinds of computers and simulators

can communicate simulation data to one another by using standard PDUs Most PDUs defined by the

current DIS protocol standard are variations of the User Defined Protocol (UDP) datagrams used by

some computers on the Internet.

The DIS protocol standard and a companion document, commonly known as the DIS enumeration documental specify the sequence in which various bits of information are transmitted in each type

of PDU.

Changes in the PDUS

2 are enabling new kinds of information to be passed. The demonstration of

the 2.0 version of the DIS protocol, for example, included electromagnetic transmissions data, which

enables the simulation of radio communication. A protocol that includes what is known strategically and

tactically—intelligence about the enemy, command and control data, what is known about an enemy’s

location—has been proposed. This would allow simulation exercises to be restarted with the same

amount of known information available to both sides.

The table3 on the next page shows some of the types of PDUs for the current standard, DIS 1.0,

and proposed revisions 2.0 and 3.0. The simulation management protocols proposed for DIS 2.0 would

allow new entities to be networked without having to add code for modeling them to each simulator.

They would also contain information that would capture the state of the simulation, allowing it to be recreated or examined at a later time DIS 3.0 PDUs may possibly contain information for reflecting

changes in the terrain, such as when a bomb has created a crater in the road. New PDUs may also

provide for additional capabilities for translating between models and simulators with different levels of

fidelity, representing unaggregated, aggregated, and disaggregated entities.

1 The latest version of what is commonly called “the DIS enumeration document” is: Institute for Simulation and Training, Enumeration and Bit-encoded Values for Use with IEEE 1278.1 Standard for Distributed Interactive Simulation—Application Protocols,

IST-CR-95-05 (Orlando, FL: Institute for Simulation and Training, 1995); available online as <URL:ftp://ftp.sc.ist.ucf.edu/public/STDS/

enum/enum95.doc>

2 U S. Army, Simulation, Training, and Instrumentation Command, “Distributed Interactive Simulation: Operational Concept, ”

draft (Orlando, FL 199), p 15, describes types of PDUs and functions originally used in SIMNET.

3 U.S. Army, Simulation, Training, and Instrumentation Command, “Battlefield Distributed Simulation-Developmental Advanced Technology Demonstration Technology Development Plan, ” draft (Orlando, FL: September 1993), p 4.

due care had been exercised in developing a simu- but is not necessarily, a validation procedure; nor

lation plan for the generation of simulation data to

be furnished to the DAB for use in acquisition decisionmaking. It comes close to being an accreditation procedure, except for the fact that, in

accordance with DoD Directive 5000.59, only the

user of M&S results—in this case, the DAB-can

accredit a model or simulation. The EXCIMS can

endorse, but cannot accredit, use of a model by the

DAB for support of acquisition decisionmaking.

The Simulation Endorsement Process could be,

is it claimed to be a validation procedure.

STANDARDS FOR DIS

The simulators, computers, and network links

used for DIS would not work together unless they

were all designed to have compatible interfaces—

i.e., to comply with interface standards. Several

types of standards are needed for DIS. Industry

standards and military standards for computer
Distributed Interactive Simulation of Combat 49

Additional PDUs Additional PDUs

DIS 1.0 PDUs Proposed for DIS 2.0 Considered for DIS 3.0

Entity state Emission C

3

I

Fire Laser Dynamic terrain

Detonation Transmitter Weather/atmosphere

Service request Signal Fidelity controls

Resupply offer Simulation management Transfer control

Resupply received Create entity Aggregate/deaggregate

Resupply cancel Remove entity Instrumentation

Repair complete Start/Resume

Repair response Stop/Freeze

Collision Acknowledge

Action request

Action response

Data query

Set data

Data

Event

Message

SOURCE: Office of Technology Assessment, 1995.

equipment and software are specified for DIS ex- may or may not be worth making. Standard-setercises. However, the DIS community has had to

invent the standards that are unique to DIS. These

include standards for protocols (procedures and

message formats) for communications among

simulators and standards for exercise management.

To some, the entire issue of standards for DIS

revolves around the creation of standards for the

message packets called protocol data units

PDUs that are transmitted from simulator to

simulator over the network (see box 10). If these

are not well designed, the simulation will not run

properly, or will cease to run at all. Enunciation of

such standards was a primary goal of the SIMNET

program from its very beginning. Refinement and

testing of DIS continues as a community-wide activity. Industry adoption of DIS protocols has led

to the availability of a variety of interoperable

simulators (see box 11).

Standards are not an unqualified good, especially if compliance is mandatory, because they

are the result of compromises among users that

ting usually falls to an organization that can set

and manage standards and facilitate making simulations available to program managers who need

them. While it might be easy to get agreement on

the need for such an organization, what the standards should be and who should manage them is

not easily agreed on. The Army, having taken

leadership in the development of DIS technology

with the SIMNET program, called for a single

manager that “will need to be the ‘policeman’ of

the system with respect to giving the equivalent of

the ‘Good Housekeeping Seal’ to configuration

models of users’ systems that wish to interact

throughout the system, and the keeper of the standards and protocols that form the electronic gateway into the electronic battlefield” (170, p. 21).

Among the other military services, there was concern that the Army might be named the manager

(115). Had this happened, it might have perpetuated a service bias to validation of simulations

and restricted access to the DIS platforms on

which the simulation exercises were performed.
50 Distributed Interactive Simulation of Combat

Industry’s acceptance of the DIS 1.0 protocols was demonstrated dramatically in November 1992

at the lnterservice/industry Training Systems and Education Conference (I/lTSEC) in San Antonio, Texas. According to the DIS Steering Committee, “the l/lTSEC is the training and simulation community’s

major annual gathering. Part of the conference is a large trade show to which all the major modeling

and simulation contractors bring their wares. In the past all the simulators demonstrated did so on a

stand-alone basis. During the 1992 show [more than 30] simulators, computer generated force devices,

and monitoring devices, from [more than 20] organizations, were linked together [by Ethernet] using the

basic DIS PDUs The virtual world consisted of a military base near the Pacific ocean (Fort Hunter-Liggett) and the adjacent waters. The scenario included maritime, air-to-air, air-to-ground, ground-to-air,

and land operations in which all the players took part. ” This was the “first major test” of the DIS standard protocols and communication architecture.1

The simulated platforms and weapon systems included E-2Cs, F/A-18s, AH-64 Apache helicopters, M1 tanks, a Dragon antitank missile, a Patriot antiaircraft missile, the U.S. S. Ticonderoga, the

U.S. S. Perry, the U.S. S. Wasp, and Soviet-made SU-25 attack aircraft.2 Most of the simulators had been

designed for other purposes before SIMNET, and many—including SIMNET simulators that used the

SIMNET protocols—had to be outfitted with separate boxes to translate messages into DIS protocol

data units. Experts had predicted it would take much longer than two years to independently develop

protocols that could be exchanged across the network quickly enough for interaction in real time.3 In

this case the exchange happened in real time even with some of the simulators translating protocols. It

was proof that industry could fully participate in the DoD simulation network, which is a necessary step

towards use of the network for design, virtual prototyping, and evaluation for procurement.

In a separate simulation, live tanks at Fort Hunter-Liggett exercised with simulated vehicles. Observers at the l/lTSEC could watch on a big screen either a view of the terrain televised by a camera on

top of a tank (roughly what the live tank crew saw) or a computer-generated image of the virtual battlefield from the tank’s point of view (roughly what a tank simulator crew would see).

1 DIS Steering Committee, The DIS Vision, A Map to the Future of Distributed Simulation, version 1, IST-SP-94-01 (Orlando, FL

The University of Central Florida, Institute for Simulation and Training, May 1994).

2 L. D. Voss, A Revolution in Simulation: Distributed Interaction in the ‘90s and Beyond (Arlington, VA: Pasha Publications, Inc.,

1993), p 106

3 K Doris, presentation, lntersewlce/industry Training Systems and Education (I/lTSE) Conference, San Antonio, TX, November 1992

SOURCE: Office of Technology Assessment, 1995

Probably these concerns have been alleviated by To standardize the VV&A of simulations,

the recent establishment of the DMSO Architecture Management Group, which will specify a

high-level architecture, including standards, for

DoD distributed simulations. The AMG is expected to specify the use of the DIS protocols developed in the DIS standards workshops at the

University of Central Florida’s Institute for Simulation and Training (see box 12).

benchmarking procedures may have to be defined.

Proposed solutions vary. The National Security

Industrial Association has suggested that “we

must depart from the concept of optimized weapon systems and relative values. . . One way to enforce this change would be to designate a scripted

benchmark which any simulation should be able

to replicate in terms of two forces of particular di-
Distributed Interactive Simulation of Combat 51

Standardization of DIS simulation and communications protocols insures interoperability of all

hardware and software that complies with the standard protocols, DoD decided in 1990 to use industrystandard simulation protocols if suitable ones existed, and the SIMNET protocols were upgraded to a

new standard known as the DIS 1.0 protocols, The new standard was been widely accepted throughout

the simulation industry, as was demonstrated in 1992 when very different simulators from over 30 different companies were interactively networked using the draft DIS 1.0 protocols in real time. A DIS community representative asked the Standards Board of the Institute of Electrical and Electronics Engineers

(IEEE) to adopt the DIS 1.0 protocols as an IEEE standard, and it did so in 1993, The IEEE standard,

called IEEE 1278 -1993,1 was subsequently recognized (i.e., endorsed), by the American National Standards Institute (ANSI). This doubtless averted proliferation of competing standards and concomitant

problems of noninteroperability.

The protocols need to be revised as telecommunications and computing technologies advance.

Some of the protocols used for DIS are widely used and fairly stable. Examples include the basic widearea networking protocols used by the Internet; these are called the TCP/lP protocols, after the acronyms for two of them, the Transmission Control Protocol and the Internet Protocol. However, the application-layer protocols used only for DIS are revised frequently, New experimental versions of the DIS protocols are defined by Government, industry, and academic participants at DIS standards workshops

hosted by the Institute for Simulation and Training (lST) at the University of Central Florida.2 This process allows both military requirements and the profit motive to motivate the development of protocols

that are usable by many types of applications and by systems embodying a variety of technologies,

Experimental versions of the DIS protocols are tested in DoD excercises, and perhaps elsewhere,

If satisfactory, a version may be proposed for adoption by the IEEE Standards Board. If adopted, many

domestic and foreign companies may produce simulation and networking hardware and software that

complies with the standard, increasing the options for nondevelopmental procurement of such products

by DoD. Standards adopted by the IEEE may be recognized by ANSI or adopted by the International

Electrotechnical Committee (lEC) of the International Organization for Standardization (ISO).

Balloting groups of the IST standards working groups have recently approved the proposal of two

draft DIS standards to the IEEE Standards Board. The first, P1278.1, is a standard for protocols; the

second, P1278.2, is a standard for communications. The respective working groups are struggling to

resolve most balloting comments (suggestions and reservations) in time to submit these draft standards

to the IEEE Standards Board in August for consideration at its September 1995 meeting.3 If they are

adopted this year, they will become IEEE standards IEEE 1278.1-1995 and IEEE 1278.2-1995 and wiII

complement IEEE 1278-1993. Balloting for a draft standard for exercise management and feedback,

P1278.3, ended June 30, but tabulation of the results and resolution of comments has not been completed as this background paper goes to press (July 1995).3 If it is adopted in 1996, it will become IEEE

standard IEEE 1278.3-1996.

1 Institute of Electrical and Electronic Engineers, IEEE Standard for Information Technology---Protocols for Distributed Interacrive Simulation Applications, Entity Information and Interaction, IEEE Standard 1278-1993 (Piscataway, NJ IEEE Customer Service

Center, 1993).

2 Details are on the World Wide Web at <URL.ftp://ftp.sc.ist.ucf.edu/public/STDS/workshop/>

3 M. Loper, personal communication, JUly 17, 1995.

SOURCE: Office of Technology Assessment, 1995
52 | Distributed Interactive Simulation of Combat

mensions engaging one another. . . [ARPA’s 73

Easting, a simulation recreating a battle of the Persian Gulf War, is an example of] an excellent first

step in developing dynamic simulation benchmarks’’ (101, p. 39).

Ultimately, the limitation of using modeling

and simulation throughout the development life

cycle of a system is the difficulty in relating program measures of performance to those of system

effectiveness and finally to measures of force effectiveness. Each organization—analytic, combat

development, training and doctrine, test and evaluation—uses its own measurement tools and has

its own perceptions of how to measure the effects.

In order to pass on lessons learned, share models

and modeling tools and resources, and coherently

track systems through their life cycle process,

mutually acceptable tools, common data dictionaries,48 and common measures of effectiveness

must be coordinated (101, p. 40).

Without such benchmarking, different simulations used to predict system effectiveness give

disparate results. One simulation may indicate

that proposed system X would increase the effectiveness of an armored division, in a particular

scenario, by half, while another simulation may

find no significant effect. Such occurrences sometimes lead to allegations of biases in ‘‘legacy’’

(long-used, service-sponsored) simulations.

However, not all differences in preference deserve

the perjorative term ‘‘bias.’’ Differing roles and

missions can lead to some very defensible differences in preference. The test and acquisitions

communities illustrate this point: the approaches

of the two communities are in direct opposition to

each other. The acquisition community uses relative changes in outcome as their primary measure

of effectiveness (MOE). Scenarios are normalized

with optimal operation assumed for all weapons to

better detect inherent differences when comparing

different systems, models, or designs. These simulations are not necessarily intended to reflect

reality. The testing and training communities,

however, seek a realistic representation of the fog

and friction of war (101, p. 3). These differences

stem from the communities’ differing missions:

the acquisition community seeks the best item, or

the best item for the money, while the testing community seeks the Achilles heel of a given item.

SCALABILITY AND SCALING

The draft DoD Modeling and Simulation Master

Plan (195) defines scalability as ‘‘the ability of a

distributed simulation to maintain time [sic] and

spatial consistency as the number of entities and

accompanying interactions increase’’ (cf. (27)) It

is desirable that DoD distributed simulations be

scalable, e.g., in order to provide large forces for

high-echelon commanders to command. To this

end, ARPA at one time wanted 100,000 entities to

be networked in its Synthetic Theater of War program (162,164). It is also desirable that distributed engineering simulations, such as those being

developed for ARPA’s Simulation-Based Design

program, be scalable to encompass, e.g., simulation of all systems on a ship (147).

However, there is a limit to the number of entities that can participate in a distributed simulation

without causing loss of temporal consistency—

i.e., desynchronization.49 The limit is not a

constant of nature, like the speed of light; it may

be raised by improving hardware, software, communications architecture, and exercise management (7, 39, 81,102, 125, 130, 132). However, it is

a real limitation, one that is recognized by the draft

DoD Modeling and Simulation Master Plan.

Limits to scalability were encountered in the

STOW-E exercise, even though STOW-E set a

new record for DIS scale (over 2,000 simulta48 Data dictionaries are documents that define and identify needed data. An entry in a data dictionary specifies what the name of a data set

means, what the data are used for, where the data are stored, how the data are transferred and used, and the form they are put in.

49 There is also a limit to the number of types of entities, although this is a limit imposed more by the affordability of pre-exercise simulation

development than by run-time constraints. See reference (117).
Distributed Interactive Simulation of Combat | 53

neous entities) by demonstrating several effective

communications techniques that had been proposed (27) to widen PDU traffic bottlenecks; these

included bundling,50 compression,51 multicasting,52 and the use of ‘‘delta-PDUs’’ (PDUs that

announce changes rather than status). One analyst

of STOW-E data wrote that ‘‘in some machines,

computer processing was overloaded after entity

counts reached 1,000, with others having a slightly higher threshold’’ (119).

OTA staff observing STOW-E at ARPA’s WISSARD (What If Simulation System for Advanced

Research and Development) facility and other

sites noted that some briefers described the STOW

program as having a goal of netting 50,000 entities in its STOW-97 demonstration (a change indicating greater concern about scalability), while

other briefers still quoted the original goal of netting 100,000 entities. ARPA’s World Wide Web

page describing STOW efforts still says the goal is

netting 100,000 entities (164).

Even if it were technically feasible to have

100,000 tank simulators drive and fight on the

same virtual battlefield, they would not represent

real divisions or corps (or higher echelon organizations) unless each small unit’s ‘‘slice’’ (share)

of division- or corps-level intelligence, communications, logistics, and other support functions is

modeled. That is, balanced representation of different types of entities is necessary for fidelity.

Attempts to increase fidelity by simulating

more details—e.g., of radio signals, or a dismounted soldier’s pose—will also stress scalability. Other things being equal, simulating more

detail will require simulating fewer entities.

FUNDING FOR DIS SYSTEMS

It is difficult to estimate how much DoD spends

on combat modeling and simulation, partly because there are disputes about how much of the

Department’s various activities are modeling and

simulation, and partly because much modeling

and simulation—including combat modeling and

simulation—is performed as an integral part of acquisition, operations, and maintenance and is not

funded as a separate account. This section reports

some estimates of funding for modeling and simulation, especially for DIS.

❚ Research and Development

The Modeling and Simulation Technology Area

Assessment (TAA) prepared by the Office of the

Director of Defense Research and Engineering in

1994 (186) tallied and proposed DoD investment

in development of models and simulations. Development of the 1994 M&S Technology Area Plan

(TAP), conduct of the 1994 M&S Technology

Area Review (TAR), and production of the 1994

M&S TAA were DoD’s first attempt to comprehensively analyze and plan the course of development of M&S technology. In the process, DoD

discovered that the Army and ARPA treated M&S

as a unique technology area; while the Air Force,

the Navy, and the Ballistic Missile Defense Organization (BMDO) considered M&S development

50 Bundling is the concatenation of the information fields of several datagrams addressed to the same destination into one longer information field, which requires only one header and set of flags. This reduces network congestion but requires delaying transmission of the resulting

packet until its information field is filled with the component information fields.

51Compression is the coding of information in a datagram so as to reduce the number of characters required. The basic principle is to use the

shortest code words for the character strings that occur most frequently in the uncompressed data. In DIS exercises such as STOW-E, compression has reduced packet size to 4 or 5 percent of the uncompressed size, in effect increasing the number of simulators that can participate by a

factor of 20 or 25.

52 Multicasting is the addressing of packets to an interest group rather than to a particular computer, or as had been done in SIMNET, to all

computers. Thus a gateway between a WAN (such as the Defense Simulation Internet) and a LAN connecting aircraft simulators can be programmed to ignore packets arriving from the WAN that are addressed to an acoustic emissions interest group; that is, the gateway will not pass

them on to the LAN, which serves no simulators that need to receive them. This reduces congestion of the LAN.
54 Distributed Interactive Simulation of Combat

($ millions)

subarea FY 1994 FY 1995 FY 1996

Architecture & standards $114.8 $88.2 $83.4

Environmental representation 68.4 76,4 64.9

Computer generated forces 44.4 54.4 49.9

Total $227.6 $219.0 $198,2

SOURCE” Department of Defense, Defense Modeling and Simulation Office, 1994

as supporting technology (tools) for other programs. The Joint Directors of Laboratories (JDL)

had not kept track of laboratory spending on modeling and simulation as a separate account. The

TAA concluded that, because of the “confusion”

then existing among the DoD Components over

the distinction between M&S technology development and M&S applications, the first TAA is

probably incomplete and “significant 6.2-6.3

funded M&S technology development work, beyond that reported in the TAR, is being conducted

in support of other technology areas, as well as in

6.4 funded Demonstration and Validation programs” (186). Nevertheless, the TAA concluded

that preparation of the first TAP, TAR, and TAA

was very productive for DoD’s M&S community.

It identified many excellent programs as well as

areas needing increased emphasis. “The TAR, in

particular, offered an excellent forum for crossfertilization among components and among programs within components. The lessons learned

from this initial effort will yield significant improvements in subsequent TAPs, TARs, and

TAAs ” (186). The 1995 TAP is expected to be released shortly after this background paper is completed.

Despite its warning of incompleteness, the

1994 TAA tallied M&S Technology Area funding

of approximately $232 million in FY 1994, $231

million in FY 1995, and $226 million in FY 1996,

before adjustments were made by and overhead

expenses were deducted for the Office of the Secretary of Defense. The TAA identified the funding

of several projects in nine program elements. The

TAA panel binned the funding into three “subareas” of technology flagged as meriting special

emphasis: Architecture and Standards, Environmental Representation, and Computer Generated

Forces. (See table 5.) These subareas are critical to

the development of distributed simulations, including DIS.

The Defense Modeling and Simulation Office’s

own 1995 Modeling and Simulation Investment

Plan projects a doubling of funding, from about $5

million in FY 1995 to about $10 million in FY

1996 (67, slide 38). (See table 6.) All of the activities listed support DIS; one—the DIS enumeration document—is exclusively for DIS.

Another DIS-related project is the Universal

Threat System for Simulators (UTSS), which will

build a data repository for threat data required by

the supported simulators. It will also build a RealTime Simulation Software (RTSS) library that

will host reusable, validated simulation software

and will catalog available threat simulations

(144). The initial data modeling effort, now complete, was gathering detailed data requirements

for each of the four simulators selected for inclusion in the initial development of UTSS: the

Army’s Modular Semiautomated Forces (ModSAF), the Navy’s F-14D Weapon Systems Trainer, the Marine Corps’s UH-1N Weapon Systems

Trainer, and the Air Force’s F-15C and F-16C

UTDs. The project, formerly a program, received

funding from DMSO in FY 1993 and FY 1994. It

is now funded solely by the Navy, as project

W2124 of program element 0204571 N, Consolidated Training Systems Development (122), for

which DoD budgeted $48 million in FY 1996 and

$38 million in FY 1997 (190).
Distributed Interactive Simulation of Combat 55

Estimated

FY 1995 FY 1996

Functional Data Administrator (FDAd)a support

Model and Simulation Resource Repository (MSRR) projectb

Data verification, validation, and certification tasks

Authoritative data sources (ADS), portable data quality tool,c

and C* core data model integration

Modeling and simulation community directories

Data security

DIS enumeration documente

$1,184,5

1,443.8

680.0

300.0

411.0

575.0

533.0

Total $5,127.3

$2,900.0

5,700.0

1,000.0 

—

—

1,000.0

—

$10,600.0

a The Defense Modeling and Simulation Office (DMSO) has been designated the Modeling and Simulation Functional Data Administrator (M&S

FDAd).

b DMSO has established a MSRR on the World Wide Web at Uniform Resource Locator (URL) <URL:http://www.dmso.mil/dmso/msrr.html> as part

of the DoD Data Repository System. It has hypertext Iinks to other MSRR nodes, including the DMSO Interim MSRR hosted on the Joint Data Base

Elements for Modeling and Simulation node at the U.S. Army Electronic Proving Ground at Fort Huachuca, Arizona, at <URL:http://huachucajdbe.army.mil/>.

c The supported activity is making computer software used for data verification at U.S. Central Command portable, i.e. , capable of being used on a

variety of computers

d Modeling and simulation community directories enable M&S developers and users to locate authoritative data sources, databases, models and

simulations,

e The latest version of what IS commonly called the DIS enumeration document Is: Institute for Simulation and Training, Enumeration and Bit-encoded Values for Use with IEEE 1278.1 Standard for Distributed Interactive Simulation—Application Protocols, IST-CR-95-05 (Orlando, FL: lnstitute for Simulation and Training, 1995). Available online at <URL:ftp://ftp.sc.ist.ucf.edu/public/STDS/enum/enum95.doc>, filedate Mar 21, 1995.

SOURCE: Department of Defense, 1995.

, Procurement

DoD has not tallied funding for procurement of

modeling and simulation equipment and services

in general or for DIS in particular. In the past, no

need for such an accounting was recognized, and

doing it would be problematic because of definitional difficulties. An expansive definition of simulation says that “everything is simulation except

combat” (191 ). According to this definition, most

of the defense budget is for simulation, but less expansive defininitions of simulation would count

only a small fraction of the defense budget as

funding for simulation, and only a fraction of that

would be funding for DIS. However, as the Army

proceeds with its program of “digitizing the

battlefield” and as DoD, guided by the MSMP,

proceeds to interface live forces, constructive simulations, and distributed simulations to one another and to the Global Command and Control

System—so that forces can train at home and

when they deploy—it will become more difficult

to identify systems that are not from time to time

involved in DIS.

Thus again at the risk of undercounting, we can

mention some of the largest DIS-related programs

in the DoD Comptroller’s 1995 Procurement Programs (P-1) report. Top ranked (in terms of FY

1997 funding) is SIMNET/Close Combat Tactical

Trainer, for which $88.5 million is requested. The

Close Combat Tactical Trainer (CCTT), a descendant of SIMNET, is the first of several planned

Army DIS programs that are collectively called

the Combined Arms Tactical Trainer (CATT) program. Its aim is to network simulators for a variety

of Army combat platforms for the purpose of collective combined arms tactical training. The second CATT program, which ranks second among

DIS programs in FY 1997 P-1 funding, is the Fire

Support Combined Arms Tactical Trainer

(FSCATT), for which the Army has requested no
56 Distributed lnteractive Simulation of Combat

procurement funds through FY 1996; but the

Army has requested $22 million in FY 1997.53

Next is the Reconfigurable Simulators program

(see figure l-8), for which the Army has requested

about $17 million in FY 1997. Some of the “simulators, all types,” for which the Army requested

about $8 million in FY 1997, might be used for

DIS on occasion.

❚ Operations

Defense authorization and appropriation bills typically provide little specific direction on funding

for modeling and simulation.54

The National Defense Authorization Act for

Fiscal Year 1995 (Public Law 103-337) authorized $89.1 billion, and the Department of Defense Appropriations Act, 1995 (Public Law

103-335) appropriated $80.9 billion, for operations and maintenance, including training. The

fraction spent on DIS activities is unknown but

presumably very small.

DARPA’s SIMNET program ended in 1989,

but the simulators, communications protocols,

and network developed by the program are still

being used and upgraded by ARPA’s Advanced

Distributed Simulation (ADS) program, the

Army’s Battlefield Distributed Simulation-Developmental (BDS-D) program and other Department of Defense (DoD) programs. The name

SIMNET still occurs in the “SIMNET/Close

Combat Tactical Trainer” line item in the DoD

Comptroller’s 1995 Procurement Programs (P-1)

report to Congress (189). The term is still applied

to simulators, networks, and simulations that still

use the SIMNET communications protocols,

which have been superseded by the Distributed Interactive Simulation (DIS) protocols.

A distributed interactive simulation is “distributed” in the sense that simulator computers at

SOURCE: U.S. Army Simulation, Training, and Instrumentation Command.

multiple training sites nationwide are connected

by means of a local-area network (LAN; see figure

2), which in turn maybe connected to a wide-area

network (WAN) such as the the Defense Simulation Internet (DSI) (see figure 3). Trainees can enter disparate tank, helicopter, or other simulators

and all participate in the same simulated combat.

The simulation is “interactive” in the sense that

humans interact with the computers to influence

the simulation. The battle does not follow a set

script, and trainees win or lose on the basis of their

performance. The trainees fight a similarly networked opposing force, which may be a combination of virtual forces-real people operating simulators-an d semiautomated forces

(SAF)---vehicles and weapons simulated by computers with humans providing operational supervision. In both the virtual forces and the

semiautomated forces, human behavior is simulated by humans, albeit humans not affected by the

many stresses of real combat.

5 3FSCATT is one of five acquisition programs that the Federal Acquisition Streamlining Act of 1994(Public Law 103-355) authorized the

Secretary of Defense to designate as a “pilot” program to determine “the potential for increasing the efficiency and effectiveness of the acquisition process using standard commerc ial practices” (148, p. 65). 5 4The National Defense Authorization Act for Fiscal Year 1994 (Public Law 103-160) did authorize appropriation of $75 million for procurement of simulation equipment for reserve components.
 Appendix A

Acknowledgments

OTA gratefully acknowledges the assistance of the following individuals and organizations. The

individuals and organizations listed do not necessarily approve, disapprove, or endorse this background

paper; OTA assumes full responsibility for the background paper and the accuracy of its contents.

William L. Alexander

Science Applications International Corp.

Robert L. Clover

Institute for Defense Analyses

James Dewar

The RAND Corporation

Robert Grow

Hughes Training, Inc.—Link Division

Marvin Hammond

Institute for Defense Analyses

Christopher Lawrence

The Dupuy Institute

Margaret Loper

Georgia Tech Research Institute

Robert J. Mills, Jr.

Simulation Technologies, Inc.

Paul Murtha

AB Technologies

Anil Nerode

Cornell Mathematical Sciences Institute

Donald Ponikvar

Science Applications International Corp.

Renée L. Priore

Institute for Defense Analyses

Steve Seidensticker

Science Applications International Corp.

Dennis P. Shea

Center for Naval Analyses

Roger Smith

Mystech, Inc.

Gershon Weltman

Perceptronics, Inc.

Ken Wimmer

Science Applications International Corp. | 57
58 | Distributed Interactive Simulation of Combat

U.S. DEPARTMENT OF DEFENSE

Defense Modeling and Simulation Office

Captain James W. Hollenbach, USN

Lieutenant Colonel Jerry Wiedewitsch,

USA

Gary Yerace

Office of the Under Secretary of Defense for

Acquisition and Technology

Director, Test Systems Engineering and

Evaluation

Modeling, Simulation, and Software

Evaluation Program

Patricia Sanders

Office of the Under Secretary of Defense for

Personnel and Readiness

Donald Johnson

Department of the Air Force

Office of the Deputy Chief of Staff, Plans and

Operations

Directorate of Air Force Modeling, Simulation,

and Analysis

Colonel Carl M. Upson, USAF

Ted Harshberger

Air Force Studies and Analysis Agency

Clayton Thomas

Department of the Army

Office of the Deputy Under Secretary of the

Army (Operations Research)

Army Model Improvement and Study

Management Agency

Army Model and Simulation Management

Office

Colonel David E. Hardin

Army Material Command

Army Material Systems Analysis Activity

Wilbert Brooks

Systems, Training, and Instrumentation

Command

Colonel James Shiflett, USA

Concepts Analysis Agency

Howard Whitley

Department of the Navy

Office of the Deputy Chief of Naval Operations

(Resources, Warfare Requirements, and

Assessment)

Assessment Division

Modeling and Analysis Section

George Phillips

Naval Research Laboratory

Advanced Information Technology Branch

Lawrence C. Schuette

U.S. Marine Corps

Marine Corps Combat Development Command

Marine Corps Modeling and Simulation

Management Office

Major Richard Powell, USMC

The Joint Staff

Operational Plans and Interoperability

Directorate (J-7)

Joint Warfighting Center

Captain Stanley F. Bloyer, USN

Commander Raoul Reese, USN

Force Structure, Resources, and Assessment

Directorate (J-8)

Robert H. Halayko

U.S. Space Command

Directorate of Analysis

David Finkleman

U.S. DEPARTMENT OF ENERGY

Lawrence Livermore National Laboratories

Conflict Simulation Laboratory

Ralph M. Toms

U.S. ENVIRONMENTAL PROTECTION AGENCY

Mary Pace

U.S. GENERAL ACCOUNTING OFFICE

Office of Policy

Christine Fosset
Appendix B

Abbreviations

21CLW 21st Century Land Warrior

A2ATD Anti-Armor Advanced

Technology Demonstration

AAR after-action review

ACT Advanced Concepts and

Technology

ADCATT Air Defense Combined Arms

Tactical Trainer

ADS Advanced Distributed

Simulation

ADS Authoritative Data Sources

ADST Advanced Distributed

Simulation Technology

AI artificial intelligence

ALSA Air-Land-Sea Agency

ALSP Aggregate Level Simulation

Protocol

AMC Army Material Command

AMG Architecture Management

Group

ANSI American National Standards

Institute

ARPA Advanced Research Projects

Agency (formerly DARPA)

ARTEP Army Test and Evaluation

Program

ASARC Army Systems Acquisition

Requirements Council

ASPT Advanced Simulator for Pilot

Training

ASTO [ARPA] Advanced Systems

Technology Office

ATACC Advanced Tactical Air

Command Center

ATCCS Army’s Tactical Command

and Control System

ATD Advanced Technology

Demonstration

ATD aircrew training device

ATM Aircrew Training Manual

ATM anti-tank missile

ATM asynchronous transfer mode

BAA Broad Agency Announcement

BBN Bolt, Beranek, and Newman

BBS Brigade/Battalion Battle

Simulation

BDS-D Battlefield Distributed

Simulation-Developmental

BEA Bureau of Economic Analysis

BFTT Battle Force Tactical Training

BOC Bureau of the Census

C2 command and control

C2 command and control

C2T2 Commercial Communications

Technology Testbed

| 1
2 | Distributed Interactive Simulation of Combat

C2T2 Commercial Communications

Technology Testbed

C4I Command, Control,

Communications, Computers,

and Intelligence

C4I Command, Control,

Communications, Computers,

and Intelligence

C4IFTW C4I for the Warrior

CAAN Combined Arms Assessment

Network

CASTFOREM Combat Arms Task Force

Engagement Model

CATT Combined Arms Tactical

Trainer

CAX computer-assisted exercise

CBD Commerce Business Daily

CBS Corps Battle Simulation

CCD charge-coupled devices

CCTT Close Combat Tactical

Trainer

CEM Concepts Evaluation Model

CFOR Command Forces

CGF computer-generated forces

CGI computer-generated imagery

CGI Common Gateway Interface

CIG computer image generation

CIG computer image generator

CINC Commander in Chief

CJCS Chairman of the Joint Chiefs

of Staff

CJCSI Chairman of the Joint Chiefs

of Staff Instruction

CL constructive/live

CMTC Combat Maneuver Training

Center

COEA cost and operational

effectiveness analysis

COFT Conduct of Fire Trainer

COMPASS Common Operation Mission

Planning and Support

Strategy

Costar Corrective Optics Space

Telescope Axial Replacement

COTR Contracting Officer Technical

Representative

COTS commercial off-the-shelf

CP command post

CPX command post exercise

CRADA Cooperative Research and

Development Agreement

CRT cathode ray tube

CV constructive/virtual

CVL constructive/virtual/live

DA Department of the Army

DAB Defense Acquisitions Board

DARPA Defense Advanced Research

Projects Agency (now ARPA)

DCSOPS Deputy Chief of Staff of the

Army for Operations

DDR&E Director of Defense Research

and Engineering

DDS Distributed Defense

Simulation

DEEM Dynamic Environmental

Effects Model

DI Demonstration Integration

DIA Defense Intelligence Agency

DII Defense Information

Infrastructure

DIS Distributed Interactive

Simulation

DISA Defense Information Systems

Agency

DISN Defense Information System

Network

DIVE Dismounted Infantry Virtual

Environment

DMA Defense Mapping Agency

DMSO Defense Modeling and

Simulation Office

DoD Department of Defense

DR&E Defense Research and

Engineering

DSCS Defense Satellite

Communication System

DSI Defense Simulation Internet

DSP Defense Support Program

DT&E developmental test and

evaluation

DTIC Defense Technical

Information Center
Appendix B Abbreviations | 3

E2DIS Environmental Effects for

Distributed Interactive

Simulations

EMD engineering, manufacturing,

and development

ENCATT Engineering Combined Arms

Tactical Trainer

ERINT Extended Range Interceptor

EW electronic warfare

EXCIMS Executive Council for

Modeling and Simulation

FAA Federal Aviation

Administration

FFRDC federally funded research and

development center

FLIR forward-looking infrared

FSCATT Combined Arms Tactical

Trainer

FWG functional working group

GCCS Global Command and Control

System

GOSIP Government Open Systems

Interconnection Protocol

GOTS government off-the-shelf

GPS Global Positioning System

HMD head-mounted display

HOTAS hands on throttle and stick

I4 WISSARD Intelligent, Imaginative,

Innovative, Interactive What

If Simulation System for

Advanced Research and

Development

I&P intelligence and planning

IAC Information Analysis Center

IADS Integrated Air Defense

Simulator

IDA Institute for Defense Analyses

IEEE Institute of Electrical and

Electronic Engineers, Inc.

IEI Illusion Engineering, Inc.

IFF identification friend/foe

IFOR Intelligent Forces

I/ITSE Interservice/Industry Training

Systems and Education

iMSRR Interim Modeling and

Simulation Resource

Repository

IP Internet Protocol

IPPD integrated product and

process development

IRIAM Integrated Radar/Infrared

Analysis and Modeling

System

IST Institute for Simulation and

Training

IUSS Integrated Unit Simulation

System

IVIS Inter-Vehicular Information

System

IW information warfare

J-6 [Joint Staff] Command,

Control, Communications,

and Computing Systems

Directorate

J-7 [Joint Staff] Operational Plans

and Interoperability

Directorate

J-8 [Joint Staff] Force Structure,

Resources, and Assessment

Directorate

JAST Joint Advanced Strike

Technology [Program]

JCS Joint Chiefs of Staff

JDC Joint Doctrine Center

JDL Joint Directors of

Laboratories

JECEWSI Joint Electronic Combat

Electronic Warfare Simulation

JICM Joint Integrated Contingency

Model

JM&S Joint Models and Simulations

JMASS Joint Modeling and

Simulation System

JMCIS Joint Maritime Command

Information System

J-METL Joint Mission-Essential Task

List

JPO Joint Program Office
4 | Distributed Interactive Simulation of Combat

JRTC Joint Readiness Training

Center

JSEP Joint Simulation Endorsement

Process

JSIMS Joint Simulation System

JSTARS Joint Surveillance Target

Attack Radar System

JTCTS Joint Tactical Combat

Training System

JTN Joint Targeting Network

JWARS Joint Warfare Simulation

JWC Joint Warfare Center

JWFC Joint Warfighting Center

JWID Joint Warrior Interoperability

Demonstration

LAM Louisiana Maneuvers

LAN local area network

LCLV liquid crystal light valve

LHN long-haul network

LOSAT Line-of-Sight Anti-Tank

M&S modeling and simulation

MARS Multiwarfare Assessment and

Research System

MDT2 Multiservice Distributed

Training Testbed

MEL Master Environmental

Library

METL Mission Essential Task List

MILES Multiple Integrated Laser

Engagement System

MLS multilevel security

ModSAF Modular Semiautomated

Forces

MORS Military Operations Research

Society

MOSAIC Models & Simulations: Army

Integrated Catalog

MSE Mobile Subscriber Equipment

MSIS Modeling and Simulation

Information System

MSMP Modeling and Simulation

Master Plan

MSRR Modeling and Simulation

Resource Repository

M-STOW Maritime Synthetic Theater of

War

MSWG Modeling and Simulation

Working Group

NMS National Military Strategy

NRL National Research Laboratory

NTC National Training Center

NTIS National Technical

Information Service

NTSA National Training Systems

Association

NTSC Naval Training Systems

Center

OASD (FM&P) Office of the Assistant

Secretary of Defense (Force

Management and Personnel)

OPFOR opposing force

OR operational requirements

ORSBM Oak Ridge Spreadsheet Battle

Model

ORTA Office of Research and

Technology Applications

OSD Office of the Secretary of

Defense

OSI Open Systems Interconnect

OSTP Office of Science and

Technology Policy

OT&E operational test and

evaluation

PDU protocol data unit

PE program element

PEO program executive office(r)

PGTS Precision Gunnery Training

System

PM program manager

PM project manager

PM CATT Project Manager for the

Combined Arms Tactical

Trainer

PM DIS Project Manager for

Distributed Interactive

Simulation

PM TRADE Project Manager for Training

Devices

PO project office

Preview Prevent Real Errors by Virtual

Investigation of Engineering

Worlds
Appendix B Abbreviations | 5

R&D research and development

RD&A Research, development, and

Acquisition

RDT&E research, development, test,

and evaluation

REA Remote Entity

Approximation

REFORGER Return of Forces to Germany

RF radio-frequency

RFPI Rapid Force Projection

Initiative

RTSS Real-Time Simulation

Software

S&T science and technology

SAF semiautomated forces

SAFOR semiautomated forces

SAIC Science Applications

International Corp.

SBIR Small Business Innovation

Research program

SDIO Strategic Defense Initiative

Office

SDIS Simulation Data Interchange

Specification

SEES Security Exercise Evaluation

System

SEI Software Engineering

Institute

SEI System Engineering and

Integration

SEM Simulation Engineering and

Modeling

SIF SSDB [Standard Simulator

Data Base] Interchange

Format

SIF/DP SSDB Interchange Format for

Distributed Processing

SIF/HDI SSDB Interchange Format for

High Detail Input/Output

SIH Soldier Information Highway

SIMITAR Simulation In Training for

Advanced Readiness

SIMNET Simulator Network

SINCGARS Single Channel Ground and

Airborne Radio System

SIPE Soldier Integrated Protective

Ensemble

SMART Susceptibility Model

Assessment and Range Test

SOFNET Special Operational Forces

Inter Simulator Network

SSDB Standard Simulator Data Base

STOW Synthetic Theater of War

STRICOM [U.S. Army] Simulation,

Training, and Instrumentation

Command

T&E test and evaluation

TAA Technology Area Assessment

TACS/TADS Tactical Air Control

System/Tactical Air Defense

System

TAP Technology Area Plan

TAR Technology Area Review

TASC The Analytic Sciences

Corporation

TCP Transmission Control

Protocol

TCP/IP Transmission Control

Protocol/Internet Protocol

TGT Tank Gunnery Trainer

THAAD Theater High Altitude Area

Defense

TOW tube-launched, optically

tracked, wire-command link

guided [missile]

TQM total quality management

TRADOC Training and Doctrine

Command

TSI Technology Systems, Inc.

TTGT Tank Team Gunnery Trainer

TTTS Tanker-Transport Training

System

TV television

TWBnet Terrestrial Wideband Network

TWG technology working group

TWSTIAC Tactical Warfare Simulation

and Technology Information

Analysis Center

UCOFT Unit Conduct of Fire Trainer

UJTL Universal Joint Task List
6 | Distributed Interactive Simulation of Combat

URL Uniform Resource Locator

USACOM U.S. Atlantic Command

USA STRICOM U.S. Army Simulation,

Training, and Instrumentation

Command

USD(A&T) Undersecretary of Defense for

Acquisition and Technology

USD(P&R) Undersecretary of Defense for

Personnel and Readiness

UTD Unit(-level) Training Device

UTSS Universal Threat System for

Simulators

VIGS Video Interactive Gunnery

System

V&V verification and validation

VL virtual/live

VV&A verification, validation, and

accreditation

VV&C verification, validation, and

certification

WAN wide area network

WISSARD What If Simulation System

for Advanced Research and

Development

WPC Warrior Preparation Center

WWW World Wide Web
Appendix C

References

and

Bibliography

The acronym URL appearing in some citations below stands for Uniform Resource Locator, a specification—in a standard format (74) used by the World Wide Web (9,124)—for resources available via the

Internet. Unless otherwise noted, a citation containing a URL refers only to an online resource; a paper

version may not be published.

1. Adam, J.A., ‘‘Virtual Reality Is for Real,’’ IEEE Spectrum 30(10):22-29, October 1993.

2. Alluisi, E.A., ‘‘The Development of Technology for Collective Training: SIMNET, A Case

History,’’ Human Factors 33(3):343-362, 1991.

3. Apostolakis, G., ‘‘The Concept of Probability in Safety Assessments of Technological Systems,’’ Science 250:1359-1354, Dec. 7, 1990.

4. Battilega, J.A., and Grange, J.K., ‘‘The Misappropriation of Models,’’ Hughes, Military Modeling, 2d ed., W.P. Hughes, Jr., (ed.) (Alexandria, VA: Military Operations Research Society,

1989).

5. Bell, T.E., ‘‘Engineering Layoffs: Facts and Myths,’’ IEEE Spectrum 31(11):16-25, November

1994.

6. Blackman, J., ‘‘An Extension of the Kolmogorov Distribution,’’ Annals of Mathematical Statistics 27:513-520, 1956.

7. Blais, C., ‘‘Scalability Issues in Enhancement of the MAGTF Tactical Warfare Simulation System,’’ ELECSIM 1995—Electronic Conference on Scalability in Training Simulation, sponsored by the Society for Computer Simulation and the Institute for Operations Research and

Management Science, April 10-June 18, 1995. Available online at <URL:ftp://198.3.157.1/pub/

elecsim95/mtws/>.

8. Boldovici, J., Simulator Motion, technical report number 961 (Alexandria, VA: U.S. Army Research Institute, 1993).

9. Boutell, T., et al., ‘‘World Wide Web Frequently Asked Questions,’’ Jan. 26, 1995,

<URL:http://sunsite.unc.edu/boutell/faq/www_faq.html>.

10. Brooks, F.P., Professor of Computer Science, University of North Carolina, Chapel Hill, NC,

“Statement,” New Developments in Computer Technology: Virtual Reality, hearing before the

Subcommittee on Science, Technology, and Space, Committee on Commerce, Science, and

Transportation, Senate, U.S. Congress, May 8, 1991, Serial No. 102-553 (Washington, DC:

U.S. Government Printing Office, 1992).
66 | Distributed Interactive Simulation of Combat

11. Brooks, W.J., ‘‘Anti Armor Advanced Technology Demonstration (A2ATD),’’ Phalanx

27(4):21-22, December 1994. Available online at <URL:gopher://triton.dmso.mil.:70/11/twg/

mors/phalanx/a2atd.txt>.

12. Carter, C., and O’Connor, M.J., ‘‘Aggregation Information Generated Using DIS Protocols

During the STOW-E and Atlantic Resolve ‘94 Exercises,’’ 12th Workshop on Standards for the

Interoperability of Distributed Simulations, vol. I: Position Papers, IST-CF-95-01.1 (Orlando,

FL: Institute for Simulation and Training, University of Central Florida, February 1995). Available online at <URL:ftp://ftp.sc.ist.ucf.edu/public/STDS/workshop/12th/papers/064.doc>.

13. Cerf, V., President, Internet Society, “Statement,” Internet Security, hearing before the

Committee on Science, Space, and Technology, House of Representatives, U.S. Congress, Mar.

22, 1994 (Washington, DC: U.S. Government Printing Office, 1994).

14. Cheung, S.E., and Loper, M.L., ‘‘Traffic Characterization of Manned-Simulators and Computer

Generated Forces in DIS Exercises,’’ Fifth Annual Conference on Artificial Intelligence, Simulation, and Planning in High Autonomy Systems. Available online at <URL:http:/

/info.computer.org:80/conferen/ais94/cheungs1.ps>.

15. Christenson, W.M., and Zirkle, R.A., 73 Easting Battle Replication—A Janus Combat Simulation, paper no. P-2270 (Alexandria, VA: The Institute for Defense Analyses, September 1992).

16. Clausewitz, K. von, On War, (A. Rapoport, ed.), (J.J. Graham, tr., 1873) (abridged) Pelican Ed.

(New York, NY: Penguin Books, 1968).

17. Corum, J.S., The Roots of Blitzkrieg, (Lawrence, KS: University Press of Kansas, 1992).

18. Crooks, H., interview with L. Voss at Illusion Engineering Inc., Westlake Village, CA, August,

1992.

19. Darling, D.A., ‘‘The Kolmogorov-Smirnov, Cramr-von Mises Tests,’’ Annals of Mathematical

Statistics 28:823-838, 1957.

20. Davis, P.K., and Blumenthal, D., The Base of Sand Problem: A White Paper on the State of

Military Combat Modeling, note N-3148-OSD/DARPA (Santa Monica, CA: RAND Corp.,

1991).

21. Davis, P.K., Generalizing Concepts and Methods of Verification, Validation, and Arredication

(VV&A) for Military Simulations, report R-4249-ACQ (Santa Monica, CA: RAND Corp.,

1992).

22. Davis, P.K., ‘‘A Framework for Verification, Validation, and Accreditation,’’ Simulation Validation Workshop Proceedings (SIMVAL II), A.E. Ritchie (ed.) (Alexandria, VA: Military Operations Research Society, April 1992).

23. Davis, P., ‘‘Aggregation, Disaggregation, and the 3:1 Rule In Classical Ground Combat,’’

ELECSIM 1995—Electronic Conference on Scalability in Training Simulation, sponsored by

the Society for Computer Simulation and the Institute for Operations Research and Management Science, April 10-June 18, 1995. Available online at <URL:ftp://198.3.157.1/pub/elecsim95/agg_3_1/>.

24. Davis, P.K., ‘‘Distributed Interactive Simulation (DIS) in the Evolution of DoD Warfare Modeling and Simulation,’’ Proceedings of the IEEE 83(8), August 1995.

25. DeLoura, M.A. ‘‘A Summary of Virtual Environments Research at UNC-Chapel Hill,’’ CyberEdge Journal, December 1992.

26. Dewar, J.A., et al., Credible Uses of the Distributed Interactive Simulation (DIS) System, draft

DRR-878-1-A (Santa Monica, CA: The RAND Corp., October 1994).

27. DIS Steering Committee, The DIS Vision, A Map to the Future of Distributed Simulation, version 1, IST-SP-94-01 (Orlando, FL: Institute for Simulation and Training, University of Central
Appendix C: References and Bibliography | 67

Florida, May 1994). Available online at <URL:ftp://ftp.sc.ist.ucf.edu/public/STDS/vision.doc>

and <URL:ftp://ftp.sc.ist.ucf.edu/public/STDS/vision.mcw>.

28. Division, Inc., as of Apr. 28, 1995. Available online at <URL:http://pipkin.lut.ac.uk/

WWWdocs/LUTCHI/people/sean/vr-sig/products/div-pp6.html>.

29. Dunnigan, J.F., How to Make War—A Comprehensive Guide to Modern Warfare, Rev. ed.

(New York, NY: William Morrow, 1988).

30. Dupuy, T.N., Numbers, Predictions, and War: The Use of History to Evaluate and Predict the

Outcome of Armed Conflict, Rev. ed. (Fairfax, VA: HERO Books, 1985).

31. Dupuy, T.N., comment at the 73 Easting Conference, Alexandria, VA, August 28, 1991, p.

II-146 of 73 Easting: Lessons Learned from Desert Storm via Advanced Distributed Simulation

Technology, J. Orlansky, and Col. J. Thorpe, USAF (eds.), document no. D-1110 (Alexandria,

VA: Institute for Defense Analyses, April 1992).

32. Dupuy, T.N., comment at the 73 Easting Conference, Alexandria, VA, August 29, 1991, pp.

III-56 and III-57 of 73 Easting: Lessons Learned from Desert Storm via Advanced Distributed

Simulation Technology, J. Orlansky, and Col. J. Thorpe, USAF (eds.), document no. D-1110

(Alexandria, VA: Institute for Defense Analyses, April 1992).

33. Efron, U., interview with L. Voss at Hughes Research Laboratories, Malibu, CA, August 1992.

34. Engel, J.H., ‘‘A Verification of Lanchester’s Law,’’ Operations Research 2:163-171, May

1954.

35. Engelberg, M., ‘‘HST Training System,’’ Aug. 11, 1994. Available online at

<URL:http://www.jsc.nasa.gov/cssb/vr/Hubble/hubble.html>.

36. Evans, D.L., et al., ‘‘Earth from Sky,’’ Scientific American 271(6):70-75, December 1994.

37. Evans and Sutherland, Inc., ‘‘AUSA ‘92,’’ videocassette (Salt Lake City, UT: Evans and Sutherland, Inc., Oct. 8, 1992).

38. Executive Office of the President, Office of Management and the Budget, Standard Industrial

Classification Manual 1987 (Springfield, VA: National Technical Information Service, 1987).

39. Ferrari, A., ‘‘Distributed Interactive Simulation in the Legion Environment,’’ ELECSIM

1995—Electronic Conference on Scalability in Training Simulation, sponsored by the Society

for Computer Simulation and the Institute for Operations Research and Management Science,

April 10-June 18, 1995. Available online at <URL: ftp://198.3.157.1/pub/elecsim95/legion/legion.txt>.

40. Fletcher, J.D., A Review of Study Panel Recommendations for Defense Modeling and Simulation, document no. D-1161 (Alexandria, VA: Institute for Defense Analyses, June 1992). Available online at <URL:gopher://triton.dmso.mil.:70/11/Library/Reports/D1161.Fletcher/>.

41. Friedman, G., and Toms, R., Security Exercise Evaluation System (SEES) V2.0, Accreditation

Project Final Report, February 1993. FOR OFFICIAL USE ONLY.

42. Frost and Sullivan, The Military and Aerospace Trainer and Simulator Market (New York, NY:

Frost and Sullivan, 1980).

43. Fulghum, D.A., ‘‘JPATS Contractor Decision Delayed Six Months,’’ Aviation Week and Space

Technology, Feb. 6, 1995, p. 24.

44. Gabel, C.R., The U.S. Army GHQ Maneuvers of 1941 (Washington, DC: U.S. Government

Printing Office, 1991).

45. Gale Research, Inc., Ward’s Business Directory of U.S. Private and Public Companies: vol. 5,

Ranked by Sales Within 4-Digit SIC (Washington, DC: Gale Research, Inc., 1993).

46. Garrett, R., ‘‘ADS: Looking Toward the Future’’ Phalanx 28(2):9-11, June 1995.
68 | Distributed Interactive Simulation of Combat

47. General Reality Company, ‘‘JOB-OFFERED: Lead Engineer,’’ posting to the Usenet newsgroup sci.virtual-worlds, Apr. 8, 1995. See also General Reality Company, ‘‘ANNOUNCE:

General Reality Company Moves To New Headquarters,’’ posting to the Usenet newsgroup

sci.virtual-worlds, May 1, 1995. Sci.virtual-worlds postings are archived; for information on

retrieving postings, see <URL:ftp://ftp.u.washington.edu/public/virtual-worlds/postings/README>.

48. Gill, T.A., Carnegie-Mellon University, ‘‘National Test Facility Confidence Assessment,’’

briefing slides, May 17, 1995.

49. Gourley, S.R., ‘‘CCTT Creates Virtual Battlefield for U.S. Army,’’ Defense Electronics

27(5):24-26, May 1995.

50. Gray, S., Department of the Army, TRADOC Analysis Command, ‘‘JANUS and CASTFOREM,’’ briefing charts, unpublished, 1992.

51. Hancock, D., ‘‘‘Prototyping’ the Hubble Fix,’’ IEEE Spectrum 30(10):34-39, October 1993.

52. Hartley, D.S., III, Non-Validation of Lanchester’s Original Equations in the Inchon-Seoul

Campaign of the Korean War, ORNL/DSRD/TM-12 (Oak Ridge, TN: Oak Ridge National

Laboratory, 1988).

53. Hartley, D.S., III, Can the Square Law Be Validated?, K/DSRD-57 (Oak Ridge, TN: Martin

Marietta Energy Systems, Inc., Data Systems Research and Development, Oak Ridge K-25

Site, 1989).

54. Hartley, D.S., III, Historical Validation of an Attrition Model, K/DSRD-115 (Oak Ridge, TN:

Martin Marietta Energy Systems, Inc., Data Systems Research and Development, Oak Ridge

K-25 Site, 1989).

55. Hartley, D.S., III, Confirming the Lanchestrian Linear-Logarithmic Model of Attrition,

K/DSRD-263 (Oak Ridge, TN: Martin Marietta Energy Systems, Inc., Data Systems Research

and Development, Oak Ridge K-25 Site, 1990).

56. Hartley, D.S., III, Predicting Combat Effects, K/DSRD-412 (Oak Ridge, TN: Martin Marietta

Energy Systems, Inc., Data Systems Research and Development, Oak Ridge K-25 Site, August

1991).

57. Hartley, D.S., III, ‘‘Using Constructive Simulations and ALSP for Training in UFL 92,’’ Proceedings of Elecsim 1994: Electronic Conference on Constructive Training Simulation, R.D.

Smith (ed.), 1994. Available online at <URL: ftp://ftp.mystech.com/pub/elecsim/

ELECSI94.ZIP> and <URL:ftp://ftp.mystech.com/pub/elecsim/ELECSI94.sea.hqx>.

58. Hartley, D.S., III, and Kruse, K.L., Historical Support for a Mixed Law Lanchestrian Attrition

Model: Helmbold’s Ratio, K/DSRD-113 (Oak Ridge, TN: Martin Marietta Energy Systems,

Inc., Data Systems Research and Development, Oak Ridge K-25 Site, 1989).

59. Hartley, D.S., III, et al., An Independent Verification and Validation of the Future Theater Level Model Conceptual Model, K/DSRD-1637 (Oak Ridge, TN: Martin Marietta Energy Systems,

Inc., Data Systems Research and Development, Oak Ridge K-25 Site, January 1994).

60. Hays, R.T., et al., ‘‘Flight Simulator Training Effectiveness: A Meta-Analysis,’’ Military

Psychology 4(2):63-74, 1992.

61. Healy, M., and Hale, C., ‘‘The STOW-E Exercise: An Implementation of the Aggregate PDU,’’

12th Workshop on Standards for the Interoperability of Distributed Simulations, vol. I: Position

Papers, IST-CF-95-01.1 (Orlando, FL: Institute for Simulation and Training, University of Central Florida, February 1995). Available online at <URL:ftp://ftp.sc.ist.ucf.edu/public/STDS/

workshop/12th/papers/018.doc>.
Appendix C: References and Bibliography | 69

62. Hoeber, F.P., Military Applications of Modelling: Selected Case Studies (Gordon and Breach

Science Publishers, 1981).

63. Hodges, L.F., et al., ‘‘The Phobia Project,’’ as of Aug. 18, 1995. <URL:http:/

/www.cc.gatech.edu/gvu/virtual/Phobia/>.

64. Hodges, J.S., and Dewar, J.A., Is It You or Your Model Talking? A Framework for Model Validation, report R-4114-AF/A/OSD (Santa Monica, CA: The RAND Corp., 1992).

65. Hollenbach, J.W., ‘‘C4I to Simulation Initiative in JWID 95,’’ briefing at the Modeling and

Simulation Working Group meeting, Alexandria, VA, Feb. 8, 1995. Briefing charts are attachment 2 to (66).

66. Hollenbach, J.W., ‘‘Memorandum for DoD Modeling and Simulation Working Group (MSWG)

Members, Subject: Minutes of the February 8, 1995, MSWG Meeting,’’ Alexandria, VA, unpublished, Feb. 27, 1995.

67. Hollenbach, J.W., ‘‘Modeling and Simulation Working Group Meeting, 10 March 1995,’’

Alexandria, VA, briefing slides, unpublished, Mar. 10, 1995.

68. Hollenbach, J.W., ‘‘DoD Modeling and Simulation Master Plan,’’ briefing charts for presentation at the Fourth Annual Industry Briefing on Modeling and Simulation, Radisson Plaza Hotel, Alexandria, VA, May 2-3, 1995, sponsored by the National Security Industry Association

Association, the American Defense Preparedness Association, the Modeling and Simulation

Industry Steering Group, and the Defense Modeling and Simulation Office. Available online at

<URL: http://www.dmso.mil/dmso/docslib/briefs/dmsobriefs/IndDayJWH.ppt>.

69. Houston Associates, Inc., Defense Simulation Internet Program Management Office, DSI

Newsletter 2(3), February 1995, <URL:gopher://triton.dmso.mil.:70/11/twg/dsi/nws-ltrs/

dsi-v2n3/dsi-v2n3.txt>.

70. Image Society, Special Interest Group on Simulator Networking (SIGNET), NewsNet Bulletin,

3rd Quarter 1994, <URL:gopher://gopher.sc.ist.ucf.edu:70/msis/files/signet/bulletin-3>, as of

April 29, 1995.

71. Institute for Simulation and Training, University of Central Florida, Enumeration and Bit-encoded Values for Use with IEEE 1278.1 Standard for Distributed Interactive Simulation—

Application Protocols, IST-CR-95-05 (Orlando, FL: 1995). Available online at

<URL:ftp://ftp.sc.ist.ucf.edu/public/STDS/enum/enum95.doc>, Mar. 21, 1995.

72. Institute for Simulation and Training, ‘‘Acronyms of U.S. Military.’’ Available online at

<URL:gopher://gopher.sc.ist.ucf.edu:70/00/msis/files/acronyms.txt>, as of Aug. 18, 1995.

73. Institute of Electrical and Electronic Engineers, IEEE Standard for Information Technology -

Protocols for Distributed Interactive Simulation Applications, Entity Information and Interaction, IEEE Standard 1278-1993 (Piscataway, NJ: IEEE Customer Service Center, 1993).

74. Internet Engineering Task Force, URI Working Group, ‘‘Uniform Resource Locators (URL),’’

T. Berners-Lee, L. Masinter, M. McCahill (eds.), December 1994; <URL:http://ds.internic.net/

rfc/rfc1738.txt>

75. Ivanetich, R., ‘‘Preliminary Definition, High Level Architecture,’’ briefing slides, Mar. 31,

1995. Available online at <URL:http://www.dmso.mil/dmso/wrkgrps/amg/amgbriefs/hladef/31march.html>.

76. Jacobs, J.F., The SAGE Air Defense Systems: A Personal History (Bedford, MA: MITRE

Corp., 1986).

77. Jacobs, J.W., et al., A Meta-Analysis of the Flight Simulator Training Research, NAVTRASYSCEN Technical Report no. 89-006 (Orlando, FL: Naval Training Systems Center, 1990),

DTIC accession number AD-228 733.
70 | Distributed Interactive Simulation of Combat

78. Jones, A.K., Director, Defense Research and Engineering, ‘‘Grace Hopper Speech,’’ keynote

speech, first Annual Grace Hopper Conference, June 9, 1994. Available online at

<URL:http://www.acq.osd.mil/ddre/docs/Grace_Hopper_Speech.html>.

79. Josephus, F., The Genuine Works of Flavius Josephus, (W. Whiston, tr.), vol. 6 (New York,

NY: David Huntington, 1815), pp. 139-140.

80. Kameny, I., ‘‘List of Acronyms (June 1993),’’ June 1993 [filedate Sep. 1, 1994?], <URL:

gopher://triton.dmso.mil.:70/00/twg/dmso-twg/twg-idb/acronyms.txt>.

81. Kerchner, R.M., ‘‘Scaling Problems Associated With Rule-Based Decision Algorithms In Multiple-Objective SituationsValue-Driven Methods As An Alternative,’’ ELECSIM 1995—

Electronic Conference on Scalability in Training Simulation, sponsored by the Society for

Computer Simulation and the Institute for Operations Research and Management Science,

April 10-June 18, 1995. Available online at <URL:ftp://198.3.157.1/pub/elecsim95/rulebase/>.

82. Lamson, R.J., ‘‘Virtual Therapy of Anxiety Disorders,’’ CyberEdge Journal, Issue no. 20, vol.

4, no. 2, March/April 1994, p. 1ff.

83. Leontief, W., ‘‘Academic Economics,’’ letter to the editor, Science, vol. 217, no. 4555, July 9,

1982, pp. 104 and 107.

84. Leontief, W., ‘‘Academic Economics Continued,’’ letter to the editor, Science, vol. 219, no.

4587, Feb. 25, 1983, pp. 904 and 907.

85. Liebholz, S.W., ‘‘Twenty Questions,’’ Military Modeling, 2d Ed., W.P. Hughes, Jr., (ed.) (Alexandria, VA: Military Operations Research Society, 1989).

86. Loftin, R.B., et al.,‘‘Virtual Environments in Training: NASA’s Hubble Space Telescope Mission,’’ July 27, 1994, paper submitted to 16th Interservice/Industry Training Systems and

Education Conference, Orlando, FL, Nov. 28-Dec. 1, 1994. Available online at

<URL:http://www.jsc.nasa.gov/cssb/vr/Hubble/shortpaper/shortpaper.html>.

87. Lottor, M.K., ‘‘Internet Growth (1981-1991),’’ Request for Comments paper RFC 1296,

January 1992. Available online at <URL:gopher://akasha.tic.com:70/11/survey>.

88. Lukes, G.E., ‘‘Methodology: Terrain Data Base Development,’’ 73 Easting: Lessons Learned

from Desert Storm via Advanced Distributed Simulation Technology, J. Orlansky, and J.

Thorpe, (eds.), IDA Document D-1110 (Alexandria, VA: Institute for Defense Analyses, April

1992).

89. Lukes, G., and McKeown, D., ‘‘Building a Cartographic Infrastructure: Simulation and Cartography,’’ paper presented to the Defense Science Board Task Force for Simulation, Readiness,

and Prototyping, Institute for Defense Analyses, Alexandria, VA, July 30, 1992

90. Marion, P., ‘‘STOW-E—First Impressions Report,’’ filedate Nov. 28, 1994. Available online at

<URL:gopher://triton.dmso.mil.:70/00/twg/dsi/dsi-docs/stow-e.txt>.

91. Marshall, S.L.A., Men Against Fire—The Problem of Battle Command in Future War (Washington, DC: The Infantry Journal, and New York, NY: William Morrow and Company, 1947).

92. Marti, J., Kantar, P., Sollfrey, W., and Brendley, K., SEMINT: Seamless Model Integration,

MR-403-OSD/A (Santa Monica, CA: The RAND Corp., 1994).

93. McCloskey, D.N., ‘‘The Insignificance of Statistical Significance,’’ Scientific American, April

1995, pp. 32-33.

94. McCue, B., U-boats in the Bay of Biscay: An Essay in Operations Analysis (Washington, DC:

National Defense University Press, 1990).

95. McDonough, J., interview with L. Voss at Illusion Engineering Inc., Westlake Village, CA, August 1992.
Appendix C: References and Bibliography | 71

96. McKeown, D., ‘‘Toward Automatic Cartographic Feature Extraction,’’ Digital Mapping Laboratory, (NATO ASI Series, vol. F65, Mapping and Spatial Modelling for Navigation, edited by

L. F. Pau, Springer-Verlag Berlin Heidelberg 1990).

97. Miller, D., BBN Labs, comments at Mini-Symposium on Distributed Interactive Simulation,

sponsored by the Military Operations Research Society and the Institute for Defense Analyses,

Radisson Plaza Hotel, Alexandria, VA, Sept. 28, 1992.

98. Miller, D., personal communication, May 3, 1995.

99. Miller, D., and Thorpe, J., ‘‘The Advent of Simulator Networking,’’ Proceedings of the IEEE,

83(8), August 1995.

100. Miller, E.S., War Plan Orange: The U.S. Strategy to Defeat Japan, 1897-1945 (Annapolis,

MD: Naval Institute Press, 1991).

101. National Security Industrial Association, Command Control, Communications and Intelligence

Committee, C3 Simulation Subcommittee, ‘‘A Survey of the Use of Large Scale Simulation to

Support the Acquisition of Tactical C3 Systems,’’ sponsored by the Principal Deputy Assistant

Secretary of Defense for Strategic and Tactical C3, December 1991.

102. Natrajan, A., and Nguyen-Tuong, A., ‘‘To Disaggregate or Not To Disaggregate, That is Not

The Question,’’ ELECSIM 1995—Electronic Conference on Scalability in Training Simulation,

sponsored by the Society for Computer Simulation and the Institute for Operations Research

and Management Science, April 10-June 18, 1995. Available online at

<URL:ftp://198.3.157.1/pub/elecsim95/disagg/>.

103. Naval Research Logistics 42(1) (New York, NY: John Wiley & Sons, in cooperation with the

Office of Naval Research, February 1995).

104. Neuberger, T.K., ‘‘Applying DIS Technology to Operational Training—A Perspective from

Navy STOW-E,’’ 12th Workshop on Standards for the Interoperability of Distributed Simulations, vol. I: Position Papers, IST-CF-95-01.1 (Orlando, FL: Institute for Simulation and Training, University of Central Florida, February 1995). Available online at

<URL:ftp://ftp.sc.ist.ucf.edu/public/STDS/workshop/12th/papers/040.doc>.

105. O’Connell, R.L., Sacred Vessels (Clarendon, England: Oxford University Press, 1991).

106. Orlansky, J., and String, J., Cost-Effectiveness of Flight Simulators for Military Training, Vol.

1: Use and Effectiveness of Flight Simulators, IDA Paper P-1275 (Alexandria, VA: Institute for

Defense Analyses, April 1977).

107. Orlansky, J., and Thorpe, J., (eds.), 73 Easting: Lessons Learned from Desert Storm via Advanced Distributed Simulation Technology, IDA Document D-1110 (Alexandria, VA: Institute

for Defense Analyses, April 1992).

108. Osipov, M., ‘‘The Influence of the Numerical Strength of Engaged Forces on Their Casualties,’’ originally published in the Tzarist Russian journal Voenniy Sbornik (Military Collection),

1915, no. 6, pp. 59-74; no. 7, pp. 25-36; no. 8, pp. 31-40; no. 9, pp. 25-37; no. 10, pp. 93-96.

Translation by R.L. Helmbold and A.S. Rehm, CAA-RP-91-2 (Bethesda, MD: U.S. Army Concepts Analysis Agency, September 1991). DTIC AD-A241 534.

109. Parish, R.M., Department of the Army, TRADOC Analysis Command, ‘‘Review of Janus

Army Results - Battle of 73 Easting,’’ memorandum for record, July 29, 1992.

110. Perla, P.P., The Art Of Wargaming : A Guide for Professionals and Hobbyists (Annapolis, MD:

Naval Institute Press, 1990).

111. Phalanx 28(2):1-36, June 1995.

112. Ponikvar, D., ‘‘Minutes of AMG Meeting, 31 March 1995,’’ memorandum, Apr. 10, 1995.

Available online at <URL:http://www.dmso.mil/dmso/wrkgrps/amg/amgmins/amg3_31.html>.
72 | Distributed Interactive Simulation of Combat

113. Pope, A.R., ‘‘The SIMNET Network and Protocols,’’ report no. 7627 (Cambridge, MA: Bolt,

Beranek, and Newman, Inc., 1991), p. 26.

114. Reich, A.J., ‘‘An Efficient Representation of Spatial Data for Terrain Reasoning by Computer

Generated Forces,’’ ELECSIM 1995—Electronic Conference on Scalability in Training Simulation, sponsored by the Society for Computer Simulation and the Institute for Operations Research and Management Science, April 10-June 18, 1995. Available online at

<URL:ftp://198.3.157.1/pub/elecsim95/spatial/>.

115. Reis, V., Director, Defense Research and Engineering, memorandum of agreement, August

1992.

116. Repass, T., ‘‘DIS: An Evolving Modus Operandi for the Department of the Navy,’’ Phalanx

28(2):31-34, June 1995.

117. Reynolds, P.F., Jr., ‘‘DISorientation,’’ Proceedings of Elecsim 1994: Electronic Conference on

Constructive Training Simulation, R.D. Smith (ed.) 1994. Available online at

<URL:ftp://ftp.mystech.com/pub/elecsim/ELECSI94.ZIP>, May 27, 1994, or at

<URL:ftp://ftp.mystech.com/pub/elecsim/ELECSI94.sea.hqx>, June 8, 1994.

118. Ritchie, A.E. (ed.), Simulation Validation Workshop Proceedings (SIMVAL II), (Alexandria,

VA: Military Operations Research Society, April 1992).

119. Rogers, D., ‘‘STOW-E Lessons Learned, Focused on the 3 Primary Army STOW-E Sites,’’

12th Workshop on Standards for the Interoperability of Distributed Simulations, vol. I: Position

Papers, IST-CF-95-01.1 (Orlando, FL: Institute for Simulation and Training, University of Central Florida, February 1995). Available online at <URL:ftp://ftp.sc.ist.ucf.edu/public/STDS/

workshop/12th/papers/026.doc>.

120. Rogers, A., Joint Staff, J-7, presentation, National Training Systems Association conference,

Alexandria, VA, March 25, 1993.

121. Rothman, P., ‘‘Army to Develop ‘Holodeck-Like’ Virtual Reality System,’’ posting to sci.virtual-worlds Usenet newsgroup, Feb. 17, 1995.

122. Sarkovitz, M., personal communication, May 5, 1995.

123. Scales, R.S., Jr., Certain Victory: The US Army in the Gulf War (Washington, DC: Office of the

Chief of Staff, United States Army, 1993).

124. Schatz, B.R., and Hardin, J.B., ‘‘NCSA Mosaic and the World Wide Web: Global Hypermedia

Protocols for the Internet,’’ Science 265:895-901, Aug. 12, 1994.

125. Seidel, D.W., et al., ‘‘CFOR Approach to Simulation Scalability,’’ ELECSIM 1995—Electronic

Conference on Scalability in Training Simulation, sponsored by the Society for Computer Simulation and the Institute for Operations Research and Management Science, April 10-June 18,

1995. Available online at <URL:ftp://198.3.157.1/pub/elecsim95/cfor/>.

126. Shiflett, J., and Lunceford, D., ‘‘DoD Applications of DIS,’’ Proceedings of the IEEE, vol. 83,

no. 8, to be published August 1995.

127. Sikora, J., and Coose, P., ‘‘What in the World is ADS?’’ Phalanx, vol. 28, no. 2, June 1995,

pp. 1 and 6-8.

128. Sikora, J.J., and Williams, M.L., ‘‘SIMVAL 94,’’ Phalanx (journal of the Military Operations

Research Society), December 1994. Available online at <URL:gopher:/

/triton.dmso.mil.:70/11/twg/mors/phalanx/simval94.txt>.

129. Silicon Graphics, Inc., ‘‘Onyx Product Guide—Technical Specifications,’’ as of Apr. 28, 1995,

<URL:http://www.sgi.com/Products/9Onyx-Product.html>.

130. Srinivasan, S., de Supinski, B., ‘‘Multicasting in DIS: A Unified Solution,’’ ELECSIM

1995—Electronic Conference on Scalability in Training Simulation, sponsored by the Society
Appendix C: References and Bibliography | 73

for Computer Simulation and the Institute for Operations Research and Management Science,

April 10-June 18, 1995. Available online at <URL:ftp://198.3.157.1/pub/elecsim95/multicas/>.

131. Starr, S.H., Director of Plans, MITRE Corp., ‘‘An Assessment of Key Technologies that Support Modeling & Simulation,’’ presented to the Defense Science Board Task Force for Simulation, Readiness, and Prototyping, Institute for Defense Analyses, Alexandria, VA, July 30,

1992, p. 2.

132. Steinman, J.S., ‘‘Scalable Parallel and Distributed Simulations Using the SPEEDES Framework,’’ ELECSIM 1995—Electronic Conference on Scalability in Training Simulation, sponsored by the Society for Computer Simulation and the Institute for Operations Research and

Management Science, April 10-June 18, 1995. Available online at <URL:ftp://198.3.157.1/pub/

elecsim95/speedes/>.

133. Sterling, B., ‘‘War Is Virtual Hell,’’ Wired, January 1993.

134. Stevens, J.E., ‘‘Flights Into Virtual Reality Treating Real-World Disorders,’’ Washington Post,

Mar. 27, 1995, p. A3.

135. Stockfisch, J.A., Models, Data, and War: A Critique of the Study of Conventional Forces,

RAND Report R-1526-PR (Santa Monica, CA: The RAND Corporation, 1975).

136. Strachan, I.W. (ed.), Jane’s Military Training and Simulation Systems, 7th Ed. 1994-1995

(Alexandria, VA: Jane’s Information Group, Inc., 1994).

137. Strachan, I., ‘‘Eyes on the Future of Simulation,’’ Jane’s Defence Weekly , Jan. 21, 1995,

pp. 24-25.

138. Sullivan, G.R., Chief of Staff, United States Army, memorandum for T.R. Franks, Director,

Louisiana Maneuvers Task Force, on ‘‘Letter of Instruction for Louisiana Maneuvers (LAM),’’

May 22, 1992. Available online at <URL: http://140.139.18.189/lamhome/charter.html>.

139. Tarr, R.W., and Jacobs, J.W., ‘‘Distributed Interactive Simulation (DIS): What Is It and Where

Are We Going?’’ presented at the 1994 Summer Computer Simulation Conference, La Jolla,

CA, July 18-20, 1994, sponsored by the Society for Computer Simulation. Available online at

<URL:http://www.sc.ist.ucf.edu/ms_web/papers/dis_paper/dis.html>.

140. Thomas., C., ‘‘Verification Revisited—1983,’’ Hughes, Military Modeling, 2d ed., W.P.

Hughes, Jr., (ed.) (Alexandria, VA: Military Operations Research Society, 1989).

141. Toms, R.M., ‘‘Results of a Verification, Validation, and Accreditation Excercise,’’ briefing,

presented at the Office of Technology Assessment Workshop on Issues in Combat Model Validation, Feb. 8, 1995.

142. Tseng, D., interview with L. Voss at Hughes Research Laboratories, Malibu, CA, August 1992.

143. Tucker, W., et al., ‘‘A Glossary of Modeling and Simulation Terms for Distributed Interactive

Simulation,’’ draft, Jan. 13, 1995, <URL:ftp://ftp.sc.ist.ucf.edu/public/STDS/general/

glossary.txt>.

144. Universal Threat System for Simulators Development Group, ‘‘UTSS Design and Development,’’ filedate Apr. 25, 1995, <URL:gopher://triton.dmso.mil.:70/00/twg/utss/devel/

DEVGRP.TXT>.

145. U.S. Congress, Senate, Committee on Armed Services, Department of Defense Authorization

for Appropriations for Fiscal Year 1993 and the Future Years Defense Program, hearings, Serial No. 102-883, part 1, May 21, 1992, pp. 673-730.

146. U.S. Congress, Office of Technology Assessment, Data Format Standards for Civilian Remote

Sensing Satellites, OTA-BP-ISC-114 (Washington, DC: U.S. Government Printing Office,

1993).
74 | Distributed Interactive Simulation of Combat

147. U.S. Congress, Office of Technology Assessment, Virtual Reality and Technologies for Combat Simulation—Background Paper, OTA-BP-ISS-136 (Washington, DC: U.S. Government

Printing Office, September 1994). Available online at <URL:ftp://otabbs.ota.gov/pub

/vr.military/>.

148. U.S. Congress, Office of Technology Assessment, Assessing the Potential for Civil-Military

Integration: Technologies, Processes, and Practices, OTA-ISS-611 (Washington, DC: U.S.

Government Printing Office, September 1994).

149. U.S. Department of Commerce, Bureau of the Census, Communication Equipment; and Other

Electronic Systems and Equipment, 1993, Current Industrial Report MA36P. Available online at

<URL:gopher://una.hh.lib.umich.edu:70/00/ebb/industry/communications/ma36p93.cen>.

150. U.S. Department of Commerce, Industrial Outlook 1994 (Pittsburgh, PA: Superintendent of

Documents, P.O. Box 371954, 1994), chapter 27. Available online at 

<URL: ftp://141.211.190.102/ebb/industry/outlook/indout27.txt>.

151. U.S. Department of Defense, Common Warfighting Symbology Version 1, MIL-STD-2525,

Sept. 30, 1994. Available online at <URL:http://www.itsi.disa.mil:

5580/A1E4711T1907/MIL2525.ZIP>.

152. U.S. Department of Defense, ‘‘Defense Acquisition Management Policies and Procedures,’’

Department of Defense Instruction, Number 5000.2, February 23, 1991.

153. U.S. Department of Defense, ‘‘DoD Modeling and Simulation (M&S) Management,’’ Department of Defense Directive, Number 5000.59, January 4, 1994.

154. U.S. Department of Defense, ‘‘DoD Modeling and Simulation (M&S) Verification, Validation,

and Accreditation (VV&A),’’ Department of Defense Instruction, Number 5000.XX, draft, June

26, 1995. Available online at <URL:http://www.dmso.mil/dmso/docslib/mspolicy/vva/>.

155. U.S. Department of Defense, ‘‘DoD Modeling and Simulation (M&S) Management,’’ Department of Defense Directive, Number 5000.59, Jan. 4, 1994, ASCII text file, <URL:gopher:/

/triton.dmso.mil:70/00/Library/Policy/dd500059.txt>.

156. U.S. Department of Defense, Acquisition Task Force on Modeling and Simulation, Cost and

Operational Effectiveness Analysis Focus Group, Final Results—Cost and Operational Effectiveness Analysis Focus Group, Jan. 31 to Feb. 2, 1994. Available online at <URL:gopher:/

/triton.dmso.mil.:70/11/twg/atf/coea/finalbfg>.

157. U.S. Department of Defense, Advanced Research Projects Agency, ‘‘Simulation-Based Design

Enabling Technologies,’’ solicitation, Broad Agency Announcement BAA 93-45, Commerce

Business Daily, Issue PSA-0942, September 30, 1993.

158. U.S. Department of Defense, Advanced Research Projects Agency, ‘‘SBD Prototype Development,’’ solicitation, Broad Agency Announcement BAA 94-39, Commerce Business Daily,

Issue PSA-1107, June 1, 1994.

159. U.S. Department of Defense, Advanced Research Projects Agency, ‘‘BAA Proposer Information Packet,’’ [for BAA94-39], ARPA listserver, list ‘‘baa94-39’’, file ‘‘pip’’, as of Aug. 16,

1994.

160. U.S. Department of Defense, Advanced Research Projects Agency, ‘‘Frequently Asked Questions,’’ [about BAA94-39], ARPA listserver, list ‘‘baa94-39’’, file ‘‘faq,’’ as of Aug. 16, 1994.

161. U.S. Department of Defense, Advanced Research Projects Agency, ‘‘TRANSTECH,’’ Special

Notice 95-06, Commerce Business Daily, Issue PSA-1315, Mar. 31, 1995.

162. U.S. Department of Defense, Advanced Research Projects Agency, Advanced Systems

Technology Office, ‘‘Advanced Distributed Simulation,’’ as of Apr. 17, 1995,

<URL:http://www.arpa.mil/asto/ads.html>.
Appendix C: References and Bibliography | 75

163. U.S. Department of Defense, Advanced Research Projects Agency, Advanced Systems

Technology Office, ‘‘Commercial Communications Technology Testbed (C2T2),’’ as of May 3,

1995, <URL:http://www.arpa.mil/asto/c2t2.html>.

164. U.S. Department of Defense, Advanced Research Projects Agency, Advanced Systems

Technology Office, ‘‘Synthetic Theater Of War,’’ as of May 12, 1995, <URL:http:/

/www.arpa.mil/asto/stow.html>.

165. U.S. Department of Defense, Defense Modeling and Simulation Office, ‘‘Architecture Management Group (AMG),’’ Mar. 27, 1995, <URL:http://www.dmso.mil/dmso/wrkgrps/amg/

amg.html>.

166. U.S. Department of Defense, Defense Modeling and Simulation Office, ‘‘AMG Membership,’’

Mar. 27, 1995, <URL:http://www.dmso.mil/dmso/wrkgrps/amg/amgmem.html>.

167. U.S. Department of Defense, Defense Modeling and Simulation Office, ‘‘Preliminary Definition of the DoD High Level Architecture for Modeling and Simulation,’’ briefing slides, May

31, 1995. Available online at <URL:http://www.dmso.mil/dmso/wrkgrps/amg/amgbriefs

/hladef/31may.html>.

168. U.S. Department of Defense, Department of the Air Force, Office of the Deputy Chief of Staff

for Plans and Operations, Directorate of Modeling, Simulation, and Analysis, ‘‘Distributed

Mission Training,’’ Issues in Air Force Simulation and Analysis, August 1994, pp. 5-6.

169. U.S. Department of Defense, Department of the Army, Army Research Laboratory, ‘‘United

States Army Research Laboratory, Federated Laboratory Concept Broad Agency Announcement’’ ARL-95 Fed Lab BAA, <URL:http://w3.arl.mil/fedlab/baa/fedlab1.html> or

<URL:ftp://w3.arl.mil/pub/fedlab/baa/fed_baa.txt>, Dec. 21, 1994.

170. U.S. Department of Defense, Department of the Army, Assistant Secretary of the Army for Research, Development, and Acquisition, Army Science Board, 1991 Summer Study Final Report,

Army Simulation Strategy (Washington, DC, December 1991).

171. U.S. Department of Defense, Department of the Army, Communications and Electronics Command, ‘‘Documentation for the WN Server,’’ (‘‘Please be patient while the Soldier Information

Highway is being paved.’’), as of May 3, 1995, <URL:http://sih.readadp.com//>.

172. U.S. Department of Defense, Department of the Army, Headquarters, Operations, FM 100-5,

1986.

173. U.S. Department of Defense, Department of the Army, Headquarters, Operations, FM 100-5,

June 1993.

174. U.S. Department of Defense, Department of the Army, Headquarters, Verification, Validation,

and Accreditation of Army Models and Simulations, Department of the Army Pamphlet 5-11

(DA PAM 5-11), 15 October 1993.

175. U.S. Department of Defense, Department of the Army, TRADOC Analysis Center; U.S. Department of Defense, Department of the Army, Space and Strategic Defense Command; Department of Defense, Department of the Navy, Naval Air Warfare Center, Training Systems

Division; U.S. Department of Defense, Department of the Army, Army Materiel Systems Analysis Activity; U.S. Department of Defense, Space Command, Center for Aerospace Analysis;

U.S. Department of Defense, Department of the Air Force, Wright Laboratory, Theater Air

Command and Control Simulation Facility; Quality Research; Colsa Corporation; and Johns

Hopkins University, Applied Physics Laboratory; A Methodology Handbook for Verification,

Validation, and Accreditation (VV&A) of Distributed Interactive Simulations (DIS), draft, January 1995. Available online at <URL:http://160.149.162.24/demo/demotree/dmsoproj/

method.doc>, file date Mar. 24, 1995.
76 | Distributed Interactive Simulation of Combat

176. U.S. Department of Defense, Department of the Navy, U.S. Marine Corps, Headquarters, Network Support Section, ‘‘Directory of jwid/ACRONYM/,’’ Apr. 4, 1995.

<URL:http://www.hqmc.usmc.mil/jwid/ACRONYM/>.

177. U.S. Department of Defense, Director, Defense Research and Engineering, Defense Science

and Technology Strategy, September 1994. Available online at <URL:http://www.acq.osd.mil/

ddre/docs/Strategy.txt>.

178. U.S. Department of Defense, Director, Defense Research and Engineering, Defense Technology

Plan, September 1994. Available online at <URL:http://www.acq.osd.mil/ddre/docs/

Techplan.txt>.

179. U.S. Department of Defense, Director, Defense Research and Engineering, Modeling and Simulation (M&S) Technology Area Plan (Executive Summary), September 1994; part of (178).

180. U.S. Department of Defense, Director, Defense Research and Engineering, ‘‘Minutes for the

March 22, 1995, meeting of the DoD Executive Council for Modeling and Simulation

(EXCIMS),’’ Apr. 17, 1995.

181. U.S. Department of Defense, Director, Defense Research and Engineering, ‘‘Strategies, Plans,

and Partners,’’ briefing charts for keynote speech at the Industry Days symposium, sponsored

by the Defense Modeling and Simulation Office, Alexandria, VA, May 2, 1995. Available online at <URL: http://www.acq.osd.mil/ddre/docs/KEYNOTE.ppt>.

182. U.S. Department of Defense, The Joint Staff, Deputy Director for Technical Operations, Force

Structure, Resources, and Assessment Directorate (J-8), Joint Modeling and Simulation Evolutionary Overview, February 1994.

183. U.S. Department of Defense, The Joint Staff, Vice Director, Verification, Validation, and Accreditation of Joint Models and Simulations, Joint Staff Instruction 8104.01, Jan. 12, 1995.

184. U.S. Department of Defense, Office of the Assistant to the Secretary of Defense/Public Affairs

Office (OSD/PA), ‘‘Deputy Secretary Announces Science and Technology Strategy,’’ press release, Available online at <URL: ftp://ftp.dtic.dla.mil/pub/deflink/bt569-94.txt>.

185. U.S. Department of Defense, Office of the Chairman of the Joint Chiefs of Staff, Department

of Defense Dictionary of Military and Associated Terms, Joint Pub 1-02 (Washington, DC: U.S.

Government Printing Office, 23 March 1994).

186. U.S. Department of Defense, Office of the Director, Defense Research and Engineering, Modeling and Simulation Technology Area Assessment, unpublished, undated (apparently late 1994).

187. U.S. Department of Defense, Office of the Secretary of Defense, Comptroller/Chief Financial

Officer, Procurement Programs, P-1, February 1994.

188. U.S. Department of Defense, Office of the Secretary of Defense, Comptroller/Chief Financial

Officer, Research, Development, Test, and Evaluation Programs, report R-1, February 1994.

189. U.S. Department of Defense, Office of the Secretary of Defense, Comptroller/Chief Financial

Officer, Procurement Programs, report P-1, February 1995.

190. U.S. Department of Defense, Office of the Secretary of Defense, Comptroller/Chief Financial

Officer, Research, Development, Test, and Evaluation Programs, report R-1, February 1995.

191. U.S. Department of Defense, Office of the Under Secretary of Defense for Acquisition, Defense Science Board, Report of the Defense Science Board Task Force on Simulation, Readiness, and Prototyping: Impact of Advanced Distributed Simulation on Readiness, Training, and

Prototyping, January 1993.

192. U.S. Department of Defense, Office of the Under Secretary of Defense for Personnel and

Readiness, Use of Simulation in Training—Report to the House Committee on National Securi-
Appendix C: References and Bibliography | 77

ty and Senate Armed Services Committee, April 1995. Available online at

<URL:http://www.sc.ist.ucf.edu/ms_web/rfi/M%26S%20Benefits/simindod.txt>.

193. U.S. Department of Defense, Office of the Under Secretary of Defense for Research and Engineering, Defense Modeling and Simulation Office, briefing to OTA staff, Alexandria, VA, May

25, 1994.

194. U.S. Department of Defense, Office of the Under Secretary of Defense for Research and Engineering, Defense Modeling and Simulation Office, meeting with OTA staff, Alexandria, VA,

July 18, 1994.

195. U.S. Department of Defense, Under Secretary of Defense for Acquisition and Technology, DoD

Modeling and Simulation Master Plan, DoD 5000.59-Paa, draft, January, 1995. Available online at <URL:gopher://triton.dmso.mil.:70/11/twg/dmso/mstrplan>.

196. U.S. General Accounting Office, Models, Data, and War : A Critique of the Foundation for

Defense Analyses, PAD-80-21 (Gaithersburg, MD: U.S. General Accounting Office, March 12,

1980).

197. U.S. General Accounting Office, DoD Simulations: Improved Assessment Procedures Would

Increase the Credibility of Results, PEMD-88-3 (Washington, DC: U.S. General Accounting

Office, December 1987).

198. U.S. National Aeronautics and Space Administration, Spaceborne Synthetic Aperture Radar :

Current Status and Future Directions, Evans, D.L. (ed.), technical memorandum 4679, April

1995. Available online at <URL:http://southport.jpl.nasa.gov/nrc/index.html>.

199. Vector Research, Inc., ‘‘Louisiana Manuevers Task Force Home Page,’’ Feb. 23, 1995,

<URL:http://140.139.18.189/lamhome.html>.

200. Vector Research, Inc., ‘‘LAM TF Glossary,’’ as of Apr. 4, 1995.

<URL:http://140.139.18.189/lamhome/toolbook/tabd/glossary.html>.

201. Voss, L.D., A Revolution in Simulation: Distributed Interaction in the ’90s and Beyond

(Arlington, VA: Pasha Publications, Inc., 1993).

202. Walker, P., ‘‘An Outline of the History of Game Theory,’’ Feb. 15, 1995, <URL:http:/

/www.honors.drexel.edu/top/class/history.html>.

203. Washer, B., et al., Janus Gaming Division, U.S. Army TRADOC Analysis Command, White

Sands Missile Range, ‘‘The Battle of 73 Easting,’’ briefing slides, unpublished, undated, received by OTA May 1995.

204. Whitley, H., U.S. Army Concepts Analysis Agency, personal communication, July 21, 1995.

205. Wiedewitsch, J.L., Deputy Director, U.S. Department of Defense, Defense Modeling and Simulation Office, Alexandria, VA, personal communication, Apr. 29, 1994.

206. Wiedewitsch, J.L., Deputy Director, U.S. Department of Defense, Defense Modeling and Simulation Office, Alexandria, VA, personal communication, Apr. 29, 1994, and May 25, 1994.

207. Williams, M.L., and Sikora, J.J., ‘‘Overview,’’ Simulation Validation Workshop Proceedings

(SIMVAL II), A.E. Ritchie (ed.) (Alexandria, VA: Military Operations Research Society, April

1992).

208. Zorpette, G., ‘‘Breaking the Enemy’s Code,’’ IEEE Spectrum, September 1987, pp. 47-51.