ADVANCED DISTRIBUTED SIMULATION
AND THE FOG-OF-SIMULATION:
LESSONS LEARNED FROM THE COCKPIT

Herbert H. Bell, Ph.D.
Robert J. Clasen, Capt, USAF

Abstract

The Armstrong Laboratory's Aircrew Training
Research Division (AL/HRA) has participated in a
number of Advanced Distributed Simulation (ADS)
exercises. These exercises included Warbreaker, the
Synthetic Theater of War-Europe (STOW-E), Multi-
Service Distributed Training Testbed (MDT2), and
Warfighter 95. Each of these exercises successfully
linked large numbers of simulators and enhanced the
technical capabilities of ADS. These exercises also
suggested that ADS offers unique training
opportunities for warriors, battle staffs, and
commanders, Unfortunately, these training
opportunities were not demonstrated to the same
degree as the technology. We believe part of the
reason for the failure to fully demonstrate the training
potential of ADS stems from problems in adequately
defining requirements, appropriately _ scoping
exercises, thoroughly testing simulations, and
maintaining real-time exercise control. This paper
describes the typical problems we encountered as a
node providing virtual simulations of tactical aircraft
in support of larger ADS exercises while at the same
time trying to provide combat mission training for
the pilots flying those simulators. In addition, this
paper provides recommendations that we believe will
enhance the training value of ADS exercises for
participants in simulated weapons systems.

Introduction

Advanced Distributed Simulation (ADS) is viewed as
a technology that will enable us to design better

* This paper is declared a work of the U.S.
Government and is not subject to copyright
protection in the United States.

systems, develop better tactics, and train better
warfighters. Since its beginnings in the Defense
Advanced Research Projects Agency's (DARPA's)
Simulator Networking project, ADS has shown
tremendous technological advances. An Institute of
Electrical and Electronics Engineers (IEEE) standard
exists for Distributed Interactive Simulation (DIS)'
and extensive simulation networks capable of
creating complex synthetic battlefields have been
demonstrated. ADS is clearly the dominant theme in
today's simulation and training communities.

We have participated in several ADS exercises
including Warbreaker, Synthetic Theater of War-
Europe (STOW-E), Multi-Service Distributed
Training Testbed (MDT2), and Warfighter 95. One
of our roles in each of these exercises was to provide
high-fidelity tactical aircraft simulators flown by Air
Force pilots. These exercises allowed us to
Participate in distributed simulation environments
with large numbers of other participants and
simulations. Each of these exercises was a
technological success. Yet each exercise was also an
expensive and often frustrating experience for us and
our pilots. The primary reasons for this were that the
techniques and procedures necessary to effectively
and efficiently design, implement, and conduct ADS
exercises were, and are still, very immature.

The purpose of this paper is to describe the recurring
problems we saw in these ADS exercises as a small
virtual simulation node. In addition, we will offer
some recommendations that we believe will reduce
the effects of some of these problems and enhance
the training value of ADS exercises.



which resources are ineffectively used and trainees
become the fodder for technology demonstrations.

References

1, Institute of Electrical and Electronics Engineers,
IEEE Standard 1278.1, Standard for Distributed
Interactive Simulation -- Application Protocols,
1995. Workshop on Standards for the
Interoperability of Distributed Simulations, March
11-15, 1996, pp. 327-335.

2. Dahmann, Ponikvar, and Lutz, "HLA Federation
Development and Execution Process," /4th

3. McAuliffe, Long, Liu, Hays, and Nocera, "DIS
Test Suite," /4th Workshop on Standards for the
Interoperability of Distributed Simulations, March
1996, pp. 561-566.

4. Institute for Simulation and Training, Standard
for Distributed Interactive Simulation--Application
Protocols, Version 2.0, Fourth Draft (Revised),
March 1994, p. 25


used in exercises such as STOW-E, many virtual
simulators cannot realistically process or display such
large numbers of entities. Consequently, even
though large numbers of entities are broadcast across
the network, many virtual simulation nodes can
realistically process and display only a tiny subset of
those entities. Because these problems can have a
negative impact on training, the size of the exercise
should be limited to "just enough" entities to make
the scenario realistic enough to meet the training
objectives

In addition, large ADS exercises require extensive
commitments of manpower and facilities to support
both exercise development and execution. Most
ADS exercises rely on existing legacy systems to
provide the weapon system simulators. Since these
legacy simulators are already fielded to support other
requirements, ADS exercises must compete for
personnel and time. This means either extending the
period of simulator operation and increasing the
number of personnel or reducing support to other
programs.

Th h Testi
Another obstacle to successfully accomplish training
in distributed simulation is insufficient testing.
Today, most distributed exercises use DIS protocols
to allow communication among disparate simulation
devices. It is up to each site to implement these
protocols correctly; our experience has shown that
this is not always done. For example, we recently
participated in an ADS exercise in which several sites
had major DIS discrepancies. These discrepancies
included incorrect versions of the DIS protocols,
incorrect entity identifications, incorrect dead
reckoning, incorrect timestamping, and errors with
PDU fields.

Noncompliance by one site impacts all other sites.
At the very least, noncompliance wastes computer
resources because each site must process bad data,
and, at the worst, noncompliant data causes
simulation devices and simulation nodes to "crash."
The only way to avoid these problems is to have
thorough DIS-compliance testing. During this

testing, engineers using the correct DIS tools must
analyze the data transmitted from each simulator to
determine DIS compliance. Discrepancies must be
reported to both the individual sites and to the
exercise director. Once the various systems are
compliant, the system configuration should be frozen
until the completion of the exercise. Unfortunately,

this rarely occurs. Typically, due to schedule
pressures or last-minute additions of some new
capability, testing is incomplete, and a mammoth
engineering effort is needed just prior to the start of
the exercise to pull everything together.

The first step in the testing process should be for the
test director to clearly specify all the DIS Protocol
Data Units (PDUs) necessary to exchange
information between the sites. The specification
must list the PDUs for each site and thoroughly
describe any experimental PDUs that will be used
during the exercise. These should be clearly
documented and distributed to all sites to avoid any
possible miscommunication. Armstrong Laboratory
(AL/HRA) recently participated in an exercise where
a requirement for an Identification Friend or Foe
(IFF) PDU existed, but because no one created a list
of which PDUs each site needed to support, we were
unaware of the requirement. Late in testing, we
noticed our F-16 simulators were being shot down by
friendly Patriot missiles because they were expecting
the F-16s to broadcast certain IFF codes. This
problem occurred too late for us to implement the
IFF PDUs, and during the exercise the F-16 pilots
had to alter flight paths to avoid the Patriots. This
“sim-ism" would have been avoided if the DIS PDU
support requirements had been explicitly defined.

Once the required PDUs have been defined, each
participating site should pass an individual DIS
compliance test, like the DIS Test Suite? Once the
individual compliance tests are complete, then sites
can join the rest of the network on a one-by-one basis
to begin interoperability testing with the other sites.

The time spent upfront thoroughly testing individual
sites more than pays for itself in the long run. It is
much easier to troubleshoot a problem when only one
site is involved as opposed to having multiple sites.
Also, unexpected PDU data can affect different sites
in different ways, making it even more difficult to
find the cause of the problem. Individual compliance
testing is also a more efficient use of resources. If
multiple sites are interconnected and a problem
occurs, then personnel at each site must spend time
troubleshooting a problem that could have been
discovered locally. In addition, simulation resources
are wasted when bad data is passed over the network.
At a minimum, network interface units must weed
out the bad data before passing it along to the local
simulation devices. At the worst, non-DIS-compliant
data can cause simulations to crash, thus eliminating



The biggest problem we have encountered in
distributed simulation exercises is the absence of
well-defined objectives for the human participants.
To date, the majority of ADS exercises have focused
on creating technologically complex synthetic
environments. As a result of this technology
emphasis, exercise objectives have typically focused
on correcting the technical shortfalls identified in
previous exercises and adding new capabilities.
Therefore, the objectives tend to emphasize the
number of participants, the addition of new
simulators, the simulation of new phenomena, and
associated improvements in computer and
communication technology. Such technological
advances are certainly necessary if ADS is to reach
its full potential. Unfortunately, in many recent ADS
exercises, it often seems as if the human participants
in the simulators are merely aids needed to
demonstrate the operation of the technology.

We believe the first step in improving the value of
ADS exercises is to identify the training audience and
the associated training objectives. Exercise planners,
training professionals, and technologists must work
together to define the training or mission objectives
for the individual warfighters and scope the exercise
to meet these objectives. Failure to develop such
detailed objectives means it is difficult to define the
federation of simulators necessary for the exercise.
In addition, it is difficult to determine how those
simulators should interact with one another and to
specify when those interactions should occur.
Hopefully, some of these deficiencies will be
eliminated as future ADS exercises adopt the Defense
Modeling and Simulation Office's (DMSO's) High
Level Architecture (HLA). HLA requires formal
definitions of all simulation interactions as part of the
federation development process.

Once the training objectives and simulator federation
have been clearly identified, the exercise planners,
trainers, and technical staffs can begin the iterative
process of designing an ADS-based exercise. During
this phase, the design team must ensure that the
simulation scenario is consistent with the training
objectives. Once the draft scenario exists, the design
team must explicitly review individual simulator
capabilities. During this review, weapon system
operators (e.g., pilots) must describe the stimuli and
responses necessary for them to perform their

assigned mission within the draft scenario. The
technical experts familiar with the simulations can
then describe how these events will appear in the
virtual world. Our experience has repeatedly shown
that neither DIS protocols nor individual simulators
are capable of faithfully reproducing all aspects of
the battlefield. In addition, different simulators may
reproduce the same aspect of the battlefield at
different levels of fidelity. As a result of these
discrepancies, changes to individual simulators, DIS
protocols, exercise scenarios, or training objectives
are typically required. These changes are necessary
to eliminate technically impossible events and to
identify workarounds to enable other events to
appropriately occur. Usually, this review and
modification process requires several iterations.

As part of defining the scenario and simulator
requirements, the design team must also specify the
communications requirements for the exercise.
Typically, communication requirements involve both
simulation data and voice communications.
Simulation data includes the exchange of platform
and weapons state information among the
participating entities. Voice communication usually
includes tactical, exercise control, and.
technical/simulation troubleshooting _ nets. In
addition, both voice and data communication may be
required to support mission planning and after action
reviews.

iate Scoping of Exerci

A second problem with distributed simulation
exercises, which is related to poor requirements
definition, is improperly scoping the size of the
exercise. Conventional wisdom seems to be the more
entities you can put on the network, the more realistic
the exercise. However, "more" is not necessarily
"better" when it comes to creating synthetic
environments. Large ADS exercises demand large
resource commitments during both the development
and conduct of the exercise. A large number of
entities necessitates an extensive network bandwidth,
which can be very costly. Also, the computers used
to process the network traffic can become bogged
down trying to meet real-time processing demands.
In exercises like STOW-E, we have seen network
interface units and plan-view displays become so
overloaded during peak periods of traffic that they
are barely useful. In addition, although the
constructive simulations may be able to
computationally handle the large number of entities



Exercise directors should use a telephone conference
call among all participating sites to convey
information during execution of the exercise.
Currently, the reliability of telephone conference
calls is much higher than the reliability of sending
digital voice data over the network, and the
importance of exercise control necessitates using the
most reliable method of communication available.
AL/HRA recently participated in an exercise that
relied solely on digital voice communication for all
voice traffic, and when we lost communication due to
compatibility problems with our digital voice unit,
we had no way to let the exercise director know that
our pilots would not be able to talk with the weapons
controller. Flying the mission was a waste of the
pilots' and controllers’ time, yet we couldn't convey
that information to the exercise director.

Even when a reliable communication link is used,
sites can still miss important information. It is
difficult for each site to have someone constantly
monitor the exercise control network--sometimes
personnel must walk away to reset a computer or
possibly answer another phone. During these gaps, if
the exercise director announces, for instance, that the
network will be down for an hour, not all sites may
get the information. These sites may have pilots in
the cockpit waiting for an expected mission to begin.
To avoid this type of situation where trainees are
potentially wasting their time, exercise directors
should use a roll call when announcing information
that is of importance to all sites. By going through
the list of participants one by one, the exercise
director can ensure that everyone is listening and gets
the word.

The exercise director should also use the control
network to closely monitor execution of the scenario.
During the confusion of large-scale exercises, it is
easy to forget about individual sites, but in order to
maximize the training potential, the exercise director
must know whether the scenario is unfolding
according to plan. For example, AL/HRA
participated in an exercise where our F-16 pilots were
to fly as part of a strike package. Enroute to the
target, we realized we had no escort and that none
was forthcoming. We later learned that the
originating site of our escorts had network problems
and was not able to carry out its mission. If the
exercise director had been aware of this and had let
us know, perhaps we could have delayed the mission
or at least exercised command and control procedures
to inform the flight of the status of the escorts. This
would have offered the opportunity for some useful

training. As it was, everyone associated with this
mission missed a valuable training opportunity and
the F-16 pilots flew a meaningless mission.

Recommendations

In order to maximize the training benefit of advanced
distributed simulation exercises, we recommend
program managers follow these guidelines:

1. First and most important, develop a list of
comprehensive training objectives.

2. From the training objectives, derive the scenario,
communication plan (data and voice), and list of
potential participating sites.

3. Define exercise management requirements, to
include security, exercise control, mission
initialization, and after action reviews.

4. Review site capabilities and ensure that assigned
missions are within the capabilities of that site's
simulators.

5. Finalize scenario, simulation data requirements,
communication requirements, and the exercise
schedule.

6. Develop a comprehensive testing plan.

7. Conduct rigorous testing, and then establish a
baseline for the system.

8. Pretrain and rehearse participants.
9. Conduct the exercise.

10. Archive exercise baseline, exercise data, and
document lessons learned.

nclusion
Based on our observations, we believe that ADS has
the potential to significantly enhance combat mission
training. However, in order to fully realize this
potential, exercise planners and trainers must work
together to define the training requirements. Once
the training audience and training objectives have
been defined, the appropriate simulation technologies
and training strategies should be identified. Failure
to adequately define training requirements, identify
simulation resources and training strategies, and test
system components leads to the fog-of-simulation in


AL/HR-TP-1996-0038

UNITED STATES AIR FORCE
ARMSTRONG LABORATORY

Advanced Distributed Simulation
and the Fog of Simulation:
Lessons Learned from the Cockpit

Herbert H. Bell

Robert J. Clasen, Captain, USAF

March 1997

Human Resources Directorate
Aircrew Training Research Division
Approved for public release; distribution is unlimited. 6001 South Power Road, Bidg 558
Mesa Arizona 85206-0904



any training opportunity for the operator of that
simulation device.

There are other DIS issues besides being able to
support specific PDU types. One of these is update
rates. The DIS standard states the default amount of
time between transmission of a simulation entity's
position is five seconds, unless the dead reckoning
tolerance has been exceeded.‘ Often, sites violate
this update rate by sending updates at different rates,
based on local requirements. Some sites update at a
slower rate in order to reduce the amount of traffic on
their local area network, but this can cause problems
at other sites that are expecting the standard rate. For
instance, if the update rates are too slow, other sites
will assume the entity no longer exists and will
remove it from the list of active entities and all visual
displays. Then, when the update does arrive, the site
must insert it into the active list again, and the entity
will suddenly reappear on the visual displays. This
causes entities to pop in and out of the visual
displays, resulting in an unrealistic training
environment.

Another example of a DJS-related problem that
should be checked during testing is the
implementation of the Start and Stop PDUs. Many
sites implement these PDUs so that all entities on the
network are frozen or unfrozen. This is fine on a
local area network, but when connected to a wide
area network, each site's Start/Stop PDUs should only
affect that site's entities. During one exercise, our F-
16 cockpits kept unexpectedly coming off freeze and
crashing, and we had to keep reinitializing them.
This was due to another site which sent out
improperly addressed Start PDUs. Problems like this
should be caught during testing to avoid wasting
valuable training time during the exercise.

Thorough testing is also required to identify and
correct fidelity mismatches and incorrect entity
identifications. For example, during testing for
MDT2, we found that one site was scoring 500- and
2,000-pound bombs as mortar shells. We also
discovered that the location of a major road differed
by as much as 300 meters between different
simulator databases. Without extensive testing,
neither of these problems would have been caught
before training began.

It is also important to test using the expected amount
of network traffic as early as possible.
Unfortunately, testing is rarely possible using all the
simulation systems in a realistic manner. The

primary reason for this is that the warriors who will
use these simulators are typically not available until
the exercise actually starts.

Such testing, however, is critical for ADS exercises
with large numbers of entities. Individual sites may
have problems processing the large numbers of
entities on the network, and, if the expected network
load is not present early enough during the testing
period, there may not be enough time to correct the
problems. This happened to us during the STOW-E
exercise. During testing, we typically only saw 100-
200 entities on the network, and we never saw more
than 800 entities at one time during the test period.
Then, during the exercise, there were 1,700 entities
present, and several of our systems choked under the
increased processing load. Our plan-view display
could not update fast enough, and eventually it
crashed, leaving us "blind" as far as seeing the
scenario. The network interface units for the F-16
simulators also had a difficult time processing the
increased amount of data. One result of these
processing problems was that the entities in the pilot's
out-the-window display appeared to jump instead of
smoothly move within the virtual scene. We may
have been able to fix these problems prior to the
exercise if we had been able to detect them during
testing.

Once DIS compatibility and interoperability
requirements are satisfied, the system configuration
should be baselined, and no software changes should
be made until after the exercise is completed. New
problems can arise or old problems that were once
fixed can suddenly reappear after a programmer
makes one seemingly innocent little change to the
software.

Exercise Control

A lack of appropriate control over unfolding events
during the execution of the exercise is another reason
why distributed simulation has failed to achieve its
full training potential. The distributed nature of ADS
exercises allows the creation of complex, synthetic
environments that offer more realism than previously
available; however, having participants scattered
geographically also makes it very difficult to
orchestrate events in a way to maximize the training
benefits.

Of course, it is impossible to have any control over
an exercise if there is no means for the exercise
director to communicate with the participating sites.


PREFACE

This paper describes the participation of Armstrong Laboratory, Aircrew Training Research
Division (AL/HRA) in a number of Advanced Distributed Simulation (ADS) exercises which
successfully linked large numbers of simulators. The purpose of the paper is to describe some of
the recurring problems that were seen in the ADS exercises. Recommendations are provided that
will enhance the training value of ADS exercises for participants in simulated weapons systems.

This work is being reported under Work Unit 1123-B2-15, Synthetic Environments for Air
Warrior Training Research. Dr Herbert H. Bell was the principal investigator. This paper
documents a presentation made to the American Institute of Aeronautics and Astronautics
(AIAA) Flight Simulation Conference which was held 29-31 July 1996 in San Diego CA. This
presentation was also published in the conference proceedings, AIAA-96-3539-CP, pp 573-578.

iii



Form Approved

REPORT DOCUMENTATION PAGE OMB No. 0704-0188

Public report jection of information is estimated t + hour per response, including the time for reviewing instructions, searching existing data sources, gathering
Coe eo Or ac and completing and rovewing ie colecion of infomation, Send comments regarang this burden estimate or any olner aspect ofthis colecion of
information, suggestions for this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite
"1204, Aringlon, VA 22208-4302, and to the Office of Management and Budget. Paperwork Reduction Project (0704-0188), Washington, DC 20503.

1. AGENCY USE ONLY (Leave blank) | 2. REPORT DATE 3. REPORT TYPE AND DATES COVERED
March 1997 Final Report - December 1995 to July 1996

4, TITLE AND SUBTITLE 5. FUNDING NUMBERS.

Advanced Distributed Simulation and the Fog-of-Simulation: Lessons Learned from

the Cockpit rE : G2302F
6. AUTHOR(S) TA - B2
WU - 15
Herbert H. Bell
Robert J. Clasen
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. RERFORMING ORGANIZATION
Armstrong Laboratory

Human Resources Directorate
Aircrew Training Research Division
6001 South Power Road, Building 558

Mesa, AZ 85206-0904 AL/HR-TP- 1996-0038

10. SPONSORING/MONITORING

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) AGENCY REPORT NUMBER

Armstrong Laboratory

Human Resources Directorate

7909 Lindbergh Drive

Brooks Air Force Base, Texas 78235-5352

11. SUPPLEMENTARY NOTES

Armstrong Laboratory Technical Monitor: Dr Herbert H. Bell, (602) 988-6561.
This paper was presented at the American Institute of Aeronautics and Astronautics (AIAA) Flight Simulation Technologies
Conference held 29-31 Jul 96 in San Diego CA and is included in the proceedings (AIAA-96-3539-CP, pp 573-578).

12a. DISTRIBUTION/AVAILABILITY STATEMENT 12b. DISTRIBUTION CODE

Approved for public release; distribution is unlimited.

13. ABSTRACT (Maximum 200 words) 7

The Armstrong Laboratory's Aircrew Training Research Division (AL/HRA) has participated in a number of Advanced
Distributed Simulation (ADS) exercises. These exercises included Warbreaker, the Synthetic Theater of War-Europe
(STOW-E), Multi-Service Distributed Training Testbed (MDT2), and Warfighter 95. Each of these exercises successfully
linked large numbers of simulators and enhanced the technical capabilities of ADS. These exercises also suggested that ADS
offers unique training opportunities for warriors, battle staffs, and commanders. Unfortunately, these training opportunities
were not demonstrated to the same degree as the technology. We believe part of the reason for the failure to fully demonstrate
the training potential of ADS stems from problems in adequately defining requirements, appropriately scoping exercises,
thoroughly testing simulations, and maintaining real-time exercise control. This paper describes the typical problems we
encountered as a node providing virtual simulations of tactical aircraft in support of larger ADS exercises while at the same
time trying to provide combat mission training for the pilots flying those simulators. In addition, this paper provides
recommendations that we believe wil! enhance the training value of ADS exercises for participants in simulated weapons
systems.

14, SUBJECT TERMS

os + + + . + . 15. NUMBER OF PAGES
ADS; Advanced Distributed Simulation; Exercises; Flight simulators; Simulators; STOW-E;

13

Synthetic Theater of War-Europe; Training; Virtual simulations; Warfighter 16. PRICE CODE

17. SECURITY CLASSIFICATION
OF REPORT

Unclassified

18. SECURITY CLASSIFICATION
OF THIS PAGE

Unclassified

19. SECURITY CLASSIFICATION
OF ABSTRACT

Unclassified

20. LIMITATION ABSTRACT

UL

NSN 7540-01-280-5500

Standard Form 296 (Rev 2-69)
Prescribed by ANS! Std Z-39-18



NOTICES

When Government drawings, specifications, or other data are used for
any purpose other than in connection with a definitely Government-related
procurement, the United States Government incurs no responsibility or any
obligation whatsoever. The fact that the Government may have formulated or
in any way supplied the said drawings, specifications, or other data, is not to
be regarded by implication, or otherwise in any manner construed, as licensing
the holder, or any other person or corporation; or as conveying any rights or
permission to manufacture, use, or sell any patented invention that may in any
way be related thereto.

The Office of Public Affairs has reviewed this report, and it is releasable
to the National Technical Information Service, where it will be available to the
general public, including foreign nationals.

This report has been reviewed and is approved for publication.

HERBERT H. BELL ELIZABETH L. MARTIN
Project Scientist Technical Director

LYNN A. CARROLL, Colonel, USAF
Chief, Aircrew Training Research Division

Please notify AU/HRPP, 7909 Lindbergh Drive, Brooks AFB, TX 78235-5352, if your
address changes, or if you no longer want to receive our technical reports. You may
write or call the STINFO Office at DSN 240-3877 or commercial (210) 536-3877.