select and activate these functions, along with the trigger for selected functions. These are
summarized in the following sections.

“Hand-thrown” Munitions

The ability to “throw” munitions is simulated by calling up a fluctuating graphic on the
screen that indicates the strength of throw desired. The direction of throw is determined by the
weapon aimpoint. The user activates the “throw” with the weapon, and the munition is launched
in the desired direction and distance. Munitions able to be launched in this manner include:

© Grenades — fragmenting and flash/bang
e Smoke - User selectable colors
e Flares — White
The user also has the ability to hand emplace C4 explosive charges to objects in the virtual
- environment that can be subsequently detonated.

“Surrogate” Weapon-Launched Munitions

Flares of various colors can be launched into the sky using the weapon as a surrogate
launching device by selecting and activating this function.

Environmental Effects

SVS allows a number of environmental effects to be set at runtime. These include:
Time of day (24 hours —day — night)

Weather

Wind strength and direction

Fog — intensity and color

These can be preset or varied during scenario execution.

Figure 6 below is an engagement detection algorithm flowchart

Training scenarios can be constructed by a number of means: force-on-force
engagements with networked human-in-the-loop SVS simulators; separate computer-generated-
forces (CGF) applications that operate in the SVS networked environment (e.g., US Army’s
ModSAF or OneSAF), or via an SVS scenario authoring capability. Separate SVS products
support scenario development (Authoring) and control (Battlemaster).

SVS Authoring

AIS Authoring station provides full scenario development capability for the SVS virtual
simulation system. Built upon the SVS Stealth simulation visualization system, the Authoring
station uses Stealth 3D visualization capabilities to assist in drag-and-drop scenario creation
through the Scenario Development Tool (SDT). SVS-internal computer-generated forces (CGF)
can populate the scenario with dynamic, responsive human entities.



A Fire Team/Squad should request the heavy weapons squad:
e Iforganic squad weapons are not effective against the enemy.
e If the team/squad receives an order from higher to engage the enemy with the heavy
weapons squad.
If the heavy weapons squad is available and not assigned another mission.
If the PL approves employment of the heavy weapons squad.

A Fire Team/Squad should move using bounding overwatch:
e Ifthe fire team or squad comes under enemy fire.
e Ifenemy contact is expected.
¢ If directed to do so.

Action Execution

For the building-clearing scenario, the following high-level goals were identified from
Klein et al. (2001):
Secure the perimeter
Approach the building
Enter the building
Clear the building
Maintain and extend security

Much of the planning aspects and situation assessment knowledge that are necessary in
clearing a building are more the purview of the PL than of the SLs or fire team leaders. We
decomposed each of these goals into smaller steps that represent the reasonable responsibilities
of SLs and fire team leaders. This resulted in the following list of expected Soldier behaviors:
Taking immediate cover
Reporting enemy location
Providing suppressing fires
Firing in self-defense
Establishing a Base of Fire
Using smoke
Reporting to PL (SLs)

Reporting to SL (fire team leaders)

Movement Techniques

Correct orders to subordinates (voice, radio, and hand and arm signals)
Providing cover to other elements

Building breaching

Building entry

Room clearing

Marking cleared rooms

Requesting support from higher

Rotating fire team responsibilities

21



U.S. Army Research Institute
for the Behavioral and Social Sciences

A Directorate of the Department of the Army
Deputy Chief of Staff, G1

ZITA M. SIMUTIS
Director

Research accomplished under contract
for the Department of the Army by

Sonalysts, Inc.

Technical review by

Richard E. Christ

NOTICES

DISTRIBUTION: This Research Note has been cleared for release to the Defense
Technical Information Center (DTIC) to comply with regulatory requirements. It has
been given no primary distribution other than to DTIC and will be available only through
DTIC or the National Technical Information Service (NTIS).

FINAL DISPOSITION: This Research Note may be destroyed when it is no longer
needed. Please do not return it to the U.S. Army Research Institute for the Behavioral

and Social Sciences.

NOTE: The views, opinions, and findings in this Research Note are those of the
author(s) and should not be construed as an official Department of the Army position,
policy, or decision unless so designated by other authorized documents.



Fire Team 1A @ @ Fire Team 28
ie

pang

Building A.

Fire Tear 2A

Fire Team 1B

Trall Fire Team and PL. enter
building. 3% SL tells trail team to
Clear next room. Once first room
is clear SL marks room with “Wolf
Tall"

3" SL — Determines that first room is clear and tells trail team to begin clearance of the
second room. 3" SL enters first room and after verifying that it is cleared marks the room
according to established procedures.

PL — Orders 2" Squad forward to the entry point. Orders 1° Squad to secure the perimeter
of the building. May ask for the heavy weapons squad to come forward to assist 1* Squad
in securing the perimeter.



Ayjngssaoons sey
y0e]U09 ut penbs
ay} sezius0oe1
yey) UoTIpuoCD
8u11033i

e yoddns

0} uOHE|NUIIS
ayy sormbar sry,

“2a0ul 0} penbs
yxau sIopIo Tg
ruonsy payduog

JOg 0} we}
poenurjuos aq OJ, | puodes sIspi0 Tg
udATS
WOYM 0} pue | Sem JOpIO BULISA09
yoeoo WO pue porspio ay} Jaye Jepi0
‘TS au) Aq possasse | sem yey BZuIpnysut | saul sy} paatooas
2q [JIM siapio ‘suoneoipul SAOUL 0} WBS} OY}
jo souanbas oy, quos aBessay | yey) ospopmouy (¢
‘oyM 0} pue
Woy pue palepio Jopi0 BuLaA0o
sem yey Surpnyjout B paataoal
‘suonjeorpul | uree} SULIAAOD oY}
JUas oBessayy | yey) oBpa_mouy (Z
sany BULIOAOD
eptaoid 0} palapso
“qUaUIOAOUT ures} wea} Jay} pue
S,Urea} JayjO | Yowa Joy ay (anbruys2} 1NOd
SIOLABYIG
paambay aomlely,
WoHeM0yay wondiissaq pajaadxq
$}U9UIUI0Z) 40 8}8q p9309}9q] MOF] yosuey, paplejoq UIAq OLIBOIIG



Once we have selected an instructional opportunity, we can address the third question:
What instructional intervention strategy should be used? There are six different types of
instructional intervention strategies that the VOC will be able to provide:
¢ Immediate feedback (both negative and positive)
¢ Delayed feedback that is only given after some period of time
¢ Student Dialog, which is a computer-hosted dialog that focuses on why the Solder
made the choice he did
Situational awareness assessment for the SL and two fire team leaders
After-action review (AAR) focused on team-level goals
¢ Introduce forced or natural consequences into the scenario, particularly in reaction to
human error

The first three items listed will be generated from the individual coach, while the unit

» coach, focused on team goals, will generate the last three. A series of pedagogical rules will help
determine the type of response employed. The instructional responses provided by the VOC for
each class of problems were modeled after an experienced O/C. Minor problems are not dealt
with immediately, but may be recorded for later use in an AAR. A major problem will be dealt
with immediately, with an intrusive feedback aimed at the appropriate individual. A catastrophic
problem would result in a pop-up message announcing the end of the problem and perhaps some
reason for ending the training trial.

Immediate feedback is the most effective type of feedback in most tutoring situations.

However, if a Soldier makes an error during a firefight, then feedback will be delayed until a
_ later in the mission, or during the AAR. The immediacy of feedback will depend upon several

factors:

e The severity of the action

e The number of humans that will be affected

¢ The ability for an intervention to have an overall positive impact on all Soldiers while

not interfering with other salient activities
e Previous actions taken by the Soldier that warranted feedback

The most obvious feedback channel is to create an auditory feedback message using
synthetic speech technology. If the Soldier's action is correct, the coach may supply some or all
of the following information:

e A statement that the Soldier's action was correct
¢ Arestatement of the Soldier's correct action
e Arationale for the correct action

If the Soldier's action is incorrect, then the coach might supply some or all of the following
information:

A statement that the Soldier's action was incorrect

A restatement of the Soldier's incorrect action

A statement of the consequences of the Soldier's incorrect action

A statement of the correct action to take (determined during the Cognitive Task
Analysis)

e A statement of the rationale for the correct action

1]



Joy ajqisuodsar soueuLlojied aye} SBM UOT}OR
SI yoe0: Jo pousg | 3p azTUsOI0y (p
“pajoajap aq 0}
UOHeUTSap aulOs 32
JeALLe ue 10} Moye Japio scout
ysnu uoReULOyuT ul UoHeUnsSap
yuoureaour Jo/pue juouraaour
Jo Ayorpored jo uorjeunsep
pue Ayemnuerp Jeaye ‘TS azqusoooy (€
“Surpring
ay) soyovoidde
1 se uoNRoO]
(01 pue juswioAour
yUas SeM aBessou s,penbs sejnonred
2t oym Surpnyour) ay} UO paseq
SUOT}eOIpUT soyeuIpsoqgns 0} sally ou) 10331,
Jeplo sAoul | JapIo aAoUur Jo/pue 0} uonEjnus
Joypue uoNsarIp ‘JS JO juouraAoul aun samber
yUSWISAOU TS azuZos0y (Z woe sy,
Jt pulysq polsaos
9q Wed STeNpIATpUT
Aue! moy
pure “12409 saptaoid
Ayryue oy Jo opis | uonRyuato ajysoy J9A09 9]qQeRIINS ‘uones0] UMouyUN
Yorya ‘uoNeso] sy} pue A[puary uaald | —jsozeaul 0} (aAour woy aly
Surpnyout ‘19009 Joy J9A09 9]qGe}INS 0} penbs siapio 3209 sayy penbg ,.]
aiqeieae sonnuq aztusooay (| Joypue) saaoy | ayelpouruyy soye], peidiiog
San eee TE pS eT :
SIOLAUYIG]
paumbay aaulesy
voHEULOjay uondiasag papadxy
syuami0;) 40 Req pajs3}3q MOT] psivy paylesaq WIA OLeUadS



Message Type

Data Needed

Trigger Conditions

Message Sent

From, to, when sent,
content, form, delivery
mechanism

User sends a message

Unit movement

Which unit,

Starting location,
Movement formation and
technique,

When movement started

Unit begins moving

Unit arrival Time of arrival, destination | Unit stops moving
location

Sound cue Spatial location, When a sound cue is
What made the sound, delivered to the user
Distance qualifier

Visual Cue Real world location, When a visual cue is
Rendered on display, presented to the user
Field of view location,
Size information,
Duration information,
What was rendered

Friendly fires Firing rate, Whenever friendly fires

Firing direction,
Round destination,

Weapon fired

Hostile fires Friendly unit that could Whenever enemy fires and
know this information, it could be detected by a
Firing rate, unit or units

Firing direction,
Round destination,
Weapon fired,

Base of Fire establishment

Time of establishment,
Entity providing cover,
Location

When established

Wall breech

Time of breech,
Mechanism of breech,
Size of breech

When a wall is breeched

Room entered

Friendly element
designator,
Room identifier

When element enters room

Other Simulation Requirements

The SVS Battlemaster shall generate scenario events in the same way that events are

generated by other simulated entities so that the VOC sees all changes induced by the
Battlemaster station. Events generated by the Battlemaster shall be identified by a host

28



Action Execution Rules and Detection

The following table contains the rules governing the details of how each expected
behavior is supposed to be executed. For each of the rules there is also presented the mechanism
by which we will detect, in the SVS simulated environment, the fulfillment or violation of the
rule. The second column therefore represents the information the simulation environment must
provide in order for the Virtual O/C to evaluate the rule or criteria in the first column. This table
is focused on what information the simulation must provide, not how it is going to extract and
provide it, the “how” is presented in the Discussions section of this report.

Take Immediate Cover

Rule

Automated detection mechanism

Begin moving within 3-5 seconds after
receiving enemy fire

Knowledge of when the enemy began
shooting at unit and when they began
moving

Seeks first available cover

There may be more than one object in the
environment that can provide cover and the
Virtual O/C will be able to determine that
the element under fire has moved to the
closest one that will provide adequate
cover.

Cover must provide adequate protection
from small arms up to 12.7mm.

Knowledge of simulated objects available
for cover, including the object’s location
and which side of it provides cover from
the enemy.

Does not expose any portion of body to
direct fire.

Friendly unit does not move from behind
cover until safe to do so.

Determines Enemy Location

Rule

Automated detection mechanism

Uses visual cues, i.e., smoke, movement to
determine enemy location

Knowledge of enemy location, and whether
or not the visible indications of the
incoming fires were rendered in the display
being viewed by the Soldier.

Uses audio cues, i.e. gunfire, personnel or
vehicular movement to determine enemy
location.

Sound cue location and direction
information.

Reporting Enemy Location

Rule

Automated detection mechanism

Reports enemy location within 30 seconds
of contact

Time of contact with enemy and time of
message sent to higher, if any

Enemy location is accurate within 100
meters (6-digit grid coordinate).

Known enemy location, location content of
the report

Reports to appropriate leader (SL or PL).

Message sent indication

Provides Suppressive Fires

Rule

| Automated detection mechanism



first. This involves comparing the location of the fire team and the visual and audio location
information. There will be some situations when the fire team knows it is being fired upon, such
as when the round impact locations are within a meter of the fire team. Conversely, there will be
situations when the rounds are impacting a great distance from the fire team. However, we must
be prepared to qualify those ambiguous situations where it is not clear who the enemy is firing at.
One way to manage this problem is to define a sphere or zone around the fire team in question
and declare that if rounds are impacting inside this zone, then the fire team is being fired at. This
begs the question of how sharply defined the zone should be, because two rounds impacting one
inch apart should be considered as very similarly placed, regardless of the fact that one was
inside the zone and one was outside the zone. There are two ways to manage this kind of
artificial discrimination. The first is to use fuzzy logic to decide whether a round’s impact
location belongs in the close-enough category. The second is to define the zone large enough so
that the outer limit of the zone is close to the too-far-away limit. Ignoring rounds that lie just
outside the zone is reasonable.

A sureness algorithm based on the image rendering data derived from the simulation
environment will allow us to know if Soldiers were able to detect the cue messages. This
sureness factor will also influence our instructional strategies. If we get a very high sureness
factor, we can safely select an instructional intervention that corrects the Soldier for missing the
cue, if indeed he missed it. However, if the sureness factor is lower than some threshold, and
evidence suggests that the Soldier missed the visual cue, then the instructional intervention
should not make the correction.

When a fire team has been fired upon, a timer can be started that indicates how long it
has been since a particular fire team has taken fire. Once a threshold has been established
regarding how long into the past you must go to ignore past engagements, it is a simple thing to
evaluate the condition that the fire team has been fired upon.

Consider the following algorithmic approach to answering the question of being fired
upon. We must characterize the firing rate and firing direction of the enemy in a quantifiable
way. One way to do this is to consider the incoming rounds or bursts of automatic rounds passing
through a spherical zone around a friendly element that defines what being fired upon means.
Figure 6 below is an engagement detection algorithm flowchart.

35



Bullding A

‘w_ Suppresste Fire
Cover ar Cover

Fire Team 14 Fire Team 1B

Both Fire Teams return fire and
move to covered positions and
provide suppressive fire, The SL
reports the contact to the PL.

SL of squad in contact ~ Establishes base of fire with his squad, establishes local security,
and adds “suppressive fires against the enemy. Reports to PL when base of fire position is

established.
Cover
e
Bulsing A

Inan attempt o slate the

bulding, Fire Team 1A and

Fre Team 28 link up 4 ~_cover

visualy. ‘Wo Suppresshe Fire

ts Goer
Fire Team 16,

‘2°¢ Squad moves up Into position to
attemptto isolate Bullang A

PL — Orders 2nd squad to link up with squad in contact, moves up to link up with the next
squad. The goal is to isolate the building so that additional enemy forces cannot enter the

building.

C-2



then those orders must be forwarded to the correct squads so that the expected behaviors can be
established.

voc

Teameader [gf | Performance
"| Coaches ‘Archives
2
->|
A SVS Simulation F— — — = = ~ Squadteeder |e | Pesfomance
¢ ‘Coach Archive
{ ‘Student Actions
Scenario Tisiructional Interventions,
Definition 0.9. feedback ,
Unit coach f¢ — —} —— — + (|

Figure 5. VOC component diagram

DISCUSSION OF AUTOMATING OBSERVATIONS AND CONTROL

In this section we will discuss the technical issues and challenges associated with
developing the situation assessment rules, behavior detection strategies, and behavior evaluation
approaches.

Automated Situation Assessment

Recall part of what a human O/C would do: observe the situation and form expectations
of Soldier behavior. Thus, we must, as much as possible, encode into the intelligent tutoring
system the human O/C’s map between their understanding of the current situation and the correct
behaviors that correspond to that situation. We call this knowledge map Situation Assessment
(SA) knowledge, and we cal! the human O/C’s conclusions about what the Soldier should be
doing Expectations or Expected Actions. To build the VOC, this SA knowledge must be built
into the system rules that trigger or activate Soldier expectation messages from the simulation.
The SA knowledge forms the software version of the human O/C’s observations of the situation,
and the rules triggered or activated by this knowledge establish the expectations of the Soldier’s
behavior.

As an oversimplified example of the SA rule, we may state, “If a fire team is taking fires

from a visible hostile unit, then the fire team may return fire IAW the ROE.” In this case, an
expectation message would be sent to the coach announcing that the enemy unit has fired on the

31



(feteanial @ Qi)

Bullding A
. Fire Team 34,

ee err

Fire Team 2A
Fire Team 1B

“FRAG OUT", Firsttwo men enter
bullding, SL enters building next.
TLreports when room Is cleared
or that there ts aproblem. SL
decides what to donext. Squad
can continue to clear by room or
askfor help.

3" SL — Calls trail fire team forward to the entry point, executes breach. Once the smoke
has cleared second man tosses in flash or fragmentation grenade and enters breach after
detonation. Soldiers begin clearing the first room according to established procedures.
Assaulting fire team leader is responsible for clearance of the first room, then determines whe

additional personnel are required for clearing room.

C-5



Provides accurate fires on enemy.

Vector details from squad to enemy and
firing vectors and round destinations of
friendly unit.

Provides volume of fire to force enemy to
cease or significantly reduce fire.

Enemy firing information

Does not expend ammunition needlessly
(ammo conservation).

Ammunition remaining, firing rates, firing
direction, rates of fire

Firing in Self-Defense

Rule Automated detection mechanism
Does not fire unless there is an imminent Enemy location, and recent enemy
threat or has been fired upon. behaviors.

Responds with the appropriate level of fire.

Firing rate, direction of fire, impact point
of rounds, weapon being used.

May require use of non-lethal ammunition.

Type of ammunition selected.

Establishing a Base of Fire

Rule

Automated detection mechanism

Able to direct accurate fires at specific
enemy known or suspected position.

Enemy location, and base of fire location

Within effective range of all assigned
weapons.

Friendly location, enemy location, range of
weapon systems available

‘Selects a location that does not limit
communication with higher.

Location of base of fire, and line of sight
obstructions

Does not expend ammunition needlessly
(ammo conservation).

Rate of fire, ammunition remaining, enemy
behavior and status

Use of Smoke

Rule

Automated detection mechanism

Uses smoke only when tactically
appropriate, i.e., blind enemy observers,
defeat trackers, screen an assault, create a
deception, conceal movement, or obscure
enemy observation posts.

Employment of smoke by Soldier.
Location of enemy, enemy action. Tactical
situation related to use of obscurants

Uses appropriate color smoke for situation.
(White for screening, red for emergency,
etc).

Type of smoke selected, tactical situation
related to use of obscurants

Uses amount of smoke consistent with the
situation. Does not use a smoke grenade

when a smoke pot is required or vice versa.

Type of smoke selected tactical situation
related to use of obscurants

Reporting to Platoon Leader

Rule

| Automated detection mechanism

E-12



e Speech recognition (SR) - Speech recognition technology continues to improve and is
a likely source of recognizing the orders or reports of a Soldier. This approach
requires the development of a relevant grammar and some field experience to
determine its practicality in a potentially very noisy environment.

e AC4lI system was developed at Fort Benning, Georgia, for use with the SVS. It
enabled the user to send and receive reports and orders and to view a dynamic map
display. A custom interface device enabled immersive users to control this system,
while others used a conventional desktop version. This approach is more flexible
than menus but less so than speech recognition.

Sound Cues

We must differentiate how sounds are associated with objects versus how they are
presented to the user. The SVS will detect internally if sound is being generated and where the
source of the sound is located relative to the user. This feeds the sound generation system.
Currently, the SVS uses standard Microsoft DirectX sound generation capabilities with four
speakers. There are higher fidelity options, (e.g., the 3D sound system developed by the Institute
for Creative Technology (ICT) that is currently being integrated into the SVS).

Tracking Human Soldier and CGF Ammunition Levels

Ammunition is tracked internal to the SVS and is used to identify when a magazine is
empty and when a Soldier is out of ammunition. At present, the SVS CGF ammunition is not
tracked. This will be added.

Enemy Engages and Misses

All non-guided munitions are not represented in the virtual world as entities. Thus, fly
out is a computation that only has an effect once it “hits” something and a detonation PDU is
issued. Possible alternatives for assessing this and providing to the VOC include:

© Use entity identification information in the enemy’s Fire PDU. Within SVS,

(assuming SVS station or SVS CGF used for OPFOR), a 5-degree cone is generated
at the fire event. Any entity within this cone is identified as the entity being shot at.
If there are multiple entities, then the first encountered is identified. Thus, to
determine if a team is being fired upon, either the SVS or the VOC should have a list
of the members of each fire team (or whatever level is desired), and, if one member is
being fired upon, then the team is under attack. These are existing data on the
network.

© Look for detonations in the immediate vicinity on objects and in buildings. These are

existing network data.

41



si-d

poyeisued
ueumy ponssi | po19jU9 sI Wool se
Jo uayods Jayjo se | payfox ABopoaseryd
oures oY} poypuepy azmusoooy (¢
woos Suliojus {  Woor UI sIaquisut
Joye UMy Yoy urea} Joy}O
Jo 1y3u e Aynuapt 0} SATBIOI “WI0OI
0} yyed Jo siskyeue | Ul 90U0 yUaUAAOUT
Jo jusWIOAOUI jo uonsarp
JO 1099 We eztudoosy (Z
20Ud}SIXO OJUT
AOU AVI Ausue sawios jurod Anuo
Aue sadedua ay} Moy UO paseq
pue wool aijua quand poled}
ay) punore sazed aq snUr SIL,
‘wool ay} SuLojua
9u0z uodn uonsenp Aressaoau TBI ATS]
peyioads v apisur yalioo ay} | jl Sazty ‘uoTeNtIS Ul UBUL IXON,,
st Ayuo ue yey) ut soaowi ‘oseryd ayy sassasse | ‘s]J29 aU puodas
uoneorput oijtoads uoneoIpur WOOI VU! yoarioa Burjjax | ‘punoze syooy pue | 94} pue Ap}o0LI00
® opraoad ysnur WOO! opisul | JUoWaAOUW! Jopes] WOO] ay} S1ojUa | WOOI SIojUa Jopeay ulool Joyua
uonemnuns sy] MOU Jopea] Weay | ures} szTUsON0y (1 Tope] wea} sy, weg} y[Nessy SIaIpjog OM]
Tre
s Surpyng ou}
ur ajoy B Suimoyq
sourey ueumMYy,
@ Aq pares3iy oq
ysnul JUSA9 STU],
SIOIABYIG
poambay aoulely,
WoneULoyuy noydiiosag pajsadxq
syaaurM10;) 40 BEG paya9}0q| MOF] posi payes9q JUIAT O11BUIIS



We conducted a partial cognitive task analysis and a detailed scenario walk-through. We
then examined the results of the scenario analysis and extracted situation triggers and behavioral
details. The last step was to attempt to develop concrete practical methods for the detection and
evaluation of the situations and behaviors that can be converted to software algorithms, rules,
heuristics, and data.

We built a prototype that incorporated a very simple cognitive model for room clearing
using the Unreal Tournament Engine (Unreal and Unreal Tournament are trademarks of Epic
MegaGames, Inc). This effort was conducted to investigate some of the issues associated with
employing the cognitive modeling technology we wished to use in constructing the VOC.

Preliminary Cognitive Task Analysis

We did not attempt a formal or exhaustive cognitive task analysis, nor a detailed training
needs analysis. These tasks should be part of any subsequent efforts. Rather, we focused on a
subset of the small-unit dismounted Infantry subject matter. Our goal was to pick a subset small
enough to allow examination of a number of issues in depth, but broad enough to cover the major
categories of actions and behaviors applicable to a small unit. We reviewed a number of reports
that focused on urban operations (i.e., Phillips, McDermott, Thordsen, McCloskey, & Klein,
1998; Klein, Phillips, McKloskey, McDermott, Battaglia, 2001; Pleban, Eakin, Salter, &
Matthews, 2001). We also examined the material prepared by the STRICOM-sponsored effort
Dismounted Warrior Network (Singer, Grant, Commaford, Kring, & Zavod, 2001). After this
document review we conducted a partial cognitive task analysis consisting primarily of
information from interviews with a subject matter expert. This information was used to develop
tactical scenarios involving a small dismounted Infantry unit approaching, securing, and clearing
a building (see Appendix C). Focusing on a specific and limited tactical scenario such as this
helped manage the scope of this effort.

Scenario Analysis

After the development team and the subject matter expert finished reviewing the
technical documentation and the results of the cognitive task analysis, we created and dissected
detailed actions required of the squads and fire teams in the building-clearing scenario. We
developed a series of sketches showing the position of each squad and fire team throughout the
scenario to force our conversations to be very concrete and specific. During these discussions
we repeatedly asked ourselves a series of questions:

Why was a certain action required?

How is an action performed incorrectly?

What would a human O/C be watching for, qualitatively and quantitatively?

How might a triggering condition be modified to change the expected behavior or
action?

e What level of granularity should be used to decompose behaviors into discrete
actions?



ARI Research Note 2005-01

The Virtual Observer/Controller (VOC): Automated
Intelligent Coaching in Dismounted Warrior
Simulations

H. George Banta and David B. Troillet
Sonalysts, Inc.
Neil T. Heffernan
Worcester Polytechnic Institute
Brian Plamondon
Advanced Interactive Systems
Scott A. Beal
U.S..Army Research Institute

Infantry Forces Research Unit
Scott E. Graham, Chief

U.S. Army Research Institute for the Behavioral and Social Sciences
2511 Jefferson Davis Highway, Arlington, Virginia 22202-3926

Army Project Number Personnel Systems and
20262785A790 Performance Technology

Approved for public release; distribution is unlimited.

iii



Does The Enemy Know Where We Are?

There are ways to answer this question that do not require sophisticated processing
techniques. If the enemy is firing at friendly units, then the enemy knows the location of the
units. A second approach could be the use of a scripted, triggered report, such as a human spy
observing the enemy observing the friendly units and reporting it to the friendly unit. These two
techniques do not solve the general question, but provide mechanisms that do not disturb the
realism of the situation.

Has The Element Of Surprise Been Lost?

This is the same question we just considered: “Does the enemy know where a friendly
unit is?” In this case, not knowing the answer to the question has different implications than the
knowledge about what the enemy knows.

Is Stealth Important?

This issue is more about an operational parameter than it is about a specific behavior,
although determining it can be handled several ways. There are doctrinal rules that specify
stealth for certain actions or procedures. These can be encoded into the VOC’s knowledge base.
Additionally, there can be an order from higher to maintain stealth. The VOC would then know
that stealth was important.

Is The Enemy’s Location Known?

A report from another unit or source of intelligence can announce that the enemy is at a
certain place. Such an announcement is plausible in the real world, can easily be scripted in a
-scenario without disrupting the sense of realism. The message can be sent as a simple piece of
data to the coach, and the report establishes a certain knowledge that friendly Soldiers know
where the enemy is. What follows is a discussion of several SA rules and how they can be
encoded into a combination of triggering expectation messages that are sent from the simulation
to the coach, along with the corresponding rules that create the correct situation dependent
expectations of behavior.

DISCUSSION OF DETAILS OF THE SVS SIMULATION ENVIRONMENT

The SVS will be required to provide specific data to the VOC. While much of the data
are contained in simulation packets transmitted among simulations on the network, some data are
available only inside the SVS. This section begins with a brief discussion of the distinctions
between these sources of data and the implications for the architecture of integrating the VOC
into the SVS. Subsequent sections define these data and suggest ways in which the SVS will
most effectively convey this information.



APPENDIX D

DETAILED SCENARIO ANALYSIS

The goal of the following table is to illustrate the types of results that occur from a
detailed analysis of the scenario. The goal of the detailed analysis is to generate a list of the
ways the Soldier Visualization System (SVS) environment will need to be instrumented to detect
the actions of the trainees to support the student evaluation portion of the Virtual O/C. This table
presents the results of an examination of the sample scenario broken down into stimulus events
(the first column, labeled Scenario Event ) and the related required detailed trainee actions (the
second column, labeled Detailed Expected Trainee Behaviors). Each detailed action is
annotated with:

e Who should perform the action (part of the action description),

e Amore general description of the correct action, later used in the development of a -

related learning objective (the column labeled Target Description),

e A discussion of how such an action might be detected in the context of the simulation

(the column labeled How Detected), and

e What information or data would be needed for that detection mechanism to work (the

column labeled Data or Information Required).

Detailed Expected Trainee Behaviors are specific actions that the trainee is supposed to
take in the context of this generic building-clearing scenario. These behaviors and the content of
the next column (Target Description) form the basis for detailed learning objectives. How
Detected and Data or Information Required columns will have some redundancy in the table, but
are repeated in the Requirements section without duplication. These two columns refine the
requirements for instrumenting the simulation.

The column labeled “Comments” is for additional information regarding how a particular
issue may be handled from the perspective of instrumenting the simulation environment or
otherwise obtaining the information described in other columns of that row of the table.



ol-a

STUY} BU} SION

0} P98N “07

yey} ospaymouy (T

auf [Te SUG

Jopeay VY weal,

aoueuLojied Jo
poud afquidasoe
Guim paysruyy pue
pores sao (S$

woeD0] parisap
sem UOTeO0T
azqusooey (p

uones0]
SUIOS Je PoALe
urea} ezTUsoooy (¢

[EZIN)
ayenbape soumsse)
syuuids puosas ¢-¢
Ul SBM JUSUISAOUL

aztudoooy (¢

SAOUI 0} Jopio
au} paatesal wea}
yey} oztudoosy (1

juoulyeaau09
pure Jaa09 radoid
pue sonbruysa)
yuaweaow Jadoid
Buisn ‘siapio usAId
Sutoq Joye uo
SoAOU YW Weo],

yuiod Aqua
0} SoAOW! YW Wea],

soueullojiod
5° polled

souBULiojiad JO
potted s1qejdoooer
up Ory

yuas sem yodel oy}
yeup azrusooay (p

as

Oo}

spuemml0)

paambay
WoeULOyay
40 BR

pa7293;0q MOH,

wondrosaq
posiey,

SIOIABYIG:
aaulely,
payoodxq
palejaq

WUaAq OLeuaag



ARI Research Note 2005-01

The Virtual Observer/Controller (VOC): Automated
Intelligent Coaching in Dismounted Warrior
Simulations

H. George Banta and David B. Troillet
Sonalysts, Inc.
Neil T. Heffernan
Worcester Polytechnic Institute
Brian Plamondon
Advanced Interactive Systems
Scott A. Beal
U.S. Army Research Institute

Infantry Forces Research Unit
Scott E. Graham, Chief

December 2008

United States Army Research Institute
for the Behavioral and Social Sciences

Approved for public release; distribution is unlimited.

20050126 026



Rule

Automated detection mechanism

Request for support must be submitted in a
timely manner.

Tactical information requiring assistance,
Message sent indication, contents of
message, when message was sent.

Request for support must be IAW unit
SOP.

Contents and form of message.

Request for support must be tactically
sound (e.g., requesting heavy weapons
support to engage enemy squad).

Message contents, enemy situation,
friendly action

Rotating Fire Teams

Rule Automated detection mechanism
Lead Fire Teams are rotated whenever Tasking of team over time
tactically possible.

Teams are rotated before teams are
exhausted.

Team effectiveness, duration as lead fire
team

SL rotates teams before TL requests to be
rotated.

SL Orders follow on team to take the lead.
Message content.

Feedback Samples

The following table presents some samples of the immediate feedback that can be

provided to an individual Soldier’s incorrect behaviors.

Take Immediate Cover

Behavior

Feedback

Target: Moves or orders squad to nearest
suitable cover

Bug: Does not take cover

You should seek a position that provides
good cover and concealment.
Take cover now.

Bug: Moves to covered location too far
away

You should have selected a position that
was not so far away.
You exposed yourself too long.

Locates the Enemy

Target: Takes appropriate action to try and
determine the location and strength of
enemy unit initiating fire

Bug: Unable to determine enemy location

H- You need to be able to identify enemy
positions quickly.
L—Do you see the enemy yet?

Bug: Unable to determine enemy strength

H- Have you determined the strength of
the enemy unit yet?
L-— What size element is opposing you?

E-16





1 Squad Suppresses the Enemy

Target: Squad returns fire IAW with ROE,
ammunition conservation considerations
and in the direction of the enemy unit.

Bug: Squad does not return fire

H -— You need to return fire and suppress
the enemy unit
L- Return fire now!

Bug: Squad expends too much
ammunition

H- Your ammunition expenditure is way
too high for the current situation
L- You're firing too much ammo

Bug: Squad returns fire on wrong location

H- You need to acquire the correct target
before engage them
L— What are you shooting at?

SL Orders a Team to Establish a Base
of Fire (BOF) Position

Target: SL orders the appropriate first
team to move to the BOF IAW with TTP
(e.g., uses BOUT technique) and other
team ordered to provide covering fires

Bug: SL does not order a team to BOF
position

H- You need to move a team up to set up
a BOF position
L — Get that BOF position set up now!

Bug: SL moves both teams at same time

H- You need to have one team cover
while the other team moves

L- You shouldn’t move both teams at
once.

Uses Smoke to Conceal Move

Target: Determines if use of smoke is
feasible, and uses smoke to conceal
movement of teams.

Bug: Smoke is available but not used.

H- You should use smoke to cover your
move.
L— You have smoke, use it!

Bug: Smoke is used, but does not conceal
movement.

H-—-The smoke was poorly placed and
did not have the desired effect.

L- The wind blew the smoke the wrong
way.

Moves to Covered and Concealed
Location

Target: Moves quickly and to location
within a 3-5 second movement (bound)

Bug: Squad moves too slowly (stays
exposed too long).

H- You need to move and get to your
next position within 3-5 seconds.
L- You moved too slow

Bug: Squad moves to a position that

H - The position you selected provides

E-17



Category

Specific Items

Reports to higher

Takes cover

User Status Location

Ammunition

WIA/KIA

Fatigued

Fired at

Enemy Action Fires a weapon

Throws a grenade

Sees friendly unit

Moves

Reinforcements arrive

Surrenders

Enemy Status Location

Becomes visible

WIA/KIA

World Object Information __| Objects suitable for cover

Building location, size layout

Terrain features

Equipment

Existence of vehicles

Location of vehicles

Weapon operational status

The following items represent more complicated types of information needed from the
simulation. The technical details associated with how this information will be obtained can be
found in the Discussion section of this report.

Identification of objects suitable for cover for a specific friendly unit and from a
specific enemy location

Orders or reports that a human Soldier has issued, and both the type and content of
the report

Sound event localization information

How much ammunition a Soldier has at any point in time (both CGF & human)

How to detect an enemy shooting at but missing friendly

Information about entities rendered on the visual display, how big they are, how long
they were rendered in the trainees visual field

Human readable names for all entities in the simulated world that might be referenced
in instructional feedback

Movement information about a human Soldier to determine if the movement path
taken was reasonable

The sweep extent and sweep speed of a human Soldier’s visual gaze and information
to determine if the sweep was enough to view an entire room during room clearing.

26



LI-d

JL suTULIA}9p 0}

jouoneunseq «

Sally Siapeay Wea}

pasn 9q TIM si], pay spunoy e| jo ospajmouy (6
‘Buoy MOY Oy
pue Aejdsip yensta
S Jopea] wre9}
ou) Uo paropuar
9U39S oY} UT
a|qIsIA sem Aursua
2y} JO yonur Moy 0} (ouyso&
SATISUSS SI BBP JO ®) uoreorput Jopeo]
Pury sty} uo paseq <Aopeay wes} ure9} JO MATA
UOTBUTULIA}Op | 0} SIQISIA JUBJeqUIOD | Ul JOSIE} aJeUTYISO]
eB Sune Aursua sysop, | BJO ospaymouy (g
uonemunis
ay apis]no
passasoid aq 0}
uorjeunojut deams
TeNsIA Mei a} 10
‘WOO 34} UT SaryTUS
Tle Jo sdoams payadsur
Tensta amp [je Woy | —ATJensta useq jou
Pealop uonENums | saey yey) WOO! ay)
9y} Woy uolsenb jo syed pamosqo
STU} 0} Jomsue Aqyensta Aue
ouysaf vroyyg | Jo opaymouy (2
Te} 00}
Aq 3urum)-pesy
yeotshyd oy s8ey
auidue Suuapuar
SIOIABYyag
paambay aaulely,
uo} euLIOyUy dondissaq payadxy
syuaurUI0Z do BEG p979930q MOTT yoda, paylejaq | = yuaAq OleUaIg



tI-a

ysnur Joquiswr wes} yutod Arua pue pueurui0s
ueumy au0 asym uonoalp pue ‘JS Jo yautaaow | Burpying 0} Joao W | JO} Wea} pea] YIM
QSB9 B SI SIY], | 39SUO JUOUIOAOUI TS azmudoosy (| | wea], smoyfoy TS SoAoU TS 6
soueuLlojied Jo
pourad ajqeidacor
ou) UTM
yuas sem podal ay}
yey} ozruso00y (p
1S oy
Woy jUas adessoul | juss sem Yodal 34}
Ul :O] aztUZOsay | oYM ozIUSOIIy (¢
aso BuTG}OUIOS yoda
Jo uonisod ay} Jo ua}.
Uy Wodar ayy se”, ayy eztusooey (Z
uontsod
Ul Sy Weal
yey ‘sanbruyoa}
suoneorumuruos
yoou109 Sutsn pue uontsod
uoneorpur | wes sem oder ay | uorysey Afou e ut | Ul st y Wea] yeu)
quas oSessap] | yey) ozTUZos0y (| “Jd 01 suodar Js | Td 0) suodey Js
(vy ural,
spzemo} pue Aulous
Spremo} yj0q
9q PInoys) Zapyoo]
SI gq Urea] alam
aztusooey (p
ay jo uonoang e ayer
amy josey e Sui st gq wes],
SIOIAByag
paimbay aamlely
PCELAUAC UN § wondiiasag poysodxq
s}UsTIUIOD Jo weg paq9930q MOF] josie payieyaq | 3uaAq OLeUaIg



SVS Immersive Architecture

— Radio
¥.
Switcher Controlled
|} By User or TED CAVTBD Spesck
OneshF Source Recognition
Svs SVS | Weapon View Weapon or C4VOther View InterSense Tracking System

System* PC AQ [ormnnnn nanan meee

:

g | svs

PC#I

‘Trigger, Button, Clip Sensor Data from Rifle

User Posture and Weapon Pointing Data

* Immersive System or Desktop SVS, Stealth, or Battlemaster

Shaded items are optional equipment ———-  Audio/Voice
—— Dara

Figure 2. SVS immersive architecture

The SVS is configured so that the user stands in front of a 7.5 x 10 foot rear-projection
screen. The computer generates an image of the synthetic environment and other objects and
entities. The image is projected onto the screen. The Soldier controls his movement through the
environment by means of a miniature joystick integrated into his weapon. The user can see and
can be seen by other entities in the environment. He can engage these entities with his weapon,
and can be engaged by them as well.

The InterSense tracking system provides weapon-pointing information used to project
round impact information into the virtual environment upon weapon firing. This system also
tracks the Soldier’s position with a 10 x 10 foot play area, and is used to monitor posture
(standing, kneeling, prone) that is reflected by the Soldier’s animated character in the virtual
environment.

In the Land Warrior version of the SVS, a second PC is used to generate an independent
line-of-sight (LOS) into the virtual world, and can be used to simulate sensors such as binoculars
or laser ranging devices, or as a weapon sighting display. This configuration has been integrated
with a simulation command, control, communications, computers and intelligence (C41) system,
a digital radio system, and a helmet-mounted display (HMD). A speech recognition system has
been proposed as an additional data source for the VOC. The latter is not a part of the basic
system, but demonstrates the ability to augment the SVS to support additional training
objectives.



Stimulus Event 1
‘occurs in the leaming environment

Domain Experts Assessment Decision
Used! fo update the Leamar Model

Figure A-1. ExpertTrain cycle of operation

Using the Domain Expert, Learner Model, Instructional Expert, and Student-Device

Interface, the student is placed in a situated learning environment. This learning environment

_ can be either a simple desktop computer simulation or a fully immersive environment. Scenario
events within this environment are communicated (via messages from the instrumented
simulation) to the domain expert that in turn invokes expectations of student behaviors. These
expectations are derived from the domain specific knowledge embodied in the domain expert.
The domain expert then monitors the student’s response to these events (also via messages from
the simulation) and assesses whether expectations were met or violated. These assessments are
then passed to the instructional expert and used to update the learner model. The instructional
expert, using the input from the domain expert and information from the learner model,
determines the appropriate instructional feedback and provides it dynamically. The cycle of
stimulus event, student action, assessment, and feedback continues throughout the exercise (see,
for example, McCarthy et al., 1995; McCarthy et al., 1994).

Another way to consider the situation assessment capabilities of an ExpertTrain-based
VOC is to recognize that it is attempting to answer the same question as the human O/C: “Given
these observed conditions, what do I expect of the student?” The resulting VOC monitoring of
student behavior allows the VOC to compare actual performance to these expectations, as
mentioned earlier. However, ExpertTrain was also designed to provide feedback in response to
specific errors identified a priori during the knowledge discovery and engineering efforts.
During these knowledge engineering efforts (conducted during the Mission and Task Analysis
phase of this effort) the “expectations” of behavior associated with any particular tactical
situation can be both positive and negative; that is, in addition to specifying good performance
(referred to as “target” behaviors or states), Sonalysts attempts to identify likely or prototypical
errors (referred to as “bugs”) in student performance. These “bugs” are often tied to more

A-2


Jas UaYyM ‘Yq 0} SHodal JS pul

BuIpping 0} saynol
0} d4j JO Spjayy Ieajo sapraoid uo1ye907]

"quouISAOUT

Jea0U09 jou saop jnq ‘pasn si oyouS
“pasn Jou yng a]qejleak st ayouTS
ising

“o]qiseay

SI OYOUIS JO ash FI SOUTULID}SG 73ad1e]

SAOU [B9DU0S 0} SYOLUS Sas)

“I9A09 OU JO 19409 100d

saplaoid yey) uonisod e 0} saaowl penbg
“(Bu0] 004

posodxa sfejs) Ajmojs 00} saaow penbg
sssng

(punog) yuawaaour puosas ¢-¢ B UTI
woHeOO] 0} pue ATYDINb sano] :3981e |,

UOTBSO]
Pajeaou0s puke paiaaos 0} sao

SUIP]ING ajejosi 0} uoTIsOd
Jo) WUaWDAOU suIZaq Gud) penbs 1x0N

“AOW 0} penbs yxou sdop10 Yq
suonsy pajduag

AO 0} urea} puosas SI3PIO 1$

aU} oures Jv sUTBa} YOG S2A0U TS

sng

Sauly SuUaA09 apraoad 0} pasapso wea)
Jayjo pue (anbruysa} [Og sesn “3-a)
ALL BM MVI AO” 24} 0} aaour 0} urea}
ysuyj oyeudoidde ay} sispio ‘Jy 33918],

AO 0} Wea} B sIapi0 1S

uoyduassag Sng pur jose]

SAOIABIG saulvl] pajoadxg payizjoqg

yudaq O1LBUag



ed

B 10 ABojouyoa} | ued se ys0q
wontugoa1 se Woy
yooeds se yons suru0o
‘skem Jo Jaqumu ore sally
Bur poysyduroose | ay aay “7
9q Ueo oBessau | :Bunsaroyut are yey) yodar
9q} JO yUsJU0O ay} | oBessour oy} ur JUAS ayy JO Jay.
0} ssaooe Suture | o10M ByEp JaAg}eY AY ay} azrusosey (Z
aouruLojiod
jo poled
a1qeidacoe ayy
Umm uoHeULOsUT
ayemooe
Uopeorput | juas sem Yoder ay) | pue oyenbape ym Td
ques oBessow] | yey) eztUSo9y (1 | Tq 0} WoOdor spuag 0) syodai Tg ,.1
Aejdstp royysty “I9A09 SuTye}

uo suoneoIput
ony Surwosut ay}

Ul apeur soaour
Jo UoNed0} ay} pue

uaas aavy pinoa| “Tg ay Aq pamata ‘Td 2U} 0} yodax
‘TS ay} som|oym Buleq Aejdstp | ay Jo quaju00 ayy
0} SATISUas Su} Ul payopual | se yons ‘sioravyaq | ay SueNtu yun
9q TT storazyaq | oom (axyours ‘yseyy quenbasqns | Awroua jo yyBuans
quonbasqns | —9]zznut “3'a) sony Sururuexo pue uoneso]
0} asuodsar SurWooUl ay} Jo 4q pazrusooa1 2Y} suTULIa}ap
Teuononysu | = suoneorpur ojqista 9q |[M uonoe pue 4} 0) uonoe
S.yoroo ay] | ay} JOU Jo JOON Sip Sunoajeq | ayeidordde soyey | Awoua ay} Saye007T
“uorysey Aju @ UT eouruLojiod Jo
Uoye} o19M suOTOR pouad a[qujda00er
yey} Sunenyeaa au) UNAIM
SIOLABH IG
paambay aaulely,
donvulLioyay wondiasaq pajaadxy
sjuamM0; 40 BEG pa}29}9q MOH] yasaey papeyaq | — yuaaq o1reuaag



friendly unit. The message is triggered by the enemy firing, which may be a scripted action
designed into the scenario or a behavior of a semi-automated hostile force. In either case, the
coach is notified of the enemy action at the same time the Soldier receives the fires. This, in
turn, activates the SA rule in the coach and establishes an expectation that the friendly unit
should shoot back. The VOC observes the Soldiers to see if their behavior matches the expected
behavior.

The encoded SA knowledge must be able to adjust the dependent and independent
aspects of the expectations. Automating this SA knowledge can be very challenging. One way
is to hard-code SA knowledge into scenario definitions and into simulation processing. This
technique can work for simple cause and effect SA knowledge, such as always requiring a
specific response when the PL asks for a report. However, this approach can also be brittle when
dealing with dynamic situations where both the actions of the Soldier and the actions of
intelligent entities in the simulated world can change the situation.

Consider the following example of how hard coding SA knowledge could be counter-
intuitive. A fire team might be required to provide cover to other elements. A more complete
expression of this behavior using specific teams and the scenario we have been discussing is:
“Bravo team provides the correct level of suppressive fires while Alpha team moves from one
location to another.” This statement seems easy to say and understand if you are a human subject
matter expert in urban operations dismounted Infantry situations. However, it is a challenge to
turn it into an automated computer-based algorithm or heuristic. A human O/C watching a
training exercise would size up the tactical situation by considering the following:

The locations of the supporting squads

The enemy’s location

The enemy’s recent or current behavior

ROE

The tactical experience of the teams

The current stealth of the teams as they move and cover

Based on an analysis of the tactical situation, the human O/C and the VOC form an
expectation of Soldier behavior that could be different than that cited above. For example, “ravo
team should not shoot as Alpha team moves to its new location unless Alpha team is being
engaged by an enemy element.” What follows is a discussion of the rules that determine one
possible action of a fire team, whether to fire on the enemy.

Should The Fire Team Shoot?

Consider the following series of notional rules, all of which can be in effect at the same
time, with differing levels of importance, depending on the situation. They all help to determine
if the fire team should fire:

¢ If the fire team is taking fire and knows the location of the enemy, then return fire

¢ Ifthe fire team sees the enemy and the ROE permits it, then the fire team should fire
at the enemy



and SVS will mean that the training value of the simulation-based exercises will not be
completely dependent on the military expertise of a human observer/controller (O/C).

Sonalysts, Inc, Worcester Polytechnic Institute, and Advanced Interactive Simulations
collaborated to investigate the feasibility of producing a training system that supplies
instructional interventions that are pedagogically sound and contextually relevant to Soldiers
engaged in small-unit training with the SVS. The proposed prototype training system would
support small unit, dismounted infantry Squad and Team Leaders. Based on our investigation,
we believe it is feasible to integrate an intelligent tutor with the SVS and produce a VOC to
provide sound instructional support to members of small dismounted infantry units engaged in
simulated urban operations exercises. Furthermore, we believe there is reason for optimism that
this approach can be extended to a variety of other first-person simulated contexts.

The intelligent tutoring technology capabilities of Sonalysts Inc. and Worcester
Polytechnic Institute have been exploited to develop a design for the VOC that attempts to mimic
the behavior of a human O/C. The VOC has to observe the situation, form expectations of
Soldier behavior, observe the Soldier, compare the observed behavior with the expected
behavior, draw a conclusion about the Soldier’s behavior, and then make an instructionally
sound assessment regarding if, when, and how to intervene. These instructional interventions
can take the form of immediate feedback to an individual Soldier, critical information for an
After Action Review (AAR), or other actions discussed later in this report.

Investigating the development of the VOC required the following major efforts: (a)
identifying the Soldiers’ behaviors that merit performance evaluations (e.g., reporting to higher,
suppressing an enemy unit); (b) developing situation triggers in the context of a training scenario
that stimulate the Soldiers’ behaviors we wish to observe and to evaluate; (c) determining how to
detect those behaviors in an automated fashion, and; (d) developing instructional strategies that
can adequately respond to both individual actions and small-unit collective behaviors.

The process we used to accomplish these four objectives included a partial cognitive task
analysis and a detailed analysis of a scenario in which a dismounted squad on patrol in urban
terrain undertakes a building-clearing mission. The situation triggers and the detailed behaviors
expected of the Soldiers were derived from these analyses. Following this, the triggers and
behaviors were closely evaluated to determine exactly how to detect them with software and
hardware. Finally, we derived specific situation assessment and behavioral evaluation rules and
criteria. These rules and criteria form the knowledge base that must be incorporated into the
VOC and also proscribe the information that must be extracted from the simulation environment
to detect the situation triggers and Soldier behaviors.

The proposed training system is comprised of a simulated environment that has been
instrumented to extract and feed the necessary situation data, information, and Soldier behavioral
data to the VOC. The VOC has been broken down into two classes of coaching modules, one for
monitoring individual Soldiers (i.e., Squad and Team Leaders) that is called the Individual
Coach, and one to monitor the entire squad that is called the Unit Coach. There will be two
kinds of Individual Coaches, one with the knowledge base for observing and evaluating Squad
Leaders and one for Team Leaders. The various instructional strategies employed by the VOC

44


APPENDIX A

EXPERTTRAIN TECHNOLOGY

Sonalysts began development of ExpertTrain in 1992 as a way to provide real-time
coaching in simulation-based, rapidly changing environments. The instructional metaphor we
used in developing this technology was that of a master-apprentice relationship. Our goal was to
put a synthetic tutor into the learning environment (i.e, embed the tutoring engine into the
simulation), provide the tutor with a detailed “memory” (i.e., a learner or student model) of each
individual student, and support the following high level requirements:

Employ event-based scenarios designed to exercise student mastery of specific
learning objectives,

Monitor and assess student performance throughout a simulation-based exercise,
Provide feedback to the student as required during and after an exercise, and
Update the learner model with mastery and instructional history information.

ExpertTrain comprises the following:

Domain Expert — A software module that represents the knowledge of an expert in the
subject matter domain (dismounted infantry operations, in this case) and assesses the
student’s performance.

Learner Model — A data repository that reflects the student’s mastery with respect to
course learning objectives and the coaching that the student has received (also known
as instructional history).

Instructional Expert — A software module that produces instructional decisions.
Considering inputs from the learner model and the domain expert, the instructional
expert determines whether to intervene in the student’s activity, what issue to address
if an intervention is warranted, which type of intervention to employ, and in the case
of the VOC, to whom to direct the intervention. This module will be extended to deal
with collective tasks, and some investigation will be conducted to determine what
additional instructional strategies may be appropriate in the dismounted infantry
simulation environment.

Student-Device Interface —- The medium of communication between the student and
the intelligent tutor system, which is usually the simulation environment. As such, its
interface creates the learning environment and provides the medium through which
the Instructional Expert communicates its instructional interventions.

A description of how these components interact during a simulation-based education session is
presented in Figure A-1.

)

A-l



will be apportioned differently to the two classes of coaches. The individual coaches will use
real-time feedback as their dominant instructional interventions, while the unit coach will
employ a wider range of strategies and will reserve for itself the ability to pause the scenario for
an instructional intervention.

Building a training system incorporating SVS and a VOC appears feasible, but should be
done in at least two phases: (a) construct a prototype with a carefully selected set of features to
support an evaluation of the instructional effectiveness of the prototype, and (b) analyze the
results of the evaluation and, based on lessons learned, decide what the next effort should be.
The evaluation could suggest that modification of the prototype to support more evaluation is
warranted, or that the concept has shown such instructional promise that it is reasonable to
construct a more capable version that could be used for actual training.

45



oa

94} J9A09 pynoys Jo o[suy “¢] sosn “3'2) dLL
Auraua oy} 0} Awrous soualazoid 8 st} WIM MW JO OD
JYBIs Jo ou] Joyaq | 0} wea) Yous aroy) Jt ISIy 2AOUT 0} SAOUI 0} UTED}
ay} yy urea} | JO} SIOJOBA PlMoys ures} yor ys1yy oyeudoidde sdO€@ %
ay ‘Afqeumsarg | JYSIs Jo our] ‘ZT jo a3paymouy (| ay} sIapio Tg | wee} & slapio Js
(a@7o40I10}0ur
puryaq quam JO
‘yue} pouopuege ue
puryaq yuam “3'a)
uonenyes njosn
ueumy woddns 0} Pliom payepnunts
suLis} [nJSuruesuT ay} Ul salynua
Ayyeonueuies ut 08 pure uoneoo]
0} Surpueyt st Ts AOd peynuepr
ay} aay ApOexo ot) Usemyjaq
T[21 07 aqqe aq isn wonepen05 (¢
-dnyreys orreusos
72 YOBOD a} 0} “yosIe}
UaAI3 UONeULIOFUT | Wo soURISIp ‘eyEp
SI} pue pjIoM | —yYBIs Jo out] Mou
Povejnuns ap | 01 pean “Auiqesms
ou} poururexdoid 40g Suneorpur
-oid aq ued | saynqinye Surpnyout (s)uonesoy
uoneULOsUI st | uoTeUIOFUT AyWUS JOM F19eIINs
suOTJEN}IS SUIS UT Pliom payejnurtg | Jo o8paymouy (7
Zr)
‘adessoul ay} puas
01 IND ® asn 0} Surpying jo
Jasn oy} Sumber 98819A09 JoyoULIad
SIOTARqIG
poambay aaurely,
uonEULiojuy aondiasag paysadxq
syuouiM0; 40 Beg p2999)9q MO] 7288] payiejaq | uaaq o1reUaag



oun Jt Aqyersedso
‘orjeuraqoid

9q ues daams

2y) Surop jueds
zosn sy} Su] MOY
nq ‘uonefnus

ay} apisino

JO opisut passesoid
9q Wed UOTeULIOs;UT
9ze3 JO UOTIAIIG

ysnonpy dooms
Jopeay wea],
ou} pip oduez
qepndue yA,
wool

a]oyM oy} JaA09
PInom o3uez
Jejnsue yey,

woor

ay} noge ezed
Tensta jo daams
Jo a8pajmouy (9

10}99A
um) JUsUIAAOUr

au) pue adessour
uayods oy) wo
Byep Sursn suop

SI uosueduios sty y,

Ppoyad ay Joaoyeyn
YIM yUa}sIsuOD

ST JUSWISAOUL
Jape] ures}

yey eztusooay (¢

“ATeuOTE}S
parepisuoa

aq 0} ou}

JO pouiod ysnoua
BuO] & 10 019Z 0}
ysnous ssojo Bu1aq
SIOJIOA APIDOTOA
uo spuadep

sty, ‘Areuones

SI Sumpouros

yey optoap 0} Asko
skemye JOU StI]

Areuoneys

90U0 “Japes]
ured} JO} ByEp
uoHeRdO] psyteioq

wr0or
0} souRIUD JO JUuOT,

ut paddojs jou
sey Jopea] wee] (p

woneoTMUMOD

sjuomm0;)

poimbay
WoneuLioyuy
40 BEG

pa}99}9q] MOF

tondiasag
yosiey,

SIOIABQag
aoulely
payaadxy
pete

yuaaq OleuaIg



SVS Data Architecture

The SVS is a distributed (networked) virtual simulation system, utilizing either the
Distributed Interactive Simulation (DIS) IEEE standard (v2.0.4) or the High Level Architecture
(HLA) to communicate (over the computer network) entity and event information that occur in
the virtual world (i.e., synthetic environment). Despite initial attempts by the Department of
Defense to transition simulations from DIS to HLA, 99% of networked simulation users run in
DIS mode. The DIS 2.0.4 consists of about 27 predefined network packets (called protocol data
units or PDUs). The SVS uses only four of these packets that make up the majority of all
packets in any typical simulation exercise. These four are Entity State, Fire, Detonation, and
Collision PDUs. The SVS uses other PDUs for simulation control and transmission of
nonstandard information.

The DIS PDUs are transmitted as UDP IP packets on a given network port. An exercise
ID is used to associate a set of PDUs with a given simulation exercise. There are commercial
toolkits that facilitate reading DIS PDUs.

EntityState (ES) PDUs are transmitted onto the network for each entity at a rate
determined by the entity’s rate of change (either angular or positional), or at some minimal time-
based rate if they are not moving enough to exceed the rate thresholds. An ES PDU is also sent
when a state change occurs (e.g., go from standing to kneeling, or alive to dead). Each
EntityState defines the entity, its physical description, and rate of movement, location, status, and
markings. Because each ES PDU contains everything there is to know about an entity, and
because each entity has‘a minimum transmission rate for these ES PDUs, one can joina
simulation exercise late and still learn of each entity in the simulation within a given time period
(normally 10-12 seconds). Another important aspect of DIS (not specific to SVS) is the concept
that each entity decides its own state, for example, if another simulator shoots at and hits “my”
simulation entity, then “I” decide if ] am wounded or-killed.

Fire PDUs are sent when an entity fires a weapon. It contains the location from which
the weapon was fired, direction of fire, what ammunition was fired, and if another entity is the
target. Detonation PDUs are sent when ammunition detonates. For example, one can tell where
a bullet hits from the Detonation PDUs. Collision PDUs are sent when an entity hits another
entity or a structure.

All of the PDU structure and content is defined by an IEEE specification. Data from
within the SVS simulation is collected and packaged for network transmission in accordance
with the defined standards. However, PDUs do not contain all data available from a simulation.
Within the SVS, information on weapon firing mode, ammunition stores, whether objects are
associated with sounds or make sounds themselves, and user control inputs, are not transmitted
over the network. Thus, the implication is that direct communication between the SVS and the
VOC will be required to provide necessary information to the VOC. In this discussion, a
definition of information source will be provided, as will an assessment of the difficulty of
modifying the SVS to provide the required information.

39



REPORT DOCUMENTATION PAGE

T. REPORT DATE (dd-mm-yy) 2. REPORT TYPE 3. DATES COVERED (from. . . to)
December 2004 Interim June 2003 - December 2003

4. TITLE AND SUBTITLE Sa. CONTRACT OR GRANT NUMBER
The Virtual Observer/Controller (VOC): Automated Intelligent DASW01-03-P-0623

Coaching in Dismounted Warrior Simulations

5b. PROGRAM ELEMENT NUMBER
0602785A

6. AUTHOR(S) ‘5c. PROJECT NUMBER
H. George Banta and David B. Troillet (Sonalysts, Inc.), Neil T. A790

Heffernan (Worcester Polytechnic Institute) Brian Plamondon
(Advanced Interactive Systems), and Scott A. Beal (U.S. Army
Research Institute)

5d. TASK NUMBER
215

5e. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT NUMBER
Sonalysts Inc.

215 Parkway North
P.O. Box 280
Waterford, CT 06385

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. MONITOR ACRONYM
U.S. Army Research Institute for the Behavioral and Social ARI

Sciences

2511 Jefferson Davis Highway, Arlington, VA 22202-3926

77. MONITOR REPORT NUMBER
Research Note 2005-01

12. DISTRIBUTION/AVAILABILITY STATEMENT
Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES:
cor: Scott A. Beal Subject Matter POC: H. George Banta

14. ABSTRACT (Maximum 200 words):
Report developed under a Small Business Technology Transfer Research (STTR) contract.

This report describes the efforts and results of examining the feasibility of creating a Virtual Observer/Controller (VOC) to
observe and critique Soldiers’ performance as they are engaged in simulated small-unit, dismounted infantry training using the
Soldier Visualization System (SVS) at Fort Benning, Georgia. The successful integration of these two technologies will mean that
the training value of the simulation-based exercises will not be completely dependent on the military expertise of a human
observer/controller. Investigating the development of the VOC required several major efforts: (a) identifying the Soldier behaviors
that merit performance evaluations; (b) developing situation triggers in the context of a training scenario that stimulate the Soldier
behaviors we wish to observe and to evaluate; (c) determining how to deiect those behaviors in an automated fashion, and; (d)
developing instructional strategies that can adequately respond to both individual actions and small-unit collective behaviors.

15. SUBJECT TERMS: STTR report, Soldier Visualization System (SVS), automated coaching, intelligent tutor, automated
feedback, virtual observer controller, observer/controller

19. LIMITATION OF | 20. NUMBER 21, RESPONSIBLE PERSON

SECURITY CLASSIFICATION OF ABSTRACT OF PAGES
16. REPORT 17. ABSTRACT | 18. THIS PAGE Ellen Kinzer
Unclassified | Unclassified | Unclassified | Unlimited Technical Publication Specialist
702-602-8047



Signals action prior to movement when
bounding moves from one covered position
to another covered position.

Signal indication, start of move indication

Reports when “Moving”.

39

Message sent indication, including “from’
and “to” and movement indications

Reports when “Set”.

Message sent indication, including “from”

Correct Orders to Subordinates

and “to” and indications of not moving

Rule

Automated detection mechanism

Uses order appropriate with the situation.

Recognize content of message sent

Uses most effective method of
communicating order.

Recognize how order was transmitted
(radio, voice, hand signal), recognize “to”
and “from” for order

Does not use voice or radio when hand and
arm signal is required.

Indication that stealth is required in
situation, method of communicating used

SL Moves Teams to Establish Base-of-Fire

Position

Rule

Automated detection mechanism

Uses team with best firing position to cover
other teams movement.

Team location, team field of view, team
weapons fan

Moves team with poor position to better
BOF position.

Current location, destination, BOF location
information

Provides Cover to Other Elements

Rule

Automated detection mechanism

Provides covering fire when required or
requested.

Enemy situation, request from friendly
unit.

Provides accurate and timely fires.

Firing rates, firing direction, impact point
of rounds.

Uses weapons/ammunition consistent with
tactical situation.

Weapon selected, target engaged.

Does not expend ammunition needlessly
(ammo conservation).

Ammunition remaining firing rates.

Breaching Operations

Rule

Automated detection mechanism

Moves tactically to the breach point.

Current location, destination, formation,
movement technique.

Reports when set at the breach point.

Message sent indication, report contents.

Has breach kit on hand and complete.

Equipment on hand

Prepares breaching charge properly.

Breach order to CGF or Soldier

Uses correct amount of explosive for
desired effect.

Breach order to CGF or Soldier

Allows adequate time for exposed

Alert message sent to squad, when charge
is set off

personnel to take cover.

E-14



1* Squad — Splits squad up to allow one fire team to move to a point to continue with isolation
building.

2" Squad — Moves to the other side of the building to complete isolation of the building.

3” Squad — SL tells his squad to prepare to move to the entry point. Moves his squad to desigi
entry point on order.

PL — Gives order to 1* and 2TM squads to lift and shift fires.

1 and 2" Squads — Lift and shift fires on order.

PL — Gives order to 3" squad to move to entry point.

3" Squad — moves squad to entry point.

Fire Team 1A, @ @

Building A

IFire Team 34
«& Fire Team 38

‘3* Squad Is now stacked at the
breaching point prepared to
execute breach. SL remains
outside

34 SL — Tells PL that his squad has arrived at the entry point and has secured a foothold.
3% Squad is now “stacked” outside the breaching point and is preparing to execute the brea


Wisher, R. A., McPherson, D., Abramson, J., Thornton, D., & Dees, J. (2001) The virtual sand
table: Intelligent tutoring for field artillery training. (Research Report 1768). Alexandria,
VA: U.S. Army Research Institute for the Behavioral and Social Sciences. (DTIC
ADA388158)


od

ABojooseryd y9eL09u]

yodar 0} Mo[sg

issng

Jouueul

Ajourn ur ‘pores St Wool 34} yet}

‘Jd 2 03 sodas AoaLI09 ‘Tg 39818],

porea]o st wool Jey} ‘Tq 0} Suoder TS

ayesun A[snorago UsyM polojyugy

I9}U9 0} MOS OOT

ising

Jouueul

Ajaury & Ut ‘pazea[o St UWOOI ay} Jey}

PIO} Buteq Joye “WIOOI sI9jUS TS :JoRIE],

WI0OI SIa}U9 JS

‘ABopoaseTyg }OaLOoUy

snjeys Suoim syodey

yodal 0} Mojs O],

issng

‘opeul us0q,

sey AUISU9 JIM 49R]U0D Je} JO Iea]O St
WOO! JY} TS S[[9} Jopeo] Wes] jade],

WIOOI ay} JO SpIsut
a0U0 snjye}s TS 0} sodas Jopea] urea],

.dS8,J BuIpURS,,
‘spuodsaz Jaquioul urea} yLINO.y

aploap 0} MOjS

ABopoaseryd yooL00uT

Pepaeu jou UsYyM Ul WITY SIOpIE
pepsou

Ayjead sem ay Way 3SB,J pUueIS SIOPIO
isang

“pores]o Sutoq Woo! Ul UOTENIS
[eo1}9e} YJIM JUa}SISUOD pue UOTYSey
Ajouy e ut pue ASojoosemyd 3901109 Sursn
‘op 0) 3eyM JoquioUr ures} SuTuTeUIOL
S][9} Ap}9o1I09 Jopes] Wes], 739818],

Joquiswl Wea}
Sulureulal 0} Japio SoAId Jopeay Wea],

aondiioseg Sng pur jase]

SIOIABYIG UII], poyoodxyq payiejyaq

JUIA OLeUIIg



These behaviors have a mixture of situation dependent aspects and situation independent
aspects. For example, reporting to higher must be done using the expected report format,
regardless of the specific content of the report. The situation assessment rules all have
uncomplicated “Then” clauses, such as “return fire.” The “Then” clause constitutes an expected
action on the part of the Soldier. However, these expected actions need further situation-based
qualification before they can be applied. The additional qualifications associated with these
expected actions are captured as a separate set of rules.

Table 4 presents a sample of action execution rules, whereas a more completed list is
found in Appendix C. For each action there is an associated set of evaluation criteria related to
those actions. The details of the automated detection mechanisms are discussed below.

Table 4
Action Execution Behavior and Evaluation Criteria Used in the VOC

Action or Behavior Evaluation Criteria

Take Immediate Cover

Seeks first available cover, within 3-5
second move.

Cover must provide adequate protection
from small arms up to.12.7mm.

Does not expose any portion of body to
direct fire.

Reporting Enemy Location

Reports enemy location within 30 seconds
of contact.

Enemy location is accurate within 100
meters (6-digit grid coordinate).

Reports to appropriate leader (SL or PL).

Provides Suppressive Fire

Provides accurate fires on enemy.

Provides volume of fire to force enemy to
cease or significantly reduce fire.

Does not expend ammunition needlessly
(ammo conservation).

Uses all weapon systems available ( AT-4,
SAW, M-240).

Action Evaluation Details
There are two kinds of Soldier behavior evaluations: outcomes and processes. Using

state information available from the simulation, we can recognize when the student has or has
not achieved a desirable state (e.g., cleared the building in a reasonable time). That state is an

22



Factor Possible values

Weather conditions. Temperate,

Extreme hot or cold
temperatures,
Humidity,

Wind speed, direction

Terrain from line of departure | Rubbled urban terrain,

to Objective. Clean clear streets.
Distance from line of departure | Less than 2km from LD to
to Objective. objective,
Greater than 2km from LD to
objective
Size of Building to be cleared. | Single story, single room
building,
Multi-story with multiple
Tooms
Proximity of other Friendly Friendly forces within
Forces. supporting range,
No forces within supporting
range
Rules of Engagement (ROE) Complex ROE,
Simple ROE

Attitude of Civilian Population | Friendly Civilian,
Belligerent Civilians,
Hostile Civilian

Urgency of mission Mission Urgent,
Mission Routine
Intelligence Available Accurate Intelligence,

No Intelligence Available,
Poor Intelligence

Situation Assessment Rules

After considering what the information needed to process the “If” clause of situation
assessment rules, we turn our attention to the “Then” clause. The “Then” clause in a situation
assessment rule only determines the category of behavior or action that the Soldier needs to
execute. We will consider the rules for determining the specific behaviors in the next section.
Table 3 presents a simple mapping of stimulus conditions and events to expected behavior
categories.

Table 3

Situation Assessment Rules used in the VOC.

Stimulus Conditions Behavior

Engaged by enemy forces Taking immediate cover
18



Reports to PL in a timely manner.

Message sent indication, recognize who the
report was sent to, when the message was
sent and when the message was required

Uses report format consistent with the
situation.

Content and form of the report

All reports are consistent with unit SOP.

Content and form of the report

Reports are required usually when one of
these actions occurs:

Change to friendly status (WIA, KIA).
Change in friendly location.

Contact with enemy forces.

Call for artillery, mortars or heavy
weapons.

Logistics reports.

The most common reports are:

ACE Report (After Contact, Ammunition,
Casualties, Equipment)

SALUTE Report (Enemy Contact)
SITREP

NBC Reports

Incident type information and type of
report sent. When unit has WIA/KIA
expectation would be for WIA/KIA report.

Reporting to SL

Rule

Automated detection mechanism

Reports to SL in a timely manner.

Message sent indication, recognize who the
report was sent to, when the message was
sent

Uses report format consistent with the
situation.

Recognize content and form of the report

All reports are consistent with unit SOP.

Recognize content and form of the report

All reports are accurate and sent to the
correct recipient.

Recognize who the report was sent to and
its content and form

Movement Techniques

Rule

Automated detection mechanism

Uses appropriate movement formation and
technique.

Enemy situation, current location,
movement formation and technique

Changes formation and technique as
required.

Enemy situation, current location, move
order, destination, movement formation
and technique

When using bounding overwatch does not
stay exposed for more than 3-5 seconds.

Time spent moving, whether or not unit
was behind cover at start and end of moves

E-13



‘VOC creates intemal

apeciaions of behaor
‘Yoo Stustion
[Sims Event Message vasecsrert
Processing Expected Bohevir
VOC Behar
Siowiation Evauton
Processing
Behavior! Evatuton
“ius Event via veer ‘Actual Behar Conchslon
intetace Sirutation
Beier takes scion va Ut
{VOC Instructional
Processing
Triervertion detveréd ia
amit)

Figure 4. VOC processing diagram

Figure 4 focuses on the VOC’s internal structures and processes. The same two kinds of
information leaving the simulation remain: about situation information messages and Soldier
Actions. In this diagram we split the various processes of the VOC and the data being used to
support those processes. At the top of the figure we see a box labeled VOC Situation
Assessment Processing. This is the component where the stimulus event messages are
processed. The output of this processing will always be the expected behaviors. During a
scenario, there may be a large number of expected behaviors in existence. On the right of the
figure is a box labeled the VOC Behavior Evaluation Processing. The data coming into that
segment of the VOC include the expected behaviors from situation assessment processing and
the actual behaviors of the Soldiers. The VOC is constantly comparing the actual behaviors with
the expected behaviors. When it reaches a conclusion about this comparison, it forwards that
conclusion to the VOC's instructional processing. The instructional processing is where a
decision is made on whether to intervene, on what subjects an intervention takes place, a
decision about which type of instructional intervention, and the content of the intervention. If an
instructional intervention is decided upon, then it is forwarded to the simulation for delivery to
the Soldier.

Figure 5 indicates that the VOC itself is made up of several discrete modules. Each of
the individual coaches will be managing their own performance archive as shown by the figure.
Because we focused on a squad, and because there are two fire team leaders in each squad, there
is a pair of fire team leader coaches indicated in the figure. The knowledge encoded in each of
these coaches will be different. During a scenario, each of the Soldiers has unique
responsibilities, and these differences must be reflected in the knowledge used by each of the
coaches. There will be differences between the fire team leader coach and the SL coach, and
both will have a different knowledge base than that of the unit coach. Because our architecture
does not preclude having more than one team, there might be a family of VOCs watching
multiple squads during an exercise. In order to function correctly, the appropriate situation
information must be forwarded to the correct VOC. If one squad has been told to approach the
building and another squad has been told to establish a base of fire for a support by fire mission,

30



accompanying audio feedback. Using appropriate weapon ballistics supplied with SVS, or
customized by the user, the round trajectory is computed, including wind effects. The trajectory
is assess for intersection with geometry in the virtual world, and if found, the entity/object is
determined and appropriate hit effects are applied, e.g., wounding or death of human entities,
“dings” on walls of buildings, puffs of dirt on the ground, etc.

Each weapon carries a standard load for the magazine, and when depleted, the user must
reload the weapon. This, in most cases, is accomplished by removing and replacing the weapon
clip. This simulates replacing an empty magazine with a full one.

SVS can be supplied with the following weapon mockups:
M-4

M-16

M-16/M-203

M-240

M-249

Remington Pump Shotgun

M-9 Pistol

Additionally, SVS functionally simulates other weapons that do not have a physical mockup
available. These include an AK-47, AT-8, and an RPG-18.

The user has the option of employing simulated tracer rounds for specifically-defined
round intervals. Tactical aiming lights (visible and infrared (IR)) can be employed. Laser range
finding is also supported using the weapon as the interface. Firing the weapon optionally creates
a muzzle flash that can be spotted by other participants in the exercise.

Standard Weapon

The standard weapon, i.e., anon-Land Warrior system, operates as described above. The
user has the option of displaying a crosshair or replica “iron sight” graphic on the display screen
that represents the aimpoint of the weapon as it moves through space.

Augmented Display Weapon

With the Land Warrior augmented display, the output of a simulated daylight video sight
attached to the weapon is displayed on a user-defined device, such as an HMD or a simulated
weapon-mounted sight. Crosshairs on this display represent the aimpoint of the weapon.

In addition to being used as a weapon sight, this second display channel can be used to
display magnified binocular imagery on a specially configured device.

Supporting Functions

The SVS system provides the user with capabilities beyond those associated with
employment of a specific weapon. Two specially integrated buttons on the rifle are used to



rd

B9q Pfnom sIyy | Joyo Wo poLayuy Jo o8paymouy (Z | 9010} Autous oy) JT
yun Auraua oy)
uonedo] s,Auiaua | Jo uoNsarp oy) ur
“MO}Oq PoyeoIpUT 24} 0} UoTeD0] | pue suONeJepisuCD
se poaliop juoimo s.penbs uoreAlasuo “I9A09
doueI9]O} B YIM. wo papaou any uonrmunue Awisua | say} penbs soye
pasn Aquo yng ‘yin Auraua 0} penbs | Jo uorsamp yooLI09 “Jou WM MVI ayy sassaiddns | urese sany Awouq
3q Pinoys sy] | Woy s[rejop Joysap | sy) eztUSos9y (] | azj suMjar penbg penbs ,.1 uonoy peydusg
“aseq ospo|Mouy,
uorenyeaa souruLlojied Jo
S,yoR0o oy} potted a[quidaooer
out payesodioour ayy uIM
pue s.qNS souvuuojied | ues sem yodel ay}
Woy peure}qO JO ported | yey} oztudoooy (p
“ange (7 wa}!
Ul SB UOTYsey sures 1s 0}
ay} UI poureygo | wold jUes aBessour | jUos sem yodal oy)
9q ued BIep asou,], Ul :O] azudoo0y | oYM oztuZos0y (¢
“(eremyos
wonEOIp Jo djay wones0]
ayy wim sdeqiod) penbs “¢
uoreuLOyUT soumful “p
oijloeds ur “UMOUY JI
adAy ysnur Tg oy} | Tun Aurous
aroym yowoidde | oy joozig “¢
Suidessoul | pauruuajop
ueAup IND 3q
SIOLAByIG
paambay aoulvay,
WOLnvULIOsay uondiiosaq poyodxy
syuawim10;) 40 vyeq pajaajaq MOF] pose Papieyeq yUdAG OLwUs0g



Behavior

Evaluation Possibilities

Action Detection

Table 6 presents one of the action execution criteria samples shown in Table 4. In this
presentation the automated detection mechanism is shown. The full tables of rules and action

criteria are shown in Appendix C.

Table 6

Action or Behavior Detection Rules Used in the VOC

TAKE IMMEDIATE COVER

Rule Automated detection mechanism

Begin moving within 3-5 seconds after
receiving enemy fire

Knowledge of when the enemy began
shooting at unit and when they began
moving

Seeks first available cover

There may be more than one object in the
environment that can provide cover and the
VOC will be able to determine that the
element under fire has moved to the closest
one that will provide adequate cover.

Cover must provide adequate protection
from small arms up to 12.7mm.

Knowledge of simulated objects available
for cover, including the object’s location
and which side of it provides cover from
the enemy.

Does not expose any portion of body to
direct fire.

Friendly unit does not move from behind
cover until safe to do so.

Determines Enemy Location

Rule

Automated detection mechanism

Uses visual cues such as smoke and
movement to determine enemy location

Knowledge of enemy location, and whether
the visible indications of the incoming fires
were rendered in the display being viewed
by the Soldier.

Uses audio cues such as gunfire,
personnel, and vehicular movement to
determine enemy location.

Sound cue location and direction
information.

Dealing With Uncertainty

There are several sources of uncertainly with which the VOC will have to reason. First,

there is the uncertainty associated with knowing when to deem that a solder has mastered a

particular skill. Then, there is the uncertainty of interpreting a Soldier’s actions. One way to deal

with the former type of uncertainty is to employ Model-tracing Intelligent Tutoring Systems

24



ca

“ayemnooe jou si poday

-aye] Yodel spuag

*yodal puas jou sa0q

isdng

sourulojied Jo pouad afqeidosor oy)
UTYIIM WOReULIOsU! a}eIndoR pue syenbape
Wm ‘Yq 0} yoda spuag 339848],

‘Td 01 suoday 1s ,.1

Wuans Auisus auTULa\ap 0} ]qQeU)
woHNBog] AuIsUa sUTULA}ap 0} a]qeur)
song

aay Supentur yun Auraus Jo

WWsuays pue uoNEd0] ay) suTUO}ep pue
Ax 0} UOTOR ayeudoidde soye] :yosaey

Ausauz oy} $3}290"]

ABE IE} 00} VOTO] P212A09 0} SSAC
43A09 S42} JOU S20q

sing

39A09 9]QRIMS jSaTBoU 0} (2AOUL

or penbs sispso Joypue) SaAoW yad1e LL

JOA09 djeIpoUU] SO¥e I,

UOT}RI0]
uMOUyUN WO} ay SOxe) penbs 1

Wondiiassg dng pue jodie] r

A ENbsipesKT:
SAOIABqIg aamlely pajyaedxy payiejaq,



Regarding the action granularity, there was considerable discussion regarding what level
of behavioral detail was appropriate. We settled on two guiding principles. The first was that
we were trying to teach Soldiers, who have some years of military experience, the knowledge
and skills that are specific to urban operations and avoid training that was accounted for earlier in
their military career. The second guiding principle was that we wanted to focus on those actions
and behaviors that could be legitimately done wrong or “badly” in view of established doctrine,
TTPs, established SOPs, and lessons learned materials. For example, we did not wish to
examine the specific path a Soldier might take moving from one point to another. We did want to
consider whether the Soldier moved from one covered and concealed place to another, and that
the Soldier did not take a path that left him exposed to enemy fire for longer than was necessary.

The cognitive task and scenario analyses identified what we needed; investigating how to
fulfill these information and modeling requirements would answer the feasibility question. We
discovered that attempting to determine whether a Soldier had fulfilled expectations could get
very complicated (see Appendix D). This issue is discussed at length later in this report.

Feasibility and Requirements Analysis

The central goal of this effort, investigating the feasibility of building a VOC, resolves
into two broad questions: (a) Can we extract sufficient information from the simulation
environment to know what is occurring, and (b) Can we model the instructionally interesting
aspects of a human O/C? Once we completed the scenario analysis we had the basis for working
out the following items that were a more detailed version of our two broad questions:

¢ Extract specific rules that governed the behaviors we identified

¢ Consider how we would be able to tell whether a Soldier was emitting the behavior

e Determine how we could initiate the situation triggers inside the simulated

environment necessary for every behavior of interest

e Examine the qualitative and quantitative measures postulated for a human O/C and

consider how these measures could be modeled in the VOC.

The two questions were transformed into rules, usually expressed as an “if-then”
statement, although this is only a notional representation because the actual knowledge
representation is more an implementation question than a design or feasibility question.

Each rule was examined to identify what information would be needed to evaluate the
tule. For example, a situation assessment rule might include an “if” clause that contains the
phrase, “The enemy engages your unit.” In this case, the simulated environment would have to
provide information about an enemy unit and whether it fired at a particular friendly unit. The
analysis of the data and information needed by the rules included enough depth and detail to
ensure that the simulation could extract the necessary data when it was needed. A sample of the
tules may be found in Appendix E.

The second item, detecting whether a Soldier fulfilled an expectation of behavior, is
closely related to the extraction of information from the simulated environment, only this time it



“uonrudoser yooods wunurxeur
BIA paystjduioose pue uoreso]
9q ued UOT}BULIOyUT Auraue 0} WY3Is Jo Juouraaoul uIsaq
si) Sumen oBessour aul] yomp sapraoid 0} spenbs Joyo
‘syreulpury wloy poyoenxe yey) uortsod uoTy200] SuImoyye ary JO
Jo sayeurpr009 uoneuoyur | ‘UOreO0] JO 24} | JY Jo aseq amoas (dOd) a1, JO | seq B saystjqeiso
pus osn ues 1S uones0] JOM Td 01 suodoy (1 B Saljuapl Ts | aseg soynusp! TS penbs ,.1
SBuryIOM o7e
samy oatssoiddns | ye Surmy st
Jl auruusjap 0} | Aurous wy AA “TT
pasn oq [TIM SIY], | soyel SUL] “OT sony Awaugq (p
(amyeay o1j1O0ds
oLeuads 8 aq We
yor) suertAto
Jo aouasqe
Jo souaseid
ay) ‘sdury Surureuior
Joyjo Buoure | uonrmurury “6
‘uoTye1apIsuod ye par
OVUT oye} Jsnu | Butoq jose], *g suuy
Sally WINjor sy} uonsalp SI penbs oy) yeyM
Jo ssouay|qeuosear Buu “ZL ye pue 4sey Moy
oy) Buissessy | saver uu “9 ‘uoym mMouy (¢
saqy JO “Yq Woy suodeam
uonsalp Zutssasse Aavay ysonbar
Joy s0uBI9]0} sn, PreMsoy
a]qeuoseal & sALIap (e1ep Japio uone90} penbs suodeam
0} pasn yoroo | sour ‘syuautaAour | s_,Aulous oy) jnoge AAvay 2AOUr
ay} Ur oNsLMey *8'a) s1olARyaq | SITS ay} aMs Moy | pynoys Tg adzR] st
SIOIABTIG
paimbay aamrely,
woHEMIOyOy wondiiasaq payaedxy
syusurn0d 40 Beg paysaj0q MOP] pais papiejeq | 3UdAq OLIeUIIS



II-d

WOTeULIOJUT
Suny q weeL,

yonur Moy pue
Flazusoooy (¢

~Auroue Sy} 0}
SANBIAI payeoo] St
q weal ory (Z

any JO uoToaIg «
aly jooey

Qou 10
ally prnoys gq weal,
Ji sauruuajop)

SOAOUW
V urea aftyn
samy oatssaiddns

SIaAog g ulea],

MOTBULIOJUI jou 10 Bui3eBua | Jo Joao] 1991109 ayy :SaA0Y]
Suny Awougq Augraua ay} Ss] (| saplaoid g weap V wee] oyu
aouRULloyiod
jo potted
aqqeidoooe uryyIM
pansst podey (7
as|e Surpjouos yodar
Jo uontsod ay Jo uayu0D
uy Hoda oy} seAy ou} aztudoo0y (¢
uoneorpur | quas sem yoder oy}
quas odessayy | yey) ozTUsos0y (Z
“(ueumy |e Jou ore
Aa} Jt) ures} oy} Jo Jequioul
qsor oy) Sustidui0o urea} yoro ‘JS ay} 0) Wodar
sIoquisul | Jo UOTBOO] Ayouy & sepraoid
payejnuns Temoy "ZZ Japeay wea}
Aue pue suoreo0] Jaquisur ay} Jurod Anua
Ss JoquIUI ule} ures} yous Surpying ay} 1
ueuMy ay} Y}0q Jo | JO UOTFBDOT uontsod (payors) uonisod
WOTeO0] By} MOU PILOD “1Z uomtsod ur are | sodoid ayy ul ore y ul are Kay)
0} pou aM sueouL {MOU SIOqUIOUI Wed} |[e | UWeay Jo siaquiaul | yey) TS 0} swodar
SIOIABIDG
paimbay aaulely
WoneUL0yoy aorndiiosaq payaadxq
syusuIMI0; 40 Bjeq pa799}9q MOTT yosieL paiejaq | juaAq oleuadg



INTRODUCTION
This report describes the work done in response to the following Phase I STTR topic:

Develop intelligent, automated coaching and feedback for training dismounted small-unit
leaders and teams within a collective virtual simulation/computer gaming environment.
The intent is to merge two training technologies — intelligent tutoring engines for
individual skill training and virtual/gaming simulations for small-unit, dismounted
operations. A synthetic, intelligent “virtual” observer/controller (VOC) shall be created
within simulations to perform the real-time coaching and feedback functions similar to
those functions executed by actual observer/controllers (O/C) or unit leaders during field
exercises within a unit or at the Army’s Combat Training Centers.

This report is comprised of six major sections: Introduction, Methods, Findings,
Discussion of problems and issues in automating observation and control, Discussion of some
details of the virtual training system of interest, and a Summary of the entire report. This
introduction section presents a statement of the problem and a narrative that illustrates what
might occur during some future operational application of the Virtual Observer/Controller
(VOC). The Methods section describes what the authors did to fulfill the requirements of the
statement of work. The Findings section presents the results of the technical investigation,
focusing on what the envisioned training system would do. The two Discussion sections delve
deeper into how the system would provide the required capabilities.

Statement of the Problem

Training using simulated environments has progressed rapidly in recent years due in no
small measure to the significant investment by the Department of Defense (DoD) in general and
the Army in particular. Simulations for small-unit dismounted warrior operations have benefited
from recent advances in technology. Some of these advances include increased graphical display
resolution and detail in the physical terrain needed for dismounted operations and in modeling
and displaying realistic human behavior. These simulation environments can provide immersive,
realistic, and engaging experiences. However, in spite of the technological advances, simulation
environments are still practice environments. Without the intervention of a knowledgeable
human mentor and the use of sound instructional design of training scenarios, poor performance
may be learned just as efficiently as good performance. Even with a human in the loop there will
be variations in training effectiveness that are a function of the human trainer’s knowledge of the
subject matter and his instructional skills.

As simulation technologies have advanced there have been corresponding advances in the
development of increasingly sophisticated simulated “mentors” or “coaches” in the intelligent
tutoring community. These tools include advanced intelligent tutoring technology where
Domain Experts (also known as Intelligent Agents) are created to monitor and assess student
performance in particular domains within a training environment. The authors have previously
developed and applied training tools to support decision-making training for the dismounted
small-unit leader in the conventional environment. Of particular interest to this project are our



6d

Woy JUSS odessour

quas sem yoda oy}

UT 0], ozlUsOdNyY | OYM azTUZOI0y (¢
‘BULIOAOO ST
tres} YOM “61
Suraow st
ures YOM “81
yusyu0o a8essaul | uoreUNsap Jopio
Sunosjop Surpreser DAOW “LT ayy JO yua}u09
@A0ge Saj0U 29g :MOUY 0} pea | 94} aztUSooay (Z :
yoa0o
0} q Urea,
Jo} Jopio0
ue ureyu09
you yy31ur
Jopso ay], 9]
uoneoo]
peqe
aaey pynoo
Japso ay, °C] asuodsar
91] ST spenbs 1040 8.18 Aq poros3iy
Jopso ay “pT 4q paptaoid usaq aq ysnul UoT}oe
rapnyour sey amy BuLaa09 1909 0} g paiduos sy,
payseos aq pinos uoleorpul | yUas Sem Japio oy} aouo Ajdusoid | urea, 27 aaou! 0}
JEM SOEISIY juos oSessoyy | yey) ozTUBOD0y (| slopio sanssi Tg | VY Wea] SIepio Tg} no ‘Ia80y,, Td
aourULOjied Jo
poued s1qe}da.0e
ou UTA
souruLojied | juas sem yodal ay}
SIOIABHI
paimbay aaulely,
DOHeMoyuy uondiiasag poyoodxy
syaourm0;) 40 weg p9q29}0q MOF] PLT RLA § payiejaq | yWaaq o1euaag



Takes cover.

Current location, destination.

Executes breach on order.

Breaching event indication, order sent
indication, message acknowledged.

Reports whether breach was successful or
not to higher.

Breaching status and time, Message sent
indication, contents of message.

Building Entry

Rule

Automated detection mechanism

Moves on order to the entry point.

Time of order, Unit location, movement
Start time and destination.

Moves tactically to entry point.

Unit formation, movement technique.

Reports when set at entry point.

Time of reaching destination, message sent
indication, contents of message.

First man tosses concussion or
fragmentation grenade if required.

Time of order to entity tossing grenade
(human or CGF)

First man moves into room moves right or
left.

Soldier (human or CGF) location, direction
of movement.

First man tells second man to move into
room right or left.

Message sent indication, contents of
message.

Second man moves into room moves right
or left.

Soldier (human or CGF) location, direction
of movement. :

TL enters room (if required)

TL location, direction of movement, enemy
actions in room, when TL entered.

TL moves right or left.

TL location, direction of movement.

TL sweeps room visually.

TL location, TL direction of gaze
information

TL acquires and engages enemy if
encountered.

Enemy actions, TL action on contact.

TL orders last man in room, or tells last
man to stand fast.

Message sent indication, recipient, message
contents.

TL reports room cleared to SL.

Message sent indication, contents of
message, when message sent, status of
toom being cleared.

SL reports room cleared to PL.

Message sent indication, contents of
message, when message sent, status of
room being cleared.

Marking Cleared Rooms

Rule

Automated detection mechanism

Room is marked immediately after
clearance.

| TL or SL marking action indication

Marking is done IAW unit SOP.

Marking procedural details

Marker is clearly visible in both day and
night conditions.

Location of marker information

Marker is not removed without permission.

Marker removal indication

Requesting Support from Higher



0c-d

suonoe a}BUIpI00.
aplomney proar 0} 0} Urea} pea] YIM
syodar snqejg | suonoe ayeurpio09 BuryiOM sutoo1
‘suonjeso] wea], ysnul sueo | sie9jo ure9} [Ie],
~AyLmdas Suroduo
aINsut 0} paaoul
Ajayeudoidde wool }xau
are spenbs Jayjo | ay} 0} souejUS ay)
ay) aou0 Burpying 0} pue Surpying
SJ9pIo SAOUT | 94} OFT Urea} aly ayy OyUT UTD}
JOO SAH] PITPUEH | [TEN at} SapsO ‘TS | SY [FEN SIOp1O “TS
“soouley} ueuNy JO
JorUOS oy) Jopun
aq Kew fay} pue
‘pazmooo aARy
‘syuouoaow penbs
Joyo Aue [yun
4n990 jouurd sy,
“Burpying oy}
Surespo anutyuos
0} TS sunnesse
SIOIABag
peambay aoulely,
UOBULIOJO] wondi1os3ag paysodxy
syusuiMi0;) Jo BBQ pa}99}9q MOF, paiey papeyaq | yuaaq o1enaag



The SVS Authoring Station provides all of the features required to visualize the
combined-arms synthetic battlefield. Freely move through the virtual world or “attach” to other
entities in your simulation. Once attached to an entity, you can observe the exercise from either
the entity’s first person point of view, or from a third person “over the shoulder” point of view.

The SDT enables the operator to create customized scenarios in any synthetic
environment, including cluttered urban areas. The SDT allows static models to be placed
through a 3-D ‘drag-and-drop’ interface. It also enables the creation of multi-state objects, such
as smoking and flaming vehicles, as well as shrouded weapons, doors that can open/close, and
fenced areas. SVS Authoring allows for the creation of “persistent objects“(POs) compatible
with ModSAF and the OneSAF such as chemical zones, lines and points. These scenarios can be
created beforehand and saved as files that can be loaded at simulation run-time.

The Authoring’s embedded computer-generated infantry forces capability enables the
user to add dynamic friendly or opposing forces to the simulation exercise. These forces can be
given scripted behaviors such as move, shoot, and follow and can further react to external events
such as engagement by opposing forces. AIS authoring is also ModSAF and OneSAF
interoperable for additional simulation development capabilities.

SVS Battlemaster

SVS Battlemaster station is a simulation monitoring and control and after-action review
tool. Built upon the SVS Stealth simulation visualization system, the Battlemaster station
extends Stealth capabilities to provide start- to-finish simulation exercise control. These
capabilities include an Exercise Controller for real-time simulation initialization and control; and
an AAR capability to review simulation activities recorded by the Battlemaster Station

As with all SVS simulation products, the Battlemaster features real-time 3D graphics and
directional audio and provides native support for Distributed Interactive Simulation (DIS)
network protocols and the High Level Architecture (HLA).

The Battlemaster station provides full simulation management and control. Simulation
-entities can be paused, teleported, revived, resupplied, and re-initialized. All distributed
environmental effects can be controlled such as wind, fog, rain, snow, and time-of-day. In
addition, the exercise controller provides an artillery tool to enable the exercise controller to
provide real-time dynamic munition effects during a scenario.

The Battlemaster’s inherent capability to record internal simulation events can be
replayed through an (AAR) feature. These recorded data files can be replayed through a VCR-
like interface that supports jumping forward and backward and pausing the simulation. The
Battlemaster supports helpful 3D display features such as firing identification lines, entity
overlays and wireframe modes that assist in the AAR capability.

Commands and signals for SVS Computer Generated Forces (CGF)

1. Indigenous SVS CGF are simple, deterministic entities that perform specific behaviors in
response to specific commands. Their behavior is scripted using a combination of 3D
graphical inputs and a graphical user interface (GUI) selection mechanism. This differs
from semi-automated forces (SAF) such as OneSAF in that SAF employ some level of
autonomous “reasoning” based on some programmed artificial intelligence. Thus, with

B-4


Individual or unit behavior or performance assessment decisions are the result of the
comparison process indicated in the center of the figure. Each action that is recognized in the top
of the figure is “offered” to each of the expectations originating from the bottom. In turn, the
expectations can “claim” an action as matching a target expectation or matching a known bug
associated with that expectation. In practice, the matching process is sensitive to learner model
mastery data. As an individual or unit gains mastery, additional performance precision is
required to match a target expectation. As additional actions are recognized, expectations
(targets or bugs) may become “completed.” When an expectation has been completed, a positive
(for targets) or negative (for bugs) assessment decision is rendered.

A4



implementation of any system that seeks to automate this processing. To do this with software
requires not only the identification of the relevant factors, but ways to extract or derive their
values and infer their importance to the simulated world.

We have already discussed the VOC’s need to assess the unfolding situation and derive
behavioral expectations. This represents one class of knowledge that must be developed. We
refer to this as situation assessment knowledge. The VOC must also be able to observe the
Soldier's behavior and compare it to predetermined expectations. This latter capability depends
on the VOC’s knowledge of what constitutes a match between an expected and an actual
behavior. A match is declared when observed behavior resembles expected behavior within
tolerances or performance qualifiers. We refer to this knowledge as “behavior evaluation
knowledge.” Together, the situation assessment and behavior evaluation knowledge comprise
the knowledge base that the VOC needs to function effectively.

In examining the various aspects of situation assessment that are required to critique a
building-clearing exercise, there is a wide range of technical complexity. The simple issues
include determining whether a Soldier responded to a request for information in a timely fashion.
The more complex issues include deciding whether the enemy’s actions are sufficient for a
particular fire team to engage them. Our investigation examined the range of issues and
concluded that they are not insurmountable. In this section of the report we will discuss the
automated measures of performance and effectiveness that the VOC will need. These measures
will be represented, at least notionally, as a series of rules and heuristics.

As we began to assess the situation assessment challenges in this domain, we realized
that 'the number and complexity of the rules that would be needed for intelligent, automated
situation assessment were high. Our approach to managing the complexities we encountered,
especially in automating situation assessment behavioral expectations, was based on a divide-
and-conquer philosophy. We were looking for ways to attack this problem with a multi-phased
approach: (a) prove feasibility, (b) build a small prototype, and (c) build and evaluate the

prototype.

Our solution amounts to breaking up the logical processes needed by the VOC into a
series of smaller steps. The VOC is always trying to answer two questions:

© What should the Soldiers be doing under the immediate conditions?
e What are the Soldiers actually doing?

The first question manifests itself in the VOC as the following generic situation
assessment rule: If (some situation exists) then (take some action).

The “If” clause is an assessment of the simulated world, in the form of a conclusion that
some specific, relevant situation exists. The “then” clause represents an action that is expected
of the Soldier. We decided to handle the processing of these situation assessment rules in several
steps. The process of evaluating the “If” clause conditions will be done separately from the
processing of the “Then” clause. The processing of the “If” clause will be further broken down
into three steps:

14



One benefit to this approach is that CTDBs are frequently built for many of the 3D databases that
the Government uses, and the MES structures also carry information about apertures, windows,
and doors.

An alternative approach is to use the SVS zone feature to define rooms and hallways.
This would be a laborious process, but tags could be applied for reporting.

The SVS could use a visual cone defined by the system field of view around the
orientation axis of the user. In the SVS, the user’s head is not-tracked to determine field of view.
Instead, the user’s body orientation is rotated to look around. This cone can be tracked and its
intersection with the room walls can be used to conceptually paint the room with the circular or
oval intersection of the cone with the wall. The total area thus painted can be assessed, and when
a specific percentage of the area has been painted, the room can be assessed as searched. This
would be a moderately challenging task to program into the SVS and would be available as
internal data only.

SUMMARY

Training using simulated environments has progressed rapidly in recent years as a result
of increased investment by the Department of Defense in general and the Army in particular.
Simulations for small-unit dismounted warrior operations have benefited from recent advances in
technology. Some of these advances include increased graphical display resolution and detail in
the physical terrain needed for dismounted operations and in modeling and displaying realistic
human behavior. These simulation environments can provide immersive, realistic, and engaging
experiences. However, in spite of the technological advances, simulation environments are still
practice environments. Without the intervention of a knowledgeable human mentor and the use
of sound instructional design of training scenarios, poor performance may be learned just as
efficiently as good performance. Even with a human in the loop there will be variations in
training effectiveness that are a function of the human trainer’s knowledge of the subject matter
and instructional skills.

As simulation technologies have advanced there have been corresponding advances in the
development of increasingly sophisticated simulated mentors or coaches in the intelligent
tutoring community. Intelligent tutoring systems have been fielded in areas ranging from the
deployment of Field Artillery units (Wisher, McPherson, Thornton & Dees, 2001), to teaching
students the details of solving Algebra problems (Anderson & Pelletier, 1991). These ongoing
tutor development efforts have enhanced the tutoring capabilities of the embedded virtual
coaches. Furthermore, there is an increasing body of evidence that these tutoring systems
produce significant improvements in instructional effectiveness and efficiency (Wisher,
McPherson, Thornton & Dees, 2001).

This report describes the efforts and results of examining the feasibility of creating a
Virtual Observer/Controller (VOC) to observe and critique Soldiers’ performance as they are
engaged in simulated small-unit, dismounted infantry training using the Soldier Visualization
System (SVS) currently in use at Fort Benning, Georgia. The successful integration of the VOC

43



References

Anderson, J. R., & Pelletier, R. (1991). A developmental system for model tracing tutors. In
Birnbaum (Ed.), The International Conference on the Learning Sciences. Association for
the Advancement of Computing in Education, Charlottesville, Virginia. New Jersey:
Lawrence Erlbaum, Inc.

Corbett, A. T., & Anderson, J. R. (1995). Knowledge tracing: Modeling the acquisition of
procedural knowledge. User Modeling and User-Adapted Interaction, 4, 253-278.

Klein, G., Phillips, J., McCloskey, M., McDermott, P., Wiggins, S., Battaglia, D. (2001).
Decision-Centered MOUT training for small unit leaders (Research Report 1776).
Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.
(TIC No. AD-A394 066)

Martin, J., & VanLehn, K. (1995) Student assessment using Bayesian nets. International Journal
of Human-Computer Studies, 42, 13-17.

McCarthy, J. E., Wayne, J. L., & Morris, J. J. (2001, June). Closed-loop adaptive education.
Paper presented at the 13" World Conference on Educational Multimedia, Hypermedia, &
Telecommunications (Ed-Media 2001), Tampere, Finland.

Phillips, J., McDermott, P.L., Thordsen, M., Mckloskey, M.J., & Klein, G. (1998) Cognitive
requirements for small unit leaders in military operations in urbantTerrain. (Research
Report 1728) Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social
Sciences. (DTIC No. AD-A355 505)

Pleban, R.J., Eakin, D., Salter, M, & Matthews, M. (2001). Training and assessment of
decision-making skills in virtual environments (Research Report 1767). Alexandria, VA:
US. Army Research Institute for the Behavioral and Social Sciences. (DTIC ADA389677)

Pleban, R. J., & Salvetti, J., (2003). Using virtual environments for conducting small unit
dismounted mission rehearsals (Research Report 1806). Alexandria, VA: U.S. Army
Research Institute for the Behavioral and Social Sciences. (DTIC No. AD-A415 298)

Russell, S., & Norvig, P. (2002). Artificial intelligence: A modern approach. New York:
Prentice Hall.

Singer, M., Grant, S., Commaford, P., Kring, J., & Zavod, M. (2001) Team performance in
distributed virtual environments (Technical Report 1118). U.S. Army Research Institute for
the Behavioral and Social Sciences. (DTIC No. AD-A396 489)

Department of the Army, (1992, April), Infantry Rifle Platoon and Squad (FM 7-8).
Washington, DC: Author.

47



sd

J0 ported

yey} aztus0o2y (p

IS

Woy JUSS oBessoul
Ul :O], ozTusOI0y

Oo}
quas sem yodal oy}
oym oztusoo0y (¢

yua}U09 asessoul yodar
Surjoojep Surpresor Joplo S.1d 9Y} JO yUa}U0O
aaoge sajou 99g | ZuIspaymouyor JS ay} aztusosay (7
uoneUuoyUt
3991109 ay} | SAOU 0} JopsO ay}
ques sem jiodal ayy | yyiMm pue Apduoid Bur3pajmouyoe
yey} oztudooay (1 syodai yg ay, | Tq 0) sHodar yg
yarod Anua poos
®B syoa]as TS oy}
JL 99s 0} sayovo09
ay} Surmoyye
(peq are Surpjmq
ayy JO saovy (aseo
3SaM pue YyNOs at} Sty} UI yoyeMaA0
“8'a) yuiod Anuo Sutpunoq)
poo e soyeorpul sanbruyoa}
yey) uoTeuLoyUT JUSUIOAOUL 4991109 spenbs ,,7 pue Surpyinq
TS oy) oad “MoOfaq peyst] | pure yuoujeaoU0D ss1 Aq papiaoid ay} ynesse 0}
Jo ‘st yutod SIOTARyaq paliejap | pue JoAo09 sodoid SI any Butzaa09 | aredaid 0} uore90]
Ayqua 3y} a34yM sump smo90 Buisn uoTe0} g0u0 uontsod | Axuo poyeusisap eB
Jopesy penbs JotAvyag Areurums | 4991109 0} ATyoINb | —yynesse 0} penbs | 0} accu 0} penbs
1} PInos Td sty} JO uoT99}9q SaAOUl penbs SoAoUur nS pe | pif S19PI0 Td

“auly JO
aseq 94} 0} psAoul

syustIO;D

pasnbay
uorEULIOjUy
40 BBQ

paysajaq MOP

uondtsosaq
psuny

SIOIABY IG

aaureay
paysadxy
papleyeq

JUIAY OLlBNIIg



The SVS software provides total system functionality that can be divided into the general
categories of synthetic environment display, human capabilities simulation, weapon
employment, and other supporting functions. Other products independent of the SVS provide
additional system capabilities such as instructor system control, scenario generation, data
logging, and replay.

VOC Concept Of Operations

This section of the report describes how the VOC processes instructional and trigger
conditions. In the following major section (Automated Measures of performance and
Effectiveness) we will discuss the situation assessment capabilities of the VOC.

The VOC’s basic operations can be described as:

Observe the situation

Form expectations of behavior

Monitor Soldier performance and compare to expectations
Intervene instructionally when expectations are violated.

This description of the VOC concept of operations focuses on instruction and intervention.

Let us consider how the VOC will be triggered into action. Much as a human O/C may
stand silently observing a training exercise for periods of time until some interesting event occurs
and then take an action, the VOC needs similar triggering mechanisms. Because the simulation
is sending a variety of messages to the VOC, these messages will be used to initiate instructional
processing. In general, there are two kinds of messages being received by the VOC, and each
trigger different processing. When a message about a change in the world is received (these are
called Expectation Messages), the VOC updates its internal situation assessment information
with the new data and then modifies expectations of behavior warranted by the change.
Following that, it performs a review of all outstanding expectations to see if there are any with
expired periods of performance. If there are, the VOC’s instructional processing is initiated.

The second kind of messages that the VOC receives is those sent in response to an action
taken by the human Soldier being trained (these are called Action Messages). These messages
initiate the behavior evaluation processing where the VOC compares the Soldier’s action,
represented by the message just received, to established expectations. The match is successful if
the expected and actual behaviors agree within appropriate tolerances (Target condition), or, if
the actual behavior can correspond to an anticipated error condition (Bug). Regardless of
whether the evaluation results in the declaration of a Target or a Bug, the evaluation processing
concludes by initiating the instructional decision-making processing.

The instructional processing seeks to answer the following questions:

e Is an instructional intervention warranted?
e Which Target(s) or Bug(s) should be addressed?
¢ Which instructional intervention strategy should be employed?



targeted coaching strategies. The expected actions (targets and bugs) are described in terms of
observed student behaviors or effects (e.g., explicit student actions or changes within the
modeled world that indicate dismounted infantry actions).

Figure A-2 illustrates the processing within the ExpertTrain tutoring engine during a
simulation-based exercise. Beginning at the bottom of the figure, certain aspects of world data
are determined to be instructionally important; that is, they represent trigger conditions for some
expected individual or unit behavior or performance. These “instructionally important world
conditions” are then “instrumented” in the simulation by a set of data-driven “sensors” or
“demons” that are sensitive to those conditions and transmit them to the domain expert from the
simulation.

Action << ActionData
Sensor Layer
Action Layer
Comparison Process
Expectation Layer
. Sensor Layer
World Data :

Figure A-2. ExpertTrain assessment process

Continuing up through the figure, these incoming messages from the simulation trigger
domain-specific rules regarding certain combinations of world data that then form the basis for
generating expectations of individual or unit behaviors. In a manner similar to a semantic
network or a production system, ExpertTrain (as the Virtual O/C) recognizes certain
combinations of events or states, links them to expectations, and prioritizes those expectations.
For example, if the sensor layer recognizes conditions A, B, and C, then ExpertTrain might
expect the student to perform actions 1, 2, and 3 with a priority of HIGH. However, if instead
the world data reveals conditions A, B, and D, then ExpertTrain might expect the student to
perform action 4 with a priority of LOW.

A similar process examines the student’s performance. Beginning at the top of the figure,
separate sensors (instrumented simulation elements) examine the states of world or individual
and unit data. The sensor findings are combined to indicate the Presence or absence of critical
student activity or behavior.





Fire Team 1A e@ @

Bullaing A,

FS) [a

cm)

Fire Team 28 enters building and
the clearing of rooms continues.

Platoon Sergeant — Calls for resupply of ammo, prepares for consolidation.
PL — Reports to Company Commander when the building is cleared and asks for additional
instructions. Insures that consolidation activities are conducted prior to movement.

C-10



outcome. For example, the coach might say, “You took too long to clear the second room.” An
outcome-based assessment will not attempt to determine if the student failed to establish the
desired state or failed in his attempt to do so. On the other hand, a process assessment would
recognize when the Soldier made an error in the process leading to that state. For example, the
O/C might say, “You waited too long after receiving the order to breach the building.” Both
process and outcome assessments have their usefulness in this domain and both will be used
where appropriate.

Table 5 presents a sample of the specific behaviors we have identified in the building- /
clearing scenario and how they might be evaluated by the VOC. For each behavior or action,
there are a number of possible evaluations that can be made. Whenever the evaluation concludes
that the behavior is as expected, a target is declared. However, when behavior does not meet
expectations, there are specific aspects to the failure (e.g., an action was taken too late) that we
can look for. These failures are called “Bug” conditions and are listed in Table 5. Following the
table, we will discuss how each of these Target and Bug conditions will be detected.

Table 5
Behavior Descriptions and Evaluation Possibilities Used in the VOC
Behavior Evaluation Possibilities
While Team A Moves: Target: Team B provides the correct level of
Team B Covers suppressive fires while team A moves
Bugs:

Shoots when not needed

Does not shoot when needed

Shoots when risk of fratricide is too high
Does not look towards enemy location
Does not look towards Team A

SL Reports to PL that Team A is | Target: SL reports to PL, in a timely fashion and

in position using correct communications techniques, that Team
A is in position

Bugs: .

Reports before hearing from Team A

Slow to forward report to PL

Incorrect phraseology

3 SL moves with lead team for Target: SL “follows” Team A over to building entry
C2 point

Bugs:

Doesn’t follow assault team

Leads assault team

Goes somewhere else

Team B moves to building entry | Target: This movement action should be identical to
location the Team A move. Refer to that move for details
Bugs: .

Moves before other squads can cover

Does not go to correct location

23



(Fein @ Q@ina2)

PA" (Qghie Team 2a]

@ Fire Team 1B

Building A

Fire Team 2A into building as Fire
Tear 2B and 1B covers
movemert.

3" Squad — Continues to clear the second room, signals 2"! Squad when to enter the
building. an Squad continues to next room to begin clearing.


APPENDIX C

BUILDING CLEARING SCENARIO

The following figures and narrative present the example building-clearing scenario used
throughout this report.

Building A

4 Squad Is moving ushg BOUT
along the street towards Bullding A
when it receNves fire from Building A.

Fire Team 18
(Fire team 14.

PL — Identifies a base of fire position and directs the squad in contact to move there. PL
also attempts to move forward to gain better situational awareness.



Based on what the VOC knows so far, two of these rules can be evaluated. In the
detailed design of these rules, there would be further qualifying information associated with
every component of the “if” clause to deal with the uncertainty that may exist. Thus the rules
will not necessarily resolve to an absolute certainty. For example, we may be confident, but not
certain, that the enemy knows the location of a particular friendly unit. Let us assume that two of
our rules evaluate so that their “If” clause is true, and that the ”Then” clauses both say return fire.
This creates the expectation that the friendly unit should fire at the enemy. If two rules evaluate
to different but simultaneous conclusions, then a voting or weighting strategy must be employed
to determine which conclusion is most important. Later in this report we will examine how a
Soldier’s actual behavior is evaluated in light of expectations. The details of how the Soldier
should fulfill the expectation are left to subsequent processing. During that processing we will
consider other factors, such as remaining ammunition, what weapon to use, and how many
rounds to fire.

We have not yet dealt with the third question: Who is affected? In our example, the
enemy engaged a friendly unit. The simulation provided information about the enemy
engagement, such as where the rounds were impacting and what evidence was available to reveal
the enemy location. Determining what friendly unit was involved requires figuring out if the
rounds were impacting close enough to any particular unit so that they would consider
themselves under attack. For a single engagement by an enemy, one friendly unit might be
considered under attack while a more distant unit might only be expected to take cover and
watch.

Situation Assessment

In this discussion we presented a series of specific situation assessment triggering events,
situation factors, and examples of each. We then described-how we detected all of these items in
an automated fashion.

The first step of the situation evaluation process was to identify the situation category and
the specifics of the situation. Table 1 contains examples of stimulus categories and specific
instances of those categories.

Table 1
Stimulus Categories and Specific Instances of Those Categories in the VOC
Stimulus Category Stimulus Examples
Enemy Actions Engage friendly element,

Movement,

Surrender,

Retreat
Orders from Higher Assault,

Retreat,

Request for report,

Move

16



If the fire team has recently seen an enemy and the element of surprise has already
been lost and the ROE allows it, then the fire team should fire at the last known
location of the enemy

¢ Ifthe enemy does not know the fire team’s position and stealth is important and the
enemy is adequately suppressed, then do not fire :

¢ Ifin hostile environment and the ROE is non-restrictive and the enemy location is
known with confidence and the fire team has been fired at, then fire at the last known
enemy location

Notice that in these rules the “If” clause is the situation condition and the “Then” clause
is the expected action that relates to the condition. Thus, the information we need out of the
simulation environment is whatever will inform the coach that the “If” condition has occurred.
In the list of rules above, the following “If” conditions must be identified:

e Fire team is being fired at
The fire team has been fired at
Fire team sees an enemy
Fire team has recently seen an enemy
ROE in effect
The enemy does not know the fire team’s position
The element of surprise has been lost
Stealth is important
The enemy’s location is known

We will now examine how the simulation environment can inform the coach in each of _
these situations.

What Does “Fired At” Mean?

The first item, Fire team is being fired at, requires that we resolve what being fired at
means. There are two aspects to this information. The simulation environment has knowledge
of what the enemy is doing because it is under the control of the simulation. However, in most
cases, the simulation cannot inform the coach of ground truth unless the Soldiers can also
perceive ground truth through their interface with the simulation environment. Thus, the coach
can only be told what the Soldiers can know about the situation. In this case, the fire team will
likely hear the sound of the gunfire and, if the rounds are being fired in their direction, will see
some indications of where the rounds are hitting. They might also see muzzle flash or smoke, if
they were looking in the right direction when the shots were fired and if the enemy’s position
made those visual indications possible. This suggests that the coach should be sent information
regarding the following items:

e The fire team should have heard gunfire

© The fire team should have heard round impact audio cues

e The visual cues of the enemy’s fires were visible

¢ Round impact visual cues were visible

33



APPENDIX B
SOLDIER VISUALIZATION SYSTEM SIMULATION DESCRIPTION

The following sections summarize the general SVS system functional capabilities.

Synthetic Environment Display

The SVS system displays synthetic environments, or databases, that are in industry
standard OpenFlight format. Additionally, SVS can import SEDRIS and TerraPage format
databases. In databases that have been appropriately constructed and labeled, the SVS can
support dynamic terrain features such as opening and closing doors, shooting out windows and
doors, blowing holes in building structures, making “dings” in building structures with rifle fire,
and shooting out streetlights and eliminating corresponding illumination.

Entities and objects that are not part of the synthetic environment database, such as
humans, tanks, trucks, aircraft, efc., are represented as models that exist in a library so that as
they are instanced by networked or local scenarios, they can be displayed appropriately. A
model of standard objects is provided along with the SVS. This includes a proprietary human
animated character set that represents own and opposing forces, as well as some neutrals
(civilians). A third-party software package — DI-Guy by Boston Dynamics, Inc. Is available as
an option.

Human Behavioral Capabilities

Part of the SVS software is dedicated to simulating the perceptual-motor capabilities of
the human interacting with the virtual environment. These include:

© Movement — This includes walking, running, crawling depending on posture, turning
left/right, walking up/down stairs, climbing over low objects, detecting collisions
with structures and objects, etc. Maximum movement rates are set based on research
on human capabilities. SVS present visual and aural stimuli, but rely on user
capabilities for detection and location (given system performance constraints).

e Health status — Human entities in the SVS can be wounded or killed. Health can be
affected by direct and indirect fire munitions, and by chemical contaminants. Effects
of specific munitions or wounds can be tailored by the user.

e Night vision devices — SVS provides rudimentary NVG (night vision goggle) and
thermal imaging simulation

Weapon Employment
Weapon employment in the SVS is designed to be as natural as possible. The user aims
the intended target in the virtual environment using either the weapons “iron sights” or, in the

case of the Land Warrior configuration of the SVS, through use of the simulated video display
sighting system. To engage the target, the user squeezes the trigger and fires the round, with

B-l



Stimulus Conditions Behavior
Determining enemy location
Reports of enemy activity Changes movement technique
Known or suspected enemy location Reporting enemy location
Returning fire or covering friendly Providing suppressing fires
move
Enemy fire or hostile intent per ROE Firing in self-defense
Preparation for move or assault of Establishing a Base of Fire
objective
Screening movement, obscuring enemy | Use of smoke
observation, signaling
Engagement by enemy forces Reporting to PL (for'SLs)
WIA/KIA
Call for Fire
SITREP
SALUTE Report
ACE Report
When beginning a movement Reporting to SL (for fire team leaders)
When Set following a move
Ordered by superior Movement Techniques
Squad ordered to new location Correct orders to subordinates (voice,
Team assaulting a building radio, and hand and arm signals)
Ordered by superior Providing cover to other elements
Ordered by superior Building breaching
SOP/TTP
Entry point secured Building entry
1TM man enters room Room clearing
Room determined clear by fire team Marking cleared rooms
leader/SL
SL determination based on mission Requesting support from higher
posture ‘
TIP/SOP Rotating fire team responsibilities
Elements fatigue
Consolidation Cross level ammunition, request
resupply, evacuate WIA
Building Cleared Report to higher
Ordered by superior Move to pick-up point
Arrive at pick-up point Report to higher

The following are samples of the situation assessment rules:

Whether a Fire Team/Squad should fire:
¢ Ifthe fire team is taking fire and knows the location of the enemy, then return fire.
¢ Ifthe fire team sees the enemy and the ROE permits it, then fire at the enemy’s last
known location.

19



yey} JO Ieajo | apIisul aou0 snje}s
‘JS 0} Japeay SI WOOL Jey) TS ‘JS 0) spodar
ures} Woy Yodey | —_s[j9} Jepeay wea], Jape] wea],
«dseq Surpuris,,
spuodsar Joquiow
wes} YyyNoy
uonenis
‘woor uy | owure A[pusliy e
popecu st Joquiour yunod WT JO paresyo Zutaq wor
Wed} JSP] JoyJOyM VIM Alpusiy oe ay} UI JUsUIasesuS
SUTULIZ}p 0} pasn Byep jo Ayisuayur
stuoneuuoyur siyy | Suny Awoug © | jo o8paymoury (|
“Ogt] Teor UL
snon3iquie jou st
}I 90uls ‘paurumexa “parea]o
9q 10U [JIM ,.0} Sutoq woor
SI Joplo ay} WoYM,, | Ul UoRENyIS ;eoT}Oe}
atp idaoxa ‘siayjo | UjIM JUa\sIsuOD pue
uy} WarayjIp ou uorysey Aju e
Joplo SI] :3}0N | Ul pue ASopoosesyd
ypol109 Suisn ‘op
soueuuojied 0} yey Joquisu
jo poulog (¢ wea} Surureuter JoquISUI We}
yusyuoo Japig (7 [193 Apoauioo | Surureuser 0} Japs0
SIapio snotaaid 99g udAIs JBPIO. (1 Jopes] Wea] | soaid Jopeay wes],
JOIABYaq,
3091109 B st SULIT spunoi
SIOIARQIg
paimbay aoulesy,
moneULioyay wondii9saq poyadxy
Buetititiey 40 Beq pajaa}aq MOR, yeguey, papiejaq yUIA OLBUaIg



SVS and VOC Information Requirements

Much of the required entity data are available on the network, (e.g., where it is, where it
is headed, its path over time, whether it employs a weapon, is hit by ammunition or collides with
another entity or object in the environment). Any information that is generated by the SVS for
distribution over the network is also available internally and upstream of when this information
would be available from the network. Obtaining data directly from the SVS would increase the
timeliness of the information. What follows are specific information needs generated by the
scenario to help define the VOC requirements. This section describes alternatives by which the
SVS can provide this information.

Techniques for Identifying Objects Suitable for Cover

The issue here is operationally defining what affords cover in the virtual environment.
These include environmental features such as trees and gullies, structural features such as
buildings, light posts, and curbs, and entities such as vehicles, barrels, and furniture.

Options for identifying these within SVS include:

e Tagging structures in underlying database: Terrain development tools such as
MultiGen can be used to tag polygonal (terrain) structures with comments identifying
them as providing cover, along with a location and volume. The SVS would locate
and maintain a list of these when loading the database, and a search process would be
undertaken by the SVS to identify them at defined choice points. While
straightforward, this would be labor intensive.

e Pre-identifying terrain features with SVS zones: The SVS uses the concept of zones
to define volumetric regions that are used to identify chemical or radiological
contamination areas, or areas in which specified sounds will be heard. These are
rectangular, cylindrical, or spherical in shape. Zones could be overlain on defined
cover areas and preprocessed and stored as a scenario file using the SVS authoring
capability. Available as broadcast data and as non-standard persistent objects, this
structure is also used for interoperability with ModSAF and OneSAF. This
application is also labor intensive.

e Use broadcast objects (entities) as cover: This is only a subset of possible cover in
addition to terrain features. It is available as broadcast data coming into SVS and is
currently available.

Recognizing Soldier Orders and Reports

Options for identifying Soldier orders and reports within the SVS include:

e Menus - The user can select screen-presented menu options using a weapon-mounted
two axis controller. While this is an artificial constraint and all options must be pre-
defined, it is the easiest option to implement. The resulting selections will then be
available as internal data available to be sent to the VOC if necessary.

40



€I-d

Uy Hodar oy} sep, ay} aztus0s0y (Z
uoneuuoyul uoned0] INesse
uoneorpul | jes sem poder ay} ayemooe WIM ye Jos uayM Tq
yuas aSessoy] | yey} OzTUSONDy (1 ‘Id 0} suoda1 Tg 0} syodai Js pit
Apear Ayjenjoe
ST urea} USA “Td 0} Sunsodar
‘1QUUULI 199.1109 ay} ‘TS 0} asuodsaz
ur ‘uorysey AjouIN Apear ul UoRoR sty?
B UI TS 0} suodar are £ay} yey} TS | 19931} 0} [Qe 9q
Jopea] ueay | 0} suodei g uray, | jsnuI uoe|NUTIS
sjtejap Joy
QAOUL FEY} 0} JOJOI ‘yurod
Os sAOU Y UTea | Anua Surpying
at 0} feotuapr uoneo0] | 0} aAou! S,gq Wea],
aq Pinoys uonoe Axwua Burpying 49A09 0} spenbs
quoulaaoul sry] | 0} SaAoul g Wea], Joyo SIOp10 Tq
aoueuLojiad Jo
poud afquidooor
ay) UTM
soueulojiod uaye} sem uoT}OR
JO potiog | yey} eztudoooy (¢
qusuIsAour
aaoge jo uoreunsap
S9}0U Jal[Iea 99g JRALLe JS aziudooay (7
“Piiom payernuns
9} UT Jaquiour
ures} UeUMY
Joyjoue MOT]OY Jonuoo
SIOIARQIG
poimbay aoulely
WonenLioyuy gondiiasaq payaadxq
syusuIMI0Z) 40 eyEg pa799}0q MOF] pose, payiejaq | = yuaAq OLIeUIIg





vi-d

iiLNO
OVUA S124 pue
WOO! OUI apeuaIs
Sasso} JOIp[OS
asreyo
au) sally pue
Surpying Jo yrod
Axjue ye odreyo
sooetd Jopeay,
urea} yJnessy
“yoda
“oray payeodar Ts hq parad3in 0q
aq JOU [JIM satu 0} poau []IM uoTe
pernba eyep pardiios sy,
pue wistueysou | Apeol si 1 sjeusis
uonsa}ap oy} V weal Jaye “peasoid 0}
Os ‘siay}o [Je ayI] | ‘hem Jodoid ayy ur Burpying | uorsstued saaid
paTpuey oq pynom | Jopeoy urea} ay} 0} | = ay} Joy 0} UTR} pure yiodai Ts
adessoul sIY] | J9pio ay} SaaI3 JG | yNEsse sIap1o Jg | sadpajmouyoe Tq
aouRULojiad Jo
pouied ajqeydaooer
ou UTM
soueuliojied | ues sem yoda oy)
JO potted | yey} aztuBoooy (p
ats oy
Wody JUes aBessoul | yuos sem Yodal ay)
Ul :O], oztUso9y | oYM oztUBod0y (¢
asjo SUTyJOWIOS yodai
Jo uontsod ay} Jo yuayU0D
SIOLABYIG
paimbay aaulety,
WoHeULIOyUy uondiasaq paysadxq
syusmMI0;) 40 Beg paj29}9q MOY] pose, payteyaq | UaAq OLenaIS



Preliminary System Requirements

This section contains a partial set of requirements for the VOC and for the simulation.
The simulation requirements have been collected from various portions of this report.

The VOC shall be able to process the situation assessment and action execution rules
contained in Appendix E. The VOC shall support the instructional strategies described in the
VOC Concept of Operations section of this report. Table 8 shows the message type, the data
needed, and the triggering conditions for the VOC.

Table 8
Simulation Message Descriptions, Data Needed, and Trigger Conditions for the VOC
Message Type Data Needed Trigger Conditions
Cover object information Type of object, Scenario initialization
What cover it provides,
Location,
Size
World Object information | Type of object, Scenario initialization,
Location of object interesting objects only

Room interior objects

Location of object,
Room region dimensions
visually blocked by object

Whenever the user can see
into the room and see the
objects in the room

Friendly unit status

Unit designator,

Unit size,

Location,

Fatigue status,
Operational Status,
Ammunition remaining

Scenario initialization and
whenever any of the data
change

Enemy disposition

Size of enemy unit,
Status of enemy unit,
Location,

Posture

Whenever this information
is told to user and only to
the degree it is told to user
via intelligence.

Enemy to friendly visibility

Friendly unit designator,
Enemy unit designator,
Line of sight indication,
Other visibility

Sent whenever friendly unit
gains information about
what enemy could know
about friendly, but only
what friendly could know

Friendly to enemy visibility

Friendly unit designator,
Enemy unit designator,
Line of sight indication,
Other visibility

Sent whenever friendly unit
gains information on enemy
unit

27


platoon is going to clear the suspected building. His squad is told to establish a base of fire. The
SL directs his men to occupy positions to provide suppressive fire. The voice of the PL tells the
SL that he should have taken a better look at the area and selected positions that allowed them to
isolate the building and cover the window where the sniper was seen. The SL directs the Alpha
team leader to a new position, and orders Bravo team to cover Alpha team’s movement.

The squad hears on the platoon net that another squad is getting into position on the other
side of the building. The Alpha team leader sees an enemy Soldier in a different building. He
reports to his SL that he sees enemy movement, and the SL sends the PL a contact report. The
VOC recognizes that an important piece of information was not in the SL's report. The SL did
not give the direction of movement of the enemy. This is a crucial piece of information since the
enemy was moving in the direction of the building the platoon is going to clear. The VOC
decides to pause the simulation while each Soldier is given a situational awareness assessment.
Each Soldier is shown a map of the area and is asked to indicate on the map where friendly units
are, where enemy units are, where the most vulnerable and strongest positions are for both sides.
After this brief individual situational awareness assessment the VOC sees that the SL did not
realize that a given sector was vulnerable, whereas the Alpha team leader did. The VOC decides
not to tell the squad about this discrepancy, but saves this information for the AAR. The VOC
resumes the training exercise after everyone has finished the situational awareness assessment.

Next, the squad hears over the radio that that another squad has breached the building and
has secured a foothold. The PL orders the 1st squad to enter the building to clear it. The SL
reminds his team that they will be using the strong wall as opposed to the opposing corners
method of placing men into position in rooms. Once they have cleared a room, the SL makes an
error of not correctly marking all the exits, and the VOC reminds the SL to do this correctly.

The fire team leaders are occasionally reminded to not to stop and shoot while they are standing
in a doorway.

At the end of this 15-minute exercise, the VOC conducts an AAR. The VOC begins the
AAR and focuses on the team’s lack of shared situational awareness. All the Soldiers are asked
to write a few sentences summarizing what they think happened. After everyone has written
their own explanation the VOC shares what it thinks caused the problem (the fire team failing to
report that enemy were moving towards the building.) The Soldiers are then able to discuss this
problem. The Soldiers’ explanations and conversations are recorded, but not analyzed by the
VOC. The SL is asked by the VOC to explain why he chose the sequence of rooms to clear that
he did. The SL is presented with a system of menus to help elicit the reasons for his choices.
The SL is also told that he should swap out his lead teams more often. The Soldiers can decide
to do another training exercise and the VOC will select another scenario for them.

METHOD

Focus was placed on squads and teams, as opposed to larger units such as platoons or
companies. Furthermore, we focused on building-clearing scenarios in urban operations. We
adapted Battle Drill 6 from FM 7-8 (1992) for our purposes.



¢ Use vector information between user and friendly forces and enemy forces to provide
a gross level of detailed information. This would not be difficult, but additional
integration effort would be required.

¢ User can know he is being fired upon by the sound of shots and muzzle flashes.
These data are not exact, but provide adequate information. This is independent of
the VOC knowing that a friendly is being shot at.

World Object Names

All objects in the simulated world that might be referenced in instructional feedback (e.g.,
large concrete planters, hills, buildings, rooms, windows) need human readable names that must
be accessible to the VOC, either by request or provided as data in any messages sent from the
simulation to the coach that relate to the entity. Techniques for tagging non-entity structures or
objects, similar to those described for identifying terrain or objects as capable of providing cover
can be used for ascribing nominal identifiers to objects of interest. As noted previously, the
tagging process would be laborious, but the SVS modifications could be accomplished.

Soldier Route Monitoring

Using pre-stored routes, currently used to control SAF, the SVS can watch an entity
move and determine if the movement was reasonable (i.e., it more or less followed the path that
was pre-stored), and then forward that information to the VOC.

The SVS currently provides a mechanism for laying down paths as part of the process of
programming CGF travel routes. The CGF subsequently follow these paths when commanded to
execute movement behaviors. Thus the line segments concatenated to form an overall path are
converted to linear equations against which CGF position is compared over time to perform path
corrections to keep it on route. This approach can be used to monitor user movement along a
predefined path and assess how well he is keeping to the defined route. A defined error metric
could be continuously passed to the VOC for assessment and correction if deemed necessary.
This error data would be internal to the SVS only. Since the mechanism to generate the error
measure is in place, modifying the SVS to perform this feature would be very simple. Detecting
that a Soldier arrived someplace requires a definition of where the place is (coordinates) and a
radius threshold for arrival at that point.

Identifying Where a Soldier Looked

The determination of line of sight and visual angle is straightforward in the SVS, but
definition of a room is problematic. We have discussed the possibility of using an alternate
representation of the environment, a compact terrain database (CTDB) format used by OneSAF
for reasoning purposes. It has terrain feature data, plus knowledge of entities and of buildings by
special representations called multiple elevation structures (MESs). This representation could be
used for many of the terrain, cover, concealment, and building reasoning tasks described above.

42



ExpertTrain applications that employ intelligent tutoring technologies to provide adaptive
training within scenario-based environments (see Appendix A and McCarthy, Wayne, & Morris,
2001). The ongoing intelligent tutor developments have enhanced the tutoring capabilities of
embedded “virtual coaches.” Furthermore, there is an increasing body of evidence that these
tutoring systems produce significant improvements in instructional effectiveness and efficiency
(e.g., Wisher, McPherson, Thornton, & Dees, 2001).

This report describes the efforts and results of examining the feasibility of creating a
VOC to observe and critique Soldiers’ performance as they are engaged in simulated small-unit,
dismounted Infantry training using the Soldier Visualization System (SVS) currently in use at
Fort Benning, Georgia (see Appendix B). The successful integration of VOC and SVS
technologies will mean that the training value of the simulation-based exercises will not be
completely dependent on the military expertise of a human O/C. The next section illustrates a
hypothetical application of the VOC training technology in some future training situation.

Narrative of a Future VOC Training Application

2LT Thomas is a new Platoon Leader (PL) in 2" Platoon, A Company, 2TM Battalion,
502" Infantry. 2LT Thomas has several new Squad Leaders (SL) in his platoon. 2LT Thomas
decides to take advantage of a new training opportunity at his base. He decides to send one of his
SL and two of his fire team leaders to a virtual training facility. 2LT Thomas suggests that the
squad conduct an exercise. One SL and two fire team leaders prepare to practice maintaining
their situational awareness during a simulated exercise. One of Soldiers puts on his virtual realty
helmet and steps into the system, while the two other men sit down at personal computers, The
Soldiers log in, and the VOC retrieves their individual learning profiles. The VOC selects the
best scenario for the Soldiers. The scenario selected is a building-clearing scenario that focuses
on situational awareness and that sharpens room clearing tactical skills. The VOC asks the
Soldiers if they want to do this exercise with other Soldiers from others units or use computer-
generated forces for their other team members. The Soldiers choose to work with the computer-

generated forces first because they are just getting used to working together as a squad.

The squad receives a mission briefing stating that they are to conduct a dismounted
patrol. The scenario places the squad on the streets of Baghdad in the early days after its capture.
After reviewing their ROE, the men see that they are actually in a street in Baghdad. They are
part of a platoon, but the only Soldiers that are visible right now are the nine members of this
squad. The other squads consist of computer-generated forces.

The SL issues an order to use bounding overwatch and to proceed up both sides of the
street. The VOC notes that the SL has used the correct formation and movement technique.
After a few minutes, a shot rings out. While most of the men immediately move to cover, the
VOC notes that the Alpha team leader took cover behind several 55-gallon drums. The voice of
the PL plays in the team leader’s headset telling him to seek real cover, not just concealment.
Meanwhile, the SL is trying to determine if anyone knows where the sniper is, and verify that
there were no casualties. One of the squad members says that he saw a sniper in the second floor
window of a building in front of them. The SL reports to the PL and receives orders that the



‘et inal me interval
‘var which the

sveraging ato be done
othe default “recent
‘hoe pant intervalTM

Tentty at roonde that

4 the total number of
ounde above 8 "oo
many threshold?

ent past
arte Yeo

Declare the average
to be“Too High"

No

De considered?

Figure 6. Engagement detection
algorithm flowchart

The first step in the algorithm is to establish the time interval over which we wish to
compute the average firing rate. This time interval should be an adjustable parameter to allow the
algorithm to be tuned based on how it performs in training exercises. Once this is done, the
algorithm counts all the rounds that have passed through the Alpha team’s sphere, using fuzzy
logic to handle near misses. If this count is zero, then it may be appropriate to increase the time
interval and count the rounds again. This “increase-interval-and-count-again” process could be
repeated until the time interval has reached a maximum value. This maximum time interval
could represent the point of time in the past before which we do not care if the enemy has fired at
Alpha team’s position. This point of time could be when Alpha team arrived at their current
location. Assuming the rounds count is not zero, the algorithm would next determine if the total
number of rounds fired is too high. This parameter simplifies the algorithm’s behavior in the face
of massive enemy fire, for example, if the enemy has poured automatic weapons’ fire into Alpha
team’s position. If this simple test fails, then the algorithm can examine the average rate of fire
over the time interval in use to see if the frequency of fires is high enough to justify returning
fire.

Have We Seen the Enemy?

Deciding whether a Soldier has seen something in a simulation environment is
problematic. As in the case of the visual cues associated with rounds impacting, the same

36



¢ What is the situation category (e.g., enemy action)?
© What is the specific situation event (e.g., enemy fired at friendly unit)?
© Who is affected (i.e., what friendly unit or Soldiers)?

The processing of the “Then” clause is also split into two steps. Initially, situation
assessment rules will only have a description of "what" the expected action is supposed to be.
"How" the expected action should be executed will be handled separately. Thus, the "Then"
clauses, except in the simplest situation assessment rules, will be types or categories of actions
(e.g., take immediate cover). The how-to details associated with a "Then" clause's expected
action will be processed separately. In the example we have been using, where the “Then”
clause is “take immediate cover,” we would consider the rules regarding “taking cover” (e.g.,
seek cover that provides adequate protection).

The benefit of this multi-phased approach is that one rule serves as an activation trigger
for other rules. This can serve to manage the growth of the problem space that must be
represented in the knowledge base. We will return to this idea in a following section.

Using our three-step process, we consider a situation and then the rule set. A fire team on
patrol has been engaged by an enemy element and has taken cover. While behind cover, the fire
team leader can see an enemy combatant and has a good line of fire. The simulation has already
informed the VOC about the engagement, the move to cover, and the fact that the enemy
combatant is visible to the friendly Soldiers. All this information has been recorded in the
VOC's internal situation assessment representation. This representation can be thought of as a
blackboard on which all the aspects of the situation are captured for use by the rules that
determine what a Soldier should be doing at any point in time. The blackboard is also where the
specific elements of the rules of engagement will be stored. These items will also be used
whenever rules are evaluated that are sensitive to ROE issues.

One of the features of the blackboard is to support triggering or activation of the rule or
tules as the situation unfolds. This triggering process handles the first two questions in our
three-part process. When the VOC is informed that the enemy engaged a fire team, the
information allows the VOC to recognize that the category of action is an enemy action. This
allows the rules associated with enemy actions to be activated. The second piece of information
provided by the simulation lets the VOC know that the specific type of enemy action is an
engagement. This further pares down the number of rules that must be examined in response to
the situation. For this discussion the fire team has already taking cover after contacting an
enemy element. This represents another piece of information sent from the simulation
environment. Consider the following collection of rules that relate to enemy actions:

¢ Ifthe fire team is taking fire and knows the location of the enemy, then return fire

¢ Ifthe fire team has recently seen an enemy and the element of surprise has already

been lost and the ROE allows it, then fire at the enemy’s last known location

¢ Ifthe enemy does not know the fire team’s position and stealth is important

¢ If ina hostile environment and the ROE is non-restrictive, the enemy location is

known, and the fire team has been fired upon, then return fire

15



¢ What should the specific content of the intervention be?

As we discuss these four questions, we will address the capabilities and interactions of
the two classes of coaches. As we have noted, the instructional processing is initiated when the
VOC's comparative processing reaches an evaluative conclusion. That conclusion provides the
data needed to evaluate the first question, such as what subject matter item is involved (usually
identified by learning objective) and either the class of problem (which bug type or category has
been identified) or an indication that the result was a target condition. This information is used
to classify the nature of the instructional opportunity. Ignoring the possibility of providing
positive feedback, we will use the data from the evaluation to classify the problems identified by
the coach’s performance evaluation as a minor problem, a major problem, or a catastrophe. We
have defined a minor problem as one that does not adversely affect the successful completion of
the mission during the training exercise. A major problem is one that might interfere with the
training goals of the scenario, thus jeopardizing completion of the tactical mission. This should
be corrected with an immediate instructional intervention. A catastrophic problem is defined as
something that requires restarting the exercise, such as the death of the SL.

Our second question dealt with choosing which instructional opportunity to pursue. A
likely event in any real world training exercise is that several instructional opportunities may
arise all at once. The following rules will be used to select among the possibilities:

e Polarity of Opportunity
- If the opportunity is for positive feedback, then the instructional weight of the
opportunity will be decreased. Otherwise, the instructional weight will be
increased.

e Instructional Recency
-If instruction of any sort has been delivered recently, then all instructional
weights will be decreased.
-If instruction on this topic has been delivered recently, then the instructional
weight associated with that opportunity would be decreased a lot.
-If no instruction has been delivered recently, then all instructional weights will
be increased.
-If no instruction on this topic has been delivered recently, then the instructional
weight associated with that opportunity would be increased a lot.

e Learning Objective Priority
-The instructional weight associated with a given instructional opportunity will be
adjusted in proportion to the priority assigned to that objective for the current
instructional evolution. Reportable objectives are higher priority than other
objectives.

e Granularity Of Action
-In the case of a negative opportunity, those closer to the smallest atomic actions
for which coaching is possible will be weighted heavier than those farther away
from atomic actions. Conversely, for positive opportunities, higher nodes are
weighted heavier than lower nodes

10



THE VIRTUAL OBSER VER/CONTROLLER (VOC): AUTOMATED INTELLIGENT
COACHING IN DISMOUNTED WARRIOR SIMULATIONS

CONTENTS

INTRODUCTION ..........:cesessssssscsesscsessssesscaeseseseesescassseneescncsuesonesscesceaeesvesnecssssouanenenseceasensoeonencees 1

Statement of the Problem
Narrative of a Future VOC Training Application ..

Preliminary Cognitive Task Analysis
Scenario Analysis
Feasibility and Requirements Analysis

System Description
SVS Simulation Description
VOC Concept of Operation....
Automated Measures of Performance and Effectivenes:
Simulation Modifications and Instrumentation..

DISCUSSION OF AUTOMATING OBSERVATIONS AND CONTROL.........sesseessesessseesiee 31

Automated Situation Assessmen
Should the Fire Team Shoot?.....

DISCUSSION OF DETAILS OF THE SVS SIMULATION ENVIRONMENT

SVS Data Architecture
SVS and VOC Information Requirements..
Techniques for Identifying Objects Suitable for Cover
Recognizing Soldier Orders and Reports ..
Sound Cues.
Tracking Human Soldier an
Enemy Engages and Misses
World Object Names.......
Soldier Route Monitoring .........
Identify Where a Soldier Looke:

SUMMARY ..



provides poor cover or no cover.

poor cover and compromises your unit.
L- This position has no cover.

1 and 2" Squad Provide Accurate
Fires During 3TM Squads Movement

Target: Squads fire on building in
accordance with the ROE

Bug: Squads do not fire on building.

H- You need to provide accurate fires to
cover the other squad’s movement.

L — Fire on the building, what are you
waiting for.

Bug: Squad has poorly placed fires with
no effect on the enemy.

H- Your fire has no effect on the enemy
position in the building.
L- What are you firing at?

1" SL Reports to PL

Target: Sends report to PL with adequate
and accurate information within the
acceptable period of performance

Bug: Does not send report.

H- You need to report all information to
higher.
L- Send that report to the PL now!

Bug: Sends report late.

H- All reports is sent as soon as possible.
L— Your report is late!

Bug: Report is not accurate.

H- Check your reports for accuracy

| before you send them.

L- Check your report, it doesn’t look
tight.

Bug: Report is sent to wrong person.

H- Verify the recipient of the report
before you send it.
L— Whom did you send that to?

E-18



Fire Team 1A @ @

Building

eae
@ Fire Team 2A
_ Fire Tear 1B

Fire Team 18 moves to corner of
Building A in preparation for squad
2to eter bullding. Fire Ream 1B
will continue tp secure the
perimeter with assistance of
squad's heavy weapons.

1 Squad — Coordinates with weapons squad and continues to secure the perimeter of the
building. It is imperative that the building remains isolated during clearance.



If the fire team has recently seen an enemy and the element of surprise has already

been lost and the ROE allow it, then fire at the enemy’s last known location.

e Ifthe enemy does not know the fire team’s position and stealth is important and the
enemy is adequately suppressed, then do not fire.

e Ifina hostile environment and the ROE are non-restrictive and the enemy location is

known with confidence and the fire team has been fired upon, then return fire.

Whether a Fire Team/Squad should take cover:
e Ifthe fire team has received fire, then they should take immediate cover.
e Ifthe fire team is moving by bounding overwatch, then each move should be from
one covered position to the next.
If the threat of enemy artillery or mortars is imminent, then they should take cover.
If the threat of enemy observation is high, then they should take cover.
If the team receives an order from higher to take cover, then they should take cover.

A Fire Team/Squad should report to higher:

If the team comes in contact with the enemy. (SALUTE Report)
If the team is engaged by enemy forces. (Contact Report)

If the team has a WIA/KIA. (Red Report)

If there is a requirement for a Call for Fire.

If there is a significant change in the situation. (SITREP)

At consolidation. (ACE Report)

When requested by higher.

At certain times (0600, 1800, etc) as stated in the unit SOP.

A Fire Team/Squad should use smoke:
e Ifthe team needs to obscure their movement from the enemy.
e Ifthe team needs to mark their location.
e Ifthe team needs to mark an enemy location.
e Ifthe team receives an order from the SL to use smoke.

Fire Teams/Squad should not use smoke if:
e Wind speed and direction are not favorable.
e¢ Smoke is not readily available, or is not available in the correct color.
e ROE prohibit use of smoke.

A Fire Team/Squad should call for mortars or artillery:
e If other weapon systems are not effective against the enemy.
e Ifthe team/squad receives an order from higher to engage with mortars or artillery.

In order to request mortars or artillery the following conditions need to be met:
e There must be a supporting mortar or artillery unit able to range the target.
e The team must have communication with the Forward Observer or firing unit.
e ROE must allow use of mortars or artillery.


is information about the Soldier’s actions. The analysis here is focused on determining whether
Soldiers’ actions can be recognized and extracted when needed. In this analysis and in the rule
analysis we just described, we were trying to specify the details of various kinds of messages that
the simulated environment must trigger and send to the VOC. Trying to describe exactly when,
where, how, and with what data a message is to be triggered, from a concrete system design
perspective, uncovers all of the cases in which something is easy to say, but difficult to automate
in software. As an example of this unexpected complexity, consider a rule that is trying to
evaluate a Soldier’s firing position. To do this, it must be possible to know when the Soldier has
arrived at their intended destination. If the Soldier runs from one covered position to another in
several short bursts, then how long should the system wait before deciding that the Soldier has
arrived. Should the system wait until the Soldier actually fires his weapon? What if he is firing
along the way to keep the enemy suppressed? Perhaps the system should wait a certain period of
time after the Soldier’s movement has ceased, but how long should that be? The details of this
analysis produced very concrete data and message triggering requirements and allowed very
specific examination of the feasibility of extracting the needed information.

The third item, situation trigger analysis, was heavily dependent on the preceding two
analyses. However, this step was more about making sure that there was some tactically
believable way, in the context of a training scenario, to create a situation that required each and
every behavior we wished to train. The capabilities of the simulated environment were
considered at this point. If the simulated environment could not support what was needed with
its current capabilities, then the technical aspects of extending the simulated environment’s
capabilities were explored before we answered the feasibility question.

Our fourth item, identifying the qualitative and quantitative measures that the VOC
should use provided the basis for a VOC that would function like an expert human O/C.
Specifically, these measures included situation assessment capabilities, behavioral evaluation
mechanisms, and instructional intervention strategies. However, these measures are only
potential requirements for the VOC and were examined for feasibility by considering how easy
or how hard it would be to model them. This effort amounted to developing the automated
measures of effectiveness and measures of performance suitable for use in the VOC. We
examined each of the situation assessment measures discussed above and identified those cases
in which the effort required to automate them would be significant when compared to the value
provided to the training system by that capability. This was an important part of the analysis
because we were building a training system and wanted to focus effort and resources where we
would get the best value from an instructional standpoint.

FINDINGS

This section of the report presents the results of our investigations. It is broken down into
two subsections that include a description of the system and various components, and
preliminary sets of requirements for the VOC and the SVS. The section concludes with a brief
discussion of future steps that seem reasonable based on our findings.



(e.g., Anderson & Pelletier, 1991). Because humans will never be perfect, and because humans
sometimes take correct actions for the wrong reasons, we must use some method of dealing with
the uncertainty about which skills a Soldier has mastered. One common way of dealing with this
is Corbett and Anderson’s (1995) Knowledge Tracing.

The second source of uncertainty causes the “credit-blame assignment problem.” If you
have a reasonably complicated task, then there will be situations when there are two plausible
explanations for a Soldier’s action. Which action do you assume they took? For instance, if you
see a Soldier moving backwards, is it because he has decided to retreat, or is he trying to
reposition to outflank an enemy? If retreating is appropriate at this point, do you credit the
retreating action, or do you treat the action as a flanking maneuver? Martin and VanLehn
(1995) offered an elegant method for making a principled guess as to an interpretation of each
action that takes into account the prior probability that a student would take each action.

. According to Russell and Norvig (2003), Bayesian networks are now acknowledged to be the

best way to model uncertainty, replacing a plethora of more ad hoc techniques used during the
past 30 years. Martin and VanLehn (1995) have already shown how to use Bayesian Networks
for this purpose in the tutoring context, and thus present a mathematically principled and
computationally tractable method for the VOC to use in handling action uncertainty.

Simulation Modifications and Instrumentation

Receiving the Soldier’s actions from the simulation amounts to a series of messages
flowing into the VOC. The messages that provide this information are grouped into two broad

" categories: expectation messages and action messages. These mirror the two kinds of

information that the VOC needs. The expected and actual information, and the supporting
messages, will include the following: (a) a list of every discrete tactical situation that needs to be
identified to the tutoring engine, the necessary expectation message, and the data that the
message needs to provide to adequately characterize the context, (b) a list of every discrete
action that the Soldier (or unit) can take, the necessary action message, and the data needed to
help characterize the action, and (c) the specific details, rules, and data associated with both the
student actions and the situation contexts in order to identify message trigger conditions. Table 7
provides a summary of the information needed from the simulation. A more complete list is
found in the

Preliminary System Requirements section on the following page.

Table 7
Simulation Information Needed From SVS to Make Assessments
Category Specific Items
User Action Fires a weapon
. Throws a grenade
Moves
Orders subordinate

25



pepseu jou usyM s}ooys
issng

SOAOUI

V wea], ayy sary aatssoiddns Jo JoAoy
paulo oy} sepraold gq Wes] 30318],

sIOA0D g Wea],
SAO] Y Wea] SITY,

Adopoaseryd yoou09uy

uonisod ut jou uaym syodey

yodar 2387]

isang

‘TS oy) 0) yoda Ajourn B sopraoid

Jopea’] Wea] oy) Julod Anus Surpying sy)
ye uontsod (payoujs) 1odoid ay) ut ore y
urea], JO SIaqUIOU! aU} [JB 90UG :JadaB],

uonsod ut oe
Aayy yey} Tg 0} suodal Jopeay y urea]

SIOpJO INOYPIM SOAO|T
uoT200] SUOIM 0} 0D

QAOUI UIB9q 0} MOTS

ising

yUSUTBS9U05

pure 9,09 Jadoid pue sonbruyse}
yuouzaaow Jedoid Suisn ‘slopio usAIs
Bureq Joye ATWO SaAOW YW Weal 7729818],

yutod Anus 0} soaoy] Y Weal,

I9A09 JOp1O JOU S90]

Jopio 9AOUI Ul UOReUNsep BuIA,

SAOUI JapsO 0} MOTS

isang

spenbs

Jamo Aq paptaoid useq sey ary SuLIA0D
20u0 Apduiosd siepio sansst Ts :3ad18],

IOA09 0}
q wea] pue cou! 0} y wea] sIapi0 Js

«AO ‘10804, Td

ABojoaseryd yoa1100u]
asuodsar MO|S
ising

uoydiissag Sng pus josie],

SAOIABYIG daUIeI], poyoodxy paiejaq

yUaAg OLIEUaIG



61-d

SIOpI0 Td
“THEI SJIOM ®B
lM Woo! paresyo
ay} ‘payreul sIapso
JO ‘syzeul TS
r Surpring
9y} 0VUT SUTAOUI
penbs Sunjnesse
ay} JO urea) ary
ye oy) woddns
0} 2AOUI 0} spenbs
JoyJO BuLIap10
‘spuodsai Tq
youueUul
Ajaurn ut ‘pareayo
SI Wood oy} yey} pazesjo

"Jd 24} 0) syodar SI WOO yey}
Ap921109 1S ‘Jd 01 suodai Tg
soueulojiod
Jo poutog (7
au0z,,
portoeds v opisut Jouueur
st Ayryuo ue yey) : Ajourny e ut ‘pareayo
uoneorpul sijtoeds payeorpur st wWoOo1 woo SI WOOL oY} Jey}
B aptaoid ysnuz | 9y} apisut souasaid | ay} optsut st TS ay} PIO} Butoq Joye
uolejnuns ou], Jopesy weezy | yey) azrUBoo0y (1 ‘wool s19}U9 TS Wool sia} TS
“apeul
uaeq sey Auroua
WIM Jord Wool ay} Jo
SIOIARqIg
peambeay aaulely
HoneULIOyUy aondiiosaq pajoodxy
syustIMI0Z 40 Beg p2399}0q MOF] yosaey, papiejaqg JUdAW OLeNa9g



System Description
In its simplest form, the model of the system is depicted in Figure 1. The simulation

presents an interface to the Soldiers and is connected via a messaging protocol to the VOC. All
communication between the VOC and the Soldier is handled by the simulation interface.

ee Information

—_ ~e Mastery Evidence,
L— ~al Instructional History
SVS Simulation -- — — | voc
| ly.
Cc ‘Student Actions
{ Performance
eo > Archive

‘Scenario Instructional interventions,
Definition 9, feedback

Figure 1. High-level system diagram.

The prototype system is focused on Soldier teams comprised of a SL and two fire team
leaders from a dismounted Infantry platoon. A PL and platoon sergeant (PSG) will be simulated
via scripted and triggered voice communications sent from the simulation environment to the
squads. The SLs and fire team leaders will engage in a building-clearing exercise hosted by the
SVS. The team members will be computer-generated forces (CGF).

The system will incorporate a VOC comprised of four separate modules: a module for
each player, i.e., the SL, fire team leaders, and a team coach that will be focused on monitoring
the performance of the fire team as a unit. The individual coaches will be closely monitoring
each individual’s behaviors and providing immediate feedback, when instructionally appropriate.
In addition to providing feedback to the individual team member, these coaches will forward
performance information to the team coach. The team coach will be focused more on diagnosing
patterns of behavior based on the information received from the individual coaches and will
provide feedback about the team to the SL.

SVS Simulation Description

The SVSTM Dismounted Infantry (DI) Immersive simulation system is a first-person
human-in-the-loop tactical training system (also see Appendix B). The term “tactical” is used to
intentionally differentiate SVS from other existing marksmanship-type trainers that do not
support unrestricted user movement through the environment. Using United States Department
of Defense standards for synthetic environments (databases) and networking protocols
(Distributed Interactive Simulation (DIS) and High Level Architecture (HLA)), the SVS DI
supports individual and collective level training. Figure 2 illustrates the SVS architecture.



sd

ABojoasergd 1991109 TIM
pue Ayduroid syodai [g ayy josey,

QAOU! 0} JapIO
au) Buspajmouyoe Tq 0} suodal 7S

*I9A09 OU IO J9A09 100d

sapraoid yey) uoytsod & 0} soaow penbs

*(Bu0] 00}

posodxe sAejs) A[MO|Ss 00} soaoul penbs

isang
(ases

SI} Ul Yo}em9A0 Burpunog) sanbruysa}
JUSUISAOU 3091109 Puy JUSU]RADU0D pu

39409 Jadoid Sutsn woes] 3991109 0}

Apyomb saaouw penbs :ya818 1, Areurmng

spenbs puc

pur 1 Aq papraoad st aij Butaa09 aauo
uorsod ynesse 0} penbs saaow JS ,,¢
oraeqeq Arguing

SUIPying atp yynesse
0} oredaid 0) uoNed0] Aqua payeusisap
B 0} AOU 0} penbs ,,€ S1API0 Td

penbs | ¢ jo Teauire ayy oy toud
payyy aze spenbs .,2 pue ,,| Woy Soll

EsnOnOVAEHpENDS

uonisod ynesse
soyoeoadde penbs ,,¢ se saaty yIys
pue yt] o1 penbs ,,2 Pue 1 S19PIO ‘Id

“AudaUd dt) UO 1DaIja

ou Yjlm Samy paseid Aj1ood sey penbg
“Surpying Uo ally jou Op spenbs

issng

AOU ey) UM aoueploooe

ut Surpying uo aly spenbg :3a818 |,

quowioaou spenbs ,,¢ 3utinp
saalj ayemaov apiaoid penbs ,,Z pur 1

: UOLBI0]
\nesse ay} 0) soaour penbs ,,¢ se sally
aatssaiddns aptaoid spenbs puc PUB <1

ayendoidde oq pmom sjeusts

suue pue puey UoyM dd10A sasn 1S
ng

“syeusis une 2 puey

1O Olpel ‘s010a Buisn syiodal 1g 339818]

uondiisseg sng pue yasiey

SIOIABYIG saute], pajsodxq papiejaq

yuaAy O11BUadg



SUONOE BJEUIPIOOD 0} UIed) Ped]
YM Suryiom suloos s1e9]d wes} [11]

“AYLMOaS

SuloZuo amsut 0} paaoul AJaye~dodde
are spenbs Joy}o ay} 290 BuTpying ay)
OU] Wed} SIT [Ie a} Slapio ‘Tg :ja81e |,

ul001
}XOU 94} 0} souRNUS ay} 0} pue SuIpying
SY} OVI WRI} oI [Tey Slapio TS

“Surpying ayy Butreajo
aNUNUOD 0} ‘Tg SuNNnesse siopso Tq

“TIS FIOM & YM wor
pareg]o a} ‘poyxseul SIapio 10 ‘seul TS

Suipying

ay} OjU! ZulAow penbs Sunjnesse

dy} JO wea} any [1ex} sy) Woddns 0} saour
0} spenbs Jayj}0 Suuapio ‘spuodsai Tg

uondiidseg Sng pus jos18 7,

SIOTABYAG solely], poyodx gy popiejeq

wUaAY OLeUaIG



Fete @® Q@H==

Point of Entry
Building A
+. ®
‘W. _ Sippresgive Fire
“~@
Fire Team 1B.

24 Squad shifts fire away from the point of
entry, 3 Squad begins to bound to move irto
position to enter building.

PSG — Orders remaining squad to move (if required).

PL — Assesses situation and provides subordinates with his assessment.

1 Squad — Continues to provide suppressive fires and adjust position as required.
PL — Determines the entry point for the building.

Cover Cover

Fire Team 14 @ r Fire Team 26
Butding A

(Gir ten

Cover
Cover

Bi Fire Team 24
Fire Team 18

Fire Team 38 tells PLthathe
hhas secured foothold. Fire
‘Team 3A moves up to point of
erty

C-3



Fire Team 1A @ e ire Team 28

EJ

Bullding

PR" Qkhie team 24)

Fire Tearn 1B

Fire Team 2A into building as Fire
Team 2B and 1B covers
movemert.

2" and 3°¢ Squads — Continue clearing the building and upon clearance of the building inform
the PL that the building is now clear.

1 Squad — Continues to secure the perimeter of the building with the assistance of the
weapons squad.

Cc-9



Pleban and Salvetti (2001) described a method of online situational awareness assessment
that allows the system to assess whether a Soldier knows where the enemy and friendly units are.
We plan to pause the simulation for all the SLs and for two fire team leaders while their shared
situational awareness is analyzed. Soldiers will be asked to drag-and-drop figures representing
the squads, platoon leaders, and fire teams, as well as suspected enemies. Because the VOC
knows the state of the world, it can provide feedback to the Soldiers, and can recognize when
members of the squad have different assessments of the enemy than other members. The latter
condition suggests weakness in the squad’s ability to communicate their shared situational
awareness.

The AAR will be focused on group level goals, but will benefit from the knowledge of
who was making etrors. The AAR is also the place to comment on patterns of behavior that do
not generate a pedagogical response during the mission. For example, “You failed to switch
your lead fire teams between clearing and security detail. While there are no set rules about
swapping, you should have made a change to the lead team earlier.”

A sixth type of instructional intervention strategy is to change the scenario. This is a type
of cheating, in which the tutor plays an all knowing O/C and can make the players suffer *
consequences for mistakes that they may have not noticed. For instance, if a point man fails to
continuously scan the environment, then the scenario will be able to make an enemy movement
that can be used later as learning experience. Under some conditions, we may choose to give
immediate feedback as a default. However, if the student is engaged in a fire fight, rules will
determine if feedback is given either at the end of the fire fight, during the AAR, or not at all.
For instance, suppose a Soldier charges into a room without first making sure his team is ready.
We would give the feedback by stating, “You rushed into the room before you heard from each
member of the team that they were ready.” However we cannot give this feedback immediately
because it would violate our rules to give it only when there is a probability that the Soldier will
attend to it.

: We also need rules to give feedback depending on the context in which it is given. Each
of these choices will change the nature of the feedback. For instance, feedback that is deployed a
few minutes beyond the event will need to identify the context in which the error occurred. For
instance, feedback might consist of the following statement: “Jn the middle of that last fire fight,
you rushed into the room before you heard from each member of the team that they were ready.”
The first part of this statement establishes the context in which the error occurred. If this error is
left for the AAR, then it might be combined with other errors of the same type. It may also
connect to a pattern of errors that occurred at the same time. The feedback statement will
include contextual information so that the feedback is linked to the appropriate event and
behavior.

It is possible that an error might not be addressed until the AAR. There are two reasons
for this. After a sufficient amount of time has passed, the urgency to make a comment may
decrease, In addition, some team errors may not be detected at particular points in time during
the mission. Therefore, the opportunity to provide immediate feedback would not emerge. For
instance, if one team moves to clear a building quickly, and the VOC detects that the squad took

12



uonisod ut ore Aayy a10jaq syioday
yiodal 0} mols

issng

Apear Ayyenjoe st ures) uy

‘IoWUBUI J99LI09 SY} UT ‘UOTYsEy AjOUTN
B UI JS 0} Sodar Jape] Wea] 739918],

Apeor
are Kay) yey} ‘TS 0} soda gq urea],

WONLDO] 1991109 0} OF jou s90q.
19A09 Wed spenbs 104} a10J9q SAO
sang

steep Joy aAour yey}
0} JaJal OS SAOU YW Wea] oY} 0} eOUSPT
9q Pnoys uonoe JuoWSAOUI ST], :3991B],

woNe00}
Anuo Zurpying 0) soaour g wea],

‘quiod Aqua Surpying 0} scour s.q
urea], 1909 0} spenbs 19y}0 sIap10 Tg

OS]O DISYMOUIOS S90H
ured} }]Nesse spesy]

Wed} }[Nesse MOTIOF },USe0q

isang

qurod Anus Supping

0} IBAO YW Wea], SMOT[OJ TS 39818],

ZD AO Urea} pea] YIM SAAOUI TS ,,€

ABojoaseryd yDoLI00NT

"Td 0) Hodar premioy 0} MO|S

V wee] woy BuLeay al0jaq syoday
sng

uontsod ut st

‘V wea], Jey) ‘sonbruyoa} suoneormumuos

yooli09 Suisn pue uoTysey uontsod
Ajoumn v uy “Tq 0} sodas Tg +9848], Ur SI y Wea] Jy} Tq 0} SHodey 1S
VW wea, Spremo} Yoo] 3,uss0q

uoneao] Auras spreMmo} Yoo] 3,Uss0q]
YBN] 00} SI eplolney Jo si uayM s}ooys
popeou uaym JOOYS 3,Uss0q]

wondioseg Sng pur josie]

SIOIABHIG saulVsy, pojyedxq papiejoq

yuaaq O1enaIg



Stimulus Category

Stimulus Examples

Friendly events

KIA,
WIA,
Element fatigue

Civilian actions

In line of fire,

Mob forms,

Assisting enemy forces,
Overt acts against friendly
forces

Equipment

Weapon malfunction,
Radio malfunction,

Out of range for
communication with higher,
Equipment missing or not
operational

Table 2 presents a list of factors that can influence situation assessment and whose value

must be determined whenever they are relevant.

Table 2

List of Factors That Can Influence Situation Assessment in the VOC

Factor

Possible values

Enemy’s experience, training
and morale.

Highly trained, high morale,
Poor Training, low morale

Friendly’s experience, training,
and morale.

Highly trained, high morale,
Poor training, low morale

Enemy Level of Resistance

Fanatic level of resistance,
High level of resistance,
Low level of resistance

Condition of friendly forces
equipment.

Equipment operational and
available,

Equipment missing or non-
operational

Quality of friendly forces
equipment.

Current first-line equipment,
Outdated equipment

Equipment available

Trucks, helicopters available,
No support available

Friendly forces fatigue.

Well-rested,
Exhausted

Light conditions, visibility.

Daylight, clear skies,
Night, cloudy, fog, rain, smoke
or other obscurants

17





uontsod

JO 0} ures} & Jopso jou sa0p Js ang
Surpting Jo e8ersa09 JoyoutLed
UWMUNXeUI pue UOHed0] AusUD 0} JYdIS
Jo aul] yaNp sapiaoid yey uoNIsod any
Jo aseq amdes b sal uap! TS :3981B

uoneso,
(dO) o41g Jo aseg saynuapr 1s

“oqR]
00} penbs suodvam Aavay sjsonbai 7g
“penbs

suodvom Aavay ysonbai jou sa0p 1S
sing

“Td Woy

suodeom Aaray ysonbar snl “preMJo}
penbs suodeam Aavoy oaoul pynoys

‘JS ‘adsR] St ao10} Atuoue ap J] 3JoduE

woned0] SuoIM UO any sumjar penbg
uontumunue yonur 00} spuedxe penbs
aij WNJal JOU ssop penbs

issng

wun Auiaua

dy} JO UOTaIIp at} UI pue sUOTEIOpIsUCD
uonPAIasuOO UOTTUNUTUE “AO

WIM AY] 21 summa penbs :ya8uey,

Aurauq ay) sassaiddng penbs ,.|

juataaowl
utdaq 0) spenbs sayjo duimojye

daly JO aseq B saystgeiso penbs |]

“J9A09 Saye] penbs
Joye ulede sary Awiaugq :uonsy paiduss

noyndioseg sng pur jad18],

SIOIABYI soulesy pajysadxg pojlejoq

WUaAg OLWENIIS



identification that defines all simulation hosts (computers). SVS shall provide a method for
placing a wolf-tail or other similar marker on a wall outside a room to indicate the room has been
cleared. SVS shall provide a method for placing a satchel charge on a wall to blow a hole in it.
Preliminary System Design

Figure 3 repeats the information shown earlier in Figure 1. It illustrates a training system
that incorporates a VOC. In this diagram the simulation is represented as a single box, which
includes all aspects of the simulation, the user interface devices, the displays, and the necessary
computers. The information flowing between the simulation and the VOC is represented by three
broken lines, and these lines indicate communication channels and protocols through which all
the information necessary for the systems operation will pass. In the lower left of the diagram
there is a box labeled Scenario Definition. For any training session, a scenario will be presented
to the Soldiers. On the far right of the diagram is a box labeled Performance Archive, where the
VOC stores all the performance information gathered during training sessions. These archives
are organized along several dimensions, including Soldier identification, the time and date of
training sessions, the scenario involved, and other pertinent information. The contents of the
archive include instructional history (e.g., feedback delivered, scenarios experienced), and
mastery evidence derived from the VOC’s action evaluation decisions. There is no user interface
provided to the VOC because all interactions with the Soldiers will be through the simulation
interfaces. Any instructional interventions generated by the VOC will be forwarded to the
simulation for display or presentation to the Soldiers.

ie Information

_-7 fs Mastery Evidence,
L— ~al Instructional History
| SVS Simulation > — 7: voc ly
a Student Actions
( Performance
ae a> Archive

Scenario Instructional Interventions,
Definition e.g., feedback

Figure 3. Preliminary system design (from Figure 1)

In Figure 4, the three lines connecting the simulation and the VOC represent a stream of
messages being passed back and forth between them. From our discussions earlier in this report,
you will recognize information regarding the situation, (indicated by the top line in the figure);
information about what the Soldier is doing (the middle line), and instructional interventions :
such as immediate feedback (indicated on the lowest line of the figure). The specific content of
all of this message traffic will be developed during a detailed design phase. The Preliminary
Requirements section of this report provides a list of what the messages must include.

29



too long to the clear the building because of the second fire team’s delay, then the VOC would
address this during the AAR.

The number of humans affected by the feedback should be a factor in determining the
content of the feedback and when it is offered. During a real mission, an O/C might intervene
when errors would destroy the value of the training exercise. The VOC should do the same. As
an extreme case, if a SL misunderstood his mission brief, and he set up his assault point in the
wrong position, then the mission may prove to be a failure at the outset. This is an instance
when the VOC should intervene immediately to reduce wasted training time.

Now we consider some specific differences between the individual coaches and unit
coaches. The individual coaches will have their own set of instructional rules to decide among
three broad instructional intervention options:

e Provide immediate feedback to the individual
e Record the error and contextual information for later use
e Inform the unit coach about the problem, potentially providing a feedback message

In the latter case, the unit coach can decide, using its own instructional rules, whether to
execute an instructional intervention, or to discuss the issue during the AAR. While individual
coaches will provide feedback on the specific errors individual Soldiers make, the unit coach will
provide feedback on higher-level team goals, such as the percentage of the building cleared in a
given time. The VOC will be able to point to specific errors committed to explain why a group
failed to meet their team goals. At other times it will not be certain why a group failed to reach
its goals. Under these conditions, the VOC may bring up an issue during the AAR, but might
leave the final analysis to the Soldiers.

Automated Measures of Performance and Effectiveness

In this section of the report we will present examples of the specific knowledge that the
VOC needs to operate, as well as the performance and effectiveness measures that it will
implement. We will also present our approach to managing the challenges we encountered due
to the inherent complexities of dismounted infantry operations.

In light of all the recent investigations into dismounted infantry operations in urban
terrain, there is a large amount of information about how Soldiers should act in a variety of
situations. Klein et al. (2001) captured a great deal of information relevant to our building-
clearing scenario, and we have drawn heavily from it. Klein et al. (2001) focused on PLs.
However, much of its content is relevant to SLs and fire team leaders. Klein et al. (2001)
captured a number of factors that can affect situation assessment and decision-making, such as
the intensity level of the conflict and the enemy’s capabilities to engage. We will evaluate the
situation in order to decide what the proper action might be for the small units we are
considering. Furthermore, the values associated with these factors and their importance can
change quickly. For example, maintaining stealth is less important while breeching a building
than it is while approaching the building. All this must be accounted for in the design and

13



SAF you can, for instance, specify a start point and an end point and the SAF will
determine its own path using terrain reasoning and obstacle avoidance. SVS CGF must
have the path completely specified, and it will blindly follow this defined path. Signal
and command data are available internal or broadcast.

The current command set for SVS CGF consists of:
e Run, walk, crouch, crawl

Stand, kneel, prone

Aim, stop aiming

Stop, resume

Timer, play sound, change path

Send signal, wait for signal

Die, revive

Shoot at point, stop shooting

Detect incoming

Detect entity

The current signal set for SVS CGF is:

(Alpha, Bravo, Charlie, Delta) Halt
(Alpha, Bravo, Charlie, Delta) Move out
Attack

BMP-Go

Delta heavy weapons go
Dismount

Follow me

Open fire

Pause

Resume

Smoke

Start

Stop

Suppress

Withdraw

Current triggering cues are manually executed via a menu selection (as from the Battlemaster
station), or as seen, may be generated by other CGF. It is possibly for commands to be generated
via the speech recognition system discussed as a potential addition to the Virtual O/C/SVS
system.

B-5



CONTENTS (continued)

APPENDIXES

Appendix A. Experttrain Technology...
Appendix B. SVS Simulation Description
Appendix C. Building Clearing Scenario...
Appendix D. Detailed Scenario Analysi:
Appendix E. Knowledge Base

. List of Tables

Table 1. Stimulus Categories and Specific Instances of Those Categories in the VOC
Table 2. List of Factors That Can Influence Situation Assessment in the VOC
Table 3. Situation Assessment Rules used in the VOC.
Table 4. Action Execution Behavior and Evaluation Criteria Used in the VOC
Table 5. Behavior Descriptions and Evaluation Possibilities Used in the VO .
Table 6. Action or Behavior Detection Rules Used in the VOC...........04
Table 7. Simulation Information Needed From SVS to Make Assessments .
Table 8. Simulation Message Descriptions, Data Needed, and Trigger Conditions

for the VOC......c.sesssesssecseeaceseesnees
Table 9. Audio Cue Messages for the VOC .
Table 10. Visual Cue Messages for the VOC ........ssssssesssssssessessssesesesssseneseeseseneetesseeneneeneneanenenes

List of Figures

Figure 1. High-level system diagram .
Figure 2. SVS immersive architecture
Figure 3. Preliminary system design
Figure 4. VOC processing diagram.
Figure 5. VOC component diagram
Figure 6. Engagement detection algorithm flowchart.



APPENDIX E

KNOWLEDGE BASE OF INFORMATION AND RULES REQUIRED FOR VOC

This appendix presents the various types of information and rules needed by the Virtual
O/C. They are grouped by type: 1) Behavior Description, 2) Action Execution and Detection, 3)
Immediate Feedback samples.

Behavior Description

The following table presents annotations of the detailed trainee responses associated with
the sample scenario, These annotations describe some of the correct (target) and incorrect (bug)
trainee behaviors that the VOC will detect and critique. Note that this table is dealing only with
individual trainee actions (i.e., actions that the Individual Coach will monitor and evaluate), not
collective team behaviors or longer-term issues such as the overall pace of movement through
the major phases of the scenario.

E-1



We will examine each of these in turn.

The information regarding the sound associated with the enemy’s fires must contain
whatever directional information the Soldier received, such as the left-to-right panning location
of the sound, a qualifier regarding what weapon system made the sound, and some indication of
its proximity to the fire team. This message must also be directed to the correct Soldier, because
if multiple squads are in the same scenario, the same sound can be to the right of one Soldier and
to the left of another. The same information is required regarding the auditory cues of rounds
impacting. Examples of the audio cues required to determine if the fire team is being fired at are
shown in Table 9.

Table 9
Audio Cue Messages for the VOC
Types of Information Specific Data
Sound cue event Indicates what type of information is being
provided: audio, visual,
Spatial location 3D coordinates
What made the sound Small arms, round impact,
Distance qualifier 30m or “nearby”

The information regarding the visual cues is somewhat more complicated. While the
simulation can inform the coach where the visual cues were located in the simulation
environment, there is no way to know if a Soldier actually saw the visual cue. What we can
know is that the visual cue was rendered on the display in the Soldier’s field of view, along with
some size and duration information, so that they could have seen it. We are making the
assumptions that the Soldiers are looking for visual cues and that an impact dust cloud is visible.
Examples of these visual cues are shown in Table 10.

Table 10

Visual Cue Messages for the VOC

Types of Information Specific Data

Visual Cue event What type of information is being provided: audio
or visual?

Real world location information 3D coordinates

Rendered on display Yes/no

Field of view location, if rendered _| Horizontal and vertical angle from center of display

Size information, if rendered In pixels

Duration Information, if rendered _| How long was the image in the display

What was rendered Smoke, flash, impact dust

The simulation’s ability to provide these data is only part of the process. There are two
more steps. The first step is to decide how sure we are that the Soldier could have perceived the
visual cue events and the second is to decide whether the fire team is close enough to these visual
and audible cues for it to mean they are being fired at. We will deal with the location question

34



WOO! aty dsams AT]enstA 3OU So0q
Atessaoau UsyM JOOYS 30U S20q,
Atessad0uun UsyM S}OOYS

MOPUIM Sosseg

soueqUs syso[g

AjyooM0oul SaAo]

asesyd Joa1I00UI S]J9%,

J9}09 0} MOTS

issng

AOU MV] Autous Aue sodeSua

pue wool sua oy) punore soze3
“Woo! sy} SuLIa}Ue uodn UONaIIp yO91109
OY} Ul sovoul ‘asesyd 3991109 Sury]s4
“WOO! SY} S19}U9 Jopes] Wed], :}9S1BT,

Azessaoau jl
Sally ‘UOTENIIS ay} Ssasse “‘PpuUNOre syOo]
pure WOO! s19}U9 Jopes’] Wes] INessy

« WBLAY2] Ul WeUL 1X9N,,
‘s]]@0 9U0 puodes ay} pur AT}9ALI09 WI0OI
JoqUa SIOIpog payeiouas 1oynduros om],

ogreyo
ay} Selly pue Zurping Jo yu1od Anua
ye o8reyo sooejd Jopeo’] urea] ynessy

J|NESSE IOPIO 0} MOTS
uorssruad

"Td 3umj03 a10jaq yNesse sIOpIC
sang

Apeal st yt speusis
‘V wea] Joye ‘Aem Jodoid op ul Jopesy
wea], 94} 0} Jop1o ay} SAIS TS :39318],

Surpring
9} JajUe 0} UTED} }[NESSe SIOPIO JS

*posood 0} uoisstutied
Saals pue yodal Ts sodpojmouyoe Tq

ABojoaseryd yoaL09uy

uontsod ut are Kay} al0jaq syoday
yoda 0} Mo[s

ising

uoneuLoyut

ayemooe YIM ‘Tq 0} sHodel Tg :yad1e],

woTeI0}
ymesse ye Jos way Tq 0} SHOdaI Tg ,,€

aoRdiosag Sng pus yas1e 7,

SIOIABYOY saulesy, poysadxy payiejoq

JUdAG OLIeUIg



collection of visual scene information is needed regarding the enemy. Was the enemy rendered
on the screen? If so, then how large was the image, where in the Soldier’s field of view was it,
and how long was it visible? All the same issues described above are in effect, with one more
complication. Was the Soldier supposed to focus his attention on the area where the enemy was
last seen? If the tactical situation required the Soldier to pay particular attention to the enemy’s
last known location, then that objective must consider the sureness factor we described above.
We would like to know whether the Soldier concentrated his visual attention on the area of
interest.

Because we are in a simulation environment, the simulation always knows what part of
the simulated world is being drawn on the output display. The Soldier has complete control over
where he is looking. We can now consider extracting direction of gaze information from the
simulation. As with the sureness factor, there is some amount of detailed information that can be
extracted, such as how much time elapsed since the enemy’s last known location was rendered

* on the display, how much of that time was not in the field of view, and some indication of where
in the field of view it was rendered. All of this detail must be examined to derive a conclusion
regarding the Soldier’s attentiveness. An algorithm that uses these data to arrive at a conclusion
must contain several parameters that can be adjusted based on experience with the algorithm in
an operational system. Combining the visual cue sureness factor with the qualitative conclusion
about the Soldier’s attentiveness provides the opportunity for a number of instructional
interventions.

The next item on our list is “Fire team has recently seen an enemy.” Again, this is an
instance where we need to mark the passage of time from one event to the next. However, in this
case, we need to start the clock only after we have confirmed that a fire team has seen the enemy.
This confirmation can be derived from indirect evidence such as the fire team firing at the
enemy’s location and/or reporting to higher that they have seen the enemy.

Rules of Engagement

The rules of engagement (ROE) will have the effect of changing the various weights and
priorities associated with the rules used to evaluate the situation or the Soldier’s behavior. As an
example, the rules of engagement may prohibit firing until they have been fired upon. In sucha
case, a fire team that has not been engaged, but fires at a human who crosses in front of a
window in a building reported to hold enemy forces, would be evaluated as incorrect. The rules
of engagement must be reported both to the human Soldiers in the exercise and to the VOC.

This can be handled by assigning specific ROE data to a scenario definition in two forms: (a) a
human readable form, and (b) as data that the coaches can accept and process into the rule base.
When a scenario is selected for an exercise the simulation will report to the human Soldier,
through its interface, the readable form of the ROE. The data form of the ROE is then forwarded
to the coach who adjusts the rules to match the ROE.

37