Table 2 (Continued)

Skill Dimension and Measure
Name

Description

Shared Situational Awareness
Map Area
Line of Sight
Surprise Attack

Collateral Damage

Square kilometers of the battlefield displayed and center point of each
resized PVD map during each mission by duty position at critical
points in time.

Frequency and duration of use of the Line of Sight tool during each
mission by duty position.

Total number of flank or rear engagements on OPFOR and BLUFOR :
vehicles during each mission. _

Total number of attacks on BLUFOR non-instrumented vehicles
and/or personnel by indirect non-line of sight weapon systems under
battalion control during each mission.

Communication
Whiteboard Use

Radio Communications
Pattern

Personnel Initiating
Whiteboard Conferences

Total number of Whiteboard files residing on each workstation for
each mission by duty position.

Frequency and duration of use of battalion command and operations-
intelligence radio nets and Whiteboard conferencing during each
mission by duty position at critical points in time.

Total number of Whiteboard conferences, lasting 3 minutes or more,
initiated during each mission by duty position.

Coordination
Overlay Use

Whiteboard Commonality

Targeting

Fire Support

Coordination

Fire Engagements

OPFOR Destruction

Total number of workstations showing the same operations overlay
file that the Squadron Commander is showing on his PVD for each
mission by duty position at critical points in time.

Total number of Whiteboard directories showing the same Whiteboard
files that the Commander and Deputy Commander have for each
mission by duty position at critical points in time.

Total number of times a SPOTREP query was conducted on the target
identified in a fire support request immediately before it was
transmitted by a staff member.

Ratio of OPFOR kills due to indirect fire from units controlled by the
squadron staff to OPFOR kills due to direct fire controlled by

squadron subordinate units.

Average range of OPFOR and BLUFOR kills by vehicle type during
each mission.

“Time from the first OPFOR engagement until OPFOR vehicle losses

exceeded 70%. In addition, the rate at which the OPFOR was killed in
5-minute intervals from the first engagement until the last OPFOR kill
during each mission.

(table continues)



Figure 10. Opposing force damage from blue forces.

The BLUFOR Damage bar graph (Figure 11) also indicates that indirect fire was the
primary cause of losses to the unit. More indirect fire systems were lost than direct fire systems,
which may indicate that those systems were easier to target by the OPFOR because they moved
less often, or that they were positioned too far forward, which increased their vulnerability.
Further analysis would be required to find the reason for this result.

29



90000 -———

20000 —|—

190000 j=

10000
eesen | — L
e000 L. —_
70008 {Unit Zone OPFOR Movement

lor Action NI

50000 }—=—-

40000 | — _

30000 ! —

0000

10000 - —

(© $0000 20000 20009 40000 S090 0000 70900 #9000 90000 100060 110000 120000 130000 140000 150000 160000 170080 180000

°

(Zone _TM@ Car DepCor_TM Control1 TM@ Control2 TM Maneuver Platoons

Note. OPFOR = opposing forces.
Figure 9. Node locations at 0900.

The locations of the nodes indicate that the Cdr and DepCdr nodes were centrally located in
the zone of action and in close proximity to one another. Control 2 was likewise centrally
located. Control 1 was located well outside of the zone of action. With the unit’s SC* system
capabilities, the ability to maintain communications was not dependent on proximity.
Operational and physical security was a consideration, and ideally Control 1 should have been
located closer to the unit’s subordinate elements.

The picture provides a quick reference as to where the commander and staff were located at
any point during the mission. If communications among staff vehicles and between the staff and
subordinate units is affected by terrain or distance, then a communications pattern indicator
similar to a range template might be useful. If physical security is a primary consideration,
adding detected OPFOR locations might also be warranted.

Communication

Overlay Use

Table 4 shows which staff officers and team commanders had the same operational overlays
available on their SC‘ system as did the unit commander. The source of the data is a record of
each time the PVD Overlay Editor is used to create an overlay, or to display or turn off an
existing overlay. The recorder also captures the time, the name of overlay file being
manipulated, and the radio call sign of the soldier working with the overlay. For Trial 1, there
were approximately 7,700 instances where the PVD Editor was used. Using the unit
commander’s radio call sign, the list was filtered several times, to identify the overlay files the
commander used. That list was compared with the overlay files that the staff officers in his

25



OPFOR, first friendly casualty, friendly losses exceed 30%, and last engagement during
mission).

b. Rationale: May indicate whether the staff estimates and other information staff members
provided to the commander allowed him to make the right decision on positioning his
maneuver forces prior to the start of the trial.

c, Output: Picture

e — Availability of ammunition and fuel during each mission.

a. Operational definition: Percent of critical ammunition types (to be determined prior to
start of mission) for 20-minute intervals for each mission, using 60% as a base amount at

any given time.
b. Rationale: May indicate if the staff made the right decision on positioning ammunition

and fuel prior to start of mission.
c. Output: Line graph

C-7



detect the unit. Also missing are unit positions, which again would provide an indication of how
much information the OPFOR could have gained before they were destroyed.

490000 oan pomennnns ep

20000 |~ t- ~

110000 |“ TGnit Zone
lof Action

100000

000 -|-—

#0000 | — - OPFOR Movement |

ro000 f——~ {— 7 | f

000

50000 —

40000 | |}

30000 i

20000 mabe ~ +
| I

"0009 Note: Tracks link where OPFOR Recon detected then killed
| i i
© %0000 20000 30000 40000 0000 60c00 70060 80080 90000 100009 110000 =20000 130000 40000 150000 169000 170000 180000

0

[zone —*-Series2 Series3 ~*-Series4 —*-Seriess —*-Series6. —+Series7 sess |
~"Series9 __Seriest0_Seriesi_Series12_- Series“ “'Seriesi4““Seriasi5_\ Serles16

Note. OPFOR = opposing forces.

Figure 15. Counterreconnaissance effectiveness.
Artillery and Counterfire Radar Coverage

The picture designed for this measure (Figure 16) depicts the range templates of the indirect
fire weapons that are controlled by the unit staff. There are a total of 12 systems — six missiles,
three rockets, and three 155mm howitzers that could be positioned by the staff. Location for
individual systems was available at 5-minute intervals. To create the range templates, data for
each individual weapon system are automatically posted to a worksheet, which plots the points
necessary to create the circle representing the range template. If a particular system has been
destroyed, the location of the system is calculated as being at 0,0 and the range template for it is
drawn around that point, as indicated in the lower left portion of Figure 16. The legend provides
the unit identification number for the individual systems. Units being supported by the weapon
systems are depicted by plotting the maneuver platoon leader positions. The OPFOR depicted
represent detected OPFOR systems. The OPFOR location data were available at 20-minute
intervals.

This measure is designed to provide information on whether the staff is effectively
coordinating the indirect fire assets it directly controls to support both its subordinate units and
the commander’s intent and concept of the operation. Typically, indirect fire weapon systems,
while geographically dispersed, are positioned where they can mass timely fires at decisive
points in the operation, and attack targets of opportunity while reducing BLUFOR exposure to
OPFOR counterartillery fires.

34



Integrating commercial business software into digital C“I systems may be a low-cost way to
allow trainers, commanders, and observers who are not computer programmers to design their
own automated performance measures. As these commercial business software packages are
becoming more capable and sophisticated, their ease of use and relatively low cost has led the
Army to widely distribute them. Increasingly, every soldier is being provided access to desktop
computers that use business software to accomplish routine tasks. These soldiers are also
experimenting with ways to use this software to collect, process, analyze, and disseminate

information about performance.

Capitalizing on this broad-based knowledge and familiarity with operating commercial
business software may be a bridge to automating performance measurement using C“I systems.
Although outcome information (i.e., MOEs) is commonly automated, more work is necessary for
process information (i.e, MOPs) to reach the same level (Salas, Bowers, & Cannon-Bowers,
1995). The U.S. Army Research Institute for the Behavioral and Social Sciences (ARI) has
sponsored a series of research projects with the objective of automating performance
measurement, particularly for battle staffs equipped with advanced digital C? systems. This
report details the assessment methodology work performed for an ARI-sponsored contract
project titled, “Performance Evaluation, Training, and Future Requirements for Digital Skills”
and referred to herein as DC‘I-3. The research effort was conducted by a project team consisting
of personnel from the Human Resources Research Organization and Litton PRC (hereafter
referred to collectively as the Project Team).

Specifically, the objective of the DC‘I-3 research project was threefold: design, develop,
and demonstrate prototype automated measures to improve training and evaluation for brigade
and below command and staff training; design, develop, and demonstrate prototype training
techniques that incorporate principles of cognitive psychology and automated performance
assessment and feedback; and identify research issues and training approaches for the future
force with a focus on embedded staff training and multi-functional leaders and soldiers. This
report documents the design, development and demonstration of the prototype automated
measures of command and staff performance.' The original prototype automated measures
package was designed for “Refinement of Methods for the Training and Assessment of Digital
Staffs (DC‘I-2),” and is described in the report Refinement of Prototype Staff Evaluation
Methods for Future Forces: A Focus on Automated Measures (Throne, Holden, & Lickteig,

2000).

There are many benefits to using automated measures, some of which are outlined in Throne
et al. (2000). Most importantly, as command staffs rely on computers in their work, the more
important it will become to assess their computer interactions as meaningful aspects of work
process and outcome. Second, automated measures are objective measures of performance —
they present “unchallengeable ‘ground truths’ ” (Brown et al., 1997, p. 3). Third, a greater
reliance on automated measures may increase the scope and precision of performance assessment
and feedback. Fourth, automated measurement and analysis may be needed for more complex
work settings, such as C? staff performance. Fifth, unobtrusive and automatic data collection
may reduce measurement error and increase the accuracy of the information presented during

' For more information on the prototype training techniques, see Deatz and Campbell. (2001). For more information
on the research issues for the future force, see Campbell and Holden (2001).



Future Staff Performance Standards

Further development efforts on automated performance measures requires research to
develop objective performance standards for future C’ organizations. The research should
include intrinsic measures where the organization is provided on-going indicators of its
performance, such as Orders Distribution, where they could determine by the systems and
resources immediately available to them if everyone is using the same operations overlay, and
extrinsic measures, such as Kill Distance, where additional post-training processing would be
required to establish the organization’s overall performance. Team behaviors and processes,
soldier-machine interface, and soldier-machine task allocation issues should be included in the
research to ensure both group dynamics and individual contributions can be isolated and
analyzed for their impact to the overall performance of the organization.

Pre-Planned Measures Implementation

Automated performance measurement capabilities need to be fully integrated into the design
of advanced digital C'lsystems. If data are not being recorded and stored, it is not available for
other purposes such as training feedback. Data storage may be an encumbrance/obstacle with
operational CI systems that dump data when they are powered down. In such instances,
alternative, dedicated data storage systems may have to be specifically developed for recording
performance data for later analysis. The feasibility of including this capability may increase and
the costs associated with it may decrease as large capacity data storage devices become more
compact, tolerant of unstable operating conditions, and power efficient. As described earlier,
handling large amounts of data associated with automated performance measures can overwhelm
operational C’ systems. Research efforts are needed to optimize collecting, processing,
assembling, and distributing data for AARs.

Summary

Digital C'l systems provide unlimited potential to assess individual soldier, small group, and
collective performance data. They have organic capabilities that need to be exploited to
automatically collect, analyze, and portray data. Such data collection, accompanied by
capabilities for analyzing and displaying the results in terms of processes and outcomes, can
support both intrinsic and extrinsic feedback. By “intrinsic feedback,” we mean the information
that immediately informs the user that something is not right, or that more information is
available, or that some critical information need is being answered. This information, provided
by means of on-board systems and remote sensors, can be provided as an operational capability
as well as during training. “Extrinsic feedback” allows the user to look back on an operation or
training exercise and identify ways to sustain or improve performance through optimal use of

information systems.

Accessing the collected performance data and making it intelligible to observers or to the
unit in training has to become a common and routine feature of all training and operational
systems. The particular data elements that are related to performance, the analytic tools that fuse
data to provide useful feedback, and the format for feedback reports need to be optimized from a
human factors point of view to allow for clear and immediate impact. Continued research and

62



450} +

400 —

360 oo, -— | soma -

| |
| | |

250

Rounds

0:00 020 O40 1:00 120 140 200 220 240 300 320 520 S40 600 620 640 7:00
Time (in Minutes) Since Mission Start

[== MRAAS_CARGO “M=MRAAS_KE —*=MRAAS_MPERM —@=QUICKLOOK

Figure 24. Ammunition available over time.

While consideration was given to trying to add a line to indicate what the minimum desired
stockage level, or unit basic load, was for these types of ammunition, the Project Team was
unable to automatically generate a line or other device compatible with the rest of the data that
would provide that information. Such information would be useful in determining whether the
staff needed to sustain or improve its performance in this area. Changing the format to graph the
percentage of the unit basic load for each type of ammunition being tracked that is available over
time would be a technique for overcoming the limitation of the current output format. Visibility
on the number of rounds would be lost, however. In some instances such as when the unit is
cross-leveling ammunition, the actual number of rounds available would be an important
management tool. Further research is required to determine the optimal format for this measure.

Performance Monitoring and Feedback

Effects of Targeting

Three output formats were developed by the Project Team for this measure: a table, a
picture, and a line graph. In the table, the categorization of HPT and the number of expected

44



Based on the approximately 225 OPFOR weapon systems that were engaged and killed
during the trial (Figure 11), only three OPFOR systems were subsequently attacked again after
they had been killed. This result may indicate that the unit was closely tracking OPFOR battle
damage and had a high degree of confidence that the SC’ system was providing them with
accurate information.

The picture output format may provide more information if the location of the shooters are
also displayed. Adding the locations of other OPFOR weapon systems that were near the
targeted systems at the time they were attacked again may also provide additional context for
analyzing the engagements. Incorporating the table into the picture would also facilitate
analysis.

ma LI |
[> i
‘ween | — ont Zone ———
ot Action | | ,
veto = 1
sa | /- —
e000 J —s
OPFOR Movement
rome {|__
_ «if.
.
soot
- a
30000 j | |
20000 | t —
rae 7 f[__| Note: Depicts Targets Engaged Aor Boing Reported ‘Destroyedt
° ~ 4 1 4 4 { {__} 4 4. 4 4 1 4
© sexc0 aotc0 sinco onco e000 F000 Toned ited Geto yGE000 110500 YaNGD 12080 YO00 160000 #6500 17000 180009
(tore + Tages]

Note. OPFOR = opposing forces.
Figure 22. Multiple fire engagements.

Maneuver Battle Sets

Figure 23 depicts the location of the unit’s maneuver platoon leader vehicles in relation to
detected OPFOR weapon systems. The unit vehicle location data are available at 5-minute
intervals. The OPFOR location data are available in 20-minute intervals. The Project Team had
originally designed this measure to plot the locations of like sets: BLUFOR platoon leader
vehicles against OPFOR platoon leader vehicles. However, since the OPFOR locations are
based on whether they had been detected or not, relying on just detected OPFOR platoon leader
vehicles would not provide sufficient reliable information about the OPFOR to indicate whether
the unit had positioned its subordinate units appropriately. Consequently, the Project Team
decided to plot the locations of all detected OPFOR weapon systems, rather than just detected
OPFOR platoon leader vehicles.

42



based on 5-minute intervals, except for the location of the OPFOR which is available in 20-
minute increments. An arrow indicating the general direction of movement for the OPFOR has

also been added.

The rationale for this measure is to ascertain if the staff has a shared understanding of what
is occurring at critical points during the mission. If the team commanders are focused on
different portions of the battlefield than the staff or if the resolution of the staff's view is
significantly different than that of the team commanders’, then the amount of shared situational

awareness among the unit comes into question.

The measure results depicted in Figure 27 indicate that at 0920 or about an hour into the
mission, the unit commander and his staff had adjusted their displays to cover most of the unit
zone of action and could also cover a significant portion of the battlefield in the area from which
the OPFOR could be expected to appear. Only one staff officer (Outlaw6) was able to see back
to areas where the unit still had forces. The team commanders (Eagle6, Fox6 Ghost6, and
Hawk6) had zoomed their displays down into the immediate fight, focusing on their subordinates

and the OPFOR.

Plotting the map area for the commander and three staff officers plus the four team
commanders on the same picture creates a fairly complex presentation. While it is not possible
to replicate it on paper, the electronic version of the output format can be modified to remove
unwanted data, which can simplify the presentation. Additionally, each of the individual map

areas can be also be highlighted if needed.

48



Fire Support Coordination

The Project Team developed two output formats for this measure: a picture and a bar graph.
The picture (Figure 31) depicts the location where each individual OPFOR weapon system was
killed by the experimental unit during Trial 1. The outlined diamond symbol represents the
location where vehicles were destroyed by direct fire weapons. The solid diamond symbol
represents the location where OPFOR were killed by indirect fire weapons. The unit zone of
action has been plotted based on a table created by an observer and an arrow indicating the
general direction of movement for the OPFOR has been added. The bar graph (Figure 32 relates
the number of direct fire kills to the number of indirect fire kills.

The rationale for this measure is that if the staff has sufficient situational awareness as well
as the ability to synchronize the fire support assets under its control, then the unit could inflict
significant damage to the OPFOR before the enemy can close to within direct fire range. The
farther away from the unit the OPFOR can be destroyed, the less risk there would be to the
experimental unit.

sono
tzot0 | —| |
‘\0t80 | AU Zone]-—|
ran
seeaco | = |
0000 > 4
wo its owron tome
° Bye” +
‘70000 0} 2% 1% ~ ——|—
i abe +
‘0000 ie” gL °
fel |
sexo “ 7 oe oe
4000 |-——} 4—— A
sexo | —|—
woe Note: Depicts Only Targets Killed
° —L { n 14 Lt
© 000 20009 sc060 ogo sooo eotco e000 S009 eo000 1G0o00 10000 10000 130900 v4Ge00 14x80 Te00N0 F700 180000

Note. OPFOR = opposing forces.

Figure 31. Fire support coordination.

As can be seen in Figure 31, a majority of the OPFOR were killed with indirect fire weapon
systems, with most of these occurring with the zone of action. There were some kills outside of
the zone of action which means that the unit was able to interdict the movement of the OPFOR.
Figure 32 relates the number of BLUFOR direct fire kills to the number of indirect fire kills.
The data indicate that there were approximately three indirect fire kills for every one direct fire
kill, which may indicate that the experimental unit was able to inflict significant damage to the
OPFOR while minimizing its losses.

52



Time of detection and time of kill could be added to the picture to give a representation of the
time lag for individual targets. This would be most beneficial for a few critical targets since
providing that information for all targets would clutter the presentation, particularly where there
are large numbers of vehicles clustered.

Decision-Making

Sensor Coverage

Figure 21 provides information about the effectiveness of the various sensors that were
controlled by the unit’s staff during Trial 1. The OPFOR data come from two sources. The
locations for OPFOR systems that are identified as “Not Detected” are derived simulation PDUs.
These locations are available at 5-minute intervals. The locations for systems that have been
identified as “Detected” are provided by data that record the location and time at which the
OPFOR vehicle is under observation and the identification of the unit or vehicle that is observing
it. The data have been filtered so that only those OPFOR vehicles that were first detected by and
are currently being observed by the unit are displayed. These data are available in 20-minute
intervals. The detected OPFOR vehicle data are filtered again to identify those vehicles that
have a “killed” status at the time they are being observed. In creating the picture, the “Not
Detected” systems are plotted first with an outlined diamond symbol. The “Detected” vehicles
are plotted next in a red diamond symbol. The detected vehicle symbol will be superimposed
over the “Not Detected” symbol. The “Dead” symbol is a black diamond which is then
superimposed over the detected symbol. The unit zone of action is then plotted from a table
created by an observer and an arrow indicating the general direction of OPFOR movement is
added.

‘20000

‘0000 |_|

[ery goa
eT @

Al

FOR Movement

20000 a }—|

|
10000 _ | Note: Depicts OPFOR First Detected by Unit ~ |
° bop 1
«2 r0000 20000 20000 2000 so000 ooo 7009 90009 90090 Te0000 110000 120000 1aeec0 +4EG50 7E0000 yecE00 +7eG00 180000

“e NelDaecied + Deleded » bead —=A0)
Note. AO = area of operations; OPFOR = opposing forces.

Figure 21. Sensor coverage at 0920.

40



Handbook (TRADOC, 1997) and at least 11 from the measures developed during the previous
project. Therefore, the team reviewed the TRADOC handbook and the work done on the DC'I-2

project.
Measures Design

Measures were reviewed and selected based on whether they were relevant to the BCR
environment, could be automated without the need of an observer, would be of interest to a
commander, and could possibly be depicted in a pictorial format. From the STAARS handbook
(TRADOC, 1997), 12 potential measures were chosen, including 4 from the C’ battlefield
operating system (BOS). An additional 8 potential measures were chosen from the DC‘'I-2
project. Finally, 9 measures were chosen that appeared in both the STAARS handbook as well
as in the DC‘I-2 project. These 9 measures were essentially measuring the same things although
they may have had different names and/or output formats. For example, the STAARS handbook
contains a measure named fratricide, which was called collateral damage i in the DC‘I-2 project.
In summary, there were 12 unique STAARS measures, 9 STAARS/DC‘I-2 measures, and
8 unique DC‘I-2 measures, for a total of 29 potential measures.

Once these 29 measures were selected, operational definitions, rationale, and sample output
formats were produced for each. Operational definitions and rationale were adapted from the
STAARS handbook (TRADOC, 1997) as well as from those developed for the DCc‘r-2 project.
Measures that had been implemented during the BCR IV were evaluated for their output formats.
The original outputs provided in the STAARS handbook were also evaluated for
meaningfulness. Those that were determined to be meaningful and could not be improved were
used for the current project as sample output formats. Measures that did not provide very
meaningful outputs were redesigned to provide the desired information in the best possible

format.

The next step was to integrate the STAARS measures with the DC*I-2 measures and
categorize them under a single framework. Whereas the STAARS measures are classified by
BOS, the DC‘I-2 measures were classified by the team process skill dimensions discussed
earlier. The Project Team again opted to use the Cannon-Bowers et al. (1995)/TRADOC (DA,
1999) framework since it had provided a meaningful base on which to build the measures for the
previous project. Through discussions among researchers and SMEs, measures were
individually categorized by the skill dimensions. Although some of the measures may have fit
under more than one skill dimension, they were categorized according to the skill dimension for
which they were most relevant.

The operational definitions, rationale, and sample output formats were then passed along to
ARI for evaluation and approval. After evaluating the measures, ARI deleted several of the
measures that dealt with tool use, added a few new measures, and refined the output formats for
some of the remaining measures. This review process led to 24 proposed measures. Figure 1
shows the 24 candidate measures presented under the team process skill dimensions framework.

12



aecace

hb =|
a .
Astery ee .

IFVAPC. ———

Tank FV
Types of Fire

Mo roct Bindirect OaNA

aNA

Figure 11. Blue force damage from opposing forces.

The OPFOR Destruction line graph (Figure 12) shows that Tank_FVs, IFV_APCs, and
artillery systems were being destroyed early in the mission at approximately the same rate. This
may indicate that the staff was able to coordinate fires throughout the depth of the battlefield.
The drop in the rate of loss on Tank_FVs at the 2-hour mark, and the rate of loss for Artillery
and Air Defense systems, starting around the 3-hour mark, may indicate that almost all of the
OPFOR targets in these classes of weapons had been destroyed by that point in time, these
systems were out of range, or were not being detected by the unit.

30



Lessons Learned: Sustain

Based on the experience of the Project Team during the FCC? implementation, as well as the
discussion for improving each of the measures provided earlier in this report, several lessons
were learned which may serve to sustain current efforts in implementing automated measures of
command and staff performance and to act as a catalyst to initiated future research. These may
be applicable to developers concerned with embedding training and, by extension, performance
assessment methodology and tools into future digital CI systems at the brigade level and below.
These lessons are provided as issues to be addressed in future research.

Data Visualization

Continued efforts are needed to optimize the presentation of automated measure results,
especially for graphical and pictorial formats, to soldiers. The goal should continue to be to
provide information that soldiers can readily understand without facilitator or trainer
intervention. Formats should be able to relate command and staff performance to battlefield
conditions. Users should be able to track or “drill down” from MOEs (outcomes) to MOPs
(processes). Finally, through the use of interactive media, soldiers should be able to control the
type and amount of automatic performance feedback information that they are receiving as well
as the format in which it is being displayed not only in AARs but also from their operational cl

systems.

Commercial Business Sofiware

Using a widely available commercial off-the-shelf business software package such as
Microsoft® Office provides inexpensive development tools to create prototype automated
measures. All of the components that would be required to provide measures of command and
staff performance to an automated AAR process are available: data storage and retrieval
(Access), charting and graphing tools (Excel), and tools to present the results to the training
participants (PowerPoint®). Applying this same type of software to operational automated
measures of command and staff performance would benefit users of such systems. Most
experienced business software users would need little or no additional training to operate the
software and to work with the results from the measures. Finally, as these commercial business
software programs incorporate additional functionality, which may make them even easier to
integrate into C‘I systems, upgrades should incur little or no additional training costs.

User Tools Development

Today’s measures, such as the prototype automated measures designed and developed
during this effort may not apply to future training conditions. There is a need for future staff
trainers to be able to design their own measures without requiring an extensive background in
programming. An automated measures development package that gives users the option of using
existing measures or modifying them, or creating entirely new measures is needed. If the tools
used to work on measures development or modification are not already integrated into
operational C'I systems used by the training participants, they should be based on whatever
commercial business software that is being widely used throughout the Army.

61



FOREWORD

The Future Battlefield Conditions (FBC) team of the Armored Forces Research Unit
(ARFU), U.S. Army Research Institute for the Behavioral and Social Sciences has a Science and
Technology Objective (STO) entitled “Force XX] Training Strategies.” The purpose of research
under this STO is to develop and demonstrate prototype training and evaluation technologies that
prepare operators and commanders to take maximum advantage of evolving digital systems.
This STO is also reflected in the FBC work package (2228) FASTRAIN: Force XXI Training
Methods and Strategies. Design of prototype commander and staff training packages that use
advanced digital technology was completed under previous work. This report describes work
under a contract entitled “Performance Evaluation, Training, and Future Requirements for
Digital Skills.” The purpose of the work described in this report was to develop and evaluate
performance assessment tools for future commanders and staffs working in a future digital

environment.

This report describes the design, development. and demonstration of prototype automated
measures designed to improve training and evaluation for brigade and below command and staff
training. The report examines implementation of these measures in a Future Combat Command
and Control (FCC”) experiment at the Mounted Maneuver Battlespace Lab located at Fort Knox,

Kentucky.

At least three audiences may be interested in this report. Materiel and training developers
could consider hardware and software issues involved in embedding these measures into future
command, control, communications, and computer systems. Training unit and training
(simulation site) personnel who conduct training of digital staffs could consider these measures
for feedback concerning commander and staff performance. Also, measurement specialists and
researchers could examine this report to inform further research into the development of
automated measures of commander and staff performance.

In addition to this report, products developed under this effort include software to run an
Observer Workstation, used to develop automated measures of commander and staff
performance during FCC?, Data gathered during FCC? is also contained on the system.

The research reflected in this report was briefed to sponsors throughout the effort and in a
final in progress review, held at AFRU, Fort Knox, Kentucky, on 25 July 2001.

‘A M. SIMUTIS
echnical Director

Vv



References

Anderson, L. B., Begley, I. J., II, Arntz, S. R., & Meliza, L. L. (2000). Training analysis and
feedback center of excellence (TAAF-X) (ARI Study Report 2001-01). Alexandria, VA:
U.S. Army Research Institute for the Behavioral and Social Sciences.

Atwood, N. K., Quinkert, K. A., Campbell, M. R., Lameier, K. F., Leibrecht, B. C., & Doherty,
W. J. (1991). Combat vehicle command and control systems: Training implications
based on company-level simulations (ARI Technical Report 943). Alexandria, VA: U.S.
Army Research Institute for the Behavioral and Social Sciences.

Brown, B., Wilkinson, S., Nordyke, J., Riede, D., Huysson, S., Aguilar, D., Wonsewitz, R., &
Meliza, L. (1997). Developing an automated training analysis and feedback system for
tank platoons (ARI Research Report 1708). Alexandria, VA: U.S. Army Research
Institute for the Behavioral and Social Sciences.

Campbell, C. H., & Holden, W. T., Jr. (2001). Reflections on the structure of the future training
system (ARI Research Note 2001-12). Alexandria, VA: U.S. Army Research Institute

for the Behavioral and Social Sciences.

Cannon-Bowers, J. A., Tannenbaum, S. I., Salas, E., & Volpe, C. E. (1995). Defining
competencies and establishing team training requirements. In R. Guzzo & E. Salas
(Eds.), Team effectiveness and decision making in organizations (pp. 333-380). San
Francisco, CA: Jossey-Bass Publishers.

Deatz, R. C., & Campbell, C. H. (2001). Application of cognitive principles in distributed
computer-based training (ARI Research Product 2001-03). Alexandria, VA: U.S. Army
Research Institute for the Behavioral and Social Sciences.

Department of the Army. (2001). Field Manual I, The Army. Washington, DC: Headquarters,
Department of the Army: Author.

Department of the Army. (1999). TRADOC Regulation 350-70, Systems approach to training
management, processes, and products. Fort Monroe, VA: Headquarters, U.S. Army

Training and Doctrine Command.

Dzindolet, M. T., Gleason, D. R., Hernandez, C., Hill, K., Jaquint, M., Jette, F., Larsen, J.,
Oxley, N. L., & Pierce, L. G. (1998). Defining training objectives and performance
measures in a Janus battle simulation (ARL-CR-433). Aberdeen Proving Ground, MD:
U.S. Army Research Laboratory.

Goehring, D. J. (1995). An automated system for the analysis of combat training center

information: Strategy and development (ARI Research Report 1676). Alexandria, VA:
U.S. Army Research Institute for the Behavioral and Social Sciences.

65



development of automated measures of performance that provide intrinsic and extrinsic feedback
is an imperative for training the Army’s future brigade-level and below forces equipped with
advanced digital C'I systems.

63



Throne, M. H., Holden, W. T., Jr., & Lickteig, C. W. (2000). Refinement of prototype staff
evaluation methods for future forces: A focus on automated measures (ARI Research
Report 1764). Alexandria, VA: U.S. Army Research Institute for the Behavioral and

Social Sciences.
U.S. Amy Training and Doctrine Command. (1997). The Standard Army After Action Review

System (STAARS): After action review (AAR) handbook, Version 2.1, 9 May 1997 [On-
line]. Available: http://www-leav.army.mil/nsc/stow/staars/handbook/htm.

67



team process skill dimensions identified earlier. The mission of the unit, a reinforced battalion
equivalent, was to defend against an attack by several OPFOR mechanized infantry brigades,
reinforced with Corps-level artillery. The area to be defended is approximated by the unit zone
of action, which is depicted on most of the pictures. Trial 1, which started at 0800 local time,
lasted approximately 7 1/2 hours.

Adaptability

CSS Locations

The picture for this measure (Figure 8) was designed to show key vehicle locations relative
to ammunition and fuel resupply vehicles. The platoon leader vehicle locations were used to
represent the location of all major combat systems in the battalion. The Project Team decided
that showing all of the vehicles would clutter the presentation. Normally, all of the vehicles
within a maneuver platoon are located within visual range of each other, so the location of the
platoon leader vehicle was thought to be an accurate representation of where the other vehicles
were located. Also shown are the ammunition and fuel resupply vehicles. The Project Team
used symbols for the resupply vehicles that were slightly larger than the platoon leader vehicle
symbols to accentuate their location and to keep them from being covered by another vehicle
type when they were plotted. Data for the vehicle locations were available at 5-minute intervals
and come from the same source that would be used to show vehicle unit marking, type, and
location on the unit member’s PVD. There were a total of 10 ammunition carriers and 6 fuel
carriers that could be depicted. Figure 8 depicts the locations of one ammunition and two fuel

carriers.

The rationale for this measure was to identify if the staff's mission planning had
incorporated sufficient flexibility to respond to unanticipated requirements, such as a platoon
running out of fuel or ammunition during heavy combat or at a critical point in the mission. If
the resupply vehicles were located where they were needed when they were needed, even from
an unforeseen requirement, an inference could be made that the staff had adequately planned

resupply.

At the time of this picture (approximately 1 hour after the trial started), there had been
insufficient activity to determine if the ammunition and fuel resupply vehicles were adequately
positioned to support the maneuver platoons. The fuel vehicles were in close proximity to the
platoons, but the one ammunition vehicle depicted was at least 10,000 meters or about 15
minutes travel time from the nearest platoon.

The picture does provide a quick view of where the resupply vehicles are in relationship to
the maneuver platoons within the unit. It does not provide the ammunition or fuel status for the
subordinate units, which would indicate if the resupply vehicles are positioned appropriately.
Adding subordinate unit supply indicators to the picture may improve its usefulness.

23



Table 2 (Continued)

Skill Dimension and Measure

Descripti
Name scription

Decision-Making
UAV Effectiveness Percentage of OPFOR vehicles first detected by the UAV under
squadron control for each UAV launch during each mission.
Note. BLUFOR = blue forces; CCIR = commander’s critical information requirements; FLOT =
forward line of own troops; FOV = field of view; OPFOR = opposing forces; PVD = plan view
display; SITREP = situation report; SPOTREP = spot report; UAV = unmanned aerial vehicle.
From Refinement of prototype staff evaluation methods for future forces: A focus on automated
measures, by M. H. Throne, W. T. Holden, Jr., and C. W. Lickteig, 2000, Alexandria, VA: U.S.
Army Research Institute for the Behavioral and Social Sciences.

Unfortunately, since the pictorial and graphical representations for a majority of the
automated measures could not be supported by the MMBL’s programming analysts due to time
constraints, the automated measures outputs provided were in tabular form (Throne et al., 2000).
However, the Project Team attempted to develop prototype pictorial formats for some of the
measures based on the data tables. These picture formats were all developed by the Project
Team using a commercial off-the-shelf spreadsheet software program, Microsoft® Excel. This
was a time-consuming process since iterative experimentation with the picture formats was
required to relate data tables to the operational context.

In summary, although a prototype automated measurement package of team process skills
was developed for DC‘I-2, true implementation of automated measures as defined by the Project
Team was not achieved. Nevertheless, considerable progress was made in matching Data
Collection and Analysis System (DCA) output data to team process skill dimensions and a start
was made on converting the measures data into automated pictorial representations of staff

performance (Throne et al., 2000).

Current Project

For the current project, the Project Team was directed to design 25 prototype automated
measures and then select 20 for development and implementation. These prototype measures
could either be new measures or measures refined from the DC'l-2 project, since the measures
from that project were not truly automated. The goal was to produce as many measures in
pictorial format as possible, since pictures are a good way to present complex measures of
command and staff performance in a manner that is simple to interpret (Meliza, Bessemer,
Burnside, & Shlechter, 1992). As mentioned previously, although this report focuses on the use
of automated measures, they should be used as one aspect of a complete measurement package.

An opportunity was provided to implement the prototype automated measurement package
during the MMBL’s Concept Experimentation Plan, FCC’, previously known as BCR. A
description of the FCC” including experimental objectives, is provided at Appendix B. The
Project Team had extensive experience operating in this environment and felt that there would be
significant advantages to continue to work with the MMBL. Among these are:



Measures Development

The Project Team’s goal was to develop fully automated measures of performance using
commercial off the shelf (COTS) software. In addition, the measures development process
would be presented through a user-friendly medium that would require very little if any
programmer expertise to implement. To accomplish this, the measures development process was
twofold: development of the measures themselves and development of a prototype observer
workstation through which observers could develop the desired measures for AARs. Both of
these efforts are described in more detail below.

Automated measures. Once the definitions and output formats had been specified for the
measures, the next step was to select a software package that could transform the data that would
be produced by the DCA into the desired format. For all the DC‘ efforts, the DCA output
provided to the Project Team has been in the American Standard Code for Information
Interchange (ASCII) format. The data in this format is readily imported into Microsoft® Excel,
which has been used to create tables and graphs. The Project Team decided to continue to use
Excel as the measures creation tool for several reasons. Excel has an integrated chart wizard that
could produce the majority of the desired output formats without a need for extensive
programming. The chart wizard would work in conjunction with a macro recorder so that the
same type of chart or graph could be repeatedly generated using different data sets without
having to go through a step-by-step process for creating each output. Changes to the design of
the measure output format would be easy to make. Charts and graphs created with the Excel
wizard could be further customized using the Microsoft® Visual Basic® for Applications
programming language, which is included with Excel. The output formats could then be
seamlessly moved to other Microsoft® Office products, such as Word for reports or PowerPoint®
presentation graphics program for presentations.

Once the Project Team had settled on using Microsoft® Office as the baseline software
package, development of the measures was initiated using BCR IV data available from the DC‘I-
2 project. Early on, it became apparent that additional data would be required to support the new
measures that had been designed for the current project. The Project Team generated a listing of
the data elements that would be required to support each measure. Coordination then was
effected with the MMBL to obtain the required additional data. Based on the Project Team’s
experience with BCR IV data, some initial decisions were made to facilitate data handling and
subsequent analysis: vehicle locations were to be provided at 5-minute intervals; status of
ammunition and fuel were to be provided at 20-minute intervals; only weapon system kills were
to be reported; engagements that resulted in misses or less than a catastrophic kill would be
excluded; only OPFOR weapon systems that were detected by the unit and its subordinate
elements would be reported; and the MMBL would package the data into three separate files to
prevent the mixing of incompatible data types.

As the Project Team began to work with the additional data, it became apparent that the size
of the data files would exceed the capabilities of Microsoft® Excel. For example, in the sample
BCR IV data set, an average mission contained 175,000 rows of data with approximately 50
columns (i.e., variables). Excel could only handle approximately 66,000 rows of data.

15



Draft definition and output format
Categorize by team process skill dimension
Meet with the SAG to select measures

Specify definition and output format for selected measures

4.
5.
6.
7.
8.

9. Assess input and output

Develop software code and run on previous BCR data files

10. Repeat steps 7 through 9, as required
11. Develop Observer Work Station
12. Add measures and outputs to DCA Library

The front-end analysis stage consisted of steps 1 and 2; the measures design stage consisted
of steps 3 through 7; and the measures development stage consisted of steps 8 through 12. The
steps are discussed by the stages in more detail below.

Front-End Analysis

The front-end analysis of the measures development began with a look at the lessons learned
from the previous project (Throne et al, 2000). These lessons learned provided direction for the
refinement of the automated measures implemented in the previous project as well as ideas for
new measures that might provide meaningful feedback. A brief summary of the lessons learned
is provided below.

One lesson learned was that an iterative process for designing, developing, testing, and
refining automated measures is required to get useful results. Programmers, SMEs, and
researchers need to follow up on all measure outputs to make certain the end product meets
everyone’s expectations. For the current project, iterative collaboration was more fully
integrated into the design and development process.

Another important point is that data reduction is a time-consuming process. During the
design and development process, the Project Team had projected that various summary tables
would provide the desired results with little additional processing. However, in analyzing the
initial results, the Project Team needed to refilter the data tables or go back to the raw data to get
the desired measure output. There is a trade-off between getting the most meaningful measures
output and having a fully automated measures development system. In a fully automated
measures development system, data reduction would not be as time-consuming, yet the data used
may not always provide the most meaningful output. One of the main purposes of this project
was to more fully automate the measures development and output process.

After incorporating the lessons learned into the measures development process, the Project
Team identified potential measures for development. The team looked for measures that could
be automated and focused heavily on measures that could be delivered in a pictorial format. The
ARI requirement was to design 25 automated measures, with at least 4 from the C? measures
outlined in the Standard Army After Action Review System (STAARS) After Action Review (AAR)

11



130000 7

420000 |} ——-}

110000 }~[Unit Zone
lof Action

400000 — i:

80000 — - a T

0000

70000 -
60000 ~ < —

50000 | T

‘40000 - ~

30000

20000 +——j —
OPFOR Movement
10000} —} 7 —}- NTL
° | i I i
‘© 10000 20000 30060 40980 $0090 e000 70000 e600 90000 00000 110000 120000 ¥30000 140000 160000 1E0000 7eGoD 1eeo00

[Zone M Ammo _® Fuel _TM Supported Units)

Note. OPFOR = opposing forces.
Figure 8. Combat service support locations at 0900.

Node Locations

The picture designed for this measure (see Figure 9) shows the location of the unit’s
commander and staff vehicles (nodes) in relationship to the location of vehicles of the maneuver
platoon leaders, scout platoon leader, and platoon sergeant. The experimental staff in FCC’,
including the commander, comprised 14 officers and non-commissioned officers located in four
vehicles. The commander had two assistant staff officers in his vehicle as did the deputy
commander in his. The third staff vehicle (Control 1) was primarily concerned with controlling
and monitoring the unit’s indirect fires and engineer support. The fourth staff vehicle (Control 2)
was responsible for monitoring the unit’s logistical situation and controlling maintenance and
resupply activities during FCC’. The Project Team used symbols that were slightly larger than
the platoon leader vehicles to denote the Control 1 and Control 2 vehicles. The Commander
(Cdr) and Deputy Commander (DepCdr) vehicle designators were increased in size over the
control vehicle symbols so that if the command vehicle were collocated with the control vehicles
they could be differentiated. Data for the vehicle locations were available in 5-minute intervals
and come from the same source that would be used to show vehicle unit marking, type, and
location on the unit member’s PVD.

The rationale for this measure was that the location of the unit’s command and control nodes
may indicate the ability of the staff to handle different requirements simultaneously while
keeping positioned to maintain communications with all subordinate elements, and maintaining
operational and physical security.

24



Table 1

Cannon-Bowers et al.’s Team Process Skill Dimension Definitions

Team Process Skill Dimension Definition

Adaptability The process by which a team is able to use information gathered
from the task environment to adjust strategies through the use of
compensatory behavior and reallocation of intrateam resources.

Performance Monitoring and The ability of team members to give, seek, and receive task-

Feedback clarifying feedback; includes the ability to accurately monitor the
performance of teammates, provide constructive feedback
regarding errors, and offer advice for improving performance.

Shared Situational Awareness The process by which team members develop compatible models
of the team’s internal and external environment; includes skill in
arriving at a common understanding of the situation and applying
appropriate task strategies.

Leadership/Team Management The ability to direct and coordinate the activities of other team
members, assess team performance, assign tasks, motivate team
members, plan and organize, and establish a positive atmosphere.

Interpersonal Relations The ability to optimize the quality of team members’ interactions
through resolution of dissent, utilization of cooperative behaviors,
or use of motivational reinforcing statements.

Communication The process by which information is clearly and accurately
exchanged between two or more team members in the prescribed
manner and with proper terminology; the ability to clarify or
acknowledge the receipt of information.

Coordination The process by which team resources, activities, and responses are
organized to ensure that tasks are integrated, synchronized, and
completed within established temporal constraints.

Decision-Making The ability to gather and integrate information, use sound
judgment, identify alternatives, select the best solution, and
evaluate the consequences.

Note. Adapted from Cannon-Bowers et al. (1995), pp. 344-346.

Once the framework was selected and the processes to be measured were defined, the
Project Team needed to define the phrase “automated measures.” However, very few
researchers had developed automated measures and most of those who had were measuring
either individual performance or outcomes of team performance, not processes. Those few who
had measured team performance automatically (e.g., Atwood et al., 1991; Leibrecht, Meade,
Schmidt, Doherty, & Lickteig, 1994) had not provided a specific definition of what constitutes
an automated measure of team performance. Therefore, based on the literature reviewed, the
Project Team developed its own definition. Automated measures were defined as measures
based on data collected from a C*I system and processed automatically by preformatted routines
to provide meaningful training and performance assessment feedback with as little observer input
as possible. The C*l system data was further defined to include, but not limited to, voice



. Degradation of Forces — Depiction of the relative combat power of maneuver forces over

time for both OPFOR and BLUFOR during each mission.

a. Operational definition: Cumulative rate of OPFOR destruction in 20-minute intervals
from the first engagement until the last OPFOR kill for each mission. In addition,
cumulative rate of BLUFOR destruction in 20-minute intervals from the first engagement
until the last BLUFOR kill for each mission.

b. Rationale: Rapid destruction of the OPFOR reduces the risk of losses to the friendly unit.
The rate of loss also may indicate battle tempo during the mission and whether the staff is
coordinating the efforts of the unit to inflict the maximum number of losses in the
shortest period of time.

c. Qutput: Line graph

. Counterreconnaissance Effectiveness — Effectiveness of the BLUFOR counterreconnaissance
effort during each mission.

a. Operational definition: Location where each OPFOR reconnaissance asset was detected
and the location where each detected OPFOR reconnaissance asset was killed

(catastrophic, firepower, or mobility).
b. Rationale: May indicate if the staff was coordinating fires effectively to negate the

OPFOR ground reconnaissance effort. If the OPFOR cannot detect the unit, the unit
reduces its vulnerability. OPFOR ground reconnaissance vehicles are high priority,
HPTs that, doctrinally, should be among the first targets attacked after they have been
detected.

c. Output: Picture

. Artillery and Counterfire Radar Coverage — Depiction of BLUFOR artillery templates (by

unit or type) overlaid on the location of the unit’s subordinate elements at a specified time.

a. Operational definition: Location of artillery systems on the battlefield and their range
templates at critical points during each mission (start of exercise, first indirect fire
engagement with OPFOR, first direct fire engagement with OPFOR, first friendly
casualty, friendly losses exceed 30%, and last engagement during mission).

b. Rationale: May indicate whether the staff is effectively coordinating the indirect fire
assets it directly controls to support both its subordinate units and the commander’s intent
and concept of the operation. Typically, indirect fire weapon systems, while
geographically dispersed, are positioned where they can mass timely fires at decisive
points in the operation and attack targets of opportunity while reducing their exposure to
OPFOR counterartillery fires.

c. Output: Picture

. Kill Distance — Distance between shooter and target.

a. Operational definition: Distance between OPFOR/BLUFOR weapon system killed and
BLUFOR/OPFOR system that killed it. Categorize data by indirect weapon and direct
fire systems separately. Scatterplot with a trend line will reveal how far the killed
systems were from the system that killed them.

b. Rationale: If the staff was effectively coordinating the fires of the weapons it directly
controlled, then the majority of the kills they obtained should be nearer the maximum
effective range of the weapon rather than the minimum effective range of the weapon.



Leibrecht, B. C., Meade, G. A., Schmidt, J. H., Doherty, W. J., & Lickteig, C. W. (1994).
Evaluation of the combat vehicle command control systems: Operational effectiveness of
an armor battalion (ARI Technical Report 998). Alexandria, VA: U.S. Army Research
Institute for the Behavioral and Social Sciences.

Mason, D. E. (1995). Identifying measures of effectiveness for Marine Corps C4] systems.
(Master’s Thesis). Monterey, CA: Naval Postgraduate School. (ADA305269).

Meliza, L. L. (1996). Standardizing army after action review systems (ARI Research Report
1702). Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social

Sciences. (ADA322044).

Meliza, L. L., Bessemer, D. W., Burnside, B. L., & Shlechter, T. M. (1992). Platoon-level after
action review aids in the SIMNET Unit Performance Assessment System (ARI Technical
Report 956). Alexandria, VA: U.S. Army Research Institute for the Behavioral and

Social Sciences.

Meliza, L. L., Bessemer, D. W., & Tan, S.C. (1994). Unit Performance Assessment System
(UPAS) development (ARI Technical Report 1008). Alexandria, VA: U.S. Army
Research Institute for the Behavioral and Social Sciences.

Morrison, J. E., & Meliza, L. L. (1999). Foundations of the after action review process (ARI
Special Report 42). Alexandria, VA: U.S. Army Research Institute for the Behavioral

and Social Sciences.

Mounted Maneuver Battlespace Lab. (1998). Battle lab experiment plan (BLEP) for battle
command reengineering III (BCR III). (Available from Mounted Maneuver Battlespace
Lab, ATTN: ATZK-MW, Fort Knox, KY 40121)

Salas, E., Bowers, C. A., & Cannon-Bowers, J. A. (1995). Military team research: 10 years of
progress. Special Issue: Team processes, training, and performance. Military
Psychology, 7, 55-75.

Salas, E., Dickinson, T. L., Converse, S., & Tannenbaum, S. I. (1992). Toward an
understanding of team performance and training. In R. W. Swezey and E. Salas (Eds.),
Teams: Their training and performance (pp. 3-29). Norwood, NJ: Ablex Publishing
Company.

Throne, M. H., Deatz, R. C., Holden, W. T., Jr., Campbell, C. H., Sterling, B. S., & Lickteig, C.
W. (1999). Prototype staff training and evaluation methods for future forces (ARI
Research Report 1745). Alexandria, VA: U.S. Army Research Institute for the
Behavioral and Social Sciences.

66



The results may indicate that, even with the inherent capabilities of the sct system to
provide a visual display of all friendly and detected OPFOR weapon systems locations that is
being constantly updated, the closer the forces move toward each other, the less valuable that
information is in protecting the force. For each side, the losses are surrounded by losses from the
other side. A trained observer may conclude that the two forces had intermingled and/or
bypassed each other which created or increased the vulnerability to flank or rear engagements.

20000

T T_ ~

bess

toe00 |=.

‘1900 | —

Sr Movernent

| | i
tom0 [ Note: Depicts LOS (< 4000 Meters) Kills — |
Lit tt Ld

0000 60000 7o000 econo | ‘00000 119000 #0000 130000 140000 150000 180000 170000 180000

Note. LOS = line of sight; OPFOR = opposing forces.

Figure 28. Surprise attack.

Figure 29 graphs the flank or rear engagements as a percentage of the total engagements
against the BLUFOR experimental unit. Twelve percent of the engagements against the unit
were from the flank or rear. Additional analysis of these engagements during an AAR by the
unit participants may provide the reason for these losses. Figure 30 provides the same
information about engagements against the OPFOR. While the total number losses for the
OPFOR was greater than the unit’s losses, the percentage difference is not nearly as great.
Again, additional analysis is required to explain why the unit was not more effective in
maneuvering forces or fires once the OPFOR had closed to within 4,000 meters of their

locations.

50



CONTENTS (Continued)

Page

Decision-Making .........cscecsessessesseesesseesessesseesesssesessessessesssecaesnessesseasseeseeseenseuseeseneenseneeneaee 40
Sensor Coverage .........0004 40
Multiple Fire Engagements. 41
Maneuver Battle Sets .... 42

CSS Available Over Time. 43
Performance Monitoring and Feedback 0... .cssesessesesteseseseeneseceeseneseeseseeneseenesnenesnensaeesesese 44
Effects of Targeting .. 44

Shared Situational Awareness . 4T
Map Area 47

Surprise Attack 49

Fire Support Coordination 52

Air Defense Coverage 53
Fratricide/Collateral Damag: 54
Common Picture 57
COnCIUSIONS ......c.sceseesesseseesessessesesuessessesuessssecsessecseesecavesecsesessussucsienesseeneeneeseeniesusenesneaneeneene 59
Lessons Learned: Improve ......c.ccssssesssscesessessssessesessessssssssscacesacsssnssesesnsacanensnesasansnsneaneateneneenease 59
Observer Input.. 59
Data File Size 59
Missing Data.. 60
Data Manipulation .........c.cssssssessessesessescssssescsscsnessssenssnsucassnencaesnsusaesneneaneneaneneaeeaeaneneavessaeenes 60
Lessons Learned: Sustait.........ccscssssessssesssssssesssesssesssssssansneacecsssncnesescaeseneaueaeseaesneneeneneaeeneanes 61
Data Visualization 61
Commercial Business Software 61
User Tools Development 61
Future Staff Performance Standards. 62
Pre-Planned Measures Implementation. 62
Summary. 62
References 65



definition for Map Area is provided along with a sample of what the measure output will look
like once it is developed. The Map Area measure can only be produced in picture format;
however, some of the other measures have more than one format available and the user can click

on the preferred format.

If the user chooses the complete list of available automated measures (see Figure 5), the
program displays all 20 of the available measures in alphabetical order. When the user selects
one of the measures, the program will go to that particular measure, as shown in Figure 7. The
list of all available measures is a shortcut for those users who are already familiar with the
measures and know which ones they want to develop.

Once the user selects a measure by clicking on it, Access automatically opens and the form
seen in Figure 2 comes up. The user again selects the desired measure and the output for that
measure is exported to Excel. From there, the user again selects the desired measure and its
output format. The measure output is then created and saved as an Excel file.

The Observer Workstation is designed to be flexible and user-friendly. It allows the user to
go almost anywhere within the program at any given time. For example, if the user selects the
wrong skill dimension from the main menu, he or she can go back to the main menu and choose
another skill dimension or go to the list of available measures. From the list of available
measures screen, the user can choose to go to the main menu of skill dimensions. If the user
selects a particular measure and decides not to develop it, he or she can go back to other
measures within that skill dimension, the main menu of skill dimensions, or the list of all

available measures.

Measures Implementation

The 20 developed automated measures along with the Observer Work Station (OWS) were
implemented during the MMBL’s FCC? experiment. The workstation was physically located in
the data analysis area where the Project Team could readily coordinate with the MMBL
programmers on FCC’ DCA data file issues. It was also linked to the MMBL’s intranet so that
data files could be electronically downloaded. The Project Team had access to an SC* system
where zones of action, OPFOR strengths, and other pertinent data could be extracted.

As the measures were tested during initial exercises, a number of changes had to be made.
The biggest change required modifying the measures’ software code to complete processing
when a DCA source file unexpectedly did not contain any data. This condition, which did not
occur in pre-implementation development testing, was a result of changes in unit organization
and equipment that the Project Team had not anticipated prior to the start of the experiment.
Processing of raw DCA data files for input to the OWS also took longer than expected. In the
case of a full mission’s data set, it took approximately 2 hours to produce the ASCII files needed
for the measures. It then took another 30 to 45 minutes to import the data into Access hosted on
the OWS. The net effect was that sample outputs from the measures were not available in time
for use in AARs during the experiment. Some sample results were shared with MMBL
personnel and contractors during the experiment to gain some informal feedback on formatting.

21



130000 ~ TIT]
#20000 a
110000 + —Unit Zone} a ae
lot Action
109000 - —— -
90000 - —
20000 +-— {-— Sane OPFOR Movement
7000 - a,
60000 -
50000 |} aa
40000 ee a ee
‘90000 deep — i
{ i
20000 | -——| - - }+—|——}—
30009 |—— i
Note: Tracks Pair Shooters with Targets
o! | ara facia
‘© 0000 20000 30000 40000 0000 60000 70000 e000 90900 100000 110000 120000 130000 140000 150000 160000 176000 180000

Note. OPFOR = opposing forces.
Figure 35. Fratricides.

Common Picture

To develop this measure (see Figure 36), the Project Team first obtained the location of
every vehicle in the unit. This location is plotted using a red or blue outlined diamond or square.
The vehicle was assigned the “Stationary” label. If there were not stationary vehicles, such as
the case for the OPFOR, then Excel assigned the “Series” label. The location of the same
vehicle 20 minutes later was then plotted. The 20-minute interval was used for two reasons:
first, the Project Team thought that vehicle movements of less than 20 minutes in duration would
not be visually detectable at the scale of terrain data being presented; and second, OPFOR
information was only available in 20-minute intervals. This location was plotted in a solid red or
blue diamond or square, and labeled as “New/Moved.” This was the same category used to
account for vehicles that may have been added to either the OPFOR or the unit since vehicle
positions had been last recorded. If a vehicle was being reported as destroyed, it was plotted in a
black diamond or square, and labeled “Dead.”

An indicator of how much information the staff and others in the unit are sharing among
themselves would be an analysis of the changes over time in the information that is being
displayed on an individual soldier’s PVD. This information would be a common starting point
for most staff actions such as providing estimates, making recommendations, alerting the
commander to changes in the situation, or developing courses of action. To start that analysis,
the first step was to determine what information changes. The Project Team focused on the
locations and status of detected OPFOR and unit vehicles since changes in locations and status
would be readily discernible on a PVD.

57



Conclusions

During FCC’, the Project Team implemented 20 measures, 19 of which were automated in
that, once an initial set of criteria had been determined (such as the critical weapon systems to
track, the number of unit and OPFOR weapons participating in the exercise, and unit zones of
action), the output formats could be generated from trial data without further observer
intervention. The orders distribution measure requires additional development to automate it.
Further refinement is required to improve the utility of some of the measures as discussed under

the particular measures.

Lessons Learned: Improve

This project built on the lessons learned from the previous projects and more fully realized
the development of fully automated measures of command and staff performance. The
implementation of the automated measures during FCC? provided an opportunity to test
generating automated measures output formats in training conditions similar to those that are
envisioned for future brigade-level and below battle staffs equipped with advanced digital Cc
systems. However, there are still issues that need to be addressed before these automated
measures can become a meaningful part of training feedback in an AAR-type environment.

Observer Input

A key role for an observer in any training event is determine before hand what information
will be needed to provide meaningful performance feedback. Even training feedback systems
that rely heavily on automated measures need someone to determine which measures from the
available library will apply to the specific event and to review the outputs to determine if the
results should be incorporated into an AAR. Observations from trained observers are also
required to put many of the measure outputs into context. All of the conditions involved with
staff performance during a training event cannot be automatically derived. For example, while
the amount of map area that is being displayed on a staff member’s PVD can be determined, the
data cannot reveal whether the staff member was actually looking at the display or attending to
other duties, which may have caused him to ignore the information being displayed. A staff
member simply could have left the display on and exited his vehicle. Also, some measures, such
as fratricide and collateral damage, invariably will require a detailed examination of all the
circumstances involved with those incidents. The measure output alerts the training participants
that a problem exists, but does not get to the root cause. Without objective command and staff
performance standards, all automated measures will need to be interpreted by SMEs who can
provide standards based on their knowledge and experience.

Data File Size

After researching the capabilities in Microsoft® Excel, the Project Team realized that the
quantity of records that would be produced during an FCC? trial could not be manipulated in
Excel as planned. Recording 1,000 vehicle positions every five minutes for an eight hour trial
produced 96,000 records which exceeded Excel’s data handling capabilities. This caused the
Project Team to go from Excel to Microsoft® Access. Adding an extra software package into the



equation led to some minor difficulties in the design of the observer workstation. Instead of
pushing a single button in order to develop a measure as planned, the user has to push a button in
each software package. This leads to redundancy for the user as well as longer measure output
development time. Additionally, more data tables were required than were planned. For
example, the largest MMBL ASCII data file was eventually split into three different files so the
information needed to build measures could be accessed more easily. Additionally, the sheer
size of the files associated with FCC’, Access databases in excess of 51 Megabytes and some
Excel output files exceeding three Megabytes, created problems in transferring, copying, and
printing operations. All of these challenges in handling data are essentially technology driven
and can be overcome with more capable computer and communications hardware and software.

Missing Data

Working with the initial set of FCC? data, the Project Team discovered that there were
numerous queries that were producing files with no data. This condition was not expected since
it had not occurred during pre-implementation testing. As a result of having blank query files,
the Microsoft® Excel macros would generate run-time errors which terminated further data
processing necessary to create the output formats. To overcome these errors, the Project Team
encoded the Excel macros to ignore empty data files. The Project Team also compared the raw
ASCII data files with the FCC? Experiment naming convention for identifying unit, vehicles, and
weapon systems. Several discrepancies were found which resulted in the Project Team changing
queries to reflect actual conditions. Finally, the Project Team, based on input from MMBL
personnel, determined that the output formats were not reflecting all activity that was occurring
during the trials. An examination of the MMBL ASCII data files revealed that the file that
matched vehicle locations with engagements was not catching every engagement. This file was
then split by MMBL programmers into two data files which corrected the problem. This, in turn,
led the Project Team to revise all queries requiring engagement data. With these changes,
problems with empty data files were eliminated.

Data Manipulation

The goal for this project was to have the outputs from the automated measures to be
available for use during the FCC? AARs which would be held each day. Based on previous
experience, the Project Team had projected that it would take about two hours to produce the
complete set of output formats. This would allow them to be used during FCC’ AARs, which
generally took place about two hours after the last significant combat action during a trial.
Unfortunately, the amount of time to create the output formats greatly exceeded that estimate. It
took an average of two hours of processing time for the software used by the MMBL to
consolidate the data and produce the initial data tables for each trial; it took another 15-30
minutes to convert these data files to ASCII and post them to an MMBL intranet web page where
the Project Team could access them; it took another 30-45 minutes to import the ASCII files into
Microsoft® Access; and finally, it took an additional 45-60 minutes to build the measures, add
titles, and save the output formats. The total amount of processing time from start to finish
exceeded four hours. As a result, the results where not available for use during the FCC? AARs.

60



If detailed explanations are required for users to understand the purpose of the picture, it will be
of no value to exercise participants in an AAR (Meliza, 1996).

The next step was to provide a user-friendly medium through which this complex process
could take place and where the final product could be either printed out or entered into a slide
presentation to be used during an AAR, similar to an “AAR Presentation Manager” (Meliza,
Bessemer, & Tan, 1994, p. 45). The Project Team decided that the best way to accomplish this
goal was to develop a prototype Observer Workstation. The development process for the

prototype is described next.

Observer Workstation. Figure 3 displays the entire measures development process and
where an Observer Workstation would fit into the progression. The process starts with the
information gathered in the MMBL by the data logger and processed through the DCA. These
refined data combined with information inserted by observers provide the information needed by
the Observer Workstation to create the measures automatically. Once the measures were
developed through the Observer Workstation, they could be used during AARs as well as for any
reports that required analysis of staff performance.

instrumented SC4

Voice
System
yan _| Observer Workstation

‘Communications

Electronic

Data
Collection
Analysis

Basic Data Tables |

Situational
‘Awareness

Refined Data
Tables

5 af

a io

‘SC Workst
Note. AAR = after action review; BLEFR = Battle Lab Experiment Final Report;

FCC2 = Future Combat Command and Control; PDU = protocol data unit; sct= surrogate
command, control, communications, and computers.

Figure 3. Automated measures development process.

Using the Microsoft® PowerPoint® presentation graphics program, a prototype graphical
user interface (GUI) was built as a means for user-friendly measure output format creation. This
GUI is what the user saw when interacting with the Observer Workstation. Sample screens can
be seen in Figures 4 through 7.

18



Table B-1 shows the call signs and associated node positions for the commander and 13
primary staff members. In this report, these participants will often be identified by their call
signs, especially in tables that appear in the Results section.

Table B-1

Staff Member Call Signs and Titles

Call Sign Title Node

Cougar6 Squadron Commander Command 1

Cougar62 Enemy Operations Officer Command 1

Cougar69 Effects Officer Command 1

Cougar3 Deputy Squadron Commander Command 2

Cougar32 Operations NCO Command 2

Cougar35 Operations Officer Command 2

Outlaw6 Battle Captain Control 1

Outlaw2 Enemy Operations NCO Control 1

Outlaw3 Friendly Operations Officer Control 1

Outlaw59 Sensor NCO Control 1

Headhuner6 Battle Captain Control 2

Headhunter2 Enemy Operations NCO Control 2

Headhunter3 Friendly Operations Officer Control 2

HeadhunterS9 Sensor NCO Control 2

Eagle6 Company Commander Constructive Simulation
Fox6 Company Commander Constructive Simulation
Ghost6 Company Commander Constructive Simulation
Hawk6 Company Commander Constructive Simulation

Note. NCO = non-commissioned officer.

Other squadron personnel involved in the experiment included: six company commanders,
six deputy company commanders, six maneuver platoon leaders, one scout platoon leader, one

scout platoon sergeant, nine gunners, and 15 drivers.

Materials

The FCC? used emulation as well as constructive and virtual simulators. Figure B-2 shows
the layout of the SC* system in the BCVs and SOVs. The SC‘ system included the following

capabilities:

* Command and Control (C’) Plan View Display (PVD), represented by the Modular Semi-
Automated Forces (ModSAF) two-dimensional PVD. On the PVD, the commander and
the staff are able to view movements of all of their own systems, as well as any opposing
force (OPFOR) units detected by satellite or other sensors. Overlays can be drawn on the
PVD, users can add labels or other notes, and there are tools that show past events and
project future movements.

B-2



PROTOTYPE AUTOMATED MEASURES OF COMMAND AND STAFF PERFORMANCE

EXECUTIVE SUMMARY

Research Requirement:

This research and development effort continues the work by the U.S. Army Research
Institute for the Behavioral and Social Sciences (ARI), Armored Forces Research Unit, Future
Battlefield Conditions Team. It focuses on the design and development of automated training
and performance evaluation techniques. A primary context for these efforts is digital brigade
and below training requirements and environments. For this project, ARI’s objective was to
design, develop, and demonstrate 20 prototype automated measures to improve training and
evaluation for brigade and below command and staff performance.

The prototype automated measures developed were implemented during the Future Combat
Command and Control (FCC?) Concept Experimentation Program experiment conducted by the
Mounted Maneuver Battlespace Lab (MMBL) at Fort Knox, Kentucky. The ARI’s purpose for
participating in this experiment was to support the MMBL and the Army’s need to gain
additional information on future staff evaluation requirements in a virtual simulation
environment, and gather feedback for improvements to the prototype automated measures
developed during this effort.

Procedure:

The Project Team reviewed The Standard Army After Action Review System (STAARS)
handbook (U.S. Army Training and Doctrine Command, 1997) and previous research literature
regarding staff processes and measures developed to assess them, especially automated
measures. This literature review provided the basis for decisions concerning staff processes to
measure, as well as guiding the procedures that the Project Team would use to design, develop,
implement, and analyze the measures. A total of 29 candidate automated measures were then
designed and presented to a Subject Matter Expert (SME) Advisory Group (SAG) for review.
The SAG selected 20 measures for development by the Project Team. The Project Team then
built a prototype observer workstation to facilitate the use of the automated measures by trainers
and other personnel during after action reviews (AARs).

The prototype automated measures were implemented during the FCC? experiment, which
took place 7 May through 24 May 2001. The experiment was conducted in the MMBL Mounted
Warfare Test Bed at Fort Knox with the 24 Squadron, 2"TM Armored Cavalry Regiment from Fort
Polk participating. The Project Team demonstrated 19 of the 20 automated measures it had
designed and developed using commercial business software in common use throughout the
Army. Development of one measure was not completed during the project.

vii



ACKNOWLEDGMENTS

This report represents the efforts of an integrated team of research scientists, military
experts, training developers, simulation technology experts, and administrative support
personnel. The team members directly supporting the U.S. Army Research Institute for the
Behavioral and Social Sciences included: military subject matter expertise from Mr. Neff
Jenkins, Mr. Mike Flynn, Mr. Bud Dannemiller (Litton PRC), and Mr. Mike Cobb (Human
Resources Research Organization [HumRRO)); programming expertise from Mr. Mitch Bonnett
(Litton PRC) and Mr. Scott Shadrick (HumRRO); and Ms. Kathleen Horn (Litton PRC) who
provided administrative support. Ms. Charlotte Campbell (HumRRO) was the Program Manager
for this project. Her contributions went beyond project supervision. She was involved in all
project decisions, contributed her expertise as a Research Psychologist, and was a reviewer of

this report.

Additionally, we had support and guidance from a variety of individuals and government
organizations, including:

« Mounted Maneuver Battlespace Lab, Fort Knox, Kentucky

Colonel Dennis Szydloski, Chief

Major Joe Burns, Chief, Advanced Concepts Research
Major Dan Ray, Concept Integration Officer

Mr. Joe Jarboe, Systems Analyst

¢ Illinois Institute of Technology Research Institute (ITRI/AB Technologies

Mr. Pat Ritter, Program Director for Research
Mr. Jim Lewis, Senior Systems Analyst

e Lockheed Martin-Marietta
Mr. Don Appler, Site Manager
Mr. Dan Schultz, Battle Master

Mr. Paul Monday, Analyst
Mr. Paul Colonna, Analyst

vi



communications, simulation protocol data units (PDUs), electronic messaging, CI system usage,
situational awareness information, and observer input.

The first step in the design process was to select candidate measures for development. For
each of the six team process skill dimensions chosen, multiple candidate measures were
identified in order to obtain supporting data on that particular skill dimension. Candidate
measures were selected from those implemented during the BCR III, measures developed by
other researchers (e.g., Dzindolet et al., 1998; Mason, 1995), or based on the lessons learned
from BCR III discussed in Throne et al. (1999). All candidate measures were designed so their
outputs could be displayed in at least one of these three formats: tables, graphs, and pictures,
with an emphasis placed on pictures.

Once the candidate measures were designed, they were passed to the Subject Matter Expert
(SME) Advisory Group (SAG) for review and approval. The SAG was a committee consisting
of participants from the MMBL, the MMBL Core Support Group, ARI, and the Project Team.
This group was organized during DC‘I-2 in an attempt to include programmers, SMEs, and
researchers in the entire measures design and development process. Based on the SAG’s
feedback, the Project Team met with MMBL system programmers to begin developing the
higher priority measures. After meeting with the programmers, some of the priority measures
were considered to be too time-consuming or expensive to develop, given the BCR IV time and
cost constraints. Of the 30 candidate measures, 19 measures were selected as feasible for
development, given the BCR-related costs and time constraints. These measures are summarized

in Table 2.
Table 2

Automated Measures Developed for Battle Command Reengineering IV by Skill Dimensions

Skill Dimension and Measure

Name Description
Adaptability
Terrain Analysis Amount of time each duty position uses each of the following tools:

Stealth Control, Terrain Intervisibility, FOV, Snail Display, and FLOT
Display during the mission.

Performance Monitoring and Feedback

SITREP Use Frequency and duration of use of the SITREP tool during each mission
by duty position.

SPOTREP Use Frequency and duration of use of the SPOTREP tool during each
mission by duty position.

CCIR Frequency and duration of use of the CCIR tool during each mission

by duty position.
Information Retrieval by — Type and frequency of information the Commander retrieves on his
the Commander own that other staff normally retrieve for him.

(table continues)



PROTOTYPE AUTOMATED MEASURES OF COMMAND AND STAFF PERFORMANCE

CONTENTS
Page
Tntroduction......scssssssesseesessesseseesesssssssscsscssesecssscnssussnsesesussussnsanesecsesassessussuceseaseessesseseessesseneeneeneaes 1
Organization of the Report 3
Background 4
DC'‘I Research... 4
DC'l-2 Research 5
Current Project......cccccscsscessesessessesseeseeseesssseesesssessesessessseseeseasessssnsssssrssnssssssssseesesseaseneeueenenaee 9
Method oeccecccceeccssseseessessessessesssesesseessesssassnssnsesesssessenecesesssnesseesssesseesesseessesessesseeesssasssssessecsseavenee 10
Measures Development Process .......sssssssssessseseeseeseessesessessesseensensesescsnsenecsessesseeneeanssesseeness 10
Front-End Analysis 11
Measures Design.... 12
Measures Development. 15
Measures Implementation ..........scsscsessesseeseeseesessesseeseeseesesssessssesnesnesecssssneeseesnesesseesessesseenens 21
Results and Discussion .........csessecssssesssssessscsssssessesesssssscnsssessssseseseseeseeseesersnesesnsenssseeesseeseseeaee 22
Adaptability .......cccccscsessestesseseessssseseeseesnsencssessssesseessssuessssesseanssnscnssncenssnecasesesoesateseenenesseese 23
CSS Locations.......cceccesssecesssssesssessessseessseesesssssseseeseesessseseessseesssssessessssessssssesssseeenearenes 23
Node Locations ........ccsesssssssessessesseessesecsseseesecseeseesecsesssesssnssuseneseesseesssseesneeesseessenesueseens 24
Communication ......ceseecsecsersesseceesecseeseesecseestesteseeseesusseseeseseeseenseseaseesssseeseesessecseesesseaseanees 25
Overlay Use....cecccecsecsessesssseesessesseesessesseeseesseseesessessessesseesssnssucsscsessisssneseasessseseasesneanenees 25

Coordination ..

Damage to BLUFOR/OPFOR System:
Degradation of Forces...........
Counterreconnaissance Effectiveness...
Artillery and Counterfire Radar Coverage.
Kill Distance wo. ccceeseesesteeesneeeenees
Sensor-Shooter Time Lag w..cecescessesesseeeeeeee



c. Qutput: Line graph and/or table

6. Sensor - Shooter Time Lag — Time between first detection of OPFOR HVT/HPT by

BLUFOR and when the OPFOR HVT/HPT was killed during each mission.

a. Operational definition: Amount of time between first detection of an OPFOR HVT/HPT
and when it was killed. Also, location where they were first detected connected to the
location they were killed. HVT/HPT targets will be identified before the start of the
exercise. Normally, they might be C? nodes, air defense artillery (ADA) systems,
artillery systems, tanks, and key logistic systems.

b. Rationale: The shorter the time interval between detection and kill, the better the staff
may be in coordinating fires.

c. Output: Picture and/or bar graph

Decision-Making

1. Sensor Coverage - OPFOR vehicles detected as well as those not detected by BLUFOR
sensors during each mission.

a. Operational definition: Location of OPFOR vehicles on the battlefield that are first
detected by squadron assets. Also include location of OPFOR vehicles on the battlefield
that are undetected.

b. Rationale: Ifthe staff is not properly deploying and monitoring performance of their
sensors, then the information that is being displayed to the commander will be
incomplete, which will prevent him from making a decision using all information that
could be made available to him.

c. Output: Picture

2. Multiple Fire Engagements - Number of OPFOR vehicles which were engaged multiple
times during each mission.

a. Operational definition: For each OPFOR target killed, calculate the total number of
additional engagements against it by BLUFOR weapon systems after it was killed. Data
collected should indicate the firing unit, echelon of its controlling headquarters, the type
of weapon used, the engagement time, and should indicate at which time the OPFOR
target was killed (firepower, mobility, and/or catastrophic). For each OPFOR target
engaged multiple times, show its location on the battlefield and the location of the units
that continued to fire on it after it was killed.

b. Rationale: The data on multiple engagements of an OPFOR target may indicate whether
the situational awareness provided by the SC* system and an effective unit fire
coordination and distribution system reduced or prevented the needless expenditure of
ammunition against targets already destroyed.

c. Qutput: Picture and/or table

3. Maneuver Battle Sets — Disposition of OPFOR and BLUFOR at a specified time period.
a. Operational definition: Location of OPFOR company command vehicles and battalion
subordinate maneuver platoon leader vehicles at critical points during each mission (start
of exercise, first indirect fire engagement with OPFOR, first direct fire engagement with

C-6



(referred to as UAVs and UGVs, respectively). Constructive simulation workstations were used
by the Sustainment Team Commander, the Effects Team Commander, and the four Maneuver
Team Commanders. The remainder of the extended training audience was in virtual simulation.

In the virtual environment, simulators were used to represent several vehicles. These
included the battalion commander and deputy commander vehicles which were represented by
the Advanced Research Projects Agency (ARPA) Reconfigurable Simulator Initiative (ARSI)
simulator and an ARSI mockup, respectively; and BCVs and SOVs which were represented by
command and control vehicle (C’V) mockups. Scout vehicles and the manned platoon vehicles
of three maneuver teams were represented by Future Combat Vehicle mockups. The virtual and
constructive environments were linked by means of distributed interactive simulation (DIS) to
form the seamless battlefield environment for the participants.

The FCC? experiment trials were based on tactical operations that an Army battalion,
equipped with an advanced digital command, control, communications, computers, and
intelligence (C*I) system, might be expected to conduct in the year 2010 and beyond. The virtual
terrain chosen for the experiment was northeastern Bosnia-Herzegovina, centered on the city of
Tuzla. This terrain is extremely mountainous with limited ground mobility corridors. Figure
B-3 shows the experiment terrain map with a battalion area of operations superimposed.

Zone of
Action

pha

a
Rea as

Figure B-3. Future Combat Command and Control terrain map.

The FCC? experiment data were collected and processed by the Data Collection and
Analysis System (DCA). Figure B-4 illustrates key aspects and functions of the DCA.
Information about SC’ system usage, electronic messaging, voice communications (but not
content), displayed situational awareness, and the status of major combat systems in the

B-4



c

unit’s performance against the desired result and to provide necessary feedback to the
commander or guidance to subordinate units to get back on track.
Output: Picture, line graph, and/or table

Shared Situational Awareness

1.

Map Area — Square kilometers of battlefield displayed for each staff member at a specified
time period.

a.

c.

Operational definition: Center points of each staff personnel’s plan view display (PVD)
screen displayed by grid coordinates at critical points during the mission (start of
exercise, first indirect fire engagement with OPFOR, first direct fire engagement with
OPFOR, first friendly casualty, friendly losses exceed 30%, and last engagement during
mission). The view size may be affected by the size of the open PVD/SC? [Surrogate
Command, Control, Communications, and Computers] window, the scale of the map
selected, and the use of SC‘ tools. Picture formats of this output should enable visual
comparison of visible map relative to total map and relative to other users.

Rationale: May indicate whether the staff has a shared understanding of what is
occurring at critical points during the mission. If the team commanders are focused on
different portions of the battlefield than the staff or if the resolution of the staff's view is
significantly different than that of the team commanders’, then the amount of shared
situational awareness among the unit comes into question.

Output: Picture

Surprise Attack — Depiction of blue forces (BLUFOR) in relation to OPFOR when BLUFOR
were attacked for flank or rear engagements during each mission.

a.

c

Operational definition: Location on battlefield of flank or rear direct fire engagements on
OPFOR and BLUFOR vehicles: attacks from a position that is greater than 45 degrees
and less than 315 degrees of the hull orientation of the vehicle being attacked. The
orientation of the vehicle is considered to be zero degrees for this calculation. Direct fire
engagements are those that occur at ranges of 4,000 meters or less. Calculate the total
number of engagements, the number of flank or rear engagements, and the number of
non-flank or non-rear engagements. Data collection should start at first direct fire
engagement with OPFOR.

Rationale: If the staff was aware of the location of the OPFOR, it would ensure that
everyone in the unit was aware of the situation and direct the unit’s maneuver to either
meet the threat head on or to a position where the OPFOR could be attacked from the
flank or rear, increasing the probability of destroying the OPFOR while reducing the risk
to the unit.

Output: Picture and/or pie chart

Fire Support Coordination - Comparison of OPFOR kills due to direct fire and indirect fire
during each mission.

a.

Operational definition: Location on the battlefield of each OPFOR kill due to indirect
fire as well as each OPFOR kill due to direct fire from units controlled by the battalion
staff. Additionally, number of OPFOR kills due to indirect fire from units controlled by

C-2



Adaptability Decision-Making
Node Locations (D) Sensor Coverage (D/S)
CSS Locations (S) Multiple Fire Engagements (D)
Communication Maneuver Battle Sets (S)
Orders Distribution (D/S) CSS Available Over Time (S)
Coordination Performance Monitoring and Feedback
Damage to BLUFOR/OPFOR Systems (D/S) Priority Intelligence Requirements (D)
Degradation of Forces (D/S) Common Map Display (D)
Subordinate Unit Graphics (S) Effects of Targeting (S)
Overlay Use (D) Shared Situational Awareness
Counterreconnaissance Effectiveness (S) Map Area (D)
Artillery and Counterfire Radar Coverage (S) Surprise Attack (D)
Kill Distance (D) Fire Support Coordination (D/S)
Sensor-Shooter Time Lag (D) Air Defense Coverage (S)
Fratricide/Collateral Damage (D/S)
Common Picture (D)

Note. D=DC*I-2 measure; S = STAARS measure; D/S = DC‘I-2 and STAARS measure.

Figure 1. Proposed measures categorized by team process skill dimensions.

These refined measures were then given to the SAG for evaluation. After reviewing the
measures, the SAG met and provided ratings on which measures should be developed now,
developed later, or not developed. The Project Team consolidated the ratings and selected the
top 20 measures, which all fell into the first two categories, for development. Table 3 shows the
top 20 measures selected for development. The operational definitions, rationale, and
recommended output formats for these measures can be seen in Appendix C.

Table 3

Selected Measures Categorized by Team Process Skill Dimensions

Skill Dimension and

Measure Name Description
Adaptability
Node Locations Location of each node in relation to major subordinate units at a specified
time period.
CSS Locations Location of types of CSS assets at a specified time period.

Communication
Overlay Use Compares Commander’s operation overlay files usage with key unit
personnel during each mission.

(table continues)

13



U.S. Army Research Institute
for the Behavioral and Social Sciences

Research Report 1779

Prototype Automated Measures of Command and
Staff Performance

William T. Holden, Jr. and May H. Throne
Human Resources Research Organization

Bruce S. Sterling
U.S. Army Research Institute

np 12

Approved for public release; distribution is unlimited.

November 2001


the battalion staff to OPFOR kills due to direct fire controlled by battalion subordinate
units.

b. Rationale: If the staff has sufficient situational awareness as well as the ability to
synchronize the fire support assets under its control, then it could inflict significant
damage to the OPFOR before the OPFOR can close to within direct fire range. The
farther away from the unit the OPFOR can be destroyed, the less risk there would be to
the experimental unit.

c. Qutput: Picture and/or bar graph

Air Defense Coverage — Depiction of air defense system range templates overlayed on the

location of the unit's critical assets taken at a specified time.

a. Operational definition: Location of air defense systems on the battlefield, their range
templates, and the critical assets they are covering at critical points during each mission
(start of exercise, first indirect fire engagement with OPFOR, first direct fire engagement
with OPFOR, first friendly casualty; friendly losses exceed 30%, and last engagement
during mission).

b. Rationale: Location of air defense systems in relation to critical assets may indicate if the
staff is monitoring the location of subordinate units and directing changes in air defense
coverage based on the commander's priorities.

c. Qutput: Picture

Fratricide/Collateral Damage — Depiction of the location, unit(s) involved, and results of

fratricide and collateral damage during each mission.

a. Operational definition: Each instance of attack on BLUFOR vehicles and/or personnel
by weapon systems under the unit’s control during a mission. Data should reflect firing
unit ID, echelon of controlling headquarters, type of weapon, time of engagement,
distance between the shooter and the victim, and damage to the targeted BLUFOR unit.
In addition, report each instance of attack on non-combatant or civilian vehicles and/or
personnel by weapon systems under the unit’s control during a mission. Data should
reflect firing unit ID, echelon of controlling headquarters, type of weapon, time of
engagement, and damage to the targeted non-combatant vehicle and/or personnel. When
presented in picture format, the data should also depict trace lines between the shooter
and victim.

b. Rationale: If the staff is keeping the rest of the unit apprised of non-combatant locations
and other friendly forces locations and activities, there should be no instances of
fratricide and very few instances of collateral damage caused by units equipped with
advanced C? systems like the SC‘ that provide users with updated situational information
every 60 seconds or less. Fratricide and/or collateral damage may indicate problems with
situational awareness within a unit. The type of ammunition employed and the range
may indicate if the unit used area-type or “dumb” ammunition against targets that should
have been engaged with point-type or “smart” ammunition.

c. Qutput: Picture and/or table

Common Picture — Difference between electronic map displays at specified time periods.

a. Operational definition: Comparison of the battlefield with BLUFOR and detected
OPFOR at critical points during each mission (start of exercise, first indirect fire

C-3



[PAM]). The number of kills by the various weapon systems for each range category was then
computed and plotted. A linear trendline was then created for each weapon system. As seen in
Figure 17, just three weapon systems accounted for all of the Trial 1 kills, and just two systems
had a sufficient number of kills to generate a trendline. All of the mortar systems (vehicle robot
mortar [VEH_ROB_MTR)]) kills occurred in the same range category, which did not generate a
discernible trend line in the graph.

The rationale for this output format was that if the staff was effectively coordinating the
fires of the weapons it directly controlled, then the majority of the kills they obtained should be
nearer the maximum effective range of the weapon rather than the minimum effective range of
the weapon.

The results obtained by this measure indicate that the unit was not maximizing the
effectiveness of the IDF_GUN weapon systems; no clear trend is discernible for the PAM
weapon systems. Both of these weapon systems were controlled by the unit’s subordinate teams.
If all of the weapon systems in the unit had been plotted, the graph might have been too crowded.

, t

Sustain

Improve

Note: Slope of Line Indicates
Performance-

0<4000 4001-8009 001-1200. 15091- 20001-25001. 30001. 35001-40001. 45001- _>50000
2000 15000 20000-25000 «30000 38000 © 40000 «45000-0000

Range (Meters) to Target

= VEH_ROB_MIS SILE Ver ROB UTR 7S VEH_ROB_155
ROB_PAM +} VEH_ROB_RKT Linear (VEH_ROB_IDF_GUN)
ineat (VEH_ROB_PAM)

A VEH_ROB
(Linear VEH_ROB_MTR)

Figure 17. Kill distance (squadron controlled weapons) graph.

Figure 18 is the other output format available for this measure. The average engagement
range for each weapon system is automatically inserted into the table. To provide a comparison
with OPFOR performance, the data for their weapon systems are also provided. Data for all
weapon systems involved in Trial 1 are reflected in this table. While this table provides the
maximum effective ranges for weapon systems which are not provided in the graph, one or two
engagements at very long ranges could skew the results somewhat. The graph provides data that
indicate how the engagements were distributed across the effective range of each system.

36



CONTENTS (Continued)

9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24,
25.
26.
27.
28.
29.
30.
31.
32.
33.
34,
35.
36.

Page
Node locations at 0900......sc:sccssessessesessessscsssssesesseesesscssessessssesesseeneeatesesesnecteaeesneaee 25
Opposing force damage from blue forces..........ccscceseessssssessssesstesseeeseesseeestesneesneesses 29
Blue force damage from opposing forces ..........sesssesseessseseessesneesneesseessesssesssesstensees 30
Opposing forces destruction by weapon classification Over tiMe..........c.sessesseeseesees 31
Blue force destruction by weapon classification OVer tiMe ..........teceeeseeeeseeseeseene 32

Degradation of forces over time

Counterreconnaissance effectivenes:

Artillery coverage

Kill distance (squadron controlled weapons) graph.........eseccsecsesseesessessesesseseseeesees 36
Kill distance table .....c..ceccccecssssssesssseessessesecsscsessecssseesessesseesnesuesneeneeseeseeaesssesseseese 37
Sensor-shooter time lag graph .........cescssceesessessessessesesteseeseeseeseenteaesesneeseesesaeeneeseense 39

39

Sensor-shooter time lag picture

Sensor coverage at 0920 40
Multiple fire engagements . 42
Maneuver Battle Sets at 0900 43
Ammunition available over time ........:csscsssssscsessecsessessseesseereesesaeeneeaeeseesesneenesneenses 44
Effects of targeting........c.scccscscssessecsessesssessessessesessesesseesseseeseesussessesesnssneenseseestesesees 46
Effect of targeting opposing force high value targets over time..........ccssessseseeseeeees 47
Map area at 0920 49
Surprise attack... 50
Opposing force (OPFOR) flank engagements against blue forces. 51
Blue forces (BLUFOR) flank engagements against opposing forces 51
Fire support COordimation.......sscesesssessseesssssesseessessscsssessessnesssessseaneesseenecenecanesaneesees 52
Blue forces fire support coordination graph ........sseseeeesessessessesseesesessecsteseesesseesees 53
Air defense coverage at 0900.0... ccsessssssessssssstsssseesesseesessssussnsssessesseeseeseeseesesanessees 54
Collateral damage ........cecessesssssessessesssesecsessessessnsseenesusssseesseseeseestesnssassneeseensestessesees 56
Fratricides 57

58

Common picture at 0920.

xii



CONTENTS (Continued)

Page
Appendix A. List of ACronyMs.......scesessesseseeseesessesseesssessscsesssecsueeseessesssseasssssensasssesenesueseee A-l
B. Future Combat Command and Control Experiment Setting... teens B-1
C. Automated Measures of Command and Staff Performance
Categorized by Team Process Skill Dimensions...........csssccscesseceeseeesteseeseeseesees C-1
D. Sample Automated Measure Macro.......csesssesssessesssesssecssesseessesseesseceeeneesesensenneess D-1
LIST OF TABLES
Table 1. Cannon-Bowers et al.’s Team Process Skill Dimension Definitions ..........c.0008 6
2. Automated Measures Developed for Battle Command Reengineering IV by
SKill Dimensions...........ccccessescsssesseseesecseessessesseseeneescsseenesaesnecaeseeaeenesneseeneeneaneeneee 7
. Selected Measures Categorized by Team Process Skill Dimensions ..............:000 13
4. Overlay Use o..ceeccecccssessesessessessessssescssessesncssessesaseseeseeussuenussustieseecesneeneeeneeaeeneenseneaee 26
5. Multiple Fire Engagements Against Targets Previously Reported as
Destroyed ......ceccscsessessessessessessssvescssesucsscssesuesseenssussussnsseeseeaeeneeneeneeeaeeseeasansaneseesse 41
6. Effects of Targeting ........c.cccsessesssseescessssessessessesssesesseenessssesseeeseeesareecsusseeneeneeneeneene 45
7. Fratricides and Collateral Damage...........essssesssesessessessesseesesscesesessenessesteateaseseeee 55
LIST OF FIGURES
Figure 1. Proposed measures categorized by team process skill dimensions ........c.:ccsseseseeeee 13
2. Measures development screen in Microsoft® ACCOSS .......:-sssssssssssssssessessesssesseseeee 17
3. Automated measures development proces .........ssessssssecseeseesssesseecnecetessecetesnecseeesees 18
4. Main menu of team process skill dimensions as seen in the graphical

user interface... 19
Complete list of available automated measures as seen in the graphical
user interface 19
Definition and available measures for the shared situational awareness team
process skill dimension as seen in the graphical user interface «0.0... 20
Definition and available output formats for the map area measure of shared
situational awareness as seen in the graphical user interface .........c.cseeseeseeeeeee 20
. Combat service support locations at 0900 wo... cessesteeeseseetesteteseseeteseseeeee 24

xi



190000, -— T

!
tL
120000 - —-

170000 }—|-—|-—-}—
|

‘
00000 }—} |
i)

‘OR Movement

20000 +f

70000 |—~

‘80000

«0000

0000 }——— oe

20000 -——

Note: Only Detected OPFOR Depicted
pop

30000

19009 20000 30000 40000 $0900 600¢ 70000 80000 90000 100000 110000 120000 130000 140000 150000 160000 170000 180000

~~ DepCar —Outlawé —Headhunters — Eagles —Fox6 — Ghosts —Hawk6_TM BLUFOR_* OPFOR AO]

Note. AO = area of operations; BLUFOR = blue forces; Cdr = commander;
DepCadr = deputy commander; OPFOR = opposing forces.

Figure 27. Map area at 0920.
Surprise Attack

There are two output formats associated with this measure — a picture and a pie chart. The
Project Team used the range and the angle of attack based on the center line of individual combat
systems to determine what constituted a surprise attack. If the engagement was less than 4,000
meters and the individual vehicle was attacked from the side or rear, then the attack was
considered to be a flank engagement and thus a surprise attack. [Normally, if the targeted
vehicle is aware of the location of its attacker, it orients its front, where its armor is thickest, to
the threat. If it is attacked from the flank or rear, it is likely that the vehicle did not know that it
was vulnerable to attack.] The picture (Figure 28) represents the locations where BLUFOR and
OPFOR targeted vehicles were attacked from the flank or rear. The two graphs (Figures 29 and
30) depict the relationship of flank engagements to non-flank engagements for both sides. The
solid pattern indicates the non-flank engagements; the diamond pattern indicates the flank
pattern. Notes have been added by the authors.

The rationale for this measure was that if the staff was aware of the location of the OPFOR,
it would ensure that everyone in the unit was aware of the situation and direct the unit’s
maneuver to either meet the threat head on or to a position where the OPFOR could be attacked
from the flank or rear, increasing the probability of destroying the OPFOR while reducing the
risk to the unit.

49



Tank [FV S®IEV_APC M?=Avtilery SO Air Defense

Figure 13. Blue force destruction by weapon classification over time.

Degradation of Forces

The graph developed for this measure (Figure 14) provides a comparison between the
combat strength of both sides during the trial. At 5-minute intervals, the cumulative number of
major combat systems that had been killed to that point is recorded. Those data are
automatically integrated into a worksheet that has the number of systems with which each side
started the mission. Losses are subtracted from the starting total and a percentage of the force
that is still alive is then calculated. The data are then plotted along a timeline from when the
mission began.

A rationale for this measure is that rapid destruction of the OPFOR reduces the risk of losses
to the friendly unit. The rate of loss also may indicate battle tempo during the mission and
whether the staff is coordinating the efforts of the unit to inflict the maximum number of losses
in the shortest period of time.

Results from Trial 1 indicate that for the first 2 hours, the rates of loss between the OPFOR.
and friendly forces were comparable and moderate. After 2 hours, the rate of loss for the
OPFOR substantially increased, while the rate of loss to the friendly side decreased considerably.
Upward spikes in the lines for both sides indicate where either destroyed systems were replaced
or additional forces were committed.

The graph provides a quick comparison of the strengths of the two sides over time. While it
covers critical combat systems, it does not break the losses down into categories of combat
systems as does the Damage to BLUFOR/OPFOR Systems measure output formats described
earlier. A potential improvement might be to add lines to indicate when the units move from
green to amber to red to black.

32



constructive simulation driving the experiment were recorded by three separate systems,
collectively referred to as the data logger. Using database extraction tools, the DCA would
initially create a series of basic data tables. These tables would be subsequently refined into
more advanced tables which answered specific questions posed by various researchers. These

refined tables were the starting point for output formats of the automated measures developed
during this project.

Voice
Communications SC* System
Electronic
Messaging
Situational Simulation PDUs

Awareness

Data Refined Data
Collection Tables

Analysis "
Basic Data Tables

Note. PDU = protocol data units; SC‘ = surrogate command, control,
communications, and computers.

Figure B-4. Future Combat Command and Control data collection system.

B-5



Shared Situational Awareness

Definition:
The process by which team members develop compatible
models of the team’s internal and external environment,
includes skill in arriving at a common understanding of the
situation and applying appropriate task strategies

Available Measures:
Common Picture
Map Area
Surprise Attack
Fire Support Coordination
Air Defense Coverage
Fratricide/Collateral Damage

Return fo Menu of Sk Dimencions

Figure 6. Definition and available measures for the shared situational awareness team process
skill dimension as seen in the graphical user interface.

Map Area

Definition:
Square kilometers of battlefield displayed for each staff member at
specified periods of time

Available Output:
Picture

‘Ret to Menu of Shaved Stuatonat Return to List of Measures Return to Menu of Skill Dimensions.

Figure 7. Definition and available output formats for the map area measure of shared situational
awareness as seen in the graphical user interface.

If the user chooses the first option, the next screen will provide a list of the skill dimensions
available for measurement. When the user selects one of the skill dimensions, the program goes
to a screen that provides a definition of the skill dimension and all available measures for that
dimension (see Figure 6). In the example shown in Figure 6, a definition is provided for the skill
dimension, Shared Situational Awareness, along with the six available measures. When the user
selects one of the measures, the program provides a definition of the measure selected as well as
all available output formats for that particular measure. For example, as seen in Figure 7, the

20



The data in the table confirm the data interpretation of the graph; that is, the IDF_GUN
system was not being employed at ranges commensurate with its capabilities, while the PAM
system average kill distance was mid-point in its effectiveness range.

BLUFOR
Weapon System. tive were
Indirect
VEH_ROB_IDF_GUN 50000 15526
VEH_ROB_PAM 50000 20827
VEH_ROB_LAM 50000 #NIA
VEH_ROB_IDF_RKT 50000 #NIA
VEH_ROB_155 27300 NIA
VEH_ROB_MTR 50000 6012
Direct
VEH_ROB_IDF_GUN(KE) 50000 #N/A
VEH_MAND_TRP_TRANS 2000 1818
VEH MAND TRP TRANS CM = 2000 979
VEH_MAND_C2 1000 #NIA
OPFOR
Weapon System Maximum Effective Range __ Average Range
Indirect
ELDO VEH 2819 24700 #NIA
ELDO VEH 2823 24700 #NIA
ELDO_VEH_2S9 5000 aN/A
ELDO_VEH_BM_21 20300 #N/A
ELDO_VEH_BM_22 140000 30253
Direct
ELDO_VEH_BMP_3 4000 #NIA
ELDO_VEH_BRDM_2 2000 NVA
ELDO_VEH_BRM_3 4000 2059
ELDO_VEH_BTR_80 4000 #NIA
ELDO_VEH_BTR_90 1000 252
ELDO_VEH_T72_MP- 1000 2104
ELDO_VEH_T80_UD 1000 #NIA
ELDO_VEH_T90 4000 2436

Note. BLUFOR = blue forces; OPFOR = opposing forces.

Figure 18. Kill distance table.
Improvements to the output formats for this measure might include providing an indicator of
the maximum effective range for the particular weapon system being plotted on the graph.

Including the number of kills for each weapon system in the table would inform the reader of the
sample size and give more power to the results, especially if the sample size is large.

37



Main Menu

Click on one of the skill dimensions below to see its definition
and available measures:

Adaptability
Communication
Coordination
Decision-Making
Performance Monitoring & Feedback

Shared Situational Awareness

Figure 4. Main menu of team process skill dimensions as seen in the graphical user interface.

Complete List of Automated

Measures
Air Defense Coverage Fratricide/Collateral Damage
Artillery and Counterfire Radar Coverage Kill Distance
Common Picture Maneuver Battle Sets
Counterreconnaissance Effectiveness Map Area
CSS Available Over Time Multiple Fire Engagements
CSS Locations Node Locations
Damage to BLUFOR/OPFOR Systems. Overlay Use
Degradation of Forces Sensor Coverage
Effects of Targeting Sensor-Shooter Time Lag
Fire Support Coordination ‘Surprise Attack

Return fo Menu of Skil Dimensions

Figure 5. Complete list of available automated measures as seen in the graphical user interface.

The goal for the Observer Workstation was that it would provide a user-friendly way for
observers with no programming skills to produce meaningful output for feedback to the staff
during AARs. When a user starts the program, an introductory screen appears. After entering
the program, the user sees the instructions screen. The user can choose to read the instructions
and then move on to the main menu, which contains a list of the skill dimensions (see Figure 4),
or if familiar with the program and the available measures, the user can select to view the
complete list of automated measures (see Figure 5).

19



*s0000 apm - | |
een _ - _
—! i
130000 i ——- aa
‘
+0000 | — oo oo - —_ 7 L 7
20000, — - MN tf — a
8006 -— :
p00 4h OPFOR Movement
7ec00 -
g

60000 y
0000 : ope
40000
30000
20000 | 4 7
10000 - + a ~ -—

| Note: Tracks Pair Shooters with Targets

° | poopy pp
© toa0 20009 30000 40000 s00e0 60000 70000 0000 60009 100000 110000 120000 190000 140000 150000 160000 170000 180900

Note. OPFOR = opposing forces.
Figure 34. Collateral damage.

Figure 35 provides the same data for fratricides during Trial 1. Again the legend was
eliminated. There was only one instance of fratricide: a friendly nation’s light truck was
destroyed by a PAM. Again the location of the shooter is indicated by the larger of the
rectangles; the target location is the smaller rectangle. The line that connects the pair has merged
into the symbols. Without other data to explain what precipitated the attack, this incident would
be analyzed during the unit’s AAR to see if tactics or procedures needed to be modified to

prevent a recurrence.

Initially, the Project Team sought to combine the fratricides and collateral damage incidents
into one picture. This combination was not technically feasible during this effort. The specific
problem was linking two different sets of shooter-target combinations, yet differentiating
between the two. Additional programming should be able to overcome this challenge.
Additionally, adding the data table information into the picture would also be useful. To gain a
full understanding of the measure’s results as it is currently formatted, both the picture and the

table need to be displayed simultaneously.

56



Findings:

Based on the FCC” implementation, additional effort is required to complete the integration
of various tools associated with the observer workstation so that raw performance data can be
automatically transformed into a finished AAR product. Additional research is required to
establish performance standards for future brigade and below battle staffs using advanced
command, control, communications, computers, and intelligence (CI) systems upon which
additional automated measures can be developed. Trained SMEs on staff operations and
procedures are still required as observers to provide a context for the results obtained by

automated performance measures.

This research effort demonstrated that commercial business software packages operating on
low-cost personal computers can be used to design, develop, and implement automated
performance measures in a laboratory environment. With additional research, they may be able
to support implementation of automated performance measurement in brigade-level and below
battle command staffs using advanced digital C‘I systems.

Utilization of Findings:

The specific audience who may find the information contained in this report beneficial
includes: (a) measurement specialists and researchers, (b) simulation system programmers
(hardware and software), (c) training unit and training site personnel, and (d) personnel
conducting the FCC? experiment AARs and preparing the FCC” experimental final report.

viii



Army Battalion Staff Participation. An intact, trained staff from a maneuver battalion
would be participating in FCC’. This staff would replicate the rank structure and
operational experience of soldiers on future battle staffs.

Surrogate Command, Control, Communications, and Computers csc’) System. Staff
members in FCC? would be equipped with individual SC‘ systems that are digitally
networked to share information. Each SC‘ workstation is equipped with: an electronic
map that displays automatic updates of friendly and opposing forces’ (OPFOR’s) location
and status; e-mail messaging; collaborative mission planning tools, including electronic
Whiteboard conferencing; voice communications; and a computer-generated virtual
display of the battlefield, to include terrain information and both friendly and OPFOR
vehicles and personnel. Conceptual designs for future Army C‘ systems include most of
these capabilities. A complete description of the sc* system can be found in Throne et
al. (2000).

Instrumentation and Data Collection System. Data to create the automated measures
could be generated by the MMBL’s DCA during the virtual simulation used in the FCC’.
The DCA is a set of tools designed to collect, reduce, and display information on
battlefield performance, command and control, communications, and other types of data
in distributed simulations. Again, based on prior efforts, the Project Team had extensive
knowledge of the type of information the DCA could produce, and what its strengths and
limitations were.

Experimental Objectives. The FCC? experiment was designed to examine the effects of
digitization on battle command at brigade and below. This supported the DC‘I-3
project’s main objective, which was to design, develop, and demonstrate prototype
automated measures to improve training and evaluation for future brigade and below
command and staff training

Method

This section provides both the measures development and the measures implementation

processes. The measures development process includes information on the front-end analysis,
measures design, and actual measures development. The measures implementation process
includes information on how the measures were implemented during the FCC’.

Measures Development Process

The measures development process is an iterative procedure that requires extensive

collaboration among researchers, SMEs, and programmers. The process implemented by the

Project Team consisted of 12 basic steps:

1.
2.
3.

Review lessons learned from previous work
Identify candidate measures

Select 25 for potential development

10



Appendix C

Automated Measures of Command and Staff Performance
Categorized by Team Process Skill Dimensions

Adaptability

1. CSS Locations — Location of types of combat service support (CSS) assets at a specified time

period.
a.

Cc.

Operational definition: Location of CSS supply points and/or resupply vehicles (e.g.,
fuel, ammunition) in relation to unit locations on the terrain at critical points during the
mission (start of exercise, first indirect fire engagement with opposing forces [OPFOR],
first direct fire engagement with OPFOR, first friendly casualty, friendly losses exceed
30%, and last engagement during mission).

Rationale: Identify if the staff's mission planning had incorporated sufficient flexibility
to respond to unanticipated requirements, such as a platoon running out of fuel or
ammunition during heavy combat or at a critical point in the mission. If the supply points
or resupply vehicles were located where they were needed when they were needed, even
from an unforeseen requirement, an inference could be made that the staff had adequately
planned.

Output: Picture

2. Node Locations — Location of each node in relation to major subordinate units at a specified
time period.

a.

c

Operational definition: Location of each node on the terrain at critical points during the
mission (start of exercise, first indirect fire engagement with OPFOR, first direct fire
engagement with OPFOR, first friendly casualty, friendly losses exceed 30%, and last
engagement during mission).

Rationale: Location of the unit’s command and control (C”) nodes may indicate the
ability of the staff to handle different requirements simultaneously while keeping
positioned to maintain communications with all subordinate elements, and maintaining
operational and physical security.

Output: Picture

Performance Monitoring and Feedback

1. Effects of Targeting — Depiction of high value targets (HVTs)/high payoff targets (HPTs)

a.

and the degree of attrition each suffered during a specified time period.

Operational definition: For targets that the commander has designated as HVTs/HPTs
(e.g., tanks, artillery, air defense systems, C2 nodes), calculate number of kills
(catastrophic, firepower, mobility) over time and location and time of each target kill.
Also calculate time first detected, time engaged, and time killed for each target.
Identification of HVTs/HPTs will be provided prior to start of exercise.

Rationale: Destruction of HVTs/HPTs by the weapon systems that are under the control
of the unit’s staff may be an indicator of the staffs ability to effectively monitor their

C-1



Numbers of
Systems |
Based on

Intelligence

Estimates

Time Since Mission Stat (Hours:Minutes)

(Tank Fv “#IFV_ APC “®TMArillery “*TMADA

Figure 26. Effect of targeting opposing force high value targets over time.

The table format could be improved by breaking down the categories of high pay targets
into individual weapon system types. This might help the staff look to sustain or sustain their
targeting efforts against particular systems, not just against categories of weapons. Also, the
killed symbols on the picture could be coded by color or pattern to reflect the categorization of

weapon systems.

Shared Situational Awareness

Map Area

Figure 27 displays the average area of terrain that was visible on the electronic map displays
of commanders and staff officers over a designated time interval of 5 minutes. The Project Team
selected the unit commander, the deputy commander, and the two control node officers in charge
(Outlaw6 and Headhunter6) as representative of the amount of the battlefield that the staff would
be viewing at critical points during the mission. The Project Team also decided to depict the
battlefield area that the four maneuver team commanders displayed during the same interval as
an indicator of where significant combat activity would be occurring. The size of the area being
displayed has been averaged for the same 5-minute interval for each of the designated
commanders and staff officers. This approach was necessitated because of the difficulty in
synchronizing the displays to the same instant for the eight soldiers being tracked. For example,
over the 7 1/2-hour duration of Trial 1, the experimental unit commander resized and/or changed
the center point of his display 905 times. The deputy commander changed his display 1,363
times while one team commander changed his display 2,664 times. Other information being
displayed includes the unit zone of action, the location of the unit’s subordinate maneuver
platoon leader vehicles, and the location of detected OPFOR systems. All data elements are

47



Stealth display, providing a 3-dimensional representation of the battlefield with all of the
systems that are visible on the PVD (i.e., friendly and detected OPFOR).

Video teleconference (VTC) capability linking the commander and the staff.

Collaborative whiteboard capability, to allow the commander to present his intent and
guidance to the staff visually and quickly. Users who are part of the whiteboard session
can show snapshots from their PVDs, draw in different colors on those images, add
clipart-style labels and icons, and type words onto the whiteboard.

Large screen display, providing a three-dimensional representation of the battlefield with
all of the systems that are visible on the PVD, Stealth, whiteboard, or unmanned aerial

vehicle (UAV) screens.

Digitized modified combined obstacle overlay (MCOO), produced automatically for the
large screen display, rather than as a manually produced intelligence overlay.

Satellite imagery, acting as the electro-optic satellite sensor to deliver a direct downlink
imagery feed.

Large Screen Display
Plan View / Stealth View

Sensor VTC / Whiteboard
Window (Internode)
Enemy Ops Friendly Ops

Plan View Plan View

Commander's Station:
Plan View (Dynamic Map) /
Stealth View (360°) including
BLUFOR and sensed OPFOR

Note. BLUFOR = blue forces; OPFOR = opposing forces; VTC = video teleconference.

Figure B-2. Surrogate command, control, communications, and computers system setup.

Vehicles and weapon systems were represented in either constructive or virtual simulation.

Constructive simulation (ModSAF) was used to generate and control the OPFOR, friendly forces
below the company level, and unmanned vehicles replicating both aerial and ground sensors

B-3



ActiveChart.PlotArea.Select
ActiveChart.ChartArea.Select
ActiveChart.Deselect

End Sub

D-6



engagement with OPFOR, first direct fire engagement with OPFOR, first friendly
casualty, friendly losses exceed 30%, and last engagement during mission) and 30
minutes after each critical point. The pictures will indicate which previously detected
OPFOR vehicles disappeared and which ones were newly detected between the two
times. Comparisons will be made among various duty positions to determine the
commonality of displays among the staff.

b. Rationale: Data will identify if the staff is sharing a common picture of the battlefield,
which may be one indicator of shared situational awareness.

c. Output: Picture series

Communication

1. Overlay Use — Depiction of what graphics each command post had on its decision map at a
specified time.

a. Operational definition: Number of staff members that are showing, on their PVD, the
same operations overlay file that the battalion commander is showing on his PVD at
critical points during each mission (start of exercise, first indirect fire engagement with
OPFOR, first direct fire engagement with OPFOR, first friendly casualty, friendly losses
exceed 30%, and last engagement during mission). These data should be reported as a
percentage by dividing the number of staff members showing the same overlay file as the
commander by the total number of staff members. If possible, report by duty position
and operations overlay file name, those staff members who are not showing the same file
as the commander at critical points in time.

b. Rationale: The data may indicate whether the commander and his staff are using the
same graphic control measures to monitor and control subordinate units. If they are not,
then there is a significant potential for miscommunication and a breakdown of situational
awareness within the unit. Measures the behaviors related to synchronizing actions,
passing information in timely and efficient manner, and facilitating performance of other
team members.

c. Qutput: Table

Coordination

1. Damage to BLUFOR/OPFOR Systems — Number and type of BLUFOR/OPFOR systems out
of action and what damaged or destroyed them during each mission.

a. Operational definition: Number of BLUFOR/OPFOR tanks, artillery, and air defense
system kills by various categories of weapons for each mission.

b. Rationale: More indirect fire kills than direct fire kills may indicate that the staff has
effectively coordinated fires. Also, the distribution of losses in the unit due to indirect
fire and indirect fire from the OPFOR may indicate whether the staff was successful
coordinating types of fire. Indirect weapon systems, due to their longer ranges, are
habitually employed to attack the other side’s indirect fire weapon systems.

c. Output: Bar graph

C-4



If R.Offset(0, 7).Value >= 150 And R.Offset(0, 7) < 180
Then

Counté = Count6 + 1

ActiveSheet.Cells(7, 11).Value = Count6é

Else

If R.Offset(0, 7).Value >= 180 Then

Count7 = Count7 + 1

ActiveSheet.Cells(8, 11).Value = Count7

End If
End If
End If
End If
End If
End If
End If
Next R

End Sub
Sub MakeSENSORSHOOTERGraph3 ()

' MakeSENSORSHOOTERGraphl Macro
' Macro recorded 4/13/2001 by holdenb
' Macro revised 5/24/2001 by holdenb

"Creates Bar Chart
Charts.Add
ActiveChart.ChartType = xlColumnClustered
ActiveChart.SetSourceData
Source:=Sheets ("Sensor Shooter_Time_Lag_ Graph") .Range("J1:K
8"), PlotBy:=x1Columns
ActiveChart.Location Where:=xlLocationAsNewSheet, Name:= _
"SensorShooterTimeLagGraph"
With ActiveChart
-HasTitle = True

.ChartTitle.Characters.Text = "Sensor - Shooter Time
Lag"

.Axes(xlCategory, xlPrimary) .HasTitle = True

.Axes(xlCategory, xlPrimary) .AxisTitle.Characters.Text =
~ "Time (Minutes) from Detection to Kill"

.Axes(xlValue, xlPrimary).HasTitle = True

.Axes(xlValue, xlPrimary) .AxisTitle.Characters.Text =
"Number"

End With

With ActiveChart.Axes(xlCategory)

D-4



130000 pop ~~ a
120000 t |
110000 | —[Unit Zone F— - 7
lof Action
ste | =P —
sooo | + - —
coca
e080
ooo0 a spot 5
soca
000 }|._| —|— |
| A L |
seoo0 1
OPFOR Movement | im
2000 — | — —_| + i i
10600 j 5 7
i Note: Effects of Terrain Masking Not Plotted
°
‘© 10000 20000 30000 40000 $0000 €0000 70000 80000 So000 Yo0000 110000 120000 130000 140000 156000 160000 170000 160000
anor _2sikR = _2is2R = 2183 —— ais —_2185R_A LAM _- Seriess A Rocket —Zone]

Note. OPFOR = opposing forces.
Figure 33. Air defense coverage at 0900.

Fratricide/Collateral Damage

This measure looks at the instances where the unit attacked and destroyed either friendly
combat forces (fratricides) or caused unintended damage to neutrals, non-combatants, and
civilians (collateral damage). In addition to noting the time, the unit involved in the incident, a
description of the target, and its location, the weapon system and ammunition employed along
with the range are also tracked.

54



Table 4 (Continued)

Team Commanders

Cougar6 Overlays - Name Eacleé Foxé Ghosté Hanké
Defend Overlay 1 v v v v
Defend Overlay 2 v v v
Defend Overlay 3 v v v v
Attack Overlay | v v v
Attack Overlay 2

v v v

Attack Overlay 3

Passage of Lines Overlay 1
Passage of Lines Overlay 2
Unit Boundary Overlay

Among the staff, the senior officer in each node (Headhunter6 and Outlaw6) were also using
the same overlay files as Cougar6 and Couger3, the Deputy Commander. Interestingly, only one
of the commander’s two assistants, Cougar69, had the same overlay files as he did. In the
Command 2 node, only the Deputy Commander had overlay files related by title to the Defend
mission of the unit. The team commanders (identified by the suffix “6” in the radio call sign),
with the exception of Ghost6, had most of the same overlay files as the unit commander,

Cougar6.

This measure identifies which members of the staff have the same overlay files as the unit
commander. The Project Team had to construct these tables manually, since a reliable way to
automatically match file names among the staff had not been developed by the time this report
was prepared. By relying exclusively on file names as the basis for the comparison, there is a
potential problem with identifying the right overlay. For example, if two staff members were
working on the same overlay file simultaneously and closed the files to a common file server
without changing or modifying the file name, the last-saved file would overwrite the first-saved
file, even though the contents of the files might be different. The first staff officer would assume
that his work on the overlay file had been saved when in fact it was not. Other staff officers
might not even be aware that changes had been made to the overlay file and that they might not
have the most current file. To counteract this possibility, the unit staff would have to establish a
standing operating procedure (SOP) for naming OPORD overlays and tracking modifications to

them.

Coordination

Damage to BLUFOR/OPFOR Systems

There are two bar graphs and two cumulative line graphs associated with this measure. The
Project Team attempted to integrate BLUFOR and OPFOR data into single bar and line graphs,
but the presentation of information was overly complex. The data used to create the output
formats are the type of weapon system that was killed, the type of weapon system that killed it,
and the time the weapon system was killed. The killed weapon system is classified into one of
four groups through the use of a look-up table that is constructed ahead of the exercise by an
observer on the OWS. The four groups are: “Air Defense” — weapon systems with a primary

27



AAR
ADA
AO
ARI
ARPA
ARSI
ASCII

BCR
BCV
BLEFR
BLEP
BLUFOR
BOS

(ou
Cv
cil
ccIR
Cdr
COTS
css

DA
DCI

DC‘I-2
DC'L3
Skills
DCA
DepCdr
DIS

FCC?
FLOT
FOV
FV

GUI

HPT
HVT

Appendix A
List of Acronyms

after action review

air defense artillery

area of operations

U.S. Army Research Institute for the Behavioral and Social Sciences
Advance Research Projects Agency

ARPA Reconfigurable Simulator Initiative

American Standard Code for Information Interchange

Battle Command Reengineering
battle command vehicle

Battle Lab Experiment Final Report
Battle Lab Experiment Plan

blue forces

battlefield operating system

command and control

command and control vehicle

command, control, communications, computers, and intelligence
commander’s critical information requirements

commander

commercial off the shelf

combat service support

Department of the Army

Prototype Methods for the Design and Evaluation of Training and
Assessment of Digital Staffs and Crewmen

Refinement of Methods for the Training and Assessment of Digital Staffs

Performance Evaluation, Training, and Future Requirements for Digital

Data Collection and Analysis System
Deputy Commander
distributed interactive simulation

Future Combat Command and Control
forward line of own troops

field of view

fighting vehicle

graphical user interface

high payoff target
high value target

A-l



role of killing helicopters and other aircraft; “Artillery” — indirect fire weapon systems whose
primary mission is to attack areas of OPFOR concentration throughout the depths of the
battlefield; “IFV_APC” — ground combat vehicles whose primary mission is carry infantry
soldiers and which may or may not have organic weapon systems capable of defeating tanks; and
“Tank FV” — tanks and other ground combat vehicles that have organic weapon systems capable
of defeating tanks with direct fire, but whose primary mission is not to carry infantry soldiers.
For those weapon systems, which did not fall into these four groupings, like attack helicopters or
dismounted infantry, the designation “#N/A” is automatically assigned by Microsoft® Excel.
Each weapon system is also classified as providing direct fire (line of sight) which means that the
firing weapon system can see its target, or indirect fire (beyond line of sight or non-line of sight).

The rationale for this measure was two-fold. First, the staff is traditionally responsible for
planning indirect fires in support of the commander’s concept of the operation. More indirect
fire kills than direct fire kills may indicate that the staff has effectively coordinated fires. Also,
the distribution of losses in the unit due to indirect fire and indirect fire from the OPFOR may
indicate whether the staff was successful coordinating types of fire. Indirect weapon systems,
due to their longer ranges, are habitually employed to attack the other side’s indirect fire weapon
systems. Second, in the expected progression of a more conventional combat operation, the first
weapon systems destroyed are those that are in close proximity or direct fire range. As the
operation unfolds, artillery and air defense weapon systems, which are generally located farther
away, are attacked later. If all classes of weapons are being attacked at the same time, the staff
may be effectively coordinating fires throughout the battlefield.

In Figure 10, the data indicate that the majority of losses suffered by the OPFOR were due
to indirect fire weapons. The only type of weapon system that experienced significant losses due
to direct fire were Tank_FVs, which would be expected to be in the lead elements of an OPFOR
attack. IFV_APCs also experienced losses due to direct fire, but not to the same extent that

Tank_FVs were experiencing.

28



The rationale for this measure was to provide an indicator of whether the staff estimates and
other information staff members provided to the commander allowed him to make the right
decision on positioning his maneuver forces prior to the start of the trial. Based on the early
information presented in Figure 23, it appears that the unit’s subordinate maneuver platoons were
positioned to block the detected OPFOR vehicles.

[| ty

i
L

soo | | - t
woo |—- | < rome 1
7

eco

{ |
tom Note: Only BLUFOR Maneuver Platoon Leaders Depicted
4 J. i 1 1 1 I. 4 i

‘© o000 20000 0000 +0000 50000 60600 78860 e6500 SEBB0 180080 11e6G0 128000 190000 ¥46550 ¥SC000 160000 170000 100000

“Bus LF LBA _> Lhe Deeded OPFOR]

Note. BLUFOR = blue forces; OPFOR = opposing forces.
Figure 23. Maneuver Battle Sets at 0900.

The picture format for this measure could be improved by changing the symbols from
vehicles to reflect standard Army symbols and labels for friendly forces and the OPFOR and
increasing their size somewhat. It would also be helpful to be able to automatically depict
OPFOR units that are expected to be committed against the unit, but have not yet been detected

or identified.
CSS Available Over Time

Figure 24 depicts the number of rounds available for four critical types of ammunition over
the course of Trial 1. The intent of this measure was to determine if the staff had adequately
planned to ensure that the unit would not run out of ammunition or fuel during a critical phase of
the operation. Since fuel was not a major factor in the outcome of any trial during FCC’, the
Project Team focused on ammunition as an indicator of the staff's effectiveness in logistical
planning to support the commander’s intent. The results indicate that with the exception of a 40-
minute period, starting at 2 hours into the mission, the number of rounds available in the unit was
fairly constant and by the end of the mission was very close to what they had available at the

start.



Therefore, the Project Team decided to use Microsoft® Access for data manipulation while
continuing to use Excel to create the charts and graphs.

Once the MMBL ASCII data files were imported into Microsoft® Access, the Project Team
built queries for each of the picture, graph, and table output formats. Queries in Access are
designed to return only the data the user requires from the larger data file. Multiple queries were
often necessary to return all the data required to build a specific measure. The Project Team
found that it was easier to manipulate several small data files to create a Microsoft” Excel chart
or graph than to work with one large complex data file. Additionally, since some of the
measures had more than one output format, even more query files would be required. To handle
the growing number of queries, the Project Team created Access macros that would control
running the queries associated with each measure. Therefore, while the Project Team developed
66 queries, there were only 20 macros — one for each of the 20 developed measures.

Once the macros were developed, a form was constructed in Access that appears
automatically when the file is opened (see Figure 2). The form does three things. First, it
provides the user a link to Access through a user-friendly interface that will be incorporated into
the Observer Workstation. In this way, the user will only see a button for each of the 20
measures, and not all the queries that need to be run in order to develop a measure. Second, it
allows the user to set the time for which the output data files would be created. Once the user
enters the time and clicks on a button, the required data to build the measure(s) are automatically
output to Excel. This is especially critical for those output formats that are essentially snapshots
in time. For each change in time, queries would be required and the form provides a simple
mechanism for setting the time for all queries. Finally, the form gives the user an option of
selecting one or all of the measures at one time

To create the measure output formats, the Project Team began by using the Microsoft®
Excel chart wizard, which was described earlier. All operator actions used to create the chart or
graph were captured by a macro recorder. The macros were then modified using Microsoft®
Visual Basic® in order to achieve the desired format. In many instances, this meant overriding
the various Excel default chart or graph characteristics. For example, a gridline pattern that
represented the terrain database was added to the pictorial. Standard chart colors and symbols
were aligned with Army standards. Background and foreground colors were eliminated or
modified. Additional Visual Basic® software code had to be written to work around some of the
data series limitations of Excel. An example of an Excel macro modified with Visual Basic is
provided in Appendix D. The Project Team also developed additional Excel files that were
needed to support the output format creation.

16



~ vt |
} ~ °
sco |-—fonZa} ~
eae rn

lee

— aft
ofits

se00 +
‘60000 ~e . |

j
= |__ oy coe

e
‘60009 - V
sex
ews - —
00 |
} LAL
000 —_|— | crromvenent
| Note: Depicts Targets Destroyed by Staff Controlled Weapons
° pe ots Destroyed fy Staft Controted Wes
2 Wem ane Ben) Ane HD ow TR fone vm HIE TaN Im ten SD FoR a fd
Te

Note. OPFOR = opposing forces.
Figure 25. Effects of targeting.

The results from the line graph (Figure 26) indicate that the unit was able to target and
destroy in a short period of time almost all of the HVT/HPT except for artillery systems. The
effects of this targeting effort can be also be seen in the results of from the Damage Measure
(Figure 11) where it shows that the unit suffered significant losses from OPFOR artillery systems
with minimum losses from other types of OPFOR weapon systems.

46



If the staff is keeping the rest of the unit apprised of non-combatant locations and other
friendly forces locations and activities, there should be no instances of fratricide and very few
instances of collateral damage caused by units equipped with advanced cil systems like the SC*
that provide users with updated situational information every 60 seconds or less. Fratricide
and/or collateral damage may indicate problems with situational awareness within a unit. The
type of ammunition employed and the range may indicate if the unit used area-type or “dumb”
ammunition against targets that should have been engaged with point-type or “smart”
ammunition. Table 7 lists the fratricides and collateral damage that occurred during Trial 1.

Table 7

Fratricides and Collateral Damage

Time Unit System Ammo Range

9:

: S | ROB_PAM PAM 30994 CIV_DI_QUAD_E
11:31 _2G20. VEH_MAND_TRP_TRANS_CMD German 35AP 880 CIV_DI_DUO_A
13:27. _2G20. VEH_MAND_TRP_TRANS CMD M792 112 CIV_DIL_COL20_C

The collateral damage reported requires further analysis. As an artifact of the virtual
simulation used to support experimentation during F' CC’, certain OPFOR units were disguised as
civilians or non-combatants. This labeling allowed those units to approach elements of the
experimental unit without drawing undue attention to themselves. On command, they would
then attack the experimental unit without warning. Further analysis may determine if this was
the case during this trial.

Figure 34 shows the location of the collateral damage incidents and the locations of the
shooters who initiated the attacks during Trial 1. The large rectangles are the shooters. The
smaller rectangles are the targets. Lines are used to pair shooters with targets. In two instances,
the shooter and target were so close together that the line between them was merged into the
location symbols. The legend was eliminated from this picture. The software code that
generates this picture was sized to accommodate 50 potential collateral damage incidents. Every
potential incident is automatically assigned a “Series” label which is displayed in the legend
along with a label for an actual incident. The Project Team determined that this information
would create confusion about the number of incidents being displayed.

55



Research Report 1779

Prototype Automated Measures of Command and
Staff Performance

William T. Holden, Jr. and May H. Throne
Human Resources Research Organization

Bruce S. Sterling
U.S. Army Research Institute

Armored Forces Research Unit
Barbara A. Black, Chief

U.S. Army Research Institute for the Behavioral and Social Sciences
5001 Eisenhower Avenue, Alexandria, Virginia 22333-5600

November 2001

Army Project Number Personnel Performance and
20262785A790 Training Technology

Approved for public release; distribution is unlimited.

iii



130000

120000

‘10000

100900

90000

20000

)PFOR Movement

70000

60000

0000

0000

30000

20000

+0000

°

‘© occa 20000 30000 9000 $0000 60000 70000 s0000 99000 100000 110000 12c000 126000 140060 150000 160000 170000 180000
210 San 212 a3 a4
ans suo ~3I44 3142 “2120
2122 2124 Zone Supported Units_* OPFOR

Note. OPFOR = opposing forces.
Figure 16. Artillery coverage.

The results depicted in Figure 16 show that the missile systems are positioned to engage
targets throughout the unit’s zone of action and most of the detected OPFOR targets outside of
the zone. Several of the unit’s weapon systems are shown as being destroyed or not available
(lower left portion of Figure 16). This may be explained by the loss of these systems to OPFOR.
indirect fire, which was noted under the results obtained by the Damage to BLUFOR/OPFOR

Systems measure.

The output format provides a clear depiction of what targets could be engaged by the
weapon systems. The legend needs to be simplified to relate the color of the range template to
the type of weapon systems being represented rather than the unit vehicle marking number.
Another improvement would be to automatically remove individual weapon systems that were
destroyed or not available from the graph which would prevent their range templates from being
plotted around the 0,0 point and remove their unit vehicle marking number from the legend.

Kill Distance

There are two output formats for this measure — a line graph and a table. The line graph
(Figure 17) is generated by categorizing the range from the shooter to the target for each OPFOR
weapon systems killed. To facilitate the presentation of information, 12 range categories were
created. Initially, the line graph was designed to report information for those weapon systems
that were directly controlled by the staff (missile, rocket, and 155mm howitzer). Since the
number of kills for those systems was limited during the experiment, three additional weapon
systems were added (indirect fire [IDF_GUN], mortar [MTR], and precision attack missile

35



120%

100%

[Fe opror Losses #-BLUFOR Losses

Figure 14. Degradation of forces over time.
Counterreconnaissance Effectiveness

The picture developed for this measure (Figure 15) reflects the location where OPFOR
ground reconnaissance vehicles were first detected and the location where they were killed. In
order to visually link these two points a straight line is plotted between them; the line does not
reflect the actual route the reconnaissance vehicle was following before it was killed. The legend
for this picture identifies the pairing of the location where the vehicle was detected and the
location where it was killed as a “Series.” This identification is assigned automatically. While
the legend could have been excluded, the number of the series could be used to identify how
vehicles were being tracked, especially in those instances where lines or vehicle locations are
being superimposed on one another. In this case, eight vehicles were being tracked (the number
of series divided by two).

This measure may indicate if the staff was coordinating fires effectively to negate the
OPFOR ground reconnaissance effort. If the OPFOR cannot detect the unit, the unit reduces its
vulnerability. The OPFOR ground reconnaissance vehicles are high priority, high payoff targets
(HPTs) that, doctrinally, should be among the first targets attacked after they have been detected.

The data presented suggest that OPFOR ground reconnaissance vehicles, while detected
soon after they had moved into the unit’s zone of action, were not killed until they had an
opportunity to move, on average, about 10,000 meters farther into the zone. What is not
available in this output format is the time that the OPFOR vehicles were detected and
subsequently killed. Those data would indicate how much time the OPFOR would have had to

33



The results obtained by this measure, at the scale being represented, required careful
examination to detect significant changes, even at the 20-minute interval used to plot vehicle
locations. Any movement, however insignificant, caused the vehicle to be plotted as
“New/Moved.” Vehicles that were “Dead” continued to be plotted as each picture was being
updated, so that the vehicles that had been killed during the current 20-minute interval could not
be discerned from those killed in the preceding 20-minute interval or even earlier.

1
M@ =BLUFOR

@ = =OPFOR

20000

)PFOR Movement
}

A [
Nv

~ aL 1
10000 |} Note: Depicts Common Relevant Picture Change (20 Minute Interval)

{ J | fo je fo fej] | | f | | |

"e000 20000 20000 40080 soeeD eon0 Toa aooon seca 190000 +19EG0 12600 TsEGGD 140000 150000 160000 1700N0 180000

wm Soies2_@ Newhtoved” @ Dead _O Sialonay mW Newhoved wm Dead
Note. OPFOR = opposing forces.

Figure 36. Common picture at 0920.

To fully realize the intent of this measure, additional information is required about the
electronic PVD filters each FCC? unit participant had set on his system at the point in time that
the picture was created. For example, a participant may have set the PVD to display aggregated
unit symbols such as platoons or companies rather than individual vehicles. Rather than having
hundreds of vehicle symbols, his display may have had only 10 or 15 unit symbols. He could
have adjusted the display so that dead or killed vehicle symbols would not be displayed. He
could have adjusted the display to remove or to leave OPFOR vehicle symbols that were not
currently being observed by some sensor. The Project Team was not able, within the time
available to the project, to integrate individual participant filter settings into the picture.
Additionally, the amount of vehicle movement that would be considered significant needs to be
defined, so that only those significant movement changes would be plotted. Also, newly killed
vehicles need to be distinguished from previously killed vehicles.

58



after action reviews (AARs). Finally, automated data collection will reduce observer workload
and resource requirements. Current AAR presentations can be very labor-intensive and products
are often requested late in the battle, which makes them difficult to prepare in time for the AAR
(Anderson, Begley, Arntz, & Meliza, 2000). Automated measurement will allow these products
to be produced in a more timely manner, allowing observers to focus on other activities, such as
evaluating overall performance and providing coaching and mentoring where needed (Morrison

& Meliza, 1999).

Although employing automated measures is beneficial, they should not be the only form of
evaluation used. A comprehensive measurement package should also include the traditional
types of performance evaluation, such as observations, surveys, and interviews. Automated
measures are the least studied aspect of a well-rounded measurement package and should only be
used as a complement to the other types of measurement. In addition, automated measures
provide measures of what happened, not why it happened. In order to provide a stronger link
between outcomes and processes, more information would be needed than that provided by
automated measures alone. This project focuses solely on automated measures because other
aspects of measurement are more fully developed. Additionally, a complementary package of
traditional measures (e.g., survey, observer, and interview questions) was already developed
during the original project entitled “Prototype Methods for the Design and Evaluation of
Training and Assessment of Digital Staffs and Crewmen,” and referred to as DC'‘I (Throne et al.,

1999).

Organization of the Report

This report has five major sections:

« Introduction. Organization of this report as well as summary of previous research and
relevant literature on team performance and evaluation. Includes a discussion of staff
processes, measures of staff processes, automated measures of command and staff
performance, and conclusions.

« Method. Description of the measures development process, including front-end analysis,
design, and development of automated measures of command and staff performance.
Also includes a description of the Future Combat Command and Control (FCC”
participants, materials, and implementation of the developed measures.

« Results and Discussion. Representative results from developed measures and discussion
of automated measures evaluation and implementation for FCC.

¢ Lessons Learned: Improve. Summary of the major lessons learned concerning
implementation of automated measures during Mounted Maneuver Battlespace Lab
(MMBL) experimentation.

e Lessons Learned: Sustain. Principles that should sustain future research and
development efforts on the use of automated measures to assess command and staff
performance.



targets for each category is manually entered into the table prior to the start of the trial by an
observer. The number of expected targets reflects an intelligence estimate by the unit’s higher
headquarters of OPFOR weapon systems that could be committed in the unit’s zone of action.
The targeted number is generated automatically by categorizing the targets that the experimental
unit engaged and then counting the number in each category. The number of targets engaged
could be higher than the number expected if the number of OPFOR participating in the trial is
increased or if the unit engages targets outside its zone of action. The picture depicts the location
of where each OPFOR vehicle or weapon system was killed. The line graph provides
information similar to the Damage to BLUFOR/OPFOR Systems measure discussed earlier. The
major difference is that the Damage measure begins counting the damage from a zero point
while this measure starts with the expected number and counts down. Another difference is that
the Damage measure provides the data in percentages, while numbers are used for this measure.

The rationale for this measure is that destruction of HVTs/HPTs by the weapon systems that
are under the control of the experimental unit’s staff may be an indicator of the staff's ability to
effectively monitor their unit’s performance against the desired result and to provide necessary
feedback to the commander or guidance to subordinate units to get back on track.

The results for the measure provided in Table 6 indicate that the number of Tank_FVs and
Air Defense systems targeted were somewhat greater than the number expected. This difference
can be seen in the picture (Figure 25 ) which depicts the location of the targets that were
destroyed. Several targets are located outside of the unit’s zone of action, which may explain
why the number of Tank_FVs targeted exceeded the expected number.

Table 6
Effects of Targeting
High Payoff Target Expected Targeted
Tank_FV 40 53
IFV_APC 153 132
Artillery 90 20
Air Defense 17 22

Note. FV = fighting vehicle; IFV_APC = infantry fighting vehicle/armored personnel carrier.

45



Other OPFOR OPFOR Flank
Engagements Engagements
N=72 N=16

82%

Figure 29. Opposing force (OPFOR) flank engagements against blue forces.

While the picture format provides a good representation of where the surprise attacks took
place, information about the location of the shooter and whether it had been detected or not
(thereby having its location displayed on the sct system PVD) might be useful in determining if
the unit was reacting quickly enough to an imminent threat to prevent losses. Including the total
number of engagements on the graphs would make them more meaningful. Additionally, they
should be combined to allow side by side comparisons.

Other BLUFOR
Engagements
N= 189
BLUFOR Flank
Engagements
N=61

Figure 30. Blue forces (BLUFOR) flank engagements against opposing forces.

51



and subsequent analyses also had to be accomplished by the DC‘l Project Team. The measure
tables and graphs were manually created using Microsoft® Excel with no automated data
processing involved.

The most important lesson learned from the DCI research was that by specifying the output
format of the automated measures, the researcher will greatly reduce the programmer’s work in
extracting data from the C'l system. Output format specification will also provide the
programmer a greater understanding of what the researcher is looking for in the measures. By
providing the programmer both the operational definition of an automated measure and the
specific format in which it is to be reported, the programmer will be better able to meet the
expectations of the researcher (Throne et al., 1999).

DC'I-2 Research

For the DC‘I-2 project, automated measures of command and staff performance became the
focus. An opportunity to implement these measures was provided by BCR IV, which took place
in April, 2000. By participating in the BCR IV, the Project Team had the opportunity to conduct
a trial implementation of the automated measures of performance. Coordination between ARI
and the MMBL at Fort Knox, Kentucky, enabled the two organizations to work together as a
team to accomplish multiple goals. The Project Team used the preliminary automated measures
developed for the DC‘ project as a starting point. For this second project, since there already
was a clear understanding of what constitutes a team, the next step was to establish aspects of
positive team performance in order to determine what to measure. Consequently, the Project
Team decided to institute a framework around which to develop the measures. The Project Team
needed to determine what kind of processes to measure that would provide an indication of level

of staff performance.

After a literature review, the Project Team found that one of the most thorough evaluations
of team processes was conducted by Cannon-Bowers, Tannenbaum, Salas, and Volpe (1995).
These authors conclude that there are eight skill dimensions common to most team-based tasks.
The definitions given by Cannon-Bowers et al. are provided in Table 1. As corroboration, the
U.S. Army Training and Doctrine Command (TRADOC) Regulation 350-70 (DA, 1999) lists
team skill requirements very similar to those outlined by Cannon-Bowers et al. Therefore, the
Cannon-Bowers et al./TRADOC framework of team process skill dimensions was implemented.
However, after careful deliberation, only six of the eight skill dimensions were selected as
potential candidates for being partially addressed by automated measures. The two dimensions
not selected, Leadership/Team Management and Interpersonal Relations, were considered either
to be more of an individual skill or to require observer input in order to be measured.



REPORT DOCUMENTATION PAGE

1, REPORT DATE (dd-mm-yy) 2. REPORT TYPE 3. DATES COVERED (from. . . to)

November 2001 Final 3 May 2000 - 2 August 2001

4, TITLE AND SUBTITLE 5a. CONTRACT OR GRANT NUMBER

Prototype Automated Measures of Command and Staff Performance DASWO01-99-D-0012 DO #8
5b. PROGRAM ELEMENT NUMBER
0602785A

6. AUTHOR(S) 5c. PROJECT NUMBER

William T. Holden, Jr. and May H. Throne (Human Resources Research |_A790_

Organization); Bruce S. Sterling (U.S. Army Research Institute) 5d. TASK NUMBER
2228

5e. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT NUMBER
Human Resources Research U.S. Army Research Institute for the
Organization (HumRRO) Behavioral and Social Sciences
66 Canal Center Plaza, Suite 400 ATTN: TAPC-ARI-IK
Alexandria, VA 22314 2423 Morande Street
Fort Knox, KY 40121
9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. MONITOR ACRONYM

U.S. Army Research Institute for the Behavioral and Social Sciences ARI

ATTN: TAPC-ARI-IK
5001 Eisenhower Avenue 11, MONITOR REPORT NUMBER

Alexandria, VA 22333-5600 Research Report 1779

42. DISTRIBUTION/AVAILABILITY STATEMENT
Approved for public release; distribution is unlimited.

13, SUPPLEMENTARY NOTES
Bruce S. Sterling, Contracting Officer’s Representative

14, ABSTRACT (Maximum 200 words): The transition to the Objective Force is characterized by challenges, such as how the Army
will train, maintain, and operate as an information-age force. A key aspect of the Objective Force is command, control,
communication, computers, and intelligence (C‘D) systems. One of the Army’s immediate needs is an approach for ensuring
that the capacity of digital information systems is fully exploited in combat units, especially among staffs. Staff members must
acquire and maintain the skills required on the digital battlefield. Closely linked to training is the need for assessment for
feedback and performance improvement, and support for design and development of training programs. Digital C‘I systems
offer an exceptional opportunity for efficient and objective methods for staff performance measurement with their potent
organic capabilities to collect, analyze, and portray information automatically. This paper describes an effort to develop
prototype performance measure output and format that exemplify the potential of digital systems to provide that feedback.
Topics covered include a rationale and definition of automated measures and how they relate to measuring skills that staffs and
teams need to possess to be successful. The methodology used to design and develop automated measures of staff performance
is discussed. Finally, representative results obtained from these automated prototype measures during U.S. Army concept
experimentation will be presented, along with lessons learned during this research effort.

15. SUBJECT TERMS

Future Combat Command and Control Staff Processes Team Performance Measures of Effectiveness
Automated Measurement cil FCC? Measures of Performance
© SecUnty CLassiricaion or [7 49. imitaTIONOF | 20. NUMBER | 21. RESPONSIBLE PERSON
Se - ABSTRACT OF PAGES (Name and Telephone Number)
16. REPORT | 17. ABSTRACT | 18. THIS PAGE Dr. Carl W. Lickteig

Unclassified Unclassified Unclassified Unlimited 98 DSN 464-2613



vehicle used, with other staff members’ overlay files, and with his major subordinate team
commander’s overlay files.

The unit operations orders (OPORDs) were transmitted to the staff and subordinate units
through the use of Whiteboard files and through the use of PVD overlay files. Use of the PVD
overlay file that contains the OPORD operations overlay has been found by the Project Team,
based on previous experimentation, to be a reliable indicator of who has received the OPORD.
The comparison between the commander and his staff, and between the commander and his
subordinate commanders may indicate whether there is a potential for miscommunication and a
breakdown of situational awareness within the unit if the commanders and staff are not using the

same overlays.

Table 4
Overlay Use
Command 1 Command 2
Cougar6 Overlays - Name Cougar62 Cougar69 Cougar3 Cougar32 Cougar35
Defend Overlay 1 v v
Defend Overlay 2 v v
Defend Overlay 3 v v
Attack Overlay 1 v v
Attack Overlay 2 v
Attack Overlay 3 v
Passage of Lines Overlay 1 v v v
Passage of Lines Overlay 2 v
Unit Boundary Overlay
N Control 1 Control 2
Cougar6 Overlays - Name Outlaw6 Outlaw2 = Outlaw3_--_ Head Hunter6 Head Hunter2 Head Hunter3
Defend Overlay 1 v v v v v v
Defend Overlay 2 v v v v
Defend Overlay 3 v v v v v v
Attack Overlay | v
Attack Overlay 2 v v
Attack Overlay 3
Passage of Lines Overlay 1 v v
v v

Passage of Lines Overlay 2
Unit Boundary Overlay

26

~ (table continues)



IDF indirect fire

IFV_APC infantry fighting vehicle/armored personnel carrier
LAM long-range attack missile

LOS line of sight

MCOO modified combined obstacle overlay

MMBL Mounted Maneuver Battlespace Lab
ModSAF Modular Semi-Automated Forces

MOE measures of effectiveness

MOP measures of performance

MTR mortar

NCO noncommissioned officer

OPFOR opposing forces

OPORD operations order

OWS Observer Work Station

PAM - precision attack missile

PDU protocol data unit

PLT LDR platoon leader

PPT Microsoft® PowerPoint®

PVD plan view display

SAG SME Advisory Group

sc Surrogate Command, Control, Communications, and Computers
SITREP situation report

SME Subject Matter Expert

SOP standing operating procedure

SOV staff operations vehicle

SPOTREP spot report

STAARS Standard Army After Action Review System
TRADOC U.S. Army Training and Doctrine Command
UAV unmanned aerial vehicle

UGV unmanned ground vehicle

VEH_ROB_ MTR vehicle robot mortar
VTC video teleconference

A-2



Table 3 (Continued)

Skill Dimension and Description
Measure Name PI

Coordination
Damage to BLUFOR/ Number and type of BLUFOR/OPFOR systems out of action and what
OPFOR Systems _damaged or destroyed them during each mission.

Degradation of Forces Depiction of the relative combat power of maneuver forces over time for
both BLUFOR and OPFOR during each mission.

Counterreconnaissance Effectiveness of the BLUFOR counterreconnaissance effort during each _

Effectiveness _mission. _
Artillery and Range of BLUFOR artillery (by unit or type) coverage at a specified time
Counterfire Radar period.

Coverage

Kill Distance Distance between OPFOR/BLUFOR weapon system killed and

BLUFOR/OPFOR system that killed it during each mission.

Sensor-Shooter Time Time between first detection of OPFOR HVT/HPT by BLUFOR and when

Lag the OPFOR HVT/HPT was killed during each mission.

Decision-Making
Sensor Coverage Range of BLUFOR artillery (by unit or type) coverage at a specified time

period. : 7

Multiple Fire Number of OPFOR vehicles which were engaged multiple times during
Engagements each mission.
Maneuver Battle Sets _ Disposition of OPFOR and BLUFOR at a specified time period.
CSS Available Over Availability of ammunition and fuel during each mission. -
Time

Performance Monitoring and Feedback
Effects of Targeting Depiction of HVT/HPT and the degree of attrition each suffered during
each mission.

Shared Situational Awareness

Map Area Square kilometers of battlefield displayed for each staff member at a
specified time period. _

Surprise Attack Depiction of BLUFOR in relation to OPFOR when BLUFOR were
attacked for flank or rear engagements during each mission.

Fire Support Total number of artillery and/or mortar rounds fired during ‘each mission.

Coordination _
Air Defense Coverage Depiction of air defense system range templates overlayed on the location
of the unit’s critical assets at a specified time period.

Fratricide/Collateral Depiction of the location, unit(s) involved, “and results of fratricide and
Damage collateral damage during each mission.
Common Picture Difference between electronic map displays at a specified time period.

Note. CSS = combat service support; BLUFOR = blue forces; OPFOR = opposing forces;
HVT = high value target; HPT = high priority target

14



U.S. Army Research Institute
for the Behavioral and Social Sciences

A Directorate of the U.S. Total Army Personnel Command

EDGAR M. JOHNSON
Director

Research accomplished under contract
for the Department of the Army

Human Resources Research Organization
Technical Review by

Joseph E. Jarboe, Mounted Maneuver Battlespace Lab
Brooke Schaab, Army Research Institute

NOTICES

DISTRIBUTION: Primary distribution of this Research Report has been made by ARI.
Please address correspondence concerning distribution of reports to: U.S. Army
Research Institute for the Behavioral and Social Sciences, Attn: TAPC-ARI-PO, 5001
Eisenhower Ave., Alexandria, VA 22333-5600.

FINAL DISPOSITION: This Research Report may be destroyed when it is no longer
needed. Please do not return it to the U.S. Army Research Institute for the Behavioral

and Social Sciences.

NOTE: The findings in this Research Report are not to be construed as an official
Department of the Army position, unless so designated by other authorized documents.



Cells(7,

10).Value = "150-179"

Worksheets ("Sensor_Shooter_Time_Lag_Graph") ._
10).Value = ">180"
Worksheets ("Sensor Shooter_Time_Lag_Graph") ._

ActiveSheet.Cells(2
ActiveSheet.Cells (3
ActiveSheet.Cells(4
ActiveSheet.Cells(5
ActiveSheet.Cells(6
ActiveSheet.Cells(7
ActiveSheet.Cells (8

Then

Then

Then

Then

Cells (8,
Cells(1,

Countl =
Count2 =
Count3 =
Count4 =
Count5 =
Count6 =
Count7 =

ecooaocoCce

For Each R

If R.Offset (0,

Count1

ActiveSheet.Cells(2,

Else
If R.O

Count2

11).Value = "Number"

11)
11)
11)
11)
11)
11)

, ii)

-Value =
-Value =
-Value =

Value

-Value =
-Value =
-Value =

In FTimel.Cells

= Countl + 1

Countl
Count2
Count3
Count4
Count5S
Count6
Count7

7).Value < 30 Then

11) .Value = Countl

ffset(0, 7).Value >= 30 And R.Offset(0,

= Count2 + 1

ActiveSheet.Cells(3, 11).Value = Count2

Else

Tf R.Offset (0,

Count3

ActiveSheet.Cells(4,

Else
If R.O

Count4

ActiveSheet.Cells(5,

Else
If R.O

Count5

ActiveSheet.Cells(6,

Else

= Count3 + 1

7).Value >= 60 And R.Offset(0,

11).Value = Count3

ffset(0, 7).Value >= 90 And R.Offset (0,

= Count4 + 1

11).Value = Count4

ffset(0, 7).Value >= 120 And R.Offset(0,

= Count5 + 1

11).Value = Count5S

7) < 60

7) < 90

7) < 120

7) < 150



Sensor-Shooter Time Lag

There are two output formats associated with this measure — a graph and a picture. The
graph (Figure 19) is created from data that provide the time the target was spotted and the time it
was killed. The difference between the two times is then graphed, based on seven 30-minute
time categories. The 30-minute categories are based on the Project Team’s previous experience
with experiments similar to FCC’, where the preponderance of targets was killed within 180
minutes of being detected. This measure is used as an indicator of the effectiveness of the staff
in coordinating the fire engagements of the unit. Unless the unit is being constrained by external
factors, such as when the OPFOR has been detected massing along an international border but
hostilities have not yet broken out or when ammunition supplies are limited, the shorter the
difference in time between when a target is detected and then subsequently killed, the better the

staff may be in coordinating fires.

The Trial 1 data indicate that the unit was able to kill the majority of OPFOR targets within
120 minutes of the time they were detected. During FCC”, resupply rates for key ammunition
types were constrained which may explain why very few targets were killed within 30 minutes of
detection or many targets were killed more than 180 minutes after they were detected. Results
from additional trials would be needed to establish a baseline to determine if the performance
indicated by this trial should be sustained or improved upon.

The picture (Figure 20) is derived from plotting the location of the OPFOR target when it
was detected and the location of where it was killed. The lines on the picture simply link the two
points and do not indicate the route that the target took before it was killed. The unit’s zone of
action is also plotted from a table created by an observer. An arrow is used to indicate the
OPFOR general direction of movement. Figure 20 complements the data presented in Figure 19.
Based on ground tactical movement speeds of 25 kilometers per hour using highways, the
clustering of dead vehicle symbols in the center of the zone of action indicate that the majority of
vehicles did not move between the time they were detected and the time they were attacked.
Vehicles that were detected outside of unit’s zone of action were able to travel greater distances
before they were killed, which would result in an increase in the sensor-shooter lag time for

those engagements.

38



A full set of measure outputs was presented to the SAG for review and to the MMBL for Battle
Lab Experiment Final Report (BLEFR) input after the F ‘CC? experiment was concluded.

Results and Discussion

To provide trainers with a variety of means to depict staff performance, 33 output formats
were designed to support the 20 automated measures. This section provides samples of each
output format. The discussion of each output format is organized into three parts:

« A description is provided of what information is being presented along with any filters
that were applied to the source database to obtain the data elements required to create the
output. A rationale for the presenting the information is also provided.

e The measure’s result is discussed briefly. The results should not be viewed as a
reflection of the unit’s performance. As described earlier in this report, the intent of the
FCC? experimentation was to gain insights into future C? issues, not evaluate specific
performance of a unit using conceptual weapons, organization, or doctrine. Performance
standards for this type of unit have not been established.

« Comments on the utility of the output format and recommendations on how it could be
improved are also provided.

Readers are further advised that the original output formats provided were developed with
color and shape as primary identifying characteristics. Generally, blue squares are used to
denote BLUFOR while red diamonds are used to denote OPFOR. Although the shapes are still
identifiable, colors cannot be reproduced in this report. In the electronic versions of the output
formats, the user can modify the characteristics or delete any data series to clarify the
presentation of information. Again, this capability cannot be replicated in this report. Full color
samples or electronic files of these measures and output formats can be obtained from the

MMBL.

For the picture output formats, the Project Team created a grid system to represent the
terrain database used during the trials. The terrain database covers an area 184 kilometers by
130 kilometers. The grid represents 10,000-meter intervals and covers the entire terrain. The
Project Team has not yet integrated tactical ground maps into pictorial representations due to
scaling problems and the potential for map information to clutter the presentation of critical
information. The unit zone of action for each trial is also drawn automatically from data in a
table created by an observer. The zone of action represents the geographical area where the
majority of the unit’s maneuver forces were positioned and where close combat with the OPFOR
was expected. The pictorial representation of the unit zone of action along with an arrow
indicating the OPFOR direction of movement has been found to adequately orient a training

participant to the activity being depicted.

The sample results provided here are from Trial 1 during FCC”. The results of all four FCC?
trials were provided to the MMBL for use in the FCC? BLEFR. Results are categorized by the

22


160 ————

BEER RESEUEEES

n Start Hours: Minutes)

SA AES lay SO Osta}

Figure 12. Opposing forces destruction by weapon classification over time.

The BLUFOR Destruction line graph (Figure 13) shows that the early rate of loss in the
Artillery category carried throughout the trial. Additional analysis is required to explain this
trend. The expected pattern would be that those elements of the unit that were closest to the
OPFOR would sustain the most losses early in the fight. That does not appear to be the case in
this trial.

Overall, the bar graphs appear more informative than the line graphs. The bar graphs show
what types of weapons (direct or indirect) caused the most casualties. The line graphs, while
showing at what time losses were occurring, would also require an observer to interpret why the
losses were happening when they did. As they are currently constructed, the line graph could not
stand alone, and this is an important goal for all of the measure output formats.

31



Appendix D
Sample Automated Measure Macro

This appendix contains the Microsoft® Excel Macro, modified by Microsoft®
VisualBasic®, that is used to create the bar graph for the Sensor-Shooter measure.

Sub MakeSENSORSHOOTERGraph (

'

' MakeSENSORSHOOTERGraph Macro

' Macro recorded 4/18/2001 by holdenb
MakeSENSORSHOOTERGraph1
MakeSENSORSHOOTERGraph2
MakeSENSORSHOOTERGraph3

Dim myFileName As String, myChart As String, myPath As String,
myTime As String, myTitle As String, ThisFile As String
myPath = "C:\FCC2\Results\"

myFileName = Format (Now(), "mmddyy")

myTime = Application.InputBox("Enter Time")

myTitle = ActiveChart.ChartTitle.Characters.Text

MsgBox "Saving Chart as" & myPath & myTitle & myFileName _

& "_" & myTime, vbInformation, "Save as:
ThisFile = myPath & myTitle & myFileName &
Sheets ("SensorShooterTimeLagGraph") .Select
Sheets ("SensorShooterTimeLagGraph") .Move
ActiveWorkbook.SaveAs (ThisFile)
ActiveWindow.Close
Application.DisplayAlerts = True
Windows ("Sensor_Shooter_Time_Lag_Graph.xls") .Activate
ActiveWindow.Close
Application.DisplayAlerts = False
Windows ("FCC2Macros.xls") .Activate

End Sub

Sub MakeSENSORSHOOTERGraphl (
,
' Macro recorded 4/13/2001 by holdenb
' Macro revised 5/24/2001 by holdenb
' Activates data source file
Workbooks.Open _
"C:\FCC2\Common\Sensor_Shooter_Time_Lag_Graph.xls"
Dim Count As Integer
Set myRange = ActiveSheet.UsedRange
Set FTime = myRange.Columns ("A")

D-1



#8 Automated Measures

ar . R ie Kill Distance ee Orders Distribution

Counterreconnaissance [i [ Sensor-Shontet Time
Fire Support Multiple Fire
ALL MEASURES
maa Exit DC41

Note. BLUFOR = blue forces; CSS = combat service support; OPFOR = opposing
forces; PPT = Microsoft® PowerPoint®.

Figure 2. Measures development screen in Microsoft® Access.

Many of the output formats incorporate information that is either not readily available from
or not tracked by the DCA. For example, a measure’s pictorial output format might require a
depiction of the unit’s zone of action. The coordinates for the zone cannot be solely determined
by obtaining the vertices of various graphical control measures plotted on the plan view display
(PVD), due to a multiplicity of linear control measures, which may or may not be applicable to
the current mission. On the other hand, an observer can readily determine which are the
appropriate points to plot. Various reference data were also needed, such as the number of
expected OPFOR and blue force (BLUFOR) weapon systems in the mission, maximum effective
ranges of weapons, call signs for various staff members, weapon system categorization, and so

forth.

To simplify creating the output J formats, a similar form to the one constructed in Microsoft®
Access was developed in Microsoft® Excel so users could select the measure they wanted to
create by clicking a button. When users click on the measure they want to create, another form
appears with the various output formats available for that particular measure. A user could then
choose a specific format or select all output formats. If only one output format is available for a
measure, then the output is created immediately.

After the output format was developed, it was informally evaluated by SMEs who were not
involved in the measures development process to better ensure that users would understand the
outputs without further detailed explanation. Based on SME input, the Project Team refined the
output formats by adding legends and changing colors until they were immediately interpretable.

17



"Routine begins to convert times into minutes

Rows ("1:1") .Select
Selection.Delete Shift:=x1Up
Columns ("D:D") .Select
Selection.NumberFormat = "General"
Range ("E1").Select

For Each R In FTime.Cells
R.Offset(0, 4).Select

ActiveCell.FormulaR1Cl = "=RC[{-4]-RC[-1]"
R.Offset(0, 5).Select
ActiveCell.FormulaR1Cl = "=HOUR(RC[-1])"
R.Offset(0, 6).Select
ActiveCell.FormulaR1Cl = "=MINUTE(RC[-2])"
R.Offset(0, 7).Select
ActiveCell.FormulaR1C1 = "=(RC[-2]*60)+RC[-1]"
If R.Value = "" Then End
End If
Next R
End Sub

Sub MakeSENSORSHOOTERGraph2 ()

' MakeSENSORSHOOTERGraph2 Macro
' Macro revised 5/24/2001 by holdenb
'
Windows ("Sensor _Shooter_Time_Lag_Graph.xls") .Activate
Dim Count As Integer
Set myRangel = ActiveSheet.UsedRange
Set FTimel = myRangel.Columns ("A")
Worksheets ("Sensor Shooter_Time_Lag_Graph").
Cellis(1, 10).Value = "Time"
Worksheets ("Sensor _Shooter_Time_Lag_Graph").
Cells(2, 10).Value = "<30"
Worksheets ("Sensor_Shooter_Time_Lag_ Graph") .
Cells(3, 10).Value = "30-59"
Worksheets ("Sensor _Shooter_Time_Lag_Graph").
Cells(4, 10).Value = "60-89"
Worksheets ("Sensor Shooter _Time_Lag Graph").
Cells(5, 10).Value = "90-119"
Worksheets ("Sensor _Shooter_Time_Lag Graph").
Celis(6, 10).Value = "120-149"
Worksheets ("Sensor Shooter Time_Lag_Graph") .

D-2



PROTOTYPE AUTOMATED MEASURES
OF COMMAND AND STAFF PERFORMANCE

Introduction

The transition to the Army’s Objective Force is characterized by challenges, such as how the
Army will train, maintain, and operate as an information-age force (U.S. Department of the
Army [DA], 2001). The foundation of the future Army includes an increased area of operation;
increased operational tempo; combining branches for close combat with fewer systems; non-
linear, asymmetrical operations; and information dominance with increased situational
awareness. A key enabler for this future force will be enhanced commanders’ situational
understanding, which will flow from the fielding of advanced command, control,
communications, computers, and intelligence (C‘l) systems.

Closely linked to the training challenge is the need for measurement, both to allow for
feedback and performance improvement and also to support the design and development of the
training programs. Measures of performance (MOPs) focused on process, and measures of
effectiveness (MOEs) focused on outcome are critical areas of interest. However, direct
observation and objective measurement of performance during training are difficult, particularly
for command and control (Cc? ') performance. Subjective methods used for assessing C?
performance are labor-intensive approaches, requiring observers with high subject matter
expertise. Even with automated data collection aids such as electronic clipboards or computer-
assisted observation tools, these methods are inefficient. They are also subject to unreliability
because of the lack of standardization among observers. Measurement problems are further
exacerbated in the information-intensive environment of digital C?. Observers, like users and
participants, can be quickly overwhelmed with the amount of information relevant to C?

performance.

Digital C‘l systems, however, offer an exceptional opportunity for more efficient and
objective methods for performance measurement, particularly for C? performance. Digital C*I
systems have organic capabilities that should allow us to exploit them to automatically collect,
analyze, and portray data. As Goehring (1995) states:

What distinguishes this approach [automated analysis by computer systems] from
previous software tool development efforts is that most of the work of the
researcher or analyst is actually accomplished by the computer software. After
careful formulation of the problem and codification, the actual analysis is
accomplished automatically. The difference from the past is somewhat subtle.
Previously, software tools processed and presented refined information to the
investigator, who then further analyzed the information. Now because of several
technological developments it is increasingly possible for much of the second
phase of the work to be accomplished fully automatically. In the first case, the
computer helped to do the work. In the second case, the computer does all the

work! (pp. 4-5)



+80

160

veo:

x20

Numbor of Kite

Direct Fire vs Indirect Fire

Wovea Fie Bindvedt Fr)

Figure 32. Blue forces fire support coordination graph.

Differentiating among the types of fires may improve the picture format for this measure.
For example, each kill for a specific type of weapon might be given a different color or pattern.
Another potential improvement might be to incorporate the bar graph into the picture to provide
a quick visual ratio of direct fire to indirect fire kills without having to display the two formats

simultaneously.
Air Defense Coverage

Prior to the start of FCC’, the Project Team surmised the 12 indirect weapon systems (Long-
range Attack Missile [LAM], Rocket, and 155mm Howitzer) controlled by the staff would be the
critical assets most vulnerable to air attack. In Figure 33, these systems are represented by the
triangular symbols. To establish if the unit’s air defense systems could protect these critical
assets, the locations of the six individual air defense weapon systems were entered into a
worksheet which automatically calculated the range template for each weapon system. The
range templates were then plotted. Since geographical information, such as mountains, was not
being factored into the display, a note was added to remind the viewer that the masking effect of
terrain was not being considered. This meant that the actual range template could in fact be more
limited than shown. The direction from which potential air attacks could come was indicated by
an arrow representing the general direction of OPFOR movement. The vehicle location data
used to create the picture is available in 5-minute intervals.

During FCC’, the unit’s air defense systems came under the direct control of the staff. The
Project Team projected that the location of the air defense systems in relation to critical assets
may be an indicator of the staff’s ability to monitor the location of those assets and to direct
changes in locations based on the commander’s priorities. With the relatively short range (5,000
meters), the air defense systems would need to be positioned very closely to the assets and to be
mutually supporting if they were to provide a defense in depth against air attack.

53



Appendix B
Future Combat Command and Control Experiment Setting

The implementation of the automated measures developed was dependent upon the Future
Combat Command and Control (FCC? ) environment. Therefore, descriptions of the participants,
the equipment used by the participants, and equipment used to collect data in the FCC?
experiment follow.

Participants

The unit participating in the experiment was an active Army cavalry squadron staff with its
subordinate company commanders participating. One company brought drivers and gunners to
man several future combat vehicle simulators. As shown in Figure B-1, the squadron staff,
which operated in a virtual simulation, included the commander and 13 staff officers and non-
commissioned officers (NCOs). The commander and staff were reconfigured into two battle
command vehicles (BCVs) and two staff operations vehicles (SOVs), or nodes. The battle
command reengineering aspect of the FCC? experiment was focused on this group of 14 soldiers.
The node functions and job responsibilities for each staff member were left to the discretion of
the squadron commander, who was allowed to reorganize the staff as he gained experience in
operating the Surrogate Command, Control, Communications, and Computers (SC) system.

VIRTUAL
SIMULATION

COMMAND 1
(Command Group

‘Commander (LTC)
Effects Ops (CPT)
Enemy Ops (CPT)

|CONSTRUCTIVE
| SIMULATION

|

|

CONTROL 1 CONTROL 2

@) Batle Captain (CPT) Battle Captain (CPT)
Sustinmant Friendly Ops (MSG) Friendly Ops (CPT)
(Commander Node Enemy Ops (SFC) Enemy Ops (SFC)

Sensor NCO (SFC) Sensor NCO (SFC)

COMMAND 2
(Deputy Commander)
Deputy Commander (MAJ)
Ops Offcer (CPT)

Ops NCO (SFC)

|
| 3

PLTLOR PSG

Scout Platoon Nodes

‘One Team had
four platons

Nareuver Team
Commander Nodos

ConbatVericles
(Three platoons)

Figure B-1. Battle Command Reengineering IV staff structure.

B-1



60 —

N40

20 ———_| -

<0 30-59 60.89 90-119 120-149 180.178 >160
Time (Minutes) from Detec tion to Kil

Figure 19. Sensor-shooter time lag graph.

0 ; “PPrtr

420000
a
_

vow | fan -
Prnaen|

sox oss

ws ; 7 |

ft

OPFOR Movement |

0000

‘70000 -

0000 -—

0000 |--—- —

49000 |—— pee pe +

ae Note: Maximum of 100 Sensor- | |

10000 |— — 4 Shooter Pairings Depicted Lt
| pepo py

‘9 10900 20000 30900 40000 $0900 60900 70900 sodee 90000 100000 110000 20000 138000 149000 150000 169000 170000 160000

°

Note. OPFOR = opposing forces.

Figure 20. Sensor-shooter time lag picture.

There are several potential improvements that could be made to the output formats. If
objective standards have been established for the time lag between when sensors have detected a
target and the time it was killed, the bars outside of the tolerance could be given a different color
or pattern to distinguish how much unit needs to sustain or improve its targeting performance.



The rationale for this measure is based on the capability of the SC‘ system, which allows the
unit commander to visualize the entire “sensed” battlefield on his PVD. The information that is
being displayed on the PVD comes from a wide variety of sources, but the sensors that are being
controlled by the staff provide a majority of the OPFOR information. If the staff is not properly
deploying and monitoring performance of their sensors, then the information that is being
displayed to the commander will be incomplete, which will prevent him from making a decision
using all information that could be made available to him.

The results depicted in Figure 21 indicate that 50 minutes into the exercise, a considerable
number of the OPFOR systems that were participating in the trial had not yet been detected.
However, most of those systems that had been detected within the zone of action were killed.

Depending on the number of OPFOR systems that have been detected and killed, the picture
could evolve into a complex presentation of information. One way to simplify the presentation
of information may be to create two different pictures. One picture would present the “Not
Detected” and “Detected” OPFOR system locations, while a second picture would present the
“Detected” and “Dead” vehicle locations.

Multiple Fire Engagements

Table 5 lists the three OPFOR weapon systems that were attacked after they were reported
as being killed. Battle damage assessment information is automatically posted as the SC" system
is updated every 30 seconds. With this information available to everyone in the unit, instances
where the target is fired upon after it is destroyed should be infrequent. The table provides
information on the target and the unit systems that participated in the multiple engagements.

One of the critical pieces of information is the time that the second attack took place. It would
be possible for an OPFOR system to be targeted by two different weapon systems during the
short period in which the results of the initial engagement had not yet disseminated.

Table 5

Multiple Fire Engagements Against Targets Previously Reported as Destroyed

Target

Mark Target Type Firer Mark Firer Type Ammo Effect Time
_101A12 ELDO_VEH_BTR_90 _2E10 VEH_MAND_TRP_TRANS CMD German 35AP KL | 9:02
_101A12 ELDO_VEH_BTR_90 _2G410P VEH_ROB_PAM PAM KL 10:46
_19H13. ELDO_VEH_2S_23  _2F38P VEH_ROB_PAM PAM 10:53
_119H13_ ELDO_VEH_2S_23 _2F19P. VEH_ROB_PAM PAM KL 18
_119H23 ELDO_VEH 2S 23 2F38P VEH_ROB_ PAM PAM KL 10:54
_119H23_ ELDO_VEH_2S_23 _2F33R VEH_ROB_IDF_GUN MRAAS MPERM: KL _ 11:08

Figure 22 complements the data presented in Table 5 by depicting the locations at which the
three multiple fire engagements took place. This information may be of use in analyzing why a
particular target was engaged more than once.

Al



.HasMajorGridlines = False
.HasMinorGridlines = False
End With
With ActiveChart.Axes (xlValue)
.HasMajorGridlines = True
.HasMinorGridlines = False
End With
ActiveChart.HasLegend = False
ActiveChart.PlotArea.Select
With Selection.Border
-ColorIndex = 16
-Weight = xlThin
.LineStyle = xlContinuous
End With
With Selection.Interior
.~ColorIndex = 2
-PatternColorIndex = 1
-Pattern = xlSolid
End With
ActiveChart.Axes(xlValue) .AxisTitle.Select
With Selection
»HorizontalAlignment = xlCenter
.VerticalAlignment = xlCenter
.Orientation = xlVertical
End With
ActiveChart.ChartTitle.Select
Selection.AutoScaleFont = True
With Selection. Font
-Name = "Arial"
.FontStyle = "Bold"
.Size = 14
.Strikethrough = False
.Superscript = False
-Subscript = False
-OutlineFont = False
-Shadow = False
.Underline = xlUnderlineStyleNone
-ColorIndex = xlAutomatic
Background = xlAutomatic
End With
ActiveChart.SeriesCollection(1).Select
ActiveChart.SeriesCollection(1).ApplyDataLabels
Type: =xlDataLabelsShowValue,
AutoText:=True, LegendKey:=False
ActiveChart.SeriesCollection(1).ApplyDataLabels
Type:=xlDataLabelsShowNone,
AutoText:=True, LegendKey:=False

D-5



Appendix A contains a list of the acronyms and abbreviations used in this report. Appendix
B contains a description of the FCC’ setting. Appendix C contains the key aspects of the
prototype automated measurement package, including operational definitions, rationale, and
recommended output formats for the candidate automated measures. Appendix D contains a
sample automated measure macro.

Background

The purpose of the Background is to provide the context for the DC‘I-3 project and the
rationale for the automated measures approach to evaluation. A brief summary of the previous
work conducted for the DC*I and DC‘I-2 Projects and the lessons learned from each are
provided. The environment in which the DC'l-3 research was implemented is also described.
For a review of additional research related to automated measures of command and staff
performance, see Throne et al. (2000).

DC‘I Research

The development of automated measures of performance was only a small portion of the
original DC‘I project. The goal of the project was to develop a prototype C’ staff training and
assessment package. For the assessment portion, the Project Team developed not only
automated measures, but also surveys, interview questions, and an Observer Data Collection

Instrument (Throne et al., 1999).

In order to develop effective C? staff performance measures, the term “staff” had to be
defined. Since information about automated measures of command and staff performance is
limited, the DC*I Project Team redefined staff performance as team performance, since a
military staff could be viewed as a team. Therefore, the definition of teams developed by Salas,
Dickinson, Converse, and Tannenbaum (1992) was used to refer to a staff. They define team as
“a distinguishable set of two or more people who interact dynamically, interdependently, and
adaptively toward a common and valued goal/objective/mission, who have each been assigned
specific roles or functions to perform, and who have a limited life-span of membership” (p. 4).

Since the measurement package was to be implemented during the Battle Command
Reengineering (BCR) III, the Battle Lab Experiment Plan (BLEP) for the BCR III (MMBL,
1998) was examined as a starting point for evaluation. In the BLEP, several research issues or
questions related to advanced digitization’s effects on battle command at brigade and below were
identified that could form the basis for developing future staff training performance standards.
Since the future battalion battle staff model that the Project Team was using did not have
published doctrine or tactics, techniques, and procedures upon which to base training and
performance evaluations, the MMBL issues were used as a starting point.

A total of 14 automated measures were designed to support six issues that the Project Team
felt could be partially addressed by automated processing of BCR III data. The intent was to use
the data collection and analysis capabilities of the MMBL to automatically produce the output
formats, which consisted of tables and graphs. This approach did not work and to produce the
results for the issues, a considerable amount of additional data examination, editing, processing,