general lessons learned by the participants about the VTP and
structured training. The participants are also questioned about
the use of the VTP for future virtual training devices, e.g. the
CCcTT.

The self-administered questionnaire, This instrument, which

can be found in Appendix C, contains the following sections:

1. Demographic Information. This section contains nine open-
ended items dealing with the respondents’: (a) military
experience, including previous O/C and EC experience and (b)
experiences with the VTP’s O/C team.

2. Train the Trainer. This section consists of two sets of

Likert-scale items. For Set 1, respondents are to indicate the
perceived quality of their O/C or EC training; while for Set 2,
they are to rate the amount of training for various VTP duties.

3. Unit Preparation. This section is comprised of five Likert-
scale items dealing with the: (a) effectiveness of various VIP
materials; (b) importance of advance visits by the O/Cs to the
unit's home-stations; (c) percentage of units adequately prepared
for their VTP training; and (d) effectiveness and utility of the
demonstration tapes.

4. Training Management. This section consists of three Likert-
scale items in which participants are to indicate their opinions
about the: (a) different TSP components; (b) format of the OCIC
Handbook’s components; and (c) allotted time for the various
SIMNET activities.

5. Training Structure. This section contains seven Likert-scale
items dealing with such aspects of the VTP’s training structure
as the utlility of: (a) the stealth preview for aiding the
units’ ability to conduct the tables; (b) of subtask repetition
in helping the units to improve their performance; and (c) of the
AAR aids in helping the units to discuss their performance.

6. Training Proficiency. This section requires respondents to
estimate on a numerical scale from 0-100% the levels of
performance for a typical unit leader and unit at the start and
end of a VIP rotation. The respondents are then to use the
following scale--0-25%, 26-50%, 51-75%, 76-100%--to estimate the
percent of unit improvement that is a function of coaching
provided by their O/C and adapting to SIMNET.

7. Unit Follow-up and the THPs. This section consists of five
items dealing with such issues as: (a) identifying the groups
(e.g., crews or squads) within each unit level (e.g., PLTs) who
should have feedback materials incorporated into their THPs; (b)
determining the effort involved in producing a THP; and (c)
ascertaining the perceived usefulness of the THP materials.
Likert-scale and open-ended items were used to examine these
issues.

11



2a. Why was the training"Too Much" for those duties you checked?

2b. Why was the training "Too Little" for those duties you checked?
Unit Preparation Interview

1a. You indicated that should be modified. Please describe how you think
it should be modified.
Orientation Guide

Orders and Overlays

Demonstration Tapes

1b. You indicated that should be replaced. Please describe what you think
the replacement should be.
Orientation Guide

Orders and Overlays

Demonstration Tapes

1c. Why should be deleted?
1d. What additional advance materials are needed?

2. Do you have any suggestions for better ways to handle advance visits and unit
coordination?
3. Do you have any suggestions on how to encourage units to better prepare for a VTP
rotation?

4. How could the Demonstration Tapes be improved?

5. Do you have any suggestions for better ways to use the Demonstration Tapes?

Training Management Interview

1a. You indicated that in the TSP should be modified. Please describe how
you think it should be modified.
Familiarization Course

Observer/Controller Handbook

Tools and Reference Materials

OCIC Handbook (Tables)



Shlechter, T. M., Bessemer, D. W., Nesselroade, K. P., & Anthony,
J. (1995). Initial evaluation of a simulation-based training
program for Army National Guard Units (Research Rep. No.
1679). Alexandria, VA: U.S. Army Research Institute for the
Behavioral and Social Sciences. (AD A 297 271)

Turecek, J. L., Campbell, C. H., Myers, W. E., & Garth, T. H.
(1995). Reserve Component Virtual Training Program (RCVTP)
orientation guide (Research Product 95-07). Alexandria, VA:
U.S. Army Research Institute for the Behavioral and Social
Sciences. (AD A292 885)

U.S. Department of the Army. (1988, November). Training the force
(FM 25-100). Washington, DC: Author.

U.S. Department of the Army, (1990, September). Battle focused
training (FM 25-101). Washington, DC: Author.

28



Appendix C
The O/C Questionnaire
DATA REQUIRED BY THE PRIVACY ACT OF 1974

AUTHORITY: Title 10, USC, Sec 4503.

PRINCIPLE PURPOSE: The data collected with this form are to be used for research
purposes only.

ROUTINE PURPOSE: This is an experimental personnel data collection form developed
by the U.S. Army Research Institute for the Behavioral and Social Sciences pursuant to
its research mission as prescribed in AR 70-1. When identifiers (name or Social Security
Number) are requested, they are to be used for administrative and statistical contro!
purposes only.

DISCLOSURE: Your participation in this research is strictly voluntary. Individuals are
encouraged to provide complete and accurate information in the interests of the research,
but there will be no effect on individuals for not providing all or any part of the information
requested.

The purpose of this questionnaire is to collect information from
Observer/Controllers (O/Cs) regarding the Virtual Training Program (VTP). We are not
interested in evaluating you or your instructional methods. It should take less than
thirty minutes to complete this questionnaire. Answer each item to the best of your
ability. You will have the opportunity to elaborate upon some of your answers during the
interview phase of this data-collection effort.

Please sign your name below indicating that you have agreed to participate in
this data collection effort. This page will be removed from the remainder of the
questionnaire before responses are examined so that your input will be identified only
by general demographic data.

Name:

PTY 59-72 c-1



their future training needs (Turecek et al., 1994).

The O/Cs observe the participants’ SIMNET performance and
lead the AARs from their workstations. These stations include a
plan view display (PVD; two-dimensional electronic map), tactical
radios, stealth vehicle display (three-dimensional view of the
virtual battlefield), and audiovisual recording and replay
systems helping the O/Cs to perform their duties. The stealth
vehicle, for example, provides a direct view of the battlefield
from an invisible vehicle moving on or above the virtual terrain.

The ECs’ duties, The ECs are primarily responsible for
operating the O/C workstations, troubleshooting problems with the
simulators, and assisting the O/C with conducting the VTP

training. These duties involve: (a) initiating different
scenarios, (b) monitoring the list of table events (master event
list), (c) creating battlefield effects, (d) controlling indirect

fires, and (e) coordinating the resolution of equipment problems
with the appropriate SIMNET staff members. The ECs also assist
the O/Cs by helping them prepare for the AARs and complete the
THPs. The ECs must then be knowledgeable with regards to the
training equipment and objectives.

Additional responsibilities of the O/C team. A few of the

military personnel have functions other than those of an O/C,
such as the field-grade officer who also functions as the Team
Chief during a VTP rotation. During the BN missions, an EC acts
as the controller for the opposing force (OPFOR). He directs the
semi-automated OPFOR vehicles during the battle and gives the
OPFOR portion of the AAR.

The ECs also: (a) operate the Janus workstations during BN
staff exercises; (b) control the Janus exercises; and (c) observe
the performance of the units’ staff. The O/C team’s military
members serve as "interactors" (i.e., role-playing as unit
personnel) during the Janus exercises (Hoffman et al., 1995).

The O/C team’s training, The military members of the 0/C

team received training on being a VTP O/C in late 1993 and early
1994. This training included the SIMUTA team’s giving a series
of "workshops" on the: (a) instructional philosophy associated
with the VTP; (b)home-station visits; (c) training tables; (d)
O/C workstation, including the operation of the equipment; and
(e) training support packages (TSPs). A course on facilitating
the AARs was also given by an instructor from the U.S. Army
Armor School. These O/Cs, additionally, received "hands-on
training" from the SIMNET site staff on operating the
workstations.

The O/C team personnel who joined the program after its
initiation only received on-the-job training (OJT). Such
training consisted of reviewing the relevant VTP training
materials and being instructed by a more experienced member of
the 0/C team ("right-seating"; Hoffman et al., 1995). Hence, the



Table G-3

Response Frequencies by O/C Personnel to the Question on the Effectiveness of

Orders and Overlays

Delete Replace Modify Good
Personnel Type n a 2 n
Officers 0 2 2 2
Ncos 0 1 3 4
Civilians 0 0 6 2
Management t) 0 6 0

*Chi-square tests revealed a significant difference (p < .02) between the
proportion of modify responses indicated by the Management group as compared to
the rest of the sample.

Table G-4

onse Frequenci o/c sonnel to thi estion on ectivenes:

Demonstration Tapes

Delete Replace Modify Good
Personnel Type a no 2 a
Officers 0 1 1 3
NCOs 0 0 2 6
Civilians 0 0 Cy) 8
Management 0 ty) 4* 2

*Chi-square tests demonstrated that significant difference( p< .02) existed
between the proportion of modify responses indicated by the Management group as
compared to the rest of the sample. :



c. Data for the Unit Preparation Questions

1. Need for additional advance materials:
Yes: 13 respondents No: 14 respondents
Not sure: 2 respondents

2. Lessons learned about handling advance visits and

unit coordination:

Must continue face-to-face interactions ...........
No recommendations (good as is) ..............-.4--
More time for unit visits ..................2 ee eee
More unit leadership involvement
Better distribution of materials
Other suggestions ...... 0... ee eee eee

A PA UO ©

3. Ways of improving units' preparation for their
VTP rotation:
Make units systematically plan and prepare for

their rotation ......... cece eee eee eee eee 13
Have O/Cs conduct unit visit sooner ............... 7
Have unit leadership emphasize the need for
plan/preparation 1.1... . 0. cece cee ee eee 6
Better distribution of materials ................-. 4
Other suggestions .......... ec eee ee eee 4
4. Ways of improving the demonstration tapes:
Produce more tapeS ........ ec eee eee ee ee eee eee 7
Produce more detailed tapes 7
Provide more cues on tapes 3
Other recommendations ............ eee eect eee ee eens 3
No recommendations made (tapes are good) ......... 14
5. Ways of improving use of the demonstration tapes:
More systematic review by units 12
Disseminate to all unit levels 11
Use during AARs : 7
Other recommendations ............ cece eee ee eee 5



6a.

6b.

Number of
Comments.

Most helpful aspects of the participants’ training:

Workstation/Equipment training ............+-.--05- 15
Conducting the AARS ....... cee eee eee eee eens 8
Other types of training ........... cee eee eee ee eee 11

‘Least helpful aspects of the participants’ training:

NONE 2... cee eee eee eee eee
Using the SIMUTA training/books
Conducting the AARS ........ 0 eee eee eee eee
Using the Unit Performance Assessment System (UPAS)
Other types of training ............. eee eee eee eee

wow AHO

Need for additional training:
On using the workstation/equipment ..............-- 7
On conducting the AARS .......... ee eee eee eee ee ee eee 5
On completing the take-home packages (THPs) . 3
On other aspects of the Virtual Training Program (VTP) 6
No need, as training was adequate ...............-- 12

Procedures for training future O/Cs and ECs:
Having a formal certification program ............- 14
Having a Mentor ........ cee eee eee eee
Reading the instructor manuals .............-------
Completing school house training ................--
Other types of training ........... cee eee eee eee eee

Could unit personnel perform the duties of an O/C or E/C?
Yes: 19 respondents No: 10 respondents

"Yes" respondents: Unit personnel would need to take:

A formal certification program for O/Cs and ECs 3
Hardware/Software training ......... eee e eee eee eee 3
Same training as VTP O/Cs and ECs plus OJT ........ 3
Other types of training ............ eee eee eee ee eee 8
"Yes" respondents: Unit personnel would still have
problems with:
Technical skill requirements . sees : wee 10
Ability 2... ccc ccc eee ee eee eee eeeeee 8
Objectivity 22... eee eee 8
EFficiency 1... .. eee ee ee eee eee eee eee 7



(Items 5a and 5b deal with the questionnaire
items on the time for different VTP activities)

5a. Reasons for indicating that the allocated time
was "too much" for:

Fam COUrS@E 2.2... ee eee eee eee eee eee eee eee eens
Table preview ....... cece eee eee eee

Troop leading procedures
Table execution ......... : :
AAR preparation .................
AARs for platoon (PLT) tables
PLT AARs for company (CO) tables
need leadership involvement (2)

Company AARs for CO tables ..............------
Battalion staff section AARS .............+-00-
Full battalion AARS .......... 0. eee eee ee ee eee

more “time discipline” needed (3)

5b. Reasons for indicating that the allotted time was

"too little" for:

Fam COuUrS@ 22... eee eee eee ee eee eee ee eens

need to train basic skills (4)
need to acclimate to SIMNET (4)

Table preview 1.1... ... cece eee eee eee
Troop leading procedures ..........----ee eens

units not prepared (9)
need more time at SIMNET (5)

Table execution ......... cece cece eee eee

AAR preparation .
0/Cs need flexibility (5)
AARs for PLT tables

PLT AARsS for CO tables ............. eee eee ee eee

leaders not present (2)
15 minutes for AAR is nonsense (2)

CO AARs for CO tables ........... cece ee ee eee

O/Cs need flexibility (3)

Battalion staff section AARS ...............4..

need more time to prepare (3)

Full battalion AARS ........ 0... eee eee eee eee

NOOOROR

Nw



5.

5a.

Comments
Changes needed to the THP materials book?
Okay as is: 9 respondents
Add to it: 9 respondents
Modify parts of it: 4 respondents
Replace parts of it: 3 respondents
Delete parts of it: 1 respondents
Not Sure: 3 respondents
Reasons for:
Modifying the THP materials book .................. 7
avoid duplication/eliminate redundancy (3)
Replacing the THP materials book .................- 5
use AAR tape (2)
Adding to the THP materials book .................-.- 11

videotape unit performance (4)

Ways for increasing the quality of the produced THPs:
Provide AAR tape ....... cece cece eee teens
Automate the process (laptop computers)
Improve the THP format ..............44-
Other ideas ....... cee eee eee eee

Ways for making it easier to produce the THPs:
Automate tools for O/CS 2.1... ec eee eee eee ee eee
Standardization 2... 0... . cece eee ee eens
Nothing (OK aS iS) ...... cc cece cee eee eee
Other WayS 2.2... cece eee eee eee eee eee eee eee

Need for a dedicated staff of specialists to produce
the THPs?
Yes: 14 respondents No: 12 respondents
Not Sure: 3 respondents



5. Estimate the percentage of unit improvement that was a function of

adapting to the SIMNET equipment.
0-25% O 26-50% O 51-75% 1 7e-100% O

UNIT FOLLOW-UP AND TAKE-HOME PACKAGES: These items deal with the
materials sent to the units' home-station after a VTP rotation.

1.

At each level of training, indicate the groups that need feedback materials in a
Take-Home Package (THP). Check as many boxes as you want for each unit
training level.

Group Uni ining Level
Platoon Company Battalion
a. Crews or Squads Oo Oo
b. Vehicle Commanders oOo O oO
c. Platoon Ldrs/Sgts Oo Oo Oo
d. Company Leaders oO oO Oo
e. Battalion Staff oO oO oO
f. Next Superior HQ (to BN staff) (2 O oO

Indicate the usefulness of the THP materials book contents in helping to make up

THPs:
Extremely useful [] Useful [] Somewhat useful C] Not useful 1

Indicate the typical quality of the produced THPs:
Very Good LI Good 0 So-so 1] Poor O Very Poor C]

Indicate the difficulty for experienced O/Cs to produce a THP:
Very Easy Easy 1 So-so 0 Hard Very Hard C1

Estimate how many THPs you worked on at each unit level, and the average
number of hours that you spent working per THP:

Platoon Company Battalion
Number of THPs
Manhours per THP

PIE 59-72 C7



The participants also had the following opinions about
training future VIP O/Cs and ECs:

1. Future members of the O/C team should complete a formal
certification program and/or be tutored by a senior O/C or EC.
This point was indicated by 71% (22/31) of the responses to the
item on training future O/Cs (see B-5).

2. Unit personnel can be trained to perform the duties of an o/c
or EC. Nearly two-thirds (19/29) of the participants made this
assertion (see B-6).

3. Unit personnel would need intensive training with: (a)
having the requisite technical skills for operating the SIMNET
equipment; (b) being objective; (c) doing a quality job; and (d)
being efficient. Such concerns were voiced by participants
regardless of their belief about whether or not unit personnel
could be trained to perform the duties of an O/C or EC. (These
findings are detailed in B-6b and B-6éc.)

4. Two notable strategies for training unit personnel to
overcome the noted deficiencies would be to have them: (a)
complete a formal certification program and (b) receive OJT with
experienced and dedicated O/C personnel. These recommendations,
however, were based on a limited number of comments (see B-6a).

Questionnaire data, The data for these items, which can be

found in Appendix F, provided support for following observations
made in relationship to the corresponding interview data:

1. O/Cs’ training was at least adequate. Eighty-two percent of
them claimed that their training was either good or very good.
Also, over a majority of participants felt that their training
for the different types of duties (e.g., SIMNET tables) was about
right. However, the data for this latter finding were only
significant for operating SIMNET workstations and conducting
SIMNET tables (see Table F-3).

2. A relatively low percentage of participants felt that the
amount of training time was about right for conducting advance
visits, especially vis-a-vis the training time for controlling
the SIMNET exercises (see Tables F-3 and F-4). As a participant
noted: "(SIMUTA) training did not focus on preparing him for
unit visits."

Unit Preparation

Interview data. The most prominent findings from these data
were as follows:

1. As presented in C-1, participants were nearly equally divided

about the need for any additional advance materials. Forty-eight
percent (14/29) of the participants thought that additional

15



EC Handbook (Tables)

Take-Home Package Materials

1b. You indicated that in the TSP should be replaced. Please describe
what you think the replacement should be.
Familiarization Course

Observer/Controller Handbook

Tools and Reference Materials

OCIC Handbook (Tables)

EC Handbook (Tables)

Take-Home Package Materials
4c. Why should be deleted?

14d. Is there some other component you need that should be added? Please describe.

2a. You indicated that in the OCIC Handbook should be modified. Please
describe how you think it should be modified.
OPORD Narratives

Execution Guidance

Table Preview

Event Guides

AAR Guides

AAR Worksheets

2b. You indicated that in the OCIC Handbook should be replaced. Please
describe what you think the replacement should be.
OPORD Narratives

Execution Guidance

Table Preview

Event Guides

AAR Guides

AAR Worksheets

2c. Why should Be deleted?

2d. Is there some other component you need that should be added. Please describe.



Appendix G

Findings for Questionnaire Items on Unit Preparation

Table G-1
Opinio ectiveness o: lateri. Uni tio:
Delete Replace Modify Good

Material a 2 a 2 a 2 2 2
Orientation

guide 1 3.7 1 3.7 6 22.2 19 70.4
Orders and

overlays 0 0.0 3 10.7 17 60.7 8 28.6
Demonstration

tapes 0) 0.0 1 3.7 7 25.9 19 70.4
Table G-2

Response Frequencies by O/C Personnel to the Question on the Effectiveness of the
Orientation Guide

Delete Replace Modify Good
Personnel Type n D n a
Officers 0 1 1 4
NCOs 1 0 1 6 ;
Civilians io} 0 0 7
Management 0 0 4* 2

*Chi-square tests showed that a significant difference (p < .02) existed between
the proportion of modify responses indicated by the Management group as compared
to the rest of the sample.



FOREWORD

Army National Guard (ARNG) units have become an increasingly
important element of post cold-war combat power. These units
however, have limited training resources and time with only 39
days allocated for training per year, including just 15 days for
annual training (AT). Congress has thus provided funding for
establishing the Virtual Training Program (VTP) at Fort Knox, KY
with the intended goal of providing ARNG units with intensive,
time-compressed training opportunities. Implementation of thé
VTP has been broadened to support the training of Active
Component units.

The U.S. Army Research Institute for the Behavioral and
Social Sciences (ARI), the Advanced Research Projects Agency
(ARPA), the National Guard Bureau (NGB), and the U.S. Army Armor
Center (USAARMC) and Fort Knox joined efforts (Memorandum of
Agreement entitled "National Guard Armor Simulation Center,"
April 1993) to develop and implement the VTP. The ARI Armored
Forces Research Unit at Fort Knox accomplished training research
and development for the VIP through a contract effort entitled
"Simulation-Based Multiechelon Training Program for Armor Units
(SIMUTA)," as part of Research Task 2124, "Strategies for
Training and Assessing Armor Commanders’ Performance With Devices
and Simulations (STRONGARM) .”

The present research report examines the VTP’s strengths and
weaknesses from the perspective of its original instructional
team. This examination involved having these instructional
personnel complete questionnaires and be interviewed regarding
their opinions on various aspects of the VTP. The information in
this report has been provided to training developers and
instructors in the 16th Cavalry Regiment at Fort Knox. It will
also be useful to personnel involved in the development and
implementation of structured simulation-based instructional
programs for future simulation-based training systems, such as
the Close Combat Tactical Trainer.

ZITA M. SIMUTIS EDGAR M. JOHNSON

Deputy Director Director
(Science and Technology)



Appendix H

Results for Questionnaire Items on Training Management

Table H-1
Opinii on the Suppo: ckage
Delete Replace Modify Good

Component n Bz a Bz 2 Bz 2 Bz
Familiarization

Course 0.0 0 0.0 7 25.0 21 75.0%
Observer/Controller

Handbook 0.0 C) 0.0 13 48.1 14 51.9
Tools and Reference

Materials 0.0 0 0.0 11 40.7 16 59.3
OCIC Handbook

(Tables) 0.0 0 0.0 15 55.6 12 44.4
EC Handbook

(Tables) 3.6 0 0.0 15 53.6 12 42.9
Take-Home Package

Materials 0.0 4 13.8 9 31.0 16 55.2
*p < .02 testing the hypothesis P = 50.
Table H-2
Opinions on the OCIC Handbook Components

Delete Replace Modify Good

Component a Bz a 2 a 2 2 B
OPORD Narratives 0.0 0 0.0 1. 39.3 17 60.7
Execution Guidance 0.0 C) 0.0 3 11.5 23 88.5%
Table Preview 0.0 0 0.0 5 18.5 22 81.5*

(Table H-2 Continued)



Table G-5

Means and Standard Deviations of Participants’ Estimates to Unit Preparation
Items 2-5

Item Stem XN M s.D
Importance of 0/Cs‘ advance visits 28 3.64% 0.488
Percentage of prepared VTP units 29 2.06" 0.753
Demo tapes emphasized unit tasks 25 4.00° 0.645
Demo tapes useful to train units 23 2.834 0.778
“1-4 scale with 4 as "very important."and 1 as “very unimportant.” *1-4 with 4 =

76-100% and 1 = equaling between 0% and 25% prepared; °1-5 scale with 5 as “very
good” and 1 as “very poor;" “1-4 scale with 4 as “extremely useful”and 1 as “not
useful.”



Appendix I

Findings for Questionnaire Items on Training Structure

Table I-1
Opinions the VIP's Struct:

Strongly Don't Agree Strongly

Disagree Agree Agree
Aspect of VTP 2 BR pn B n RB peu BR
Stealth preview C) 0.0 1 3.4 1 37.9% 17 58.6*
VTP tables C) 0.0 4 13.8 13 44.8% 12 41.4*
Progression 1 3.4 0 0.0 19 65.5* 9 31.0*
Critical subtasks 0 0.0 1 3.4 17 58.6* 11 37.9%
Subtask repetition C) 0.0 0 0.0 18 62.1* 1 37.9%
AAR structure 0 0.0 1 3.4 8 27.68 20 69.0*
AAR tools 0 0.0 1 3.4 13 44.8% 15 51.7*
*p < .02 testing the hypothesis P = .50 for the combined percentages in the rating

categories "Agree" and "Strongly Agree."

Table I-2

o/e to the Question on Usefulness Subtask

Strongly Don't Agree Strongly

Disagree Agree Agree

a a 2 n

Officers 0 0) 1
NCOs C) 0 6 2
Civilians 0 0 2* 7
Management 0 0 5 1
*Statistical tests indicated a significant difference (p = .03) between the

proportion of strongly agree responses for the Civilian group as compared to
proportion for the rest of the sample. (Significant differences vis-a-vis the
different O/C groups were not found for the other Training Structure items.)



participants, this option should be used as a last resort and
only after the unit personnel have been thoroughly trained to be
an O/C.

Another ramification is that VTP-like instructional programs
should be developed for the training systems of the future; e.g.,
the CCTT. Such future generations of the VIP should, of course,
be tailored to meet the capabilities of the future training
systems. This development should also involve a close
relationship between the user and the instructional design team.
And, this development can be enhanced vis-a-vis the lessons
learned by this investigation’s participants, who were the first
generation of VTP instructors.

In closing, the VTP, despite its minor flaws, was a
rewarding experience for the sampled instructional personnel. As
noted by an O/C:

"Professionally, the VIP was the most rewarding experience
that I have had with regards to people & outcomes...."

26



Procedure

Each participant completed the self-administered
questionnaire and was interviewed during one two-hour session.
Three Armored Forces Research Unit (AFRU) personnel--two research
psychologists and a graduate student intern--conducted these
sessions with the stipulation that two of them be present at a
session.

At the beginning of each session, the participant was
informed about the purpose of this investigation and the specific
procedures that would be followed. He/she was also given the
opportunity to ask any questions about this investigation. The
participant then read and signed the consent form that was
attached to the questionnaire. Also, each session took place at
a private area in the AFRU. Hence, ethical guidelines prescribed
by the Army Research Institute and the American Psychological
Association were followed during this data collection process.

The participant then completed the self-administered
questionnaire without any distractions or interruptions from the
AFRU personnel. After approximately twenty minutes, the AFRU
personnel went over the questionnaire with the participant to see
if he/she had any problems with completing it. The AFRU
personnel also checked to see if the participants had answered as
many questions as possible. Because of their duty position, some
participants might not have been able to answer all the
questions.

After a short break, each participant was then interviewed
for approximately ninety minutes. The lead interviewer asked
each question as it was stated on the interview form, while the
second AFRU person took notes and asked the participant to
elaborate upon some of his/her answers. The lead interviewer
also recorded the participant’s responses onto an interview form.
In addition, each interview was tape-recorded with the
participants’ permission. After the interview was completed, the
participant was thanked for his/her participation and allowed to
return to his/her regular duties.

Scoring and Data Analyses
Interview data Three judges--the AFRU personnel who

interviewed the participants--scored and analyzed these data by
using the following scheme:

1. The lead interviewer recorded onto a master scoring sheet a
paraphrased account of the participant’s response(s) to a
question. If another participant had made a similar response,
the lead interviewer then put a tick mark by the “original
response.”

2. The assistant interviewer reviewed the notations made by the
lead interviewer.

12



significantly different from 50% for the Fam course and the AARs

for PLT AARs during CO exercises.

2. However, when interviewed, nearly 72% (21/29) of the
respondents stated that changes should be made to the allocation
of time for the different VTP activities. The most cited change
involved making the time for AARs more flexible. (These findings
are delineated in D-1 and D-1a.)

3. Making the AAR time more flexible was also noted by several
participants in reference to their reason(s) for indicating on
the questionnaire that too little time was spent on the AARs (see
D-5b).

4. A consensus was not obtained regarding the importance of
having units repeat the Fam course upon return trips to the VTP.
Thirty-five percent (11/31) of the responses to this interview
item concerned the importance of always taking the Fam course;
26% (8/31) noted the relationship between taking the Fam course
and the degree of turbulence within the unit; 23% (7/31)
indicated that taking the Fam course depended upon the time
between VTP rotations. (These data can be found in D-2.)

5. Further evidence was provided about the lack of preparation
by the units for their VIP rotations. Problems with unit
preparation was the most noted reason given by participants for
claiming on the questionnaire that too little time was spent
during the VTP on troop leading procedures (see D-5b). .

inin ta
Interview data, These data demonstrated the following:

1. The participants had few problems with the basic arrangement
of the VTP tables. Ninety percent (26/29) of them thought that
there was adequate task overlap and repetition among the VTP
tables and that the VTP tables did tend to increase in difficulty
(see E-6 and E-1). Sixty-six percent (19/29) of them,
consequently, felt that the order of the tables was fine. This
last finding is manifested by the “no” responses to item E-3.

2. They, however, expressed some concern about the arrangement
of the PLT tables. Thirty-one percent (9/29) of the comments to
these items dealt with the fundamental tables being difficult for
the VIP participants to complete than the offensive/defensive
tables. In addition, six comments were about the PLT tables
being harder than the CO tables. (See E-2 for account of the
findings cited in this paragraph) .

3. Questions remain about adding more tasks or subtasks to the
VIP tables. As shown in E-4, forty-eight percent (14/29) of the
participants did not believe that any MTP tasks or subtasks
should be added to the VTP tables; while 41% (12/29) felt that
some MTP tasks (e.g., movement techniques or formations--see E-

18



important criterion for selecting an O/C, accounting for 29%
(12/42) of the items on selecting O/Cs. Finally, tactical
knowledge and computer skills were a particular concern for
selecting ECs as manifested by these categories representing 51%
(28/55) of the responses to the item on selecting ECs.

Trai he Trai

Interview data. The participants’ responses to these items
indicated the following about their training to be a VTP O/C or
EC:

1. Their training was, at least, adequate. This point was
manifested by the relatively high percentage of participants (36%
or 12/33) who believed that they did not need any additional
training (see B-4). In addition, 29% (9/31) of them felt that
all of their training time was well spent (see B-3)

2. The two most prominent types of training received by the
participants involved using the SIMNET or Janus
workstations/equipment and doing an AAR, with comments about
using the workstation accounting for 50% (22/44) of the responses
to the item on types of training received (see B-1).

3. The participants indicated that their training on using the
workstations/equipment was the most helpful part of their O/C or
EC training. This response category represented 44% (15/34) of
the responses to the item on the most helpful aspects of 0/C or
EC training. Also, 24% (8/34) of the comments on the helpful
aspects of their training concerned the training on doing an AAR.
(See B-2 for the data cited in this paragraph.)

4. Ironically enough, the participants also felt that they
needed more training on using the workstations/equipment and
doing an AAR. As presented in B-4, these categories accounted
for 36% (12/33) of the responses to the item on training not
provided.

5. Several participants correspondingly expressed a need for
more hands-on training in using the SIMNET and Janus
workstations. These comments were made by participants when
asked to provide reason(s) for claiming on the questionnaire that
they received too little training time on learning how to operate
these pieces of equipment (see B-7b).

6. As presented in B-3, 59% (13/22) of the comments regarding
the least helpful aspects of training dealt with the
"instructional program" provided by the contracting team; i.e.,
training on the workstation or on the Unit Performance Assessment
System (UPAS-see Meliza, Bessemer, & Tan, 1994 for a description
of this system), and AAR. (The denominator 22 comes from
subtracting 9, the number of the “none responses,” from the
data.)

14



References

Bessemer, D. W. (1991). Transfer of SIMNET training in the Armor
Officer Basic Course (Tech. Rep. No. 920). Alexandria, VA:
U.S. Army Research Institute for the Behavioral and Social
Sciences. (AD A233 198)

Bessemer, D. W., Shlechter, T. M., Nesselroade, K. P., & Anthony,
J. (1995). Effectiveness of structured training in
Simulation Networking (SIMNET). Paper presented at the 17th
Interservice/Industry Training Systems Conference,
Albuquerque, NM.

Campbell, C. H., Campbell, R. C., Sanders, J. J., & Flynn, M. C.
(1995). Methodology for development of structured
simulation-based training (Research Product 95-08).
Alexandria, VA: U.S. Army Research Institute for the
Behavioral and Social Sciences. (AD A296 171

Elliott, G. S., & Sanders, W. S. (1996). Training in a digitized
battalion task force: Lessons learned and implications for
future training. Alexandria, VA: U.S. Army Research
Institute for the Behavioral and Social Sciences. Manuscript
in preparation. :

Garvey, E. E., & Radgowski, T. (1988). SIMNET-D standing
operating procedure (Report No. 6929). Cambridge, MA: BBN
Systems and Technologies.

Hoffman, R. G., Graves, C. R., Koger, M. E., Flynn, M. E., &
Sever, R. S. (1995). Developing the RCVTP: History and
lessons learned (Research Rep. No. 1675). Alexandria, VA:
U.S. Army Research Institute for the Behavioral and Social
Sciences. (AD A296 -153)

Meliza, L. L., Bessemer, D. W., & Tan, S. C. (1994). Unit
Performance Assessment System (UPAS) development (Tech. Rep.
1008). Alexandria, VA: U.S. Army Research Institute for the
Behavioral and Social Sciences. (AD B191 576

Morrison, J. E., & Holding, D. H. (1990). Designing a gunnery
training strategy (Tech. Rep. No. 899). Alexandria, VA: U.S.
Army Research Institute for the Behavioral and Social
Sciences. (AD A226 129

Shlechter, T. M., & Anthony, J. (1996). An examination of the
value of demonstration tapes for the Virtual Training
Program (Research Rep. No. 1688). Alexandria, VA: U.S. Army
Research Institute for the Behavioral and Social Sciences.

27



(1996) --with the unit preparation aspects of the VTP?

The O/Cs and ECs can also provide some unique insights into
this program’s strengths and weaknesses. They may have
difficulty, for example, with developing the THPs. Or, they may
not use some VTP materials because of factors that were not
readily apparent to the developers. Given their high degree of
experience with the VTP, these individuals can also provide
valuable lessons learned for possible expansion of structured
simulation-based training.

Objectives of This Evaluation

This investigation was thus designed to examine,
systematically, the opinions of the VTP’s instructional personnel
about various aspects of this program. Information collected
during this investigation should help answer questions about the
VTP’s training value, and correspondingly help determine the need
for making any changes to this program. This investigation’s
data could also help guide the design and development of
structured training for future training systems; e.g., the Close
Combat Tactical Trainer (CCTT).

Method

rtici

The participants were 29 members (6 officers, 8 non-
commissioned officers--NCOs, 9 civilian ECs and 6 management
personnel) of the O/C team. The management personnel included 4
officers, 1 NCO, and 1 civilian. In addition, this sample
consisted of 27 males and two females who were civilian ECs.

These participants were chosen because they had been with
the O/C team for more than six months. All of the military
participants had been associated with the VTP from October 1993
to Summer 1995. Nearly all of the civilian personnel had been
with the VIP since January 1994. (See Appendix A for a listing
of these participants’ military experience and their experience
as VTP O/Cs and/or ECs.)

instruments

The instruments consisted of a structured interview form and
a self-administered questionnaire. The development process for
both instruments involved: (a) developing a draft outline of
each instrument; (b) assigning individual authors to prepare
specific sections; (c) reviewing the draft sections; (d) piloting
the instruments; (e) rewriting any problematic sections; and (f)
determining the procedures for administering the instruments.

The structured interview form. As shown in Appendix B, this

instrument contains the following sections:



Table H-2 Continued

Delete Replace Modify Good

Component a 2 2 B a B a 2
Event Guides 0 0.0 0 0.0 2 7.4 25 92.6*
AAR Guides 0 0.0 1 3.7 2 7.4 24 88.9%
AAR Worksheets 0 0.0 5 17.9 4 14.3 19 67.9
*p < .02 testing the hypothesis P = 50.
Table H-3
Opinions on the Time Available for SIMNET Activities

Too About Too

little right much
Type of activity a Bz a 2 B
Familiarization Course 1. 39.3 17 60.7 0.0
Table Previews 1 3.6 27 96.4% 0.0
Troop Leading Procedures 15 53.6 13 46.4 0.0
Table Execution 2 6.9 27 93.1* 0.0
AAR Preparation 6 20.7 23 79.3% 0.0
Pp < .02 testing the hypothesis P = 50.
Table H-4

ini i \vailable for AARs _b

Too About Too

little right much
Unit echelon a B 2 2 a Bz
Platoon 1 3.6 27 96.4% 0 0.0
Platoon in Company Exercise 7 25.9 18 66.7 2 7.4
Company 4 14.8 22 81.5* 1 3.7
Battalion Staff Section 4 14.8 23 85.2* 0 0.0
Full Battalion 4 14.8 20 74.1% 3 11.1

R < .02 testing the hypothesis PB = 50.



7a.

Comments
Changes to the AAR agenda or responses
indicated by the "yes" respondents:
Make it made more flexible ................0.00005-
Need for additional presentation tools
Need to automate presentation tools ..
Other TeaSONS .... eee eee eee ee ee eee

orun

Most useful AAR aids:

Data logger with voice (linked to stealth) ........ 20
Events poster ............ eee eee sees Lee cece e eee eee 16
Magnetic tanks, vehicles, etc 5
Other 2. eee ee eee eee e eee eens 6
Suggested additions or changes to the AAR aids:

UPAS oe ce ee ee ee ee eee teen teenies 9
Bigger/wider plan view display screen 6
Visual aids ........ cece cee eee eens 4
Modified Semi-Automated Forces (MODSAF) 1.5 ....... 3
Other changes or additions ...............e eee eeeee 7
Data for the Training Proficiency Questions

Aspects of leader performance that increased the most:
Command and control .. . 19
Situational (battlefield) awareness ............... 8
Leadership/Confidence in one's ability ..........-. 6
Other ASPECtS ©... Lecce teen eee 1
Aspects of leader performance that increased the least:
Troop leading procedures .......... 22. e cece 8
At BN level, delegating authority 7
Coordination of PLT fires : 3
Navigation and map reading skills 3
Other ASPECtS .... eee eee ee ee ee ee 4
Aspects of unit performance that increased the most:
Reporting ...... eee eee ee eee eee . oe : 12
Actions on contact 11
Formations 2.2... .. eee cece eee eet tees 10
Teamwork/Cohesion 9
Other aspectS ........ eee eee eee 3



Perspectives on the Virtual Training Program From Members of Its
Initial Observer/Controller Team

Introduction

This report examines the value of the Virtual Training
Program (VTP) as viewed by members of the program’s original
observer/controller (O/C) team. This team, made up primarily of
active duty military personnel, served as the program’s
instructional personnel from its implementation in early 1994 to
the present time. Nearly all of the team's original military
members rotated to other assignments during the summer of 1995.
During that summer, these members of the O/C team were surveyed.

The VTP

Army National Guard (ARNG) units have become an increasingly
important element of post cold-war combat power. These units,
however, have limited training resources and time with only 39
days allocated for training per year, including just 15 days for
annual training (AT). Congress has thus provided funding for
establishing the VIP at Fort Knox, KY with the intended goal of
providing ARNG units with intensive time-compressed training
opportunities. Implementation of the VTP has been broadened to
support the training of active component (AC) units.

The VTP instructional design and development processes were
accomplished by the Simulation-Based Multiechelon Training
Program for Armor Units (SIMUTA) contractor team, which consisted
of a consortium of subject-matter experts (former military
personnel), instructional designers, and evaluators. This
contractual effort was monitored by the U.S. Army Research
Institute's Armored Forces Research Unit at Fort Knox. | (See
Hoffman, Graves, Koger, Flynn, & Sever, 1995 for a detailed
description of the SIMUTA project.)

raining Systems Associated With the VTP

Providing VTP participants with intensive time-compressed
training involves the use of the Simulation Networking (SIMNET)
and Janus training systems. Since the original O/C team directed
a relatively few Janus training rotations, this report focuses on
the VTP’s SIMNET components.

SIMNET is used mainly to support platoon (PLT) and company
(CO) training. It consists of the integrated use of training
simulators with combat and combat support: simulations operating
under constraints similar to those found in battlefield
conditions. SIMNET also employs the use of modified semi-
automated forces (ModSAF), allowing computer generated vehicles
(e.g., opposing forces) to operate on the SIMNET battlefield.
(ModSAF is currently available in version 1.5 but VTP exercises
have not been updated to this version.) SIMNET is thus
considered to be a virtual training system.



ner. mm

The final set of interview questions indicated the following
about the participants’ views toward the VTP:

1. The VTP’s cost-effectiveness, structured design, and AAR
feedback system were the three most beneficial aspects of this
program. These three response categories accounted for 91%
(70/83) of the participants’ comments regarding the most
beneficial aspects of this program (see H-1a to H-1ic).

2. The three most problematic aspects of this program involved:
(a) aspects of the VTP’s hardware/software; (b) lack of
standardization across echelon with such items as the overlays
and orders; and (c) aspects of the training development process.
These three items represented 84% (67/80) of the comments on the
item dealing with needed modifications to the VTP (see H-2a to H-
2c).

3. Regarding the issue of standardizing the OPORDs, the
participants also had problems with the narrative format for
presenting this information. OPORDs are typically presented in a
paragraph format with each paragraph delineating another aspect
of the mission.

4. Use of structured training and increased knowledge of
military operations and doctrine were the two most mentioned
lessons learned by these participants. These two categories
accounted for 59% (17/29) of stated lessons learned by these
participants (see H-3).

5. Nearly 80% (23/29) of the participants thought that both ARNG
and AC units could benefit from the VTP. A primary reason stated
for this belief was that both ARNG and AC units faced operating
tempo constraints (see H-4).

6. Seventy-two percent (21/29) of the participants believed that
structured training should be developed for mission planning and
preparation and for future simulation systems, e.g., the CCITT.
Regarding the CCTT system, 12 participants cautioned that the
VTP's instructional content must be tailored to the CCTT
capabilities. (Data presented in this paragraph relate to those
described in H-5 and H-5a)

7. The participants' positive feelings toward the VTP were
finally manifested in their responses to the last interview
question. Sixty-nine percent (20/29) of them made positive
comments about the VTP, while the remaining 31% (9/29) thought
that some aspects of the VIP should have been designed
differently. Regarding the latter, problems were noted with the
SIMUTA instructional design team. As mentioned by one
participant, "(L)ots of promises by the SIMUTA folks which just
weren’t delivered."

24



Ordered Categories

These items were as follows: (a) #2 from the Train the
Trainer set; (b) #1 from the Unit Preparation set; (c) #1-#3 from
the Training Management set; and (d) #1 from the Unit Follow-Up
and Take-Home Package set.

Differences in distributions of responses among the Warthog
Observer/Controller groups (Officers, Non-commissioned Officers,
Civilians, and Management) were tested by the rank-order Kruskal-
Wallis test for k = 4 independent samples with a = .05. Follow-
up Mann-Whitney pairwise comparisons used a = .02.

Commonly, when group differences were not statistically
significant, single-sample tests of proportions were performed
just as done for unordered categories. Binomial tests of the
proportion of responses in a selected category (or sometimes two
categories together) for all groups of O/Cs combined were done
with the null hypothesis p = .50 and a_= .02. This tested
whether a statistically significant majority of O/Cs agreed on
the same response, or disagreed with that responses.

Again, some sets of items asked the same question about
members of a class of entities. When group differences were not
found statistically significant for all items in the set,
differences of O/C opinion among the entities were of interest.
Then, the combined samples of ordered responses were compared
among items using the Friedman test for k related samples with a
= .05. Pairwise comparisons used the Wilcoxon signed-ranks test
with a = .02.

Numerical Estimates

Numerical estimates were provided for items la-2e from the
Training Proficiency set and item 5 for the last set of items.
Descriptive statistics for these items were obtained using the
SPSS Explore and Tables procedures. For individual variables,
O/C group means were compared by analysis of variance with a =
-05, and follow-up pairwise t-tests with a = .02.

For Training Proficiency item groups la-le and 2a-2e, 4x5
x 2 repeated-measure multivariate analyses of variance were done,
with a = .05 for each main effect and interaction. In these
analyses the four O/C groups served as the between-subjects

E-2



Appendix F
Data for Questionnaire Items on Train the Trainer
Table F-1

Response Frequencies of the Judgements by 0/C Personnel Regarding the Quality of

Judgements Officers NCOs Civilians Management
(n=6) (n=8) (n=9) (n=6)

Very Poor 0 0 0 0

Poor 0 0 1 1

So-So 1 0 2 C)

Good 5 5 2 5

Very Good 0 3 4 0

Significant differences among groups not found ( p > .05) in

the chi-square and Kruskal-Wallis tests.

Table F-2

Mi -Whitne: ar. of 0. +s Judgem, i the
Their Training

Type of
Personnel Cases Mean Ranks Sum of vu Ww Z Probability

Ranks Value Value Value Level

Officer/ 12 12.13 145.5

Management

Vs. 67.5 145.5 -1.72 08
Civilian 17 17.03 289.5

Table F-3

the Amoun: ainin eceived C or ties



Too About Too
little right much

Type of duty pal 2B a 2 2
Conducting advance

visits 11 45.8 13 54.2 0.0
Operating SIMNET

workstations 7 24.1 22 75.9% 0.0
Operating Janus

workstations 10 34.5 19 65.5 0.0
Controlling SIMNET

tables 3 10.3 24 82.8* 6.9
Controlling Janus

exercises 8 27.6 19 65.5 6.9
Conducting AARs 7 26.9 18 69.2 3.8
*p < .02 testing the hypothesis PB = 50.
Table F-4
lative is_of endal to)
Coordinance
Type of duty 2 Mean Ranks
Conducting advance 23 2.11*

visits

Operating SIMNET 23 2.46

workstations
Controlling SIMNET 23 2.85

Tables
Conducting AARs 23 2.59

*P < .02 for the comparison between conducting advance
visits and conducting AARs.



Area of Proficlency

‘Overall performan:

Occupying and defending BP
Executing battle drills WS
Making reports — SS
Executing formations and changes -; WS
T T
tL) 20 a 60 80 100
Proficiency Estimate
Legend

W\ End Start

Figure 2. Estimates of unit proficiency before and after
training in the VTP.

1. This program was quite effective for training unit leaders
and units. The participants indicated that both unit leaders and
units progressed from proficiency levels of approximately 30% to
those around 70% for leaders and 80% for units. These reported
differences between the participants’ starting and ending levels
of proficiency were statistically significant (see Appendix

Jd).

2. Support was provided for the interview data’s findings
regarding the most effective and least effective aspects of the
VTP. Estimates of the unit leaders’ proficiency increased the
most for command and control (controlling unit fire and maneuver)
and situational awareness (monitoring the battlefield situation)
with approximately 43% increases for both. These estimates
increased the least for troop-leading procedures with a 33.90%
increase. In addition, the estimates of unit performance
increased the most for reporting and the least for defensive
tactics; i.e., occupying and defending battle positions. (See
Tables J-2 and J-3 for a more detailed account of these results.)

3. Coaching by the VTP instructional personnel was an important
factor in this program’s instructional effectiveness. Nearly
three-quarters of these participants thought that more than 51%
of unit improvement was a function of this factor.

21



Appendix K

Data for Questionnaire Items on Unit Follow-up
and Take-Home Packages (THPs)

Table K-1
ero: ed it Li and o/c Per:
Platoon Company Battalion
Level Level Level
o/c
Group N  Mean* sD No Mean* sD N Mean sD
Officers 6 77.50 47.83 5 34.00 15.17 5 18.60 6.50
NCOs 8 36.75 12.66 1 1.00 0.00 5 11.60 3.21
Civilians 8 21.00 20.47 5 9.80 8.81 2 31.50 33.23
Management 4 3.75 4.27 2 2.00 0.00 4 8.75 8.38
Total 26 36.23 35.69 13.9 °417.23 (17.42 1615.56 12.50

*Significant differences among means at .02 level.

Table K-2

Results of Significant (p < .05) Mann-Whitney Comparisons in the Number of THP:

Worked by 0/C Personnel

Comparisons U-Value W-Value Z-value 2
Platoon Level

Officers vs

Civilians 4.00 40.00 2.62 0.01
Officers vs.

Management 0.00 10.00 2.57 0.01
NCOs vs.

Civilians 12.00 48.00 2.11 0.03
NCOs vs.

Management 0.50 10.50 2.66 0.01

Company Level
Officers vs
Civilians* 1.00 16.00 2.43 0.02

"Only comparison which can be made because of the small ns for NCO and Management
Personnel (see Table K-1).



participants indicated that unit leaders and units became more
proficient during the course of their VIP rotation, and that this
improvement was not simply a function of adapting td the SIMNET
terrain and equipment. Correspondingly, the participants had few
problems with most aspects of the VTP.

This report’s findings also suggested the need for fine-
tuning the VTP. Among the most notable suggestions made by the
participants for fine-tuning the VIP were: (a) reorganizing the
handbook for the O/C In-Charge, (b) standardizing the operations
order narratives across echelons and to military doctrine, and
(c) using afteraction review tapes to improve the THPs. The
participants also noted that the units had to be better prepared
for their VTP rotations.

Utilization of Findings:

This report has ramifications for military trainers and
instructional designers. The utility of using structured
instructional programs (e.g., the VTP) for simulation-based
training systems has been further substantiated. Consequently,
such instructional programs could be incorporated into the
simulation-based training systems of the future, e.g., the Close
Combat Tactical Trainer.

viii



PERSPECTIVES ON THE VIRTUAL TRAINING PROGRAM FROM MEMBERS OF ITS
INITIAL OBSERVER/CONTROLLER TEAM

EXECUTIVE SUMMARY

Research Requirement:

Army National Guard (ARNG) units have become an increasingly
important element of post cold-war combat power. ARNG soldiers
must be trained for their new roles in the post cold-war
military. To support the needed ARNG training, Congress has
provided funding for establishing a Virtual Training Program
(VTP), which utilizes the available training technologies at
Fort Knox, KY, including the Simulation Networking (SIMNET)
system. Implementation of the VTP has been broadened to support
the training of Active Component units.

This research effort examined the opinions of the VTP’s
instructional personnel about various aspects of this program in
order to ascertain the VTP’s strengths and weaknesses.
Correspondingly then, the data collected from this effort would
help determine the need for making any changes to this program.
This investigation’s data would also help determine the value of
incorporating aspects of the VTP into future instructional

programs.

Procedure:

Twenty-nine members (6 officers, 8 non-commissioned
officers--NCOs, 9 civilian, and 6 management personnel) of the
VTP’s instructional team participated in this investigation. The
officers and NCOs primarily served as observer/controllers
(0/Cs), while the civilians were the exercise controllers (ECs).

The participants completed self-administered questionnaires
and structured interviews regarding their opinions about the

following aspects of the VTP: (a) train the trainer, (b) unit
preparation, (c) training structure, (d) training proficiency,
and (e) unit follow-up and take-home packages (THPs). These two

instruments also ascertained information about the participants’
previous military experience and duties as an O/C or EC. The
participants were also asked questions during their interview
session about such matters as the: (a) criteria for selecting an
O/C or EC, (b) beneficial and problematic aspects of the VTP, and
(c) usefulness of structured instructional programs (e.g., the
VTP) for simulation-based training systems.

Findings:

Findings from the questionnaires and interviews provided
‘further support for the VTP’s instructional value. The

vii



1. Demographic Information. This section contains questions
about the participants’ feelings on the: (a) value of their
previous experiences in helping them to become an effective 0/C
or EC and (b) qualifications for being selected as an O/C or an
EC.

2. Train the Trainer. This section consists of two parts. The
first part contains questions about the participants’ training to
be a VTP O/C or EC. The second part deals with the participants’
reasons for indicating on the self-administered questionnaire
that they received either "too much" or "too little" training for
the sampled VIP functions.

3. Umit Preparation. This section is comprised of three parts.
Part 1 deals with the participants' responses to the self-
administered questionnaire items on modifying, replacing, or
deleting the different VIP preparation materials. Part two
contains questions about: (a) the need for additional materials;
(b) ways of better handling the advance visits; and (c) ways of
encouraging units to be better prepared for a VIP rotation. The
third part pertains to the participants’ opinions about the
demonstration tapes.

4, Training Management. This section contains three parts with
each part pertaining to a different set of questionnaire items on
this topic. Part I probes the participants’ reasons for
indicating on the self-administered questionnaire that the VTP
components need to be modified, replaced, or deleted. Part II
does the same for items dealing with the OCIC Handbook
components. Both parts also contain a question about the need
for developing additional components. Part III consists of
questions about the allocation of time for the different VTP
activities and about re-familiarizing units with the VTP.

5. Training Structure. Questions about the VIP tables’ C-W-R
framework are asked in this section. In addition, this section
includes questions about the use of Mission Training Plan (MTP)
subtasks and the AAR process.

6. Training Proficiency. This section asks the participants to
identify the components of both leader performance and unit
performance that increased the most and least during VTP
rotations.

7. Unit Follow-Up and the THPs. Participants are quizzed in
this section about: (a) tailoring the THPs for different units
and levels within a unit; (b) changing the THP materials book;
and (c) making it easier to produce the THPs. They are also
asked to comment on the use of these materials by ARNG and AC
personnel and the need for a dedicated staff of specialists
(e.g., operations analysts, writers, graphic/media editors, etc.)
to produce the THPs.

8. General Comments. This section contains questions about the

10



CONTENTS (Continued)
Page
APPENDIX DATA FOR QUESTIONNAIRE ITEMS ON
TRAIN THE TRAINER wee F-1
FINDINGS FOR QUESTIONNAIRE ITEMS
ON UNIT PREPARATION . oe G-1
RESULTS FOR QUESTIONNAIRE ITEMS ON
TRAINING MANAGEMENT . . H-1
FINDINGS FOR QUESTIONNAIRE ITEMS ON
TRAINING STRUCTURE : . I-1
RESULTS FOR QUESTIONNAIRE ITEMS ON
TRAINING PROFICIENCY toe ee J-1
DATA FOR QUESTIONNAIRE ITEMS ON UNIT
FOLLOW-UP AND TAKE-HOME PACKAGES (THPs) K-1
LIST OF FIGURES
Figure Estimates of leader proficiency before and
after training in the VTP woe ee 20
Estimates of unit proficiency before and
after training in the VTP tote 21



Table K-5

Participants’ Opinions about the Usefulness Of the THP Materials Book, the Qualit:

the Produced s the HP etio) 8
Item Mean sD
Usefulness of THP materials book 2.89% 0.885
Quality of the Produced THPs 4.07% 0.716
Ease to Produce the THPs 3.75° 0.701

“4=extremely useful, 3=useful, 2=somewhat useful, 1=not useful. "S=very good,
4=good, 3=so-so, 2=poor, l=very poor. “S=very easy, 4=easy, 3=so-so, 2shard, l=very
hard.



materials were not needed; while 45% (13/29) thought that
additional materials were needed.

2. Participants tended to believe that the procedures associated
with advance visits and unit coordination could be improved.

This point was manifested by 58% (19/33) of the responses to the
item on better ways for handling unit visits, although
approximately one-quarter (8/33) of the participants did note the
importance of continuing the face-to-face interactions with the
units. (This paragraph’s data can be found in C-2.)

3. The most cited procedures for improving units visits were:
(a) having more time for unit visits; (b) getting the unit
leadership more involved; and (c) providing better distribution
of materials to the units (see C-2).

4. The participants were concerned about units’ lack of
systematic planning and preparation for their VIP rotation. This
category accounted for nearly 40% (13/34) of the comments on ways
to improve units’ VTP preparation (see C-3).

5. A consensus was not reached regarding the quality of the
developed demonstration tapes (see C-4). Forty-eight percent
(14/29) of the participants believed that the demonstration tapes
were fine. The remaining 52% made the following comments on ways
of improving these tapes: (a) have more tapes; (b) make the
tapes more detailed; and (c) have more cues on the tapes. The
last suggestion was based on a limited number of responses.

6. The two most salient suggestions for improving use of these
tapes involved: (a) having units systematically review them and
(b) disseminating tapes to all unit levels. These findings
parallel those found in Shlechter and Anthony’s (1996) interviews
of VIP participants. The participants also suggested that the

tapes be used during the AARs. (Findings cited in the paragraph
refer to the data presented in C-5).
Questionnaire data. The pertinent data associated with

these items can be found in Appendix G. These data revealed the
following additional information about participants’ opinions on
unit preparation:

1. The orientation guides and demonstration tapes were fine as
more than 70% of the participants found them to be adequate.

2. The orders and overlays were problematic. Sixty percent of
the participants wanted these materials to be modified or
replaced. In addition, nearly half of the participants stated,
during their interviews, that the overlays and orders needed to
be modified in order to be error-free and conform to Army
doctrine (See C-6a for an account of these comments.)

3. Management personnel, however, were the most likely to have
problems with the different unit preparation materials.

16



U.S. Army Research Institute
for the Behavioral and Social Sciences

Research Report 1691

Perspectives on the Virtual Training Program
From Members of Its Initial
Observer/Controller Team

Theodore M. Shlechter, Ronald E. Kraemer,
David W. Bessemer, and Billy L. Burnside
U.S. Army Research Institute

James Anthony
University of Louisville

DTIC QUALIEY UNGPHCTED B

19960619 072 says

Approved for public release; distribution is unlimited.



TRAIN THE TRAINER: For each item, check the box that best represents your opinion
regarding the effectiveness of your training as an O/C or E/C.

1. The quality of your O/C or E/C training was:
Very good L] Good LJ soso 1 Poor O Very Poor

2. The amount of training that you received for each of the following duties was:

Too About Too
Little Right Much
a. Conducting advance visits Oo oO Oo
b. Operating the SIMNET workstations oO Oo Oo
c. Operating the Janus workstations oO | oO
d. Conducting SIMNET tables oO oO oO
e. Conducting Janus exercises O O Oo
f. Conducting AARs Oo Oo O

UNIT PREPARATION: For each item, check the box that most closely represents your
opinion regarding unit preparation for a VTP rotation.

1. Indicate your opinion about the effectiveness of each of the following materials:
Good Needstobe Needstobe Needs to be
as is Modified Replaced Deleted

a. Orientation Guide oO oO oO Oo
b. Orders and overlays Oo Oo oO oO
c. Demonstrationtapes O Oo O |

2. The O/Cs’ advance visits to units’ home-stations were:

Very Important [J Important [1 Unimportant [J Very Unimportant

PT# 59-72 c-3



Structure of the VIP tables, A structured instructional

program is not an inherent component of SIMNET. The
effectiveness of previous SIMNET training programs has
correspondingly been a function of the instructional techniques
employed by instructors or unit leaders (Bessemer, 1991). Some
units may then have benefited from SIMNET use while others may
not have. The SIMUTA instructional design team thus had to
develop a structured set of tables in order to maximize SIMNET's
training capabilities for all units.

Structured simulation-based training is characterized by
focusing on specific training objectives (Campbell, Campbell,
Sanders, & Flynn, 1995). All VIP training tables have thus been
designed so that units perform actions (critical subtasks)
associated with specific training objectives (tasks) and cues.
Examples of critical subtasks included: (a) reaching the starting
point on time; (b) executing fires when the enemy crosses the
trigger line; and (c) conducting displacement as directed.

“Turn-key” aspects of the VIP tables Maximizing SIMNET's
capabilities has also involved creating "turn-key" training
tables, which entails preparation of a unit’s training scenarios
and associated support materials before participation in VTP
training (Turecek et al., 1995). "Turn-key" tables thus involve
providing the units with their operation orders, overlays and
other planning materials before coming to the SIMNET facility.
The VTP participants should then be able to focus on executing
the training tables rather than on managing the training support,
and they thus spend more time at the SIMNET facility on mission
execution rather than on preparation and planning (Campbell et
al., 1995).

" -W-R Based on
Army training recommendations (Morrison & Holding, 1990), the
VTP tables flow from the two cornerstone NTC BN missions in a C-
W-R sequence of learning. A unit’s later VTP tables are thus
designed to be more difficult and demanding than their earlier
ones. Regarding the latter point, table developers also
instructed the original military members of the O/C team to
provide less coaching/mentoring as the units progressed through
the training tables.

Units should also face more challenging critical subtasks as
they progress from the fundamental (mostly crawl) training tables
to the more complicated offensive and defensive (mostly walk/run)
tables. The offensive/defensive tables also include some
repetition of fundamental critical subtasks, while the
fundamental tables are designed to include some walk-level
critical subtasks. The VTP participants should then have the
opportunity to repeat training on selected subtasks. (The
difficulty of the different critical subtasks was estimated by
the SIMUTA team's military subject matter experts--Campbell et
al., 1995).



In addition, the SIMNET facility contains 12 0/C
workstations. Since a typical BN contains 12 PLTs, there is one
0/C station per PLT. These stations include a plan view display
(PVD; two-dimensional electronic map), tactical radios, stealth
vehicle display (three-dimensional view of the virtual
battlefield), and audiovisual recording and replay systems
helping the O/Cs to perform their duties. The stealth vehicle,
for example, provides a direct view of the battlefield from an
invisible vehicle moving on or above the virtual terrain.

For more information about SIMNET, the reader is referred to
the following sources (Garvey & Radgowski, 1988, Turecek,
Campbell, Myers, & Garth, 1995). A more detailed description of
the Janus systems is provided by Elliott and Sanders (1996).

VTP" a Tr i T

Overview of the VIP tables As indicated, the SIMNET
portion of the VIP is geared toward PLT and CO training.

Approximately one hundred PLT and CO training tables (a table is
defined as a short structured training exercise) have been
created, including three fundamental, nine offensive, and six
defensive tables for each of the armor PLT, mechanized infantry
PLT, armor CO, and armor CO team (CO TM). For the scout PLT,
three fundamental, six offensive, and three defensive tables have
been created. In addition, four fundamental, seven offensive,
and four defensive tables have been created for the cavalry troop
(CAV TRP).

These tables are based on two cornerstone BN missions
(movement to contact and defense in sector) frequently executed
at the National Training Center (NTC). The SIMNET PLT and CO
tables have been designed to be nested within these two BN
missions, allowing for multiechelon unit training. BN training
exercises have also been created in SIMNET for these two
missions. (Please note that the SIMUTA team did not refer to the
BN exercises as tables.)

hi 1 Each table
consists of a preparation phase, an execution phase and an after
action review (AAR), and is designed to be conducted in about two
hours. The participating units should spend one half-hour on
preparing for the mission, one hour on executing the mission, and
another half-hour on participating in an AAR of the exercise.
The times for the preparation and AAR may vary depending on the
mission. The AAR process for a CO, for example, may last 45
minutes with 20 minutes being spent on an informal PLT AAR and 25
minutes on a more formal CO AAR. The former AARsS include all
members of the different PLTs within a CO; the latter AARs only
include the CO commander, executive officer, PLT leaders, and
tank commanders. (See Hoffman et al., 1995 for a more detailed
description of the different training tables, and a description
of the two BN missions.)



4a.

Data for Training Structure Questions

Do the VTP tables increase in difficulty?
Yes: 26 respondents No: 3 respondents

Exceptions to the crawl-walk-run table sequence:
Some fundamental tables are more difficult than
some PLT offensive/defensive tables ..............
NONE 22. ce eee eee eee eee een eeeee
PLT tables are harder than CO tables ..............
Other exceptions ........ cece eee eee eee eee eee

YHA IV

Need to change the order of the tables?
Yes: 10 respondents No: 19 respondents

Need to add any mission training plan (MTP) tasks
or subtasks?
Yes: 14 respondents No: 12 respondents
Don't Know: 3 respondents

Additional MTP task indicated by the "yes"

respondents:

Movement techniques (formations) .... 4
Tasks determined by task analysis ...............-- 4
Other tasks ....... eee eee eee eee eee eee 7

Need to add critical tasks or subtasks that are
presently undefined?
Yes: 15 respondents No: 12 respondents
Don't know: 2 respondents

Need to have more or less task overlap and repetition
among the VTP tables?
Okay as is: 26 respondents More overlap: 1 respondent
Don't know: 2 respondents

Need to improve the AAR agenda or process?
Okay as is: 18 respondents Yes: 11 respondents



2b. What aspects of unit performance increased the least during VTP rotations?

Unit Follow Up and Take-Home Package Interview

1a. Should parts of the THP be tailored for each type and level of audience? What is
needed for each group at each level besides the THP memorandum and matrices?
Platoon-Level Training

Company-Level Training

Battalion-Level Training

‘1b. How should Reserve component units use the THP materials to follow Up their
VTP training with activities at home station?

1b. How should Active component units use the THP materials to follow Up their VTP
training with activities at home station?

2. What needs to be modified, replaced, or added to the THP Materials book?

3. What could be done to increase the quality of the THPs that are produced? Would
additional materials done in other media be beneficial?

4. What could be done to make it easier to produce THPs?

5. Is a dedicated support staff of operations analysts, writers, graphic/media editors,
etc., needed to produce THPs of consistent quality in a reasonable time frame?



7. The use of AAR tapes would also increase the quality of the
produced THPs. This comment accounted for 47% (17/36) of the
responses regarding improving the THPs. Use of automated
processes (e.g., laptop computers) was another prominently
mentioned way of improving the THPs, representing 31% (11/36) of
the responses to the item on improving the quality of the THPs.
(This paragraph’s data are presented in G-6.)

8. Automated tools for O/Cs would make it easier for the O/Cs to
produce the THPs. This point accounted for 57% (21/37) of the
responses to the item on facilitating the THP process. Also
mentioned was the need to standardize the process of producing
the THPs. (See G-7 for the data cited in this paragraph.)

9. The participants were split on needing a dedicated staff of
operations analysts, graphic editors, etc. to produce the THPs.
Forty-eight percent (14/29) of the participants indicated a need
for such a staff; while 41% (12/29) felt the opposite. The
remaining 10% (3/29) were not sure about the need for having a

dedicated staff to produce the THPs. (This paragraph’s data can
be found in G-8.)
Questionnaire data. The following information about the THP

materials were obtained from these data (see Appendix K):

1. Members of the O/C team typically produced 21 THPs with a
typical officer and NCO completing 45.5 and 25.21 THPs,
respectively. Statistical differences (see Table K-1) were found
with regards to the numbers of PLT and CO THPs produced by the
different types of participants with the officers completing the
most PLT and CO THPs.

2. On the average, a THP took approximately 2.5 hours for the
0/c instructional personnel to complete. The O/C instructional
personnel took approximately the same amount of time to complete
the THPs for the PLT, CO, and BN tables.

3. A significant number of participants felt that the THP
feedback should be given to: (a) crews/squads for PLT tables but
not for BN tables; (b) vehicle commanders for PLT tables; (c) CO
commanders for CO and BN tables; and (d) BN staff members and
next higher HQ for BN tables but not for PLT tables.

4. These participants had positive feelings about the procedures
associated with producing the THPs. Over two-thirds of them
indicated that producing a THP was either easy or very easy and
that the THP materials book was either useful or very useful in
helping them to produce a THP.

5. High quality THPs were apparently produced. Approximately
85% of the participants felt that they produced either good or
very good THPs. (The data reported for points 4 and 5 were
statistically significant.)

23



3. Estimate the percent of units that were adequately prepared for their VTP

training:
0-25% 26-50% O 51-75% 0  76-100% O

With regard to emphasizing the tasks that the units performed in the tables,

the demonstration tapes were:
Very good L] Good 1 So-soOM Poor veryPoor

With regard to helping the units go through more training tables during their VTP
rotation, the demonstration tapes were:
Extremely useful [] Useful C1 Somewhat useful [1 Not useful C1

TRAINING MANAGEMENT: For each item, check the box that best represents your
opinion regarding each component of the VTP.

1.

Indicate your opinion of the following Training Support Package (TSP)

components:
Good Needstobe Needstobe Needs to

as is Modified Replaced _ be Deleted
Familiarization Course oO oO
Observer/Controller Handbook
Tools and Reference Materials
OCIC Handbook (Tables)
EC Handbook (Tables)
Take-Home Package Materials

>paoo®
oooooo
ooo0o00

oo00000
oo000

Indicate your opinion regarding the format of the following components of the
OCIC Handbook:
Good Needstobe Needstobe Needs to be
as is Modified Replaced Deleted

a. OPORD Narratives O oO oO
b. Execution Guidance Oo oO oO
c. Table Preview oO oO oO Oo
d. Event Guides oO oO O Oo
e AAR Guides Oo oO oO Oo
f. AAR Worksheets Oo oO oO oO

PT# 59-72 c-4



4. Unit improvement was seemingly not an artifact of units’
becoming more acclimated to either the SIMNET terrain or
equipment. Approximately 70% of the participants thought that
less than 50% of unit improvement was a function of these
factors. Management personnel were the least likely to think
that unit improvement was a function of adapting to SIMNET’s
equipment and terrain.

low-' P ri
Interview data, These data indicated the following about

unit follow-up and THP materials as viewed by VIP instructional
personnel:

1. The THP memorandum and matrices should be tailored for
different types of units. This opinion was voiced by 59% (17/29
of the participants (see G-1).

2. Additional materials besides the THP memorandum and matrices
were not needed for the crews and squads. This assertion was
made by nearly all of the respondents (see G-la, G-lb- & G-1c).

3. Different types of exercises needed different types of
additional THP materials. For example, a few respondents
expressed a need for the THPs associated with PLT exercises to
consist of: (a) checklists rather than matrix; (b) UPAS
performance data; and (c) tapes of unit performance. (Data
associated with these findings can be found in G-2).

4. The participants stated that the ARNG units ‘should use the
THP at their home-station to: (a) focus on tasks needing
training; (b) plan and prepare for future home-station
activities; (c) crosswalk with unit Mission Essential Task List
(METL); and (d) follow FM 25-100/101 (U.S. Department of the
Army, 1988; 1990). As presented in G-3, the first two categories
accounted for 64% (23/36) of the responses to the item on home-
station use of the THPs with the first category accounting for
39% (14/36) of the responses.

5. AC units should also use the THP materials as described
above. Ninety percent (26/29) of the respondents made this claim
(see G-4).

6. A clear answer was not provided for making changes to the THP
materials book. As presented in G-5, 31% (9/29) of the
respondents thought that this book should not be changed; while
31% (9/29) thought that it should include additional materials;
e.g., demonstration tapes. The remaining participants thought
that the book should be modified (14% or 4/29), replaced (10% or
3/29), deleted (3% or 1/29), or were not sure about what to do
with these books (10% or 3/29). Modifications to these books
primarily involved eliminating redundant materials, and using AAR
tapes was noted as the way of replacing the books (see G-5a).

22



Appendix E
Statistical Procedures For Questionnaire Items

The questionnaire items required three types of responses:
(a) unordered categories, (b) ordered categories (c) numerical
estimates. Summarized below are the statistical analyses done on
items of each type. All procedures used the Basic and Advanced
modules of SPSS for Windows 6.1.2 (Norusis, 1993).

a ri

This section deals with the following items: (a) #1 from
the Train the Trainer set; (b) #2-#5 from the Unit Preparation
set; (c) #1-#7 from the Training Structure set; (d) #3-#5 from
the Training Proficiency set; and (e) #2-#4 from the Unit Follow-
Up and Take-Home Package set.

Differences in distributions of responses among the Warthog
Observer/Controller groups (Officers, Non-commissioned Officers--
NCOs, Civilians, and Management) were tested by the Pearson chi-
square test for k = 4 independent samples with a = .05. Follow-
up pairwise comparisons used Fisher’s exact test with aw = .02.

Commonly, when group differences were not statistically
significant, single-sample tests of proportions were done.
Binomial tests of the proportion of responses in a selected
category (or sometimes two categories together) for all groups of
0/Cs combined had the null hypothesis p = .50 anda = .02. This
tested whether a statistically significant majority of O/Cs
agreed on the same response, or if a majority disagreed with that
responses.

Some sets of items asked the same question about members of
a class of entities. When group differences were not found
statistically significant for all items in the set, differences
of O/C opinion among the entities were of interest. Then, the
combined proportion of responses in a selected category were
compared among items using the Cochran Q test for k related
samples with a@ = .05. Pairwise comparisons by the McNemar test
used a@ = .02.



PERSPECTIVES ON THE VIRTUAL TRAINING PROGRAM FROM MEMBERS OF ITS
INITIAL OBSERVER/CONTROLLER TEAM

CONTENTS
Page
INTRODUCTION 1
THE VTP . 1
Training Systems Associated With the vrp 1
The VTP’s SIMNET Exercises (Tables) 2
The VTP’s O/C Team 4
The VTP’s Training Management “Tools” . 6
Review of the VTP’s Components 7
The VTP’s Training Value 7
OBJECTIVES OF THIS EVALUATION . 9
METHOD 9
Participants 9
Instruments 9
Procedure 12
Scoring and Data Analyses. 12
RESULTS AND DISCUSSION 13
Demographic Information 13
Train the Trainer 14
Unit Preparation 15
Training Management 17
Training Structure 18
Training Proficiency . 19
Unit Follow-up and THP Materials 22
General Comments woe eee 24
CONCLUSIONS 25
REFERENCES 27
APPENDIX A. THE ASSESSED DEMOGRAPHIC INFORMATION A-1
B. STRUCTURED INTERVIEW FORM . B-1
C. THE O/C QUESTIONNAIRE C-1
D. DATA FOR THE STRUCTURED INTERVIEWS D-1
E. STATISTICAL PROCEDURES FOR QUESTIONNATRE
ITEMS . foe E-1

ix



3. Discrepancies between judges were resolved by discussing
their differences and listening to the tapes.

4. All three judges, independently, conducted a content analysis
of each item with the stipulation that a response category
consisted of three or more similar responses. Those responses
which could not be included into a distinct response category
were categorized as “other” responses.

5. The three judges met to reach a consensus on the: (a)
response categories and (b) number of responses per category.

(See Appendix E for a discussion of the data analyses associated
with the questionnaire items.)

Results and Discussion

This part of the report contains sections on: (a)
demographic information; (b) train the trainer; (c) unit
preparation; (d) training management; (e) training structure; (£)
training proficiency; (g) unit follow-up and THPs; and (h)
general comments. These topics, excluding the sections on
demographic information and general comments, are further
segmented by the participants’ interview responses and
questionnaire data. Those two sections consist only of the
participants’ interview responses.

D ic I ion

These data, (see item set A of Appendix D‘), indicated the
importance of 0/C team members’ having previous military field
experience. Such experience was the most frequently mentioned
criterion for selecting a VTP O/C and an EC, accounting for 43%
(18/42) of the responses regarding the criteria for O/C and/or EC
selection. It was also cited as the most helpful experience to
these participants in their role as an O/C or EC. For example, a
participant noted that being the Chief of Simulation at Fort Knox
was great preparation for his duties as a member of the O/C team.

Several other salient findings surfaced from the
participants’ responses to these items. These participants
Claimed, for one thing, that O/Cs should have: (a)? been an
instructor; (b) received training at the NTC, and (c) previously
operated computers. In addition, proper schooling was an

1 Results for the structured interviews are presented in
Appendix D. Text notation in the text refers to the numbering
scheme found in this Appendix.

2 Sentence seriation--throughout the results and discussion
section--is based upon the percentage of the item’s responses for
that category. More comments were thus found for “being an
instructor” than for the two other cited response categories.

13



(Items 4a-4d deal with the questionnaire items
on the OCIC Handbook’s format)

4a. Reasons given for indicating a need to modify:

Operation order (OPORD) narrative ............+.---
standardize to current military doctrine (7)
correct errors (5)

Execution guidance

Table preview

Event Guides

AAR Guides 2.0... cece cece eee eee eens

AAR Worksheets

Reasons given for indicating a need to replace the:
OPORD narratives ........ cece eee eee eee eens
Execution guidance
Table preview .....
Event GuideS ......... cece eee cee eee eee eter eens
BAR Guides 2.0... cece eee een eens
AAR Worksheets

Reasons given for indicating a need to delete the:
OPORD narrativeS ....... eee eee eee eee ee
Execution guidance .......... eee eee ee eee eee eee
Table preview 2... 0... eee eee eee eens
Event GuideS ......... cece cece eee eee eens
BAR GuideS 2.2... cece cee ee eee eee teenies
AAR Worksheets

not used/duplication of event guides (7)

Need to add to the OCIC Handbook:
Yes: 10 respondents No: 19 respondents

FOOOOR wowurasA

OrROOrHSO



Table K-3

Number of Hour: er it Le’ tel ersonnel
Platoon Company Battalion
Level Level Level

o/c
Group N Mean sD N Mean spd N Mean sD
Officers 6 2.67 0.82 5 2.60 0.89 5 2.53 1.61
NCOs 8 2.88 1.13 1 3.00 0.00 5 3.33 1.18
Civilians 8 2.63 1.06 5 2.40 1.67 2 1.82 0.25
Management 4 1.25 0.50 2 2.00 0.00 4 3.25 2.06

Total 26 2.50 1.07 13 2.46 1.13 16 2.87 1.49
Table K-4

‘ inions ed for 101 Com) ion
Tables
Platoon (PLT) Company (CO) Battalion (BN)
Tables Tables Tables

Type of Units Need No Need Need No Need Need No Need
Crews /squads 203+ 8 9 19 4 25*
Vehicle Commanders 23* 6 14 15 5 24*
PLT Leaders/Sergeants 27* 2 27* 2 13 16
co leaders 13 16 27* 2 26* 3
BN Staff 6 23* 15 14 28* 1
Higher Headquarters 5 24* 15 14 8 21
‘Number of participants. *response frequency is significantly (p < .02) greater

than 50%.



2b.

2c.

4a.

5a.

Second most in need of modification:

Hardware/Software ........ cee eee eee eee eens 10
Training development process
Standardization of OPORDs seen : a :
Other ASPECtS ©... Lee ee eee eee eee eee eee 4

Third most in need of modification:
Hardware/Software ......... eee eee eee eee eee
Training development process ..........-.--+-+--eee
Standardization ......... cee e eee eee ee
Other aSPectS .... eee eee eee eee eee ee eee

Wonwo

Major lessons learned from being an O/C or EC:
Structured training used ......... cece eee eee eee eee
Knowledge of operations, tactics, doctrine ........
Knowledge of RC/AC structure and training .........
Hardware/Software ........ cece eee eect e eee
Other leSSONS ........ eee eee ee eee eee ee eee tee eee

bP BOW

Does structured training like the VTP best apply to
RC or AC training?
RC: 6 respondents AC: 0 respondents
Both: 23 respondents

Reasons given:
Both, RC/AC training constrained by OPTEMPO ....... 5
RC, because it's simpler/basic, only PLT and CO

tactics training most units will get, and

biggest bang for the bucks (time/resources) ...... 7
Other TeaSONS ..... eee eee eee eee eee eee 8

Should structured training be developed for mission
planning and preparation:
Yes: 21 respondents No: 7 respondents
Not Sure: 1 respondent

"Yes" respondents: Method of structured training:

At Army school-houses .......... 2. cece eee eee eee 6
After execution skills are acquired oe . a 4
Other ideas ......... eee eee eens 14



Table A-2

f-R BN i n a
OFFICERS NCOs CIVILIAN MANAGEMENT
(N=6) (M=8) PERSONNEL PERSONNEL
(N=9) (N=6)

O/C Rotations

PLT/CO

M 106.33 49.00 2.00 6.00

sD 59.20 25.69 2.00 6.78
(n=6)? (n=8) (n=1) (n=4)

BN

M 16.50 13.75 0 20.00

sD 6.89 4.79 0 11.55
(n=6) (n=4) (n=0) (n=4)

Janus

M 12.67 12.17 5.00 15.00

sD 7.20 6.15 0) 12.20
(n=6) (n=4) (n=2) (n=6)

EC Rotations

PLT/CO

M 5.00 25.86 48.78 5.80

SD 3.46 18.71 28.49 6.83
(n=4) (n=7) (n=9) (n=5)

BN

M (0) ) 9.11 20.00

sD 0 0 8.31 17.67
(n=0) (n=0) (n=9) (n=3)

Janus

M 7.25 9.29 14.00 9.00

SD 6.55 8.38 9.83 14.00
(n=4) (n=7) (n=8) (n=4)

"Number of respondents by group for that item.



Appendix J

Results for Questionnaire Items on Training Proficiency

Table J-1

Estimates of the Proficiency Levels of Unit Leaders and Units at the Start and End
of Their VIP training

Start End

Area of Proficiency n M sD M sD
Unit Leaders

Troop-leading procedures* 27 © 34.81 14.24 68.70 13.34
Monitoring the battlefield situation* 270 26.11 14.10 68.81 13.50
Controlling unit fire and maneuver* 27 24.44 15.71 67.96 19.67
Explaining reasons for actions in AARs* 27° 38.70 21.15 77.96 16.36
Overall performance* ~ 27 36.30 12.06 74.37 13.21
Units

Executing formations and changes* 28 36.18 18.72 81.36 10.51
Making reports* 28 «31.43 16.43 80.89 12.10
Executing battle drills* 270 31.11 18.47 78.33 12.56
Occupying and defending battle positions* 26 30.77 17.19 69.73 18.81
Overall performance* 27 34.93 16.16 76.15 13.74

*Statistical tests revealed a significant difference (p < .05) between the
starting and ending estimates of unit proficiency.

Table J-2
if es for i ed Starti i Lev: icienc’

Difference Scores
Area of Proficiency a M sD

Unit Leaders

Troop-leading procedures 27 33.89 16.60
Monitoring the battlefield situation 27 42.70 16.03
Controlling unit fire and maneuver 27 43.52 21.61
Explaining reasons for actions in AARs 27 39.25 23.17
Overall performance 27 38.07 16.65

(Table J-2 Continued)



6a.

Appendix B
Structured Interview Form
Demographic Information Interview

What previous experience was most helpful to you in your role as an O/C (EC)?

6b.
VTP O/Cs?

In addition to O/C experience, what other qualifications are important for selecting

6c.

What qualifications are important for selecting ECs?

da.

Train the Trainer Interview

Briefly describe the training you received for your role as an O/C or EC?

1b.

What parts of your training proved to be most helpful?

1c.

What parts of your training proved to be least helpful?

1d.

Did you need any training that was not provided?

te.

Do you have any suggestions for how future O/Cs and ECs should be trained?

1f. Dedicated O/Cs and ECs may not be available in future programs like the VTP.
Can unit personnel be trained to perform these duties? If yes, what training will be
needed?



6c. "No" respondents: Unit personnel deficient in:
Technical skill requirements ............2--ee ee eee
Ability 22... cece ee eee eee eee eee e tenes
Objectivity
Efficiency

(Items 7a-7b deal with the participants’ responses to
the questionnaire items on time for O/C training)

Ja. Reasons given for indicating that training for the
0/C team was "too much:"
Conducting advance visits
Operating the Simulation Networking (SIMNET)
workstations 2.1... eee e eee eee eter eeee
Operating the Janus workstations ................--
Conducting SIMNET tables ........... eee ee ee eee eee
Conducting Janus exercises .......... eee eee eee eee
Conducting AARS ....... cee eee ee eee teers

7b. Reasons given for indicating that the training
for the O/C team was "too little:"

Conducting advance visitS ............ sees eee eee
training did not focus on unit visits (6)?
Operating the SIMNET workstations ................-

more hands-on training required (5)

Operating the Janus workstations ................--
need more hands-on as skills deteriorate (5)
learned while conducting the exercises (3)

Conducting SIMNET tables ............ 2s eee eee eee
learned while conducting the exercises (3)

Conducting Janus exercises
re-familiarization training (4)

did not receive any formal training (3)

Conducting AARS ....... eee eee eee eee eee eens
did not receive any formal training (3)

2 Listing of response category(ies) with number in
parentheses denoting response frequency for the category.

D-3

NPP OO

BHRWoOO



4a) should be added. Also, 52% (15/29) of the participants
wanted the VTP tables to contain other critical tasks or
subtasks; while 41% (12/29) did not feel that any other critical
tasks or subtasks should be added to the VIP tables (see E-5).

4. The AAR agenda seemed to be appropriate. This claim was made
by 62% (18/29) of the participants (see E-7).

5. However, as discussed for the Training Management data,
participants wanted more leeway in conducting the AARs. This
point was noted by participants who wanted to make changes to the
AAR agenda (See E-7a).

6. The participants indicated that the stealth display with
voice, and the events’ poster were the two most useful AAR aids.
They also thought that their AARs could be improved with the
addition of UPAS, bigger/wider PVD screen, visual aids, and

ModSAF 1.5. (The data associated with this paragraph’s findings
are presented in E-8 and E-9).
Questionnaire data, These data, as presented in Appendix I,

showed that nearly all of these participants either agreed or
strongly agreed with the questionnaire statements dealing with
the VTP’s structure. The following statements can then be made
about these findings:

1. The VTP’s instructional personnel did feel that the training
tables were progressively more difficult.

2. The stealth preview and AARs were valuable instructional
tools.

3. The AARs, correspondingly, did promote discovery learning by
the VTP participants.

4. The VTP participants’ performance was aided by such VIP
design features as subtask repetition and the progression in
difficulty of the VIP tables (The ECs tended to have stronger
feelings about the value of subtask repetition than did the other
respondents) .

ing P. icien

Interview data. These participants indicated the following
opinions about the effectiveness of the VTP for training unit
leaders and elements:

1. The VTP was most effective in helping unit leaders to become
better able to command and control their elements. This category
accounted for 56% (19/34) of the responses to the item dealing
with improvements in leader performance (see F-1).

2. As also shown in F-1, this program was also noted for helping
foster positive changes within the unit leaders. Forty-one

19



Number of

(Items 3a-3d deal with the questionnaire items
on the Training Support Packages (TSPs) :

3a. Reasons given for indicating a need to modify the:

Fam COULSE 2... eee eee eee teen eee eee
changes needed to SIMUTA materials (3)

O/C Handbook ......... cee cece eee eee eens
reorganize it (9); update materials (3)

Tools and reference materials .............ee ee eeee
reorganize them (3)

OCIC Handbook (tables) .......... eee eee ee eee eee eee
reorganize it (8); combine with EC Handbook (7)

EC Handbook (tables) ........... cece eee eee eee eee eee
reorganize it (7)
combine it with the OCIC Handbook (4)

THP Materials 1.2... . cece eee cece eees
standardize the materials (5)

3b. Reasons given for a need to replace the:
Fam COUrSE 2... eee eee eee ee ee eee ee ee eee eens
O/C Handbook ....... cee eee eee eee eee
Tools and reference materials .............-2 eee eee
OCIC Handbook ........ cece eee eee eee ee eee
EC Handbook (tables)
THP materials ............

3c. Reasons given for indicating a need to delete the:
Fam COUrS@ . 2... eee eee eee eee eee ee eens
O/C Handbook
Tools and reference materials .............---+0--5-
OCIC Handbook (tables) ........ eee ee eee eee ee ee eee
EC Handbook (tables) ......... eee cece ee eee eee eee
THP materialS ......... eee eee eee eee eee eee eee eee

3d. Additional TSP components needed:
NONE 62. eee eee eee eee ee eens
User manuals for the SIMNET stations and Janus.....
Short orientation course
Training aidS ........ cece eee eens
Other COMPONENtS ...... eee eee eee eee eee eens

12

20

23

25

23

20

ot oo momo) socooaoo

Ww Wo


5b.

6b.

If "no" response, then why not?
Not efficient use of resources ............-eeeeeee 6

Should structured training like the VTP be developed
for future simulations like the Close-Combat
Tactical Trainer (CCTT)?
Yes: 21 respondents No: 7 respondents
Not Sure: 1 respondent

Ideas from "yes" respondents about changing the VTP:

Tailor VIP to CCTT capabilities ................... 12
Set up like SIMUTA .......... ccc e eee eee 4
Other ideaS ..... ec eee eee eee 4

Final comments about the VTP:

Positive 20
Negative : (e)
Needs Modification .......... eee eee eens 9



Research Report 1691

Perspectives on the Virtual Training Program From Members
of Its Initial Observer/Controller Team

Theodore M. Shlechter, Ronald E. Kraemer,
David W. Bessemer, and Billy L. Burnside

U.S. Army Research Institute

James Anthony
University of Louisville

Armored Forces Research Unit
Barbara A. Black, Chief

U.S. Army Research Institute for the Behavioral and Social Sciences
5001 Eisenhower Avenue, Alexandria, Virginia 22333-5600

Office, Deputy Chief of Staff for Personnel

Department of the Army
May 1996
ES
Army Project Number Training Systems and Education

20363007A793

Approved for public release; distribution is unlimited.

iii



DEMOGRAPHIC INFORMATION: Please provide the following information about your

military service.

1. SC/MOS:

2. Rank/Grade:

3. Time in Military Service: Years Months

4. Time in SC/MOS: Years

Months

5. When did you report to the Warthog team? Year Month

®

If yes, then where?

Did you have previous O/C experience? Yes No

For how many months?.

7. Estimate the number of rotations that you have served in each of the following

roles during SIMNET training:
o/c
Armor Plt
Mech Plt
Scout Pit
Company
Battalion

eoaong

LT |

E/C

HTT

8. Estimate the number of rotations that you have served in the following roles during

Janus training:
o/c
Battalion

Interactor

9. List any other roles in which you have served on the Warthog team:

PTF 59-72



General Interview

1. Briefly describe the three most positive or beneficial aspects of the VTP in order of
importance (most important first).

2. Briefly describe the three aspects of the VTP that are most in need of modification,
in order of importance (most important first).

3. Briefly describe three major lessons you learned while serving as a VTP O/C or EC.

4. Does structured training like the VTP best apply to Reserve or Active unit training?
Why?

5. The VTP focuses on execution. Should structured training be developed for mission
planning and preparation? How do you think such training should be done?

6. Should structured training like the VTP be developed for future simulations like the
CCTT? Do you have ideas about specific program changes needed for the CCTT?

7. Do you have any other comments you would like to provide about the VTP?
Besides your comments here, feel free to submit more detailed comments in writing.



factor, with the five items (aspects of performance) and gains
(performance at the start vs. the end of training) as within-
subjects repeated-measure factors. The Pillai statistic was used
for multivariate significance tests. When no group differences
or interactions were found statistically significant, follow-up
paired comparisons were done on means for the groups combined
using dependent-sample t-tests. The ten comparisons among the
item means at both the start and end levels used a = .01. Tests
of gain for each item used a = .02.



ECs received limited formal training on operating the SIMNET
workstations and on supporting the Janus exercises.

The VTP’s Training Management Tools

Overview of the training management tools, Exercise or
training management tools are the VIP’s final component. The
purpose of these tools is to help the o/c team follow the
instructional guidelines set forth under SIMUTA. The use of
these materials would then serve to standardize the VTP process
(Hoffman et al., 1995).

Two other points must be noted about the VTP’s management
tools. One, these tools are contained in a "Materials Library."
Two, this library is comprised of a set of handbooks for each
echelon with each set consisting of the volumes--except for the
Familiarization (Fam) Course volume--that are summarized below.

These handbooks have been
designed to prescribe the basic scheme for: (a) planning the
unit’s rotation; (b) conducting the advance visits; (c)
overseeing the units’ table execution; (d) preparing the THPs;
and (e) coordinating the training with the SIMNET staff.

n ial These
materials include forms to record information about the different
VTP missions completed by each element within an echelon and the
elements’ battle rosters. The PLT volume also contains separate
sets of materials for the different types of PLTs.

Handbooks on the advance materials, These handbooks consist

of: (a) instructions for unit preparation, (b) narratives of the
different operation orders (OPORDs), and (c) descriptions of the
@ifferent critical subtasks associated with each table. The PLT
volumes also contain separate sets of OPORDS and table
descriptions for the armor PLT tables, mechanized infantry PLT
tables, and scout PLT tables.

Handbooks on the THP material, These handbooks contain
information on the procedures for completing the THPs for each

unit. These procedures include having the 0/Cs indicate in the
THPs those subtasks which units need either to "train to sustain"
or "train to improve," representing satisfactory or
unsatisfactory performance, respectively.

Fam Course manual, A Fam Course manual only exists for the
PLT set of volumes as this course is designed to help crews
become familiar with the SIMNET simulator modules and terrain
database. This manual delineates the O/Cs’ roles in this course
as helping crews to: (a) locate the SIMNET switches and knobs;
(b) navigate the SIMNET terrain database; (c) identify friendly
and enemy vehicles in the SIMNET database; and (d) engage enemy
vehicles with direct and indirect fire.



Number of

Comments
(Items 6a-6c deal the participants’ responses to
questionnaire items on unit preparation materials.)
6a. Reasons given for indicating a need to modify the:
Orientation Guide ....... 6. cece eee cee eee ee eee 10
more materials need to be added (3)
Orders and OverlayS .......... cece eee eee eee eee 32
need to be standardized (13)
need to correct errors (12)
Demonstration TapeS ........ eee eee eee eee eee teens 12

add additional tapes (3)

6b. Reasons given for indicating a need to replace the:
Orientation Guide ......... cee eee eee eee eee eee eee
Orders and OverlayS ......... eee cece eee eee eens
Demonstration TapeS ........-. cee eee eee eee eee eee

6c. Reasons given for indicating a need to eliminate the:
Orientation Guide ......... cee eee ce eee eee
Orders and Overlays .............- peek eee eee e sees
Demonstration Tapes ..............-2006- bbe ee eee ee

D. Data for the Training Management Questions

1. Need to change the time allocated for the different
VTP activities:
Yes: 21 respondents No: 4 respondents
Maybe: 4 respondents

la. Changes recommended by the "yes" respondents:
Time for AARs should be flexible ..................
Time for different activities should depend on
unit’s level of skill/experience ..............
More time needed for troop leading procedures .....
Other reaSONS ....... eee eee eee

2. Comments about re-familiarizing units with SIMNET:
Familiarization (Fam) course is always important
Depends on percentage of old/new personnel
Depends on time between rotations
Other actions ......... cece eee eee

rs

°


la.

1b.

Ic.

Number of

Comments
Aspects of unit performance that increased the least:
Situational awareneSS ........ 2 cece cece ee eee eee 7
Gunnery ........--------- oe 7
Occupy/Defend battle positio 4
Communication/Crosstalk ........ cece cece eee eee 4
Other aspectS ©... eee eee eee eens 11

Data for the Unit Follow-up and Take-Home Package Questions
Should parts of the THP be tailored to each type and
level of audience?

Yes: 17 respondents No: 12 respondents

Should parts of the THP be tailored just to PLT tables?

Crews or squadS ....... cece eee eee eee eee ee eee eee 3
All others 2... cee eee eee eee eee eee eee 26
Should parts of the THP be tailored just to CO tables?

Crews or squadS ......... cece cee eee eee eee eee 1
Bll others 2... eee eee eee eee ee eens 26
Should parts of the THP be tailored just to BN tables?

Crews or squadS ...... cee eee ec ee ee ee eee eee 1
All others 2.1... . ccc eee eee eee eee eee eens 28

Additional THPs needed for PLT (all groups):
Checklist rather than matrix ..............-- ee eens
UPAS data ... eee eeeeeeeeee eens
TAPES 2c cette eee ete t ene eens
Feedback from unitS .......... cee eee eee eee eee eee
Other additions .......... cee cece eee eee tee eee

Awww ps

Ways for the reserve components (RCs) to use the THPs:
Focus on tasks needing training .................6% 14

Plan/Prepare for future home station training .. 9
Crosswalk with unit mission essential task list 5
Follow FM 25-100/101 ......... eee eee eee cece eee 4
Other WayS ......... cee eee te eens 4
Ways for the active components (ACs) to use the THPs:
Same as RC (ACs have more time and resources) ..... 26
Perform other activities ............ eee cece eee eee 3



claimed that they were more proficient after training than they
were before training.

Potential problems with the VIP’s training value The VTP

is seemingly then an efficient and effective instructional
program. This conclusion, however, is tentative because direct
comparisons of the VTP’s efficiency and effectiveness with other
training programs have not been conducted. Shlechter et al.
(1995), for example, were not able to collect data on a control
group of units who had gone through SIMNET training but not
through the VTP. The improvement noted by Shlechter et al.
(1995) could then have been partially a function of the
participants’ adapting to the SIMNET environment.

Also, Hoffman et al. (1995) have observed several potential
limitations with the implemented VIP. First, the developed BN
exercises were not fully linked with the PLT and CO tables, which
correspondingly led to minor problems with conducting
multiechelon training. Second, immersion vis-a-vis a constant
mission may have been carried too far, at least for the units who
completed the tables during their AT. These units claimed to be
bored with the repetition of missions across echelons. (A PLT--
CO-BN progression of missions was not implemented for the ARNG’s
week-end training drills, due to time limitations.) Third, the
0/Cs--especially the newer ones--were not thoroughly trained with
regards to incorporating “coaching” in relationship to the
previously discussed C-W-R framework. Finally, the original 0/C
team should have been more heavily involved in the development
process, thereby creating a product more suitable to the Army
culture (VTP development was well underway when O/Cs began
arriving) .

Shlechter and Anthony (1996) have suggested another
potential shortcoming with the VTP. They found that some ARNG
units did not systematically view the VTP demonstration tapes,
because of time constraints associated with their home-station
training. In this and other regards, time constraints may limit
units’ opportunity to prepare fully for their VTP rotation(s).

ini h E As
indicated, information about the VTP’s training value was
obtained from independent evaluators, its developmental team,
and its training participants. Questions remain about the VIP’s
training value as viewed by its original set of instructional
personnel. Examining these individuals' views would provide
further insights into the following issues:

1. Do units and unit leaders become more proficient as the
result of participation in the VTP?

2. Do problems--as suggested by Hoffman et al (1995)--exist with
the VTP’s structure and with the O/Cs’ training?

3. Do problems exist--as suggested by Shlechter and Anthony



REPORT DOCUMENTATION PAGE

1. REPORT DATE 2. REPORT TYPE
1996, May Final

3. DATES COVERED (from. . . to)
April 1995-March 1996

4, TITLE AND SUBTITLE

Perspectives on the Virtual Training Program From Members of Its

Sa. CONTRACT OR GRANT NUMBER

5b. PROGRAM ELEMENT NUMBER

Initial Observer/Controller Team 0603007A.

6. AUTHOR(S) 5c. PROJECT NUMBER
A793

Theodore M. Shlechter, Ronald E. Kraemer, David W. Bessemer, 5d. TASK NUMBER

Billy L. Burnside (ARI), and James Anthony (University of 2124

Lousiville) Se. WORK UNIT NUMBER
HOL

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

U.S. Army Research Institute for the Behavioral and Social Sciences
Armored Forces Research Unit

ATTN: PERI-IK.

Fort Knox, KY 40121-5620

8. PERFORMING ORGANIZATION REPORT NUMBER

9, SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)
U.S. Army Research Institute for the Behavioral and Social Sciences
5001 Eisenhower Avenue

Alexandria, VA 22333-5600

10. MONITOR ACRONYM
ARI

11. MONITOR REPORT NUMBER
Research Report 1691

12. DISTRIBUTION/AVAILABILITY STATEMENT
Approved for public release; distribution is unlimited.

13, SUPPLEMENTARY NOTES

Institute’s Armored Forces Research Unit.

James Anthony is a graduate student at the University of Louisville working as a consortium fellow at the U.S. Army Research

14, ABSTRACT (Maximum 200 words):

This report examined the Virtual Training Program’s (VTP) strengths and weaknesses from the perspective of its original
instructional team. These instructional personnel completed questionnaires and were interviewed regarding their opinions on various
aspects of the VIP. Findings from the questionnaires and interviews provided further support for the VTP’s instructional value. The
participants indicated that unit leaders and units became more proficient during the course of their VTP rotation, and that this
improvement was not simply a function of adapting to the Simulation Networking (SIMNET) terrain and equipment.
Correspondingly, the participants had few problems with most aspects of the VTP.

15. SUBJECT TERMS

Armor training Simulation-based training Simulation Networking Training methods

‘SECURITY CLASSIFICATION OF : 49. LIMITATION OF

ABSTRACT

16. REPORT 17. ABSTRACT | 18. THIS PAGE
Unclassified | Unclassified | Unclassified | Unlimited

20. NUMBER | 21. RESPONSIBLE PERSON
OF PAGES (Name and Telephone Number)

84



Appendix A

The Assessed Demographic Information

Table A-1
- Sal i i nd Prior xperien
OFFICERS NCOs CIVILIAN MANAGEMENT
(N=6) (N=8) PERSONNEL PERSONNEL
(N=9) (N=6)*

Years in

Service

Me 11.91 15.40 18.68 17.50

spe 2.47 2.29 7.44 4.28
(n=6)¢ (n=8) (n=8) (n=6)

Years in

Sc/MOS

M 11.08 15.00 20.42 16.08

sD 1.80 2.29 4.09 5.83
(n=6) (n=8) (n=6) (n=6)

Years of Prior

O/C Experience

M 1.50 3.50 0.75 1.50

sD 0.87 0 0.35 0.99
(n=3) (n=1) (n=2) (n=3)

*The management personnel consisted of 4 officers, 1 NCO, and 1
civilian, who were not included in the other three groups. »M =
mean. °SD = standard deviation. “Number of respondents by group
for that item.



Table J-2 Continued

Difference Scores

Area of Proficiency a M sD
Units

Executing formations and changes 28 45.18 20.39

Making reports 28 31.43 17.97

Executing battle drills 27 47.22 19.76

Occupying and defending battle positions 26 38.96 18.47

Overall performance 27 41.22 16.90

Table J-3

ignifican isons <_.05 i e. Scores
Comparisons Cases at Mean SES £
Diff

Unit Leaders

Troop Leading Procedures vs.
Controlling Unit Fires 27 26 9.63 3.62 2.66

Troop Leading Procedures vs. 27 26 8.81 2.50 3.52
Explaining Reason during AARS

Controlling Unit Fires vs. 27 26 5.44 2.34 2.33
Overall performance

Units
Occupying and Defending vs. 26 25 7.96 2.56 3.11
Maintaining Formations
Occupying and Defending vs. 26 25 12.00 2.22 5.40
Making Reports
Occupying and Defending vs. 26 25 9.30 2.41 3.86
Executing Battle Drills
Occupying and Defending vs. 25 24 3.60 1.71 2.10
Overall Performance
Making Reports vs. 27 26 8.96 2.00 4.48

Overall Performance
(Table J-3 Continued)



4. The O/Cs’ advance visits were very important. This item had
a mean score of 3.64 with 4 as "very important."

5. The demonstration tapes effectively prepared the units for
their VTP rotation. These materials received a mean score of
4.00 with 4 as "good" on emphasizing VTP tasks, and they received
a mean score of 2.86 with 3 as "useful" on helping units go
through more training tables during their VTP rotation.

6. Units were not prepared for their VTP rotations. That is,
the majority of participants felt that fewer than 50% of the VTP
units were properly prepared.

Training Management

Since nearly all of these interview and questionnaire items
are complementary, they are discussed jointly. The following
findings emerged from these data regarding the TSP materials:

1. The participants had mixed feelings about these materials.
While the participants clearly indicated on their questionnaires
that the Fam course was good as designed; their feelings about
the other TSP materials were not as clear (see Appendix H).
Also, as shown in D-3d, fewer than 40% (11/29) of them believed
that the TSP did not need any additional components. The
participants, correspondingly, noted the need for user manuals
for the SIMNET workstations and Janus, a short orientation
course, and training aids (see D-3d).

2. The organization of the TSP materials seemed to be
problematic. Reorganizing the materials was the prevalent reason
given for modifying the Tools and Reference Materials, OCIC
Handbook, and the EC Handbook (see D-3a).

3. The participants were generally satisfied with the different
components of the OCIC handbook. A significant proportion of
participants indicated on the questionnaires that the Execution
Guidance, Table Preview, Event Guides, and AAR Guides were good.
In addition, 66% (19/29) of the participants stated that the OCIC
handbook did not need any additional components (see D-4d).

4. As discussed for unit preparation, the OPORD narratives could
be modified by making them more doctrinally correct and by
eliminating other errors in them (see D-4a).

The following findings have also surfaced regarding training
Management issues:

1. The questionnaire data showed that the time available for
most SIMNET activities was about right. Over 60% of the
participants made this assertion for the Fam Course, Table
Previews, Table Execution, and for preparing and conducting the
AARs. However, the proportion of "about right" responses was not

17



U.S. ARMY RESEARCH INSTITUTE
FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency Under the Jurisdiction
of the Deputy Chief of Staff for Personnel

EDGAR M. JOHNSON
Director

Technical review by

Martin A. Leppert
Bruce S. Sterling

NOTICES

DISTRIBUTION: Primary distribution of this report has been made by ARI. Please address correspondence
concerning distribution of reports to: U.S. Army Research Institute for the Behavioral and Social Sciences,
ATTN: PERI-STP, 5001 Eisenhower Ave., Alexandria, Virginia 22333-5600.

FINAL DISPOSITION: This report may be destroyed when it is no longer needed. Please do not return it to
the U.S. Army Research Institute for the Behavioral and Social Sciences.

NOTE: The findings in this report are not to be construed as an official Department of the Army position,
unless so designated by other authorized documents.



3.

Indicate your opinion about the allotted time for the following SIMNET activities:

Too About Too
Little Right Much
a. Familiarization Course oO oO
b. Table Previews oO oO Oo
c. Troop Leading Procedures oO Oo O
d. Table Execution Oo Oo Oo
e. AAR Preparation oO oO Oo
f. AARs for Platoon Tables | oO oO
g. Platoon AARs for Company oO oO |
Tables
h. Company AARs for Company Oo O Oo
Tables
i. Battalion Staff Section AARs oO Oo oO
j. Full Battalion AARs oO Oo oO

TRAINING STRUCTURE: For each item, check the box which best represents your
opinion regarding the following statements about the structure of the VTP.

1.

The stealth preview aided the units’ ability to conduct the tables.
Strongly agree CL] Agree C1 Don't agree C1 Strongly Disagree

The VTP training tables progressed from less to more difficult.
Strongly agree Agree L] Don't agree O1 Strongly Disagree 1

The progression in difficulty enhanced the VTP's effectiveness.
Strongly agree L] Agree 1 Don't agree 0 Strongly Disagree 1

The critical subtasks identified key aspects of unit performance.
Strongly agree OD Agree O1 Don't agree O Strongly Disagree oO

Subtask repetition helped the units to improve their performance across tables.
Strongly agree CL] Agree C1 Don'tagree 0 Strongly Disagree 11

The AAR structure promoted discovery learning by the units.
Strongly agree CL] Agree O1 Don't agree LI Strongly Disagree oO

The use of the AAR tools helped the units to understand their performance.
Strongly agree CX] Agree LI Don'tagree O Strongly Disagree 11

PTY 59-72 c-5



Conclusions

This investigation has provided further support for the
VTP’s instructional value. As discussed, these VIP instructional
personnel believed that unit leaders and units became more
proficient during the course of their VIP rotation. Both units
and unit leaders seemingly improved from a novice level, i.e.,
approximately 35% mastery level) to an intermediate level of
tactical proficiency; i.e., approximately 75% mastery level
This improvement was apparently not a function of VTP
participants’ becoming more acclimated to either the SIMNET
terrain or equipment. The VTP is then an effective program for
training ARNG and AC units to become more tactically proficient.
In addition, this program has been noted by these VTP
instructional personnel as being cost-effective. One O/C
declared: "It (the VIP) pays off. Wish I had it when I was a CO
commander."

The participants, correspondingly, had few problems with
most aspects of the VIP. They felt that the tables did progress
in difficulty, and that this progression helped to make the VTP
an effective instructional program. Also, these O/Cs and ECs
seemed to be satisfied with their training to be an O/C or EC and
with most of the instructional materials associated with the VTP.

However, these participants suggested fine-tuning the VTP.
They especially noted the need to: (a) reorganize the OCIC
handbook, (b) standardize the OPORD narratives across echelons
and to military doctrine; (c) use the AAR tapes to improve the
THPs; (d) allow the O/Cs to have more flexibility in preparing
for and conducting the AARS; (e) have more hands-on training on
using the SIMNET and Janus equipment; and (e) eliminate
duplication among the VIP training support materials. These VTP
instructional personnel also claimed that the typical unit is not
prepared for their VTP training. Perhaps then, as suggested by
Shlechter and Anthony (1996), a structured instructional program
is needed for helping units prepare for their VTP rotation.

Some of the problems listed above may have been rectified by
the current O/C team. In reviewing this report, a current VTP
O/C has noted that units coming to the VTP during the Winter
months of 1996 are distinctly more prepared for their VTP
rotation than those coming prior to this period. This
improvement has been attributable to a structured workshop for
helping VIP participants prepare for their VTP rotations, which
has recently been developed by the 0/C team. A certification
course has also recently been developed by the O/C team for
training new personnel.

This investigation’s data also have ramifications for future
use of the VTP and other similar structured instructional
programs for simulation-based training systems. One ramification
is that unit personnel, in case of resource constraints, can
possibly be trained to be a VIP O/C. However, as stressed by the

25



3a. Why was the time "Too Much" for those activities you checked?

3b. Why was the time "Too Little" for those activities you checked?

3c. Do you think the allocation of time among the VTP program activities should be
changed? If so, how?

3d. After the units’ first VTP rotation, what refamiliarization is needed when units return
for additional rotations?

Training Structure Interview

2. Do the VTP tables consistently increase in difficulty? What are the exceptions?
Should the order of the tables be changed?

4. Should some MTP tasks or subtasks be added to the VTP? Are there other critical
tasks or subtasks that should be added that are presently undefined?

5. Should there be more or less task overlap and repetition among the VTP tables?

6. Should the AAR agenda or process be improved in some way?

7. What AAR aids were most useful? What changes or additions are needed?

Training Proficiency Interview
(For the following questions, do not limit answers to areas in the questionnaire items.)
1a. What aspects of leader performance increased the most during VTP rotations?

1b. What aspects of leader performance increased the least during VTP rotations?

2a. What aspects of unit performance increased the most during VTP rotations?



percent (14/34) of the responses to the item cited above
concerned improvements in the unit leaders’ situational awareness
(of the battlefield) and self-confidence.

3. The VIP had the least impact on helping unit leaders to
understand the importance of conducting troop-leading procedures
and delegating authority to subordinates. The BN leaders were
the focus of the latter criticism. (Data associated with these
findings are presented in F-2.)

4. Units became more proficient with regards to reporting (27%
or 12/45), conducting actions on contact (24% or 11/45),
conducting formations (22% or 10/45), and being a cohesive
fighting force (20% or 9/45). Each of these categories
represented approximately 20% (9/44) of the responses to the item
dealing with improvements in unit performance (see F-3).

5. The units’ situational awareness, gunnery abilities,
defensive tactics, and cross-talk skills improved the least.
These categories accounted for 67% (22/33)of the comments on the
item regarding aspects of unit performance that increased the
least (see F-4).

Questionnaire data, The following points were deduced from
the information presented in Figures 1 and 2. (Numerical values

associated with these figures are detailed in Appendix J.)

Area of Proficiency

Overall performance —

Explaining reasons for a

Controlling untt fire and maneuver

Monitoring the battlefleld situation

‘Troop-leading procedures

100

IN

Figure 1. Estimates of leader proficiency before and after
training in the VTP.

20



The VIP's O/C Team

Overview of the O/C team, At the time data were collected

for this report, the O/C team was staffed with 21 military O/Cs
and 11 exercise controllers (ECs). The O/Cs were primarily AC
personnel, ranging in rank from Lieutenant Colonel (05) to
Sergeant First Class (E7). The field-grade officers served
primarily as 0/Cs for the BN exercises; the CO-grade officers
served primarily as 0/Cs for CO and PLT tables; and the
noncommissioned officers served primarily as O/Cs for the PLT
tables. The ECs were civilian government employees who were also
training analysts (TAs). The original group of 0/Cs joined the
VTP in late 1993 and early 1994, while the ECs joined the team in
early to mid 1994. (Further demographic information about the
original O/C team is found in the method section.)

The O/Cs' duties. The O/Cs have the following

responsibilities associated with implementing the VTP:

1. Preparing VTP participants for their training by visiting
them at their home-station (Unit Preparation). During such
visits, the O/Cs provide the unit with all required training
support materials, including overlays and operation orders,
associated with completing VIP exercises. They also provide the
unit with suggestive VTP preparation techniques, e.g., rock
drills or rehearsals. In addition, the O/Cs supply the units
with demonstration videos that introduce them to SIMNET and show
exemplary units conducting successful VIP missions.

2. Providing units with a preview of each training table at the
O/C workstation (Table Preview). This preview, which occurs
during the time allocated for the preparation phase, involves
giving the units an operation order, providing them with a quick
overview of the battlefield situation, and answering their
questions about the mission.

3. Monitoring execution of the VIP training tables. This
responsibility involves role playing higher elements, e.g. a CO
Commander for a PLT set of tables. It also involves providing
feedback to the unit about their performance as they complete the
tables.

4. Conducting the AARs. According to SIMUTA’s instructional
design, the O/Cs are supposed to conduct the AARs by following a
a specific preplanned agenda. This agenda includes a discussion
by the unit of its performance. The O/Cs have been instructed to
facilitate this discussion through the use of discovery learning
techniques and by using the workstation's apparati, e.g., the
stealth vehicle display.

5. Completing the Take Home Packages (THPs). The THPs consist
of the O/Cs' observations regarding units' performance during

training. This package is sent to units at their home-stations
following their participation at the VIP, and is used to assess

4



EH.

Data for the General Questions

(Items la-1c deal with the three most positive or
beneficial aspects of the VIP to the participants)

la.

1b.

1c.

Most beneficial aspects:
Cost-effective/Quality training ...................
Structured training .......... cece eee eee eee eee eee
Audio/Visual (AAR) feedback ............. eee eee eee
Dedicated O/C and EC team ......... eee eee eee eee eee
Hardware/Software 2.2... eee eee eee eee eee eee

ww moo

Second most beneficial aspects:

Cost-effective/Quality training ..................- 12
Structured training ........ 0. ee eee eee eee eee eee 13
Audio/Visual (AAR) feedback ...........-. 2 ee eee eee 3
Dedicated O/C and EC team ........ ee eee ee eee eee eee 1
Hardware/Software 1.1... 6. cee eee eee eee eee 2

Third most beneficial aspects:
Cost-effective/Quality training
Structured training . : + : wee
Budio/Visual (AAR) feedback ............ 2. eee eee eee
Dedicated O/C and EC team ........ cee eee eee eee
Hardware/Software

BW oO oO

(Items 2a-2c deal with the participants’ opinions about the
three aspects of the VIP that need to be modified)

2a.

Most in need of modification:
Hardware/Software ........ ccc eee eee eee eee 12

Training development process 7
Standardization of OPORDs?® os : 3
Other aASpectS .... eee eee ee ee ee ee ee eee eee 6

3? Standardization involves making sure that the OPORDs are

consistent across echelons and conform to Army doctrine.

D-13



It

Overall Performance

"The standard error of the difference mean.

Table I-4

Participants’ Opinions on the Percentage of Unit Improvement Attributable to
Coachi dapti to ain and ipment

0-25% 26-50% 51-75% 76-100%

be
Bb
to
b
bo

Material 2a zB a

Function of coaching
Management Group 2 33.3 1 16.7 3 50.0 0 0.0
Other Groups 1 4.3 4 17.4 13 56.5 5 21.7

Adapting to terrain
Management Group 5 83.3% 1 16.7 0 0.0 0 0.
Other Groups 5 21.7 13 56.5 3 13.0 2 8

Adapting to equipment
Management Group 6 100.0% 0 0.0 0 0.0 0 0.0
Other Groups 5 21.7 12 52.2 4 17.4 2 8.7

*Statistical tests showed that the cited proportion for Management Group was

Table J-3 Continued
Comparisons Cases oft Mean SE*
Diff
Executing Battle Drills vs. 26 25 5.46 2.50 2.19
significantly different ( p < .02) than those for the other groups.



TRAINING PROFICIENCY: The following items identify specific aspects of leader or
unit performance.

4. On anumerical scale from 0% to 100% where 100% represents the best possible
performance and 0% the worst, estimate the level of performance for a typical
unit leader at the start and end of his VTP rotation.

e.

Troop-leading procedures
Monitoring the battlefield situation
Controlling unit fire and maneuver

Explaining reasons for their actions
during the AARs

Overall performance

Start

End

2. Using the same scale as for item set 1, now estimate the level of performance

for a typical unit at the start and end of their VTP rotation.

Start_

e,

Maintaining formations and
executing formation changes

Making reports
Executing battle drills

Occupying and defending
battle positions

Overall performance

End

3. Estimate the percent of unit improvement that was a function of coaching by the
O/Cs or E/Cs.
0-25% O 26-50% O 51-75% O

4. Estimate the percent of unit improvement that was a function of
adapting to the SIMNET terrain data base.

76-100% 1

0-25% L) 26-50% O 51-75% 1 + 76-100% O

PTE 59-72



Appendix D

Data for the Structured Interviews

1 Numbering scheme used in this appendix differs from that
used for the structured interviews (see Appendix B.)

Number of
Comments
A. Data for the Demographic Questions*
| 1. Most helpful previous experience:
| Military/Field experience ............ eee eee eee eee 22
| INSCLUCEOL 2... ee eee ete eee eee eens 11
National Training Center (NTC) ............e-eeeeee 7
| Computer 2... cece eee ee eee eee eee eee 5
Other experiences ........ 0. cece eee eee eee 7
| 2. Important qualifications for selecting an
observer/controller (0/C):
| Field experience at appropriate level ............. 18
| Proper schooling ..........0c cece eee cece eee e ee eee 12
| Computer/Equipment background .............++++-eee 4
Job experience as field O/C 4
Other experiences ........ cc cece eee eee eee teens 4
3. Important qualifications for selecting an
exercise/controller (EC):
| Military/Field experience : oe : oe oe 21
Tactical knowledge ........... 2c eee eee eee eee eee ee 15
| Computer literacy ........ cece cee ee eee eee 13
| Education/Technology skills 6
B. Data for the Train The Trainer Questions
| 1. Training received:
| Workstation/Equipment training 22
Conducting an after-action review (AAR) ........... 10
| Doctrinal (tactics, techniques, procedures) 4
Other types of training ..................200- . 8

| D-1



Handbooks for the Observer/Controliler In Charge (OCIC The
materials in these handbooks include: (a) OPORD narratives;
(b) execution guidance; (c) table preview structure; (d) AAR
worksheets; (e) events guides for the OCIC; and (f) AAR guides.
Each events guide delineates the OCIC’s actions during a table.
The PLT set of volumes also includes a separate OCIC handbook for
the armor, mechanized infantry and scout PLTs.

Handbooks for the ECs, These handbooks contain the events
guides to be used by the EC. They also include the SIMNET
planning sheets, which are used by the EC for initiating vehicle
and route specifications. Again, separate handbooks have been
created for the different types of PLTs.

h “i mpon:

As discussed, the VTP is a structured training program which
includes use of: (a) the SIMNET and Janus training systems, (b)
structured training exercises/tables; (c) the O/C team, and (d) a
library of training support/management materials. Also, embedded
into this program are procedures for preparing the units for
their VTP rotation. However, a "formal program" for training new
members of the O/C team had not been established at the time of
data collection for this report (See Campbell et al., 1995;
Hoffman et al., 1995.; and Turecek et al., 1995 for further
information regarding these components. )

Th ‘s Trainin

Instructional efficiency, Because of the VIP’s training
efficiency, PLT and CO participants have favorably received this
program (Hoffman et al., 1995). Also, Bessemer, Shiechter,

Nesselroade, and Anthony (1995) have noted that the VTP
structured training is more efficient than some past SIMNET
training programs. For instance, VIP participants have tended to
complete two-to-three more SIMNET exercises during a given period
of time than have participants in relatively unstructured
training.

Instructional effectiveness, Shlechter, Bessemer,
Nesselroade, and Anthony (1995) used a multimethod-multisource
approach to provide empirical information regarding the VTP's
instructional effectiveness. Trained observers collected data
from nine units; fourteen VTP O/Cs completed standard rating
forms regarding the performance of 38 armored force units; and
280 training participants completed Likert-scale items regarding
their training experience. Data from the different sources
showed that the units further developed their collective tactical
skills across the training period. The observers found that the
sampled units took significantly less time, made fewer errors,
and needed less coaching as their training progressed. The
instructors indicated that most units had a greater likelihood of
getting more proficient in critical subtasks than either not
improving or getting worse in them. And, the participants