References

Alonso, D. L., & Norman, K. L. (1998). Apparency of contingencies in single panel and pull-
down menus. International Journal of Human-Computer Studies, 49, 59-78.

Anderson, J. R. (1980). Cognitive psychology and its implications. San Francisco: W. H.
Freeman.

Andrews, D. H., & Bell, H. H. (2000). Simulation-based training. In S. Tobias & J. D. Fletcher
(Eds.), Training and retraining: A handbook. for business, industry, government, and the
military (pp. 357-384). New York: Macmillan Reference.

Atkinson, R. K., Derry, S. J., Renkl, A., & Wortham, D. (2000). Learning from examples:
Instructional principles from the worked examples research. Review of Educational
Research, 70(2), 181-214.

Ausubel, D. P. (1968). Educational psychology: A cognitive view. New York: Holt, Rinehart,
and Winston.

Baddeley. A. (1994). The magical number seven: Still magic after all these years?
Psychological Review, 101(2), 353-356.

Berliner, D. C., & Cahen, L. S. (1973). Trait-treatment interaction and learning. In F. N.
Kerlinger (Ed.), Review of Research in Education (Vol. 1, pp. 58-94). Itasca, IL: F. E.
Peacock.

Briggs, P. (1990). Do they know what they’re doing? An evaluation of word-processor users’
implicit and explicit task-relevant knowledge, and its role in self-directed learning.
International Journal of Man-Machine Studies, 32, 385-398.

Carroll, J. M. (1997). Toward minimalist training: Supporting the sense-making activities of
computer users. In M. A. Quinones & A. Ehrenstein (Eds.), Training for a rapidly
changing workplace (pp. 303-327). Washington, DC: American Psychological
Association.

Carroll, J. M., Mack, R. L., Lewis, C. H., Grischkowsky, N. L., & Robertson, S.R. (1985).
Exploring exploring a word processor. Human Computer Interaction, 1, 283-307.

Centric, J. H., Wampler, R. L., & Dyer, J. L. (1999). Observations of Infantry courses:
Implications for Land Warrior (LW) training (ARI Research Note 2000-04). Alexandria,
VA: U.S. Army Research Institute for the Behavioral and Social Sciences. (DTIC No.
AD-A372 853)

Chandler, P., & Sweller, J. (1996). Cognitive load while learning to use a computer program.
Applied Cognitive Psychology. 10, 151-170.

53



Table E-14
Descriptive Statistics on Time on Map Lesson/Exercises and Map Final Exam (times in minutes

and fractions of minute)

Condition | M [SD [Man | Min-Max | N
Zoom, Pan, Find Me

Low Demand 15.17 2.64 14.78 11.03-26.00 43
High Demand 13.92 2.40 13.77 8.40-20.53 39
Find X

Low Demand 14.03 2.86 13.55 9.08-22.98 42
High Demand 13.37 2.43 13.43 7.88-20.52 37
Display

Low Demand 22.29 3.88 22.27 14.88-29.42 42
High Demand 21.66 4.09 21.64 12.75-29.72 36
Range Azimuth °

Low Demand 2.98 0.97 3.03 0.82-6.62 43
High Demand 4.23 1.41 4.32 1.63-9.23 40
Final Exam on Map

Low Demand 15.07 2.12 15.40 11.28-19.35 39
High Demand 15.33 2.63 15.16 10.00-20.72 32
Exploratory 22.02 5.37 20.75 12.23-42.38 79
Map Training *

Low Demand 54.54 8.68 55.30 38.20-74.82 42
High Demand 53.65 8.43 54.40 33.02-72.62 36
Exploratory 12.63 6.32 11.08 2.10-34.88 79
Total Time *

Low Demand 69.28 9.21 69.40 50.27-83.90 39
High Demand 70.81 10.26 70.65 45.90-96.53 32
Exploratory 34.65 8.57 33.58 19.65-59.00 79

Note. Range-Azimuth times are based only on the lesson as there were no practice exercises.
® Zoom and Pan. F(1, 74) = 5.34, p = .0235; Low faster than High.

> Range and Azimuth. F(1, 75) = 20.14, p = .0000; Low faster than High.

© Final Exam. F(2, 138) = 46.34, p = 0000, Low and High faster than Exploratory.

4 Map training. F(2, 143) = 622.65, p = .0000; Exploratory faster than High and Low.

© Total time. F(2, 138) = 296.45, p = .0000; Exploratory faster than High and Low

E-14



the abbreviations in Figure 5. Figure 5 graphically illustrates the map training and testing
sequence for all conditions.

Lesson I — Zoom Pan (6 chunks). Four buttons or functions were presented (zoom in, zoom
out, pan, and find oneself). Two (zoom out and find oneself) were only one-step (e.g., click
on the button and the map zooms out). Both the zoom in and pan buttons had two additional
steps. After clicking on the button you then had to click on the map to completely activate
the function. To deactivate the function, you had to click on the button again.

Lesson J — Find X (6 chunks). The lesson was on how to find others (units and individuals),
called find X. Only one entity (unit or individual) could be displayed at a time with this
function. Only one button was used, but it triggered a series of embedded menu selections.
In this process, the soldier had to learn five additional critical elements (find unit, go to lower
echelon, go to higher echelon, find individual soldiers, find leaders at a specific echelon).

Lesson K - Display (9 chunks). The lesson was on how to display units and individuals. The
function was designed as a pre-planning tool. Individuals could decide who they wanted to
display on their map during a mission, and then make these selections during mission
preparation. Only one button was used, but it triggered three menu selections (company
menu, platoon submenu, and squad submenu). In determining the number of chunks of
information, we counted each menu as one chunk of information (3 chunks). Three menu
elements were similar to the find X (Lesson J), and therefore were counted as one chunk of
information. These were display unit, go to upper echelon, and go to lower echelon. Within
two of the three menus (company and platoon), there were two new categories of menu
selections: one led to a further breakdown of the unit and the other led to a block of “all”
selections. For example, with the platoon submenu, further breakdowns could be found for
Co Hgs, Ist platoon, 2d platoon, and 3d platoon. The “all” selections within the platoon
submenu were: all platoons, all platoon leaders, all platoon sergeants, and all squads. So
within the company menu and the platoon submenu, each of these two sections was
considered a critical chunk of information, for an additional 4 chunks of information. (The
labels themselves were clear, once you understood the concept). Based on this rationale, the
total number of chunks was 9. The rationale for defining the number of chunks for this type
of task was consistent, but somewhat subjective.

Lesson M - Rg Azm (5 chunks). The last instruction was on determining range and azimuth.
Again, there was a single button, two clicks were required to establish the points or
units/individuals on the map, soldiers had to know where to click to get an accurate reading,
and another click was required to cancel the function.

The menus in the find X and display functions were hidden from the soldier’s view, so

the sequence of selections was not apparent from the map display (see Figure 4). The instruction
within the Low and High Demand lessons revealed the various menu contingencies to the
soldiers (see Appendix D for illustrations). Thus soldiers were formally taught and shown the
different paths available to them with these functions. This technique was found to be
particularly beneficial for individuals with low spatial abilities, although the technique also
helped those with high spatial abilities (Alonso & Norman, 1998).

18



U.S. Army Research Institute
for the Behavioral and Social Sciences

A Directorate of the U.S. Total Army Personnel Command

EDGAR M. JOHNSON
Director

— NE

Technical Teview by

William R. Sanders
Robert J. Pleban

NOTICES

DISTRIBUTION: Primary distribution of this Research Report has been made by ARI.

Please address correspondence concerning distribution of reports to: U.S. Army
Research Institute for the Behavioral and Social Sciences, Attn: TAPC-ARI-PO, 5001

Eisenhower Ave., Alexandria, VA 22333-5600.

FINAL DISPOSITION: This Research Report may be destroyed when it is no longer
needed. Please do not return it to the U.S. Army Research Institute for the Behavioral

and Social Sciences.

NOTE: The findings in this Research Report are not to be construed as an official
Department of the Army position, unless so designated by other authorized documents.



Map: Exercise and Final Exam Scores

Table E-10
ANOVA on Map Exercises and Map Final Exam Scores: High-Low-Exploratory Conditions by

Soldier Group

Factor | af ] MS | F | Dp

Zoom, Pan, Find Me

High-Low Demand 1, 74 305.29 2.66 1071

Soldier Group 3,74 227.95 1.99 1233

Interaction 3,74 33.26 0.29 8326

Find X

High-Low Demand 1,71 362.46 1.32 .2540

Soldier Group 3,71 782.61 2.86 .0431

Interaction 3,71 224.42 0.82 -4878

Display

High-Low Demand 1, 70 2992.16 4.07 0474

Soldier Group 3,70 1846.66 2.51 0655

Interaction 3,70 573.97 0.78 5086

Final Exam on Map

High-Low- 2, 142 1704.01 6.84 0014

Exploratory

Conditions

Soldier Group 3, 142 819.17 3.29 .0226

Interaction 6. 142 92.18 0.37 8971
E-10


Section B
Survey on Problem Solving Behavior

Below are 15 activities that you may engage in from time to time. Next to each item, please circle the ONE
response that BEST DESCRIBES how you typically approach each activity. That is, do you USUALLY:

1. Complete the activity (or give up) without referring to instructions, a map, or asking for directions;

2. Proceed at first without reading instructions, a map, or asking for directions, but will refer to these if
you run into a problem;

3. Read the instructions, a map, or ask for help before beginning;

4. Find someone else to perform the task for you (a friend, spouse, etc.).

Complete Only refer to Read Find Not

task or give instructions, instructions, someone else applicable

up without map, etc. if map, etc. to do it

instructions, there are before

map, etc. problems beginning

Set up a new computer 1 2 3 4 N/A
Change a tire on a new vehicle 1 2 3 4 N/A
Change settings on a new digital watch 1 2 3 4 N/A
Drive an unfamiliar route to a new destination more 4 2 3 4 N/A
than 100 miles from home
Cook a frozen dinner 4 2 3 4 N/A
Fill out tax forms 1 2 3 4 N/A
Program a new VCR 1 2 3 4 N/A
Find your seat at a ballgame or concert 1 2 3 4 N/A
Play a new video or computer game 1 2 3 4 N/A
Take a new, over-the-counter medication 4 2 3 4 NIA
Put together a new bicycle 4 2 3 4 N/A
Learn a new computer software application 1 2 3 4 NIA
Put up a new tent 1 2 3 4 N/A
Respond to a survey 1 2 3 4 N/A
Put together a piece of furniture that requires assembly 1 2 3 4 NIA

Please rate how well each of the following statements describes you. Circle any number from 1 to 10. |
i
Does not Somewhat Describes |
describe Descriptive me |
me at all completely |
| seem to have a “knack” or “feel” for finding my way 1 2 304 «5 6 7 8 9 10 |
around a computer program |
j
| Even if | already know one way to perform a task on the 1 2 3.4 «5 6 7 8 9 10
computer, | usually figure out a shortcut that will allow me
to do the same task with fewer steps __



Table C-21

Mean Number of Tasks Checked in each Response Option to the Working Independently Items

All
OSUT BNCOC ANCOC IOBC Soldiers

Response Option M M M M M

(SD) (SD) (SD) (SD) (SD)

(Range) (Range) (Range) (Range) (Range)

Tasks usually 3.10 2.35 1.87 2.24 2.45
completed without (3.32) (2.65) (2.27) (2.55) (2.79)
instructions* (0-10) (0-10) (0-9) (0-10) (0-13)
Tasks usually 4.71 4.46 4.63 4.98 4.69
attempted wo/ (2.45) (2.74) (3.55) (2.68) (2.79)
instructions until (0-10) (0-13) (0-13) (0-12) (0-13)
problem occurs?
Tasks usually 4.96 5.50 5.93 5.24 5.36
performed only after (3.06) (3.02) (3.30) (3.03) (3.08)
reading instructions ( (0-12) (0-12) (0-13) (0-11) (0-13)
or given guidance)°
Tasks usually given to 0.17 0.29 0.20 0.24 0.23
someone else to do® (0.48) (0.65) (0.48) (0.53) (0.54)

(0-2) (0-3) (0-2) (0-2) (0-3)
Tasks where soldier 0.06 0.40 0.37 0.31 0.27
could not respond (e.g., (0.32) (0.89) (0.96) (0.64) (0.73)
not done before) --- (0-2) (0-5) (0-4) (0-3) (0-5)

Not applicable®

Note. Because soldiers were required to select one of the response options listed for each of the
13 tasks, means in each column sum to 13.
* Completed without instructions: F(3, 160) = 1.55, p = .204.
> Attempted without instructions until problem occurs: F(3, 160) = .26, p = .853.

© Performed only after reading instructions: F(3, 160) = .80, p=.50.

“Task usually given to someone else: F(3, 160) = .45, p = .72.
Not applicable: F(3, 160) = 2.10, p = .10.

c-9



it would be possible to either follow step-by-step instructions or plunge right in and attempt the
task independent of directions. Following our development of the tendency to work
independently scale, it was learned that a similar scale has been developed in Dutch by Glasbeek
(as cited by DeMul & Van Osstendorp, 1996), who found that individuals with an exploratory
learning style were more suited for minimal manuals than traditional manuals. De Mul and Van
Oostendorp (1996) modified and administered the scale in an experiment of “exploration-
supportive” facilities in a computer interface. We were unable to obtain a copy of the scale.
However, a brief description in De Mul and Van Oostendorp indicates that our approach to
assessing this construct was somewhat different from theirs. De Mul and Van Oostendorp
presented participants with statements that each described an approach to learning a new
apparatus, and had them rate the frequency with which they used each approach. In contrast, we
presented soldiers with fifteen activities and asked them to indicate which of three learning
approaches they would use for each task. A detailed description of our scale is in the method
section.

De Mul and Van Oostendorp (1996) found no relationship between scores on their
“learning style” scale and performance with a novel computer interface. However, the
researchers indicated that the interface may have been too simple for differences to emerge, and
observed that they had a ceiling effect in which participants essentially mastered the interface
after performing a few tasks. In the present experiment, it was expected that a tendency to work
independently would be related to success at mastering the interface for the soldiers in the
Exploratory map learning condition. The relationships between tendency to work independently,
time spent exploring, and scores on the map final exam were examined.

Computer-Based Training (CBT)

All training in the experiment was computer-based, as it was envisioned that this form of
training will eventually be common with the Army’s digital systems. Research has shown that
CBT can be more efficient and effective than traditional classroom instruction, and that it adapts
well to differences in the learner’s rate and level of competency (Fletcher, 2000; Gibbons &
Fairweather, 2000). Also it was of interest to know the challenges posed in writing courseware
for a digital environment.

Of particular interest was prior training research with CBT on the Army’s digital systems.
Sanders’ (2001) identified principles of learning from behaviorist, cognitive and constructivist
theories that could be applied to designing computer-based training for command, control,
communications, computers, and intelligence (C41) systems, such as the Force XX] Battle
Command Brigade and Below (FBCB2) system. In a follow-on effort, Deatz and Campbell
(2001) demonstrated how one might incorporate some of these cognitive principles (advance
organizers, chunking, frames, rehearsal, etc.) within CBT. But this was only a demonstration,
not a training program. Sanders (2001) referred to a web site that provided training on FBCB2.
However, in reality this medium was similar to an electronic version of a technical manual. It
was not an interactive training medium where specific skills were taught, practiced and tested
within an operational context (hypothetical real-world missions and situations), and where
transfer to new situations was examined. Schaab and Moses (2001) cited observations of
soldiers using and training with the All Source Analysis System (ASAS), a digital system for



Table E-25
Correlations between Background Measures and Final Exam Scores

Criterion Measures and Training Conditions
Background Measures Code Exam Map Exam
Low High Low High Exp
(n= 84) | (n=83) (n= 40) (n= 34) (n= 80)
Computer background
-Own computer 10 21 .06 14 18
-Use computer features ll 17 25 .20 .23*
-Self-rating of expertise 18 .28** .29* .32* .28**
-Icon score .26* .29** .35* .29* A2**
-Short cut/Knack 17 13 .38** .03 .24*
BR AT 01 -.10 -.12 -.04
Military knowledge
-Symbols .28** .21* .23 28 .33**
-Map Reading .24* .29** 01 .36* .20
Tendency to work
independently -.01 01 -.09 -.05 -.07
Multiple R — all predictors 42 46 51 52 52
F 1.77 1.95 1.20 95 2.83
af 9, 73 9,71 9, 30 9, 23 9, 69
Pp .088 059 333 504 .007
Multiple R — stepwise 33 Al 38 46 AT
F 4.99 TAS 6.52 4.08 10.94
af 2, 80 1,49 1, 38 2, 30 2,76
Pp 009 -001 015 027 .000

*p<05. * p<ol. **p<.00l.

E-19



Table C-10

Percentage of Soldiers Indicating Different Levels of Computer Skill

Self-Ratings of Computer Skill

Good w | Good w 1Progm | Several Bill
Soldier N | Novice | 1 softw | several Lang + Progm Gates
Group program | Soft Progr | Software | Lang+Soft | hire me
OSUT 48 38% 25% 35% 0% 2% 0%
BNCOC 48 44% 25% 29% 2% 0% 0%
ANCOC 30 40% 20% 40% 0% 0% 0%
IOBC 42 12% 24% 45% 14% 5% 0%
Table C-11
Descriptive Statistics on Self-Ratings of Computer Skill
Self-Ratings of Computer Skill
Soldier N M Mdn SD Range | Interquartile
Group Range
OSUT 48 2.04 2.00 0.97 1-5 1-3
BNCOC 48 1.89 2.00 0.91 1-4 1-3
ANCOC 30 2.00 2.00 0.91 1-3 1-3
IOBC 42 2.76 3.00 1.01 1-5 2-3

Note. Scale: Novice = 1, One software program = 2; Several software program = 3, One

program language + software = 4, Program languages + software = 5; Bill Gates hire me = 6.
F(3, 164) = 7.38, p = .0001; IOBC mean rating higher than ANCOC, BNCOC and OSUT.
Chi-square test was also significant, 2 (12) = 27.48, p = .0065.

Table C-12

Descriptive Statistics on Icon Test Score

Icon Test Scores
Soldier Interquartile | Mean %
Group M Mdn Range SD Range Correct
OSUT 8.92 8.50 1-15 3.39 6.5-12 50%
BNCOC 7.97 7.75 1-15 3.48 5.5-10.75 44%
ANCOC 9.53 9.50 2-15 3.12 75-11 53%
IOBC 11.17 11.00 4-15.5 2.58 10-11 62%

OSUT; ANCOC mean score higher than BNCOC.

Note. FG, 164) = 7.89, p = .0001. IOBC mean score higher than ANCOC, BNCOC, and



The map final exam (Table 7) covered all seven functions/buttons. The questions also
required that the soldiers identify units and individuals; the material covered in Phase I of the
experiment. Feedback, percent correct, was provided upon completion of the exam. There was
no feedback on individual test items.

Table 7
Final Exam on Map Functions

© The map final exam had 22 items, with some two-part questions, for a total of 30 points.
Items worth 2 points required soldiers to perform two functions to complete the task. Partial
credit was given when one of the two functions was performed correctly on a 2-point item.
Test items differed from those in the exercises.

e Ofthe 22 items, 3 directly tested knowledge of the BR numbering system by asking soldiers
to view the codes for nine squad members and identify which, if any, were from a different
squad.

e The remaining items asked soldiers to accomplish a variety of tasks such as locating points
on the map, displaying and locating individuals and units, and determining range and azimuth
between two points. With the exception of the range/azimuth tasks, there was typically more
than one way to accomplish a task, although there was clearly a most efficient function for
each task. Therefore, soldiers were not graded on the method used. Instead, each task
required them to click on a map symbol or location, or to type a response in a text box to
indicate their successful accomplishment of the task. The one-minute time limit per question
made it difficult for soldiers to respond successfully without using the most efficient function
for task completion.

e This was the first time soldiers were required to execute range/azimuth tasks, although those
in the Exploratory condition had the opportunity to work with this function.

25


Mew Sita: Yow av the commande, Company A Tek iplay al pltoen
IBACK |; Isadore Ck on 34 Patt then Cr WEXT



100

95 ~ a Low Demand
s Se
.
90 ‘ ~o
N

” 7 -
3 7 ’
5 85 High Demand , :
(3) 4
x oO
5
o@ 80
=

75

70

Wpn/Unit Sym Co/Sqd BR Ldr/Unit Codes
Battle Roster Rifle-Wpns Codes Final Exam

Code Exercises and Final Exam

Figure 8. Mean scores on the code exercises and code final exam for the High and Low Demand
conditions. (Scores with significant differences between High and Low are highlighted.)

On the rifle and weapon squads score, it is important to remember that the Low Demand
score represents the combined score from two separate exercises, as the instruction for the Low
condition was divided into two blocks of instruction (rifle and weapon squads codes). This part
of the instruction also involved the greatest differences in working demand for the two
conditions (an estimated 9 chunks per lesson versus 18 chunks per lesson, reference Table 2),
and the greatest potential for confusion. Soldiers had to learn two squad sets of positions, each
associated with numbers 0 through 8. In both cases, the squad leader was coded a 0, but, as
indicated below, this was the only commonality across the number assignments in the two types
of squads.

Number Rifle Squad Position Weapons Squad Position
0 Squad Leader Squad Leader
1 A Team Leader 1st Machine Gunner
2 SAW Gunner, A Tm Assistant Gunner to 1st Machine Gunner
3 Grenadier, A Tm 2d Machine Gunner
4 Rifleman, A Tm Assistant Gunner to 2d Machine Gunner
5 B Team Leader 1st Javelin Gunner
6 SAW Gunner, B Tm Assistant Gunner to Ist Javelin Gunner
7 Grenadier, B Tm 2nd Javelin Gunner
8 Rifleman, B Tm Assistant Gunner to 2d Javelin Gunner

32



Appendix B

Pre-experiment Instruments

B-1



100 100
1OBC, ANCOC iu]
& BNCOC
90 e 90
80 80
70 e 70
60 osut A & @ 08 60
g A «
2 ANCOC
§ ° osut 50
5 @ sncoc
a 40 40
30 : 30
% Use % Own % Not Icon Score:
Computer i ¥
20 P Computer Novices % Correct 20
40 40
0 0

Figure 6. Summary of computer usage and ownership, and subjective and objective indices of
computer expertise.

Military Knowledge: Military symbols, the battle roster system, and map reading.
OSUT soldiers were expected to score lowest on these initial tests, and ANCOC soldiers were
expected to perform better than BNCOC soldiers. It was expected that IOBC soldiers would
score as high as ANCOC and in some cases, higher. In general, the test results were consistent
with these expectations.

The mean percent correct on each test for each soldier group is in Table 10. Clearly, each
group of soldiers was more knowledgeable about map reading than symbols, although ANCOC
soldiers performed well on the unit symbol test.

Table 10
Mean Percent Correct on Military Knowledge Tests
Test Soldier Group All Groups
OSUT BNCOC ANCOC IOBC
Symbols
Weapon 2% 12% 15% 29% 14%
Unit 1% 15% 73% 54% 31%
Map Reading 55% 86% 85% 78% 75%

28



6. What is the function of the following icons?

wr

(oa

= &

ub

e

ws

uP

gy

B-4


D-24



Table E-19
Correlations Among Map Scores (Low and High Demand)

Map Scores
Zoom/Pan Find X Display Exam
Zoom/Pan 1.00 .33** 36*** A5***
Find X 1.00 33** 31**
Display .61***
Exam 1.00

Note. N=71-82. (some missing data on some variables).
*p<.05. **p<.0l. *** p<.001.

Table E-20
Correlations Among Map Times (Low and High Demand)
Map Lesson Times
Zoom/Pan Find X Display Range/Azm | Exam
Zoom/Pan. 1.00 61 FF 58*F* .22* 61 ***
Find X 1.00 7148 42*t* .75***
Display 1.00 53*** .734**
Range/Azm 1.00 55***
Exam 1.00
Map Exercise Times
Zoom/Pan 1.00 BOF FF ST FFF NA ATF
Find X 1.00 54tte NA 56tt*
Display 1.00 NA 66***
Range/Azm NA NA
Exam 1.00

Note. N= 71-82 (some missing data on some variables). There were no range/azimuth

exercises. Correlations between lesson and exercise times for the map functions were as follows:
Zoom/Pan/Find Me = .30*; Find Others = .31*; Display =.18

*p<.05. **p<.0l. *** p<.001.

Table E-21

Correlations Between Map Scores and Map Times (Low and High Demand)
Map Map Times
Scores Map Lesson Map Exercise/Exam
Zoom/Pan 15 w/ zoom/pan lesson time -.35*** w/ zoom/pan exercise time
Find X -.04 -.46*#*
Display 12 -.43*F*
Exam -.14 w/ total time in map training | .06 w/ time to take exam

Note. N=71-82 (some missing data on some variables).
*p<.05. **p<.0l. *** p<.00l.

E-17



BR Rifle & Wpn

WpnlUnit Co-Sqd qd Members Lars & Units Final Exam
‘Symbols
0 10 20 30 40 50 60 70 80

Mean cumulative time in minutes
Segments correspond to blocks of instruction

Figure 12. Cumulative time to complete training on codes, and times for each lesson by soldier
group (includes soldiers who had to repeat exercises).

Of additional interest was a shift in the relative times for the soldier groups within each
condition on the rifle and weapons squad block of instruction. When all soldiers were examined,
this interaction approached significance, F(3, 159) = 117.54, p =.0060, Table E-4. As shown in
Figure 13, for all soldier groups except IOBC, soldiers took less time to complete these blocks of
instruction when they were in the Low Demand condition than in the High Demand condition.
For the IOBC soldiers, the times were the same. However, the discrepancy between the High
and Low Demand conditions decreased (see Figure 13) when the analysis was restricted to the
soldiers who did not repeat the exercise, since the times decreased for the ANCOC, BNCOC, and
OSUT groups. In this case, the interaction effect did not approach significance (see Table E-7).
The “interaction with all soldiers” apparently resulted from the additional time required by
soldiers who had to repeat exercises in the High condition, and the increased time was in
proportion to the number of soldiers within each course who repeated the exercises.

In summary, based on the mean scores and mean times, more time was required when
performance was low. For example, the score on the rifle/weapons squads instruction was
significantly lower for the High Demand condition than the Low Demand condition, and soldiers
in the High Demand condition took significantly longer to complete this section as well. In
addition, OSUT soldiers often had the lowest scores, and typically required the most time.

38



Unlike companies, ©

there are-only four
possible:codes that
maybe used for
platoons:

' Platoon and Squad Level Battle Roster Codes
For example.

‘The platoons are'listed in.
‘Sequence: :

quad Level Battle Roster Code

za Plamen
ts Squad
adSqed 2
ZaSqad 3
Weapons Sqiedé_

D-7



Working Memory and Exploration in Training the
Knowledge and Skills Required by Digital Systems

Introduction

The Army is now automating the battlefield, creating what is known as digitized forces.
Computer software is being embedded in vehicles, to include tanks, command and control
vehicles, and Infantry fighting vehicles (General Accounting Office [GAO], 2000). In addition,
the dismounted soldier will eventually have a wearable computer, such as that in the Land
Warrior (LW) system (Goodman, 1999). Soldiers and leaders must use this software to execute
tasks such as sending messages and orders, and creating overlays; tasks which were formerly
conducted without a computer. In addition, these computer technologies give soldiers and
leaders additional capabilities; for instance, real-time monitoring of the location and movement
of units using digitized maps. In this report, we refer to this equipment as “digital systems,” and
the required soldier skills as “digital skills.”

This large-scale fielding of computer technologies raises training questions. In particular,
we need to know how to best train the diverse population of soldiers and officers who employ
these systems. One approach is to use computers as a means of delivering training. Much is
known about learning and cognition, yet questions remain regarding designing for interactive
multi-media training systems. Issues exist about how tasks “should be broken down into practice
activities, what activities the student should engage in first, how the activities should be
organized, how much practice the student needs in each of the activities, and so on. Poor
instruction often arises from the guesswork in the specification of these details” (Woolf &
Regian, 2000, p. 353). Theories of learning and training provide general guidance (Sanders,
2001), but not the technical details required for the designers of interactive multi-media
(computer-based) training. Empirical work, however, can address these unknowns, and reduce
some of the guesswork involved during the actual creation of these training programs.

The experiment described in this report focused on three factors that can influence the
design of computer-based instruction for digital systems.

© The type of task to be taught/trained: Declarative and procedural knowledge

© The instructional techniques: The volume of information presented before the user
works with the material and the degree of instructional guidance provided

e Learner characteristics: Differences in Army experience and knowledge

Featured in many of the Army’s digital systems is a computerized map that displays the
battlefield locations of units as well as tactical overlays (Sanders, 2001). This is true of the
computer display in the Land Warrior (LW) system. A unique feature of the LW system’s map
is that it must show individuals, down to individual squad members, as well as units. In order to
use the map effectively, soldiers must know the codes or the means by which units and
individuals are depicted. A challenge with the instruction described here was to develop a


FOREWORD

The research presented in this report was conducted by the Infantry Forces Research Unit
(IFRU) of the U.S. Army Research Institute for the Behavioral and Social Sciences (ARI) under
its TRAINMOD: Training Modernization (#204) work package. TRAINMOD involves
addressing training issues associated with new Infantry systems. New Infantry systems, such as
the Land Warrior (LW), have computer software that soldiers and leaders must use to execute
their mission. It is anticipated that much future training on these and similar “digital systems”
will be done with some form of interactive multi-media or computer-based training (CBT).
Under TRAINMOD, a series of experiments was begun on how to develop CBT to train the
target population on these systems. This report describes the results from the first experiment. It
examined the impact of the amount of information presented in a lesson as well as the
effectiveness of an exploratory mode of learning.

The results provide insights into the design of effective and efficient CBT for digital
systems. A training technique that divided instructional content into relatively small chucks and
then presented training exercises was more effective than a technique that presented large
volumes of information before soldiers had training exercises. In addition, an exploratory mode
of learning showed promise as it required the least time, but it did not yield the highest
performance. Variations in exploratory modes of learning that incorporate more traditional
training techniques may be more effective. The different training techniques were applied to two
different domains: learning “facts” and learning “software interface procedures.” Soldiers who
participated in the training had diverse military backgrounds and were representative of the chain
of command within an Infantry platoon, from the platoon leader to the rifleman. The research
also showed how CBT could be designed to incorporate tactical system software for background
instruction and demonstration screens, and as interactive screens for performance exercises,
yielding a high-fidelity training environment.

The results were briefed to the LW Training Working Group, the Project Manager-
Soldier Electronics/Land Warrior (PM-SE/LW), and the LW Manpower and Personnel
(MANPRINT) Working Group in May, June, and August of 2001 respectively. The findings
provided the PM-SE/LW with needed information on how computer-based training could be
incorporated in the design of future training for the Land Warrior system.

Vet Vat Semel
TA M. SIMUTIS
echnical Director



It might be expected that the number of soldiers who had to review a lesson and repeat
the corresponding exercise would be reflective of the impact of working memory demands on
performance. The number of soldiers who had to repeat each block of instruction is in Table 11.

Table 11
Number of Soldiers who had to Repeat Code Exercises

Block of Instruction/Exercises Low Demand (7 = 84) High Demand (x = 83)
Exercises that were the same in both conditions

Weapon Symbols 101%) 3 (4%)

Battle Roster System 3 (4%) 4 (5%)

Exercises that varied with condition
(Information in smaller chunks for Low Demand)

Company BR 0 (0%) 46%)
Platoon — Squad BR 0 (0%)

Rifle Squad Codes 2 (2%) 13 (16%)
Weapons Squad Codes 4 (5%)

Key Leader Codes 12 (14%) 11 (13%)
Unit Codes 17 (20%)

Note. Number of soldiers who had to repeat exercises within each course regardless of training
condition: 34 in OSUT; 20 in BNCOC; 16 in ANCOC; 6 in IOBC.

With respect to remediation, as expected, the first two blocks of instruction showed no
difference between the two conditions, as the amount of information presented was the same
under both conditions. With the next blocks of instruction on company through squad BR, no
soldiers had to repeat the exercise in the Low Demand group, as compared to 5% in the High
Demand group. With the rifle/weapons squad instruction, twice as many in the High Demand
group had to repeat the exercise. These percentages were consistent with the expectation that the
High Demand condition would not enable soldiers to acquire the information as easily as the
Low Demand condition. However, this pattern was not repeated on the last blocks of instruction
on key leader and unit codes. In fact the percentages of soldiers in the Low Demand condition
who had to repeat key leader and unit codes increased. They were 14% and 20% respectively;
much different from the 0% to 5% remediation rates for Low Demand in the other exercises.

An item analysis was conducted to try to determine what caused this increase. The
analysis showed that the exercise question format in conjunction with the total number of
questions in each exercise affected the results, even though the overall item difficulty was the
same for both conditions (.89 probability of being correct in both cases). An unexpected item
format may have “caught soldiers off guard.”

Exercises for the key leader and unit codes had two types of questions. These two
formats are illustrated in Figure 11 for the unit questions. In the format shown at the top of
Figure 11, the item stem was the unit symbol/BR number combination with the multiple-choice
alternatives being word descriptions of possible units. Of the 12 questions, 7 were this format.

35



Table C-7

Descriptive Statistics on the Sum of Feature Use Ratings

Soldier Sum of Feature Use Ratings

Group N M Mdn SD Range Interquartile Range
OSUT 48 19.48 21.5 759 0-28 16-21.5
BNCOC 48 17.50 19.0 7.53 0-28 14-23
ANCOC 29 20.31 21.0 6.08 0-28 18-25
IOBC 42 20.57 21.5 5.13 2-28 17-21.5

Note. The 7 features were rated on a 0 to 4-point scale, ranging from “never used” to “daily
use.” Maximum score was 28 representing daily use of all 7 features; minimum score was 0
indicating a soldier never used any of the 7 features.
F(3, 163) = 1.84, p = .1409.

Table C-8

Percentage of Soldiers Indicating Different Levels of Typing Skill

Self Ratings of Typing Skill

Soldier Hunt & Peck Hunt & Peck Type Slowly Type Quickly
Group Slowly Quickly
OSUT 6% 48% 25% 21%
BNCOC 19% 35% 38% 8%
ANCOC 17% 33% 30% 20%
IOBC 0% 36% 19% 45%

Note. ¥°(9) = 27.83, p = .0010

Table C-9

Descriptive Statistics on Self-Ratings of Typing Skill
Soldier Self-Ratings of Typing Skill
Group N M Mdn SD Interquartile Range
OSUT 48 2.60 2.00 0.89 2.0-3.0
BNCOC 48 2.35 2.00 0.89 2.0-3.0
ANCOC 30 2.53 2.50 1.01 2.0-3.0
IOBC 42 3.09 3.00 0.91 2.0-4.0

Note. FG, 164), = 5.18, p = .0018. IOBC mean rating higher than BNCOC, ANCOC, and

OSUT.



CODES

Examples of blocks of
instruction for codes: Lessons

a Beg and exercises.
Lesson A:

Weapon and Unit
Symbols

the Bsa
‘ie or light

‘Examples of Unit Sizes
Lareerent
ms Sempany
eee Been
© saad

D-2



U.S. Army Research Institute
for the Behavioral and Social Sciences

Research Report 1783

Working Memory and Exploration in Training the
Knowledge and Skills Required by
Digital Systems

Jean L. Dyer
U.S. Army Research Institute

Robin S. Salter
Auburn University
Consortium Research Fellows Program

December 2001

Approved for public release; distribution is unlimited.

20020306 132



Mailbox

D-28



Research Report 1783

Working Memory and Exploration in Training the
Knowledge and Skills Required by
Digital Systems

Jean L. Dyer
U.S. Army Research Institute

Robin S. Salter
Auburn University
Consortium Research Fellows Program

Infantry Forces Research Unit
Scott E. Graham, Chief

U.S. Army Research Institute for the Behavioral and Social Sciences
5001 Eisenhower Avenue, Alexandria, Virginia 22333-5600

December 2001
en SS
Army Project Number Personnel Performance and
20262785A790 Training Technology

Approved for public release; distribution is unlimited.

iii



ove fo gid FLORES for supplies and then tothe objective ia
‘next tr strevas wil simulate that movement. Click Next fr.

D-29



The final exam for codes is described in Table 5. As noted in that table, a single item
format was used to test rifle and weapons squad members. The soldier had to identify the
individual’s squad (1st, 2d, 3rd, or weapons) as well as the individual duty position. The item
was scored as being correct only if the soldier answered each part correctly. A “percent correct
score” was given upon completing the code final exam. There was no feedback on individual
test items.

Table 5
Final Exam on Codes

Description of Code Final Exam Items

© The final exam on codes had 25 multiple-choice items. The number of test items for each
code was: Leader — 6; unit — 6; rifle squad — 7; weapon squad — 6. Items differed from
those in the exercises.

¢ A single format was used for testing rifle and weapon squad members. The code (weapon
and BR) for an individual was presented. The soldier had to identify both the individual’s
squad (1st, 2nd, 3rd or weapons) and his duty position in the squad. The first row of
response buttons displayed the squad options. The last three rows of response buttons
displayed the individual positions (both rifle and weapons squad).

¢ For all unit and key leader items, soldiers were presented with a description of the unit or
leader, and then had to select the correct code from a total of 5 options.

For most map exercises (Table 6), there were at least three practical applications of each
function. Soldiers received performance feedback immediately after each response. They were
also informed of the percentage of items they answered correctly at the end of each exercise.
Examples of map lessons, exercises, and final exam questions are in Appendix D.

23



9. Which squad is represented by this battle roster

number?

Glick on the correct answer

Lesson E:
Rifle Squad Battle
Roster Codes

WARRIOR

Rifle: Squad Battle Roster Codes

‘The fifth position is the individual soldier. a

The infantry rifle squad has nine
codes, representing each of tie
‘nine squad member positions.

D-9



is a 5 Letter/Number
Combination:

Sth Position
INDIVIDUAL

4th Position,
SQUAD

ist Position
BATTALION

2d Position

COMPANY

3d Position Each soldier has

PLATOON aposition number
code.

At what unit level do these two battle roster numbers differ?

AA1434
AA1424

Click on the correct answer

D-4



formula, for instructional designers working with computer-based training programs, when faced
with a similar issue of how much information to present in a “lesson” and when it is assumed
that most of the information is unfamiliar to the learner.

We wanted to know how much information could be presented within a training lesson
without overloading the soldier, without giving him too much to remember and process at once,
and therefore creating problems in learning. Within each phase of the experiment, two working
memory demand conditions were generated, based, in part, on the concept of Miller’s number
seven. The Low working memory demand condition (called Low Demand) contained lessons
that typically presented about seven or fewer new concepts or codes to learn. The High working
memory demand condition (called High Demand) had lessons with more than seven concepts or
codes. More details are in the Method section on how we “quantified” the number of chunks.

Four groups of soldiers from Fort Benning, GA, with differing degrees of Army
experience and knowledge, participated in the experiment. The most inexperienced were the
soldiers from Infantry One Station Unit Training (OSUT). These were young soldiers in the
midst of their basic and advanced individual training, who had not been assigned to an Army
unit. Two groups of noncommissioned officers (NCOs) also participated. One group was
enrolled in their Basic Noncommissioned Officer Course (BNCOC); the other in their Advanced
Noncommissioned Officer Course (ANCOC). Lastly, there were soldiers enrolled in the Infantry
Officer Basic Course (IOBC). These soldiers could have been commissioned through the U.S.
Amny Military Academy, through the Reserve Officers’ Training Corps (ROTC), or through
Officer Candidate School (OCS). OCS commissions NCOs as officers.

Working memory is influenced by the meaningfulness of the material (Anderson, 1980).
The more meaningful the material, the greater the memory capacity. The more familiar soldiers
are with weapon symbols, the battle roster system, or the Army’s organizational structure, then
the more meaningful the coding scheme and the fewer demands placed on their working
memory. Of the four soldier populations participating in the research (IOBC, BNCOC, ANCOC,
and OSUT), the OSUT soldiers have the least Army experience. The hierarchical structure of
the Army and even the structure of an infantry rifle squad are not ingrained concepts for those in
OSUT. On the other hand, ANCOC soldiers are senior noncommissioned officers with typically
12 years in the Army. Based on these differences in Army experience, the material was expected
to be most challenging for the OSUT soldiers.

In addition, it was reasonable to expect that some soldier groups would be affected more
strongly than others by the High Demand condition. Aptitude-treatment interactions have long
been of interest to researchers (e.g., Berliner & Cahen, 1973; Corno & Snow, 1986). With
respect to cognitive load and instructional design, Kalyuga, Chandler, and Sweller (1998)
demonstrated that integrated diagram and text presentations of electrical circuits were needed to
reduce cognitive load for inexperienced individuals, but the diagram only format was more
effective for experienced individuals.

Aptitude-treatment interactions were not the primary focus of the experiment. However,
as the soldiers in OSUT had the least military knowledge, it was expected that differences
between the Low and High working memory demand conditions would be greater for them than



Table 13
Factor Analysis (principal components) Results on Background Variables

Component
1 2 3 4
“Computer” “BR” “Military “Work
Knowledge” | Independently”

Eigenvalue 3.78 1.66 1.54 1.01
% Variance 31.52 13.83 12.84 8.34
Factor Loadings

Own Computer 67 24 12 34

Frequency of Use 77 -.04 -.12 35

Self-rate Skill 17 -.14 .06 10

Icon Score 73 -.16 01 -.01

Knack for Computers 80 -.14 -.26 -.05

Use Shortcuts w/ 79 -.10 -31 -.03

Computers

Weapon Symbols Score 33 -.07 7 -.22

Unit Symbols Score 5 .28 53 -.35

Map Reading Score .03 48 AT -.02

BR Use 09 79 -.27 07

BR Training 14 -77 -.26 -.09

Work Independently -.16 .02 44 75

Note. Factor loadings that were at least .70 are bolded and were used to characterize each factor.

The first and main component clearly reflected computer use and perceived skill, from
the computer survey as well as the knack and short cut items, which dealt specifically with a
computer. It was not possible to combine the computer survey indices, due to the different scales
and measurement procedures for each. However, the 10-point rating scales for knack and
shortcut were summed (bivariate correlation between these two ratings was .76, see Table E-23).
Thus it appeared that the knack and shortcut variables reflected computer expertise and
background more than they reflected a general tendency to work things out independently.

The second factor reflected experience with the BR system and these two items were
combined (summed, bivariate correlation was .49). The symbols and map reading scores tended
to load on several factors. However, it was decided to combine the weapon and unit symbol
scores, yielding one score for knowledge of military symbols (bivariate correlation was .42).
The work independently measure appeared to be relatively independent of the other measures,
loading most heavily on the fourth component. In sum, for the regression analysis, the original
12 background variables were reduced to 9 variables listed below.

Computer ownership

Frequency with which soldiers used key computer features
¢Self-rating of computer expertise

47


FIND & STRUCTURE
Ki FINDING UNIES

The Find X fonction i besed on unit organization andthe chain of command,
oing from higher units and commanders to lower ants and leaders. and
sdendual soldiers.

D-16



Table C-13

Weapon Symbols: Percent who Answered Correctly and Percent who Made no Response
‘Weapon
Symbol OSUT BNCOC ANCOC IOBC All Groups

% Soldiers Answering Correctly
M16 Rifle/M4 4% 17% 10% 45% 19%
M240B/M60 4% 15% 10% 24% 13%
‘Anti-tank- 2% 17% 20% 24% 15%
Javelin
M203 0% 2% 3% 31% 9%
Mortar 0% 13% 33% 21% 15%
% Soldiers Not Responding

MI6Rifle/M4 96% 44% 10% 26% 48%
M240B/M60 96% 54% 30% 36% 57%
Anti-tank 98% 69% T3% 62% 76%
Javelin
M203 98% 71% 70% 33% 69%
Mortar 100% 58% 57% 38% 65%

100% minus % Correct and % No Response.

Note. For each soldier group, the percentage of soldiers answered but answered incorrectly is

Table C-14

Weapon Symbols: Frequency Distribution
# Correct OSUT BNCOC ANCOC IOBC

% Soldiers Within Group

0 96% 58% 53% 38%
1 0% 25% 27% 21%
2 2% 15% 13% 10%
3 2% 0% 3% 21%
4 0% 2% 3% TTM%
5 0% 0% 0% 2%
Mean (5 symbols) 0.10 0.62 0.77 1.45
SD 0.51 0.89 1.04 1.48
N 48 48 30 42

Note. FG,164) = 13.13, p = .0000; IOBC mean higher than OSUT, BNCOC, and ANCOC;

ANCOC and BNCOC means higher than OSUT. Because of the nonnormal distribution of
scores, a Chi-square analysis was also conducted. Scores were divided at 0 and greater than

zero. 17(3) = 38.75, p = 0001

C-5



As indicated in Table 10, most soldiers did not know the individual weapon symbols.
The highest weapon score was for the IOBC soldiers; 45% of them correctly identified the
symbol for the M16 rifle. Many soldiers did not even attempt to answer these items (see Table
C-13). The non-response percentages were very high for the OSUT soldiers (at least 96%). The
percentage of BNCOC soldiers who did not respond to these items was lower, but also high,
ranging from 44% to 71%. Even within ANCOC and IOBC, some weapon symbols elicited a
high non-response rate. Groups differed significantly on weapons (see Table C-14). IOBC
soldiers had higher scores than each of the other groups, and BNCOC and ANCOC were higher
than OSUT. Overall, soldiers were most likely to know the symbol for the M16 rifle (19%), and
least likely to know the symbol for the M203 grenade launcher (9%).

With regard to the unit symbols, ANCOC and IOBC soldiers exhibited more knowledge
of these symbols than weapon symbols. The unit symbol scores for OSUT and BNCOC were
low and similar to their weapon symbol scores. As might be expected, there was an inverse
relationship for the groups between non-response rates and unit symbol scores (see Table C-15).
For example, OSUT soldiers had a very high non-response rate (85%) and averaged less than 2%
correct on unit symbols. Again, there were significant differences between groups (see Table C-
16). The order from high to low was ANCOC, IOBC, BNCOC, and lastly OSUT.

Map reading scores were generally high (see Table 10). Although OSUT soldiers scored
the lowest, their average score was 55% correct. IOBC, ANCOC, and BNCOC soldiers each
scored significantly higher than OSUT soldiers (see Tables C-17 and C-18).

With regard to the BR system, only the noncommissioned officers indicated either being
trained on or having used this system. These percentages were lower for BNCOC than ANCOC
soldiers, with 21% of ANCOC soldiers saying they had been trained and 38% stating they had
used the system (see Table C-19). None of the OSUT soldiers had any experience with this
system, and less than 8% of the IOBC soldiers indicated experience. No statistical tests of
significance were computed on BR data because of the disproportionate numbers of soldiers and
OSUT and IOBC without BR experience.

In summary, the soldiers entered the experiment with different levels of military
knowledge. All groups showed good map reading skills, with OSUT the lowest in this area.
Most ANCOC soldiers and about half the IOBC soldiers knew the unit symbols, but OSUT and
BNCOC soldiers did not know these symbols. None of the groups knew all the weapon symbols.
About one-third of the ANCOC soldiers had had experience or training with the battle roster
system, with the experience of the other soldiers much less or non-existent.

Tendency to work independently. Responses to the fifteen tasks on the tendency to work
independently scale were examined to determine if any items should be deleted as not
representing “typical” tasks. A breakdown of responses to each item is in Table C-20. Two
items were eliminated from further analyses because of atypical response patterns. Responses to
the item “take a new, over-the-counter medication” did not vary. More than four-fifths (83%) of
the soldiers indicated they would read the instructions before taking a new medication. Another
item, “fill out tax forms,” generated a unique distribution of responses. One-third (33%)

29



100 100

95 95
90 90
85 85
B80 Pe, Low Demand. 80
2 .
3 N
Oo 75 -l , 75
x . a a High Demand
& 70 Ll 70
=
65 @ | Exploratory | ¢5
60 60
55 55
50 50
Zoom/Pan Find X Display Final Exam

Map Exercises and Final Exam

Figure 14. Mean scores for map exercises and map final exam for High Demand, Low Demand,
and Exploratory conditions. (Scores with significant differences among training conditions are
highlighted.)

100
1OBC
96 BNCOC
OSUT
90
ANCOC
2 «85
3
g
S 80 N,
2 1oBc
5
2 75 .
| ancoc
70
>y
65 BNCOC & OSUT
60
Zoom Find X Display Final Exam

Map Exercises and Map Final Exam

Figure 15. Mean scores on map exercises and final exam for each soldier group. (Scores with
significant soldier group differences are highlighted.)

41



Table E-8
Descriptive Statistics for High and Low Demand Conditions on Time to Complete Code
Instruction and Code Final Exam for Soldiers who did not Repeat an Exercise (in minutes and

fraction of a minute)

Condition | M I SD Mdn l N
Weapon and Unit Symbols

Low Demand 5.98 1.46 5.89 84
High Demand 5.94 1.44 5.76 80
Battle Roster Numbering System

Low Demand 4.85 1.80 4.68 81
High Demand 4.72 1.18 4.68 79
Company through Squad BR

Low Demand 9.29 1.83 9.27 84
High Demand 8.89 2.39 8.68 78
Rifle and Weapons Squads Codes*

Low Demand 16.98 4.00 16.66 78
High Demand 19.34 6.51 18.61 70
Key Leader and Unit Codes

Low Demand 15.06 4.25 14.34 58
High Demand 15.09 4.34 14.59 72
Final Exam on Codes

Low Demand 9.71 2.90 8.61 54
High Demand 9.26 3.55 8.18 60

Note. Final exam times included only those soldiers who did not repeat any exercises.
2 Rifle/Weapons Squads. F(1, 140) = 8.71, p = 0037; Low condition faster than High.



As with codes, the map exercise scores and final exam score correlated significantly with
each other. The six correlations ranged from .31 to .61 (see Table E-19). Correlations among
the map times (Table E-20), computed separately for lessons and exercises, were all significant.
But they were typically higher than the map score intercorrelations (12 of the 16 correlations
were higher than .50).

Of considerable interest was that the times on the map lessons did not correlate with
scores (Table E-21). On the other hand, time spent on the map exercises correlated significantly,
but negatively with map exercise scores. The correlations for Zoom/Pan, Find X, and Display
ranged from -.35 to -.43. In summary, high map exercise scores were indicative of greater speed
on the exercise itself, but did not relate to time spent acquiring knowledge of the map functions.

For the Exploratory map condition, there was a significant but small relationship (r = .26)
between the final exam score and the time spent exploring (Table E-22). There was no
relationship between the map final score and the time to take the final exam.

Scores and background variables. Of particular interest was whether any of the
measures of military knowledge, computer experience, and tendency to work independently
correlated with the two final exam scores. As cited previously, the 12 background variables
were grouped into the three categories listed below.

Computer background from survey
-Computer ownership
-Frequency with which soldiers used key computer features
-Self-rating of expertise
-Icon score
*Military knowledge
-Score on weapon symbols
-Score on unit symbols
-Score on map reading
-Prior use of battle roster system
-Training on the battle roster system
¢Tendency to solve problems on own
-Score on tendency to work independently scale
-Self-rating on tendency to develop computer shortcuts
-Self-rating on tendency to have a “knack” in working with computers

A principal component analysis of these measures was conducted prior to regression
analyses on the exam scores. The purpose of this analysis was to determine if any of the
background variables could be combined for the regression analysis. Factor scores were not
used in the regression analyses, because it was of more interest to examine the contribution of
the “original scores” rather than a weighting of scores within a test battery that could change
with time. The principal components analysis showed four components, accounting for 67% of
the variance (see Table 13), and partially supported the original grouping of measures cited
above. As described below, it was possible to combine some variables (add scores), reducing the
predictors in the regression analysis from 12 to 9.

46



Of interest is the lack of a significant interaction between soldier group and training
condition on the final exam. From high to low within each soldier group, soldiers in the Low
Demand condition had the highest scores, those in the High Demand condition were next, and
those in the Exploratory condition had the lowest scores. Thus the training conditions had the
same relative effect on soldiers from all courses on the map final exam. It should be noted,
however, that for ANCOC’ the relative order of the High and Low Demand conditions was
reversed; but Exploratory scores were the lowest as was the case in the other courses.

Although soldiers were not required to repeat map training if they scored less than 80%
on an exercise, the number of soldiers below this level was tabulated to see if there were
differences between the High and Low Demand training conditions. These results are in Table
12. They support the expectation that performance would decrease when individuals were
required to master a large amount of information within a block of instruction. The High
Demand condition showed the most impact on the display exercise. Information on the display
function was the last to be presented in the single block of instruction for the High Demand
condition®. Thus it was not unexpected that at this point in training soldiers were “overloaded”
with information, which had a negative impact upon performance. In terms of the soldier
groups, OSUT soldiers were more likely to score below 80% on the exercises than the other
groups. On average 33% of OSUT soldiers scored below 80% compared to 11% to 22% for the
other groups. We also examined the percentage of soldiers who scored below 80% on the map
final exam. This percentage was 41% for High Demand, 45% for Low Demand, and 75% for
Exploratory.

Table 12
Number of Soldiers who did not Achieve a Score of 80% Correct on the Map Exercises
Condition Exercise (in order of presentation)
Zoom/Pan/Find Me Find X Display
Low Demand 1 of 43 (2%) 8 of 42 (19%) 11 of 42 (26%)
High Demand 5 of 39 (13%) 9 of 37 (24%) 16 of 36 (44%)

Note. Soldiers in High Demand took these three exercises after receiving all map instruction.
Soldiers in the Low Demand took the exercises immediately after the corresponding lesson.

Times on map exercises and map final exam. Time to complete the lessons, exercises,
and final exam was also examined. A two-way ANOVA (map training condition by soldier
group) was conducted on each block of instruction, the map final exam, map training, and total
time in the map phase (Table E-13). There were main effects but no interactions. Illustrations of
the time results are in Figure 16. Means are in Tables E-14 and E-15.

Several significant differences occurred between the training conditions on map times.
First, the soldiers in the Low Demand condition were significantly faster than those in High
Demand in the zoom/pan block of instruction/exercises, F(1, 74) = 5.34, p = 0235, and in the
time to complete the range/azimuth instruction, F(1, 75) = 29.87, p = .0000.

2 This change in order was not significant, but is pointed out for descriptive purposes.
3 In reality, the last information presented was on the range-azimuth function, but there was no exercise to assess
proficiency on this task.

42



Table C-20

Summary of Item Responses for Tendency to Solve Problems Independently

% Soldiers Responding to Each Alternative

Complete | Instructions Read Find Not
Item or Task With No Only If | Instructions | Someone |Applicable|
Instructions | Needed First Else To Do It

Change a Tire on a New 57% 30% 13% -- —
Vehicle
Find Your Seat at a 33% 41% 21% 1% 4%
Ballgame or Concert
Change Settings on a New 27% 48% 25% —- _
Digital Watch
Cook a Frozen Dinner 24% 15% 58% 1% 1%
Put Up a New Tent 19% 51% 30% -- 1%
Play a New Video or 18% 48% 27% 1% T%
Computer Game
Put Together a New 15% 45% 40% -- 1%
Bicycle
Respond to a Survey 15% 23% 58% 4% 1%
Program a New VCR 11% 38% 48% 1% 1%
Put Together a Piece of
Furniture that Requires 8% 33% 58% 1% -—
Assembly
Drive Unfamiliar Route to
a New Destination More 8% 26% 65% -_ 1%
Than 100 Miles From
Home
Set-up a New Computer TTM% 41% 39% 10% 4%
Take a New, Over-The- 4% A% 83% 1% 8%
Counter Medication
Learn a New Computer 4% 31% 55% 4% 6%
Software Application
Fill Out Tax Forms 3% 11% 49% 33% 4%

Note. N= 168. Items ordered by percent responding to “complete with no instructions.” The
two items/tasks shaded in gray were eliminated from additional analyses (take a new over-the-
counter medication, and fill out tax forms). Row percentages sum to 100%, within rounding

error.



roster number has five positions and the firsttwo
letters represent the battalion code and the second
position is the company code.

sa

{tn this lesson, you
feared about
platoon and

_ squad codes SER

Ag ta Ssaaion, same

Be eu ba20 battalion, Sain Intantry| Company Asx ie 420 Squ2c }
and his bate roster number. so far. is AC132X_

3. Which platoon or section Is represented by this
battle roster number?

Click on the correct answer

Select the battle roster number that represents an
dividual from the following unit.

Individual from the 3d Squad, 1st Platoon of Company B, 24
Battalion, 54th Infantry.

D-8



In the other format, shown at the bottom of Figure 11, the item stem was a verbal description of a
unit; the alternatives were different unit symbol/BR combinations. There were 5 questions in
this format. Three had options where the codes had the identical BR number associated with
different unit symbols. These three questions created problems for all soldiers (answered
correctly by 70% or less within each condition). The item analysis showed the soldiers had the
BR number correct, but did not discriminate between the unit symbols. For example, in Figure
11 the correct BR is AB231 for B Co, 3d Platoon, 1st Squad. However, the unit symbol options
included squad, platoon, and company symbols. Soldiers often incorrectly checked the code
with the company or platoon symbol, vice squad symbols, that also had the correct BR number.

8. Identify the unit represented by this symbol.
Least

Difficult

MEE comparyc

MERE 1st Paton of Company
MER) 15st sound, 1st Platoon, Company A|
|| Company A

HEBER) 1st Pratoon of Company A HT Correct

Click on the correct answer

2. Identify the symbol that represents this unit.

More
14st Squad, 3d Platoon, Company B Difficult

Ap231] [AA11] [AB234]p [AB234]
_~ ie | om \am

Click on the correct answer \

Incorrect
Figure 11. Illustrations of the two question formats in the key leader and unit code exercises.

Correct

The single exercise in the High Demand condition had twice as many items as each
exercise in the two Low Demand conditions. As a result, two soldiers’ overall performance
could have been the same in the High and Low Demand conditions, but one would have passed



Codes: Exercise and Exam Times

Table E-4
ANOVA on Time to Complete Code Instruction and Code Final Exam: High-Low Demand

Conditions by Soldier Group (all soldiers)

Factor | df [ MS | F | Pp
Weapon and Unit Symbols
High-Low Demand 1, 160 0.89 0.43 5140
Soldier Group 3, 160 36.46 17.41 0000
Interaction 3, 160 2.19 1.05 3725
BR Numbering System
High-Low Demand 1, 159 1.06 0.43 5123
Soldier Group 3, 159 3.47 1.41 .2420
Interaction 3, 159 3.76 1.53 .2098
Company through Squad BR
High-Low Demand 1, 159 0.19 0.04 8431
Soldier Group 3, 159 47.74 10.02 0000
Interaction 3, 159 3.14 0.66 5787
Rifle and Weapons Squads Codes
High-Low Demand 1, 159 764.15 16.32 .0001
Soldier Group 3, 159 606.17 12.95 -0000
Interaction 3, 159 117.64 2.51 .0605
Key Leader and Unit Codes
High-Low Demand 1, 159 0.37 0.01 -9055
Soldier Group 3, 159 235.27 8.87 0000
Interaction 3, 159 4.25 0.16 9228
Final Exam on Codes
High-Low Demand 1, 159 9.59 0.99 -3200
Soldier Group 3, 159 19.03 1.97 -1198
Interaction 3, 159 15.69 1.63 1849
E-4



Map: Times on Lessons-Exercises and Final Exam

Table E-13
ANOVA on Time to Complete Map Training: High-Low-Exploratory Conditions by Soldier
Group

Factor | df l MS l F | P
Zoom, Pan, Find Me
High-Low Demand 1,74 29.61 5.34 0235
Soldier Group 3,74 31.72 5.72 .0014
Interaction 3,74 3.14 0.57 6385
Find X
High-Low Demand 1,71 5.85 0.91 3432
Soldier Group 3,71 28.96 4.50 .0059
Interaction 3,71 1.78 0.28 8415
Display
High-Low Demand 1,70 11.54 0.94 3355
Soldier Group 3,70 113.85 9.28 .0000
Interaction 3,70 2.06 0.17 9177
Range and Azimuth (lesson only, no exercises)
High-Low Demand 1,75 29.87 20.14 £0000
Soldier Group 3,75 1.22 0.82 4862
Interaction 3,75 1.05 0.71 5500
Final Exam on Map
High-Low-Exploratory 2, 138 395.46 46.34 0000
Soldier Group 3, 138 16.86 1.97 1205
Interaction 6. 138 3.47 0.41 8738
Map Training

(All lessons and exercises for Low and High Demand, Working with map | for Exploratory)
High-Low Exploratory 2, 143 31,051.71 622.65 .0000
Soldier Group 3, 143 476.51 5.55 .0000
Interaction 6, 143 63.06 1.26 2774
Total Time in Map Phase
High-Low Exploratory 2, 138 22,359.18 296.45 -0000
Soldier Group 3, 138 577.50 7.66 .0000
Interaction 6, 138 43.07 0.57 -7528



Table E-22
Correlations Between Map Scores and Map Times (Exploratory condition only)
Time Exploring Time taking Exam
Final Exam Score .26* 13

Note. N=.79. Correlation between time exploring and time taking exam was .07.

* p<.05
Table E-23
Correlations Among Background Variables
Computer Survey Military Knowledge Work Independently
Own | Freq | Self | Icon | Wpns | Unit | Sym ] Map | BR BR | BR | Knack | Short | K&S | Work

Rate Read | Use | Tm Ind
Own 1.00 | .66 | 42 | 33 | .23 29 [31 | 12 16 | .16 | .19 33 36 37 .00
Freq 1.00 | 45 [49 | .12 20 =| .19 | -.09 | .06 | .06 | .06 55 52 57 7.05
Rate 1.00 | 52 | 30 27 | 33 | .00 =.04 [03 [-.01 | 57 55 60 -.12
Icon 1.00 | .22 32 | 32 .03 -.02 [-.04 | -.03 | 53 1 35 -.05
Wpns 1.00 | 42 | .86 | .13 -.13 | -.09 | -.13 | .09 08 09 -09
Unit 1.00 | 83 | .22 07 | .16 | .12 | .20 16 19 | -.03
Sym 1.00 | .21 04 | .04 | .00 | 17 14 17 03
Map 1.00 |.18 | .11 | .17 | -.06 08 | -.08 12
BR 1.00 | 49 | .81 03 09 06 | -.05
Use
BR 1.00] 91 | .07 09 09 04
Tm
BR 1.00 | .05 -10 .09 -.07
Knack 1.00 -76 93 -AT
Short 1.00 | .94 -.18
K&S 1.00 | -.19
Work 1.00
Ind

Noie. N= 168. Shaded variables were not included in multiple regression. Shaded variables
were replaced by the variable immediately following them in the table.
Correlations greater than .16 were significant at .05 levels; those greater than .31 were significant

at the .01 level.

Table E-24

Correlations Between Tendency to Work Independently and Map Measures (Exploratory

condition only)

Map Final Exam Time Taking Map Exploratory Time
Score Final Exam
Tendency to Work 04 15 -.12
Independently
Note. N=79.

E-18



REPORT DOCUMENTATION PAGE

2. REPORT TYPE
Final

1. REPORT DATE (dd-mm-yy)
December 2001

3, DATES COVERED (from. . . to)
January 2000 — September 2001

4, TITLE AND SUBTITLE
Working Memory and Exploration in Training the
Knowledge and Skills Required by Digital Systems

5a. CONTRACT OR GRANT NUMBER

5b. PROGRAM ELEMENT NUMBER
62785

6, AUTHOR(S)
Jean L. Dyer (U.S. Army Research Institute) and Robin S.
Salter (Auburn University)

5c. PROJECT NUMBER
A790

5d. TASK NUMBER
203

Se. WORK UNIT NUMBER
HOL

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)
USS. Army Research Institute for the Behavioral and Social
Sciences

Infantry Forces Research Unit

P.O. Box 52086

Fort Benning, GA 31995-2086

8. PERFORMING ORGANIZATION REPORT NUMBER

‘9. SPONSORING/MONITORING AGENCY NAME(S) AND

10. MONITOR ACRONYM

ADDRESSES) U.S. Army Research Institute for the Behavioral
and Social Sciences ARI
5001 Eisenhower Avenue 11. MONITOR REPORT NUMBER
Alexandria, VA 22333-5600
Research Report 1783

12, DISTRIBUTION/AVAILABILITY STATEMENT
Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES

14, ABSTRACT (Maximum 200 words): Variations in computer-based training (CBT) procedures were compared in
training the skills and knowledge required of a prototype map interface for the Land Warrior system. Soldiers
from four Infantry courses participated, representing the chain of ‘command within an Infantry platoon, from
platoon leader to rifleman. ‘Soldiers were first trained on codes that uniquely identified individuals and units on
the map. Then soldiers learned how to use map functions such as pan, zoom, determine range, and find
individuals and units. Lessons that contained a large volume of information before soldiers could apply that
information and commit it to memory resulted in low scores on both the code and map exercises. Breaking up the
content into smaller chunks of information tended to be more effective. Although soldiers who learned the map on
their own via an exploratory condition had the lowest map performance, exploratory learning may have potential
as these soldiers spent relatively little time “exploring.” The results demonstrate the importance of adapting to
individual differences in the leaning rate of soldiers. They also provide insights regarding how to design effective
and efficient CBT for digital systems.

15. SUBJECT TERMS Working memory, Exploratory training, Computer-based training, Rifle platoon, Infantry,
Land Warrior, Digital skills, Digital systems, Information overload

x ee ST
2 : ONDE J 19. LIMITATION | 20.NUMBER | 21. RESPONSIBLE PERSON
OF ABSTRACT | OFPAGES (Name and Telephone Number)
18. THIS PAGE 132 Jean L. Dyer
Unclassified | Unlimited (706) 545-5589



MAP DISPLAY:
FINAL EXAM

Preven

Message |".

Maltbox
Timor |

D-26



Section A

COMPUTER SURVEY
Age: Rank:
Years and Months in Army: ___years ____months
Are you returning to the same position you held when you left your unit? Yes____ No_

If Yes, what is that position?
If No, what position are you going to?

4. When did you use computers in your education? (Circle all that apply)
Grade School Jr High High School Technical School College Did Not Use

2. Where do you currently use a computer? (Circle all that appl
Home/barracks/BOQ —_Unit/Work Site _Library/Learning Ctr/Training Facility Do Not Use

3. For each of the following questions, circle the response that best describes you.

a. Do you own a personal computer? Yes No

b. How often do you: (circle how frequently you use each)
eUse a mouse? Daily, Weekly, Monthly, Less Often, Never
Play computer games? Daily, Weekly, Monthly, Less Often, Never
Use icon-based programs/software? Daily, Weekly, Monthly, Less Often, Never
Use programs/software with pull-down menus? Daily, Weekly, Monthly, Less Often, Never
Use graphics/drawing features in software packages? —_ Daily, Weekly, Monthly, Less Often, Never
Use E-mail (at home or at work)? Daily, Weekly, Monthly, Less Often, Never
Use the Internet? Daily, Weekly, Monthly, Less Often, Never

4. Which of the following best describes your typing ability? (check V one)
____ Hunt and peck slowly
_____ Hunt and peck quickly
___ Type slowly while not looking at the keyboard
___ Type quickly while not looking at the keyboard

5. Which of the following best describes your expertise with computers? (check v one)

Novice

Good with one type of software package (such as word processing or work calendars or slides)
Good with several software packages

Can program in one language and use several software packages

Can program in several languages and use several software packages

Expert — Bill Gates would hire me

If you are good with one or more software packages, please list them.

If you can program in one or more languages, please name these languages.



Correlations among Measures

Scores and times. The relationship among the exercise and final exam scores and times
to complete the exercises and training was a primary question for both the code and map
training. The correlation between the two final exams, for all participants, was .23 (p < .001),
significant, but not substantial.

In summary, for both the code and map phases of training, the typical relationships
among scores and times were as follows:
e Exercise and final exam scores correlated
e Times correlated with each other
e Scores correlated negatively with times (higher scores were associated with faster
times).
Specific results for the code and map phases of training are discussed in turn.

Code correlations. Product moment correlation coefficients among all the code scores
(exercises and final exam) were calculated, a total of 15. Nine were significant (Table E-16).
The highest correlations were between rifle/weapons squads and leader/unit, and between
leader/unit and the exam, each r = .43). Other significant correlations ranged from .15 to .37.
One exercise score, the initial symbol exercise (weapon and unit symbols), did not correlate with
any other exercises nor with the final exam. This could have been because soldiers were given
immediate feedback on their answers and had an opportunity to learn the symbols as a function
of the feedback during this exercise. Each symbol was presented three times, so corrective
feedback could have improved soldier performance during the exercise itself.

Correlations among the times to complete the code blocks of instruction (lessons and
exercises) and the code final exam were also computed, a total of 15. Each correlation was
significant (Table E-17). In addition, the times for each of these blocks of code training were
correlated to the total code time. These correlations were also significant. Of the 21
intercorrelations that were computed, 12 were at least .50, with the highest being .86
(rifle/weapons squad time with final exam time). In general, the magnitude of the correlations
among the code time measures was stronger than among the code scores.

For code training, separate lesson and exercise times were not available. Only the total
time to complete each lesson and exercise combination was recorded in the database. For each
block of instruction, the correlation between score and time was both negative and significant,
meaning higher scores were associated with faster times (see correlations on the diagonal in
Table E-18). These correlations were considered moderate, with four of the five correlations
ranging from -.32 to -.46. The correlation between the code final exam score and code final
exam time was -.15, also significant but small.

Map correlations. For the map scores and times, the Low and High Demand conditions
were examined separately from the Exploratory condition, as the Exploratory condition did not
have lessons or exercises. In contrast with the code training, separate times for the map lessons
and exercises were available.

45



By clicking on the Lower ‘selection you have activated anct
‘aubenent, which we cal

ons wil show the platoon symbol and bate racer
roche in the company you tected, In his case és A

D-21



z D e
° i
° r s 9
m P A
0 P 7
fel] + (el fe"
a x E—E y —
Low Demand a —>
z
° ei}? ] ek
° i s 9
m
a Pp A
P a ' z
a x a m
a ‘ [|
High Demand °

Exploratory

Figure 5. Instructional sequence for map training conditions. [The E refers to an exercise.]

No remediation was provided with the Low and High Demand map conditions. If
soldiers scored less than 80%, they proceeded to the next lesson or exercise. Soldiers were told,
however, that the desired score was at least 80%. There were two reasons for no remediation.
First, there was no requirement for soldiers to learn the map functions as a prerequisite for
performing a later task, as was the case with symbols and BR. Knowledge of symbols and BRs
was essential to successfully completing the map tasks. Second, experimental time was limited.
There was a high probability that some soldiers would not have completed the experiment if
remediation had been required. The exercise scores were examined, however, for the numbers of
soldiers who achieved the 80% desired criterion.

For the Exploratory map condition, soldiers were given minimal guidance and
information. They were informed of the seven functions they had to learn through exploration,
shown the location of each button on the map, and told the purpose of each button. There was no
instruction on how these functions or the map display worked. They were given a set of tasks
they could try to execute during their exploration, but were not required to do these tasks. They
had 60 minutes to work with the map. However, they could progress to the final exam whenever
they felt confident they knew how each button/function worked. The soldiers in this condition
never had any external feedback on whether they fully understood each function. They used
their own judgment on when to proceed to the final exam.

19


c Pit R w L uv
° $d i e e n
f a a i
B B ' P a t
RE Re hkl erie Hlew po
E E el[n [e]} r fe E

e@-or3Kn

E High Demand
[E | = Exercise

L
c | pit r| Ww e| vu
o | sd ije ajo
a a} i
B B 1 Pp e t
R R e ° r
n

EJ EJ EJ

Figure 3. Instructional sequence for code training conditions.

Lesson C — Co BR (5 chunks). The five numbers used to identify companies within a
maneuver battalion.

Lesson D- Pit Sd BR (8 chunks). The four numbers used to identify platoons were presented
first, followed by the four numbers used to identify squads.

Lesson E - Rifle (9 chunks). The numbers assigned to individuals within a rifle squad (0-8)
were presented and defined. These individuals were divided by fire team as well. At this
point, the weapon symbol was combined with the BR number, as shown in Figure 2
previously. Soldiers were able to self-test themselves with a sample of codes before
proceeding to the exercise.

Lesson F — Weapon (9 chunks). The numbers assigned to individuals within the weapons
squad (0-8) were presented and defined. The weapon symbol was combined with the BR

number, as shown in Figure 2. Soldiers were able to self-test themselves with a sample of
codes before proceeding to the exercise.

Lesson G ~ Leader (6 chunks). The codes for three key leaders at both the company and
platoon levels were presented. These were the company commander, executive officer, first

15



IARRIOR

4 The range azimuth mode wil help you determine a direction and distance to,

Stee pinta asian pea

‘You are the Fal sder of A Company and you want whniow how far
‘and in what dieection yout fest squad leader located.

Range and Azimuth Mode
Review

1. Cliek on the “Range Azimuth” button, The range ezimuth capabilty is
now functional and the button changes to "Cancel".

2. Click on a start point and then an end point. The range and azinnuth will
‘appear in meters and degrees. At this point, the button also
automatically changes back to "Range Azimuth".

3. Individuals: measure fromuto the base stem

4, Units: measure fromto the lower left comer of the symbol.

5. Can only measure one cistance at a time.

6. To determine another range azimuth, click on the “Range Azimuth*
button again.

{for teszon #. The nal exam, however, wl have its
atin vil be wed.

D-25



Both subjective and objective indices of computer skill were derived from the survey.
The subjective index was the soldier’s own rating of his skill on a six-point scale from “novice”
to “Bill Gates would hire me.” The other index was based on a quiz that required soldiers to
identify 18 icons commonly used in Windows-based software. Significant differences were
found among the groups on both indices; self-rating, F(3, 164) = 7.38, p = .0001; icon quiz, F(3,
164) = 7.89, p = .0001. For both indices, the IOBC soldiers were higher than soldiers in each of
the other courses. In addition, on the icon quiz, IOBC scores were higher than BNCOC scores.
Table 9 presents summary descriptive statistics on each index. Additional information is in

Appendix C.

Table 9

Means on Self-ratings of Computer Skill and on Icon Scores
Soldier Self Rating Icon Score
Group N M SD M SD
OSUT 48 2.04 0.97 8.92 3.38
BNCOC 48 1.89 0.91 7.97 3.49
ANCOC 30 2.00 0.91 9.53 3,12
IOBC 43 2.76 1.01 11.17 2.58

At the group level, there was agreement between the objective and subjective of indices
of computer skill. From high to low, the group order was similar: IOBC, ANCOC and OSUT,
and then BNCOC.! To illustrate the relationship between the objective and subjective indices at
the group level, consider the IOBC group, which was highest on both indices, and the BNCOC
group, which was lowest on both indices. For IOBC soldiers 20% indicated they could program
in a language such as C++ or Visual Basic, and only 12% rated themselves as computer novices.
In addition, 62% of the IOBC soldiers correctly answered over half the items on the icon test.
On the other hand, 44% of soldiers in BNCOC rated themselves as novices and only 2%
indicated they could program. Only 33% of the BNCOC soldiers correctly answered over half
the items on the icon test.

As asummary, Figure 6 graphically illustrates how the soldier groups ordered on four
major variables from the computer survey: percent soldiers using a computer, percent soldiers
owning a computer, percent soldiers indicating they were not computer novices, and the mean
percent correct on the icon test. IOBC soldiers were the highest on each dimension. Of interest
is that even though OSUT soldiers had the lowest usage and ownership rates, they were similar
to BNCOC and ANCOC with respect to self-ratings and icon scores. The graph also illustrates
that high rates of soldiers owning and using a computer within a group did not necessarily
translate to high self-ratings for the group as a whole.

' This statement should not be interpreted to mean that each group differed significantly from each other group on
each of these indices.

27



Table E-6
Descriptive Statistics for Soldier Groups on Time to Complete Code Instruction and Code Final

Exam (in minutes and fraction of a minute, for all soldiers)

Soldier Group | M [SD | Man | Min-Max | N
Weapon and Unit Symbols *

IOBC 5.03 1.28 4.87 2.02- 8.59 42
ANCOC 5.68 1.69 5.27 3.78 -12.58 30
BNCOC 6.18 1.19 5.94 4.28 -11.77 48
OSUT 7:18 1.63 7.31 2.52 -11.18 48
BR Numbering System

IOBC 4.45 2.28 4.05 2.22-16.42 4]
ANCOC 5.10 1.46 4.81 2.93-9.37 30
BNCOC 4.92 1.17 4.98 2.33-7.32 48
OSUT 5.06 1.22 4.90 1.56 -8.57 48
Company through Squad BR°

IOBC 7.89 1.56 7.73 5.97-13.70 4l
ANCOC 9.03 1.98 8.92 6.10-13.63 30
BNCOC 9.71 2.57 9.25 6.17-20.18 48
OSUT 10.36 2.28 10.16 3.50-17.37 48
Rifle and Weapons Squads Codes ©

IOBC 16.01 3.44 15.35 10.67-27.32 4]
ANCOC 17.17 6.07 15.82 10.28-37.60 30
BNCOC 19.47 6.05 18.53 11.07-42.77 48
OSUT 24.46 10.72 22.33 5.82-67.40 48
Key Leader and Unit Codes *

IOBC 13.74 4.63 12.38 8.07-25.23 41
ANCOC 15.79 5.14 16.21 8.57-24.68 30
BNCOC 16.36 3.72 16.54 10.17-28.22 48
OSUT 19.32 6.45 18.30 5.13-48.03 48
Final Exam on Codes

IOBC 9.56 4.04 8.28 5.52-26.17 4]
ANCOC 8.60 1.58 8.15 6.23-12.80 30
BNCOC 9.97 3.05 9.44 5.56-23.72 48
OSUT 10.30 3.04 9.83 4.60-17.53 48

Weapon/Unit Symbols. F(3, 160) = 17.41, p = 0000; IOBC, ANCOC, and BNCOC each

faster than OSUT. IOBC faster than BNCOC.
> Company-Squad. F(3, 159), = 10.02, p = .0000; IOBC and ANCOC faster than OSUT.

IOBC faster than ANCOC and BNCOC.
Rifle/Weapons Squads. F(3, 159), = 12.95, p = .0000; IOBC, ANCOC, and BNCOC faster

than OSUT; IOBC faster than BNCOC.
Key Leader/Unit. F(3, 159), = 8.87, p = .0000; IOBC, ANCOC, and BNCOC faster than

OSUT; IOBC faster than BNCOC.

°

o



effective in acquiring the interactive skills and insights required to work with digital interfaces
typical of Army systems. Some form of guided exploratory learning could also be a good way of
adapting to individual differences in groups of soldiers with diverse military backgrounds.

Not unexpectedly, you can include too much information in a block of instruction. The
concept of the number seven can be used as a rough guide to determine when too much new
information may be presented. And it should not be viewed as a formula for instructional design. ,
As pointed out by Ausubel (1968), blocks of instruction should not be made small just to reduce
the error rate. They should not fragment content, ignore the need to consolidate and integrate
information, or fail to provide individuals with the logic of the body of content that is being
taught. In addition, chunks of meaningful material can easily vary with the background of the
learner. Thus large bodies of material may be easily comprehended by some individuals when
familiar information is included, but be overwhelming for others.

The experiments showed how the computer-based training could be designed to
incorporate military tactical system software as background instruction and demonstration
screens, and as interactive screens for performance exercises. High-fidelity training is a positive
by-product of this technique.

Although the multi-media instruction was effective, the medium per se is not an
automatic panacea for training the digital skills required of soldiers and leaders who must use the
tactical software embedded in new equipment. Measurement techniques and procedures are
needed that account for the multiple, yet valid approaches a soldier can use to accomplish a task.
In addition, challenges lie in developing problem-solving scenarios that require soldiers to go
beyond demonstrating technical skill proficiency to demonstrating effective employment of
digital skills in combat-like settings.

51


Table E-3
Descriptive Statistics for Soldier Group on the Code Exercises and Code Final Exam Scores
(% Correct)

Soldier Group l M [SD | Mdn | Min-Max | N
Weapon and Unit Symbols

IOBC 97.32 3.99 100 83-100 42
ANCOC 97.08 4.66 100 79-100 30
BNCOC 97.39 6.35 100 63-100 48
OSUT 96.96 5.07 100 71-100 48
BR Numbering System

IOBC 97.56 5.38 100 80-100 41
ANCOC 93.00 16.43 100 20-100 30
BNCOC 95.00 8.25 100 70-100 48
OSUT 94.79 9.45 100 60-100 48
Company through Squad BR

IOBC 98.69 2.22 100 93-100 41
ANCOC 96.44 6.37 97 67-100 30
BNCOC 95.97 11.11 100 30-100 48
OSUT 95.90 6.78 98 63-100 48
Rifle and Weapons Squads Codes *

IOBC 92.33 12.09 97 31-100 41
ANCOC 91.52 10.44 94 57-100 30
BNCOC 88.51 11.65 91 40-100 48
OSUT 81.67 18.13 88 34-100 48
Key Leader and Unit Codes”

IOBC 92.77 8.54 96 71-100 41
ANCOC 91.53 7.70 92 71-100 30
BNCOC 90.54 9.44 92 67-100 48
OSUT 85.16 12.18 88 38-100 48
Final Exam on Codes ©

OBC 92.68 9.33 96 48-100 41
ANCOC 91.20 10.10 96 60-100 30
BNCOC 87.92 12.25 92 44-100 48
OSUT 82.92 11.95 84 56-100 48

Rifle/Weapons Squads. F(3, 159) = 6.26. p = .0004; ANCOC, BNCOC, and IOBC greater
than OSUT.
b Key Leaders/Unit. F(3, 159) = 5.51, p = .0012; ANCOC, BNCOC, and IOBC greater than
OSUT.
© Final Exam. F(3, 159) = 6.49, p = .0003; ANCOC, BNCOC, and IOBC greater than OSUT;
IOBC greater than BNCOC.

E-3



Table C-15

Unit Symbols: Percent who Answered Correctly and Percent who Made no Response
Unit Symbol OSUT BNCOC ANCOC IOBC All Groups

% Soldiers Answering Correctly
Squad 2% 27% 79% 65% 39%
Platoon 0% 13% 67% 55% 29%
Company 2% 4% BB% 43% 26%
% Soldiers not Responding

Squad 88% 42% TMM% 12% 41%
Platoon 85% 38% 0% 12% 38%
Company 83% 44% 0% 24% 42%

Note. For each soldier group, the percentage who answered, but answered incorrectly is 100%
minus the percentage correct plus the percentage of no responses.

Table C-16

Unit Symbols: Frequency Distribution
# Correct OSUT BNCOC ANCOC IOBC

% Soldiers Within Group

0 97% 69% 13% 33%
1 0% 21% 10% 12%
2 2% 8% 20% 14%
3 0% 2% 57% 41%
Mean (3 symbols) 0.04 0.44 2.20 1.62
SD 0.29 0.74 1.09 1.32
N 48 48 30 42

Because of the nonnormal distribution of scores, a Chi-square analysis was also conducted.

Scores were divided at 0 and greater than zero. 7G) = 73.22, p= .0001. It appeared that each
group differed from each other group, based on the difference between the observed and

expected frequencies.

Table C-17

Map Reading: Percent Soldiers who Answered Correctly

Item OSUT BNCOC ANCOC IOBC All Groups
8-digit grid #1 33% 96% 93% 78% 73%
8-digit grid #2 29% 94% 97% 71% 70%
Terrain feature 40% 75% 87% 11% 66%
Hard-surface road 81% 81% 67% 66% 75%
Distance in meters 60% 71% 67% 90% TA%
Grid azimuth 79% 94% 97% 95% 91%



The first three points on the scale were felt to be indicators of a tendency to work
independently, that is, performing the task without instructions or guidance; proceeding at first
without instructions, but referring to them if there was a problem; and lastly reading instructions
before beginning the activity. Soldiers were also given the option of indicating that they would
find someone else to perform a task for them, and they could provide a rating of “not applicable”
for tasks that they had never performed. The tendency to work independently items are at
Appendix B.

In addition to assessing tendencies toward working independently, the survey also asked
soldiers to provide self-ratings of their ability to navigate through a computer program (computer
knack) and their ability to identify efficient methods (short cuts) for performing computer tasks.
For each ability, soldiers rated themselves on a 10-point scale ranging from “does not describe
me at all” (1) to “describes me completely” (10).

Criterion measures. Criterion measures were scores on the practical exercises embedded
in the code and map training, final exam scores on the code and map, number of soldiers who did
not reach the 80% criterion on the exercises, and time required to complete exercises and
training. The composition of these exercises and exams is in Tables 4 through 7.

All code exercises were in a multiple-choice format, with 60 seconds allowed to respond.
Prior to each coding exercise and the code final exam, soldiers were shown the type of item(s)
they would encounter. They were also told how many exercise items they had to answer
correctly, corresponding to 80%, before progressing to the next lesson. If they did not achieve
this level, they went through a review and retook the exercise. The reviews were specially
developed to highlight the critical parts of each lesson. They were not simply a repetition of the
lesson, as is often done with computer-based training.

With the first weapon and unit symbols exercise, correct/incorrect feedback was provided
immediately after answering each question. When soldiers answered incorrectly, the feedback
also provided the correct response. In addition, upon completing the exercises, soldiers were
informed of the percentage of items they answered correctly. With all other coding exercises,
feedback was provided only at the end of the exercise. In this case, all items answered
incorrectly were displayed, and the correct answer was shown. Again, soldiers were informed of
the percentage of items they answered correctly. On the final exam, soldiers were only informed
of the percentage of questions answered correctly. Examples of exercise and final exam
questions on the coding system are in Appendix D.

21



Message

sf Find Mots a on
iB splay area on you.

se pee f
froma

{ Zoom out uot you have located the squad leader. 11 Sa
Company A. Once you have his symbol on your map disp
‘pymibol and click NEXT to op the tine,



coding scheme that would uniquely and graphically identify individual soldiers, as such a
scheme does not currently exist within the Army.

An early prototype of the map display and layout proposed for the LW system was
simulated, Some modifications were made to the prototype for purposes of the experiment, but
basic functions such as panning and zooming, displaying individuals, and determining distances
on the map were retained, and are similar to the most recent version of the system software. The
experiment provided insights into the particular interface that was simulated. More important, it
allowed an examination of the generic training issues mentioned previously. Participants were
soldiers with differing Army backgrounds and experience.

Purpose

The training experiment had several purposes that have general applicability to many
Army digital systems. Participants in the experiment represented the soldiers within an Infantry
rifle platoon (young soldiers, noncommissioned officers, and lieutenants). Thus the research
provided information on how soldiers from these different populations, who vary considerably in
Army experience and training, would learn and apply the concepts taught about a digital map.
Would one group of soldiers learn the information and procedures faster and to a higher level of
proficiency than the other groups?

Different computer-based training procedures were also compared. In reality, many
instructional applications are characterized by intense production schedules and limited resources
(Lohr, 2000), leaving little or no opportunity for research on comparing the effectiveness of
training alternatives. The research reported here examined the impact of the demands placed on
working memory, and the impact of an exploratory mode of training. Soldiers were faced with
two types of knowledge requirements. One was learning the unique symbols that represented
individuals and units on the map. The second was learning how to manipulate the map interface
to solve tactical problems representative of those required in the field. With both types of
knowledge requirements, the amount of information presented before soldiers had an opportunity
to apply it was varied. In other words, some soldiers were confronted with large amounts of
information prior to any practical exercises (high demands on working memory). For others, the
same information was divided into smaller chunks of information prior to practical exercises
(low demands on working memory). How much information could be presented within a
training session without overloading the soldiers?

With the map interface, some soldiers used an exploratory mode of learning the map
functions. Basically, they had a period of time to determine how each function worked, without
any exercises or feedback on whether they had learned all aspects of each function. Would
soldiers learn the map functions by working on their own as well as soldiers given formal
instruction?

A corollary to the above issues was whether the different soldier populations participating
in the research would benefit more from one training approach than another. If that were the
case, then training on digital systems should be designed to adapt to these differences.



Table C-18

Map Reading: Frequency Distribution

# Correct OSUT BNCOC ANCOC IOBC
% Soldiers Within Group

0 8% 0% 0% 0%

1 8% 0% 0% 0%

2 17% 0% 0% 2%

3 23% 2% 3% 17%

4 17% 19% 20% 19%

5 17% 40% 43% 33%

6 10% 40% 33% 29%

Mean (6 items) 3.29 5.17 5.07 4.69

sD 1.74 0.81 0.83 1.14

N 48 48 30 42
FG,164) = 24.43, p = .0000. IOBC, ANCOC, and BNCOC means each greater than OSUT
mean.
Table C-19
Experience with the Battle Roster System: % Soldiers

OSUT BNCOC ANCOC IOBC All Groups
Trained on Battle 0% 13% 21% 5% 9%
Roster System
Used Battle Roster 0% 31% 38% T% 18%
System
C-7



3, What does this symbol represent?

6. What does this symbol represent?

IP sezeoes: (203 Grenade | ‘Sarvetier:
moe Ee

newer

Ss
Lesson B:
Battle Roster
Numbering System



WORKING MEMORY AND EXPLORATION IN TRAINING THE
KNOWLEDGE AND SKILLS REQUIRED BY DIGITAL SYSTEMS

Contents
Page
[ntrOdUctiONn .......csssssssscsesesssssseesesesnessencenentenseesscesnssesaensssenesscsnsacsscesessneneneeseestaes 1
PULpOSe .......ee0e . 2
Types of Knowledge 3
Demands on Working or Short-term Memory.. 3
Exploratory Modes of Learning and Training — Problem-solving .. 6
Individual Differences in Exploratory Tendency. 7
Computer-Based Training . 8
Method .. 9
Participant 9
Procedure .. 10
Experimental Design ... 11
Measures ........:seee0+ 20
ReSUILS ..csseccccscsssesessocesssesssscsessesessvsscsecsesecsesecsesscssenessesssssecnssnsnesscnennenssscnsacecsanenenseneseeeneeeee 26
Pre-experiment Measures ... 26
Code Training Results 31
Map Training Results .. . 39
Correlations among Measures . 45

Discussion and Conclusions

References .....
Appendixes

Appendix

A. Symbols and Battle Roster System..
B. Pre-experiment Instruments .
C. Soldier Background Results.
D. Examples of Lessons, Exercises, and Exams.
E. Results on Code and Map Training



Table E-2
Descriptive Statistics for High and Low Demand Conditions on the Code Exercises and Code

Final Exam Scores (% Correct)

Condition [ M SD | Mdn | Min-Max | N
Weapon and Unit Symbols

Low Demand 97.70 4.87 100 63-100 85
High Demand 96.69 5.36 100 71-100 83
BR Numbering System

Low Demand 94.64 11.24 100 20-100 84
High Demand 95.78 8.71 100 60-100 83
Company through Squad BR

Low Demand 97.65 3.38 100 87-100 84
High Demand 95.74 10.15 100 30-100 83
Rifle and Weapons Squads Codes*

Low Demand 93.09 7.07 94 60-100 84
High Demand 82.89 17.60 88 31-100 83
Key Leader and Unit Codes

Low Demand 89.73 9.09 92 54-100 84
High Demand 89.71 10.58 92 38-100 83
Final Exam on Codes

Low Demand 87.19 12.00 92 48-100 84
High Demand 89.30 11.31 92 40-100 83

Rifle/Weapons Squads. F(I, 159) = 23.53, p = .0000; Low scores greater than High.

E-2


Appendix D

Examples of Lessons, Exercises, and Exams

Disclaimer.
The border on the displays contains the words “Land Warrior.”
The symbols and map interface DO NOT correspond to the current version
of the Land Warrior system.
The experiment was based on a prototype version of the Land Warrior map.
For purposes of the experiment,
we generated our own detailed version of how the map might function
and the symbols used to identify individuals and units.

D-1



display to chow al squad leaders fom the 3d Platoon, Company
e the platoon leadot and platoon sergeant for the 3d Platoon.
PSG

The platoon leader 24 Platoon, A Company. Dipl
ikon 24 Squad,

D-27



Examples of additional lessons and final exam questions
are not provided. The exam questions followed the
formats illustrated. However, there was no repetition of
specific items given in the training exercises.

MAP LESSONS,
EXERCISES, AND EXAM.

Panning and Locate

D-12



Corno, L., & Snow, R. E. (1986). Adapting teaching to individual differences among learners.
In M. C. Wittrock (Ed.), Handbook of research on teaching (3rd ed., pp. 605-629). New
York: MacMillan.

Deatz. R. C., & Campbell, C. H. (2001). Application of cognitive principles in distributed
computer-based training (ARI Research Product 2001-03). Alexandria, VA: U.S. Army
Research Institute for the Behavioral and Social Sciences.

DeMul, S., & Van Oostendorp, H. (1996). Learning user interfaces by exploration. Acta
Psychologica, 91, 325-344.

Department of the Army. (1994, 9 September). Personnel doctrine (FM 12-6). Washington,
Dc: Author.

Department of the Army. (1997, 30 September). Operational terms and graphics (FM 101-5-1,
MCRP 5-2A). Washington, DC: Author.

Dyer, J. L. & Martin G. H. (1999). The computer background of Infantrymen: FY99 (ARI
Research Report 1751). Alexandria, VA: U.S. Army Research Institute for the
Behavioral and Social Sciences. (DTIC No. AD-A372 716)

Dyer, J. L., Westergren, A. J., Shorter, G. W., & Brown, L. J. (1997). Combat vehicle training
with thermal imagery (ARI Technical Report 1074). Alexandria: VA: U.S. Army
Research Institute for the Behavioral and Social Sciences. (DTIC No. AD-A342 559)

Egan, D. E., & Greeno, J. G. (1973). Acquiring cognitive structures by discovery and rule
learning. Journal of Educational Psychology, 64, 85-97.

Fletcher, J.D. (2000, August). Defense training and the second revolution in learning. Paper
presented at the Conference on Distance Teaching and Learning, Madison, WI.

Fober, G. W., Bredthauer, J. L., & Dyer, J. L. (2000). The computer background of soldiers in
Infantry courses: FY99-00 (ARI Research Report 1762). Alexandria, VA: U.S. Army
Research Institute for the Behavioral and Social Sciences. (DTIC No. AD A381 507)

General Accounting Office. (2000, July). Battlefield automation: Army needs to update
fielding plan for first digitized corps. Washington, DC: Author.

Gibbons, A. S., & Fairweather, P. G. (2000). Computer-based instruction. In S. Tobias & J. D.
Fletcher (Eds.), Training and retraining: A handbook for business, industry, government,
and the military (pp.410-442). New York: Macmillan Reference.

Goodman, G. W. (1999, October). Revolutionary soldiers. Armed Forces Journal
International, 56-64.

54



exception of the code exam score for Low Demand, some measure of computer expertise or
background was in the stepwise regression equations. With the exception of the map exam for
Low Demand, the score on symbols or map reading (military knowledge) was in the stepwise
regression. Battle roster experience never correlated highly, perhaps because so few soldiers had

experience with the BR system.

Discussion and Conclusions

Despite substantial differences in military knowledge and experience among the soldier
groups as well as the minimal knowledge in some areas by all groups, neither these differences
nor the limited knowledge appeared to greatly hinder performance. For instance, soldiers
learned the weapon and unit symbols even though many did not attempt to answer these
questions on the initial military knowledge test as they had never been exposed to this material in

their military training.

For both the code and map training, the blocks of instruction that had the greatest
differences in working memory demand were the ones that resulted in differences in
performance. For the coding training, this was the rifle and weapons squads block(s) of
instruction. In the map training, this difference was ultimately reflected in performance on the
Display exercises. In each instance, those soldiers in the Low Demand condition performed

better.

Another factor that indicates the Low Demand condition was more effective than High
Demand was the variability in scores. We checked for conditions where the variability of scores
for a given group was at least 1.5 times greater than another. For both the company-squad BR
and rifle-weapons squad code exercises, the scores for the High Demand condition were at least
2.5 times more variable than those for the Low Demand condition. Thus not only did the Low
working demand condition result in a higher level of performance than the High condition, it also
resulted in more consistent scores.

Those individuals using the unguided exploratory means of learning the map performed
more poorly than those with more formal instruction. However, the time devoted to exploratory
learning was 70% less than that in the Low and High Demand conditions. Exploratory times
were at least twice as variable as those in the Low and High Demand conditions, and the time
required to complete the map final exam was longer. Both findings were additional indications
of lower proficiency. In summary, exploratory learning, with minimal guidance, was not the
most effective mode of digital map training for the diversity of soldiers in the research - the
soldiers to receive the Land Warrior system. Although the tendency to work independently scale
did seem to identify soldiers who tended to work without guidance, it did not necessarily identify
those who were good at employing the appropriate, information-seeking strategies on their own.

Differences in scores and times for the four soldier groups appeared on many measures.
Typically, IOBC soldiers scored the highest, then ANCOC, then BNCOC, and lastly OSUT.
That OSUT soldiers scored lowest was not unexpected, given their limited military experience
and knowledge. Times also varied with soldier group, although the pattern was not as consistent

49



Figure 1. Multi-media classrooms where the experiment was conducted.

Experimental Design

The experimental design is summarized in Table 1. Phase I was individual and unit code
training. Phase II was map interface training. The same information and concepts were taught
and applied in the High and Low Demand conditions. The only difference in the High and Low
Demand conditions was how this information was partitioned across the lessons and exercises.
The content of the lessons and exercises was identical. Table 1 also shows that the Exploratory
condition was only in the Map phase.

Table 1
Experimental Design for the Code and Map Training Conditions
Experimental Conditions
Low Demand Code | High Demand Code & | Low Demand Code & | High Demand Code &
& Low Demand Map | High Demand Map Exploratory Map Exploratory Map
SB & 8 SB
Code Lessons and Exercises: Phase I
Two common blocks of instruction
Six blocks of Three blocks of Six blocks of Three blocks of
instruction instruction instruction instruction
Final Exam on Codes
Map Lessons and Exercises: Phase II
Four blocks of One block of Brief explanation of | Brief explanation of
instruction instruction each function; no each function; no
instruction instruction
Final Exam on Map Functions

Note. Each block of instruction concluded with practice exercises. There were no exercises for
the instruction on range and azimuth map functions.

11


corresponding instruction. Those in High Demand received all three blocks of instruction and
then had all the exercises.

Scores on map exercises and map final exam. A two-way analysis of variance (training
condition by soldier group) was conducted on each exercise score and the map final exam score
(see Table E-10). There were main effects, but no significant interactions. There were two
significant effects for training condition, and two effects for soldier group. Means for each
training condition are in Table E-11. The means reflect the first attempt at the exercise as no
repeats were allowed because of time constraints. Final exam scores include the Exploratory
condition.

Due to time constraints in conducting the experiment, not all soldiers were able to
complete the map training. For Low Demand, three soldiers did not finish the training. For High
Demand, five soldiers did not finish the training. The actual numbers completing each exercise
and the map final exam are shown in Table E-11.

Significant differences occurred for training condition on the display exercise and the
map final exam, F(1, 70) = 4.07, p = .0474 and F(2, 142) = 6.84, p = .0014, respectively. These
are shown graphically in Figure 14. On the display exercise, soldiers in the Low Demand
condition scored higher than those in the High Demand condition. On the final exam, clearly,
soldiers in the Exploratory condition had the lowest scores, although the difference between Low
and High Demand conditions was also significant. The fact that no remedial training was
provided for the Low and High Demand conditions (vice the symbols training) may have
affected scores for soldiers in these conditions on the final exam. The overall downward trend of
scores, shown in Figures 14 and 15, may simply represent the cumulative effect of working at a
computer for several hours learning new information.

Means for the four soldier groups on the three map exercises and map final exam are in
Figure 15 and Table E-12. The map exercise means are based on only the Low and High
Demand conditions, while the final exam mean includes the Exploratory condition as well.
Significant differences among soldier groups occurred on the find X exercise [F(3, 71) = 2.86, p
= 0431], and the final exam [F(3, 142) = 3.29, p = 0226]. With find X, BNCOC did better than
ANCOC and OSUT. On the final exam, IOBC did better than BNCOC and OSUT. Thus, on
each measure, OSUT scored lower than the top group.

On the display exercise, the difference among courses approached significance, F(3, 70)

= 2.52, p= .0655. There was considerable variability in scores in BNCOC and ANCOC, which
could have contributed to the lack of an effect at p < .05.

40



The map training times show that
Exploratory soldiers spent 12 minutes
working with the map during training,
= significantly less time than the 54
minutes for those in High and Low
Demand. However, those in High and
Low conditions spent significantly less

High Demand

Low Demand “OW ¢
time in the map final (15 vs. 22
minutes).
6 15 3 rr Cy 75
Map Cumulative Times in Minutes
‘edn an Fi Esa Tine fr i ap Condens
100 7 7 ~
Bers —T_ Distribution of total map times by training

sof | Ea 25% 75%
o_Mesan Tene

condition shows where the middle 50% of
= the times were concentrated for each
condition, as well as the extreme times. The
° —_—— minimal overlap between Exploratory times
and High and Low Demand times is clear.

Total Map Time in Minutes

Total Time in Map

‘Exploratory HighDemand __Low Demand
‘Map Condition

Over both Low and High Demand
conditions, IOBC soldiers finished
training significantly faster than
OSUT soldiers.

‘ZoomiPan Find X Display Range!

Times on the map final exam show
that Exploratory soldiers in each
soldier group took about 5 minutes
more on the exam than soldiers in
High and Low Demand conditions.

i
i

ToBC "ANCOG BNCOS osuT

Figure 16. Illustrations of time spent on the Soldier Group
map exercises and map exam. ane Taneg Caton

44



Table C-22
Analysis of Variance on Tendency to Work Independently: Task Approach Strategies by Soldier
Group and Map Training Conditions

Factor I af [ MS | F I P
Complete With No Instructions
Map Conditions 2, 156 0.352 0.044 957
Soldier Group 3, 156 11.83 1.48 .222
Interaction 6, 156 3.07 0.384 888
Instructions Only If Needed
Map Conditions 2, 156 0.951 0.116 0.890
Soldier Group 3, 156 3.64 0.444 0.722
Interaction 6, 156 2.36 0.287 0.942
Read Instructions First
Map Conditions 2, 156 2.54 0.264 -768
Soldier Group 3, 156 10.66 1.11 347
Interaction 6, 156 9.73 1.01 419
Find Someone Else To Do It
Map Conditions 2, 156 0.151 0.500 607
Soldier Group 3, 156 0.078 0.259 855
Interaction 6, 156 0.263 0.870 519
Not Applicable
Map Conditions 2, 156 0.311 0.570 567
Soldier Group 3, 156 0.970 1.77 154
Interaction 6, 156 0.066 0.120 994
Table C-23
Mean Ratings on Facility with Computer Items
All
OSUT BNCOC ANCOC IOBC Soldiers

Item M M M M M

(SD) (SD) (SD) (SD) (SD)

(Range) (Range) (Range) (Range) (Range)

Have a “knack” or 5.58 5.06 5.23 6.19 5.52
“feel” for finding my (2.11) (2.86) (2.36) (2.33) (2.46)
way around a (1-10) (1-10) (1-10) (2-10) (1-10)
computer
Even if know one way 5.48 4.54 4.77 5.71 5.14
to perform a task, (2.56) (2.88) (2.79) (2.78) (2.77)
usually figure out a (1-10) (1-10) (1-10) (1-10) (1-10)
shortcut

Note. No significant differences among soldier groups. Knack: F(3, 164) = 1.77, p = 1559.
Shortcut: F(3, 164) = 1.80, p=.1499. Ratings ona 10-point scale, 1 = does not describe me at
all to 10 = describes me completely.

C-10



for the other soldier groups (BNCOC, ANCOC and IOBC). The soldiers’ knowledge of Army
symbols, the battle roster numbering system, and map reading was assessed, and provided
objective indices of relevant prior knowledge. Yet not all dimensions that could affect the
soldiers’ performance (e.g., knowledge of the structure of military organizations) were assessed.
Nor was there any attempt to directly assess differences in how the groups structured their
military knowledge as has been done with other knowledge domains (e.g., computer
programming, Ye & Salvendy, 1994).

Exploratory Modes of Learning and Training - Problem-solving

The map functions allowed comparison with another approach to training. Instead of
explaining or describing how the functions worked, we wondered what would happen if we let
the soldiers discover how these functions worked with no guidance, instruction, or feedback. In
other words, treat the map tasks as problem-solving tasks. Present the interface, state what each
function does, but not describe the procedures involved. Explore — figure it out on your own
with no hints or guidance on what to do. The goal was to determine how each function worked.
The individual worked with the map interface until personally satisfied that he understood each
function. Then he progressed to the map final exam.

The map functions in the experiment did not require great insight or creativity, although
it was presumed that soldiers had the basic computer skills required to work with menus and
icons. However, even when a computer environment is relatively simple, the level of complexity
can increase significantly if the user makes an error, venturing down an incorrect path. Greif
(1994) suggested that the complexity of a computer environment is not a stable characteristic.
Instead, level of complexity may be altered by user error, task novelty, and slight modifications
in system configuration.

There were several reasons for having an exploratory training condition for the map
interface (procedural skills). One, it is often stated that soldiers can and will figure things out on
their own. If you just give them the opportunity, they will decipher how a software package
works. Two, if successful, an exploratory mode would provide more training flexibility and an
alternative that might work very well for some soldiers. Third, research on exploratory training
had shown potentially positive and negative aspects, which were important to investigate in the
soldier populations of interest. Fourth, computer software seemed an appropriate content domain
for exploratory learning,

Research on exploratory learning has revealed both advantages and disadvantages over
traditional training. One reported advantage is improved transfer of learning (Carroll, 1997;
Egan & Greeno, 1973; Kamouri, Kamouri, & Smith, 1986). This improvement in transfer is
believed to result from the use of analogical reasoning and hypothesis testing when acquiring
procedural knowledge. It has also been reported that exploratory learning can be accomplished
in the same or less amount of time as instruction-based training (Carroll, 1997; Kamouri,
Kamouri, & Smith, 1986).

The reported drawbacks of exploratory learning include so called “exploration traps”
(Payne & Howes, 1992) where learners may select inefficient methods to accomplish a goal, or



Computer icon score

¢Short/Knack (Self-rating on tendency to develop computer shortcuts plus
Self-rating on tendency to have a “knack” in working with computers)

*BR experience (Prior use of BR system plus Training on the BR system)

*Symbols (Score on weapon symbols plus Score on unit symbols)

Map reading score

«Tendency to work independently scale

The relationships between the nine background variables and the two final exam scores
were examined separately for each of the training conditions for the codes and the map functions.
Initially, all background variables were entered into the multiple regression to establish the
maximum value of the multiple R. Next a forward stepwise regression was conducted, to
determine which variables contributed the most to the prediction. The regression results are
documented in Table E-25. The multiple Rs for the stepwise solutions ranged from .33 to .47,
with the highest R for the Exploratory condition on the map final exam (see Table 14). Results
showed that only measures from the computer and the military knowledge dimensions
contributed to the prediction of the final exam scores. The BR and work independently measures
did not predict.

The Exploratory map condition where soldiers learned on their own was of particular
interest. Could we predict what type of soldier(s) was (were) best suited for an Exploratory
training condition? The computer icon score was the strongest correlate for the Exploratory
condition (r = .42), followed by military knowledge of symbols, although this score did not
increase the R substantially (see Table 14). Perhaps, experience with computers made it easier
for soldiers to know what and how to examine the interface on their own. The tendency to work
independently scale did not correlate with the map score for the Exploratory group, even though
it was developed for that purpose.

Table 14
Stepwise Multiple Regression Results

Criterion Measures and Training Conditions

Code Final Exam Map Final Exam
Low High Low High Exploratory
(n= 80) (n= 83) (n= 40) (n= 34) (n= 80)

Predictor R | Predictor R | Predictor R | Predictor R | Predictor R

Symbols .26 | Icon Score 29 | Short/Knack .38 | Map Read .36 | Icon Score .41
Map Read .33 | Map Read .41 Self-rate .46 | Symbols .47

R 33 Al 38 46 AT

Note. Probability of F to enter was set at .10. Probability of F to remove was set at .15.
Variables are listed in order of their entry. Resulting R value is also shown at each step.

In general, the stepwise Rs were not high (.33 to .47), nor were the full equation Rs (.42
to .52). For each exam, the R for Low Demand was lower than that for High Demand. With the

48



ACKNOWLEDGEMENT

The research described in this report could not have been accomplished without the
support and hard work of many individuals. The team at TRW, Mr. Richard Wampler, Mr.
James Centric, Mr. Victor Williams, Ms. Diadra Swinson, and Mr. Steve Livingston, was
responsible for developing the storyboards and writing the computer-based courseware used in
the training experiment. This was a more complicated effort than usual, because four
experimental conditions were developed, and numerous edits were required to establish the
necessary experimental control. Dr. Gene Fober and Ms. Jennifer Bredthauer were among the
primary editors and reviewers of the storyboards and the final applications. In addition, they
assisted in execution of the experiments. We are also indebted to Ms. Lee Capraro in the
Directorate of Operations and Training in the U.S. Army Infantry School who graciously let us
use the multi-media classrooms under her control. The experiment could not have been
conducted without this invaluable support.

vi


Table C-4

Percentage of Soldiers Using a Computer in Different Phases of Their Fe ormal Education

% Use Computer
Grade Junior High Technical
Soldier Group | School High School | Schoo! | College | Not Use
OSUT 54% 11% 83% 2% 23% 4%
BNCOC 6% 15% 58% 2% 40% 19%
ANCOC 0% 10% 24% 0% 52% 24%
IOBC 29% 50% 76% 0% 95% 0%
Table C-5
Number of Educational Settings Where Soldiers Used a Computer
Soldier # Educational Settings (% soldiers)
Group 0 1 2 3 4-5
OSUT 4% 21% 27% 33% 15%
BNCOC 19% 52% 23% 4% 2%
ANCOC 24% 69% 3% 3% 0%
IOBC 0% 24% 29% 21% 26%

Note. Mean number of settings for each group was: OSUT — 2.33; BNCOC — 1.19;, ANCOC-

0.86; IOBC - 2.52. F(3, 163) = 26.90, p = .0000. All courses differed from each other except

OSUT and IOBC.
Table C-6
Percentage of Soldiers Indicating Computer Ownership and Current Use of a Computer
% Use
% Own a Computer Where Currently Use Computer
Soldier Group | Computer* Now”
Home Work/Unit | Trng Facility
OSUT 58% 63% 56% 6% 21%
BNCOC 67% 98% 69% 63% 42%
ANCOC 90% 97% 90% 76% 38%
JOBC 93% 95% 95% 21% 24%

® Own: ¥7(3) = 19.63, p = .0002.

> Use: ¥7(3) = 34.99, p = .0000.

C-2



Measures

Pre-experiment measures. Before starting the training, soldiers completed a survey on
their computer background and experience, and a short quiz on map reading, weapon symbols,
and unit symbols. They were asked whether they had ever used the Army’s BR numbering
system. Participants also answered a series of questions directed at their normal tendency to
work independently — to solve everyday type problems without requesting help or assistance.
Lastly, they were asked to rate the extent to which they have a “knack” for learning computer
programs on their own and a tendency to figure out computer shortcuts. These instruments are at
Appendix B.

The computer survey was one that had been given previously to over 2000 soldiers as
part of a three-year trend analysis of the computer background of soldiers (Dyer & Martin, 1999;
Fober, Bredthauer, & Dyer, 2000). The weapon and unit symbols tested in the quiz were
identical to those included in the training, except that the mortar symbol was used instead of the
SAW. Thus these measures constituted a test of their current knowledge in that area. The OSUT
soldiers were not expected to know these symbols as that is not a subject covered in initial
training. In prior research (Centric, Wampler, & Dyer, 1999), it was found that military symbols
were not taught in BNCOC. Consequently, high scores from the BNCOC soldiers were not
expected either. The map reading skills tested were common to all soldiers, so relatively high
scores were expected on these items.

The questions on tendency to work independently were developed in an attempt to
identify those individuals who tend to figure things out on their own and therefore might be good
in an exploratory learning environment. Many soldiers take this independent approach first,
rather than reading instructions or asking for assistance. A search of the literature during our
planning for the research showed no prior measures of this tendency. Therefore, a scale was
developed in which participants were asked how they would usually “approach” each of fifteen
activities for which instructions are typically available (e.g., changing the settings on a new
digital watch), or an opportunity to ask for directions (e.g., finding your seat at a ballgame or
concert). In order to assess whether individuals possessed a general tendency to solve problems
on their own, only three of the fifteen items described computer-related tasks. The remaining
twelve activities were quite varied (e.g., putting together a piece of furniture, responding to a
survey). The scale used to rate the 15 tasks was as follows:

. Complete with no instructions: Complete the activity (or give up) without referring
to instructions, a map, or asking for directions.

. Read instructions only if needed: Proceed at first without reading instructions, a map,
or asking for directions, but will refer to these upon encountering a problem.

. Read instructions first: Read the instructions, a map, or ask for help before
beginning.

. Find someone else to do it: Find someone else to perform the task you (a friend,

spouse, etc.).
. Not applicable

20



Practice Exercises for Land Warrior Map Exploration

The following practice exercises will help you familiarize yourself with the Land Warrior map functions.
This is NOT a test, and you will not be scored on your responses.

In addition to completing the tasks listed below, continue practicing all available functions. Do not
proceed to the Final Exam until you feel you are ready to perform a series of complex, timed exercises.
Practice Exercises:

Pan around the map display until you locate Gow Hill.

Zoom out until BM (Benchmark) 147 is visible on your map display.

Using the FindMe function, display your symbol.

Determine the range and azimuth FROM YOU to BM 147.

Zoom in on your symbol until Mount Paran is no longer visible on your map display.

Using FindX, find Company B. When you find the unit symbol on the map, click on it.

Using FindX, find the B Team Grenadier for Co. C, 4° Plt, 18 Sqd, and click on his symbol.
You are the commander, Company A. Display all of your platoon leaders.

oO OMNnNOAaR WN BS

Use pan to locate the second platoon leader. Click on his symbol.

Note on availability of individuals and units in the map area:
¢ Company D is not available.
e Only four complete squads are available for display:

> Company A, 3” Platoon, Weapons Squad

> Company B, 2TM Platoon, 2" Squad

> Company C, 1* Platoon, 1 Squad

> Company C, 2TM Platoon, Weapons Squad

Instructions for the Map Final Exam
At beginning of each task, there is a red banner that says:

Click NEXT to begin the exercise,

Once you click NEXT, the timer for the exercise starts. You must click NEXT to start each exercise. You
will not progress unless you click NEXT, and none of your responses will be scored.

Tip. Take the time to figure out the answer or determine your approach to solving the question before
clicking the NEXT button.

In the Range and Azimuth questions, you must physically move the cursor to each answer box by
clicking on it. You cannot tab to the other box.

Be patient; don’t rush when answering.

B-7



Appendix A

Symbols and Battle Roster System

Individuals

Acitio ACii2 ACIS Acta ACi25
Weapon Symbols
Rifle Squad Automatic M203 Grenade M60 or M240 Antitank Weapon

Weapon (SAW) Launcher Machine Gun Javelin

SAW symbol generated for purposes of the experiment; no official Army symbol existed.

Battle Roster. 1st two characters indicate Battalion, 1st number is code for company, 2nd
number is code for platoon, 3rd number is code for squad, 4th number is code for individual
Leaders coded as 0.

Units
a 0200 0
. ACI
Unit Symbols
Squad Platoon Company

Battle Roster. Characters and numbers as cited for individuals. Truncated at the
. appropriate level. This procedure was developed for purposes of the experiment; not
an official Army procedure.



Weapon Symbol Plus Battle RosterNumber Equals Identifying Code

* + AN1140 = t

AN1110

Symbol for Code for squad leader
M16 rifle, M4 patie Roster from 1st squad, 1st
Carbine platoon, A Company
$ * AN3223 ~ $
AN3223
Code for grenadier
bol fc
Ros aronade Battle Roster from A team, 2d squad,
Launder Number 2d platoon, C Company
2. cy
AB333
+ =
AB333
Symbol for Battle Roster
Rifle Squad Number Code for 3d squad, 3d
platoon, C Company

Figure 2. Examples of individual and unit codes resulting from combining symbols with the
battle roster (BR) numbering system. [BR: first two letters designate the battalion, 1st number the
company, 2d number the platoon, 3d number the squad, and last number the individual within the squad.
This last number also indicates whether the soldier is in A or B team. The first code shown above is fora
squad leader. A zero (0) in the last position with non-zeros in the preceding positions indicates a squad
leader. He carries an M16 rifle or M4 carbine. The second code is for a grenadier. The third code
illustrates a unit — the squad. The BR number is truncated at the squad position.]

Table 2 compares the Low and High Demand conditions by lesson. A chunk was defined
as a unique weapon or unit symbol (graphic) or as a unique number or position that defined a
duty position. Chunks, for purposes of the research, represented uniquely identifiable duty
positions or units. The number of chunks for each lesson is depicted. Exercises followed each
lesson. For example, with the Low Demand condition, lesson C was presented followed by the

13



reported they would find someone else to perform this task for them, while less than 10%
responded this way to any of the other tasks. Further analyses were conducted on the remaining
13 items.

To examine whether soldier groups differed in how they responded to the remaining 13
items, five new variables were created. For each soldier, the number of tasks performed in
accordance with each response option was calculated. In other words, we tallied the number of
tasks completed with no instructions, the number where instructions were read only if needed,
the number where individuals found someone else to do it, and the not applicable responses.
Since thirteen tasks were analyzed, possible scores for each response ranged from zero to
thirteen. Mean scores for each approach are shown in Table C-21 by soldier group.

The tendency to work independently is an individual difference variable that should not
relate to soldier group. Nor should this construct relate to the experimental map training
conditions. To test for possible differences, a soldier group by map condition ANOVA was
performed for each of the five task approaches. There were no significant differences in mean
scores for each approach by soldier group or map condition (Table C-22).

On average, the soldiers indicated they would read instructions for 41% of the thirteen
tasks (M= 5.36, SD = 3.08). They would put off reading instructions until a problem arose for
36% of the tasks (M = 4.69, SD = 2.79). Soldiers only mentioned two to three tasks (19%) that
they would perform without referring to instructions (M = 2.45, SD = 2.79).

A k-means cluster analysis was conducted to determine the percentage of soldiers who
would fall into what could be considered an independent style of solving problems versus a more
dependent style. The “not applicable” and “find someone else to do it” responses were excluded
from the analysis. Two clusters of soldiers emerged. The typical response by soldiers in each
cluster to the tasks in the scale was calculated. The results are shown in Figure 7. The figure
also shows that fewer soldiers were classified as solving tasks independently than in a more
dependent manner (29% vs. 71%).

im Not Read Directions
8 = Read If Needed 8
[3 Read Directions First

Mean # of Tasks Performed
With Each Approach

More Independent Less Independent
N=48 Cluster N=120

Figure 7. Profile of task completion strategies by soldier tendency to work independently.

30



as was the case for the scores. But in general, IOBC soldiers finished quickest, and OSUT
soldiers took the most time. As indicated by the time data and observations made during the
experiment, the CBT format adapted well to the differences in learning rate exhibited by the
soldiers. The experiment simply reinforces findings from previous CBT research about the
efficiency of this training approach, as it avoids the lock-step process inherent in teacher-led
classroom instruction. This increased efficiency can be viewed as particularly valuable when the
target user population is quite diverse, as it is with many of the Army’s digital systems.

Of interest was the lack of statistical interactions between the experimental conditions
and the soldier groups. This was not expected. Prior research (Corno & Snow, 1986) indicates
that high ability and/or or highly motivated individuals perform well in discovery learning
contexts or in situations where they have some degree of control, whereas less able individuals
require more teacher direction and instructional support. Although we had no measure of soldier
ability per se, the soldier groups did differ substantially in military experience and knowledge.
Consequently, it seemed reasonable for the High Demand conditions to have a greater impact on
OSUT soldiers than the other groups, given the limited military background of trainees. Instead,
when the High Demand condition impacted scores, it had a similar detrimental effect on each
group. Also it seemed plausible that the IOBC soldiers might do as well under an Exploratory
condition as the other conditions, given their university education. Although they were the
highest performing soldier group under this condition, their performance dropped relative to their
performance in the Low and High Demand conditions. A similar pattern of results occurred for
the other groups.

A very practical question is whether any of the training conditions resulted in satisfactory
levels of performance. The criterion was 80% correct. For codes, soldiers in both conditions
averaged over 80% correct on the code final exam. The averages for each course were greater
than 80% as well. A factor that could have contributed to soldiers meeting the code criterion on
the final exam was that they had to repeat a block of instruction if they did not achieve 80% on
the exercises the first time.

For the map exam, the final level of performance was not as high, falling below 80% for
each training condition and for each group of soldiers. There was no requirement for soldiers in
the Low and High Demand conditions to repeat map exercises when they failed to reach the 80%
criterion. If this had been required, it is likely that the final scores in these two conditions would
have been higher, resulting in an even greater discrepancy with the Exploratory condition.

A second practical question that remains unanswered in the present research is whether
the CBT lessons and exercises had a lasting effect. This issue could not be addressed in the
context of the current effort.

Given the relatively short time soldiers spent “exploring” the map interface and the small
but significant relationship between exploratory time and the final exam score, exploratory
modes of training bear further investigation. Exploratory learning should not be automatically
rejected as a training approach on the basis of this research. However, the results do support
prior research on the limitations of “pure” or “unguided” exploratory modes of instruction.
Combining some elements of formal instruction with an exploratory mode might prove very

50



sergeant, platoon leader, platoon sergeant, and radio-telephone operator (RTO). The weapon
symbol was combined with the BR number (all leaders had the rifle).

e Lesson H — Unit (3 chunks). The final lesson focused on codes for units, specifically the rifle
company, platoon, and squad. Although there are official symbols for each of these types of
units (presented in Lesson A), there is an official BR type number as well. For purposes of
the training, we simply truncated the BR number at the appropriate point to reflect the unit.
That is the squad BR number omitted the individual position (the last number). The platoon
BR number omitted squad and individual numbers. The company BR number omitted the
platoon, squad and individual numbers, consisting of only the battalion letters and the
company number.

Ifa soldier scored lower than 80% on any code exercise, he was given remedial training
on the corresponding lesson. He then retook the exercise.

Map functions. A total of seven map functions was taught: zooming in and out, panning,
finding oneself, finding others, displaying others, and range/azimuth determination. Figure 4
illustrates the map display. With the exception of range/azimuth, all functions were on the top
toolbar. Range/azimuth was at the bottom of the display. The map tasks reflected procedural
knowledge rather than the declarative knowledge of the coding system.

| LandWarrior - Training Experiment

ON

Te ON, ou

Layeis Draw Labels Display

i
y
4
ACID HLS
Video a

Message

Mailbox \\
i ira :
| Range/Azimath {) ij 4

Figure 4. Map display. [Functions included in the experiment have a thick border.]

16



Algha Team SAW Gamer Alpha
Alpha Tesm Grenacier TEAM
Alpha Team Rifleman

Sens Bess Tears ueacsr
~The sequence of the individuals, Breve Tea SAW Gummer Braye
| inleachiobthe firsteams abways avo Tear Greneaer Tear

Bravo Team Riftemsn

Rifle Squad Battle Roster,Codes:—
To:complete the graphic symbol:for each: squad membé
‘Take the individual's battle roster numbec:.

and add the symbol for the #reapon camied by the individual.

$

© to form the complete graphic-Symbol for that parficular squad. {

member. ;
$ Bravo Team Grenadier’: ”

from Company Ast

AC1117 Platoon, 1st Squad:



the exercise the first time and the other failed it. For example, if a soldier missed all three unit
items in Low Demand (3 of a total of 12 questions, 75% correct), he would have had to repeat
the exercise. If a soldier missed all three items in High Demand (3 of a total of 24 questions,
87% correct), he would have passed.

An analysis of the leader items was also conducted. But there was no distinct pattern
associated with the item stem as occurred with the unit questions. There is no readily available
explanation for the relatively high percentage of soldiers who had to repeat the leader questions
within the Low Demand condition.

Considering all conditions and exercises, OSUT soldiers were most likely to repeat
exercises, followed by BNCOC and ANCOC, and lastly IOBC. About half the OSUT soldiers
(48%) repeated at least one exercise, compared to 33% of the BNCOC and ANCOC soldiers and
to 12% of the IOBC soldiers. This represented a significant difference among the courses, 73)
= 13.07, p= .0044. Ofall the soldiers who had to repeat an exercise(s), in two-thirds of the cases
they only had to repeat one. In addition, for those who had to repeat exercises, OSUT soldiers
were most likely to repeat more than one exercise, while IOBC soldiers were least likely to do so
(19% vs. 2% respectively).

Times on code exercises and code final exam. Time to complete each code block of
instruction (lessons and exercises) and the final exam was also examined. A two-way analysis of
variance (training condition by soldier group) was conducted on time (see Table E-4). The time
data included the time spent repeating exercises for those soldiers who did not meet the 80%

criterion.

There were main effects, but no significant interactions (Table E-4). The only significant
difference between the High and Low Demand conditions on time was on the rifle and weapons
squad member blocks of instruction. Soldiers in Low Demand completed the same instruction in
less time than those in High Demand, F(1, 195) = 16.32, p = .0001 (Table E-5).

For the soldier group, there was a significant effect on four blocks of instruction (weapon
and unit symbols; company-squad codes, rifle-weapons squad member codes, and key leader and
unit codes). Descriptive statistics are in Table E-6. IOBC and ANCOC were significantly faster
than OSUT and BNCOC in each of these blocks of instruction (Table E-6), with OSUT soldiers
consistently taking the most time to complete a lesson and exercise and IOBC consistently
requiring the least time. The impact of these times on the cumulative time to complete the code

training is shown in Figure 12.
Time data were also examined for those soldiers who did not have to repeat an exercise.

Although the times decreased for those blocks of instruction when those requiring remediation
were eliminated, the same overall effects for condition and soldier group remained.

37



ID WAPRIOR

2. At what unit level do these two battle roster numbers

differ?
AA1342

Lesson

Company Level
Battle Roster Codes

There are five possible
codes that may.be
used forthe company. |



Van Oostendorp, H., & De Mul, S. (1999). Learning by exploration: Thinking aloud while
exploring an information system. Instructional Science, 27, 269-284.

Wallace, R. M., Kupperman, J., Krajcik, K, J., & Soloway, E. (2000). Science on the web:
Students online in a sixth-grade classroom. The Journal of Learning Sciences, 9(1), 75-

104.

Woolf, B. P. & Regian, J. W. (2000). Knowledge-based training systems and the engineering of
instruction, In S. Tobias & J. D. Fletcher (Eds.), Training and retraining: A handbook
for business, industry, government, and the military (pp. 339-356). New York:
Macmillan Reference.

Wright, D., & Lickorish, A. (1994). Menus and memory load: Navigation strategies in
‘interactive search tasks. International Journal of Human-Computer Studies, 40, 965-

1008.

Ye, N., & Salvendy, G. (1994). Quantitative and qualitative differences between experts and
novices in chunking computer software knowledge. International Journal of Human-

Computer Interaction, 6(1), 105-118.

56



4. |dentify the correct company.

Click on the correct answer

Lesson
Platoon and Squad
Level Battle Roster

Codes

‘ist Position

BATTALION

2d Position

D-6



Table E-12
Descriptive Statistics for Soldier Group on the Map Exercises and Final Exam (% correct)

Soldier Group L.TM@ [SD | Mdn [__Min-Max | N
Zoom, Pan, Find Me

IOBC 97.22 6.63 100 75-100 21
ANCOC 88.89 17.72 92 33-100 15
BNCOC 93.84 6.75 92 83-100 23
OSUT 92.03 10.80 100 67-100 23
Find X *

JOBC 88.89 9.98 92 67-100 21
ANCOC 81.94 24.32 87 8-100 12
BNCOC 92.75 11.32 100 58-100 23
OSUT 80.07 20.11 92 13-100 23
Display

IOBC 90.00 20.11 87 13-100 20
ANCOC 84.37 19.31 87 38-100 12
BNCOC 69.57 33.25 87 0-100 23
OSUT 73.37 30.22 87 0-100 23
Final Exam on Map”

IOBC 78.70 15.04 70 40-100 41
ANCOC 72.00 10.67 73 33-90 25
BNCOC 68.11 17.98 72 17-90 44
OSUT 68.11 18.30 70 23-97 44

Find X. F(3, 71) =2.86, p = .0431; BNCOC scores greater than ANCOC and OSUT.
Final Exam. F(3, 142) = 3.29, p = .0226; IOBC scores greater than BNCOC and OSUT.

c

E-12



Table E-15
Descriptive Statistics on Time in Map Lessons/Exercises and Map Final Exam by Soldier Group

(times in minutes and fraction of minutes)

Condition LM |..SD | Maa | Min-Mex [ON
Zoom, Pan, Find Me*

IOBC 12.84 2.06 12.72 8.40-17.28 21
ANCOC 14.64 2.01 12.82 9.75-16.48 15
BNCOC 15.25 2.02 14.72 11.83-19.18 23
OSUT 15.47 3.03 14.92 11.37-26.00 23
Find X

IOBC 12.29 1.94 12.77 7.88-15.10 21
ANCOC 13.16 2.01 12.82 9.75-16.48 12
BNCOC 14.02 2.97 13.75 9.08-22.98 23
OSUT 15.02 2.67 14.57 10.77-20.52 23
Display ©

TOBC 19.00 3.05 19.53 12.75-23.80 20
ANCOC 20.63 4.22 21.42 12.95-26.23 12
BNCOC 23.23 3.06 23.92 16.00-29.42 23
OSUT 24.09 3.68 24.65 15.38-29.72 23
Range and Azimuth

IOBC 3.43 1.41 3.03 1.97-6.62 21
ANCOC 3.26 1.25 3.52 0.82-5.57 15
BNCOC 3.68 1.52 3.38 1.38-9.23 24
OSUT 3.84 1.19 3.75 1,52-5.58 23
Final Exam on Map

IOBC 17.50 4.58 16.67 10.00-32.13 41
ANCOC 18.71 6.32 17.62 11.65-42.38 25
BNCOC 20.23 5.75 19.41 11.77-37.57 44
OSUT 18.54 5.06 17.65 10.37-32.28 41
Map Training *

IOBC 29.69 19.51 26.69 2.1-59.33 40
ANCOC 30.88 21.95 22.68 4.82-68.48 25
BNCOC 34.06 22.75 28.13 6.53-72.62 45
OSUT 35.94 23.73 34.88 3.1-74.82 45
Total Time °

JIOBC 47.60 17.56 46.21 20.30-74.25 40
ANCOC 50.24 20.71 50.27 20.95-93.12 25
BNCOC 54.73 20.49 49.87 26.00-96.53 44
OSUT 52.14 20.86 55.15 19.65-82.50 41

Zoom and Pan. F(3, 74) = 5.72, p= .0014; IOBC faster than ANCOC, BNCOC, and OSUT.
> Find X. F(, 71) = 4.50, p = .0059; IOBC faster than BNCOC and OSUT; ANCOC faster
than OSUT.
© Display. F(3, 70) = 9.28, p = .0000; ANCOC and IOBC faster than OSUT; IOBC faster than
BNCOC.
a Map Training. F(3, 143) = 5.55, p= .0000, IOBC faster than BNCOC and OSUT.
© Total Time. F(3, 138) = 7.66, p = 0000; IOBC faster than BNCOC and OSUT.

E-15



Table 6

Description of Map Exercises

Exercise & Condition

Description of Map Exercise Items

Low Demand Condition

Zoom In and Out, Pan,
and Find oneself

QO)

# Items = 12 G items/tasks for each of the four functions listed on the left).
Soldiers performed tasks that used each of the four functions. For each
question, soldiers were told which function they should use. For example,
an item on the “pan” function stated: “The 2" Platoon Company C is north
and west of you. Pan the map display until the unit symbol is visible near
the center of the map display. Once the platoon symbol is centered on your
map display, click on the symbol.”

Soldiers were scored on whether they used the required function. If asked
to click on a symbol, they were not penalized if they failed to do so. The
immediate feedback told them whether they had correctly used the required
function, as well as whether they had clicked the correct map symbol.

Find X
(kK)

# Items = 12.

Soldiers had to use the “Find X” function to locate individuals or units.
For example: “Using Find X, find ISG Company C. When you find the
corresponding symbol on the map, click on it.”

To receive credit in this exercise, soldiers had to click on the correct map
symbol once it was located. If they found the wrong individual/unit or
failed to click on the symbol, they were told that they had not completed
the task successfully.

Display Others
@)

# Items = 16 (only 8 counted toward a score).

The 8 tasks that were scored required soldiers to use the “Display” function
to display groups of individuals or units. For example: “You are the
commander, Company A. Display all platoon leaders. Click on the 37 Pit
Ldr.” Soldiers were not penalized if they failed to click on the correct
symbol, as long as they displayed the correct group of individuals or units.

The questions that were not scored asked soldiers to click on a particular
unit or individual already displayed on the map. For example, “You are
still the commander, Company A. Click on the 2 Platoon Leader.”
Soldiers did receive immediate feedback for these items.

High Demand Condition

Exercise items were the same as those in the Low Demand condition.

Zoom In/Out, Pan, Find
oneself, Find X and
display others

(GL)

Soldiers were exposed to all map exercises at a single point in time. The
exercises were in the same order as in the Low condition. The items were
not intermingled as was done with the High Demand Code condition.
Consequently, soldiers received a score for each of the three separate
exercises, not a single score across all exercise items.

Note. There were no practice exercises for Lesson M on determining range and azimuth. It was
determined that no exercises were required, as the words associated with the range/azimuth button were

almost self-explanatory.

24



Contents (continued)

Figure 1.

10.

11

12.

13.

14.

15.

16.

Page
LIST OF FIGURES
Multi-media classrooms where the experiment was conducted .......-.::seseeeeees 11
. Examples of individual and unit codes resulting from combining symbols
with the battle roster numbering SYStem ........-.:scsseseeeseeeeseeseeeeseeseesesseeneseeneenenrenes 13

Instructional sequence for code training conditions

Map display. 16
. Instructional sequence for map training CONdItIONS .............-.:ssesseeseeceseeesenneeee 19
Summary of computer usage and ownership, and subjective and objective
indices of computer EXPertise ........--scceesesesstesseecsseesesenessnsesnessneessnenennecnnnensneennes 28
Profile of task completion strategies by soldier tendency to work
Independently ............ssesessessseseeeseeseeeseneesneensnecnseessussnessnesssesssessssessaseensatanessnesans 30
Mean scores on the code exercises and final exam for the High and Low
Demand conditions ..........ecsssssssessessesssessesseesnseneeeesnecacnseseeneenscnrensressresessssesseestaes 32
Example of an exercise item on squad member COdeS .........-ssssssssescsssssseesceeesnees 33
Mean scores on the coding exercises and final exam for each soldier group...... 34
. Illustrations of the two question formats in the key leader and unit code
EXELCHSES oeeceesecesesseceeseesecseesessecsesscssecsscsuessecsscenseneecnenecneesneeascnssnecenesanessessessestesses 36
Cumulative time to complete training on codes, and times for each exercise
by soldier group ......sscsssssssssssssessnseessneeesaneeessneecssneecssnsessesessnessssasennnasscesonsacesanseneas 38
Mean times on rifle and weapons squad code training for soldier group
as a function of High and Low Demand conditions .........-sesssecsseecsseersscsseesnes 39
Mean scores for map exercises and map final exam for High Demand,
Low Demand, and Exploratory Conditions .........--ssessssesssssseesssecssssssseeeeneesaneennes 41
Mean scores on map exercises and map final exam for each soldier group........ 41

Illustrations of time spent on the map exercises and map exam...

xi



30 30

High Demand - All

28 28
2
3 26 26
3 High Demand -
4 No Repeat
2 24 lo Repeat 24
> 22 °° 22
° Low Demand - All
5 20 20
o
3 18 Low Demand - 18
it No Repeat
% 16 16
=
=
5 14 14
g 12 12
= 10 10

OBC ANCOC BNCOC OSUT

Soldier Group

Figure 13. Mean times on rifle and weapons squad code training for soldier group as a function
of High and Low Demand conditions.

Map Training Results

With the map training, the primary interest was in determining if there were differences
between the High and Low Demand conditions on the exercises, and if there were differences
among the High, Low, and Exploratory conditions on the final exam. As with the codes, course
differences and interactions with courses were also of interest. The criterion variables were
scores on the exercises and the final exam, and time to complete the training. Although soldiers
were not required to repeat an exercise if they did not achieve an 80% score, the number of
soldiers who did not achieve this level was examined.

Comparisons were made between High and Low Demand conditions on the following
exercises:

= Zoom, Pan and Find Me
= Find X
= Display

As described in the Method section, the difference between the High and Low conditions was
that soldiers in Low Demand went through each exercise immediately after receiving the

39


Contents (continued)

Page
LIST OF TABLES

Table 1. Experimental Design for the Code and Map Training Conditions ........ceecceeeeee 11

2. Operational Definitions of Chunks of Information for the Code Training
COnditiONS .....ccsescssesscseseseeseeeesesesecsnsessesessenesscseenecaesneansnesesneseacencsnensaneaeenenennennensas 14

3. Operational Definitions of Chunks of Information for the Map Training
Conditions .....sccsssssessesseessesensecstesstessssssccseensecsneessnescnesseasuessneesescasecsnecencenseceneeanets 7
4. Description of Coding Exercises .........:scssssssssssssssssescsnseseseanescenannescnnssseserssnsssess 22

5. Final Exam on Codes

6. Description of Map Exercises........-ssecsessscssssscsessssesecessneeesenssnsenecansenseereesenensennans 24

7. Final Exam on Map Functions

8. Use of Computers by Soldier Group.....-.--s:ssssecsssssessseesesssseeseeeneerensenneenneenneenneess 26

9. Means on Self-ratings of Computer Skill and on Icon Scores

10. Mean Percent Correct on Military Knowledge Tests .........-ssssssssseessesseeenerneesnes 28
11. Number of Soldiers who had to Repeat Code Exercises .......:.ss:ssssssssstssseensesnes 35
12. Number of Soldiers who did not Achieve a Score of 80% Correct on
the Map Exercises .......scssssesssssesssssseneeeeneccssscsssecessnsssnssensnsccanscensssceansecsssssnesestss 42
13 Factor Analysis Results on Background Variables.............-csssssscssersnteenteess 47
14. Stepwise Multiple Regression Results.........-.--sssssscssessssscesersntecsnstessnetsnnentsanens 48
x


The weapon symbol associated with each duty position helped to distinguish some
positions. However, the rifle weapon symbol was common to five soldiers in both the rifle and

weapons squads.

An item analysis was conducted on the rifle and weapons squad exercises to ascertain
whether soldiers in the High Demand condition had difficulty with certain items or if the lower
performance was spread across questions. The results indicated the lower exercise scores
reflected both factors. Questions on rifle squad member codes had two parts. The stem for all
items was a weapon symbol/BR code, as shown in Figure 9. The soldiers had to identify the rifle
squad (1st, 2d, or 3d) as well as the individual position within the squad. The first three such
items in the exercise created more difficulties for soldiers in the High than the Low Demand
condition. In particular, those in the High condition tended to identify only the squad member
himself, not the squad. The item difficulties or percent correct responses for these first three
questions were .48, .72, and .78 for High Demand versus .80, .92, and .84 for Low Demand.
This level of difficulty for High Demand did not occur again; the item difficulties showed a
learning effect (a recognition by soldiers that they had to identify the squad as well). It is
important to note that the High Demand condition’s exercise involved both rifle squad and
weapons squad questions. In contrast, the Low Demand condition had two separate exercises,
one on rifle squad and one on weapons squad.

2. Identify both the individual squad member and
his squad.

Figure 9. Example of an exercise item on squad member codes.
When the item difficulties for the remaining rifle squad items were examined, the mean

difficulty was .83 for High Demand and .93 for Low Demand, indicating a lower level of
performance for the High Demand condition. A similar pattern occurred on the weapons squad

33



Table E-11
Descriptive Statistics for High and Low Demand Conditions on the Map Exercises, and for High,
Low and Exploratory Conditions on the Map Final Exam (% correct)

Condition | M [SD | Mdn {| Min-Max l N
Zoom, Pan, Find Me

Low Demand 94.96 6.59 100 75-100 43
High Demand 91.45 13.98 100 33-100 39
Find X

Low Demand 87.70 15.96 92 33-100 43
High Demand 84.91 18.20 92 8-100 37
Display *

Low Demand 84.23 20.30 87 13-100 42
High Demand 71.18 34.44 87 0-100 36
Final Exam on Map”

Low Demand 78.27 13.02 82 40-100 40
High Demand 75.27 15.22 80 23-71 34
Exploratory 66.62 17.70 2B 17-80 80

Display. F(1, 70) = 4.07, p = 0474; Low scores greater than High scores.
Final Exam. F(2, 142) = 6.84, p = .0014; Low scores greater than both High and Exploratory
scores, High scores greater than Exploratory scores.

s

E-11



The coding system. In designing the experimental conditions for training individual and
unit codes, we viewed the specific code required to uniquely identify each position as a chunk of
information. Admittedly, this was a subjective decision, as the codes were not independent of
each other. But this provided a basis for partitioning the instructional materials, as there are no
firm guidelines for determining “chunks” of meaningful information. Compared to many
memory experiments, the tasks were not simply memory tasks. Specific concepts and rules
provided a scheme and format for deciphering the codes. In addition, following the presentation
of each block of information, training exercises were provided that allowed the soldiers to work
with the particular codes that had just been presented.

Soldiers had to learn a coding system for every individual within an Infantry platoon plus
key leaders at the company level. The code or symbol system involved combining the symbols
for the primary weapon (Department of the Army [DA], 1997) carried by individuals in each
duty position with what is known as the battle roster (BR) numbering system (DA, 1994). These
weapon symbol/BR combinations were developed specifically for the research. Figure 2
provides examples of these unique codes. These codes could be considered facts or declarative
knowledge (Anderson, 1980).

Five weapon symbols and three unit symbols were taught. The BR numbering system
was explained. The system provides for a five-character alphanumeric code that uniquely
identifies individual duty positions from the rifleman to the battalion commander. The training
focused on selected positions within the battalion. Soldiers learned the BR numbers for all nine
members of a rifle squad, all nine members of the weapons squad, the platoon leadership (3 duty
positions), plus the company leadership (3 duty positions). In addition, the coding scheme
covered unique codes for units at the squad, platoon, and company level. Consequently, for a
battalion, soldiers were taught to identify the code for any company commander, executive
officer, first sergeant, platoon leader, platoon sergeant, squad leader, rifleman, etc.

The BR roster number itself provides a unique code for each duty position. We added the
weapon symbol to help soldiers distinguish among the nine individual squad members. For
example, by presenting the M203 weapon symbol, the soldier immediately knew the individual
was a grenadier, even though he may have forgotten the unique number for a grenadier (a3 ora
7). A description of the battle roster system and the weapon symbols used in the training is in
Appendix A.

12



fail to remember the procedures they used to complete a task. Individuals may not possess the
appropriate information-seeking skills (Wallace, Kupperman, Krajcik, & Soloway, 2000). Even
when students had to use web technologies to solve problems, their information-seeking
behavior was simplistic (Wallace et al., 2000). Although using the web was enjoyable,
information seeking evolved into a search to find a good Web site. Students did not use search
engines well nor relevant information they found during the search.

In addition, individuals who lack experience with a task may not possess the
metaknowledge to recognize what they do not know. It has been reported that diversity of
experience has a positive influence on information seeking during exploration (Briggs, 1990).
High ability individuals may do well in an exploratory environment, but those with lower ability
apparently require more learning support (Shute, Lajoie, & Gluck, 2000). Some research has
addressed methods of providing a “supportive” exploratory environment in which potential
“traps” are minimized and the benefits of exploratory learning are optimized (DeMul & Van
Oostendorp, 1996; Trudel & Payne, 1995, Van Oostendorp & De Mul, 1999). For example,
Trudel and Payne improved exploratory learning of a computer simulated digital watch by
restricting the number of keystrokes allowed during a learning session.

Individual Differences in Exploration Tendency

Within the context of research on exploratory behavior and curiosity, computer software
systems have been referred to as the “modern labyrinths for humans,” and are ideal for research
on exploratory behavior (Greif, 1994, p. 287). Research in this area (e.g. Carroll, 1997; Carroll,
Mack, Lewis, Grischkowsky, & Robertson, 1985; Greif, 1994; Trudel & Payne, 1995) is still
relatively young, and has focused upon issues related to software design (e.g. the complexity of
software systems, providing help features to facilitate error management), and instructional
design methods (such as minimal manuals, mode restriction, and adaptive task complexity).

Outside the realm of computer research, individual differences in curiosity and
exploratory behavior have been studied in a wide variety of settings, using diverse methods and
stimuli (Henderson, 1994). Much of this research has been conducted among children, and it is
widely accepted that the predisposition to explore differs among children. However, research
has also demonstrated that the tendency to explore is influenced by setting and the type of
exploratory stimulus involved. For example, a child’s tendency to wander and explore a new
physical environment does not necessarily predict a tendency to explore other novel stimuli, or a
preference for complex over simple stimuli (Henderson, 1994). Research on individual
differences in exploring computer systems has focused primarily on the interaction between
experience (expertise) and characteristics of the interface such as complexity (see Greif, 1994)
and help features (e.g., De Mul & Van Oostendorp, 1996). However, little attention has been
given to the question of whether some individuals are naturally more inclined to learn their way
around a computer system through exploration of the interface.

As we were planning the present study, we were unable to find a measure of this
tendency to bypass manuals and instructions, and learn through active exploration. For the ‘
remainder of this report we will refer to this construct as a tendency to work independently.
Therefore a scale was developed to assess how participants approach a variety of tasks for which


WORKING MEMORY AND EXPLORATION IN TRAINING THE
KNOWLEDGE AND SKILLS REQUIRED BY DIGITAL SYSTEMS

EXECUTIVE SUMMARY

Research Requirement:

Much future training for equipment in the Army’s digitized forces will be done with
some form of interactive multi-media or computer-based training (CBT). Knowing how to
develop effective CBT for the diversity of soldiers and leaders employing these digital systems is
critical. When designing CBT, instructional designers make important conceptual decisions that
influence the learning process. One decision is how much material to include in a lesson.
Another decision is the extent to which the training should be highly structured as compared to
giving individuals the flexibility to learn on their own. The research described here examined
these issues within the context of prototype software for the future Land Warrior system. It
provided guidelines and insights into how to design CBT programs that will be successful with
the diversity of soldiers and leaders in the Army.

Procedure:

Soldiers, a total of 168, from four Infantry courses at Fort Benning, GA participated,
including those in basic training, noncommissioned officer courses, and the basic officer course.
Within each course, soldiers were randomly assigned to the training conditions. All training was
conducted via CBT; there was no instructor. Soldiers progressed at their own rate. The
experiment had two phases. In the first phase, two variations of CBT were developed to train
soldiers to learn unique codes for identifying units and individuals displayed on a digitized map.
One condition placed high demands on soldiers’ working memory by presenting much
information before soldiers had an opportunity to apply this information. The other condition
placed low demands on working memory, by breaking up the lessons and exercises into smaller
chunks. In the second phase of the experiment, soldiers were trained on map skills. Low and
high demand conditions were implemented again, and an exploratory condition was added. In
the exploratory condition, soldiers were informed of the map functions they had to learn, but not
how to execute the required steps.

Findings:

Consistent differences in the soldiers from the different courses occurred in both
experimental phases, with officers typically achieving the highest scores in the shortest amount
of time, and Infantry trainees scoring the lowest, taking the most time. In both training phases,
differences in favor of the low working memory demand condition occurred for the instructional
segments where the amount of information presented was the most discrepant between the high

vii


Appendix E
Results on Code and Map Training

Codes: Exercise and Final Exam Scores
Table E-1

ANOVA on Code Exercises and Code Final Exam Scores: High-Low Demand Conditions by
Soldier Group

Factor | af | MS | F | Dp
Weapon and Unit Symbols

High-Low Demand 1, 160 43.08 1.59 2091
Soldier Group 3, 160 1.97 0.07 9745
Interaction 3, 160 3.10 0.11 9516
BR Numbering System

High-Low Demand 1,159 75.47 0.76 3847
Soldier Group 3,159 121.05 1.22 3049
Interaction 3,159 179.82 1.81 1475
Company through Squad BR

High-Low Demand 1, 159 141.85 2.46 1186
Soldier Group 3, 159 75.18 1.30 .2748
Interaction 3, 159 4.17 0.07 9746
Rifle and Weapons Squads Codes

High-Low Demand 1,159 3,843.57 23.53 .0000
Soldier Group 3,159 1,023.15 6.26 0004
Interaction 3, 159 177.31 1.09 3570
Key Leader and Unit Codes

High-Low Demand 1, 159 0.39 0.004 9482
Soldier Group 3, 159 501.69 5.51 .0012
Interaction 3,159 12.58 0.14 9371
Final Exam on Codes

High-Low Demand 1, 159 186.86 1.48 .2240
Soldier Group 3, 159 814.49 6.49 .0003
Interaction 3,159 23.28 0.18 .9061



Section C
Map Skills, Operational Symbols and Battle Rosters

Directions: Given the 1/50,000 scale map in front of you and using a protractor, answer the
following questions as accurately as possible.

1. What is the 8-digit grid coordinate for the SP?
2. What is the 8-digit grid coordinate for the OBJ?
3. What terrain feature is the OBJ on?

4. If you traveled on a grid azimuth of 80 degrees from the SP, what is the name of the first hard
surfaced road you encounter?

5. What is the straight-line distance in meters from the SP to the OBJ?

6. What is the grid azimuth in degrees from the SP to the OBJ?

The remaining questions do not require a map:

8. Draw in the elements that will complete each unit symbol:

a) squad b) platoon c) company

9. Inthe spaces provided, name the following weapon systems:

f f
\ i

10. Have you ever received training on the Army’s battle roster system? Yes. No.
11. Have you used the battle roster system in a duty position within a unit? Yes No.
B-6


In the Low Demand map condition, the functions were divided into the four blocks of
instruction listed here.

Zoom in and out, pan, and find oneself
Find others (called Find X)

Display others

Determine range and azimuth

In the Low Demand condition, exercises followed each block except for range and azimuth. In
contrast, in the High Demand map condition, instruction on all functions was presented first, and
then individuals were allowed to practice at the end of this presentation. Again, there were no

exercises for range and azimuth.

It was harder to define “chunks of information” with the map functions than with codes.
Clearly distinguishable units or entities were harder to define, as some procedural steps were
self- cueing. Chunks primarily reflected the number of functions and the number of steps within
each function (see Table 3). The button/function itself was always counted as one chunk of
information as the soldier had to discriminate it from the other buttons/functions on the map

display.

Table 3
Operational Definitions of Chunks of Information for the Map Training Conditions
Low Demand High Demand
Lesson w/ # Content Description Lesson w/ #
Exercise Chunks Exercise Chunks
Zoom in, zoom out, pan, and find oneself
I 6 functions. Two functions had one step; two had
three steps.
Find X function, with 5 types of menu
J 6 selections: find unit, go to lower echelon, go to
higher echelon, find individual soldiers, & find LJ,K,& 26
leaders at specific echelon. M

Display function. It triggered 3 menu
selections; with 3 ways of working within each

K 9 menu, 2 unique selections within two of the
menus, plus a set of procedures covered in
Lesson J.
Range/azimuth function: 2 clicks to establish
M 5 points on map, where to click for accuracy and

how to cancel.

As shown in Table 3, the information presented in the High Demand condition was
greater than that in each Low Demand condition. Additional information on each lesson follows.
The lesson summaries describe the map training conditions. The lesson titles are consistent with

17



Types of Knowledge

Anderson (1980) distinguished between declarative knowledge (knowing that), and
procedural knowledge (knowing how). Declarative knowledge reflects the facts, concepts, and
principles we know; while procedural knowledge reflects the skills we know how to perform, to
include problem solving, reasoning, and means-ends analysis. For Anderson, procedural
knowledge or cognitive skill learning occurs in three stages: a cognitive stage where the
description of the procedure is learned, an associative stage where the method for performing the
skill is worked out, and an autonomous stage where the skill becomes more and more rapid and
automatic. Procedural knowledge can have both motor and cognitive components. A specific
task, obviously, can be primarily motor (riding a bike) or primarily cognitive (programming a
computer). Both forms of knowledge were included in the experiment.

The first phase of the experiment involved learning a coding system for uniquely
identifying individuals and units; that is, facts or declarative knowledge. This system combined
standard Army symbols with the battle roster (BR) numbering system. The second phase
focused on learning the procedural skills to manipulate a digital map interface, specifically how
to manipulate the software interface to solve tactical problems in locating personnel and ground
features, and to determine range and azimuth. There was no intent to train these procedural skills
to an autonomous stage.

Both phases compared training that placed differing demands on working memory. With
the procedural knowledge in the map phase, it was also possible to use an exploratory training
mode where there was minimal instructional guidance.

Demands on Working or Short-term Memory

One of the challenges for instructional designers of computer-based training is to present
new information so it can be kept in short-term or working memory for a sufficient period of
time to be processed and retained and retrieved as meaningful material in long-term memory.
We know that if information in short-term memory becomes inactive, it will be lost unless it is
also in long-term memory. As pointed out by Anderson (1980), it is helpful to “think of short-
term memory as a working memory holding only knowledge currently in use” (p. 166). But our
short-term capacity is limited. When new information exceeds the short-term memory
capabilities of learners, they cannot hold it in short-term memory long enough for it to be
processed, elaborated upon, and transformed into long-term memory. “For example, if we lose
track of the telephone number, we are unable to dial it (Anderson 1980, p. 166).”

Andrews and Bell (2000) made a similar point in discussing simulation-based military
training. “Structural models focus on identifying the mental structures, such as working, short-
term and long-term memories that play roles in information storage and retrieval. One of the
difficulties in designing virtual learning environments is understanding how much new
information should be presented at one simulation session before learning overload occurs.
Because simulation-based training can present so many new environmental cues at once to the
learner, it is important that a solid understanding of learner’s capacities be taken into account
during the instructional design phase. If too much information is presented too quickly, the


Table 4

Description of Coding Exercises

Exercise & Condition

Description of Exercise Items

Low and High Demand Co:

nditions

Weapon & Unit Symbols
(A)

# Items = 24 (each of the 8 symbols presented 3 times). Response was
selected from a multiple-choice format of the eight symbol names.

BR Numbering System
@)

# Items = 10. Each item presented two BR numbers. Soldiers had to
identify the echelon (battalion, company, platoon, squad, individual) at
which the numbers differed.

Low Demand Condition

Company BR # Items = 10. Soldiers were shown a BR number and had to identify the

(C) correct company, from a list of the five companies: Company A through
D and HHC.

Platoon and Squad BR # Items = 20 in two formats. In the first format, soldiers were shown a

(D) BR number and asked to identify the squad or platoon. In the second

format, a verbal description of an individual from a specific unit (e.g.,
“the 1 Squad, 34 Platoon of Company A”) was shown. Soldiers had to
select the correct BR number from five choices.

Rifle Squad Codes

# Items = 20. Soldiers were shown a code (weapon symbol plus BR

(E) number). They had to identify both the individual and his squad. In the
first row of response buttons, the three squads were listed (Ist, 2d, and 3d)
in random order. In the last two rows, all nine duty positions in the rifle
squad were listed, in random order (same order for all questions).
Soldiers had to answer both parts correctly to “pass” an item.

Weapons Squad Codes # Items = 15. Soldiers were shown the code of a soldier in a weapons

(F) squad. They had to select the correct duty position from the nine
positions in the weapons squad (presented in a random order).

Key Leader Codes # Items = 12 in two formats. In one format, the leader code was shown,

(G) and the soldier had to select the description of the leader’s position from
five options. In the other format, the leader’s position was described, and
the soldier had to select the correct code from five options.

Unit Codes # Items = 12 in two formats. In one format, the unit code was shown, and

(H) the soldier had to select the correct unit description from one of five
options. In the other format, the unit was described, and the soldier had to
select the correct code from one of five options.

High Demand Condition

Exercise items for High Demand were the same as those for Low Demand.

Combined Company # Items = 30.

through Squad BR (C&D) _| The company, platoon and squad code items were intermingled.

Combined Rifle and # Items = 35.

Weapons Squad Codes The rifle squad and weapons squad items were intermingled.

(E&F)

Combined Key Leader and
Unit Codes

# Items = 24.
The key leader and unit items were intermingled.

(G&H)

22



Mailbox

=U Using Find X fre: 15C Company C. When you ind the Cortes
fem he on, cick oni tem cick on the EXT baton,

"iad: 186 Company C. When you find he corresponding symbol
‘tick oni then lick on the NEXT bution,

pany C. When you find the earresponding syeibel
‘00H, thon ick an the NEXT burn,

D-18



Rifle Squad Battle Roster Codes

To: complete. the graphic symbol for each squad member:
Take the individual's battle roster number...

and add the symbol for the weapon carried by the individual...

form the complete graphic symbol for that particular squad

Bravo Team Grenadier.
from Company 'A, 1st

ACI117 ‘Platoon; 1st Squad

LAND WARRIOR

Identify both the individ
squad.

41. Identify both the individual squad member and
his squad.

‘Bravo Tesm:|f Aiphs Team:

‘SAW Gunnor. | “Ridemen
‘Squad: ‘Bravo Team: |BUBravo Tesi
Loader ‘Grensdier Lender



learner’s cognitive structures can be overwhelmed to negative effect (p. 380).” In computer-
based training experiments on combat vehicle identification, Dyer, Westergren, Shorter, and
Brown (1997) found that when too many vehicles were presented (8 vehicles each with 8 aspect
angles for a total of 64 images), the training time increased and soldiers did not reach criterion
easily. In this context, the recommendation was that no more than 6 vehicles, preferably 4 to 5,
should be presented in a training session. In quite a different context, Trafton and Reiser (as
cited in Atkinson, Derry, Renkl & Wortham, 2000) examined the effects of alternating examples
and practice to a blocked condition where all examples were presented followed by all practice
conditions. They hypothesized that the alternating condition allowed the participants to maintain
an example in working memory during the practice session. Results showed that the alternating
example and practice condition was more effective.

The concept of meaningful chunks. Miller (1956) argued in his classic article on “The
magical number seven, plus or minus two” that short-term memory was limited not by the
number of physical units (letters or words), but by the number of meaningful chunks. The
number of chunks that could be held in immediate memory was postulated to be seven, plus or
minus two. This span of immediate memory limits the amount of information that individuals
can receive, process, and remember. But it can be expanded through recoding or grouping
information into larger chunks that make the information more meaningful. Miller
acknowledged in that article, as have others (Anderson, 1980), that it is difficult to define what
constitutes a chunk of information. Nonetheless, the amount of information we can hold in
short-term memory represents a limitation on our mental capacity.

Since Miller’s (1956) article, more is known now about short-term memory, and slightly
different theoretical frameworks have been proposed. The extent and nature of recoding depends
on previous learning (Anderson, 1980; Baddeley, 1994; Ye & Salvandy, 1994). The number of
chunks is dependent on the degree to which the material within each chunk is integrated or
related (Baddeley, 1994). Short-term memory paradigms now involve retrieval from both long-
term and short-term memory (Shriffin & Nosofsky, 1994). Many other factors such as attention,
interference from similar inputs, and decision-making affect an individual’s short-term
processing of information (Baddeley, 1994; Shriffin & Nosofsky, 1994).

Applying the concept of chunks. In light of the substantial amount of research on short-
term memory capacity, the many factors that can influence processing of information, and the
difficulty in defining “chunks” (Simon 1974), it may seem unrealistic, and it probably was
simplistic, to try to design experimental conditions that specifically varied the amount of material
to be learned, based on the concept of chunks. Nonetheless, the concept was used to help
structure the experimental conditions. It must be noted that most of the material presented in the
experiment was unfamiliar to the soldiers. They did not know the coding scheme we presented;
they had never seen the map interface that was presented to them. From that perspective, they
had had no prior opportunity to “recode” this material, which would have helped them process
larger amounts of information. On the other hand, the material was not equivalent to the
nonsense syllables and strings of digits or words used in much of the work on short-term
memory. The soldier’s military background was relevant to the training content. The
experiment should not be considered as a formal attempt to psychologically define or measure
chunks. But it was hoped that the results would provide some guidelines, but certainly not a



ceo AK

FIND ZX STRUCTURE
xX . FINDING UNITS

Fea
eR companies
ate
eta >
toa»

Fou

SEP teeter

Hs

Te ind a company. select the down arow.slect he compory, and then
[BAGK) Sees ttticoes ee Sppear inthe comer ct you mop.

ing indlviduals similar to nding unis. Shown fshow you would fi the
(oon Lvader of ts Platoon, A Company.

|| The Find X function allows you to find a unit or an individual within a unit

To activate the Find X function to find a particular unit or person, click the
down arrow next to the Find X button

Continue your choices until you find the individual or unit you seek, then
click on it.

The corresponding symbol and battle roster graphic will hen appear
centered on your map,

‘lick NEXT after completion of roading dhe teview points.

D-17



ng it Squad. we have displayed ail members othe Tat Squad and
‘her batie ster members.

D-22



Table E-9
Descriptive Statistics for Soldier Groups on Time to Complete Code Instructions and Final Exam
for Soldiers who did not Repeat an Exercise (in minutes and fraction of a minute)

Soldier Group | M | SD l Mdn l N
Weapon and Unit Symbols “

IOBC 5.03 1.29 4.88 42
ANCOC 5.44 1.09 5.23 29
BNCOC 5.99 0.76 5.92 46
OSUT 7.09 1.54 7.22 47
Battle Roster Numbering System

IOBC 4.45 2.28 4.05 4]
ANCOC 4.91 1.25 4.76 28
BNCOC 4.78 1.06 4.85 45
OSUT 4.99 1.20 4.83 46
Company through Squad BR”

IOBC 7.90 1.56 7.73 4]
ANCOC 8.88 1.81 8.87 29
BNCOC 9.44 2.26 9.24 46
OSUT 10.13 2.07 10.07 46
Rifle and Weapons Squads Codes ©

IOBC 16.03 3.48 15.37 40
ANCOC 16.66 4.85 15.58 28
BNCOC 18.38 4.33 17.93 43
OSUT 21.43 7.14 20.97 37
Key Leader and Unit Codes“

IOBC 12.83 3.66 12.35 37
ANCOC 14.52 5.09 12.43 23
BNCOC 15.39 2.88 15.13 39
OSUT 17.57 4.64 16.63 31
Final Exam on Codes

TIOBC 9.62 4.28 8.28 36
ANCOC 8.52 1.62 8.15 20
BNCOC 9.73 2.47 9.35 33
OSUT 9.69 3.45 8.63 25

Note. Final exam times included only those soldiers who did not repeat any exercises.

® Weapon & Unit symbols. F(3, 156) = 23.47, p = .0000; IOBC, ANCOC, and BNCOC each
faster than OSUT, and IOBC faster than BNCOC.

> Company-Squad BR. F(3, 154), = 9.77, p = .0000; IOBC faster than OSUT and BNCOC.

© Rifle/Weapons Squads. F(3, 140), = 8.64, p = .0000; IOBC, ANCOC, and BNCOC faster
than OSUT.

4 Key Leader and Unit. F(3, 122), = 7.91, p = 0000; IOBC and ANCOC faster than OSUT;,
IOBC faster than BNCOC.

E-9



Computer Training Research
(U.S. Army Research Institute, Ft. Benning, GA)

Log-In Code:

Name (optional)

Please Do Not Begin Until Instructed

PRIVACY ACT STATEMENT

Public Law 93-573, called the Privacy Act of 1974, requires that you be
informed of the purpose and uses to be made of the information collected.

The Department of the Army may collect the information requested in this
questionnaire under the authority of 10 USC 2358. Providing information in
this questionnaire is voluntary. Failure to respond to any particular questions
will not result in penalty.

This packet contains the paper and pencil portion of the Computer Training
Study. The package is divided into three sections:

Section A: Survey on computer experience
Section B: Survey on problem solving behavior
Section C: Quiz on map reading, symbols, and battle roster system

e We will provide instructions for the computer training portion of the study
once everyone has completed Sections A through C. Please do not
attempt to log onto the computer until instructed to do so.

B-2


Appendix C
Soldier Background Results

Table C-1

Descriptive Statistics on Age of each Soldier Group
Soldier Group N M Madn SD Range
ANCOC 29 31.59 30 3.40 27-39
BNCOC 48 27.75 27 3.32 22-38
IOBC 42 23.14 23 1.92 21-29
OSUT 48 19.58 19 2.36 17-27

FG, 163) = 137.59, p=.0000. Mean age for each soldier group differed from the mean of each
other group.

Table C-2
Number of Soldiers by Rank in Each Soldier Group
Rank Soldier Group
OSUT BNCOC ANCOC IOBC
Private All 48 NA NA NA
Sergeant 15
Staff Sergeant 33 0
Sergeant First Class 28
2d Lieutenant All 42

Note. No data on rank from 1 soldier in ANCOC.

Of the IOBC students, 17 (41%) received their commission through Reserve Officer Training
Corps, 24 (57%) from the US Army Military Academy, and 1 (2%) from the US Air Force
Academy.

Table C-3

Descriptive Statistics on Years in Active Service
Soldier Group N M Mdn SD Range
ANCOC 29 11.51 11.9 2.21 9.00- 18.17
BNCOC 48 TAT 7.0 2.60 4.25-19.67
JOBC 42 1.93 0.8 2.29 0.00-7.17
OSUT Not asked because not applicable

Note. For the OSUT soldiers, 31 (65%) had a high school education, 3 (6%) had technical
school training, 10 (21%) had less than 4 years of college, 3 (6%) had a college degree, and 1
(2%) had a GED.

C-1


military intelligence personnel, but CBT was not used in that training. In summary, we did not
find any research on the training effectiveness of different CBT approaches for training digital
skills with the Army’s “new” digital systems.

Also of interest was whether any prior CBT research had examined the effects of
discovery learning or the role of working memory. Some of the research cited earlier in this
report did address CBT-related issues (e.g., Trudel & Payne, 1995), but most did not. Trudel and
Payne simulated a commercial digital watch. They found that imposing a keystroke limit during
the learning process or forcing subjects to explore one mode at a time were better training
techniques than allowing subjects to explore the simulated watch on their own without any
constraints. Some additional examples of research that examined CBT in conjunction with
working memory are cited below.

Chandler and Sweller (1996) did not examine CBT per se, but did focus on the demands
on working memory when learning a computer-aided design software package. They examined
the impact of the design and use of computer manuals on cognitive load. When high
interactivity with the computer was required, a manual format that required holding text
information in working memory while searching and working with the computer screen was less
effective than a manual-only approach. When low interactivity with the computer was required,
then the format of the training manual did not impact leaning. Shute (as cited in Shute et al.,
2000) found an aptitude-treatment interaction involving working memory capacity in the
learning of a flight-engineering tutor. Individuals with high working memory capacity and high
general knowledge consistently performed well; those low on both dimensions performed poorly.
The interaction effect occurred with those high on one dimension and low on the other.
Specifically, individuals with low working memory capacity but much general knowledge
benefited from being assigned to an abbreviated training condition, rather than an extended
period of training. Apparently, the extended condition led to either boredom or fatigue. Those
with high working memory capacity but low general knowledge performed best in the extended
practice condition, as they were able to benefit from the extra practice. In examining the use of
computer software for a shopping task (determining which store had the best prices for differing
numbers of items), Wright and Lickorish (1994) found that as the number of steps needed to
solve a problem increased (more demands on working memory), the more likely participants
were to use various memory aids available to them. However, which memory or navigational
aids participants were most likely to use was not predictable. The authors concluded that the
“cognitive consequences of moving through texts displayed on computer screens are far from
trivial” (p. 1003), whereas the cognitive consequences of page turning with books and manuals
seem trivial.

Method

Participants

Soldiers from four U.S. Army Infantry School courses participated. These courses were
the IOBC, BNCOC, ANCOC, and Infantry OSUT. The Infantry OSUT soldiers were in the 11th
week of OSUT, which is a 14- to 16-week course. Both BNCOC and ANCOC soldiers were



Mailbox
Tita

Message

.\
Maitbox_|\},

sar
Paar SE 7
———— Winstead Hitt
=. M4 Wy

Bravo Team, 34 Squad, 151 Platoon,
we corresponding aymbol an the nap, click ont,



exercises for C. Then lesson D was presented followed by the exercises for D. However, with
the High Demand condition, lessons C and D were presented first, followed by exercises C and
D. Under the Low Demand condition, less information was presented before soldiers worked on
exercises involving application of or memory for the information in the lessons.

Table 2

Operational Definitions of Chunks of Information for the Code Training Conditions

Low Demand High Demand
Lesson w/ | # Chunks Content Description Lesson w/ | # Chunks
Exercise Exercise
A 8 5 weapon symbols and 3 unit symbols. A 8
B 5 The 5 positions in the BR numbering system. B 5
Cc 5 The 5 numbers for company BR position.
D 8 The 4 numbers for platoon position and 4 C&D 13
numbers for squad position.
E 9 The 9 numbers for individuals in rifle squad. E&F 18
F 9 The 9 numbers for individuals in weapons
squad.
G 6 The 3 key leaders at both company and
platoon echelons. G&H 9
H 3 BR number for 3 echelons (units).

Differences in the High and Low Demand conditions started at Lesson C (see Table 2 and
Figure 3). As indicated in Table 2, the information presented in some High Demand conditions
was double that presented in the Low Demand conditions.

A summary of each lesson follows. For continuity, the abbreviations in the summary
titles below correspond to the abbreviations in Figure 3.

e Lesson A -Symbols (8 chunks): Five weapon symbols (M16 rifle/M4 carbine, M203 grenade
launcher, M249 squad automatic weapon, M240B machine gun, and Javelin antitank
weapon) and three unit symbols (squad, platoon, and company) were presented. Except for
the M249 squad automatic weapon (SAW), all symbols were the same as those in the
doctrinal manual on symbols FM 101-5-1 (DA, 1997). As there is no unique symbol for the
SAW, one was generated for that purpose. Soldiers were told this was not an approved Army
symbol, and was being used strictly for the instruction they were receiving.

e Lesson B — BR (5 chunks). The concept of the Battle Roster (BR) Numbering System was
presented with an explanation of the five positions referenced in the letter/number string that
uniquely defines individual duty positions. The battalion has a two-letter code; followed by a
company single number code; a platoon/section number code; a squad number code; and an
individual soldier number code.

14



There were also differences when the time devoted to all training (lessons and exercises
and exploring the map) was examined. Here, High and Low Demand condition times were four
times slower than Exploratory times, F(2, 143) = 622.65, p = .0000. On average, soldiers in the
Exploratory condition spent 12.5 min working with (learning about) the map functions on their
own. The fastest time was 2 min; the longest was 35 min (Table E-14). This compares to
averages of 54 min for the soldiers in the Low and High Demand conditions to complete all the
lessons and exercises.

Although the soldiers in the Exploratory condition completed their instruction very
quickly, they took more time to complete the map final exam than the other two conditions (22
min vs. 15 min, see Table E-14). Exploratory times on the final exam were significantly slower
than both the Low and High Demand times, F(2, 138) = 46.43, p = .0000. As illustrated in
Figure 16, this increased amount of time was the same for all soldier groups. It appeared that
soldiers in the Exploratory condition tried to make up for what they did not learn when taking the

exam itself.

Of additional interest is that data on the final exam indicate the Exploratory group spent
more time reading and studying the questions than the other two groups. The map exam had 22
items each with a time limit of one minute. The maximum times in the High and Low Demand
conditions were 20 minutes, meaning that all soldiers within these conditions completed all items
plus all instructions within 20 minutes. However, the maximum time for the Exploratory
condition was 42 minutes, with 59% of the soldiers (47 of 79) taking more than 20 minutes. The
only opportunity for additional time during the exam was in examining the questions prior to
officially answering a question. It is not known what soldiers were doing while examining the
questions --- were they trying to interpret or understand the question or to determine their
strategy in how to answer it? Regardless, this increased time reflects a lower level of proficiency

for the Exploratory condition.

There were also significant differences in time among the soldier groups. They differed
on time to complete three different blocks of map training: zoom/pan, F(3, 74) = 5.72, p=
0014; find X, F(3, 71) = 4.50, p = 0059; and display , F(3,70) = 9.28, p = 0000. On all three
blocks of instruction (zoom/pan, find X, display), IOBC soldiers were consistently faster and
OSUT consistently slower. In addition, ANCOC was significantly faster than OSUT on find X
and display. IOBC was significantly faster than BNCOC and ANCOC on zoom/Pan, and faster
than BNCOC on display (Tables E-13 and E-15). On the time to complete training (lessons and
exercises for the High and Low Demand groups and the time working with the map for
Exploratory), IOBC was significantly faster than BNCOC and OSUT, F(3, 143) = 5.55, p
=,0000. The same pattern occurred on the total time spent on the map phase of training. IOBC
was significantly faster than BNCOC and OSUT, F(3, 138) = 7.66, p =.0000.

In summary and as illustrated in Figure 16, the major impact on time was the training
conditions, although soldier groups differed significantly on the map times as well. Soldiers in
the Exploratory group did not explore very long, but spent more time on the exam than soldiers
in the other two training conditions.

43



1 The tn Platoon Company B is west of you
{symbol forthe 15¢ Platoon Company 8 ls

ANT) JALARR

G) The purpose of fis lesson isto teach you howto
locate other units of members of your unit or other
units using the Fi



Results
Pre-experiment Measures

Computer background. In general, the computer backgrounds of the soldiers (see survey
in Appendix B) were very similar to that obtained in prior research (Dyer & Martin, 1999; Fober
et al., 2000). The major deviation from these previous findings was that BNCOC soldiers in the
experiment were lower on subjective and objective measures of computer expertise than the
OSUT soldiers. In the prior research, OSUT soldiers were lower than BNCOC.

Table 8 shows the percentage of soldiers who used a computer at some time in their
formal schooling, the percentage currently using a computer, and the percentage that own a
computer (see also Appendix C). The lack of use of computers in public school settings directly
reflects the ages of the soldiers in the different groups. The youngest groups (OSUT and IOBC)
were more likely to have been exposed to computers at the grade school, junior high, and high
school levels (see Table C-4). On the other hand, the ANCOC soldiers were the least likely to
have been exposed to computers even during high school. The picture changed somewhat with
college experience as most colleges require students to use computers. The IOBC soldiers were
the mostly likely to have used computers in college (95%), followed by ANCOC (52%), then
BNCOC (40%) and lastly OSUT (23%).

Table 8
Use of Computers by Soldier Group
% Used Computer
Soldier Group During Their Formal | % Currently Using a % Owning a
Education Computer Computer

OSUT 96% 63% 58%
BNCOC 81% 98% 67%
ANCOC 76% 97% 90%
IOBC 100% 95% 93%

The percentage of soldiers within each group currently using computers was very high, at
least 95% for all groups but the OSUT soldiers (63%). For each group, the usage percentages
tended to be slightly higher than the percentage of soldiers indicating they owned a computer.

As shown in Table 8, the ownership rates were highest for IOBC and ANCOC, lower for
BNCOC, and lowest for OSUT. Of interest is that over half the OSUT soldiers said they owned
acomputer. Both the computer ownership and computer use variables showed significant soldier
group differences (see Table C-6).

With regard to all computer features used (mouse, games, internet, etc.), there were no
significant differences among the groups (see Table C-7 in Appendix C). The groups did differ
on their self-ratings of typing skill, with the IOBC soldiers most likely to say they could type
quickly (45%) and BNCOC the least likely to have this skill (8%), 70) = 27.83, p= .0010.

26



mid-way through their courses, which are 7 weeks and 10 weeks, respectively, in length. All
Infantry officers, regardless of commissioning source, attend the IOBC, a 16-week course. Some
officers had graduated from IOBC; others were waiting to attend. These differences did not
affect the results.

The experimental design called for 48 soldiers from each course. The actual number of
participants was 48 each from OSUT and BNCOC, 42 from IOBC, and 30 from ANCOC.

There were considerable differences among the soldiers in the courses as reflected in
their rank, time in military service, and age (see Appendix C). All Infantry OSUT soldiers held
the rank of private. All IOBC soldiers held the rank of 2d lieutenant. All but one of the ANCOC
soldiers held the rank of sergeant first class. BNCOC soldiers held the ranks of sergeant (31%)
and staff sergeant (69%). ANCOC soldiers had been in the Army a mean of 11.51 years (SD =
2.21); BNCOC a mean of 7.47 years (SD = 2.60), and IOBC a mean of 1.93 years (SD = 2.29).
The OSUT soldiers were not asked about their prior service time, as they were trainees.
However, they were questioned about their educational background. Over half (65%) had a high
school education, one soldier had a general education development (GED) diploma, and the
remaining 33% had some education beyond high school (college or technical school). The mean
ages of the groups differed significantly, F(3, 163) = 137.59, p = .0000. OSUT soldiers were the
youngest (M= 19.58, SD = 2.36). IOBC was the next oldest group (M = 23.13, SD = 1.92).
BNCOC was the next oldest group (M= 27.75, SD = 3.31). ANCOC was the oldest group (=
31.59, SD = 3.40). Considering all soldiers, the youngest was 17; the oldest was 39.

Procedure

Soldiers within each of the four courses were randomly assigned to one of four
experimental training conditions. The experiment was executed four times; once for each group
of soldiers. Three multi-media rooms in the Infantry School were used. Each room had 16
computers, allowing a maximum of 48 soldiers to participate at a given time (see Figure 1). Four
hours were allowed for the experiment. Software for each of the experimental conditions had
been loaded on the computers prior to beginning the experiment.

Before starting, soldiers were briefed on the scope of the experiment. They were

encouraged to take a break after completing the code exam and before beginning the map
training. They could take other breaks if desired, but only after an exercise or lesson.

10



Greif, S. (1994). Computer systems as exploratory environments. In H. Keller, K. Schneider, &
B. Henderson (Eds.). Curiosity and Exploration (pp. 286-306). New York: Springer-
Verlag.

Henderson, B. (1994). Individual differences in experience-producing tendencies. In H. Keller,
K. Schneider, & B. Henderson (Eds.). Curiosity and Exploration (pp. 213-226). New
York: Springer-Verlag.

Kalyuga, S., Chandler, P., & Sweller, J. (1998). Levels of expertise and instructional design.
Human Factors, 40(1), 1-17.

Kamouri, A. L., Kamouri, J., & Smith, K. H. (1986). Training by exploration: Facilitating
transfer of procedural knowledge through analogical reasoning. International Journal of
Man-Machine Studies, 24, 171-192.

Lohr, L. L. (2000). Designing the instructional interface. Computers in Human Behavior, 16,
161-182.

Payne, S. J., & Howes, A. (1992). A task-action trace for exploratory learners. Behaviour and
Information Technology, 11, 63-70.

Miller. G. A. (1956). The magical number seven, plus or minus two: Some limits on our
capacity for processing information. Psychological Review, 63, 81-97.

Sanders, W. R. (2001). Cognitive psychology principles for digital systems training (ARI
Research Report 1773). Alexandria, VA: U.S. Army Research Institute for the
Behavioral and Social Sciences.

Schaab, B. B., & Moses, F. L. (2001). Six myths about digital skill training. (ARI Research
Report 1774). Alexandria, VA: U.S. Army Research Institute for the Behavioral and
Social Sciences.

Shriffin, R. M., & Nosofsky, R. M. (1994). Seven plus or minus two: Acommentary on
capacity limitations. Psychological Review, I 01(2), 357-361.

Shute, V. J., Lajoie, S. P. & Gluck, K. A. (2000). Individualized and group approaches to
training. In S. Tobias & J. D. Fletcher (Eds.), Training and retraining: A handbook for

business, industry, government, and the military (pp. 171-207). New York: Macmillan
Reference.

Simon, H. (1974). How big is a chunk? Science, 183, 482-488.

Trudel, C. L., & Payne, S. J. (1995). Reflection and goal management in exploratory learning.
International Journal of Human-Computer Studies, 42, 307-339.

55



Table E-5
Descriptive Statistics for High and Low Demand Conditions on Time to Complete Code
Instruction and Code Final Exam (in minutes and fraction of a minute, for all soldiers)

Condition | M [SD | Mdn | Min-Max | ON
Weapon and Unit Symbols

Low Demand 6.02 1.49 5.90 2.02-10.38 85
High Demand 6.15 1.80 5.82 2.72-12.58 83
BR Numbering System

Low Demand 4.96 1.86 4.69 2.22-16.42 84
High Demand 4.78 1.24 4.85 1,57-7.25 83
Company through Squad BR

Low Demand 9.39 1.83 9.27 5.97-13.70 84
High Demand 9.27 2.78 8.88 3.50-20.18 83
Rifle and Weapons Squads Codes*

Low Demand 17.37 4.30 16.92 10.28-28.47 84
High Demand 21.94 9.93 19.68 5.82-67.40 83
Key Leader and Unit Codes

Low Demand 16.55 4.76 16.29 8.57-29.27 84
High Demand 16.38 6.10 14.95 5.13-48.03 83
Final Exam on Codes

Low Demand 10.03 3.00 9.11 5.57-23.72 84
High Demand 9.41 3.30 8.58 4.60-26.17 83

Note. Includes time for soldiers who repeated exercises. No repetition of the final exam was
allowed.
* Rifle/Weapons Squads. F(1, 159) = 16.32, p = 0001; Low faster than High.



Table E-7

ANOVA on Time to Complete Code Instruction and Code Final Exam: High-Low Demand

Conditions by Soldier Group for Soldiers who did not Repeat an Exercise

Factor df | MS | F P
Weapon and Unit Symbols

High-Low Demand 1, 156 0.09 0.06 -8084
Soldier Group 3, 156 34.95 23.47 .0000
Interaction 3, 156 1.06 0.71 5449
BR Numbering System

High-Low Demand 1, 152 0.71 0.31 5791
Soldier Group 3, 152 2.31 1.00 3929
Interaction 3, 152 4.03 1.75 1599
Company through Squad BR J
High-Low Demand 1, 154 6.19 1.60 2075
Soldier Group 3,154 37.79 9.77 .0000
Interaction 3, 154 1.94 0.50 6816
Rifle and Weapons Squads Codes

High-Low Demand 1, 140 214.18 8.71 .0037
Soldier Group 3, 140 212.38 8.64 .0000
Interaction 3, 140 25.83 1.05 3722
Key Leader and Unit Codes

High-Low Demand 1, 122 0.03 0.00 9627
Soldier Group 3, 122 126.41 791 .0000
Interaction 3, 122 21.17 1.32 .2694
Final Exam on Codes

High-Low Demand 1, 106 8.99 0.84 3601
Soldier Group 3, 106 7.83 0.73 -5331
Interaction 3, 106 13.14 1.23 3001

Note. Final exam times included only those soldiers who did not repeat any exercises.



and low demand conditions. On the map final exam, soldiers in the exploratory condition had
the lowest scores, with those in the low demand condition achieving the highest scores.
However, those in the exploratory condition completed their training and testing in half the time
required in the low and high working memory demand conditions. In summary, the low demand
condition was the most effective for code and map skills.

Utilization of Findings:

Not unexpectedly, too much information can be included in a block of instruction. But
blocks of instruction should not be made small just to reduce the error rate. They should not
fragment content or ignore the need to consolidate and integrate information. The exploratory
mode of training showed potential, particularly for the interactive skills and insights needed by
soldiers in working with tactical system interfaces, but might be more effective when combined
with some traditional modes of instruction. The experiments showed how CBT could be
designed to incorporate tactical system software as background instruction and demonstration
screens, and as interactive screens for performance exercises. High-fidelity training is a positive
by-product of this technique. Although the multi-media instruction was effective, the medium
per se is not an automatic panacea for training the digital skills required of soldiers and leaders.
Measurement techniques and procedures are needed that account for the multiple, yet valid
approaches a soldier can use to accomplish a task. In addition, challenges lie in developing
problem-solving scenarios that require soldiers to go beyond demonstrating technical skill
proficiency to demonstrating effective employment of digital skills in combat-like settings.

viii



age S :
Nese Fvineteas Hist ,

Mailbox SSS = BS

it demonstrate clicking he



Code Training Results

With the code training, the primary interest was in determining whether there were
differences between the High and Low Demand conditions and among the soldier groups, as well
as any interactions between these two factors. The criterion variables were scores on the
exercises and final exam, time to complete the training, and number of soldiers who had to repeat
an exercise because of failure to score at least 80%.

Comparisons were made on the following five exercise scores plus the final exam:

. Weapon and unit symbols

. Battle roster system

. Company, platoon and squad codes (for the Low condition, this score combined
the company exercise and the platoon/squad exercise)

. Rifle and weapon squad codes (for the Low condition, this score combined the
rifle squad exercise and the weapons squad exercise)

. Key leader and unit codes (for the Low condition, this score combined the key

leader exercise and the unit code exercise)

Because the first two blocks of instruction (weapon and unit symbols, the BR system) were
identical for the High and Low Demand conditions, no differences were expected on these. In
addition, because soldiers in each condition had to repeat an exercise if they scored below 80%,
differences between High and Low Demand conditions were not expected on the final exam.
Consequently, there were only three variables for which differences in the High and Low
conditions were likely to occur: Company-squad codes, rifle and weapon squad members; key
leader and unit codes. All analyses, unless stated otherwise, are based on the first time soldiers

took the exercise.

Scores on code exercises and code final exam. A two-way analysis of variance (training
condition by soldier group) was conducted on each exercise score and the final exam score (see
Table E-1, Appendix E). There were main effects for both factors, but no significant
interactions. There was one significant effect for the Low-High condition, and three significant
effects for soldier group.

The only score with a difference between the High and Low Demand conditions was on
the exercises involving the rifle and weapon squads, F(1, 159) = 23.53, p= 0000. In this case,
the soldiers in the Low Demand group scored better than those in the High Demand group.
Figure 8 shows the means for both groups on the five exercise scores and the final exam. The
graph shows clearly the impact of the High Demand condition on rifle and weapons squad code
scores. Descriptive statistics for each condition are in Table E-2.

31



Table E-16

Correlations

Correlations Among Code Scores (Low and High Demand)

Code Scores
Sym BR Co/Sqd Rifle/Wpn | Ldr/Unit Exam

Sym 1.00 .07 -.01 .00 -.06 08
BR 1.00 13 .21** .15* T*
Co/Sqd 1.00 .24** .23** .24**
Rifle/Wpn 1.00 43*F 37
Ldr/Unit 1.00 ABE
Exam 1.00

Note. N= 167.

*p<.05. **p<.0l. *** p<.001.

Table E-17

Correlations Among Code Times (Low and High Demand)

Code Times
Sym BR Co/Sqd Rifle/Wpn | Ldr/Unit_ | Exam Total

Sym 1.00 OFF | SOFFE 48*** 53 *** .19* 63 ***
BR 1.00 50*** A0*** AQ*t* .15* 50***
Co/Sqd 1.00 55*** 60*** .23** 74 tee
Rifle/Wpn 1.00 62*** .20** 86***
Ldr/Unit 1.00 27*** 82***
Exam 1.00 Agree
Total 1.00

lessons and exercises were not available.
*p<.05. **p<.0l. ***p<.001.

Note. N= 167. Times represent total of lesson and exercise times, as separate times on code

Table E-18

Correlations Between Code Scores and Code Times
Code Code Times
Scores Sym BR Co/Sqd Rifle/Wpn | Ldr/Unit | Exam Total
Sym ~.32*** | -11 -.15* -.05 -11 04 -.10
BR -.03 -.33*** | -.17* -.12 -.14 -.08 -.19*
Co/Sqd -.05 -.09 ~.46*** -.16* -.12 03 -.18*
Rifle/Wpn | -.19* -.15 -.278** -.34*** -49*** | -.09 ~42***
Ldr/Unit | -.17* -.10 -.23** -.56*** -.17* -.05 ~.3944*
Exam -.29*** | -.18* ~.37*** ~.37*F* -.344t* -15* -.42***

*p<.05. **p<.0l. *** p<.001.

E-16

Note. N= 167. Correlations between time and scores on same topic are in bold (diagonal)



a:
Using the Display

Function

D-20



items. Here the mean item difficulty was .82 for High Demand and .93 for Low Demand. It
appeared that the significantly lower scores for the High Demand condition resulted from
soldiers encountering different item formats within an exercise as well as a general lower level of
performance, presumably from the higher demands placed on working memory. Lastly, it is
important to stress that all exercises were preceded by a presentation of the type of items the
soldiers would encounter during the exercises, so soldiers would understand what was required
of them.

Significant differences occurred among the soldier groups on two of the five code
exercise scores and the code final exam, F(3, 159) = 6.49, p = .0003. The two exercise scores
with significant soldier group differences were the rifle and weapons squad score [F(3, 159) =
6.26. p = .0004] and the key leader and unit score [F(3, 159) = 5.51, p = .0012]. For each, IOBC,
ANCOC, and BNCOC soldiers scored higher than OSUT soldiers (see Table E-3). On the final
exam, IOBC also scored higher than BNCOC.

The means for the four soldier groups on all code scores are in Figure 10. The graph
shows the lack of significant differences among the groups on the first part of the instruction.
Beyond that point, the lower performance of the OSUT soldiers is demonstrated, as is the lower
performance by BNCOC on the final exam. Of additional interest is the general ordering of the
soldier groups from high to low as IOBC, ANCOC, BNCOC, and then OSUT.

100
95
lOBC
90 ANCOC
3 BNCOC
=
8 85
PS OSUT
5
o 80
=
75
70
Wpn/Unit Sym Co/Sqd BR Ldr/Unit Codes
Battle Roster RifleWpns Codes Final Exam

Code Exercises and Final Exam

Figure 10. Mean scores on the coding exercises and code final exam for each soldier group.
(Scores with significant soldier group differences are highlighted.)

34