Technical Report 1142

How Formal Training Affects Soldier Attitudes and

Behaviors Towards Digitization

February 2004

John S. Barnett

U.S. Army Research Institute

United States Army Research Institute

for the Behavioral and Social Sciences

Approved for public release; distribution is unlimited

Best Available Copy 20040416 075
U.S. Army Research Institute

for the Behavioral and Social Sciences

A Directorate of the U.S. Army Human Resources Command

ZITA M. SIMUTIS

Director

Technical Review by

Brooke Schaab, ARI

Donald R. Lampton, ARI

NOTICES

DISTRIBUTION: Primary distribution of this Technical Report has been made by ARI.

Please address correspondence concerning distribution of reports to: U.S. Army

Research Institute for the Behavioral and Social Sciences, Attn: DAPE-ARI-PO,

5001 Eisenhower Ave., Alexandria, VA 22304-4841.

FINAL DISPOSITION: This Technical Report may be destroyed when it is no longer

needed. Please do not return it to the U.S. Army Research Institute for the Behavioral

and Social Sciences.

NOTE: The findings in this Technical Report are not to be construed as an official

Department of the Army position, unless so designated by other authorized documents.
REPORT DOCUMENTATION PAGE

1. REPORT DATE (dd-mm-yy)

February 2004

2. REPORT TYPE

Final

3. DATES COVERED (from... to)

January 2003 - October 2003

4. TITLE AND SUBTITLE 5a, CONTRACT OR GRANT NUMBER

How Formal Training Affects Soldier Attitudes and

Behaviors Towards Digitization 5b. PROGRAM ELEMENT NUMBER

20262785

6. AUTHOR(S)

John S. Barnett (U.S. Army Research Institute)

5c. PROJECT NUMBER

A790

5d. TASK NUMBER

234

5e. WORK UNIT NUMBER

H01

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

U.S. Army Research Institute for the Behavioral and Social

Sciences

ATTN: DAPE-ARI-IF

5001 Eisenhower Avenue

Alexandria, VA 22304-4841

8. PERFORMING ORGANIZATION REPORT NUMBER

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

U.S. Army Research Institute for the Behavioral and Social

Sciences

5001 Eisenhower Avenue

Alexandria, VA 22304^841

10. MONITOR ACRONYM

ARI

11. MONITOR REPORT NUMBER

Technical Report 1142

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES

14. ABSTRACT CMax/mom 200 words;:

This research measured how formal training affects Soldier patterns of behavior and attitudes towards

digitization. A set of questionnaires was administered to 24 enlisted Soldiers and 12 officers attending

formal classroom training for the Force XXI Battle Command, Brigade and Below (FBCB2) digital system.

One questionnaire was administered before training, and a second after training was complete. The

questionnaires assessed Soldier opinions of FBCB2 usefulness and also their attitudes associated with

using the system. Soldier responses before and after training were compared to assess how training

affected their attitudes and behaviors towards digitization. Results showed Soldiers generally feel FBCB2

is useful and worth the additional effort required to learn the systems. It also indicated that formal training

in digital systems has a significant positive effect on Soldiers' attitudes and behaviors towards digital

systems. The results also seem to indicate that training may help Soldiers avoid maladaptive behavior

patterns that have been identified in other areas. Behaviors such as disuse or over-reliance on automated

systems, a significant problem in other areas, were relatively rare for Sddiers attending formal training.

15, SUBJECT TERMS

Digitization Training Automation FBCB2 Digital Training

SECURITY CLASSIFICATION OF

16. REPORT

Unclassified

17. ABSTRACT

Unclassified

18. THIS PAGE

Unclassified

19. LIMITATION

OF

ABSTRACT

Unlimited

20. NUMBER

OF PAGES

21. RESPONSIBLE PERSON

(Name and Telephone Number)

John S. Barnett

407/3843974
Technical Report 1142

How Formal Training Affects Soldier Attitudes and

Behaviors Towards Digitization

John S. Barnett

U.S. Army Research Institute

Simulator Systems Research Unit

Stephen L. Goldberg, Chief

U.S. Army Research Institute for the Behavioral and Social Sciences

5001 Eisenhower Avenue Alexandria, VA 22333-5600

October 2003

Army Project Number Personnel Performance and

2O262785A790 Training Technology

Approved for Public release; distribution Is unlimited.

Ill
IV
FOREWORD

Digital computer networks currently being procured and fielded by the U.S. Army have

the potential to significantly improve the tactical performance of combat units. Digitization of

the force promises to give everyone from commander to frontline Soldier a near perfect view of

the battlefield, thus significantly reducing the proverbial "fog of war." Improved information

flow makes it possible to quickly plan, coordinate, and execute operations at an increased

operating tempo, so that tasks that once took hours now take minutes.

However, the promise ofthis new technology relies on its skilled application. Leaders

and Soldiers must learn the best ways to apply digitization in order to gain the anticipated tactical

advantages. Formal training in digital systems provides an essential foundation for this learning.

Training provides digital system operators and leaders with the basic skills needed to employ

digitization.

This report is part of a series aimed at identifying the best methods for measuring

Soldiers' proficiency with digital technology. It measures how formal training affects Soldier

patterns of behavior and attitudes towards digitization. The results ofthis investigation can be

used to focus scarce training resources and time on those digital skills which have the greatest

impact on unit performance.

The work described in this report is a portion ofresearch task 234, Defining and

Measuring Digital Skill Proficiency, sponsored by the Program Executive Office for Simulation,

Training, and Instrumentation (PEOSTRI) Project Manager for Training Devices. The results of

this work were briefed to III Corps' Battle Command Training Directorate at Fort Hood, Texas

on 19 November, 2003.

MICHAEL G. RUMSEY

Acting Technical Director

V
Acknowledgement

The author would like to acknowledge the contributions of Dr. Larry Meliza, U.S. Army

Research Institute, for his help in constructing the questionnaires used in this research. I would

also like to thank the staff and instructors of the Battle Command Training Center, Fort Hood,

TX, for their support of this research.

VI
HOW FORMAL TRAINING AFFECTS SOLDIER ATTITUDES AND BEHAVIORS

TOWARDS DIGITIZATION

EXECUTIVE SUMMARY

Research Requirement:

The U.S. Army's use of networked computer systems to enhance mission

accomplishment, collectively known as digitization, has the potential to effect substantial

improvements in how military operations are conducted. However, to capitalize on the benefits

of digitization, units must invest considerable effort to train Soldiers in the skills they need to

exploit the advantages of digital systems. Given that training time is often a scarce resource, it is

important to understand how training affects outcomes. The present investigation explores how

formal classroom training affects Soldiers' behaviors and attitudes towards digitization.

Procedure:

A set of questionnaires was administered to Soldiers attending formal classroom training

for the Force XXI Battle Command, Brigade and Below (FBCB2) digital system. The

participants were 24 enlisted Soldiers and 12 officers attending either FBCB2 Operator New

Equipment Training (OPNET) or leader training. One questionnaire was administered before

training began, and a second similar questionnaire after training was complete. The

questionnaire assessed Soldiers' opinions ofthe useftilness and difficulties associated with using

FBCB2, and also their attitudes associated with using the system. Soldier responses before and

after training were compared to assess how training affected their attitudes and behaviors

towards digitization.

Findings:

Soldiers generally feel FBCB2 is useful, and is worth the additional effort required to

leam the system. They reported it is most useful for planning, preparation for combat, and

movement, but less during combat (i.e. enemy contact). These phases (planning, preparation for

combat, and movement) also were reported to entail the highest workload. Soldiers ranked the

most usefiil planning capabilities as sending and receiving messages and overlays, and route

planning. The non-planning related features ranked as most useful were sending and receiving

messages, navigating to a location, checking the location of enemy units, and avoiding threats

such as minefields. Overall, Soldiers reported that those features ofFBCB2 which involved the

highest workload and were most difficult to work with were also those rated the most useful. In

addition, Soldiers' responses indicated they are unsure ofthe accuracy of FBCB2 elements,

possibly because this topic is not covered in training.

Soldiers' responses after training were significantly different than before training for the

majority ofthe topics covered in the class. However, topics not covered during the course

showed little change from pre- to post training. Agreement among Soldiers about aspects of

digitization was generally higher after training than before, although for some questions

agreement was fairly low both before and after training.

Vll
The results also seem to indicate that training may help Soldiers avoid maladaptive

behavior patterns which have been identified in other areas. Behaviors such as disuse or overreliance on automated systems, a significant problem in other areas, were relatively rare in

Soldiers attending formal training.

Utilization of Findings:

These findings can be used to show commanders the value of formal training over other

training methods, such as on-the-job training or discovery learning. They show that formal

training provides more benefits for Soldiers than simply learning to operate digital systems.

Formal training affects Soldiers' willingness to employ digitization and may encourage them to

use digital systems more.

The findings can also be used by trainers to decide what elements of digitization and

digital systems are most difficult to learn and require the most training emphasis. In addition,

since some of the questions asked Soldiers about the usability and workload associated with

different digital tasks and tools, this information can be used by program managers to improve

system usability.

vni
HOW FORMAL TRAINING AFFECTS SOLDIER ATTITUDES AND BEHAVIORS

TOWARDS DIGITIZATION

CONTENTS

Page

INTRODUCTION 1

BACKGROUND 1

Digitization 1

Digitization Training 1

Patterns of Behavior Towards Automation 3

METHOD 4

Participants 4

Materials 5

Procedure 5

RESULTS 7

Questionnaire Section One: Usefulness of FBCB2 7

Questionnaire Section Two: Attitudes and Behaviors Towards FBCB2 10

Questionnaire Section Three: FBCB2 Monitoring Behavior 18

Questionnaire Section Four: Free Response Section 20

DISCUSSION 22

The Effect of Training 22

Soldier Ratings of Usefulness 22

Soldier Patterns of Behavior Towards Digitization 23

Teamwork Issues 24

Methodological Considerations 24

Future Research 25

Summary of Findings and Recommendations 25

SUMMARY 25

REFERENCES 27

Appendix A Acronyms List A-1

Appendix B Pre-Training Questionnaire B-1

IX
CONTENTS (continued)

Page

Appendix C Post-Training Questionnaire Q.]

List of Tables

Table 1. List OfEnlisted Participants' MOS 4

Table 2. List OfOfficer Participants' MOS 4

Table 3. Hours per Week Respondents Use Personal Computers at Work, at Home, and

Gaming 5

Table 4. How Soldiers Ranked The Usefulness OfFBCBl Planning Capabilities (Question 1)

8

Table 5. How Soldiers Ranked The Usefulness OfFBCBl Capabilities Other Than Planning

(Question 2) g

Table 6. How Soldiers Ranked When FBCB2 Was Most Useful (Question 3) 9

Table 7. How Soldiers Ranked When FBCB2 Workload Was The Highest (Question 4) 9

Table 8. How Soldiers Ranked The Accuracy OfFBCB2 Display Elements (Question 5j .... 10

Table 9. Frequencies OfSoldier Responses To Question 17 "If You Wanted To Hand-Off

Monitoring The FBCB2 Display To Someone Else, Who Would You Hand It Off To?"

21

Table 10. Frequencies OfSoldier Responses To Question 18 "What Feature Of The FBCB2 Is

The Most Difficult To Work With?" 21

List of Figures

Figure 1. Example of Force XXI Battle Command Brigade and Below (FBCB2) display 2

Figure 2. Example of question from section one of the questionnaire 7

Figure 3. Comparison of frequencies of Soldier responses before and after training for

statement 6 "Sometimes the FBCB2 does things for no reason or that I don't

understand." U

Figure 4. Comparison of frequencies of Soldier responses before and after training for

statement 7 "I don't use FBCB2 if I can avoid it." 12

Figure 5. Comparison of frequencies of Soldier responses beft! e and after training for

statement 8 "FBCB2 really helps me do my job." 13

Figure 6. Comparison of frequencies of Soldier responses before and after training for

statement 9 "FBCB2 isn't worth the amount of effort you have to put into it.".... 13

Figure 7. Comparison of frequencies of Soldier responses before and after training for

statement 10 "I only use FBCB2 to do a few things." 14

Figure 8. Comparison of frequencies of Soldier responses before and after training for

statement 11 "I know the information I get from FBCB2 is correct." 15

Figure 9. Comparison of frequencies of Soldier responses before and after training for

statement 12 "If the FBCB2 isn't working right, it's best to try to get it working rather

than revert to analog." j5
CONTENTS (continued)

Page

Figure 10. Comparison of frequencies of Soldier responses before and after training for

statement 13 "FBCB2 is useful to do a lot of things, but I still check to make sure the

information it gives is correct." » 17

Figure 11. Comparison of frequencies of Soldier responses before and after training for

statement 14 "Sometimes there are easier ways to do things than what's taught in

class or shown in the book." 18

Figure 12. Comparison of frequencies of Soldier responses before and after training for question

15 "How often would you look at the FBCB2 display after you crossed the Line of

Departure (LD)?" 19

Figure 13 Comparison of frequencies of Soldier responses before and after training for question

16 "How often would you hand-off the requirement to monitor the FBCB2 display to

someone else?" 20

XI
Xll
HOW FORMAL TRAINING AFFECTS SOLDIER BEHAVIORS AND ATTITUDES

TOWARDS DIGITIZATION

Introduction

This research investigates how Soldiers' attitudes and behaviors towards digital systems

change as they undergo formal training. The effort had several objectives. The first was to

determine how formal training affected Soldiers attitudes and behaviors towards digital systems,

and also how it changed their use ofthe systems. The second was to identify what capabilities of

digital systems, in this case the Force XXI Battle Command, Brigade and Below (FBCB2)

Soldiers found more or less useful. The last objective was to identify if Soldiers exhibited certain

maladaptive and adaptive behavior patterns towards digital systems which have been identified

in other areas where users interact with automated systems. The effort did not assess training

methods, rather, it measured the results ofthe training.

Before describing how this research was conducted, I will provide some background on

U.S. Army digitization, how digitization is trained, and also discuss maladaptive and adaptive

patterns of behavior towards automation.

Background

Digitization

In the U.S. Army, digitization consists of networked computer systems which are used to

improve communication, mission plarming, and integration among Battlefield Operating Systems

(BOS). There are a number of digital systems, most of which are tailored to support a specific

BOS, but which can share information with other systems (for a more detailed description of

digitization, see Barnett, Meliza & McCluskey, 2001).

The digital system studied in this research was the FBCB2 system. FBCB2 is used by

lower echelon (as the name implies, below brigade level) maneuver units both in Tactical

Operations Centers (TOC) and in individual combat vehicles. In TOCs, FBCB2 is used to plan

missions, send orders and battlefield graphics, and coordinate combat support functions. On

vehicles, it is used to receive orders and graphics, plan routes and navigate to specific locations,

as well as enhancing Soldier's situation awareness ofthe battle space. Most vehicles with

FBCB2 are able to automatically report their position using onboard Global Positioning System

(GPS) equipment. Thus, commanders and Soldiers can view the position oftheir vehicles and

others in their unit in real time. An example ofthe FBCB2 display is shown in Figure 1.

Digitization Training

Formal training for digital systems is conducted at the Battle Command Training Center

(BCTC) at Fort Hood, TX. BCTC provides Operator New Equipment Training (OPNET), leader

training, and staff officer training for a number of different digital systems. OPNET for FBCB2

is normally approximately 40 hours oftraining conducted over five days; 36 hours of instruction

with a four-hour practical examination. There are usually three instructors per class; one

instructor presents the material while the other two assist students. The leader course is similar

to OPNET training, but includes more tactical considerations.

The class normally consists of 20-25 Soldiers using desktop computers (training systems

or "white boxes") with FBCB2 software. The computers are connected over a local area network

(LAN) so that each classroom simulates an interconnected unit. For one 8-hour block of
instruction the students moved to another classroom to work with real FBCB2 hardware ("green

boxes") to learn equipment inspection and pre-combat checks. The desktop computers use a

mouse and keyboard for input, while the actual systems use a touch screen and track ball-like

device. Although "white boxes" have different input methods than "green boxes," Soldiers

report no problem switching between input devices.

' ';::! OZJo;-!..Qfr. i'c^i X0:

uwcLflsaPitp ^

I >k.

»^' •** Jt^ *

Figure 7. Example of Force XXI Battle Command Brigade and Below (FBCB2) display.

Most of the blocks of instruction are followed by a practical exercise where the students

are able to practice what they learned. The blocks of instruction and exercises follow a natural

progression including system startup, pre-combat checks, movement, assembly area operations,

and actions on the objective. The training assumes the Soldiers have a basic tactical knowledge

including familiarity with reports, map symbology, and battlefield graphics. It also assumes a

basic knowledge of computer operation, such as using a computer mouse and keyboard, as well

as familiarity with common desktop computer terminology. Previous experience with common

desktop office software is helpful since the interface uses many conventions from office

software, particularly the drawing tools used to draw battlefield graphics.

However, many Soldiers work with FBCB2 before they receive formal training because

the formal training is often difficult to squeeze into the unit training schedule. They often learn

about FBCB2 through on-the-job-training (OJT) or through a tutorial included with some of the
FBCB2 software. Thus, some of the Soldiers attending BCTC classes have digitization

experience prior to classroom training.

Patterns ofBehavior Towards Automation

Research has shown that people's attitudes and behaviors towards automation tend to

form pattems. A decade of study in commercial aviation shows certain patterns in how people

react to the introduction of automation into their work structure, and how they interact with the

automated systems. Many of these behavior patterns interfere with the proper use of automation.

Some users may avoid using automation all together, and thus lose the advantages automation

offers. Conversely, other users may rely too much on automation, which often leads them into

unpleasant or dangerous situations (Parasuraman & Riley, 1997). Similar pattems are beginning

to show in other areas where automation has been introduced, such as surface transportation

(Parasuraman & Riley, 1997) and medicine (Sheridan & Thompson, 1994).

Not surprisingly, initial studies and field observation seem to indicate Soldier's attitudes

and behaviors towards digitization follow similar pattems. A recent study by Bamett & Meliza,

(2003) showed concerns voiced by Soldiers about digitization, such as erosion of skills, system

reliability, and increased workload, mirror concerns expressed by commercial air crews

regarding automation.

If Soldier's behaviors towards digitization follow known pattems of behavior in relation

to automation, this information can be useful in terms of integrating digitization into U.S. Army

units. Commanders, trainers, and evaluators can use this knowledge to help them assess their

unit's level of digital training and tailor training to be most effective.

Typical behavior pattems. A review of previous research and theory suggests that users

seem to exhibit certain behavior pattems in response to working with automated systems (see

Parasuraman & Riley, 1997). These pattems may reflect their attitudes towards automation, and

may change as attitudes change. For example, people are often skeptical of automated systems

at first, and will resist learning about them and avoid using them if they can. Others will use the

systems, but only in a limited way. Sometimes people with more computer experience may

actually rely too much on automation, which often causes its own set of problems. Conversely,

other users rely on it in a more realistic way, possibly due to more experience or expertise with

the systems.

Thus, the four pattems of behavior of interest in this research are:

• Avoidance or disuse behaviors

• Limited use behaviors

• Ovemse or over-reliance behaviors

• Expert behaviors

In addition to the other research questions, the present research sought to identify if

Soldiers exhibited any of these four behavior pattems. Previous field observations had identified

disuse and limited use behavior pattems in some digital units. I wanted to see if Soldiers

attending formal digitization training exhibited any of these behaviors, and, if so, how the formal

training affected the behaviors.
Method

Pa?'ticipants

Participants were 36 Soldiers attending classes at the BCTC. The sample contained 24

enlisted Soldiers from Sergeant (E5) to Master Sergeant (E8), and 12 officers from Second

Lieutenant (01) to Major (04). Soldiers below E5 were not selected to participate because it

was felt their participation would take up too much class time. There was a wide range of

Military Occupational Specialty (MOS) codes, indicating participants came from a diverse

background. Tables 1 and 2 presents the list of MOS's.

Table 1. List OfEnlisted Participants' MOS

Enlisted MOS

MOS Series Functional Area MOS Series Functional Area

11B/M Infantry 75H Administration

12B Engineering 77F Petroleum

14S Air Defense 79S Recruiting

19K Armor 88M Transportation

44B 52C 63B/H/Z Maintenance 95B Military Police

Table 2. List OfOfficer Participants' MOS

Officer IVIOS

MOS Series Branch MOS Series Branch

12A/C Engineer 74A Civil Affairs

14B Air Defense 91A Chemical Corps

25A Signal Corps 92A/D Ordinance

38A Quartermaster

For enlisted Soldiers, the range of years of service ranged from 2 years ofservice to over

20 years (M=l 1.6, 5D=5.86). Officers' years ofservice ranged from less than a year to 22 years

(M=7.77,5D=7.18).

Participant Computer Experience. When asked about experience with FBCB2 and

general computer experience, enlisted Soldiers reported they had used FBCB2 for less than 12

months {M=\A6, SD=2.S7) with many (15) reporting they have no experience with FBCB2.

When asked how many hours a month they use FBCB2, most (16) reported they did not use it,

while those that did averaged 2.6 hours a month (SD=5.34). Officers reported they used FBCB2

six months or less (A/=l. 1, S'Z>= 1.78), with the majority (7) reporting they had not used it yet.

The distributions for all of these statistics were positively skewed, probably because ofthe large

number who reported no FBCB2 use.

Participants were also asked how many hours a week they used computers at work and at

home, and also how many hours they played computer games. Enlisted Soldiers reported they

used computers at work an average of 15.3 hours a week (SD=] 8.8), and at home an average of

12.4 hours (SD=\3A). They also reported they played computer games an average of 3.7 hours a

week (S[)=7.67). Officers said they used computers at work an average of 22.5 hours a week

(S'D==16.7) and 12.8 hours at home {SD=9.7). Officers also said they played computer games an

average of 4.5 hours a week (SD=5.7). Again, these statistics were positively skewed, which can
probably be attributed to the number of participants who reported no computer use at work or at

home. Table 3 provides a comparison of how many hours a week Soldiers who responded to

these questions use PCs at work, at home, and for gaming.

Table 3. Hours per Week Respondents Use Personal Computers at Work, at Home, and Gaming

Respondent 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Hours/Work 60 48 40 40 30 25 25 20 15 10 10 10 8 6 0 0 0 0

Hours/Home 4 25 20 5 30 20 10 20 20 40 15 4 6 1 7 1 1 0

Hours/Gaming 1 3 15 2 15 0 10 0 3 30 4 0 14 0 1 1 1 0

A cross correlation matrix was constructed to identify relationships between participant

demographics and both digital and general computer experience. Scatterplots showed the

variables were not linearly related, thus Spearmans' Rho was chosen over the Pearson Product

Moment Correlation. Significant correlations included a positive correlation between rank and

hours of computer use at work (rs = .381, p.< .026), a negative correlation between time in

service and playing computer games (rs = -.447, p.< .008), a high positive correlation between

months of using FBCB2 and hours a week using FBCB2 (rs = .935, p.< .000), and a positive

correlation between home computer use and playing computer games (rs = .676, p.< .000).

Materials

The materials consisted of a set of questionnaires, one administered pre- and one posttraining. In addition to demographics, the questions were organized into four sections. The first

section asked Soldiers questions about the usability, workload, and accuracy of FBCB2 and had

them rank order a set of responses. The second section consisted of Likert-type questions that

asked Soldiers to agree or disagree with statements related to typical behaviors towards

automation. Section three asked questions about how Soldiers monitor digital systems and had

Soldiers select from a series of responses. The last section asked several free response questions

about system monitoring and about which tasks were the most difficult to perform.

The questionnaires also solicited demographic information as well as information on the

individual's digital and general computer experience. Similar questions were asked before and

after training. The order of questions on the post-training questionnaire was changed, and some

of the questions were reworded to reduce the chance of the respondents being influenced by

remembering the question from the pre-training questionnaire. The questions are discussed in

the results section. Examples of the questionnaires are in Appendix B.

Procedure

The sets of questionnaires were distributed among the participants at the beginning of the

classroom training. We decided that it would be too time consuming and disruptive of the class

to collect the first questionnaire and then distribute the second questionnaire at the end of

training. This would require the participants to remember participant numbers which would put

an undue burden on both the participants and instructors. Therefore, we delivered both

questionnaires in an envelope which was kept at the student workstation throughout the training.

To ensure the questionnaires were completed in the correct order, they were clearly marked on

the first page with instructions about the order of completion. In addition, the second

questionnaire was printed on paper of a different color (yellow) than the first questionnaire
(white). The second questionnaire was also folded and sealed to further reduce the probability of

Soldiers competing the questionnaires in the wrong order.

At the beginning of the class, participants were briefed on the purpose of the research and

reminded that their participation was voluntary. Participants were then instructed to complete

the first (white) questionnaire and return it to the envelope. After training was completed at the

end of the week, they were instructed to remove the second (yellow) questionnaire, break the

seal, complete the questionnaire, and return it to the envelope. The envelopes were then

collected.

Both questionnaires and the envelope were identified by a participant number.

Participants were briefed that they could mark their names on the envelope if they wished to

prevent mix-ups but not to put their names on the questionnaires. Once the envelopes were

returned, we separated the questionnaires from the envelopes to ensure participant anonymity.
Results

The questionnaire was in four sections, each soHciting information in different ways.

Each of these sections will be discussed in turn.

Questionnaire Section One: Usefulness ofFBCB2

The first section asked Soldiers questions about FBCB2 and had them rank order a set of

responses. The questions sought to identify how Soldiers ranked the importance of aspects of

FBCB2, and also how their view of the importance of these functions changed with training.

Figure 2 shows an example of this type of question. The complete questionnaires are found in

Appendix B.

1. What do you consider the most useful planning capabilities with FBCB2? Please rank order from most

useful (number 1) to least useful

Plan a route Plan support measures

Send/receive messages/plans /overlays Line-of-sight tools

Other

Figure 2. Example of question from section one of the questionnaire.

There are three questions I want to answer about this section; (1) how did Soldiers rank

the statements, (2) how much agreement was there among the Soldier's ratings, and (3) was there

a significant difference between how a statement was rated pre-test and post-test?

To answer the first question, I initially looked at the mode of the rankings of statements.

Unfortunately, many of the statements were multi-modal with two and sometimes three

statements receiving the same frequency of rankings. Therefore, I calculated the mean rank

using Kendall's W test. Kendall's W, or Kendall's Coefficient of Concordance, determines the

agreement among raters and, as a byproduct, calculates the mean rank of each statement. Mean

rank is more descriptive than mode in this case because it avoids the problem of multiple modes

and provides a better idea of how statements were rated by the population. Since the participants

ranked the statements from 1 (highest rank) to k (lowest rank), statements with lower numbers

were rated higher.

In addition, Kendall's W also answered our second question about how much agreement

there was among the raters. Kendall's W statistic rates agreement as values from zero to one,

with zero being no agreement among raters, to one being perfect agreement. The test for

significance of the W statistic is a chi-square which determines how the calculated statistic

differs from chance. (Siegel, 1956). The results of these analyses are discussed next.

Question 1. The first question asked, "What do you consider the most useful planning

capabilities with FBCB2?" The response statements were, "Send/receive messages/plans/

overlays," "Plan a route," Line-of-sight tools," and "Plan support measures." The analysis of

this question is summarized in Table 4, which shows the mode, mean rank, and Kendall's Wfor

each statement both before and after training. The Soldiers generally ranked the usefulness of

these capabilities as shown in the Table, except that "Plan support measures" got a higher rating

than "Line-of-sight tools" on the pre-test, but was rated lower on the post-test. Kendall's W
showed there was moderately low agreement about the rankings on the pre-test, but agreement

was much higher for the post-test. In both cases, the statistic differed significantly from chance

and exceeded the p<.05 confidence level.

Table 4. How Soldiers Ranked The Usefulness Of FBCB2 Planning Capabilities (Question J)

Statement" Pre-training Post-training '^^^^^^^m. Mode M Rank^ W Mode MRank W

Send/receive messages/overlays 1 1.70 .247

(X'=20.0,

df=3,

p<.000)

1 1.47 .398

(X'=35.8,

df=3,

p<.000)

Plan a route 1 2.26 2 2.23

Line-of-sight tools 4 3.07 4 3.10

Plan support measures 3 2.96 1 3.20

""Mean Rank from Kendall's VVtest.

' Kendall's Coefficient of Concordance (W).

Question 2. The second question was, "What do you consider the most useful FBCB2

capabilities other than planning?" and asked respondents to rank order statements from most

useful (number one) to least useful. The statements are listed in Table 5, along with the analysis

of the data. The Kendall's W shows there was little agreement among respondents either before

or after training, although there was more agreement after training than before. The modes show

the responses to be multi-modal, and there is little variance between the mean ranks, both of

which probably reflect the lack of agreement among Soldiers as to the usefulness of FBCB2

capabilities.

Table 5. How Soldiers Ranked The Usefulness OfFBCBl Capabilities Other Than Planning

(Question 2)

Statement

Send/receive messages

Navigate to a location

Check location of enemy units

Avoid threats (minefields, etc.)

Check location of subordinate unit

Logistics/personnel reporting

Check status of subordinate unit

"Statements are listed in post-training rank order.

Mean Rank from Kendall's VVtest.

" Kendall's Coefficient of Concordance (W).

Pre-training

M Rank"

4.11

3.83

3.52

3.17

3.37

5.28

4.72

VV

.126

(X'=20.0,

df=6,

p<.002)

Mode

Post-training

MRank

3.15

3.27

3.47

3.60

3.82

4.97

5.73

W

.203

(X'=36.5,

df=6,

p<.000)

One possibility for the lack of agreement is that the participants came from a variety of

different backgrounds and therefore used FBCB2 for different purposes. There were 17 different

MOS's among 36 participants, which is a clearly diverse group. This may account for them

rating the usefulness of certain functions differently. To test this idea, I ran a Kendall's Won a

select group of Soldiers with similar MOS's. I chose participants with infantry (11 series) and

armor (19 series) MOS's since these Soldiers should use raCB2 for similar purposes.

Unfortunately, even by combining infantry and armor MOS's, the n was still too small to give a

confidence interval better than p<.05. Thus this test gave us no useful information.
Question 3. This question asked Soldiers "When is FBCB2 most useful to you?" and

provided a list of combat phases for them to rank order. The phases and analysis of the data are

shown in Table 6. The table shows there was moderate agreement prior to training, but

agreement was considerably higher post-training. Post-test modes and mean ranks show that

Soldiers feel that FBCB2 is most useful during planning, preparation for combat, and movement

phases, whereas it is least useful during assembly area operations and actions on the objective.

Table 6. How Soldiers Ranked When FBCB2 Was Most Useful (Question 3)

Statement ^ Pre-training Post-training

^^^^^^^^^^^^^^^P Mode MRank" W' Mode MRank W

Planning 1 3.04 .230

(x'=38.5,

df=6,

p<.000)

1 2.17 .402

(X'=72.3,

df=6,

p<.000)

Preparation for combat 4 3.93 2 2.63

Movement 1 2.29 3 2.93

Attack 4 4.07 6 4.68

Defense 6 4.57 4 4.90

Assembly area operations 7 5.07 7 5.33

Actions on the objective 7 5.04 7 5.35

' Statements are listed in post-training rank order.

''Mean Rank from Kendall's Wtest

' Kendall's Coefficient of Concordance (W).

Question 4. This question asked Soldiers, "When is raCB2 workload the highest?" and

provided the same list of combat phases as the previous question. The data analysis is

summarized in Table 7. The analysis shows that rater agreement was fairly low both before and

after training, although it did increase slightly after training. It is interesting to note that the

rankings of combat phases for this question are roughly similar to the previous question which

asked about when FBCB2 was most useful. Although the ranking for the first two phases in

question 3 are reversed for question 4, the mode and mean rank statistics show these were very

close in ranking. The similarity between the ranking of these phases suggests that when FBCB2

is most useful, it also has the highest workload.

Table 7. How Soldiers Ranked When FBCB2 Workload Was The Highest (Question 4)

Statement * Pre-training Post-training

^^^^^^^^^^^^^^^B Mode MRank" W' Mode MRank W

Preparation for combat 3 3.77 .213

(X'=33.1,

df=6,

p<.000)

2 2.42 .260

(X'=40.5,

df=6,

p<.000)

Planning 1 2.81 1 2.62

Movement 1 2.62 3 3.96

Attack 6 4.04 4 4.23

Defense 7 4.96 5 4.50

Assembly area operations 7 4.92 7 5.12

Actions on the objective 7 4.88 7 5.13

* Statements are listed in post-training rank order.

"Mean Rank from Kendall's Wtest.

"^ Kendall's Coefficient of Concordance (W).
Question 5. The last question in this section asked, "On the FBCB2 display, which of the

following is usually closest to ground truth (most accurate)?" The responses and the analysis of

the data is shown in Table 8. For this question the M^ statistic for pre-training agreement dvcs not

meet the p<.05 confidence level. However, the post-training W statistic does provide .sufficient

confidence, and shows that even after training the agreement among Soldiers was low. This

analysis suggests Soldiers are unsure about the accuracy of FBCB2.

Table 8. How Soldiers Ranked The Accuracy OfFBCB2 Display Elements (Question 5)

Statement" Pre-training Post-training 1 ^^^^^^^^^^^^^^^B Mode M Rank^ W Mode MRank W

Friendly icons (center of mass) 3 3.19 .047

(X'=4.9,

df=5,

p<.423)

1 2.57 .161

(x'=16.9,

df=5,

p<.005)

Terrain features 1 3.48 1 3.05

Friendly icons (individual

platforms)

1 2.95 2 3.33

Overlays 3 3.86 1 3.43

Line-of-sight analysis 3 3.48 2 3.86

Enemy icons

a c » , 1- . . • . .

5 4.05 6 4.76

''Mean Rank from Kendall's VVtest.

" Kendall's Coefficient of Concordance (W).

Differences after training. I also wanted to determine if Soldier ratings of usefulness,

workload, and accuracy of FBCB2 changed after training. Therefore, for every question I

compared how Soldiers ranked each item before training with the ranking they gave it after

training using the Wilcoxon Signed Ranks test. The Wilcoxon Signed Ranks test is a

nonparametric test which determines if a significant difference exists between matched pairs of

scores using Z scores (Siegel, 1956). For all questions in this section I were interested in

differences which exceeded the p<.05 confidence level (two-tailed).

Question one had no differences which exceeded the set confidence level. Question two

asked which were the most useful FBCB2 capabilities other than planning. There were two

responses which differed significantly; "Check status of subordinate units" (Z=-2.45, p<.014),

and "Send/receive messages" (Z= -2.21, p<.027), both of which were ranked as less important

after training.

Question three, which asked when FBCB2 was most useful also had two responses which

were ranked differently after training. "Planning" (Z= -2.58, p<.01) and "preparation for

combat" (Z=-3.24, p<.001) were both rated higher after training. Question four asked when

FBCB2 workload was the highest. "Preparation for combat" (Z=-3.12, p<.002) increased in

ranking after training, and "movement" (Z=-3.34, p<.001) decreased after training. Question

five only had one statement which differed significantly. This question asked about the accuracy

of display elements in FBCB2. Soldiers ranked "friendly icons (center of mass)" more highly

after training than before (Z=-2.56, p<.011).

Questionnaire Section Two: Attitudes and Behaviors Towards FBCB2

The second section consisted of Likert-type questions that asked Soldiers to agree or

disagree with statements. Soldiers were given nine statements and asked whether they strongly

disagreed, disagreed, agreed, or strongly agreed with the statement. A fifth option was "Don't'

10
know." The nine questions were cross-correlated using Spearman's Rho to ensure there were no

problems of singularity, that is, questions which measure the same construct (Tabachnick &

Fidell, 1996). The highest correlation (rs = .676, p.<.000) was between questions 7 ("FBCB2

really helps me do my job.") and 12 ("Even when it's not working perfectly, digitization is better

than analog.") on the post-training questionnaire. Since none ofthe questions were highly

correlated (.9 or greater) with each other, I assumed there was no singularity.

This section sought to identify Soldier attitudes and behaviors towards digitization and

determine ifthey are similar to attitudes and behaviors reported towards automated systems in

other areas. The behaviors of interest were avoidance or disuse behaviors, limited use behaviors,

and overuse behaviors. For a discussion of use, disuse, and overuse of automation, see

Parasuraman and Riley (1997). In addition, several ofthe questions sought to identify Soldiers

who exhibited behaviors thought to show a deeper understanding of automation strengths and

weaknesses (expert behaviors).

The first statement in this section was "Sometimes the FBCB2 does things for no reason

or that I don't understand." This question seeks to measure disuse/avoidance behaviors, and it

was expected that Soldiers who agree with this statement would tend to avoid using FBCB2.

Figure 3 shows a bar graph comparing Soldier responses pre- and post-training. The dark bars

"Sometimes the FBCB2 does things for no reason or that I don't understand"

HPre-training Q Post-training

25

20

Strongly Disagree Disagree Agree Strongly Agree

are pre-training responses and the light bars show post-training responses.

Figure 3. Comparison offrequencies of Soldier responses before and after training for statement

6, "Sometimes the FBCB2 does things for no reason or that I don't understand."

The figure shows that before training responses were fairly evenly split between

agreement (56%) and disagreement (44%), and also that 20 out of 36 respondents answered

11
disagreeing with the statement. This suggests Soldier felt they had a better understanding of

FBCB2 after training.

The next question directly assessed avoidance behaviors by stating "I don't use FBCB2 if

1 can avoid it." Figure 4 shows the responses to this statement before and after training.

"I don't use FBCB2 If I can avoid it"

iPre-training D Post-training !

Strongly Disagree Disagree Agree Strongly Agree Dont Know

Figure 4. Comparison of frequencies of Soldier responses before and after training for statement

7, "I don't use FBCB2 if I can avoid it."

In this case, the figure shows most Soldiers disagree that they avoid using FBCB2 both

before and afte--raining. Before training, four Soldiers agreed (18% of those who responded)

whereas 18 di ?,d (82%) and 14 did not know or did not respond. After training, 29

disagreed (91' id only 2 agreed (9%). A higher percentage of the Soldiers responded after

training (29 out... 36, or 81%), versus before training (only 22 out of 36, or 61%).

The next statement was also designed to assess avoidance behaviors. The statement was

"FBCB2 really helps me do my job." It was felt that Soldiers who avoided using FBCB2 would

disagree with this statement. Figure 5 shows how Soldiers responded to this statement.

This figure shows the patterns of responses before and after training were roughly the

same. A high percentage of Soldiers who responded agreed with the statement before (96%) and

after training (93%). More Soldiers strongly agreed with the statement after training (3 before, 7

after), but also one more Soldier disagreed after training (1 before, 2 after).

The next statement was "FBCB2 isn't worth the amount of effort you have to put into it."

This statement also sought to assess avoidance behaviors, and it was feU that Soldiers who

agreed would probably avoid using FBCB2. Responses to this statement are shown in Figure 6.

12
"FBCB2 really helps me do my job"

IPre-training D Post-training |

20

IS

10

21

„

10

7

; 1

2

3

strongly

0

Disagree

0 a^M'-'--'-

Agree Strongly Agree

IIDon't Know

Figure 5. Comparison of frequencies of Soldier responses before and after training for statement

8, "FBCB2 really helps me do my job."

' FBCB2 isn't worth the amount of effort I have to put into it"

I Pre-training D Post-training |

25

20

10

1

6

1

7

•;•! 15

1

14 14

^^1 4

1

0 0 0 \.-.-.-.\

Strongly Disagree Disagree Agree Strongly Agree Don't Know

Figure 6. Comparison of frequencies of Soldier responses before and after training for statement

9, "FBCB2 isn't worth the amount of effort you have to put into it."

13
The figure shows that most Soldiers disagreed with this statement both before and after

training. There were more "strongly disagree" statements after training, and fewer "don't know"

statements.

Question 10 attempted to assess whether Soldiers used FBCB2 in a limited way.

Research from other areas on automation use suggests some users employ a few automated tools,

but do not leam the entire range of automated aids open to them. Soldiers were asked to respond

to the statement "I only use FBCB2 to do a few things." The frequencies of responses to this

question are shown in Figure 7.

"I onfy use FBCB2 tr. few things"

Strongly Disagree Disagree Agree Strongly Agree Dont Know

Figure 7. Comparison of frequencies of Soldier responses before and after training for statement

10, "I only use FBCB2 to do a few things."

The figure indicates that prior to training the majority of responses was "don't know"

with the remaining responses leaning towards disagreement. However, after training the

majority of responses (50%) strongly disagreed with the statement and another 36% disagreed.

In fact, there was only one "agree" response.

The next several questions dealt with overuse behaviors. Sometimes users tend to trust

automation too much (Sarter & Woods, 1997) or use it when manual procedures are easier or

more appropriate (Parasuraman & Riley, 1997). Question 11 sought to assess how Soldiers

viewed the accuracy of FBCB2 information. The statement was "I know the information I get

from FBCB2 is correct." The intent of the statement was to suggest that information from

FBCB2 is always correct. I believed that Soldiers who agreed with this statement would tend to

place too much trust in the automated system, which has been referred to as "automation bias"

(Mosier, Skitka, Heers, & Burdick, 1998). Figure 8 shows the responses to this statement.

14
As Figure 8 shows, most Soldiers were not sure about the accuracy ofFBCB2 before

training, although a number agreed that they "know" information from FBCB2 is correct. After

training the majority of Soldiers disagreed that they "know" FBCB2 information is correct. This

suggests the training may have given them a more realistic view ofinformation accuracy, and

consequently it may help Soldiers avoid automation bias.

I know that infonnation I get from FBCB2 is correct"

I Pre-training D Post-training

25

20

IS

3

0 0

strongly Disagree Disagree Agree Strongly Agree Don't Know

Figure 8. Comparison offrequencies of Soldier responses before and after training for statement

11, "I know the information I get from FBCB2 is correct."

Next, statement 12 was designed to identify between avoidance or disuse behaviors. The

statement said, "Ifthe FBCB2 isn't working right, it's best to try to get it working rather than

revert to analog." I expected that Soldiers who disagreed would tend to avoid using digital

systems, reasoning that manual (analog) is better and ifthe system isn't working they should

immediately revert to manual. Originally, I had hoped this question would also identify overuse

behaviors, since one ofthe typical behaviors of overuse is focusing on fixing a malfiinctioning

system at the expense of performing the immediate task (Parasuraman & Riley, 1997).

However, 1 realized that Soldiers who did not tend towards overuse could also agree with the

statement. The responses are shown in Figure 9.

The responses both before and after training are very similar, as shown in the figure. The

main difference is fewer Soldiers chose the "don't know" response after training. Like other

questions, it suggests Soldiers had a better idea ofthe parameters ofthe system after training and

consequently expressed their opinion rather than choose "don't know." Most Soldiers (60%)

agreed with the statement.

15
Statement 13 said, "FBCB2 is useful to do a lot of things, but I still check to make sure

the information it gives is correct." Like statement 11 mentioned above, this statement was

designed to identify overuse or automation bias behaviors. I expected that Soldiers who

disagreed with the statement were biased in favor of automation and would show overconfidence

behaviors. A bar graph of responses is shown in Figure 10.

The figure sh( vs that the principle change from pre-training to post-training is fewer

Soldiers chose the "don't know" response after training. The pattern of agreement is roughly the

same, with most Soldier agreeing with the statement. Before training, 63% of those responding

chose agree or strongly agree, while after training 80% chose these responses. However, on the

other hand, after training 11% of the Soldiers chose to disagree with the statement. The results

from this statement seem to suggest that after training Soldiers had a better idea of how far they

could trust FBCB2 than before training.

"If the FBCB2 isn't working right, its best to try to get it working rather than revert to analog"

Strongty Disagree Disagree Agree Strongly Agree Don't Know

Figure 9. Comparison of frequencies of Soldier responses before and after training for statement

12, "If the FBCB2 isn't working right, it's best to try to get it working rather than revert to

analog."

The last statement was designed to measure Soldier's overall level of expertise with

FBCB2 and digitization in general. It is based on the idea that experts prefer shortcuts and use

more abbreviated procedures (Eberts, 1994) and when none are available they may develop their

own. The statement was, "Sometimes there are easier ways to do things than what's taught in

class or shown in the book." Responses to this statement are shown in Figure 11. As the figure

shows, before training most of the Soldiers (83%) chose neither to agree or disagree. After

training, they were fairly evenly split between agreement and disagreement.

16
"FBCBa is useful to do a lot of things, but I still check to make sure the information it gives is

correct"

IPre-training Post-training [

Strongly Disagree Disagree Agree Strongly Agree Doni Know

Figure 10. Comparison of frequencies of Soldier responses before and after training for

statement 13, "FBCB2 is useful to do a lot of things, but I still check to make sure the

information it gives is correct."

Differences between pre- and post-training ratings. To identify significant differences

between pre- and post-training responses, a Wilcoxon signed-ranks test was performed for

matched pairs. The Wilcoxon signed-ranks test measures the direction of differences between

ratings of related pairs, but also considers the magnitude of the differences. Therefore, the

Wilcoxon is a more powerful test than the Sign test (Siegel, 1956). Even so, response rates for

this section were fairly low because some respondents either did not answer or chose the "don't

know" response. In either case, these pairs of responses were eliminated from the analysis. This

accounts for the low n of most of the statements in this section.

Two statements, 10 and 11, showed significant differences. Statement 10 "I only use

FBCB2 to do a few things," showed significantly less agreement with the statement after training

(z = -3.22, p<.001), which suggests Soldiers were willing to use FBCB2 for more tasks or use

more of the tools, or both.

In addition, statement 11, "The information on FBCB2 is almost always correct, so I

don't bother checking it," also showed significantly less agreement after training (z = -3.464, p<

.001). This could mean that Soldiers had a better appreciation of the accuracy of digital

information, or at least of the need to cross-check information. Since failing to cross-check

information from automated systems is a behavior associated with over use or over reliance on

automation, the training may reduce the chances of Soldiers engaging in these behaviors.

Statement six, "Sometimes the FBCB2 does things for no reason or that I don't

understand," (see Figure 3) had relatively few usable responses (n = 15). The low n is primarily

17
a result of "don't know" responses which were not counted in this analysis. Although the results

do not meet the error probability criteria of p.< .05 (z = -1.82, n.s.), an examination of Figure 3

shows there were more disagreement responses after training than before, which suggested there

may be a difference between pre- and post-training responses which is not evident, perhaps due

to an intervening variable. Our search for this variable revealed that Soldiers who had previous

FBCB2 experience responded differently from those with no previous experience. When I

anahved each of these groups independently, I found Soldiers with previous FBCB2 experience

sho\ d a significant difference in responses before and after training (z = -2.43, p < .015).

These Soldiers reported they had a better under.'- anding of FBCB2 after training.

Sol- etimes there are easier ways to do thinqs on FBCB2 than what's taught In class or

shown in the book"

Strongly Disagree Disagree Agree Strongly Agree Dont Know

Figure 11. Comparison of frequencies of Soldier responses before and after training for

statement 14, "Sometimes there are easier ways to do things than what's taught in class or shown

in the book."

Statement 14, "Sometimes there are easier ways to do things on FBCB2 than what's

taught in class or shown in the book" (see Figure 11), also has a low response rate (« = 15) and

does not meet the .05 confidence level (z = -1.63, n.s.).

Questionnaire Section Three: FBCB2 Monitoring Behavior

Section three consisted of two questions. For each question Soldiers were asked to select

from a series of responses. The first question was, "How often would you look at the FBCB2

display after you crossed the Line of Departure (LD)?" The purpose of this question was to

gather data on how Soldiers employed the FBCB2 display during movement. The possible

responses were (a) "Never," (b) "Every few minutes," (c) "Every few seconds," (d) "If I'm

lost/confused or when approaching the next decision point," and (e) "Other." Soldiers responses

are shown in Figure 12.

18
"How often would you look at the FBCB2 display after crossing the Line of Departure?"

|wPre-training D Post-training |

Never Every Few Seconds Every Few Minutes If Lost/Confused Other

Figure 12. Comparison of frequencies of Soldier responses before and after training for question

15, "How often would you look at the FBCB2 display after you crossed the Line of Departure

(LD)?"

The bar graph shows that the pattern of Soldier responses changed little from before to

after training. Most Soldiers said they would check the FBCB2 display every few minutes after

crossing the LD. After training, fewer Soldiers said they would never check the display, and

more said they would check it every few seconds. Also, after training more Soldiers admitted

they would check the display if they were lost or confused, or approaching a waypoint. There

were also fewer "other" answers after training. Most of the Soldiers who marked "other" before

training added that they didn't know how often they would monitor the display.

The second question in this section asked about sharing tasks with others, often known as

"task shedding" (Sirevaag, Kramer, Wickens & Reisweber, 1993). Specifically, I wanted to

know how Soldiers handled the requirement to monitor FBCB2. The question asked, "How

often would you hand-off the requirement to monitor the FBCB2 display to someone else?"

Soldiers could choose (a) "Never," (b) "If I'm doing something else," (c) "Every few minutes,"

(d) "Always," or (e) "Other." The responses to this question are shown in Figure 13.

Like the previous question, the pattern of responses was very similar both before and

after training. Most Soldiers agreed they would delegate the monitoring task if they were

involved with another task. A few Soldiers before training said they would "always" delegate

FBCB2 monitoring, whereas after training no one chose this response. The "other" responses

after training mostly fell into the "Don't know," or "Depends on the situation" category.

19
"How often would you hand-of the requirement to monitor FBCB2 to someone eise?"

IPre-training DPosl-lraining |

25

20

IS -L10

Never If Doing Something Else Every Few Minutes Always Other

Figure 13. Comparison of frequencies of Soldier responses before and after training for question

16,"How often would you hand-off the requirement to monitor the FBCB2 display to someone

else?"

Pre- post-training differences. I also wanted to know how many Soldiers changed their

answers after training, and if so, if the changes formed a pattern. To answer these questions, I

examined Soldier responses before and after training and developed a matrix of all possible

combinations of responses. I then counted the frequency of response for each possible

combination. I identified how many Soldiers had the same response pre- and post-training versus

how many changed their responses. For those cases where Soldiers changed their responses, I

performed a chi-square test to determine if any response pattern occurred significantly more

often than the others.

For question 15, 20 out of 35 Soldiers (57%) had similar responses pre- and post-training.

Of those who changed their responses, there was no discemable pattern (y^ [7, A^=15] = 7.93,

n.s.). Question 16 had 24 Soldiers make the same response (67%) while 12 changed responses.

The changed responses had no significant pattern for this question either (x^ [6, A^=12] = 3.17,

n.s.).

Questionnaire Section Four: Free Response Section

The last section had two free response questions. The first question asked for more

information about task shedding the requirement to monitor FBCB2. The question asked, "If

you wanted to hand-off monitoring the FBCB2 display to someone else, who would you hand it

off to?" There were a number of different responses to this question, although responses tended

to fall into several categories. Table 9 shows the major categories of responses and the

frequencies of response in each one. As the table shows, the most popular answer to this

20
question before and after training was either hand it off to another crew member (in a vehicle

crew) or to the commander or leader. Before training, several respondents said they didn't know

who they would hand the monitoring task to. Probably the most interesting finding from this

question is that after training a number of respondents said they would hand it off to a trained

user. This suggests that after training Soldiers realized the need to have FBCB2 operated by

trained users.

The second question in this section asked, "What feature of this FBCB2 is the most

difficult to work with?" Again, there were a number of responses which tended to fall into

categories. Before training most Soldiers reported they didn't know, which suggests they had

little experience with the system before training, while a few reported using the touch screen in

rough terrain or while wearing chemical protective gear was difficult. After training, Soldiers

had a better idea of how difficult tasks were.

Table 9. Frequencies OfSoldier Responses To Question 17, "If You Wanted To Hand-Off

Monitoring The FBCB2 Display To Someone Else, Who Would You Hand It Off To?"

Response category Frequency

^^^^^^^^^^^^^^H Pre-training Post-training

Don't know 3 0

Another crew member 12 9

Commander/leader 10 9

An assistant 2 5

A trained user 0 4

The two most frequent response categories after training involved using overlays and

working with messages and reports. Somewhat surprisingly, after training several Soldiers

reported they found all features of the FBCB2 easy to use. Response frequencies for this

question are suntmiarized in Table 10.

Table 10. Frequencies OfSoldier Responses To Question 18, "What Feature Of The FBCB2 Is

The Most Difficult To Work With?"

Response category Frequency

^^^^^^^^^^^^^W Pre-training Post-training

Don't know 22 2

Using the touch screen 3 1

Messages and reporting 0 5

Planning 1 2

Using overlays 2 8

No features are difficult 0 4

21
Discussion

As mentioned in the introduction, there were a number of objectives to this research. The

first was to determine how training affected Soldiers attitudes and behaviors towards digital

systems, and also how it changed their use ofthe systems. The second was to identify what

capabilities of digital systems, in this case FBCB2, Soldiers found more or less useful. The last

objective was to identify if Soldiers exhibited certain behavior patterns towards digital systems.

I will discuss how this research addressed each of these objectives.

The Effect ofTraining

Formal training seems to have a significant and generally positive effect on Soldier

attitudes and behaviors towards digitization. After training. Soldiers reported they felt they

understood the system better, they were more likely to use it, and used more ofthe features. This

supports the idea that Soldiers who understand how to use digital tools are more likely to employ

them Soldiers also reported they better understood FBCB2 weaknesses after training.

There was a statistically significant difference between pre-training and post-training

attitudes and behaviors. Of the five questions in section one, four ofthem had rankings which

differed significantly from pre- to post-training. Only question one; "What do you consider the

most useful planning capabilides of FBCB2" showed no significant difference. When Soldiers

were asked to rank order the usefulness of FBCB2 capabilities, there was more agreement among

Soldiers after training than before for each question. This suggests the training gives Soldiers a

better appreciation of FBCB2 capabilities. When asked when FBCB2 was most useful, the

responses "planning" and "preparation for combat" were rated significantly higher after training,

so it may be that Soldiers better understood these concepts after training.

One point that could be raised is that 40 hours of computer experience alone could have

affected attitudes and behaviors independent ofmaterial presented in class. While this is

possible, some evidence from this research contradicts this point. Some ofthe questions I asked

concerned topics not covered in the class material. These topics had less change, or no change,

from pre- to post-training. For example, questions 15 asked about monitoring FBCB2 after

crossing the LD, and question 16 asked about how Soldiers would hand-off monitoring

requirements. Neither ofthese topics are covered in OPNET, and neither ofthese questions

showed significant differences between the pre-test and post-test. This suggests that it is the

material presented in class which changes Soldiers' attitudes and behaviors, rather than simple

experience working with digital systems. Therefore, structured, formal training may have a more

significant effect than unstructured experience with the system.

Soldier Ratings of Usefulness

Soldiers generally feel FBCB2 is useful, and is worth the additional effort required to

employ digitization. They reported it is most useful for planning, preparation for combat, and

movement, but less during combat (i.e. enemy contact). Once maneuver units are in contact with

the enemy, they are focused on the enemy and typically do not have time to view digital

displays. Also, once the enemy is seen, the enemy in range has higher priority than any enemy

displayed on digital systems which is out of range. These phases (planning, preparation for

combat, and movement) also were reported to have the highest workload. This naturally follows

since those features that are used least will not have high workloads associated with them.

22
Soldiers ranked the most useful planning capabilities as sending and receiving messages

and overlays, and route planning. The most useful non-planning related features ranked as most

useful were sending and receiving messages, navigating to a location, checking the location of

enemy units, and avoiding threats such as minefields. Overall, Soldiers reported that those

features of FBCB2 which had the highest workload and were most difficult to work with were

also those rated the most useful.

Soldiers did not agree on the accuracy of different features of FBCB2 either before or

after training. Although agreement was higher after training, it was still fairly low (Kendall's W

= .161 after training). This is probably because accuracy was not covered in training and

Soldiers may not have had enough experience with the system to estimate accuracy.

Soldier Patterns ofBehavior Towards Digitization

One ofthe goals ofthis research was to see if Soldiers exhibited behaviors towards

digital systems that were similar to behaviors exhibited by people in general towards automated

systems. The behaviors I was interested in were disuse or avoidance, limited use, overuse or

over-reliance on automation/digitization, and expert use. The second section ofthe questionnaire

asked questions which were designed to measure these behavior patterns.

Previous observations of Soldiers and leaders using FBCB2 for exercises showed

considerable disuse behaviors. However, Soldiers in this investigation reported few disuse

behaviors, and the majority reported a willingness to employ FBCB2 for the majority of combat

tasks. This apparent disparity has several possible explanations. First, the Soldiers in this

investigation were undergoing formal training for the digital system, whereas Soldiers in the

observational research reported much oftheir training was informal OJT. Soldiers in formal

training may have felt more committed to using digital systems since they were expending effort

to learn to use the systems. Second, the Soldiers in the observational research effort were from a

different unit from the Soldiers in the present investigation. The different units may have had

different command emphasis on the value of digitization. Since commanders and leaders often

have a significant impact on the organizational climate of a unit (Bass, 1981), different command

emphasis on digitization could have been reflected in the Soldier's willingness to use digital

systems in the present effort.

One question attempted to identify limited use behaviors, "I only use FBCB2 to do a few

things." Although there were a fair number of agreement responses to this question before

training, after training the majority ofresponses disagreed or strongly disagreed. This suggests

these Soldiers have moved beyond limited use and are willing to employ digital systems for

many combat tasks.

I also wanted to see if Soldiers exhibited overuse or over-reliance in digital systems.

Two questions assessed this behavior, "I know the information I get from FBCB2 is correct,"

and "FBCB2 is useful to do a lot ofthings, but I still check to make sure the information is

correct." Both ofthese questions dealt with cross-checking the accuracy ofinformation from the

digital system, because one ofthe most observable behaviors associated with over-reliance in

automation is automation bias, that is, assuming the information from the automated system is

correct without checking it. The answers to both ofthe questions indicate that Soldiers are not

biased in favor of automation and they report they check the information they get from FBCB2.

23
The last behavior set I was interested in was expert behaviors. In fact, there is relatively

little information about what constitutes general expert behavior with computer systems. Most

expert behavior measures involve speed and accuracy with particular software applications, but

not expert computer use in general. One behavior that is thought to indicate expertise is that

experts tend to take advantage of "hot keys" and shortcuts more often than do others (Eberts.

1994), and frequently will develop their own shortcuts for tasks. Therefore, I asked Soldiers to

agree or disagree with the statement, "Sometimes there are easier ways to do things on FBCB2

than what's taught in class or shown in the book." I expected that Soldiers who were more

expert and tended to use shortcuts would agree with this statement.

The results showed that before training most Soldiers chose the "don't know" option, but

after training there was an almost even split between agree and disagree with the statement.

When the "don't know" responses are eliminated, 48% of the Soldiers agreed with the statement

after training, which suggests to us that about half ofthe Soldiers feel comfortable enough

working with FBCB2 to use shortcuts. Somewhat surprisingly, there were no correlations found

between this variable and previous computer experience (or indeed any other demographic).

In general. Soldiers in this investigation did not exhibit any of the maladaptive patterns of

behavior seen in other areas, nor indeed observed in other digital units. Given that this is a field

effort with many extraneous variables beyond our control, it is best to be cautious in drawing

conclusions. However, several of the statements which attempted to assess maladaptive

behaviors showed that Soldiers' responses showed less ofthese behaviors after training than

before. Of particular note is that Soldiers learned to take information they got from FBCB2 with

caution, thus avoiding one ofthe major problems of automation overuse.

Teamwork Issues

An additional set of questions was asked which dealt with how Soldiers would hand-off

monitoring ofthe digital system. The majority of Soldiers said they would hand-off monitoring

requirements if they were doing something else, and they would hand it off to another crew

member or their commander. The responses were fairly similar both before and after training,

which may be because this topic is not covered in the classes.

Methodological Considerations

There are several caveats related to using the results of reported herein. The first is that

there was a somewhat restricted range since only Soldiers E5 and above were given the

questionnaires. This means the results may not be representative of the views of lower-level

enlisted Soldiers.

The second consideration is that the participant group contained Soldiers who may have

used digital systems (FBCB2) in different ways. Some Soldiers in the participant group

belonged to maneuver units, typically platoons or companies, while others operated FBCB2 in

TOCs. Soldiers in platoons and companies typically would have FBCB2 in their combat

vehicles and would use them to move and engage the enemy. On the other hand. Soldiers in

TOCs typically would use FBCB2 to monitor and support combat. Therefore, the way Soldiers

used FBCB2 in maneuver units versus in TOCs may be significantly different. This suggests

that this could have been a confounding variable which affected how Soldiers answered the

questions. This could explain in part the low agreement ratings (Kendall's W) and multi24
modalities ofsome ofthe questions. In the future I intend to include a question asking Soldiers

whether they use the digital system in a maneuver unit or in a TOC.

Future Research

It might be instructive to perform similar research on digital systems other than FBCB2

and identify similarities and differences. In addition, future research might involve a more indepth investigation of how Soldiers employ FBCB2 functions and tools, and ofits perceived

utility. Another variable which might be included in future research is whether Soldiers use

FBCB2 in maneuver units or TOCs. As mentioned above. Soldiers may utilize FBCB2

differently in these different environments, and an understanding ofthese difference may have an

impact on training.

Summary ofFindings and Recommendations

Formal training. Formal training seems to have a significant impact on Soldier attitudes

and behaviors towards digital systems. Classes on digital systems seem to give Soldiers a

positive attitude towards digital systems or reinforce a previous positive attitude. Soldiers report

they better understand the system and tend to use it more and use more features. Training

introduces Soldiers to features of digital systems which are useful to them that they might not

otherwise learn through discovery learning or OJT. In addition, training seems to steer Soldiers

away from maladaptive behavior patterns like those seen with automation in other areas.

Soldiers are willing to use digital systems and tools, but at the same time avoid placing too much

confidence in the system.

FBCB2 usability. Future design improvements for the FBCB2 interface should focus on

improving the usability of messages and reports, and also the use of overlays. These are the

functions Soldiers report they use most, and are also the most difficult to use and require the

highest workload.

Summary

This research shows that formal training in digital systems has a significant positive

affect on Soldiers' attitudes and behaviors towards digital systems. Not only does formal

training better prepare Soldiers to use correct procedures to produce digital products, it also gives

Soldiers more confidence in their ability to employ digitization. Soldiers who complete formal

OPNET courses report they are more likely to use digitization and use more ofthe tools

available. In addition, Soldiers seem to have a better appreciation of digitization's strengths and

weaknesses after training.

The effort also sought to find out if Soldiers exhibited certain maladaptive behaviors

towards digitization similar to those seen in other domains, such as disuse or avoidance, limited

use, or overuse. I found that this sample of Soldiers exhibited few ofthese behaviors before

training, and even less after training. In addition, I asked these Soldiers about behaviors thought

to be associated with computer expertise. At the end oftraining, about 39% of Soldiers in this

sample reported they exhibited these behaviors, although there was no correlation with previous

computer experience.

25
26
References

BamclL J. S.. Metiza, L. L. (2(){)3). AtiU^milion integralkm: Cainparhigflighi deck anfoinanon

and ILS. Army digitization. Paper to be presented a! the Inicrscrvice/induslry Training,

Simulation, and Ethication Conlerencc (I/ITSEO. December i-4, 2003.

Barnelt, J, S., Meliza, L. L. & McCluskey, M, R. {200!), Defming digital proficiency

measurement targetsfor U.S. Army units (ARl technical repon 1! !7). Alexandria. YA:

Army Research Institute for the Behavioral and Social Sciences,

Bass, B. M. (1981), Smgdill 'x handhook (rf leadership: A survey cftheory and research. New

York: The Free Press.

Eberts. R. E. (1994) User interface design. Englewood Cliffs, NJ: Prentice HaJi.

Mosicr, K. L., Skitka, L. J., Heer.s, S. & Biirdick, M. (1998). Aulomaiioii bias: Deci-sion making

and perfoniiance in high-tech cockpits, huernational Journal ofAviation Psychology 8

(I). 47-6-3.

Parasuraraan, R. & Riley, V. (1997). Human and automation: Use, misuse, disu.se and abuse.

Human Factors 39 (2), 230-253.

Sarter, N. B. & Woods, D. D. (1997). Team play with a powerful and independent agent:

Operational experiences and automation surprises with the AirbiLs A-20. Human Factors

39 (4) 553-569.

Siegel, S- (1956). Nonparametric statisticsfor the social sciences. New York; McGraw Hill.

Sirevaag, E. h, Kramer, A. F., Wickens, C. D. & Eeisweber, M. (1993). Assessment of pilot

performance and mental workload in rotary wing mrcraft. Ergonomics 36 (9), 1121-1140.

Tabachnick, B. G. & Fidell, L. S. (1996). Using multivariate statistics. 3'*^ Ed. New York: Harper

Collins.

27
28
Appendix A

Acronyms

BCTC. Battle Command Training Center

BOS. Battlefield Operating System.

FBCB2. Force XXI Battle Command, Brigade and Below.

GPS. Global Positioning System.

LAN. Local Area Network.

LD. Line of Departure.

MOS. Military Occupational Specialty.

OC. Observer-Controller.

OJT. On-the-job-training.

OPNET. Operator new equipment training

PEOSTRI. Program Executive Officer, Simulation, Training, and Instrumentation.

SOP. Standard Operating Procedure.

TOC. Tactical Operations Center.

TTP. Tactics, Techniques, and Procedures.

A-1
A-2
Appendix B

Pre-Training Questionnaire

B-1
B-2
PAR 1ICTPANT # FT 60-49A

• U.S.Aie^f RESEARCH MSTITUTE

FOR THE BlHAi|il^i:ipi>^S^^

Sirnutatc>rj|p^m& Research U^

Instructions - Pre-Training Questionnaire

• DO THIS ONE FIRST - Leave the other (sealed) questionnaire in the

packet (you'll do it later)

• No Names! (You will be known only by the participant number, above).

• Fill out this survey AT THE BEGINNING of the class. You will be asked

to fill out another one after the class is over.

• Your participation is voluntary. You don't have to fill out the

questionnaire if you don't want to.

• When you have finished, please return this questionnaire to the

envelope

Not to be shown to unauthorized persons. Not to be reproduced without the specific permission of the

Technical Director, U.S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES

B-3
QUESTIONS ABOUT YOU

(A) Your Rank: (B) Your MOS:

(C) How long have you been in the Army (total) Years Months

(D) What digital system do you normally work with? (circle all that apply)

FBCB2 ASAS AFATDS AMDWS MCS/(Lite) CSSCS Other

(E) How many months have you been using FBCB2?

(F) About how many hours a month do you use FBCB2?

(G) Have you ever taken an on-line tutorial for FBCB2? YES NO

Please continue to the next page

B-4
Section 1

• For each of the following questions, please rank order the list of items from first to last.

1. What do you consider the most useful planning capabilities with FBCB2? Please rank order from most

useful (number 1) to least useful

Plan a route Plan support measures

Send/receive messages/plans /overlays Line-of-sight tools

Other Other

2. What do you consider the most useful FBCB2 capabilities other than planning? Please rank order from

most useful (number 1) to least useful

Navigate to a location Send/receive messages

Avoid threats (minefields, NBC areas, etc.) Logistics/personnel reporting

Check location of subordinate units Check location of enemy units

Check status of subordinate units Other

3. When is FBCB2 most useful to you ? Rank order from most useful (number 1) to least useful

Planning Assembly area operations

Preparation for combat Movement

Attack Actions on the objective

Defense Other

4. When is FBCB2 workload the highest? Rank order from most work (number 1) to least work

Planning Assembly area operations

Preparation for combat Movement

Attack Actions on the objective

Defense Other

5. On the FBCB2 display, which of the following is usually CLOSEST to ground truth (most accurate)?

Please rank order from most accurate (number 1) to least accurate

Friendly icons (center of mass) Friendly icons (individual platforms)

Line-of-sight analysis Enemy icons

Terrain features Overlays

Other

Please continue to the next page

B-5
Section 2

• For each of the following statements, circle whether you STRONGLY AGREE, AGREE, DISAGREE, or

STRONGLY DISAGREE with the statements, or circle DON'T KNOW if you're not sure.

6. "Sometimes the FBCB2 does things for no reason, or that I don't understand"

Don't Strongly Disagree Agree Strongly

Know Disagree Agree

7. "I don't use FBCB2 if I can avoid it"

Don't Strongly Disagree Agree Strongly

Know Disagree Agree

8. TBCB2 really helps me do my job"

Don't Strongly Disagree Agree Strongly

Know Disagree Agree

9. TBCB2 isn't worth the amount of effort I have to put into if

Don't

Know

Strongly

Disagree

Disagree

10. 1 only use FBCB2 to do a few things"

Don't

Know

Strongly

Disagree

Disagree

Agree Strongly

Agree

Agree Strongly

Agree

11. "I know that infonnation I get from FBCB2 is correct"

Don't Strongly Disagree Agree Strongly

Know Disagree Agree

12. "If the FBCB2 isn't working right, it's best to try to get it working rather than revert to analog"

Don't Strongly Disagree Agree Strongly

Know Disagree Agree

Please continue to the next page

B-6
13. TBCB2 is useful to do a lot of things, but I still check to make sure the information it gives is correcf

Don't Strongly Disagree Agree Strongly

Know Disagree Agree

14. "Sometimes there are easier ways to do things on FBCB2 than what's taught in class or shown in the

book"

Don't Strongly Disagree Agree Strongly

Know Disagree Agree

Section 3

• For each of the following questions, please circle your answer.

15. How often would you look at the FBCB2 display after you crossed the Line of Departure (LD)? (Circle)

If I'm lost/confused or when approaching the next decision point Never

Every few minutes Every few seconds Other

16. How often would you hand-off the requirement to monitor the FBCB2 display to someone else? (circle

one)

Never If I'm doing something else Every few minutes

Always Other

Section 4

Please answer the questions by filling in the blank.

17. If you wanted to hand-off monitoring the FBCB2 display to someone else, who would you hand it off to?

18. What feature of this FBCB2 is the most difficult to work with?

You're done! Return this questionnaire to the envelope. Thanks!

B-7
B-8
Appendix C

Post-Training Questionnaire

C-1
C-2
PARTICIPANT # PT 60-49B

F0K-lHE:BeH|WiA|'M^M

Instructions - Post-Training Questionnaire

• This is the second questionnaire - you should have already completed

one at the beginning of class (if you did not, tell the experimenter or

instructor)

• No Names! (You will be known only by the participant number, above.

Please make sure this number matches the one vou filled out at the start of

class).

• When you have finished, please return this questionnaire to the

envelope -then turn the envelope in to the experimenter or instructor

Not to be shown to unauthorized persons. Not to be reproduced without the specific permission of the

Technical Director, U.S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES

C-3
Questions About You

(A) About how many hours a week do you use a computer at work?

at home?

(B) About how many hours a week do you play computer games?

(C) If you play computer games, which ones do you play (circle all that apply)

Card games

Role-playing games

Other

Strategy games Arcade-type games

Simulation games Sports games

Other

Please continue to the next page

C-4
Section 1

• For each of the following questions, please rank order the list of items from first to last.

1. What do you consider the most useful planning capabilities with FBCB2? Please rank order from most

useful (number 1) to least useful

Plan a route Plan support measures

Send/receive messages/plans /overlays Line-of-sight tools

Other . Other

2. What do you consider the most useful FBCB2 capabilities other than planning? Please rank order from

most useful (number 1) to least useful

Navigate to a location Send/receive messages

Avoid threats (minefields, NBC areas, etc.) Logistics/personnel reporting

Check location of subordinate units Check location of enemy units

Check status of subordinate units Other

3. When is FBCB2 most useful to you ? Rank order from most useful (number 1) to least useful

Planning Assembly area operations

Preparation for combat Movement

Attack Actions on the objective

Defense Other

4. When is FBCB2 workload the highest? Rank order from most work (number 1) to least work

Planning Assembly area operations

Preparation for combat Movement

Attack Actions on the objective

Defense Other

5. On the FBCB2 display, which of the following is usually CLOSEST to ground truth (most accurate)?

Please rank order from most accurate (number 1) to least accurate

Friendly icons (center of mass) Friendly icons (individual platforms)

Line-of-sight analysis Enemy icons

Terrain features Overlays

Other

Please continue to the next page

C-5
Section 2

• For each of the following statements, circle whether you STRONGLY AGREE, AGREE, DISAGREE or

STRONGLY DISAGREE with the statements, or circle DON'T KNOW if you're not sure.

6. "I'm never sure what FBCB2 will do next"

Don't Strongly Disagree Agree Strongly

Know Disagree Agree

7. "FBCB2 really helps me do my job"

Don't Strongly Disagree Agree Strongly

Know Disagree Agree

8. "1 don't use FBCB2 if I can avoid it"

Don't Strongly Disagree Agree Strongly

Know Disagree Agree

9. "Digitization isn't worth the amount of effort I have to put into It"

Don't Strongly Disagree Agree Strongly

Know Disagree Agree

10. There are only a few tasks FBCB2 is useful for"

Don't Strongly Disagree Agree Strongly

Know Disagree Agree

11. The information on FBCB2 is almost always correct, so I don't bother checking if

Don't Strongly Disagree Agree Strongly

Know Disagree Agree

12. "Even when it's not working perfectly, digitization is better than analog"

Don't Strongly Disagree Agree Strongly

Know Disagree Agree

Please continue to the next page

C-6
13. "As long as I check the information you get from FBCB2 to make sure it's correct, digitization is really useful"

Don't Strongly Disagree Agree Strongly

Know Disagree Agree

14. "Sometimes I make up my own shortcuts that get things done faster"

Don't Strongly Disagree Agree Strongly

Know Disagree Agree

Section 3

• For each of the following questions, please circle your answer.

15. How often would you look at the FBCB2 display after you crossed the Line of Departure (LD)? (Circle)

If I'm lost/confused or when approaching the next decision point Never

Every few minutes Every few seconds Other

16. How often would you hand-off the requirement to monitor the FBCB2 display to someone else? (circle

one)

Never If I'm doing something else Every few minutes

Always Other

Section 4

Please answer the questions by filling in the blank.

17. If you wanted to hand-off monitoring the FBCB2 display to someone else, who would you hand it off to?

18. What feature of FBCB2 is the most difficult to work with?

You're done! Return this questionnaire to the envelope.

The purpose of this research Is to find out how people learn to use digitization. This research will

help us to find the best ways to train and evaluate the use of digital systems. Your participation has made

this research possible THANKS!

C-7