U.S. Army Research Institute

for the Behavioral and Social Sciences

Research Report 1808

Web-Based Collaborative Learning: Communication

Between Learners Within a Virtual

Tactical Operations Center

James Belanich

U.S. Army Research Institute

Kara L. Orvis

Consortium Research Fellows Program

Robert A. Wisher

Office of the Secretary of Defense

June 2003

Approved for public release; distribution is unlimited.
U.S. Army Research Institute

for the Behavioral and Social Sciences

A Directorate of the U.S. Total Army Personnel Command

ZITA M. SIMUTIS

Director

Technical review by

Elizabeth Brady, U.S. Army Research Institute

Mark Sabol, U.S. Army Research Institute

NOTICES

DISTRIBUTION: Primary distribution of this Research Report has been made by ARI.

Please address correspondence concerning distribution of reports to: U.S. Army

Research Institute for the Behavioral and Social Sciences, Attn: TAPC-ARI-PO, 5001

Eisenhower Ave., Alexandria, VA 22333-5600.

FINAL DISPOSITION: This Research Report may be destroyed when it is no longer

needed. Please do not return it to the U.S. Army Research Institute for the Behavioral

and Social Sciences.

NOTE: The findings in this Research Report are not to be construed as an official

Department of the Army position, unless so designated by other authorized documents.
REPORT DOCUMENTATION PAGE

1. REPORT DATE (dd-mm-yy)

June 2003

2. REPORT TYPE

Interim

3. DATES COVERED (from... to)

November 2001 - January 2003

4. TITLE AND SUBTITLE 5a. CONTRACT OR GRANT NUMBER

Web-based Collaborative Learning: Communication Between

Leamers within a Virtual Tactical Operations Center 5b. PROGRAM ELEMENT NUMBER

633QQ7A

6. AUTHOR(S)

James Belanich (U.S. Army Research Institute), Kara L. Orvis

(George Mason University), and Robert A. Wisher (Office of the

Secretary of Defense)

5c. PROJECT NUMBER

A792

5d. TASK NUMBER

208

5e. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

U.S. Army Research Institute for the Behavioral and Social Sciences

Advanced Training Methods Research Unit

5001 Eisenhower Avenue

Alexandria, VA 22333-5600

8. PERFORMING ORGANIZATION REPORT NUMBER

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

U.S. Army Research Institute for the Behavioral and Social Sciences

5001 Eisenhower Avenue

Attn: TAPC-ARI-ll

Alexandria, VA 22333-5600

10. MONITOR ACRONYM

ARI

11. MONITOR REPORT NUMBER

Research Report 1808

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES

14. ABSTRACT ^Max/mum 200 words;:

In 2001, the U.S. Army Armor School designed a blended course for reserve officers with both distributed

learning and in-residence components. The distributed portion consisted of an asynchronous phase, where

officers completed lessons on their own time, and a synchronous phase, where they used a virtual tactical

operations center (VTOC) simultaneously but from different locations. One primary method of communication

during synchronous VTOC training was computer-based text messaging. The text messages for five teams

consisting of seven to nine members were recorded. The analysis of these text messages revealed that a

majority ofthe communication was related to the operational task. The second most frequent type of text

messaging was social in nature, and this type of communication peaked during the initial training sessions and

during the final session. The least frequent type of communication was technology-related text messages, those

where the user was having difficulty with the system and asking for help or a person replying to a help query.

Technology-related communication occurred modestly during the initial sessions, buttailed off dramatically as

sessions progressed. The students, instructors, and course designer all took part in interviews about their

opinions on the distributed learning components of the course. The overall ratings of the distributed -learning

portion of the course were positive.

15. SUBJECT TERMS

distributed learning, distance learning, virtual tactical operations center, text chat, text messaging, collaboration,

training technology, synchronous communication

i :. '. I ,k

,n, '-SECURrTY'djAsdRcknONOF ^ W

16. REPORT

Unclassified

17. ABSTRACT

Unclassified

18. THIS PAGE

Unclassified

19. LIMITATION OF

ABSTRACT

Unlimited

20. NUMBER

OF PAGES

25

21. RESPONSIBLE PERSON

(Name and Telephone Number)

James Belanich

703-617-2362
Research Report 1808

Web-Based Collaborative Learning: Communication

Between Learners Within a Virtual

Tactical Operations Center

James Belanich

U.S. Army Research Institute

Kara L. Orvis

Consortium Research Fellows Program

Robert A. Wisher

Office of the Secretary of Defense

Advanced Training Methods Research Unit

Franklin L. IVIoses, Chief

U.S. Army Research Institute for the Behavioral and Social Sciences

5001 Eisenhower Avenue, Alexandria, Virginia 22333-5600

June 2003

Army Project Number Personnel, Performance

2O262785A790 and Training

Approved for public release; distribution is unlimited.

Preceding Page Blank 111
IV
FOREWORD

The U.S. Army Research Institute for the Behavioral and Social Sciences (ARI), as part

of its WEBTRAIN project, is investigating the use of distributed-leaming technology for use by

soldiers. ARI seeks to provide guidance to the Army as it transforms from classroom-based

instruction to more distributed, soldier-centric, and collaborative instruction methodologies.

With the U.S. Army transforming many ofits courses to a distributed-leaming format,

new methodologies are needed to take full advantage of current technology. In this report, the

distributed-leaming portion ofthe Armor Captains' Career Course for Reserve Component

Officers was examined. Findings ofthis report are relevant to course designers interested in

implementing collaborative tools for distributed learning. This research was briefed to Assistant

Deputy Chief of Staff-Training, TRADOC on 17 October, 2002. Some portions ofthe research

were published in joumal form (Orvis, Wisher, Bonk & Olson, 2002) and as an ARI Research

Note (Bonk, Olson, Wisher, & Orvis, 2002). The results and recommendations ofthis research

were also presented at the Army Science Conference on 3 December 2002.

MICHAEL G. RUMSEY

Acting Technical Director
VI
WEB-BASED COLLABORATIVE LEARNING: COMMUNICATION BETWEEN

LEARNERS WITHIN A VIRTUAL TACTICAL OPERATIONS CENTER

EXECUTIVE SUMMARY

Research Requirement:

The U.S. Army Armor School (USAARMS) redesigned the Armor Officers' Advanced

Course for Reserve Component officers (ACAC RC) so that the first phase could be delivered

via web-based instruction instead of via the previous method, written correspondence. The

Director ofthe USAARMS requested that the U.S. Army Research Institute's Advanced

Training Methods Research Unit (ARI-ATMRU) evaluate the distributed learning phase.

Procedure:

During synchronous collaborative instruction, students used a computer-based text

messaging fiinction to communicate with other students in a Virtual Tactical Operations Center

(VTOC). The content ofthe text messages between students was analyzed across six training

sessions. Additionally, to determine strengths and weaknesses ofthe distributed learning portion

ofthe course, interviews were conducted with two groups offour students, three instructors, and

the course supervisor.

Findings:

Three types oftext communication were identified: task-related, social, or technologyrelated. Task-related messages involved statements oriented toward the course exercise. Social

interactions were messages not related to the task, but involved personal content. Technologyrelated interactions involved the equipment used for the exercise. The relative frequency ofthe

different types oftext messaging changed across sessions. During the initial session, moderate

levels of on-task, social, and technology-related communication occurred. Over the next few

sessions, the level ofsocial communication dropped, but rebounded during the final session,

while the frequency oftechnology-related communication dropped and remained low. The

relative frequency of on-task communication peaked during the middle sessions, but was usually

the most frequent type of communication across sessions.

The interviews with students, course instructors, and the course supervisor revealed

additional information. For example, during the synchronous distributed learning phase, the role

ofthe instructor was seen as a facilitator instead of as a lecturer. Also, to limit the frequency of

problems that may interfere with instruction, the course supervisor recommended that course

designers should not stretch the limits ofthe instructional technology.

Utilization of Findings:

The change in relative frequencies in the nature oftext messaging provides insight to the

development of a virtual training team, and should be useful to both course developers and

instructors interested in such social interaction and problem solving. In addition, the findings

Vll
from the participant interviews regarding tlie facilitative role played by instructors should be

particularly useful to instructors of distributed courses.

vui
WEB-BASED COLLABORATIVE LEARNING: COMMUNICATION BETWEEN

LEARNERS WITHIN A VIRTUAL TACTICAL OPERATIONS CENTER

CONTENTS Page

INTRODUCTION 1

WEB-BASED COLLABORATIVE LEARNING 1

ARMOR CAPTAINS' CAREER COURSE (AC3) BACKGROUND 2

METHOD 4

PARTICIPANTS 4

PROCEDURE 5

RESULTS AND DISCUSSION 6

TEXT MESSAGING DATA 6

INTERVIEWS WITH STUDENTS 9

INTERVIEW WITH COURSE INSTRUCTORS 11

INTERVIEW WITH DL EDUCATION ADVISOR 13

SUMMARY 14

COMMUNICATION TRENDS 14

PARTICIPANT REACTIONS 15

REFERENCES 1

APPENDIX A - FOCUS GROUP QUESTIONS A-1

APPENDIX B - INSTRUCTOR INTERVIEW QUESTIONS B-1

IX
X
Web-based Collaborative Learning:

Communication between Learners within a Virtual Tactical Operations Center

INTRODUCTION

The U.S. Army is embracing distributed learning (DL) as an agent of change in the areas

oftraining and education. The Army is in the process of developing more than 525 DL courses

by 2010. Hundreds of digital training facilities are currently available in both the regular Army

and reserve components to provide access to technology and learning content (Program

Management Office, 2000). As of April 2003, the Army University Access Online program, also

known as eArmyU, has enrolled 35,000 soldiers, equipped them with laptops and Internet access

accounts, and encouraged them to earn a college degree while serving on active duty. In the

National Defense Authorization Act of 2002, Congress passed legislation allowing reservists to

be compensated for completion of asynchronous DL courses. Clearly, the interest in DL from an

infrastructure and policy perspective is growing, although questions surrounding its optimal use

remain.

While delivery ofDL over the Web is still relatively new, the benefits have already been

demonstrated in many ways, such as decreased cost and increased time management (Army

Logistician, 2001; Sanders & Bumside, 2001; Deloughry, 1995; Howard, 1992; Kulik, & Kulik,

1991; Metzko, Redding & Fletcher, 1996). Metzke, Redding, & Fletcher (1996) reviewed 30

studies on DL instruction and found that students needed a median of 30 percent less time to

complete distributed courses as compared to traditional instruction. In addition, students who

completed courses via DL attained a higher level of achievement when compared to traditional

instruction, an average increase of nearly 0.5 standard deviations.

While DL has its advantages, there are some areas where it still lags behind face-to-face

instruction, such as ease of communication and the development of deeper relationships at a

faster pace (Olson & Olson, 2000). In the current research, we analyzed the use oftext

communication for Web-based collaboration by students during synchronous DL.

Web-based Collaborative Learning

Collaboration is a key part ofthe "Learner-Centered Psychological Principles" promoted

by the American Psychological Association (APA, 1997). Through collaborative tasks, such as

discussing, summarizing, clarifying, and integrating course content into an overall framework,

learners acquire knowledge and gain a deeper understanding (Deatz & Campbell, 2001;

Palincsar, 1998). Collaboration among learners allows for the synergistic building of

knowledge. Numerous studies have shown that the use of collaborative learning exercises leads

to improved knowledge acquisition as compared to traditional classroom instruction without

collaborative exercises (Alkhateeb & Jumaa, 2002; Bonk & Wisher, 2000; Fisher & Coleman,

2001-2002; Kang, 1998).

The use of collaborative tools during on-line exercises can be helpful, not only for

completing the task at hand, but also for social functions such as team building and group

cohesion (Bonk & Wisher, 2000; Kang, 1998). People in DL courses sometimes feel isolated

because there is a lack of natural interaction with class members as compared to traditional
classrooms where learners have the opportunity to meet one another. Collaborative tools can help

to reduce these feelings ofisolation experienced by distributed learners (Muilenburg & Berge,

2001).

Two ofthe more frequent Web-based tools used for collaboration are text messaging

(e.g., instant messaging or text chat) and e-mail. Text messaging allows one learner to transmit a

string oftext that immediately is presented to another online learner. It is considered

synchronous because there is essentially no delay in information transmission between the sender

and the receiver. E-mail, however, is considered asynchronous because there is typically a delay

in the receipt ofthe message, since only one participant might be online at a given time. When

used properly, both e-mail and text messaging are useful in DL courses (Bonk & Wisher, 2000;

Kang, 1998).

Asynchronous text communication has advantages, such as allowing for more time

flexibility. However, because ofthe immediacy ofthe communication users feel more connected

with synchronous communication. Pena-Shaff, Martin, & Gay (2001) found that synchronous

text messaging provided for more task collaboration than communication via asynchronous

methods. In addition, participants also showed increased socializing using text messaging in

comparison to asynchronous communication. Hudson and Bruckman (2002) concluded that the

"realtime" aspect oftext messaging is a key factor leading to effective social interactions. These

studies suggest that when users simultaneously work together, or collaborate, text messaging is

an effective communication mode.

In the course examined in this report, a text messaging system was used during

synchronous online training, allowing the participants to communicate spontaneously. In the

current research, we assessed the use oftext messaging and the perspectives ofstudents,

instructors and the course administrator through the use of group interviews.

Armor Captains' Career Course Background

The purpose ofthe Armor Captains Career Course (ACS) is to train captains both for

command positions and positions as assistant operations officers in a tactical operations center

(Sanders &. Bumside, 2001). In an earlier version ofthe course for reservists, the first oftwo

phases was presented via written correspondence before a two-week in-residence training phase.

This training schedule was required because training in the reserves is limited to 39 days per

year, generally as one weekend per month and one two-week period. The course requirements,

therefore, had to be tailored for such an intermittent training schedule. The use ofDL, with its

ability to offer training anytime and anywhere, offers obvious flexibility within such a restricted

training schedule.

The current version ofthe AC3-DL course uses a blend oftwo instructional phases. The

first phase was divided into one asynchronous Internet-based set oflessons (Phase la), followed

by a series ofsynchronous, collaborative, Internet-based exercises during weekend training

sessions (Phase lb). The final phase (Phase 2) consisted offace-to-face resident training for

two-weeks. According to Sanders and Bumside (2001), the students who participated in early

iterations ofthe AC3-DL course finished the DL lessons in significantly less time than students

who completed the traditional correspondence version. On average, students who attended the
AC3-DL course also achieved slightly higher scores on a knowledge test than students who

completed the traditional classroom-based course (Sanders & Burnside, 2001). This finding

approached statistical significance (p=.055), suggesting that the conversion ofthe course from a

traditional classroom-based course to a blended learning approach yielded better learning.

The initial phase ofthe AC3-DL (asynchronous lessons) consisted oflessons provided

over the Web followed by end-of-lesson tests. The soldiers could access course material and take

the tests at any time from any computer with Internet access. During this phase, students learned

basic terms and concepts for use later in the course. Students needed to pass each multiple-choice

test with a score of at least 70 percent correct in order to proceed to the next lesson. Ifstudents

scored lower than 70 percent, they were allowed to retake the tests until successful. Students

received feedback immediately from the computer and via email from the instructor. This selfpaced set oflessons was comparable to a 3-credit college level course. The soldiers in Sanders

and Burnside (2001) averaged 7.5 months to complete this phase.

For synchronous exercises (Phase lb), students "convened" in the virtual tactical

operations center (VTOC) for training sessions lasting between four and eight hours on two

consecutive weekend days. The VTOC was a web-based rendition of an actual tactical

operations center (see Figure 1). This phase consisted ofstudents in separate locations

completing collaborative exercises. The lessons, scheduled for weekend training sessions,

involved each student performing a specific role in time-limited exercises requiring the

participation of all students. To collaborate during this phase, students used: a) audio

conferencing where they could speak to any ofthe other team members and be heard by all ofthe

team members; b) a text messaging application where they could type a text message to either all

members or a subset ofthe team; c) a virtual image ofthe team so that the function of each

member could be identified; and d) shared applications. For additional detail ofVTOC and the

AC3-DL, see Sanders and Burnside (2001) and Sanders and Guyer (2001).

In the VTOC, there were four shared applications: a shared text program where students

could write operation orders, warning orders and other planning documents; a shared bookshelf

where students could display field manuals; a Mapedit program where students could display and

modify map images; and a 3-dimensional virtual envirormient which allowed the students to

"walk" and symbolically comment on the terrain ofthe mission. With these collaborative tools,

they collectively solved problems concerning military operations and generated work products,

such as a mission analysis.

The final phase ofthe course consisted ofin-residence collaborative exercises, where

students met face-to-face for the first time. In this phase, the students engaged in real-life tactical

operations exercises, applying the skills and competencies that they learned during the DL phase

ofthe course. As with previous phases, the instructors monitored each student's performance.
i^ffiilliiiSKIiiiiimlii

File Edit Viev- Favoiitec Jools Help

aiL«£:.

=*aschelto.B

Close Save UiD .-'.14-

pi.

788322 n Selected SY ICufSor NONE IFiistjr!

J iT

V Public ^ ^KU*i ififSHH *

(Veil oriii-'^ti I the ^/irtual Tacticai up^ratinrr, c eiitei

<enter your chat text here> M

V 2(5 People ) C> ^ 1 ! [> ' 8 j < I

IJlckname Interests

"IfeuyerC Fort Knox

.,„^PaschettoG Tactics

SS JairtConF

J|'6nUvBiTaik«|

m !<^ Internet Zi

Figure 1. The Virtual Tactical Operation Center, as it appears on the computer screen of a user.

In the current research, two different types of data were collected: text communication

and reactions to the DL phase ofthe course. First, during the collaborative exercises of Phase

lb, all ofthe text messages among the students were recorded and analyzed. The analysis ofthe

text messaging data revealed insight into the ongoing process ofVTOC training team

development and performance. Second, midway through the face-to-face classroom section

(Phase 2), the researchers conducted interviews with students, instructors, and the course advisor

to obtain reaction data regarding the DL components ofthe course (Phase la and Phase lb).

This provided a qualitative assessment ofthe blended learning approach used during this course.

METHOD

Participants

A total of 41 students who enrolled in the AC3-DL course conducted by the U. S. Army

Armor School at Fort Knox, Kentucky, participated in this research. All participants were male

reservists in the Army National Guard. The participants were geographically dispersed across

the United States and had never previously met face-to-face. For the synchronous phase, the

course administrator assigned participants to five teams consisting of between seven and nine

members.
Following completion ofthe two distributed learning phases and midway through their

participation in the face-to-face section ofthe course (Phase 2), the researchers interviewed eight

students from one ofthe teams. The researchers also interviewed two ofthe course instructors

and the course supervisor.

Procedure

Text messages. During VTOC exercises in Phase lb, groups were scheduled to meet for

12 hours on each ofseven weekends at monthly intervals. The first session introduced the

technology, and the remaining six sessions were task-oriented training using the VTOC. The

researchers recorded and analyzed six sessions oftext messaging data, each session spanning two

days. Each session began with 30 minutes devoted to a communications check, followed by

approximately 12 hours of online training per weekend.

The definition of one text message was an act ofmessaging with "a single, uninterrupted

verbalization, typed in the message box," in accordance with Lebie, Rhoades, and McGrath

(1996). Based on a variation ofthe coding scheme used by Lebie et al., two raters categorized

each text message as one ofthree types: task-related, social, or technology-related. Task-related

messages involved statements related to the VTOC exercise and the course topic. Social

interactions were messages involving personal content that were not related to the training task.

Social topics included hobbies, holidays, and plans for getting together during the in-residence

phase at Fort Knox. Technology-related messages involved discussion ofthe VTOC equipment

used for the exercise, which included how the interactive tools were used or ifteam members

were having problems with the technology. Table 1 shows examples of each type oftext

message.

Table 1

Examples ofTask, Social, and Technology Interactions in the Text Messaging Data

Task • "Remember in the BDE OPORD-the BDE CMDR wants

this to occur at about this time"

• "I don't see anything about obstacles in the CLOSE

section"

• "I think obstacles in the Close section ofthe COA

statement is a necessary evil"

Social • "Kids are great we made breakfast for Mom (wife)"

• "Did you go out for a run last night?"

• "Tell her I said "Happy Mothers Day"

• "3 miles in 24 mins all hills"

• "If God had meant for us to run, he wouldn't have given

us tanks"

Technology • "Cannot talk or hear...will try to recormect "

• "Is anyone talking right now? I think I dropped audio"

• "Going to reboot"
Inter-rater reliability. Prior to final coding ofthe text message, inter-rater reliability was

assessed. Approximately 15 percent of all messages were selected at random, and each ofthe

two raters separately coded 1,034 messaging acts. Cohen's Kappa ofinter-rater reliability

between these two raters was deemed acceptable at k=. 90. Cohen's Kappa is a measure ofthe

degree to which two raters concur when placing items into nominal categories. The range of k is

0 - 1.0, and like other correlation measures higher values indicate greater inter-rater reliability.

Accounting for the proportion of agreement expected by chance, it is better than using percent

agreement between two raters (Fleiss, 1981).

Focus groups. Midway through the final phase oftraining, one team of eight students

was divided into two groups offour to participate in focus groups. During the focus groups, one

researcher asked each ofthe two groups a set of questions while a second researcher recorded the

students' responses. The questions covered the asynchronous phase and the use ofthe VTOC

during the synchronous portion ofthe class (Phase lb). The questions are presented in Appendix

A. All participants who were interviewed had worked together in the synchronous phase ofthe

course and in the face-to-face setting during Phase 2.

Researchers also interviewed two course instructors and the DL Education Advisor. The

questions asked to the instructors are presented in Appendix B. A structured set of questions was

not prepared for the course supervisor. Instead, a researcher asked the supervisor about her

opinion ofthe AC3-DL course, specifically, and DL courses in general to discern what advice

she would give to potential developers of DL projects.

RESULTS AND DISCUSSION

Two sets of data were assessed. The first set was quantitative data generated from the

analysis ofthe text messages during the synchronous VTOC training in Phase lb. The second

set was qualitative data from the transcripts ofthe interviews conducted during Phase 2. These

two sets of data apply to different aspects ofthe research; therefore, the results fi-om each data set

below was followed immediately by a discussion ofthe findings for that particular data set.

Text Messaging Data

A total of 6,806 text messages were recorded across all sessions. Two raters coded 6,705

ofthe 6,806 messages into one ofthe three interaction categories; 101 ofthe messages lacked

sufficient information for coding, and therefore were not included in the following analysis. Of

the 6,705 messages, 54 percent were rated as task, 30 percent were rated as social, and 16

percent were rated as technology related (see Table 2).

Percentages ofthe types ofmessages were used instead offrequency counts in order to

standardize the results across the six sessions because the number ofindividuals and teams

differed across sessions. Data for some teams were not available for all sessions, and a few data

files were accidentally truncated during the recording process, resulting in some losses. For

example, for one team, we may have recorded both days ofsessions one through four and only

the Saturday data ofsession five. For all sessions, the data of at least two teams were recorded.
Table 2. The percentage oftext messages types during synchronous VTOC training.

Type of Message # ofMessages Percentage of

Total Messages

Task 3614 54%

Social 2003 30%

Technology 1088 16%

Total 6705 100%

The percentages of all three message types changed over the six sessions oftraining with

the VTOC. Task-related messages ranged from slightly over 40 percent ofthe messages during

the first session to over 60 percent during the middle sessions, and then fell to about 40 percent

during the last session. Complementary to this trend, social text messaging was more frequent

during the first and last training sessions than during the middle sessions. The level of

technology-related messages steadily decreased over sessions, from 30-40 percent during the

first three sessions to less than 5 percent during the last session. Different trends were revealed

for task, social, and technology messages, as illustrated in Figure 2.

70

60

50

40

0)

c

u

ua 30 -

a>

D.

20

10

Average Across Groups

Percentage of Chat Instances

3 4

Sessions

Figure 2. The percent oftext messaging types (task, social, and technology related) across the six

Phase lb sessions using the VTOC.

Three separate ANOVAs were conducted to test for statistically significant differences

between the six sessions for each individual category ofinteraction. For these analyses, the

percentage ofmessages within each category (task, social, and technology) for each participant

was compared across sessions. There were significant differences across sessions for task
messages (F=5.41, p<.05), social messages (F=14.20, p<.05), and technology messages (F=6.54,

p<.05). Post hoc analyses (Tukey's test of honest significant difference) were conducted to

determine where the significant differences occurred across sessions. For task-related messages,

there was a significantly smaller percentage of messages in Session 1 compared to Sessions 3, 4,

and 5. In addition, task messages during Session 4 were more likely than during Sessions 1, 2,

and 6. For social messages, there was a significantly greater percentage during Session 1

compared to Sessions 2 and 3, while a significantly higher percentage ofsocial messages

occurred during Session 6 than during any ofthe other sessions. Finally, for technology-related

messages. Session 2 produced a significantly higher percentage than Sessions 4, 5, and 6.

Additionally, a lower percentage oftechnology-related messages occurred during Session 6 than

during sessions 1,2, and 3. All other comparisons were not significant.

The results document that the use oftext messaging methodology for tracking group

interaction is viable. During the initial sessions, almost equal time was spent on the task, getting

to know one another, and figuring out how the equipment worked. As the team progressed into

the middle sessions, the soldiers focused on the task more and needed less help with the

equipment. As the training sessions concluded, almost no help was needed regarding the

equipment. Also, during the last session the social interactions increased (the participants were

either saying good-bye or planning for meeting one another during the next phase oftraining).

The results provide insight into collaborative learning and how the teams and communication

change over time.

Previous research indicates that when Web-based groups develop, the types of

interactions among members change over time. In research by Lebie, Rhoades, and McGrath

(1996), the change was similar to the pattern demonstrated by participants in the current

research. In both, the frequency of communication regarding how to use the technology or how

to complete the functions of a particular task decreased over time as the users became more

familiar with the equipment or more skilled with the technology. In addition, the frequency of

on-task communication was maintained at a high level across all sessions, and the frequency of

social interaction started high and decreased during the initial sessions.

One ofthe trends oftext messaging was the steady decrease in technology-related

messages as sessions progressed. This most likely was due to increased familiarization with the

equipment and decreased equipment difficulties over sessions. Technology-related problems are

one ofthe main barriers to distributed education (Muilenburg & Berge, 2001; Lebie, Rhoades, &

McGrath, 1996; Olson & Olson, 2000). Some ofthe difficulties in the AC3 course were due to

some students not being familiar with the technology. When students are introduced to new

technology they may have difficulty at first, however, previous experience using technology

should increase the pace at which new skills are acquired (Schaab & Moses, 2001). It might be

helpful to start the course with technology related training for those who are not familiar with the

technology. Once these skills are learned, they may then be maintained and more easily transfer

to novel situations and new training exercises.

Lebie, Rhoades, and McGrath (1996) found that text-based commimication across a

distributed network followed similar patterns to interactions in face-to-face groups. However,

the patterns took longer to develop in the distributed groups. This suggests that the nature of
communication in both distributed groups and face-to-face groups may be similar, but that social

relationships may develop more slowly.

Interviews with Students

Each student participated for approximately 70 hours in synchronous online instruction.

They each also had a personal computer available for use and access to the Internet at both their

home and work settings. Overall, the students favorably evaluated their experience working in

groups online.

Students noted technological obstacles. During the focus group sessions, students noted

several technological problems and obstacles related to the AC3-DL course. The map editor and

virtual messaging rooms were responsible for the majority ofstudents' technology problems.

Such problems can frustrate students, interfere with learning, and negatively impact course

completion rates (Olson & Olson, 2000).

The decreased frequency oftechnology-related text messages across sessions suggests

that students overcame the difficulties that they experienced with the VTOC. This would be in

agreement with Lebie, Rhoades and McGrath (1996), who demonstrated that the need for

technological assistance was reduced as users became more experienced with the program. In

addition, Olson and Olson (2000) demonstrated that students work more effectively as they

overcome technological problems. The initial difficulties, therefore, may be considered

"growing pains," or part ofthe learning curve where students become proficient with the

particular technology.

Low course attrition during Phase lb. Attrition is a pervasive problem in DL courses in

higher education and training environments (Olson & Wisher, 2002; Phipps & Merisotis, 1999).

In the focus group discussions, students were asked ifthey had ever considered dropping out and

why. Seven ofthe eight students never considered it because they needed the course for career

progression. Lack oftime, was the reason given by the one student who contemplated dropping

out, but did not. The most common reason for completing the course was that the course

provided opportunity for career progression.

Cohesiveness in the learning environment. Students' opinions were mixed regarding the

effectiveness ofthe team building process during the synchronous sessions. One group reported

that, although they had never met face-to-face during Phase lb, they were truly a team before

arriving for Phase 2. These students said that text messaging allowed for both training task

communication emd social interaction. It enabled group members to get to know one another's

strengths and weaknesses. In addition, everyone in this group stated that allowing different

students to assume various leadership roles helped them learn to be "followers" and trust others.

The second group, however, stated that they did not feel much of a sense of group cohesiveness,

but instead were like "individuals struggling to work together as a team." At this point, there is

no clear reason for the difference in opinions between the groups, and the question ofwhat

influences group cohesiveness warrants further investigation.

Instructor's role as afacilitator. The eight students agreed that the role ofthe instructor

during Phase lb was more of a facilitator than a lecturer. In accordance with the original course
design goals, a key role ofthe online instructors was to provide direction and guidance that

facilitated learning instead ofjust lecturing. The students stated that feedback from the instructor

was important, and therefore they sought feedback on a consistent basis throughout the course in

order to assess their own progress.

One popular pedagogical technique used by the instructors in this course was early online

introductions, allowing students and the instructor to get to know each another. The purpose of

these introductions was to develop common ground among the students and instructors. Also, as

instructors monitor the students' text messaging, they can make a concerted effort to get

everyone to participate and to contribute to the real-time discussions. According to the students,

these facilitative techniques used by the instructors were successful in helping the students

succeed in the course.

Perceived advantages. Since the course was designed for reservists, most of whom had

full-time jobs and families, it was not surprising that the students mentioned convenience as the

primary advantage for the online course. The use of a DL format allowed the students to

complete the course according to their ovra schedules.

Also important to students was the "active" learning environment embedded in the

course. They appreciated the immediate feedback and mentoring from instructors, and most

students were not hesitant to contact the instructor via e-mail for such feedback. Some students

also stated that they learned to work as a team while online, and that the online environment

fostered more thoughtful comments than normally found in conventional classroom settings.

Perceived disadvantages. The students stated that a key disadvantage ofthe course

involved the length ofthe asynchronous and synchronous components. This is in agreement

with both Sanders and Guyer (2001) and Sanders and Bumside (2001), who reported that

students felt the asynchronous portion (Phase la) ofAC3-DL was too long and filled with too

much detailed information. During Phase la, the tests for each completed section used different

formats, which students found confusing. In addition, when students did complete a module,

they could not proceed to the next module without instructor permission, which detracted from

the "learn anywhere, learn anytime" benefit of online learning. This situation led most ofthe

students to skip portions of Phase la just to get through it, while others simply dropped the

course'.

In contrast, the students claimed that the synchronous portion (Phase lb) was too short.

Some students, for instance, mentioned that the length of Phase lb did not allow every group

member to play all roles in the simulated exercises. They suggested that additional VTOC

sessions would allow all members to experience the XO (Executive Officer) and S-3 (Operations

and Training) positions.

' Initial attrition rates for the first phase of AC3-DL was 50 percent for students who actively participated and

approached 75 percent if all students who enrolled were included (Sanders & Bumside, 2001). Discussions with

AC3-DL course developers and instructors indicated that the attrition rate was significantly reduced in later cohorts

as the course delivery methods were modified.

10
Students were concerned with the reliabiUty ofthe VTOC system and usefulness ofsome

ofits components. At times, the program removed students from the messaging rooms without

warning, causing the loss of valuable training time while disrupting workflow. In addition, the

map editing tool tended to freeze some computer systems, and the terrain viewer was not found

useful. Most students stated that they did not like the avatar representing their physical presence

in the VTOC operations areas and that the VTOC was not compatible with the Macintosh

computer platform.

To alleviate some ofthese problems, students recommended that the course designers and

instructors focus on basic technology shown to function effectively rather than the latest gadgets

or innovations. This is not surprising, since research shows that online instructors and students

tend to rely on simple tools such as e-mail, Web links to course material, and posting lecture

notes online (Bonk, 2001; Bonk 2002). Significantly fewer instructors use online chatrooms,

multimedia lectures, online examinations, animation, and streaming video. Using new

technology (e.g., voice over the Internet) can be a mistake ifit does not add perceived relevance

and effectiveness to the course or solve a key problem.

In addition to technological concerns, students mentioned a few other disadvantages. For

instance, the online examinations during Phase la offered minimal feedback other than test

scores. They also stated that some ofthe modules could have been delivered and tested in

smaller chunks, thereby focusing on specific accomplishments. Students seemed concerned with

information overload and a reduced sense of accomplishment early in the course, which could

result in attrition.

Interviews with Course Instructors

The interviews with course instructors yielded primarily positive results. Topics covered

included: a) their role during Phase la and Phase lb, b) the instructional techniques they used,

and c) the perceived advantages and disadvantages of presenting the course in its current form.

Interviews with the course instructors provided further insight into the pros and cons ofteaching

this complex, Web-based course.

Instructor's role asfacilitator. The online instructors did not view their roles as much

different than in the classroom. While instructors reported that they served more of a facilitative

role, providing students with the means, tools, and guidance to learn effectively, they contended

that the only major differences between teaching online and teaching in a classroom were that: a)

they could not see their students and b) instead ofwriting grades on students' assignments, they

sent them e-mails. Two instructors emphasized that they allocated the bulk oftheir time to

course planning, but the third instructor indicated that he had spent much more time on the

administrative aspects ofthe course. They all mentioned the importance of facilitating good

decision-making and problem solving so that students can apply what they learn to real-life

exercises.

Instructional techniques. The instructors stated that particular instructional strategies and

pedagogical approaches were useful in specific online environments. During the asynchronous

phase, instructors used directive and one-way instructional techniques aimed at teaching basic

concepts and information, while during the synchronous phase instructors used indirect

11
questioning, prompting, and nudging. It is possible that during the asynchronous phase, the

students may have needed a structured approach because they lacked background knowledge. In

contrast, during the synchronous phase the students may have had a deeper level of

understanding to build from and were able to respond well to constructivist techniques. This

difference in teaching techniques may have been due to the asynchronous-synchronous

difference, or it may be due to other variables, such as the students' level of understanding

during a particular phase.

The instructors indicated that they were genuinely interested in student progress

throughout the course and made attempts to be available to the students. For example, they sent

out weekly reminders about assignments and used indirect questioning and prompting to engage

students. Where possible, they also attempted to place each person in a leadership position within

their groups to boost individual confidence levels. Instructional tactics, such as selecting

students to be in charge of activities, were intended to boost student participation during the

synchronous component. The instructors also found that matching weaker students with strong

leaders was beneficial. In fact, they noted that this often resulted in the respective groups

supporting poor performers on their own with little help needed from the instructors.

Perceived advantages. As a whole, the instructors stated that the online course fit nicely

into a small group instruction model and strongly complemented the Army's "crawl, walk, run"

philosophy of learning. In effect, the asynchronous portion (Phase la) provided the basic

foundation ("crawl"), the synchronous portion (Phase lb) allowed the students to put their

knowledge to use in electronic and paper formats ("walk"), and the resident portion (Phase 2)

prompted the students to fully apply their knowledge and skills in real-life scenarios ("run").

No instructor reported difficulties with the technology or the instructional methodology.

One ofthe advantages ofthe Web-based course mentioned was the ability to provide detailed

feedback to students, which promoted greater learning and application of knowledge by students.

In terms ofthe asynchronous phase, the instructors claimed that students were effectively

mentored as they progressed through the course learning new concepts and ideas. Moreover,

they felt that feedback was promptly provided during Phase la and was based on progress as

well as performance. Most ofthis feedback was provided through e-mail. Additionally, the

synchronous component (Phase lb) offered students immediate feedback from both instructors

and peers using either messaging or voice communication. Overall, the course instructors gave

the course design a positive rating.

The instructors stated that another key advantage ofthe synchronous course sections

involved teaching students how to work with others as a team to solve a problem. In addition to

problem solving and teamwork, instructors contended that using Web-based communication

applications enhanced students' communication skills. The rise of communications technology

and the need for team skills in most work settings support claims that distance technologies have

a positive impact and promote important skills.

As an added benefit, instructors mentioned that interactive courses could give students

the most current and updated material, equipping reservists with skills and training equal to that

12
of active duty soldiers. The DL course also provides an additional avenue for those who want to

advance their military careers and earn credits towards an advanced university degree.

Overall, the instructors stated that they enjoyed teaching the course online and using the

technology. The instructors reported that the students going through the DL course were better

trained than those taking it through a previous correspondence version. They added that DL

students gain general skills such as problem solving and group communication that are

applicable to any position in the Army and could not be gained from a correspondence course.

In fact, they recommended replacing all correspondence courses with distributed learning

versions, especially for Army recruiters and commanders spread out across the United States.

Perceived disadvantages. While these instructors did not indicate many disadvantages

with the overall course, they did report fairly high attrition rates during the asynchronous phase

(Phase la). They sensed that part ofthe problem was not being able to control the size of

modules. As a result, these students had to fit a fairly robust and demanding curriculum into

their already full lives. According to the instructors, early modules in the asynchronous phase

were particularly difficult. However, those who made it to the later modules usually had the

stamina and motivation to complete the entire course.

Interview with DL Education Advisor

Overallperspective. The DL Education Advisor for the Armor School provided some

valuable information concerning the design of DL courses, which echoed many ofthe findings

from interviews with students and instructors. The DL Education Advisor indicated that she was

quite pleased with the program and was an avid supporter. At the same time, she was interested

in additional course evaluation, specifically how to improve student completion rates, fine-tune

course production and system resources, and enhance online Web-based instruction tools and

strategies. She emphasized the fact that anyone involved in the development of DL technology

must be flexible and adaptable. The technology is changing so rapidly that one caimot just look

at where the technology is now, but must also consider where it will be a year from now.

The DL Education Advisor said that applying the skills learned during the course was key

to its success. She pointed to practical exercises that were embedded in the course to help

students learn the content. For the purpose oftraining students to command companies and

perform other duties at battalion and brigade levels, it was imperative to focus on bottom-line

command readiness.

Advice. For those wanting to replicate aspects ofthis program, the DL Education

Advisor provided several caveats and tips. She offered six important considerations in the

design of DL courses. First, all courses should involve direct, e-mail feedback. Her rationale

was that students need to feel connected to other students and to the instructor. Second, she

claimed that courses should have meaningfiil content that students can directly apply to real-life

exercises. Concerning the AC3-DL course, meaningfiil content typically involved combat

situations and combat readiness. Third, there should be minimal extraneous content (e.g., extra

graphics and non-essential information) to limit student confiision and course complexity. Her

team found that students skipped optional or peripheral materials ifthey felt overwhelmed.

Fourth, designers of DL courses should carefiiUy analyze their target audiences so they can

13
accurately determine what the students want and need. Fifth, it is vital to create an active

learning environment with a balance between flexibility and learner accountability. Not

surprisingly, she readily admitted that the instructor is a key part ofthat environment, providing

direction and feedback. With prompt instructor feedback, students were not isolated in their

online learning endeavors. Lastly, designers should limit their visions and not stretch the

expectations oftechnology too far beyond the tools and options that have been proven to work.

When technology is pushed to the limits ofits capabilities, then breakdowns are more likely.

The DL Education Advisor argued that there would always be room for improvement in the

future, but that one has to start somewhere.

Limitations. The DL Education Advisor noted several problems with the current system.

First, many students wanted printed copies ofthe course materials. Given the online availability

ofthe course materials, however, she felt that this would amount to an excessive waste of paper.

Second, the learner-management system was not flexible enough for most ofthe students and

instructors. For instance, some students voiced frustration that they could not move on to

another volume ifthey missed too many items on the gate test. Third, since there was not

ubiquitous access to the Internet, some activities and events were not always available to

students across settings. Fourth, in addition to Internet access, some students wanted access to

course materials via CD-ROM. The DL Education Advisor noted that the reason for Internet

delivery ofthe course was so that student's progress through the course could be easily tracked.

SUMMARY

Previous research specific to this course found that the blended approach, a combination

of Web-based and face-to-face lessons, used in the AC3-DL course was at least as effective as

the correspondence version (Sanders & Bumside, 2001; Sanders & Guyer, 2001). In the initial

version ofthe AC3-DL course, the synchronous section was not started until the asynchronous

phase was completed. The course, however, has been modified once again, so that the

asynchronous and synchronous phases are intermingled. This approach seems to be time

efficient, allowing students to complete the training in a shorter timeframe.

Communication Trends

The communication trends noted during synchronous training (Phase lb) provide insight

to the developmental process during virtual team training. During the initial sessions, time and

resources must be allocated for social bonds to develop and for the familiarization oftechnology.

Course developers and instructors who incorporate computer-supported collaborative teams into

curricula should be aware ofthe time/resource requirements for both social interaction and

developing reasonable skills with the course technology.

In the current research, a majority ofthe messaging discussions focused on the task.

Although always high, there was an increased likelihood that task-related text-messaging

instances occurred during the middle sessions rather than the beginning or end. This was

unanticipated, because the exercises across all meetings, similar in content and difficultly,

required the same attention on task. However, the findings on social interaction may point to a

reason for the inverted U shape found in the task data. During the first and last session, there

were elevated levels ofsocial text messages. The text messaging data showed that the first

14
session was influenced by initial greetings and getting to know one another conversations.

During the last session, individuals spent quite a bit oftime saying goodbye and planning to meet

up and carpool to the "in-residence" training which followed. This social chat trend was

complementary to the task related messaging trends; when social messaging decreased, task

related messaging increased.

Participant Reactions

Qualitative interviews were useful in identifying aspects ofthe course that made it

successful or inhibited it from being even more successful. While the participants stated that

length of Phase la was a problem leading to high levels of attrition, they also acknowledged that

all course material should be covered. The recommendation ofthe participants was that material

should be segmented into more sections, so that they can be learned in smaller chunks.

The course supervisor and the instructors offered a different perspective to the course.

One suggestion was that the development of a DL course should focus on content and not on the

tools oftechnology. This is in agreement with prior research (Wisher & Cumow, 1999)

demonstrating that additional non-content related information did not improve learning.

Interviews with students, course instructors, and the course supervisor, revealed some

interesting findings. During the synchronous DL phase, the role ofthe instructor was that of a

facilitator instead of a lecturer. Also, learning how to use the technology was a concern to both

the students and the instructors. Therefore, the course supervisor suggested that for instructional

purposes course designers should not attempt to stretch the technology to the limits ofits

capabilities because that increases the frequency of problems. Taken together, the results from

the analysis oftext chat communication and participant reactions provide a broad picture of an

effective blended approach to course delivery.

15
16
REFERENCES

Alkhateeb, H. M., & Jumaa, M. (2002). Cooperative learning and algebra performance of eighth

grade students in united arab emirates. Psychological Reports, 90, 91-100.

American Psychological Association. (1997). Learner-centeredpsychological principles: A

frameworkfor school redesign and reform. Washington, DC: Author. Retrieved May 7,

2002 from http://www/apa.org/ed/lcp.html.

Army Logistician (2001). Deployable campus enhances training. (PB 700-01-5) 33 (5), p.42.

Bonk, C. J., Olson, T. M., Wisher, R. A., & Orvis, K. L. (2002). Reflections on blended

distributed learning: the armor captains career course. (Research Note 2003-06).

Alexandria, VA; U.S. Army Research Institute for the Social and Behavioral Sciences.

Bonk, C. J. (2002). Online training in an online world. Bloomington, IN: CourseShare.com.

Retrieved on January 15, 2003, from http://publicationshare.com/docs/corp_survey.pdf.

Bonk, C. J. (2001). Online teaching in an online world. Bloomington, IN: CourseShare.com.

Retrieved on January 15, 2003, firom http://publicationshare.com/docs/corp_survey.pdf.

Bonk, C. J., & Wisher, R. A. (2000). Applying collaborative and e-learning tools to

military distributed learning: a researchframework. (Technical Report 1107).

Alexandria, VA; U.S. Army Research Institute for the Social and Behavioral Sciences.

Deatz, R. C, & Campbell, C. H. (2001). Application ofcognitive principles in distributed

computer-based training. (Research Product 2001-03). Alexandria, VA: U.S. Army

Research Institute for the Social and Behavioral Sciences.

Deloughry, T. D. (December 8, 1995). Making cormections. The Chronicle ofHigher

Education. A21 -A24.

Duffy, T. M., & Cunningham, D. J. (1996). Constructivism: Implications for the design and

delivery of instruction. In D. H. Jonassen (Ed.), Handbook ofresearch on educational

communications and technology (pp. 170-198). New York: Scholastic.

Fisher, M., & Coleman, B. (2001-2002). Collaborative online learning in virtual discussions.

Journal ofEducational Technology Systems, 30(1), 3-17.

Fleiss J. L. (1981). Statistical methodsfor rates andproportions. Inded. (pp. 38-46). New York:

John Wiley.

17
Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and

computer conferencing in distributed education. The American Journal ofDistributed

Education, 15 (1), 7-23.

Hillman, D. C. A. (1999). A new method for analyzing patterns ofinteraction. The American

Journal ofDistributed Education, 13 (2), 37-47.

Howard, F. S. (1992). Distributed Training Pilot-MOS 75B Distributed Training Course

Training Effectiveness Analysis. (TRAC-WSMR-TEA-92-031) US Army TRADOC

Analysis Center-White Sands Missle Range.

Hudson, J. M. & Bruckman, A. S. (2002). IRC francais: the creation of an internet-based SLZ

community. Computer Assisted Language Learning, 15(2), 109-134.

Kang, I. (1998). The use of computer-mediated communication: Electronic collaboration and

interactivity. In C. J. Bonk & K. S. King (Eds.), Electronic collaborators: Learnercentered technologiesfor literacy, apprenticeship, and discourse (pp. 315-337). Mahwah,

NJ: Erlbaum..

Kulik, C. C, & Kulik, James A. (1991) Effectiveness of computer-based instruction: an

updated analysis. Computers in Human Behavior, 7, 75-94.

Lebie, L., Rhoades, J. A., & McGrath, J. E. (1996) Interaction process in computer-mediated and

face-to-face groups. Computer Supported Cooperative Work, 4, 127-152.

Mezko, J., Redding, G. A., & Fletcher, J. D. (1996). Distributed learning and the reserve

components. Institute for Defense Analysis (Document D-1941).

Moore, M. G. (2001) Surviving as a distance teacher. The American Journal ofDistance

Education, 15(2), 1-5.

Muilenburg, L., & Berge, Z. L. (2001) Barriers to distributed education: a factor-analytic study.

The American Journal ofDistributed Education, 15 (2), 7-24.

Olson, G. M. & Olson, J. S. (2000). Distance matters. Human-Computer Interaction. 15,

139-178.

Olson, T. M. & Wisher, R. A., (October, 2002). The effectiveness of web-based instruction: an

initial inquiry. International Review ofResearch in Open and Distance Learning (Online

Journal). Retrieved January 15, 2003 from http://www.irrodl.Org/content/v3.2/olsen.html.

Orvis, K. L., Wisher, R. A., Bonk, C. J., & Olson, T. M. (2002). Communication patterns during

synchronous web-based military training in problem solving. Computers in Human

Behavior, 18,7^3-795.

18
Palincsar, A. S. (1998). Social constructivist perspectives on teaching and learning. Annual

Review ofPsychology, 49, 345-75.

Pena-Shaff, J., Martin, W., & Gay, G (2001). An epistemological framework for analyzing

student interactions in computer mediated communication environments. Journal of

Interactive Learning Research, 12(1), 41-68.

Phipps, R., & Merisotis, J. (1999). What's the Difference? A Review ofContemporary Research

on the Effectiveness ofDistance Learning in Higher Education. Institute for Higher

Education Policy. Retrieved January 15, 2003, from the World Wide Web:

http://www.ihep.org.

Program Management Office, The Army Distance-learning Program (2000). The Army

Distance-learning Program Brochure, Fort Monroe, VA: Department ofthe Army.

Retrieved from http://www.tadlp.army.mil/brochure.pdf

Sanders, W. R. & Bumside, B. L (2001). Assessment ofinitial delivery ofthe armor captains'

career course (distributed learning). (Research Report #1775). Alexandria, VA; U.S.

Army Research Institute for the Social and Behavioral Sciences.

Sanders, W. R., & Guyer, C. W. (2001). Commanders' survey: Armor captains 'career course

(distributed learning). (Research Report #1771). Alexandria, VA; U.S. Army Research

Institute for the Social and Behavioral Sciences.

Schaab, B. B., & Moses, F. L. (2001) Six myths about digital skills training. (Research

Report #1774). Alexandria, VA; U.S. Army Research Institute for the Social and

Behavioral Sciences.

Wisher, R. A., & Cumow, C. K. (1999). Perceptions and effects ofimage transmissions during

internet-based training. The American Journal ofDistributed Education. 13 (3), 37-51.

19
20
APPENDIX A - FOCUS GROUP QUESTIONS

Questions to Students

1. What was the most faciUtative aspect ofthe onhne chats?

2. What were the advantages oftaking this course onhne versus in a classroom?

3. How did you view the role ofthe instructor in this course? Did this view change over the

course ofthe class? What did you like best about the instructor? What did you like least

about the instructor?

4. How did the tools foster student interaction?

(terrain viewer, map edit, text chat, voice chat, bookshelf, word, role-playing as Avatar,

e-mail, Spearhead simulation)

5. What advice could you give people just entering this course concerning the most

effective ways to interact with others online?

6. How did you know that you were "off-track" when communicating with others online?

7. Did you have a sense that someone was facilitating or leading the chars? How did you

know?

8. When did you use the online chats versus the audio chats?

9. What were the most common technology-based problems that you encourtered

throughout the course?

10. What methods did you use to develop a sense of "team identity" with your other team

members?

11. How often did you seek feedback from the instructor? How often did you seek feedback

from peers? What methods did you use to obtain this feedback?

12. What percentage oftime during the course did you work without interaction with another

team member or the instructor?

13. Did one person emerge as the "leader" ofthe team during the course? How did this

individual behave to indicate to you that he was the leader?

14. How did you and the other group members resolve conflicts that arose?

15. Do you see the impact ofwhat you learned in Phase I training in Phase II? Do any ofthe

simulation skills transfer? Ifso, how do they help you in Phase II?

A-1
16. How often do you play simulation-based computer games? Did that help you in Phase II

training?

17. Did you ever think of dropping out? If so, when and why?

A-2
APPENDIX B - INSTRUCTOR INTERVIEW QUESTIONS

1. What tools do you think were the most helpful in getting students to learn online?

2. What instructional strategies or pedagogical approaches were particularly useful? How

do you know that they were useful?

3. How did you assess student learning? How would you change your assessment of

student learning in the future?

4. What did you like best about teaching online?

5. What do you like least about teaching online?

6. What advice would you give other instructors who were considering teaching a course

online?

7. What advice would you give potential funders of online projects?

8. What goals did you have in mind when teaching this class?

9. Where and when do you think the most learning took place?

10. Would you teach this way again? Why or why not?

11. What aspects ofthe online course do you think students had the most problems with?

Were you able to deal with those problems effectively?

12. How did you view your role as an instructor? Do you think your role would have been

different if you had taught in a classroom instead of online? How would it be different?

13. How often did students request feedback? How did you provide this feedback?

14. Did you have any students that were not as actively participating as the rest? What

methods did you use to increase their participation?

15. What percentage oftime did you allocate to plarming and preparation? Online teaching?

Administration? Interaction with students? Interaction with content?

16. Who typically drops out ofthe live and online courses? Why?

17. What are the key outcomes you want from the online course?

B-1