the AC3 DL class, and the combined scores for the five AOAC RC classes. Tables presenting
detailed Resident Knowledge Test scores for the AC3 DL class and each of the five AOAC RC

classes are provided as Appendix H.

Table 6

Resident Knowledge Test Scores for AC3 DL and AOAC RC Classes

AC3 DL (n=15) AOAC RC (n=149)

Raw Score Percent Raw Score Percent

Correct Correct
Mean 37.67 60.76 34.55 55.72
Median 37.00 59.68 35.00 56.45
Std Dev 5.04 8.13 5.90 9.53

Statistical analyses were conducted to compare test scores across the AC3 DL and AOAC
RC classes. A review of the AOAC RC data suggest that Resident Knowledge Test scores are
not normally distributed, so that parametric statistics are inappropriate to test for equality of
variance. A Kruskal-Wallis H test for equality of variance failed to find a significant difference
across the AOAC RC classes (Chi-Square = 3.88, df = 4) so that the AOAC RC scores can be
pooled for comparison to AC3 DL test scores. The mean Resident Knowledge Test raw score
(62-item test) for AC3 DL students was 37.67, compared to a mean raw score of 34.54 for
AOAC RC students. A distribution-free (nonparametric) test was used to compare Resident
Knowledge Test scores between the AC3 DL and AOAC RC programs. The Mann-Whitney U
rank-order test was used where all cases are ranked in order of increasing size, and U (the
number of times a score from group 1 precedes a score from group 2) is computed. The Mann-
Whitney U test comparing AC3 DL vs. AOAC RC Resident Knowledge Test scores approached
significance, U = 781.50, n= 15/149, p = .055, providing evidence that students in the AC3 DL
program did better on the Resident Knowledge Test than did students in the AOAC RC program.

AC3 DL Volume IV and IV EOV vs. AOAC RC End of Course Scores

Table 7 provides a very rough estimate of students’ subject matter knowledge at the
completion of a block of training. The first and second columns provide a comparison of the
AC3 DL combined Volume IV and V EOV mean score vs. the AOAC RC End of Course
percentile scores summed across four AOAC RC classes. End of course scores were not
available for AOAC RC Class 01. Statistical testing of the size of the difference between the
AC3 DL and AOAC RC scores is not appropriate, as each course completed different tests.

10



8. What is your view of the quality of AC3 DL ASYNCHRONOUS portion of the training?
0/0% VERY HIGH

12/80%___ HIGH

2/13% ___ ACCEPTABLE

1/7% LOW

0/0% VERY LOW

Student Comments:

e High. Some programming problems with course work.

e High. I thought that there were too many errors in the testing questions even for a pilot
program. I have been told that many of the errors identified by the pilot students are still
in the courseware.

High. Good info, too detailed.
High. Course material is very good. Again, why not ona CD. Time is wasted on
connectivity, and pulling up each web page.

9. What is your view of the quality of AC3 DL SYNCHRONOUS portion of the training?
2/13% VERY HIGH

9/60%, HIGH

3/20%, ACCEPTABLE

1/7% LOW

0/0% VERY LOW

Student Comments:

¢ Low- poor software (Blaxxun/ Map Edit)

e High. Work out bugs in Map Edit! Do not do VTOC every month for several months in
arow. I miss drills and have more conflicts with civilian job.

10. What is your view of the quality of AC3 DL RESIDENT portion of the training?
5/33%. VERY HIGH

9/60%, HIGH

U/7%. ACCEPTABLE

0/0% LOW

0/0% VERY LOW

Student Comments:
e High. You are definitely in the ballpark. Lets see more on medium brigade concepts,
future of force projection and Operations Other Than War!

11. What is your view of the quality of AC3 DL training overall?
3/20% VERY HIGH

9/60% ___ HIGH

3/20% ACCEPTABLE

0/0%. LOW

0/0% VERY LOW

1-3



Phase IA Correspondence Course Training Times

Appendix G

Resident Phase of Course

Correspondence Phase (Mean/Std Dev)

Class Date Size Months Since Months Since Months in
Start Finish Correspondence
Training Phase
AC3 DL 01 Oct 2000 15 13.00/2.45 5.47/3.62 7.53/3.09
AOACRCO1 Oct 1999 = 32 27.40/13.22 10.93/5.71 16.47/10.07
AOACRC02 = Jun 2000 «25 ~—.29.06/10.62 13.16/6.09 15.90/8.15
AOACRC 03 Jun 2000 = 30 31.51/11.26 14.79/6.47 16.73/10.68
AOACRC04 = Jun 2000 28 ~—29.06/9.00 13.22/6.26 15.83/7.24
AOAC RC 05 Nov 2000 34 30.63/10.68 13.65/6.39 16.99/9.31
AOAC RC Total 149 = 29.56/11.02 13.14/6.24 16.42/9.11

Five AOAC RC cases were not included where Months Since Start = Months Since Finish.



13. What are the specific benefits of being in the resident portion of the course?

Student Comments:

e Face to face is most valuable, learning from each other, the ability to concentrate on
subjects. The instructor can tell if students got it, or need instruction again. It is really
valuable for students to be able to compare products, can't hide in resident portion of the
course. VTOC facilitates later resident training.

¢ The two week resident portion of the course is necessary for the legitimacy of the course,
can't have a pure DL course. Eight of the students in the resident training are on vacation

time, two are on leave.

14. What are your overall perceptions about the course (ASYNCHRONOUS,
SYNCHRONOUS, RESIDENT?

Student Comments:

© This is the only way some could take the course, could not do a resident course. AGR
student was able to stay in unit and do this course. "I feel I learned a whole lot more —
better skills than if I had gone the book route."

CD will help/can go back and check which students could not with web based material.
Problems of VTOC groups, problem when one member can't participate.

Would like to be able to search source documents.

Armor School representative mentioned that for the next iteration of the course students
will be allowed into synchronous training as soon as they complete relevant Phase 1
content, like after Volume 3.

Asynchronous knowledge test, frustrating that you don't get feedback.

Six credit hours not accepted everywhere.

eoeoee

15. How well did the TACOPS simulation support learning in the Resident portion of the
course?

a. Would it have helped you if you had a copy of the TACOPS simulation to use
before you got to Ft. Knox?

Student Comments:
e None of the students wanted it ahead of time.

b. Would it be useful to you to take a copy of TACOPS with you?

Student Comments:

© Students like it, but would like to have it give unit specific feedback. Twelve students
said they would like to take it with them.

16. What other factors do you think impact either positively or negatively on the course?

Student Comments:
e No responses beyond what was already provided.

4



personnel reviewed all test items to ensure the content was covered both in AOAC RC and AC3
DL lessons. The Resident Knowledge Test was administered to AC3 DL and AOACRC
students on the day of their arrival for resident training at Fort Knox. The test scores thus
represent the knowledge that students retained over time since they completed the non-resident
portion of their training. One limitation of this assessment is that it does not provide a pure
estimate of AC3 DL Phase IA training, since AC3 DL students completed both Phases IA and IB
of the course prior to their arrival at Fort Knox, and Phase IB training could contribute to higher
Resident Knowledge Test scores for AC3 DL students.

Company Team Test. It was desired to obtain estimates of AC3 DL students’ knowledge
of company team operations based exclusively on Phase IA training, prior to Phase IB training,
for comparison with AOAC RC students’ company team knowledge. The AC3 DL students
complete company team instruction during Volumes IV. and V of Phase IA training, and their
knowledge is routinely assessed with EOV tests. In contrast, AOAC RC students complete their
company team instruction during resident training at Fort Knox, and their knowledge is assessed
with an End of Course test. By comparing AC3 DL EOV scores to AOAC RC End of Course
scores the learning achieved for students in the two courses can be compared, using estimates
taken immediately at the completion of training. A serious limitation of this comparison is that
scores from two different tests are being compared. The average test scores for the two courses
can be documented, but a statistical comparison would not be valid.

Small Group Instructor (SGI) Assessment. One active duty SGI conducted Phase II of
AC3 DL. This SGI had previously handled the resident portion of AOAC RC, and was thus able
to make comparisons of the two groups of students. The SGI was asked to compare the two
groups of students through a detailed structured interview addressing specific strengths and
weaknesses of each group. The AC3 DL SGI Interview Guide was prepared and used to provide
the structure for the interview (see Appendix E).

Data Collection and Analysis Approach

The goal of the present research was to assess the Phase IA asynchronous correspondence
course portion of the AC3 DL program. Measures of student performance had to be obtained on
a non-interference basis during the conduct of the AC3 DL and AOAC RC programs. One
constraint on the research was that several factors potentially impacting student performance
could not be controlled. High rates of student attrition, additional practice provided by AC3 DL
Phase IB synchronous training, and differences in the length of delay between correspondence
course training and resident training for AC3 DL and AOAC RC students could all contribute to
differences in student performance. For these reasons observed differences in AC3 DL and
AOAC RC student performance on the 62 item Resident Knowledge Test, and their performance
during resident training will likely reflect contributions from both AC3 DL Phase IA
asynchronous and Phase IB synchronous training, as well as other factors.

Personnel from USAARMS extracted AC3 DL within-course measures from the student
management database and provided them to ARI personnel for analysis on a regular basis. These
data included lesson pre-test and post-test scores, and EOV test scores. The USAARMS
personnel arranged for the administration of the Resident Knowledge Test, and ARI personnel



References

AB Technologies Inc. (1998, July). Armor Officer Advanced Course: Reserve Component
Distributed Learning Course Design. Alexandria, VA.

Alden, J. (1998). Trainer’s Guide to Web-Based Instruction. American Society for Training and
Development, Alexandria, VA.

Draves, W. (2000). Teaching Online. Learning Resources Network, River Falls, Wisconsin.

U.S. Army Armor School (2000). AOAC-RC Non-Completion Rates 1995-1998, (unpublished
briefing slide), Fort Knox, Kentucky.

Burnside, B., Wardell, C., & Sanders, W. (1999). Armor Captains’ Career Course (Distance
Learning) Assessment Plan — Phase IA (unpublished research plan), Fort Knox,
Kentucky.

15



12.

Student Comments:

Acceptable. Offers a good alternative, but I don't feel chain of command understands
time requirements.

Please provide any comments you would like to make on AC3 DL training.

Student Comments:

It is good training and we had a great division with outstanding equipment. We were
given some great info. Also, Phase I should be done with a disk, not online then you mail
your test and answers in or just do the test (online) not the courseware!

Asynch phase would be enhanced if we had access to material on and off line. On a CD
ROM with codes given at the end of a Gate to allow a Gate test online.

Smaller blocks of instruction - some blocks were 9-12 hours blocks. If these could be
broken down to 1-2 hour blocks, you may have 6 additional lessons, max it would be
easier for the student.

1) Need more one on one feedback from SGI's on student progress.

2) Enforce standards (i.e. suspense dates for work completion)

Too long to complete, a two-year process. At maximum it should be a 1 1/2 year
process.

Have a downloadable version of the ASYNCHRONOUS portion. That way we can
study off line. Phase I has too many OPORD's.

I learned a lot of information in a very short period of time. I did not start the course
with a lot of practical experience - therefore I had to do a lot of independent reading to
round out my learning to succeed in the course. I am happy to say I participated in the
Pilot Program.

Compress the timeline. Run Phase I in tandem with Phase II. Need more student/ SGI
interaction during Phase I. Support staff needs more people to cover longer hours for
those students who work after 5 pm.

This is one of the best courses I have ever participated in. It needs to have dedicated,
continuous funding to continue to be offered and improved for Reserve Officers.

A lot of straight OPORD's and tactics. However, as an armor CO CDR I spend more
time with training issues and plans, soldier issues, recruiting, armory activities, and
paperwork than I do with OPORD's. Need to focus on other aspects of command also.
Detailed AAR to follow course.

While most RC Officers are computer literate, not all have computers or even need them
for home use. Some use PC's at work for emailings, etc, but would not be allowed to do
coursework. Therefore they would have to purchase one just for this course. Solution:
possibly issue laptops with hand receipt.

Overall the course was excellent. A couple of items that should be considered: Make
the asynchronous material available off line (i.e., on a CD ROM). The tests should still
be online, however study material not tied to the internet would be help. I see the same
benefit with either method. Secondly, work the software conflicts with VTOC, Map Edit
etc, so they are not training distracters.

Army is saving thousands of dollars (per) soldier and end user is not receiving (any)
benefits through resources or pay. Currently it is a third job to a busy schedule
(Command responsibilities, civilian job, and then AC3).

1-4



FOREWORD

The Armored Forces Research Unit (AFRU), U.S. Army Research Institute for the
Behavioral and Social Sciences (ARI) has a long history of conducting research and
development with innovative training methods and delivery techniques. The U.S. Army Armor
School (USAARMS) has recently developed a revised Armor Captains' Career Course (AC3),
formerly known as the Armor Officers’ Advanced Course (AOAC), for Reserve Component
(RC) officers using the Total Army Training System conversion process. The primary delivery
means for the revised course is distance learning (DL) techniques. In June 1999, the Director of
the USAARMS requested that the ARI AFRU at Fort Knox provide Technical Advisory Service
to assist in assessment of the new AC3 DL program (Memorandum entitled “Technical
Assistance in Support of Armor Captains’ Career Course Distance Learning,” dated 24 June
1999), The request referred specifically to an assessment of the Internet-delivered initial phase
of the course. In response, the ARI AFRU formed a small team to work with USAARMS
personnel on the AC3 DL assessment. This effort was completed as part of AFRU’s Work
Package 205, “Assessment of Force XXI Training Tools and Techniques.”

This report describes the assessment of the initial delivery of the AC3 DL program. The
primary objective of this assessment initiative was to determine whether AC3 DL students learn
at least as effectively as students in the previous AOAC RC program for subject material that is
common to both course versions. A secondary objective was to identify additional material
covered as well as any additional capabilities and benefits provided by AC3 DL as compared to
AOAC RC. The research was conducted by comparing AC3 DL and AOAC RC students’
knowledge of course material, by analyzing archival course data, and by surveying and
interviewing AC3 DL students and their instructor.

The information provided in this report will be useful to the USAARMS and other
organizations involved in training soldiers and leaders through DL methods. It includes
feedback on RC personnel and training policies that can impact successful completion of AC3
DL and future courses delivered through DL means. The detailed results of this assessment were
provided to senior USAARMS personnel on 11 April 2001.

NA

‘A M. SIMUTIS
echnical Director



8. Duty Position

ARPLDR 0 3 (2.0%)
S4/G4 1 (6.7%) 3 (2.0%)
SPEC STAFF 0 12 (8.1%)
SEC STAFF 0 9 (6.0%)
SPECIALTY PLDR 1 (6.7%) 8 (5.4%)
OTHER PLDR 0 4 (2.7%)
LINE CO XO 4 (26.7%) 25 (16.8%)
HHC XO 1 (6.7%) 6 (4.0%)
CO CDR 6 (40.0%) 59 (39.6%)
S1V/G1 0 7 (4.7%)
$2/G2 0 6 (4.0%)
$3/G3 0 7 (4.7%)
No Code Given 2 (13.3%)
9. Years in Position

<1 yr. 10 (66.7%) 68 (45.6%)
1-2 yrs 5 (33.3%) 54 (36.2%)
2-3 yrs 16 (10.7%)
3-4 yrs 5 (3.4%)
4-5 yrs 2 (1.3%)
>5 yrs 4 (2.7%)
10. Years in Regular Army

<l yr. 100 (67.1%)
1-2 yrs 10 (66.7%) 9 (6.0%)
2-3 yrs 2 (13.3%) 9 (6.0%)
3-4 yrs 0 16 (10.7%)
4-5 yrs 1 (6.7%) 11 (7.4%)
>5 yrs 1_ (6.7%) 3 (2.0%)
11. CTC Rotations

0 10 (66.7%) 100 (67.1%)
1-3 5 (33.3%) 43 (28.9%)
4-6 0
7-9 1 (0.7%)
>10 5 (G.4%)
12. Yrs Experience at PLT
Level
NA 0 4 (2.7%)
1-2 yrs 4 (26.7%) 36 (24.2%)
3-4 yrs 8 (53.3%) 53 (35.6%)
5-6 yrs 3 (20.0%) 37 (24.8%)
>6 yrs 0 19 (12.7%)



Table 7
AC3 DL Volume IV and V Post-Test vs. AOAC RC End of Course Scores

AC3 DL AOACRC AOACRC AOACRC AOACRC AOACRC

VolIV&V Total 02 03 04 05

(n=15) (n=126) (n=27) (n=33) (n=28) (n=38)
Mean 85.40 82.15 82.69 82.21 80.75 82.75
Median 88.00 82.35 83.20 81.60 81.30 82.40
StdDev 7.27 5.22 5.80 4.73 4.34 5.88

Note. Numbers reflect test score percentage correct 0 — 100%.
AC3 DL Resident Training End of Course Written Survey

A 12-item AC3 DL resident Course Survey was developed and administered to the AC3
DL class at the completion of Resident training. The Resident Course Survey, annotated with
student responses, is provided as Appendix I. Students were very positive regarding the quality
of the asynchronous, synchronous, and resident portions of the AC3 DL program, with 80% of
the students rating the overall quality of AC3 DL training as “High or Very High.” The
information gained from the survey can help training developers to better understand the
constraints students face in completing the course. As an example, 86% of students reported that
they used their own personal computer to work through course materials, which suggests that
training must be flexible enough to allow delivery across a wide variety of platforms. Students
also provided valuable feedback regarding problems associated with lengthy lessons.

AC3 DL Resident Training End of Course Class Interview

An interview designed to gather course lessons learned was conducted at the completion
of resident training using a 16-item structured AC3 DL Resident Course Interview Guide. This
guide, annotated with student responses, is provided as Appendix J. Students stated that the lack
of a compact disk (CD) or downloadable files (as an alternative to accessing course materials on
the Internet) limited their ability to participate in the asynchronous self-study portion of the
course. Also, the sheer volume of material covered, large blocks of instruction, and size of
practical exercises were seen as limiting factors. Students indicated that AC3 DL should be
treated like other courses, with formal recognition of course completion, and the awarding of
retirement points. Students also stressed that the resident portion of the course is necessary for
the legitimacy of the course.

AC3 DL Resident Training End of Course Instructor Interview

An interview designed to gather course lessons learned from the AC3 DL class instructor
was conducted at the completion of the resident course. Part 1 of the interview addresses a broad
range of AC3 DL program management issues. Part 2 of the interview examines AC3 DL

11



11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

Number of CTC Rotations:
1.0 2. 1-3 3. 4-6 4. 7-9 5.>10

Years Experience at Platoon Level:
1.N/A 2. 1-2yrs. 3. 3-4yrs. 4. 5-6 yrs. 5. >6 yrs.

Years Experience at Company Level:
1.N/A 2. 1-2yrs. 3. 3-4yrs. 4. S-6 yrs. 5. >6 yrs.

Years Experience on staff:
1.N/A 2. 1-2yrs. 3. 3-4 yrs. 4. 5-6 yrs. 5. >6 yrs.

Most Recent CTC Rotation:
1.N/A 2. 1-2yrs. 3. 3-4 yrs. 4. 5-6 yrs. 5. >6 yrs.

Highest Enlisted Rank achieved:
1.N/A 2.E-3 3. E-4 4.E-5 5. >E5

Other Military Courses Completed: Mark all that apply.
1. Air Borne/Air Assault 2. Ranger/Special Forces 3. BMOC/Other CSS
4. SPLC 5. TC3/ NBC Defense/Other

How long ago did you start the correspondence phase of the course?
1. 1-3 months 2. 4-6 months 3. 7-9 months 4. 10-12 months
5. 13-15 months 1&2. 16-18 months 1&3. 19-21 months 1&4, 22-24 months
1&5. 25-27 months 2&3. 28-30 months 2&4. 31-33 months 2&5. 34-36 months
3&4. 37-40 months 3&5. 41-43 months 4&5. 44-47 months
1,2 &3. 48-50 months 2,3 & 4. > 50 months

How long ago did you finish the correspondence phase of the course?
1. 1-3 months 2. 4-6 months 3. 7-9 months 4. 10-12 months
5. 13-15 months 1&2. 16-18 months 1&3. >18 months

What is your age?
1. 22 2. 23 3. 24 4. 25 5. 26 1&2. 27

1&3. 28 1&4, 28 1&5. 30 2&3. 31 2&4, 32 2&5. 33
3&4, 34 3&5. 35 4&5. 36 1,2 &3. 37 = 2,3&4. 38 = 3,4&5. 39
1,3 &5. 40 = 1,2&4. 41 1,2&5. 42 2,385. 43 = -1,3&4. 44 1,4&5. 45

B-2



program. The AC3 DL program teaches captains to command companies, and also provides
training so that they can serve as assistant operations officers at battalion and brigade level. In
contrast, the AOAC RC program is limited to company command training. While the AOAC
RC program addressed only 15% of the tasks taught in the 18-week Active Component (AC)
resident course, the AC3 DL program addresses 92% of the AC resident course tasks (peace
keeping tasks were not included). The additional material covered in AC3 DL compared to
AOAC RC could provide more opportunities for practice of Company Team Operations used as
subject matter in comparing the two courses.

The AC3 DL program provides additional training capabilities compared to the AOAC
RC program. These include an enhanced instructional design based on sound cognitive |
strategies, use of audio, video, and animation to train complex cognitive skills, and delivery of
multiple levels of timely feedback - computer and instructor graded. Both the students and the
SGI stated that AC3 DL is superior to AOAC RC as it includes peer motivation and peer tutoring
as part of the learning process, and better prepares students for resident training. The SGI stated
that AC3 DL students have more confidence, more experience speaking in front of peers, and are
more likely to make a decision, and that these skills are a product of the experiential AC3 DL
training. While the DL training approach offers a number of training opportunities, it also
requires up-to-date computer hardware, software, and Internet access for training delivery. This
required instructional technology cannot always be provided the student’s RC unit, so that this
training support requirement may be passed on to the learner.

Conclusion

This report provides training developers and Army leaders with a better understanding of
the capabilities and challenges of training programs such as AC3 DL. The AC3 DL course is
capable of covering approximately 92% of the 18-week AC Resident Course which is a great
improvement compared to only about 15% coverage for the previous AOAC RC course. The
AC3 DL SGI stated that the AC3 DL course is superior to the previous AOAC RC program as it
includes peer motivation and peer tutoring capabilities as part of the learning process, and better
prepares students for resident training. The present research identified student attrition and
course design features as potential challenges for training developers to overcome. Data for both
AOAC RC and AC3 DL courses suggests that there is a high rate of students failing to complete
the courses. With regard to course design, students asked that the synchronous VTOC training
sessions be rescheduled so that each closely follows and reinforces the material presented in
asynchronous training. Students also asked for delivery of some course material on CD ROM
media to alleviate the on-line requirement. The on-line course design requirement was viewed
by students as limiting their access to training, and contributing to massive printing of course
reference material that could easily be provided on CD ROM. The present research also
included surveys and interviews with students and the course instructor which indicated that
further development of course materials was needed to reduce the length of Phase IA
asynchronous training modules to facilitate course completion. The current research effort
identified course implementation issues regarding training time, and training compensation.
While current AC student training occurs during duty hours and is compensated, the AC3 DL
training occurred during students’ personal time, and was not compensated. These issues must
be considered when planning for the transition of Army training to DL delivery.

13



13. Yrs Experience at Co

NA 2 (13.3%) 21 (14.1%)
1-2 yrs 10 (66.7%) 67 (45.0%)
3-4 yrs 2 (13.3%) 39 (26.2%)
5-6 yrs 1 (6.7%) 15 (10.1%)
>6 yrs 0 7 (4.7%)
14. Yrs Experience on Staff

NA 9 (60.0%) 58 (38.9%)
1-2 yrs 5 (33.3%) 56 (37.6%)
3-4 yrs 0 24 (16.1%)
5-6 yrs 1 (6.7%) 8 (5.4%)
>6 yrs 0 3_ (2.0%)
15. Yrs Since Last CTC

NA 10 (66.7%) 85 (57.0%)
1-2 yrs 1 (6.7%) 30 (20.1%)
3-4 yrs 0 8 (5.4%)
5-6 yrs 0 9 (6.0%)
>6 yrs 4 (26.7%) 17 (11.4%)
16. Highest Enlisted Rank

N/A 4 (26.7%) 51 (34.2%)
E3 2 (13.3%) 19 (12.8%)
E-4 5 (33.3%) 34 (22.8%)
E5 1 (6.7%) 30 (20.1%)
Beyond E-5 3 (20.0%) 15 (10.1%)
17. Other Military Courses

Air Borne/Air Assault 5 (33.3%) 52 (34.9%)
Ranger/Special Forces 4 (26.7%) 11 (7.8%)
BMOC/Other CSS 3 (20.0%) 16 (10.4%)
SPLC 5 (33.3%) 20 (13.2%)
TC3/NBC Defense/Other 1 (6.7%) 52 (34.9%)
Bad 0

18. Months since starting

AC3-DL Correspondence

Mean 13.0 29.6
Median 11.0 26.0

Std Dev 2.5 11.2

19. Months since Finish

AC3 Correspondence Phase

Mean 5.6 13.4
Median 6.0 11.0

Std Dev 3.2 6.4

20. Age

Mean 31.3 32.5
Median 31.0 32.0

Std Dev 2.7 3.1

F-3



Table 3

AC3 DL Volume IV Lesson Pre-Test and Post-Test Scores (n=15)

Lesson

1

Total

Pre/Post

Pre
Post
Pre
Post
Pre
Post
Pre
Post
Pre
Post
Pre
Post
Pre
Post
Pre
Post
Pre
Post

Mean

51.80
92.33
48.00
95.40
50.67
92.67
32.47
90.33
57.40
93.73
52.73
88.40
35.73
84.80
43.60
82.93
46.55
90.08

Median

60.00
93.00
50.00
100.00
54.00
94.00
30.00
90.00
70.00
95.00
57.00
89.00
30.00
90.00
45.00
86.00
49.00
90.00

Std Dev

17.25
9.26
18.59
8.29
13.23 |
6.77 |
17.08
9.80
24.49
8.04
16.55
5.57
17.33
9.85
14.08
7.07
18.93
0.90

Table 4 presents the Volume V Lesson Pre-test and Post-test score summary. Students
averaged a score of 40.20% correct across the nine Volume V Lesson Pre-tests. The average
score of 88.41% correct across the Volume V Lesson Post-tests exceeds the 70% or greater
criterion and thus provides evidence that students did learn the lesson material using the AC3 DL

program.

Table 4

AC3 DL Volume V Lesson Pre-Test and Post-Test Scores (n=15)

Lesson Pre/Post Mean Median Std Dev

1 Pre 34.53 36.00 13.70
Post 85.67 85.00 5.60

2 Pre 41.67 40.00 17.90
Post 86.33 85.00 4.81

(Table Continues)



CONTENTS (Continued)

Page
D. AC3 DL Resident Course Student Interview..........scsessssesseeseeeeeeeseneenees D-1
E. AC3 DL Small Group Instructor Interview........cssssssssesesseeseeeeeteessensennee E-1
F. Demographics Questionnaire SUMMaY.........sessessseeeeeneesneeseeneesenneeneenees F-1

G. Phase IA Correspondence Course Training Times ...

H. Resident Knowledge Test ScOres........ssssssssssessssssescnsesseessesneeneeeneenteenseneenees H-1

I. AC3 DL Resident Course Survey Questions: Annotated... I-1

J. AC3 DL Resident Course Interview Questions: Annotated............s:ss0 J-1

K. AC3 DL Small Group Instructor Interview: Annotated... K-1

LIST OF TABLES

Table 1. Demographics Comparison: AC3 DL vs. AOAC RC Classe .......-sssssesssesesesesseeneens 5

2. Phase IA Correspondence Phase Training Times Summary (Mean/Std. Dev.)........ 6

3. AC3 DL Volume IV Lesson Pre-Test and Post-Test Scores (n=15).. 8

4. AC3 DL Volume V Lesson Pre-Test and Post-Test Scores (N=15)........ssssesseeeesseseees 8

5. AC3 DL Volume IV and V EOV Test Scores (N=15) .....csssessssessersesetesseeseennsnteneenenees 9
6. Resident Knowledge Test Scores for AC3 DL and AOAC RC Classe........ssssseeeees 10
7. AC3 DL Volume IV and V Post-Test vs. AOAC RC End of Course Scores............. 11



AC3 DL SMALL GROUP INSTRUCTOR INTERVIEW (PART 2)

Compare AOAC RC and AC3 DL students addressing specific strengths and weaknesses of each
group.

1. Decision making - skill needed to make choices and solve problems.

2. Planning - involves forecasting, setting goals, developing strategies, establishing priorities,
among other skills whose goal is to support a course of action.

3. Communicating - the exchange of information from one person to another which is expressed
in oral, written, or graphic forms.

4. Technical and tactical proficiency - knowing the job and tactical doctrine.

5. Use of available systems - familiarity with techniques, methods, and tools that give you an
edge including computer usage but also analytic techniques.

6. Supervising - controlling, directing, evaluating, coordinating the actions of subordinates.
7. Professional ethics - loyalty to the nation, the Army, the unit; service and integrity.

8. Teaching/counseling - improving performance of subordinate by overcoming
problems, gaining new skills, modeling behaviors.

9. Soldier team development - creating strong bonds between leader and soldiers and among
soldiers.

E-2



Appendix D

AC3 DL Resident Course Student Interview

1. What factors, if any, limited your participation in the ASYNCHRONOUS self-study portion
of the AC3 DL course? (equipment, employment, Army policies)

2. What Reserve Component policies would support participation in AC3 DL self-study
ASYNCHRONOUS training?

3. How did you manage your time to be able to participate in the ASYNCHRONOUS portion of
the course?

4. What factors, if any, limited your participation in the SYNCHRONOUS collective training
on weekends portion of the AC3 DL course? (equipment, employment, Army policies)

5. What Reserve Component policies would support participation in AC3 DL collective
weekend SYNCHRONOUS training?

6. How did you manage your time to be able to participate in the SYNCHRONOUS portion of
the course?

7. Did you get any compensation for participation in the course ($, points, or time)?

8. What types of NON-MONETARY compensation would increase future participation in AC3
DL weekend SYNCHRONOUS training?

9. What types of MONETARY compensation would increase future participation in AC3 DL
weekend SYNCHRONOUS training?

10. What are some difficulties or issues with the enrollment process that you experienced?
11. Was it easy to access the website? Hardware or software problems?

12. Besides the members of the Resident Class, do you know of anyone else who took the AC3
DL course?

13. What are the specific benefits of being in the resident portion of the course?

14. What are your overall perceptions about the course (ASYNCHRONOUS,
SYNCHRONOUS, RESIDENT)?



ASSESSMENT OF INITIAL DELIVERY OF THE ARMOR CAPTAINS' CAREER COURSE
(DISTANCE LEARNING)

EXECUTIVE SUMMARY.

Research Requirement:

The U.S. Army Armor School (USAARMS) has recently introduced a revised Armor
Captains' Career Course (AC3), formerly known as the Armor Officers’ Advanced Course
(AOAC), for Reserve Component (RC) officers using The Army Training System Conversion
(TATS-C) process. In June 1999, the Director of the USAARMS requested that the U.S. Army
Research Institute for the Behavioral and Social Sciences' Armored Forces Research Unit (ARI
AFRU) at Fort Knox provide Technical Advisory Service to assist in assessment of the new AC3
Distance Learning (DL) program. The request referred specifically to an assessment of the
Internet-delivered Phase IA portion of the course. In response the ARI AFRU formed a small
team to work with USAARMS personnel on the AC3 DL assessment. The primary objective of
this assessment was to determine whether AC3 DL students learn at least as effectively as
students in the previous AOAC RC program for subject matter that is common to both course
versions. A secondary objective was to identify additional material covered as well as any
additional capabilities and benefits provided by AC3 DL as compared to AOAC RC.

Procedure:

The research procedure included the collection and analysis of available student
performance data for the AC3 DL and AOAC RC programs from archival sources, the
administration of a single test (called the Resident Knowledge Test) of company team
knowledge to both AC3 DL and AOAC RC students, and the collection of survey and interview
data from the AC3 DL students and their instructor. An emphasis was placed on using readily
available data, or data that could be obtained with a minimum of distraction to students. The 15
RC officers who completed the first iteration or class of the AC3 DL program provided the data
for the DL approach. One hundred and forty-nine officers from five AOAC RC classes were
used as the traditional paper booklet correspondence course comparison group for this research.

Findings:

Students in the AC3 DL course completed the correspondence phase of training in a
shorter period of time than did AOAC RC students (7.53 vs. 16.42 months), and had a
significantly shorter delay between the completion of correspondence training and the start of the
two-week resident phase of training (5.47 vs. 13.14 months) based on Mann-Whitney U tests of
significance. Students in the AC3 DL course had a mean score of 37.67 correct on the 62-item
Resident Knowledge Test, compared to a mean score of 34.54 for AOAC RC students. A Mann-
Whitney U test of this difference approached significance, providing some evidence that students
in the AC3 DL class did as well as, or better than, students in the AOAC RC program on
company team material covered in both courses. Survey and interview responses from AC3 DL
students and their instructor were generally positive regarding the course, and described it as an

vii



ACKNOWLEDGEMENTS

The authors wish to acknowledge the essential contributions made by personnel in the
U.S. Army Armor School (USAARMS) who participated in all phases of the planning and
execution of this research. The research design was greatly shaped by the insights and
recommendations of these personnel. The research provides empirical estimates of learning that
would not be possible without the volunteer support of students from both AOAC Reserve
Component (RC) and Armor Captains' Career Course (AC3) Distance Learning (DL) classes.
Much of the value of the present research comes from documenting the detailed comments and
recommendations provided by the AC3 DL students. The value of this search for good ideas
from the students closest to the issues cannot be overestimated. Dr. David Bessemer, U.S. Army
Research Institute for the Behavioral and Social Sciences, Armored Forces Research Unit
provided valuable guidance in designing the statistical analysis plan for the present research.
The statistical comparisons presented in the research could not have been accomplished without
the continuous participation of Mr. George Paschetto of the USAARMS in the data management
role. The present research required data collection from six separate classes. CPT Chester
Guyer not only carried out the data collection, but also shared his insights as an RC officer to
educate the research team on the world of RC training issues. The AC3 DL program represents a
new approach to officer education. A number of training techniques have been implemented into
the DL program to provide learning opportunities. CPT Dan Dwyer served as the Small Group
Instructor managing all phases of the AC3 DL program, and provided valuable insights regarding
the effectiveness of the DL training techniques, and the challenges facing the Army in
developing a new generation of DL instructors. Dr. Connie Wardell served as the senior
USAARMS representative in all phases of the present research, from research plan development,
data collection coordination, and through the final review of the research findings.

vi



3 Pre 30.60 30.00 19.88

Post 92.00 90.00 7.75
4 Pre 45.20 40.00 21.09
Post 88.07 90.00 6.73
5 Pre 38.33 40.00 16.87
Post 90.07 90.00 7.27
6 Pre 30.87 35.00 16.64
Post 83.67 90.00 8.55
7 Pre 44.47 50.00 11.93
Post 88.00 92.00 5.28
8 Pre 38.80 47.00 19.82
Post 84.60 87.00 8.68
9 Pre 57.33 60.00 34.53
Post 97.33 100.00 7.04
Total Pre 40.20 40.00 21.03
Post 88.41 90.00 0.08

AC3 DL Volume IV and V EOV Test Score Summary

The AC3 DL students complete company team instruction during Volumes IV and V of
Phase IA training, and their knowledge is routinely assessed with EOV tests. The average
Volume IV EOV Test score was 85.8% correct. The average Volume V EOV Test score was
85.0% correct. These EOV scores exceed the 70% or better criterion and provide evidence that
students did learn the lesson material using the AC3 DL program. Table 5 presents the Volume
IV and V EOV score summary. The EOV tests were repeated until the test score exceeded the
70% criterion. Only one student took the Volume V EOV test a second time to reach the 70%

criterion.
Table 5

AC3 DL Volume IV and V EOV Test Scores (n=15)

Measure Mean Median Std Dev
Volume 4 85.80 88.00 5.88
Volume 5 85.00 89.00 8.63

AC3 DL vs. AOAC RC Resident Knowledge Test Comparison

The Resident Knowledge Test is a 62-item multiple choice test of company team
operations. The Resident Knowledge Test was administered to one AC3 DL class (n=15), and
five AOAC RC classes (total n=149). Table 6 presents the Resident Knowledge Test scores for



ASSESSMENT OF INITIAL DELIVERY OF THE ARMOR CAPTAINS' CAREER COURSE
(DISTANCE LEARNING)

CONTENTS.
Page
INTRODUCTION .....ccssssssssssscsesssscsssesnessesssessnessessscsscssscssessscsnessecssessucssscssscesecanecacaneeneeaenneenees 1
Requirement 1
Objectives .. 1
Background 1
METHOD 1... cesscssscssessscssecseessccssssstssssssscssssssessueesscsnecsscsnsesscsnecsuccsecesecsuccsucescssccanecsecuceanenneeneenaes 2
Participants.
Within-Course Measures ........s.-sseseeee

Between-Course Comparative Measures.
Data Collection and Analysis Approach..

RESULTS un ecscssesssssesesseesecnececsessvssesscsnssecssssscsussasssssuessscenssnssassnseucsucnessecsecneenscaseneeneeneanecnsnesasenanee 5

Demographics Comparison for AC3 DL and AOAC RC Students.........sssssssecseereeses
AC3 DL Volume IV and V Lessons Pre-Test and Post-Test Score Summary .
AC3 DL Volume IV and V EOV Test Score Summary...........
AC3 DL vs. AOAC RC Resident Knowledge Test Comparison.
AC3 DL Volume IV and IV EOV vs. AOAC RC End of Course Scores .
AC3 DL Resident Training End of Course Written Survey .
AC3 DL Resident Training End of Course Class Interview.
AC3 DL Resident Training End of Course Instructor Interview..

SUMMARY AND DISCUSSION ....csscessssssssssessssssscsnecsecssecssecseecueesnecseecnseeneennesaneeneennenaeenneenenne 12

AC3 DL vs. AOAC RC Training Effectiveness ..
Additional AC3 DL Material, Capabilities, and Benefit:

Conclusion
REFERENCES .u...sscssesseseseessssssssstsssessscssssscsssscassencsasessecssessecsnecsscssecanecsscessscuecenecasecnecaenneeneenes 15
APPENDIX A. List of ACronyms.........scccesssessssecsessesssseseeseseesesssesssseeseesneneensenteneeneenenneneenee A-1
B. Demographics Questionnaire ...........cessecssssssesstessessneessesseeeseensessnseneeneeneenee B-1

C. AC3 DL Resident Course Survey

ix



AC

AC3
AC3 DL
ADT
AFRU
AOAC
ATRRS

BMOC
BN

cD
DL
EOV
IDT
RC

SAF
SGI

TATS-C
USAARMS

VTOC

Appendix A
List of Acronyms

Active Component

Armor Captains' Career Course

Armor Captains! Career Course (Distance Learning)

Active Duty Training

Armored Forces Research Unit

Armor Officers' Advanced Course

U.S. Army Research Institute for the Behavioral and Social Sciences
Army Training Requirements and Resources System

Battalion Maintenance Officer Course
Battalion

compact disk

Distance Learning

End of Volume
Interactive Duty Training
Reserve Component

Semi-Automated Forces
Small Group Instructor

The Army Training System Conversion
U.S. Army Armor School

virtual tactical operations center

A-l



U.S. Army Research Institute
for the Behavioral and Social Sciences

Research Report 1775

Assessment of Initial Delivery of the Armor
Captains’ Career Course (Distance Learning)

William R. Sanders and Billy L. Burnside
U.S. Army Research Institute

20010906 00)

August 2001

Approved for public release; distribution is unlimited.



¢ Synchronous. Example of good techniques is division of labor between students - put
perceived stronger students in more demanding positions (ex. S3) first, then rotate with
other students. Assessment of student strength comes through Synchronous performance,
and through dialogue between SGI and student during the course. There is a strong
relationship between demonstrated student motivation in Phase I and motivation
displayed in VTOC.

e Resident. The SGI wants to integrate more of the Resident class with other simulation
stuff on post - but only if it maintains the leadership training focus, don't put student in
the role of tank loader.

4. How well did TACOPS work for orders execution? Any improvements needed?

SGI Response:

e The TACOPS simulation met the SGI's needs, he is satisfied with it. The SGI wants to
get it into the hands of students earlier. All students got a CD copy of the game to take
with them, to share with units and peers.

e Inthe future SGI wants to maximize the use of Semi-Automated Forces (SAF) suite.

5. How many distributed classes of students can one instructor handle? What class size?

SGI Response:
e Class size that can be handled is Phase I (1 - 60) SGI to student ratio, VTOC (1-12/15)

and two VTOC classes at the same time, equal to two weekends each month. Resident
course (1-12/15), and Fort Knox has resources for two classes simultaneously. SGI relies
on AC3 DL Instructor Toolset to track progress of students each day, first thing in the
morning. Identify if students are going into Gate tests that should be scored very quickly
to maintain motivation.

¢ It is better to start Synchronous sessions early on after students complete Volume III
materials which covers MDMP, Bd Offense, other. It is best to practice the Phase I
content right after students see it in the courseware.

6. How many hours per week does it take to handle 15 students in training? In Synchronous
VTOC training? In Resident training?

SGI Response:

e Asynchronous. Management of Asynchronous is not a big time requirement, 80% of
student work is automatically scored.

e Synchronous VTOC. One person could handle 60 Asynchronous and 2 VTOC groups at
the same time. Goal is to train 150 students per year. Motivated students could get
through training in 12-14 months. Next resident courses begin in June (2 classes), will
also occur in July. Anticipate five resident classes, 1-15 SGI to student ratio, could be
two classes at the same time.

e Resident Course. The AOAC RC was more PowerPoint lecture format, where SGI
would tell the student what they should do, show them what to do, and then "now you do
it." In AC3 DL the students know each other already, and the SGI knows the students.
AC3 DL Resident class is taught using a different technique, here the students are quickly

K-2



The AC3 DL design document (AB Technologies, 1998) provides the strategy underlying
development of this training. As part of the course strategy, it is noted that the use of Internet
and computer-based instruction in Phase IA training does not eliminate the requirement for the
instructor. On the contrary, this training strategy calls for increased instructor involvement
through various technology tools, and a resulting requirement for additional instructor training
on the use of administrative and tracking tools. The design of the AC3 DL course reflects the
theoretical perspective that learning can best be achieved through a hierarchy of progressive
levels of knowledge acquisition and application (AB Technologies, 1998). The course design
incorporates strategies, and corresponding templates, for each type of knowledge being trained
(facts, concepts, procedures and principles). The DL training strategy requires up-to-date
computer hardware, software, and Internet access for training delivery. This required
instructional technology cannot always be provided at the student’s RC unit, so that this training
support requirement may be passed on to the learner.

The AC3 DL training appears to meet key factors identified by Alden (1998) for
transitioning to Internet and computer-based instruction. Internet-delivered instruction makes
sense given the high costs associated with bringing students to the traditional resident training
course. Particularly for the RC, resident training longer than two weeks is often not an option.
Access to the Internet is generally available, and with skillful course design traditional classroom
content and activities can be effectively presented at a distance under a remote instructor’s
guidance. One key factor identified by Alden (1998) is that success in Internet and computer-
based instruction requires that participating students really want to complete the training. This
requirement again points out the crucial role that instructors play in assessing student needs, and
providing the guidance necessary to motivate the student as an individual, and as a member of
his class peer group. Contrary to common opinion, Draves (2000) points out that Internet-based
instruction can actually be more personal and more interactive than traditional classroom
courses, because the instructor does not have to personally deliver the information content of the
course (lectures, graphics, text, video). The instructor is thus free to focus on assessing student
performance and guiding individuals and groups of students through the materials.

The AC3 DL program represents a prototype for future Army training. Both the AC3 DL
and AOAC RC versions of the course were delivered concurrently for approximately one year,
providing the opportunity for comparison. The initial or pilot delivery of the AC3 DL program
was completed in November 2000.

Method

The assessment methods included the collection and analysis of available student
performance data for the AC3 DL and AOAC RC programs from archival sources, the
administration of a single test of company team knowledge to both AC3 DL and AOAC RC
students, and collection of survey and interview data from the AC3 DL students and their
instructor. An emphasis was placed on using available data, and data that could be obtained with
a minimum of distraction to students. The research measures are described below, followed by a
discussion of their collection and analysis. Definitions for the acronyms used in this report are

presented in Appendix A.



ASSESSMENT OF INITIAL DELIVERY OF THE ARMOR CAPTAINS' CAREER COURSE
(DISTANCE LEARNING)

Introduction

Requirement

The U.S. Army Armor School (USAARMS) has redesigned the Armor Captains' Career
Course (AC3), formerly known as the Armor Officers' Advanced Course (AOAC), for Reserve
Component (RC) officers using The Army Training System Conversion (TATS-C) process (AB
Technologies, 1998). In June 1999, the Director of the USAARMS requested that the U.S. Army
Research Institute's Armored Forces Research Unit (ARI AFRU) at Fort Knox provide Technical
Advisory Service to assist in assessment of the new AC3 Distance Learning (DL) program. The
request referred specifically to an assessment of the Internet-delivered initial portion of the
course (Phase IA). In response the ARI AFRU formed a small team to work with USAARMS
personnel on the AC3 DL assessment. This report provides an initial assessment of the AC3 DL

program.
Objectives
The primary objectives of this research were to:

e Determine whether AC3 DL students learn through successful completion of Phase
JA requirements.

e Determine whether AC3 DL students learn at least as effectively as students in
traditional AOAC RC for subject material that is common to both course versions.

e Identify additional material covered as well as any additional capabilities and
benefits provided by AC3 DL as compared to AOAC RC.

Background

The previous AOAC RC program consisted of 24 paper-based sub-courses with multiple-
choice knowledge tests taken by correspondence, followed by a two-week resident Active Duty
Training (ADT) period at Fort Knox. The focus of the course was strictly on company team
operations. The redesign of AOAC RC has resulted in an expanded course delivered largely
through DL methods. The redesigned AC3 DL consists of three major phases: Phase IA
providing 240 hours of asynchronous (with an instructor "in the loop") Internet lessons delivered
over a one year period; Phase IB providing 120 hours of synchronous lessons using a virtual
tactical operations center (VTOC), and a small group instructor during seven consecutive
Interactive Duty Training (IDT) weekends; and Phase II providing 120 hours of resident training
during an ADT period at Fort Knox. The expanded AC3 DL is modeled after the Active
Component AC3 and trains officers to command companies and to perform as assistant
operations officers at battalion and brigade levels.



3. How did you manage your time to be able to participate in the ASYNCHRONOUS portion of
the course?

Student Comments:

Real juggling act - 10 pm to 2 am, or right after work.

BN OPORD took 30 hours to write, easy to procrastinate.

e The majority of students printed out a lot of the course materials.

¢ Map Editor updates needed to be provided - all software needs regular updates.

4. What factors, if any, limited your participation in the SYNCHRONOUS collective training
on weekends portion of the AC3 DL course? (equipment, employment, Army policies)

Student Comments:

e People in command positions can't miss a drill.

e Problem: when writing an OPORD you would normally have a battle staff to support
you. Provide a Battle Book - Battle Book helped tremendously - Map Edit only allows 6"
view of map, and it crashed a lot.

e Progression - should we start w/TF, CO, BN? Start big and work down, or start at the
bottom and work up? Trying to write CO/TM orders - student says they are "swinging in
the dark."

© VTOC is good - everybody worked from home. It would be much better if students
worked in teams because it is too easy to "hide" at home.

e So many students dropped that you lost key positions.

Command and control through a port hole.

5. What Reserve Component policies would support participation in AC3 DL collective
weekend SYNCHRONOUS training?

Student Comments:
e Can't have commanders’ company/team doing Synchronous training instead of drill.

e Ifyou could get small groups together for VTOC - real world together - then do it.
e Ifyou have the chance to work face to face do it. Like S3 and S2 tasks, will need to get
chain of command to support it.

6. How did you manage your time to be able to participate in the SYNCHRONOUS portion of
the course?

Student Comments:

e There is not an understanding of the time requirements associated with the course. The
chain of command does not realize the time demands.

e Problem if you don't have weekends off, or rotate days off

e Need to educate leaders of time demands.

e Student want a document that indicates the time requirements for each lesson - course
outline - would help with time management - don't know size of lesson until you open it.



Appendix E

AC3 DL Small Group Instructor Interview

AC3 DL SMALL GROUP INSTRUCTOR INTERVIEW (PART 1)
1. How many AOAC RC classes have you instructed? Approximately when?

2. How did this class compare to other RC classes you have instructed. Specific
differences, if any? Specific areas:

- Orders Preparation

- Orders Briefing

- Doctrinal Knowledge (manuals)
- Tactical Knowledge (application)

3. What techniques did you develop and use that are unique to asynchronous delivery?
Synchronous delivery? Resident delivery?

4. How well did TACOPS work for orders execution? Any improvements needed?
5. How many distributed classes of students can one instructor handle? What class size?

6. How many hours per week does it take to handle 15 students in asynchronous
training? In synchronous VTOC training? In Resident training?

7. What improvements, if any, are needed in the course?

E-1



immersed in the subject matter "Here it is, do it" (OPORD provided to students the first
night?). The AOAC RC students did not know each other until the end of the Resident
training, and only focused on Company level subject matter. Now, the students know
each other at the start.

7. What improvements, if any, are needed in the course?

SGI Response:

Asynchronous. Changes to the courseware are needed, and the students have reported
many of these issues. Chunking of information will take place, the original lessons were
too long. The student would log-in, see the length of the lesson, and log-off. Material is
being broken up into bite-sized chunks. Before doctrinal changes are introduced the SGI
wants to see them introduced in the Resident AC course.

Synchronous. The VTOC training, the Map Edit software needs to be changed or fixed,
it might be 10 year old technology, so that students will not continue to be frustrated with
map graphic tasks. The SGI wants the map editing and text tools to be available in the
VTOC environment, currently students have to exit VTOC to access the map and text
tools.

Resident. Use of more simulations is desired, SGI is happy with the current class format,
might want to bring in non-armor speakers.

K-3



15. How well did the TACOPS simulation support learning in the Resident portion of the
course?

a. Would it have helped you if you had a copy of the TACOPS simulation to use
before you got to Ft. Knox?

b. Would it be useful to you to take a copy of TACOPS with you?

16. What other factors do you think impact either positively or negatively on the course?

D-2



improvement over paper-based correspondence courses. Concerns were identified in specific
areas, such as the length of specific portions of AC3 DL. These findings can be used in

restructuring specific sections of the training.

Utilization of Findings:

The present research provides an initial assessment of the AC3 DL training program.
Results of the assessment provide evidence that the course is at least equally effective in
presenting material previously taught in the AOAC RC program. The research also identified
additional material that the AC3 DL program covers, which was not included in the AOAC RC
program. This report provides training developers and Army leaders with a better understanding
of the capabilities and challenges of training programs such as AC3 DL. It also provides course
design, development, and implementation insights which may be generalizable to a broad range
of Internet-delivered DL programs. The findings suggest policy decisions that would promote

training program success.

viii



student competencies with regard to nine elements of leadership. The interview guides annotated
with SGI comments are provided as Appendix K. In general, the SGI viewed AC3 DL students
as more skilled than AOAC RC students with regard to Orders Preparation, Orders Briefing,
Doctrinal Knowledge (manuals), and Tactical Knowledge (application). One advantage of DL
instruction was that in Phase IA the SGI was able to appoint a Student Leader who could
communicate with other students via email. The Student Leader is a person recognized as
moving through the material quickly and successfully, who serves an important role as a
motivator for the other students, identifying tactics, techniques, and procedures for getting
through the courseware. Students were not linked via email for the AOAC RC correspondence
phase of training, and Student Leaders were not appointed. The SGI stated that AC3 DL is
superior to AOAC RC as it includes peer motivation and peer tutoring as part of the learning
process, and better prepares students for resident training.

Summary and Discussion

The goal of the present research was to assess the Phase IA asynchronous correspondence
course portion of the AC3 DL program. The findings provide evidence that AC3 DL students
learn at least as effectively as students in the traditional AOAC RC course for subject material
that is common to both course versions. The research also identified additional material covered,
training capabilities, and training benefits provided by AC3 DL beyond those offered by the
traditional AOAC RC program. Several factors potentially impacting student performance could
not be controlled. High rates of student attrition, additional practice provided by AC3 DL Phase
IB synchronous training, and differences in the length of delay between correspondence course
training and resident training for AC3 DL and AOAC RC students could all contribute to
differences in student performance. Data should be collected from future AC3 DL classes to
increase the reliability of performance estimates, and to identify how changes introduced into the
AC3 DL course impact student performance and perceptions.

AC3 DL vs. AOAC RC Training Effectiveness

The comparison of AC3 DL and AOAC RC student test scores provides evidence that the
AC3 DL students were at least as well prepared as the AOAC RC students. The AC3 DL
students’ Volume IV and V Lesson Post-test scores, and EOV Test scores exceeded the
performance standard of 70% or better, and thus provide evidence that students did learn the
lesson material during the AC3 DL training. The assessment of AC3 DL and AOAC RC
students’ knowledge of company team operations upon their arrival at resident training revealed
a small advantage for the AC3 DL program. The mean Resident Knowledge Test raw score for
AC3 DL students on the 62-item test was 37.67, compared to a mean raw score of 34.54 for
AOAC RC students. The comparison of AC3 DL Volume IV and V EOV test scores vs. AOAC
RC End of Course scores revealed that average score for students in both courses exceeded the
70% or better performance standard.

Additional AC3 DL Material, Capabilities, and Benefits

The present research identified additional material covered, training capabilities, and
training benefits provided by AC3 DL beyond those offered by the traditional AOAC RC

12



Appendix I
AC3 DL Resident Course Survey Questions: Annotated
AC3 DL RESIDENT COURSE SURVEY QUESTIONS: Annotated

Instructions to AC3 DL participants

The Armor School, with assistance from the Army Research Institute, is gathering information to
examine the quality of Army training programs. We need your help to identify how well
elements of the program meet training needs. Please read each question carefully and place a
check mark next to your best answer.

By answering the questions to the best of your ability, you are helping the Armor School to
improve the way AC3 DL training is delivered.

1. Do you believe web-based training is effective for AC3 DL content?
13/87% YES

0/0%___ NO

2/13% NOT SURE

Student Comments:

e Yes. It has its limits. There must be professional development along side of it. Senior
leaders to double-check your progress.

e Yes. CD and then test points would be better.

e Not Sure. Why not issue a CD with course material and do exercises, exams, tests on the
web. It would help the student manage his time more efficiently. Being a Guardsman is
all about time management and balance.

2. How did you link to the AC3 DL website for the distributed portion of the course?
13/86% Personal Computer

1/7% Army Computer

1/7% Other: Explain

Student Comments:
e Other. Division laptop.
e Personal Computer. State Firewall prevented use of Army computer.

3. Do you belong to an active reserve unit (drill 1 weekend a month)?

15/100%__ YES
0/0% NO _ If your answer is NO then skip to Question #7

Student Comments:

e Yes, at least two weekends a month.
e Yes. And I attempted to attend every drill in addition to weekend VTOC exercises.

1-1



4. Does your unit have Internet-linked computers available to support Distance Learning?
9/60% YES

5/33% NO
1/7% NOT SURE

Student Comments:

¢ Yes. But it is Windows NT and must be scheduled for in advance.

e Yes, however they are few and generally not available for use by troops for purposes
other than admin.

¢ Yes, State Firewall prevented use. DL class not functional yet, no courseware or
instructor.

5. Did AC3 DL weekend ASYNCHRONOUS training time detract from your other unit
responsibilities?

4/27% YES, A LOT

6/40% =~ YES, SOME.

5/33% NO

Student Comments:

e Yes, Some. I was in a good situation to do it, pertaining to my civilian job. I didn't have
to work weekends.

e Yes, a lot - 4,865 web pages

6. Did AC3 DL weekend SYNCHRONOUS training time detract from your other unit
responsibilities?

5/33% YES, A LOT

9/60% YES, SOME

1/7% NO

Student Comments:
e Yes, Some. I was in a good situation to do it, pertaining to my civilian job. I didn't have

to work weekends.
e Yes, alot - XO and CDR of same unit on drill weekends a problem.

7. Do Reserve Component Officers have the computer skills necessary to work through AC3 DL
training?

15/100% _ YES

0/0% NO

0/0% NOT SURE

Student Comments:
e Yes. Very simple skills are only needed.
e Yes. Some, most could do it. Many don't have good enough systems though .

e Yes. They should!

1-2



Appendix C

AC3 DL Resident Course Survey

Instructions to AC3 DL participants

The Armor School, with assistance from the Army Research Institute, is gathering information
to examine the quality of Army training programs. We need your help to identify how well
elements of the program meet training needs. Please read each question carefully and place a
check mark next to your best answer.

By answering the questions to the best of your ability, you are helping the Armor School to
improve the way AC3 DL training is delivered.

1. Do you believe web-based training is effective for AC3 DL content?
YES

NO

NOT SURE

2. How did you link to the AC3 DL website for the distributed portion of the course?
Personal Computer

Army Computer"

Other: Explain

3. Do you belong to an active reserve unit (drill 1 weekend a month)?

YES
NO If your answer is NO then skip to Question #7

4. Does your unit have Internet-linked computers available to support Distance Learning?
YES

NO

NOT SURE

5. Did AC3 DL weekend ASYNCHRONOUS training time detract from your other unit
responsibilities?

YES, A LOT

YES, SOME

NO

6. Did AC3 DL weekend SYNCHRONOUS training time detract from your other unit
responsibilities?

YES, A LOT

YES, SOME

NO

C-1



Research Report 1775

Assessment of Initial Delivery of the Armor
Captains’ Career Course (Distance Learning)

William R. Sanders and Billy L. Burnside
U.S. Army Research Institute

Armored Forces Research Unit
Barbara A. Black, Chief

U.S. Army Research Institute for the Behavioral and Social Sciences
5001 Eisenhower Avenue, Alexandria, Virginia 22333-5600

August 2001
EEE — ——
Army Project Number Personnel Performance
20363007A792 and Training

Approved for public release; distribution is unlimited.

iii



Appendix K

AC3 DL Small Group Instructor Interview: Annotated

AC3 DL SMALL GROUP INSTRUCTOR INTERVIEW (PART 1)

1. How many AOAC RC classes have you instructed? Approximately when?

SGI Response:

One RC resident class summer 1999. Several Resident AC courses.

2. How did this class compare to other RC classes you have instructed. Specific
differences, if any? Specific areas:

- Orders Preparation

- Orders Briefing

- Doctrinal Knowledge (manuals)
- Tactical Knowledge (application)

SGI Response:

General. We are comparing apples and oranges, AOAC RC course only had 15% tasks
overlap with Resident AC course. The AC3 DL has 100% overlap. The AOAC RC
teaches Co CDR skills only, not staff skills. Different skill set compared to AC3 DL.
The AC3 DL student is more knowledgeable - training provides more learning
experiences.

Orders Preparation. The AC3 DL guys are better due to repetition, they have done this in
Asynchronous and in VTOC.

Briefings. AC3 DL probably better - because they had briefed in VTOC - this provided
experiential learning.

Doctrinal Knowledge (manuals). The AC3 DL much better - because of the
Asynchronous course.

Tactical Knowledge (application). The AC3 DL and AOAC RC classes very similar.
AC3 DL did not have time to practice tactical operations prior to Resident training. At
the conclusion of Resident training AC3 DL students are better than AOAC RC.

3. What techniques did you develop and use that are unique to Asynchronous delivery?
Synchronous delivery? Resident delivery?

SGI Response:

Asynchronous. The more feedback the SGI provides during Asynchronous (like Gates
testing) the greater the motivation for the student, there is value in providing detailed
feedback. The SGI appoints Student Leadership during Asynchronous, a person
recognized as moving through material quickly and successfully. This can be a powerful
motivator, keeps some track of other students. Identify this Student Leadership to other
students’, he identifies TTPs for getting through courseware. It is important to establish
peer motivation in the Asynchronous portion of the AC3 DL course, prior to entering
Synchronous portion.

K-1



7. Did you get any compensation for participation in the course ($, points, or time)?

Student Comments:
e Three students said they got drill credit for time spent in Synchronous training.

¢ One student said he got compensation for training.

8. What types of NON-MONETARY compensation would increase future participation in AC3
DL weekend SYNCHRONOUS training?

Student Comments:
e Three students indicated they would want to do this in lieu of drill.
e Originate a Performance Certificate - proof you were there.

9. What types of MONETARY compensation would increase future participation in AC3 DL
weekend SYNCHRONOUS training?

Student Comments:
e Six students said they would like to get paid for both DL and Drill = 2 weekends each

month.

10. What are some difficulties or issues with the enrollment process that you experienced?

Student Comments:
¢ Noreal problems getting in.

11. Was it easy to access the website?, Hardware or software problems?

Student Comments:

© Biggest problem is being online - during the day NPRNET is really slow. I live at home -
wait till late to start, ex. would be great when waiting for flight - would like files in PPT.

e Kill the Shockwave feature - it covers over stuff we would want to print. Shockwave -
bad when you had to "hover" the mouse over a selection. If you print you might only get
six questions.

12. Besides the members of the Resident Class, do you know of anyone else who took the AC3
DL course? Do you know if they had any problems completing the course?

Student Comments:

e Example of the size of attrition, student is one of five to finish, students didn't realize
how much time it would take, had to be through Volume 7 to go to Phase 2. Lost a lot of
students in volume 2 and 3, the huge size of the volumes. This course could be a big
problem, it could cause people to retire. If you want to progress you have to sacrifice.

e DLis definitely a bigger commitment. Being able to do the course from a CD would
make it a lot easier.

e A lot of people are running away to the Infantry course.

e The Armor School representative mentioned that there are opportunities to get a Masters
degree over the web. Some schools will give six credit hours for completion of AC3 DL.

J-3



led the analysis of the data. The USAARMS personnel provided the results of company team
tests (End of Course test for AOAC RC students and EOV IV and V tests for AC3 DL students)
to ARI personnel for analysis. Personnel from USAARMS and ARI jointly developed,
administered, and analyzed the AC3 DL student survey, along with the structured interview
questions for both the AC3 DL students and SGI. The USAARMS and ARI personnel also
observed resident portions of AC3 DL.

Results
Demographics Comparison for AC3 DL and AOAC RC Students

Both AC3 DL and AOAC RC class students completed a 20-item Demographics
Questionnaire at the beginning of resident training. Data from the questionnaire can be used to
compare the characteristics of students in the two different courses. Where demographic
differences exist between student groups, differences in test performance might be attributable to
different student characteristics as well as the contribution of the different course formats. Table
1 presents a summary of key AC3 DL and AOAC class demographics for comparison. The data
suggest that the students in the two classes did not differ significantly in their characteristics.
The full table summarizing Demographics Questionnaire data from the AC3 DL and AOAC RC

classes is presented as Appendix F.

Table 1

Demographics Comparison: AC3 DL vs. AOAC RC Classes

AC3DL(n=15) AOAC RC (n=149)

Current Rank
2" Lieutenant ) 1 (0.67%)
1* Lieutenant 10 (66.7%) 60 (40.27%)
Captain 5 (33.3%) 87 (58.39%)
Major 0 1 (0.67%)
Branch
Armor 13 (86.7%) 108 (72.48%)
Infantry 1 (6.7%) 27 (18.12%)
Field Artillery 0 5 (3.36%)
Engineer 0 1 (0.67%)
Other 1 (6.7%) 8 (5.37%)
Age
Mean 31.33 32.15
Median 31.00 32.00
Std Dev 2.47 3.41



the single AC3 DL class. A review of the AOAC RC data suggest that Months Since Finish, and
Total Months in Correspondence Training are not normally distributed, so that parametric
statistics are inappropriate to test for equality of variance. The Kruskal-Wallis H test uses a non-
parametric rank ordering method to compare for equality of variances across multiple
independent sample scores, and was used for this evaluation. The Kruskal-Wallis H test for
equality of variance failed to find a significant difference across the AOAC RC classes for
Months Since Training (Chi-Square = 3.05, df = 4) and for Total Months in Correspondence
Training (Chi-Square = 8.153, df = 4) so that the AOAC RC times can be pooled for comparison

to AC3 DL training times.

A distribution-free (nonparametric) test was used to compare Months Since Finish, and
Total Months in Correspondence Training between the AC3 DL and AOAC RC programs. The
results of a Mann-Whitney U rank-order test conducted on the data revealed that the AOAC RC
students had a significantly longer delay (13.14 months) than AC3 DL students (5.47 months)
after finishing Correspondence Training and beginning the two-week Resident Training Phase of
instruction (U = 350.00, n= 15/144, p= .000). It must be noted that AC3 DL students also
completed Phase IB VTOC training after Phase IA correspondence training, and prior to
beginning the two-week resident training phase of instruction, which should provide additional
training on the company team material covered during correspondence training. AOAC RC
students reported spending significantly more months in the correspondence phase of training
(16.42 months) than did the AC3 DL students (7.53 months) (U = 354.50, n = 15/144, p = .000).

AC3 DL Volume IV and V Lessons Pre-Test and Post-Test Score Summary

The AC3 DL students complete company team instruction during Volumes IV and V of
Phase IA training. Volume IV contains eight lessons, and Volume V contains nine lessons.
Knowledge of the course material is routinely assessed prior to instruction with a lesson pre-test,
and after the student has worked through the lesson materials with a post-test. The pre-test is
only taken once. The post-test can be repeated until the score exceeds the 70% or better
criterion. Table 3 presents the Volume IV Lesson Pre-test and Post-test score summary.
Students who score lower than 70% on a lesson or volume post-test are required to retake the test
until they can meet this criterion. The AC3 DL student management data base records only the
most recent post-test score, and does not record an accurate count of the number of times
students repeated the post-test to meet the 70% or better passing criterion. Apparent high levels
of learning with AC3 DL might be due in part to the restricted range of passing scores recorded
(must be 70% or greater), and might also be due to repeated practice in taking the post-test.
Students averaged a score of 46.55% correct across the eight Volume IV Lesson Pre-tests. The
average score of 90.08% correct across the Volume IV Lesson Post-tests exceeds the 70%
criteria and thus provides evidence that students did learn the lesson material using the AC3 DL

program.



REPORT DOCUMENTATION PAGE

2, REPORT TYPE
Final

4. REPORT DATE (dd-mm-yy)
August 2001

3. DATES COVERED (from. . . to)
June 1999 — March 2001

4. TITLE AND SUBTITLE

Assessment of Initial Delivery of the Armor Captains’ Career Course
(Distance Learning)

5a. CONTRACT OR GRANT NUMBER

5b. PROGRAM ELEMENT NUMBER

0603007A.
6. AUTHOR(S) 5c. PROJECT NUMBER
William R. Sanders and Billy L. Burnside A792
(U.S. Army Research Institute) 5d, TASK NUMBER
205

5e. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

U.S. Army Research Institute for the Behavioral and Social Sciences
ATTN: TAPC-ARI-IK

2423 Morande Street

Fort Knox, KY 40121

8. PERFORMING ORGANIZATION REPORT
NUMBER

9, SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)
US. Army Research Institute for the Behavioral and Social Sciences

10, MONITOR ACRONYM

ATIN: TAPC-ARI-IK ARI
5001 Eisenhower Avenue 141. MONITOR REPORT NUMBER
‘Alexandria, VA 22333-5600 Research Report 1775

42. DISTRIBUTION/AVAILABILITY STATEMENT
Approved for public release; distribution is unlimited.

13, SUPPLEMENTARY NOTES

14. ABSTRACT (Maximum 200 words):

The U.S. Army Research Institute for the Behavioral and Social Sciences (ARI) Armored Forces Research Unit (AFRU) at Fort
Knox supports the U.S. Army Armor School (USAARMS) by conducting training research and development. The USAARMS
has recently introduced a revised Armor Captains’ Career Course (AC3), formerly known as the Armor Officers’ Advanced
Course (AOAC), for Reserve Component (RC) officers, delivered primarily through Distance Learning (DL). In June 1999, the
Director of the USAARMS requested that the ARI AFRU at Fort Knox provide Technical Advisory Service to assist in
assessment of the new AC3 DL program. The request referred specifically to an assessment of the Internet-delivered Phase IA
portion of the course. Results of the assessment provide evidence that the course is at least equally effective in presenting
material previously taught in the AOAC RC program. The research also identified additional material that the AC3 DL
program covers, which was not included in the AOAC RC program. This report provides training developers and Army leaders
with a better understanding of the capabilities and challenges of training programs such as AC3 DL. It also provides course
design, development, and implementation insights which may be generalizable to a broad range of Internet-delivered DL
programs.

15, SUBJECT TERMS
Distance Learning Armor Captains’ Career Course Internet Instruction Reserve Component Training

19, LIMITATION OF 20. NUMBER 21. RESPONSIBLE PERSON

ABSTRACT

16. REPORT 17. ABSTRACT | 18, THIS PAGE OF PAGES gorvay Le Sephone number)
Unclassified Unclassified Unclassified Unlimited 51 DSN 464-3450



Resident Knowledge Test Scores

Appendix H

Table 1
AC3 DL and AOAC RC Resident Knowledge Test Scores (Raw Scores)

AC3DL_ | AOAC AOAC AOAC AOAC AOAC

01 02 03 04 05
N 15 32 25 30 28 34
Mean 37.67 36.13 33.68 34.47 33.86 34,32
Median 37.00 37.00 33.00 36.00 34.50 35.00
Std Dev 5.04 5.89 4.27 6.94 5.54 6.31
Table 2
Mean Scores for AC3 DL and AOAC RC Resident Knowledge Test (Raw Scores)
AC3 DL AOAC
N 15 149
Mean 37.67 34.55
Median 37.00 35.00
Std Dev 5.04 5.90
Table 3
AC3 DL and AOAC RC Resident Knowledge Test Scores (Percentage Scores)
AC3 DL | AOAC | AOAC AOAC AOAC AOAC AOAC
Total__| 01 02 03 04 05
N 15 149 32 25 30 28 34
Mean 60.76 55.72 58.27 54.32 55.60 54.61 55.35
Median | 59.68 56.45 | 59.68 53.23 58.06 55.65 56.45
Std Dev 8.13 9.53 9.50 6.89 11,19 8.93 10.18
Table 4
Mean Scores for AC3 DL and AOAC RC Resident Knowledge Test (Percentages)
AC3 DL AOAC

N 15 149
Mean 60.76 55.72
Median 59.68 56.45
Std Dev 8.13 9.53

H-1



Appendix B

Demographics Questionnaire

Instructions: Use the AIMS sheet to answer this questionnaire. Fill in the oval completely. If
there is a question which has no appropriate response to your situation, ask for guidance.

There are several questions which have more than (5) five possible responses. In order to answer
you will have to blacken the oval numbers indicated. (i.e. For question number 2, if you have 6
years time in grade you would blacken both oval number 1 and oval number 2.)

1. Current Rank: 1, 2LT 2. 1LT 3. CPT 4. MAJ

2. Time In Grade:
1. <1 yr. 2. 1-2 yrs 3. 2-3 yr. 4. 3-4yrs. 5. 4-5 yrs. 1 &2.>5 yrs.

3. Commission Source:
1.StateOCS 2. FederalOCS 3.ROTC 4. Service Academy _ 5. Direct

4. Basic Branch:
1.AR 2.IN 3.FA 4.EN 5.AV_ 1&2. Other

5. Year Graduated from OBC:
1. 1999 2. 1998 3.1997 4, 1996 5. 1995 1&2. 1994

1&3.1993 184.1992 1&5.1991 —- 2&3.1990 2&4.1989 2&5. 1988
3&4.1987 3&5.1986 4&5. 1985

6. Component:
1.NG_ 2. Reserve 3. IRR/ING

7. Status:
1. M-Day 2. AGR 3. Technician

8. Current Duty Position:
1. Armor Platoon Leader 2. Specialty Platoon Leader 3. Other Combat Arms PL

4. Line Company XO 5. HHC XO 1&2. Company Commander
1&3. S1/G1 1&4. $2/G2 1&5. $3/G3 2&3. S4/G4
2&4. Special Staff 2&5. Secondary Staff

9. Years in Position:
1. <1 yr. 2. 1-2 yrs 3. 2-3 yr. 4. 3-4yrs. 5. 4-5 yrs. 1 &2.>5 yrs.

10. Years of Regular Army Experience:
1. <1 yr. 2. 1-2yrs. 3. 2-3 yrs. 4. 3-4 yrs. 5. 4-5 yrs. 1&2.>5 yrs.

B-1



U.S. Army Research Institute
for the Behavioral and Social Sciences

A Directorate of the U.S. Total Army Personnel Command

EDGAR M. JOHNSON
Director

‘Technical Teview by

Connie Wardell, US Army Armor School
Scott B. Shadrick, US Army Research Institute

NOTICES

DISTRIBUTION: Primary distribution of this Research Report has been made by ARI.
Please address correspondence concerning distribution of reports to: U.S. Army
Research Institute for the Behavioral and Social Sciences, Attn: TAPC-ARI-PO, 5001
Eisenhower Ave., Alexandria, VA 22333-5600.

FINAL DISPOSITION: This Research Report may be destroyed when it is no longer
needed. Please do not return it to the U.S. Army Research Institute for the Behavioral
and Social Sciences.

NOTE: The findings in this Research Report are not to be construed as an official
Department of the Army position, unless so designated by other authorized documents.



AC3 DL SMALL GROUP INSTRUCTOR INTERVIEW (PART 2)

Compare AOAC RC and AC3 DL students addressing specific strengths and weaknesses of each
group.

1. Decision making - skill needed to make choices and solve problems.

SGI Response:
e The AC3 students are more likely to make a decision, a product of the VTOC experiential

training and the Asynchronous tools like "Firefight."

2. Planning - involves forecasting, setting goals, developing strategies, establishing priorities,
among other skills whose goal is to support a course of action.

SGI Response:
© Hard to compare. The AOAC RC never planned anything till they arrived at Resident
training, then it was just company level of responsibility. The AC3 DL has had lots more

planning through VTOC.

3. Communicating - the exchange of information from one person to another which is expressed
in oral, written, or graphic forms.

SGI Response:

e The AC3 DL students have more confidence, experience speaking in front of peers, they
know the SGI better.

e Written projects are part of a group effort, individual written products are not evaluated.
SGI wants to see how students communicate the Order during two test points. The SGI
takes on the role of Platoon Leader and tells students to communicate the essential
information he needs. The SGI looks for clarity of presentation and level of
understanding.

e Inthe use of Graphics the AC3 DL students are better. They have experience in
Asynchronous and Synchronous using Map Edit, and the Asynchronous course includes
FM 105-1 map symbols, an improvement over AOAC RC, so students get the required
knowledge and there is less time devoted to teaching and correcting use of map symbols.

4. Technical and tactical proficiency - knowing the job and tactical doctrine.

SGI Response:

e The AC3 DL has to be better because Asynchronous is better than previous AOAC RC
paper-based course. The AC3 DL is more current and provides better delivery of
training.

e Inthe AOAC RC paper-based course students would find the questions at the back of
each book and then search the book to find the matching answer. It was a "no brainer."

K-4



Demographics Comparison: AC3 DL vs. AOAC RC Classes

Appendix F

Demographics Questionnaire Summary

AC3 DL (n= 15)

AOAC RC (n= 149)

1. Current Rank

2LT 0 1 (0.7%)
1LT 10 (66.7%) 60 (40.3%)
CPT 5 (33.3%) 87 (58.4%)
MAJ 0 1_(0.7%)
2. Time in Grade

<1 yr. 4 (26.7%) 19 (12.8%)
1-2 yrs 2 (13.3%) 29 (19.5%)
2-3 yrs 2 (13.3%) 33 (22.1%)
3-4 yrs 2 (13.3%) 30 (20.1%)
4-5 yrs 3 (20.0%) 25 (16.8%)
>5 yrs 2 (13.3%) 13. (8.7%)
3. Commission Source

State OCS 10 (66.7%) 51 (34.2%)
Fed OCS 0 4 (2.7%)
ROTC 5 (33.3%) 86 (57.7%)
Academy 0 7 (4.7%)
Direct 0 1 (0.7%)
4. Branch

AR 13 (86.7%) 108 (72.5%)
IN 1 (6.7%) 27 (18.1%)
FA 0 5 (3.4%)
EN 0 1 (0.7%)
Other 1 (6.7%) 8 (6.4%)
5. Year Officer Basic

Course

Mean 1994 1991.5
Median 1994 1992

Std Dev 2.91 8.24
Min/Max 1998/1987 1998/1987
6. Component

NG 14 (93.3%) 138 (92.6%)
Reserve 1 (6.7%) 10 (6.7%)
IRR 0 1_(0.7%)
7. Status

MDay 12 (80.0%) 141 (94.6%)
AGR 2 (13.3%) 4 (2.7%)
Technician 1_ (6.7%) 4 (2.7%)



Appendix J

AC3 DL Resident Course Interview Questions: Annotated

AC3 DL RESIDENT COURSE INTERVIEW QUESTIONS: Annotated

1. What factors, if any, limited your participation in the ASYNCHRONOUS self-study portion
of the AC3 DL course? (equipment, employment, Army policies)

Student Comments:

Better if on CD - courseware/test/chance to click to get online.

Mismatched questions are a problem. Questions not linked to right answer. Volume 2
had 10 questions, three mismatched - can't get better than 70%.

Examine how much of the course could be CD ROM delivered.

Consider making files downloadable.

Oregon Armory has a firewall, can't connect. Problem with tech support on weekends.
SGI provided great feedback and grading.

Big blocks of instruction are a problem. Start late and see a nine-hour block of VTOC
and sign off - can't do big blocks. School representative responded that the school is
changing big blocks of instruction into smaller chunks.

If there is a technical problem with the system on the weekend then we loose really
valuable time.

Want course online and redundant course material on CD ROM. Other course example -
get stack of CD's and coordinate with school to get access codes.

Issue: the sheer volume of information - how to cut down Practical Exercises - but still
have them available for practice.

Relook the course content - make sure it is all relevant. I skip Fire Fights.

2. What Reserve Component policies would support participation in AC3 DL self-study
ASYNCHRONOUS training?

Student Comments:

Not everyone has PC's - need to have unit hand receipt PC's, printers, paper. Make sure
that training is compatible 800/600 format with laptops, not developed for big 17/19"
screens.

Need Service Pack 3 or higher to make NT work.

Nobody got compensated for Asynchronous. The way to support it is to treat it as
attending school = points (is this retirement/training hours?).

Regular resident course soldiers get points, DL Asynchronous and Synchronous students
don't. Solution might be to give retirement points, get centralized money.

Book based correspondence courses like CS might be worth 20 retirement. Point is to
treat DL like other courses. Indicate course completion on Form 1059 - will get
retirement points, how many credit hours?

J-1



7. Do Reserve Component Officers have the computer skills necessary to work through AC3 DL
training?

YES

NO.

NOT SURE

8. What is your view of the quality of AC3 DL ASYNCHRONOUS portion of the training?

VERY HIGH
HIGH
ACCEPTABLE
LOW

VERY LOW

9. What is your view of the quality of AC3 DL SYNCHRONOUS portion of the training?
VERY HIGH

HIGH

ACCEPTABLE

LOW

VERY LOW

10. What is your view of the quality of AC3 DL RESIDENT portion of the training?
VERY HIGH

HIGH

ACCEPTABLE

LOW

VERY LOW

11. What is your view of the quality of AC3 DL training overall?
VERY HIGH

HIGH

ACCEPTABLE

LOW

VERY LOW

12. Please provide any comments you would like to make on AC3 DL training.

C-2



Participants

The 15 RC officers who completed the first iteration of the AC3 DL program provided
the data for the DL approach. One hundred and forty-nine officers from five AOAC RC classes
were used as the traditional paper booklet correspondence course comparison group for this
research. Demographic data for research participants were collected using a demographics
questionnaire developed for this purpose (see Appendix B).

Within-Course Measures

Lesson Test Scores. Each AC3 DL lesson includes a short test administered once as a
pre-test and again as a final or post-test. A comparison of pre-test and post-test scores thus
provides a measure of student learning within each lesson. Each test is automatically scored
within the courseware. The pre-test is administered once at the beginning of each lesson and the
post-test is administered at the end of each lesson until the student passes it (scores 70% or
higher). Only the final post-test score is maintained in the AC3 DL student management
database. Pre-test scores and post-test scores were recorded for comparison. The repeated
testing and 70% or better criterion could contribute to artificially inflated AC3 DL test scores.
Data are not available to identify the number of times each post-test was taken.

End of Volume (EOV) Test Scores. The AC3 DL lessons are organized into eight
volumes. Each volume after the first ends with an EOV test on which the student must score
70% or higher before moving to the next volume. The EOV tests include a variety of question
types, some of which are scored automatically and the remainder of which are scored manually
by the Phase JA instructor. Students retake portions of the EOV tests until they score 70% or
higher. The data that will be examined here are EOV test scores and number of times each EOV
test was taken.

AC3 DL Survey and Interview. The USAARMS and ARI personnel prepared a twelve-
item AC3 DL Resident Course Survey to identify how well elements of the program meet
training needs (see Appendix C). A 16 item AC3 DL Student Interview Guide was also prepared
to identify Army policies and incentives that could impact RC officer participation in the AC3
DL training program (see Appendix D). Both the survey and interview were administered to
students during the two-week resident portion of the AC3 DL program.

Between-Course Comparative Measures

Resident Knowledge Test. A central goal of the present research was to assess whether
the AOAC RC and AC3 DL students were equally prepared by their respective training programs
prior to their arrival at resident training. Both courses include training in knowledge of company
team operations, and it was decided to compare the two courses with regard to student mastery of

this subject matter.

The USAARMS and ARI personnel jointly developed a 62 item multiple choice Resident
Knowledge Test of company team operations, with half the items coming from the AOAC RC
correspondence materials and half from AC3 DL lesson tests. The USAARMS and ARI



5. Use of available systems - familiarity with techniques, methods, and tools that give you an
edge including computer usage but also analytic techniques.

SGI Response:
e The AC3 DL may have more experience with computer usage.

6. Supervising - controlling, directing, evaluating, coordinating the actions of subordinates.

SGI Response:

e Synchronous VTOC training first addresses MDMP, the SGI plays the role of the XO
and/or CDR, and walks students through wargaming the COA analysis. First the SGI
shows the students what to do, then appoints students to play the role of XOs or S3 which
does involve supervising, controlling, and directing students in subordinate staff roles.
The previous AOAC RC course did not train staff processes, course content was all
Company Offense (for commanders), then Co TM Defense, OPORDS, written exam.

7. Professional ethics - loyalty to the nation, the Army, the unit; service and integrity.

SGI Response:
e Not part of the AC3 DL or AOAC RC courses really. No difference.

8. Teaching/counseling - improving performance of subordinate by overcoming
problems, gaining new skills, modeling behaviors.

SGI Response:
e The AC3 DL includes peer motivation, peer tutoring as part of the learning process. The
AOAC RC did not provide for student teaming prior to the Resident course.

9. Soldier team development - creating strong bonds between leader and soldiers and among
soldiers.

SGI Response:

e The AC3 DL Asynchronous phase provides the opportunity to identify Student Leaders.
Students are teamed in the Synchronous VTOC portion of training. At the Resident class
the students were all well past the point of introductions, the bond between students was
there when they showed up. They didn't have to feel each other out. The AOAC RC
students did not know each other prior to Resident Class.



Student Attrition. Student attrition from the AC3 DL course appears to be high.
However, in developing quantitative estimates of attrition it is important to recognize the factors
that contribute to these estimates. It is important to note that some students were enrolled by |
their unit leaders, did not participate in training from the outset, and might thus be distinguished
from those students who did make an attempt to work through the training materials, but failed.
Also, some students who failed to complete the AC3 DL course in the first class might still
complete the course in a later class, and thus should not be considered in attrition estimates. As
of October 1999, 60 students were enrolled in the AC3 DL course and approximately 30 were
actively participating in Phase I asynchronous training (Burnside, Wardell & Sanders, 1999).
Fifteen students completed the AC3 DL course in the first class, which would correspond to an
attrition rate of 75% if we include all students enrolled, and 50% if we limit the population of
students to only those who were actively participating as of October 1999. Attrition estimates
for students in the AOAC RC course are also high. The AOAC RC Amy Training
Requirements and Resources System (ATRRS) course data for the four years 1995-1998, reveals
that 1,296 students were enrolled and that 785 failed to graduate, yielding a 61% attrition
estimate (U.S. Army Armor School, 2000).

Phase IA Correspondence Course Training Times. Given that the correspondence
phases of the AC3 DL and AOAC RC courses are self-paced there are many different start and
finish dates for students, and total time spent in correspondence training will vary (see Appendix
G). Table 2 presents a summary of AC3 DL and AOAC RC course correspondence training start
and finish times, and an estimate of total months devoted to correspondence training. It should
be noted that students filled out the demographics information survey and completed the 62-item
Resident Knowledge Test on the first day of the two-week resident training portion of
instruction. The delay or “retention interval” between correspondence training and resident
training differed across students and courses.

Table 2

Phase IA Correspondence Phase Training Times Summary (Mean/Std. Dev.)

Course Students Months Since Months Since Total Months in

Start Finish Correspondence Training
AC3 DL 15 13.00/2.45 5.47/3.62 7.53/3.09
AOACRC 149 — 29.56/11.02 13.14/6.24 16.42/9.11

Note. Five AOAC RC cases were not included where Months Since Start = Months Since
Finish.

Statistical analyses were conducted to compare Months Since Finish, and Total Months
in Correspondence Training between the AC3 DL and AOAC RC programs. It is desirable to
merge the data from all five AOAC RC classes to increase the reliability of comparisons to this
type of training. The first step required for merging the data is to ensure that test score variances
are not significantly different across the five classes. If there is not a significant difference
across classes, the variances can be pooled for a comparison between all AOAC RC classes and