Table 1

Usefulness Ratings for the Principles of War Module by Lesson Section
(in percent)

Very Somewhat Somewhat Very
Useful Useful Useless Useless
Learning
Objective 37.5° 12.5 12.5 37.5
Definition 62.5 25.0 0.0 12.5
Doctrinal
Information 62.5 12.5 12.5 12.5
Battle of the
Bulge 25.0 12.5 : 25.0 37.5
React to
Contact 12.5 25.0 50.0 12.5
Review 12.5 0.0 0.0 87.5

* n=8 forall cells.

As can be seen from Table 1, sections devoted to defining and explaining the
principle (doctrinal information) were seen as the most useful sections. The review
section was seen by virtually all subjects (87.5%) as useless. This was due in large part
to the fact that the review section did not summarize key points, but simply instructed
students to go back to the appropriate section(s) in the lesson if they felt they needed to
review an area. For the learning objective and to a lesser extent the Battle of the Bulge
and react to contact sections, subjects seemed somewhat split in their reactions of the
usefulness of these sections. For the learning objective section, one half of the subjects
thought that it was very or somewhat useful and the other half thought that it was
somewhat or very useless. For the Battle of the Bulge and react to contact sections, sixty-
two percent of the subjects felt that these sections were either somewhat or very useless.

Relevance of specific lesson sections. Sixty-two percent (62.5%) of the
experimental subjects felt that the information presented in the module seemed mostly

relevant to the stated instructional goals. Table 2 summarizes the relevance ratings given
by the subjects for each lesson section.

13



information related to the subject’s prior experience with TCC and computers (see
Appendix B for all questionnaires).

TCC Principles of War Module Evaluation Questionnaire. Two versions of this

paper-and-pencil instrument were constructed. Subjects in the control condition
completed a three item survey which included items tapping the subject’s exposure to the
concepts on the end-of-module quiz, difficulty of quiz items, and comprehension of the
items.

Subjects in the experimental condition completed a more detailed version of the
control questionnaire which tapped the interest level of the module, usefulness of
information presented in the module, usefulness of specific lesson sections, i-e., learning
objective, definitions, doctrinal information, examples, and review; relevance of
information in the module and of specific lesson sections; effectiveness of examples,
usefulness of the self-assessments; difficulty and fairness of quiz items, and overall
instructional value of the module.

Principles of War Computer-Based Instructional Module. The Principles of War
module consisted of nine lessons. Each lesson covered a specific principle of war:
objective, offensive, mass, economy of force, maneuver, unity of command, security,
surprise, and simplicity. For continuity purposes and programming ease, each lesson was
organized in an identical format. Lessons started with a screen detailing the learning
objective. This was followed by sections which focused on: definitions (of the specific
principle of war), doctrinal information, historical examples (Battle of the Bulge), react to
contact drills, lesson review, and self-assessment questions. (See Figures 1-5 for screen
design examples.) Each lesson requires approximately seven to ten minutes to complete.
Once the subject completes all nine lessons, he or she is then presented, in this prototype
module, with a 27-item multiple-choice quiz. Due to time constraints on the part of the
course developers, the end-of-module quiz items were identical to the self-assessment
items used at the end of each of the nine lessons, but presented in a different order. The
quiz requires approximately 15 - 20 minutes to complete.



Subjective Value of the Lesson Sections

Extensive feedback was solicited from the subjects on the organization of the
module and the value of the subject content. The feedback indicated that certain
sections clearly need to be modified. The review section was seen by the subjects as
virtually worthless. The learning objective screen, although viewed by half of the
subjects as very or somewhat useful, can also be improved. In both instances,
modifications can be made rather easily. The review section must include key points
summarizing major issues presented in the lesson. The learning objective section can be
presented at the beginning of the module as part of the introduction to the instruction
which all students are exposed to when they log on for the first time. Since all lessons
have the same action, conditions, and standards, there is no need to repeat the same
generic screen (section) for each lesson.

The Battle of the Bulge and react to contact sections, while appearing to have
some relevance and utility for learning the course material, could be greatly improved.
particularly the react to contact drill. The drill scenarios that were presented were seen by
the subjects as boring. This is understandable since the geography, mission, enemy, and
situation were always the same. Using different geographic settings and mission
scenarios, manipulating the enemy composition and situation, and tailoring the scenarios
for a specific principle of war would greatly enhance the value of this section.

Frequently, the subjects had little idea what principle of war they were viewing since the
scenarios were always the same. For many of the scenarios, it was only when the subject
heard the post-scenario narration that he realized what principle of war he was viewing.

The Battle of the Bulge section attempted to provide different examples to try to
match the principle of war that was being taught. However, subjects, at times, still did
not grasp the connection between the example and the principle or felt that some of the
examples were too similar to adequately allow the them to distinguish the key aspects of
the principle of war in question.

For both the Battle of the Bulge and react to contact sections major changes
would be required. The work required to correct these shortcomings would be extensive
and current course production deadlines make it unlikely that major changes will be made
in these sections.

The feedback received on the definition and doctrinal information sections
seemed to be, for the most part, very useful and relevant. Although the phrasing of terms
and information was seen by a few subjects as too complex or awkward, course
developers indicate that no changes can be made since the definitions must be exactly
those used in the FM.

With regard to the self-assessment and quiz items, several modifications are

required. First, all self-assessment items whose stem includes the phrase “Which of the
following is not...” should be changed to a positive statement, e.g., “Which of the

17



FOREWORD

In June 1995, Army Training XXI was formally acknowledged as the training component
for Force XXI, the Army’s redesigned force of the next century. Army Training XXI consists of
three axes: Warfighter XXI, Warrior XXI, and the Warrior Network (WARNET XXI). The
objective of Warfighter XXI and Warrior XXI is to help develop or reengineer the three pillars of
Army training: institutional, self-development, and unit training with a specific focus on digitized
training. WARNET XXI provides for the linkage of training acquisition, new equipment training,
and digitization of systems training support products.

Concurrent with these efforts was the implementation of the WARNET Pilot initiative
The role of WARNET Pilot is to synchronize the development and delivery of new technologies
in training. More specifically, the intent is to use distance learning and information technologies to
pilot the delivery of high-quality, standardized training to soldiers and units where and when they
need it. The WARNET Pilot Team was formed at the U.S. Army Infantry Center, under the
Directorate of Operations and Training (DOT) - U.S. Army Infantry School (USAIS) with the
support of the National Guard Bureau to manage multimedia distance learning and distributive
training development for USAIS. In concert with other training activities at USAIS, the
WARNET Pilot Team was given the responsibility for developing Infantry-specific courses.

In November 1995, the U.S. Army Research Institute for the Behavioral and Social
Sciences Infantry Forces Research Unit (IFRU) was contacted by the Office of the Special
Assistant to the Commanding General, Army National Guard to provide technical assistance for
the WARNET Pilot Team course development effort. This report describes IFRU’s work in the
formative evaluation of the computer-based Tactics Certification Course (TCC) - Principles of
War Module. The TCC is the first core Infantry resident course to undergo the conversion from a
resident classroomylecture-based format to a distance learning format. This report provides a
model for evaluating the effectiveness of newly developed courses with multimedia formats. In so
doing, it provides the means for ensuring that the distance learning technologies selected for the
TCC are optimally employed. The results of this research were briefed to course developers, the
WARNET Pilot Team, and USAIS-DOT staff.

ZITA M. SIMUTIS EDGAR M. JOHNSON
Technical Director Director



5. How effective were the React to Contact drills in helping you to
understand the principles of war?

Very Somewhat Somewhat Very

Effective Effective Ineffective Ineffective
6. How useful were the self assessments (the short, end-of-lesson quizzes) in

helping you to learn the principles of war?

Very Somewhat Somewhat Very

Useful Useful Useless Useless
7. How difficult were the items on the final quiz for you?
Very Somewhat Somewhat Very
Difficult Difficult Easy Easy
8. Do you feel that the items on the final quiz adequately assessed your
knowledge of the principles of war?
YES
NO
9. Overall, how would you rate the instructional value of the principles

of war module?

Excellent Good Fair Poor

Please provide us with any additional comments that you feel may help us in
evaluating the effectiveness of this module.

B-4



Table 2

Relevance Ratings for the Principles of War Module by Lesson Section
(in percent)

Very Somewhat Somewhat Very
Relevant Relevant Irrelevant Irrelevant
Learning
Objective 37.5° 12.5 12.5 37.5
Definition 75.0 12.5 12.5 0.0
Doctrinal
Information 75.0 25.0 0.0 0.0
Battle of the
Bulge 12.5 37.5 12.5 37.5
React to
Contact 37.5 37.5 12.5 12.5
Review 12.5 0.0 0.0 87.5

*n=8 for all cells.

As was the case for the ratings of usefulness, sections devoted to defining and
explaining the principle (doctrinal information) were seen as the most relevant, in terms
of providing information supporting the stated instructional goals. The learning objective
and the Battle of the Bulge sections seemed to draw mixed reactions from the subjects.
One half of the subjects thought that these sections provided very or somewhat relevant
information. The other half thought that these sections provided somewhat or very
irrelevant information. For the react to contact section, seventy-five percent of the
subjects felt that this section presented information that was very or somewhat relevant to
the stated instructional goals. Finally, the review section was seen as very irrelevant by
virtually all respondents (87.5%).

Effectiveness. test difficulty, and instructional value ratings.. The majority of the
experimental subjects felt that the Battle of the Bulge example (62.5%) and the react to
contact drills (75%) were very or somewhat effective in helping them to understand the
principles of war. All subjects thought that the self-assessment questions at the end of
each lesson were either very or somewhat useful in helping them learn the principles of
War.

14



References

Buskist. Ww. Cush, D., & DeGrandpre, R. J. (1991). The lif and times of PSI. Journal
of Behavioral Education, 1(2)..215-234

Corey, J. R.. & McMichael. J. S. (1970). Using personalized instruction in colleve
courses. New York: Meredith Corporation.

Fletcher. J. D. (1992). Individualized systems of instruction. In M. C. Alkin (Ed.).
Encyclopedia of educational research (6th ed.. pp. 613-620). New York:

Macmillan.

i Fletcher. J. D.. & Atkinson. R. C. (1973). Evaiuation of the Siz
initial reading. Journal of Educational Psvchologv. 63

Kulik, J. A. (1976). PSI: A formative evaluation. In B. A. Green. Jr.. (Ed.).
Personalized instruction in higher education: Proceedines of the second national
conference. Washington. DC: Center for Personalized Instruction.

Kulik. J. A.. Kulik. C. C.. & Cohen. P. A. (1979). A meta-analysis of outcome studies
of Keller's. personalized system of instruction. _American Psvchologist. 34.
307-318.

Marreili. A. F. (1995. September). Writing multiple choice items. Intemational Society
izz Performance and Instruction. 34(8). 24-29.

McMichael. J. S.. & Corey. J. R. (1969). Contingency management in-an introductory
psychology course produces better learning. Journal of Applied Beh
Analvsis. 2. 79-83.

| Morris. C.. & Kimbrell. G. (1972). Performance and
in an introductory psychology course. Psve

Ragosta, M.. Holland. P. W.. & Jamison. D. T. (1982). Computer

and compensatory education: The ETS:LAUSD studv (Final report

i NJ: Educational Testing Service. (ERIC Document Reproduction Service No. ED
| 222 169).

Sheppard. W. C. & MacDermot. H. G., (1970). Design and evaluation of a programmed
. course in introductory psychology. Journal of Applied Behavior Analvsis. 3.
5-11.

Smith, P. L.. and Ragan. T. J. (1993). Instructional design. New York: Macmillan

Reproduced From

ding Page Blank
Best Available Copy Preceding


go back and review selected sections in the lesson, was not seen by the subjects as very
useful or relevant.

Self-assessment items. Although most subjects felt the self-assessment section
was useful and relevant, they did not like the questions whose stems included the phrase
“Which of the following is not...” Better distracter items are needed, preferably those that
are related to the principle of war being studied instead of using definitions from other
principles of war. For many items the distracters are so obviously unrelated to the
principle being assessed that the subject could easily eliminate one or two alternatives.

Final exam. The final exam was problematical in that it was composed of the
self-assessment items from each of the lessons but presented to the subject in a different
order. The present format potentially confounds the treatment effect with any incidental
learning that may have occurred from previous exposure to the items during the self-
assessment portions of the lessons.

Audio. The volume was not consistent throughout the lessons. Certain sections
were extremely loud and many subjects had to turn the volume down on these sections
and turn it up for the next section (s). The fluctuations proved to be very distracting for
the subjects.

Navigation. Some subjects had problems navigating in the program, e.g., getting
disoriented at the end of the lesson, having trouble returning to the main directory, or
difficulty exiting a lesson and entering the next lesson.

Motivation. Some subjects felt that the instruction was boring, particularly the
Battle of the Bulge and react to contact sections (though not necessarily useless,
irrelevant, or ineffective). This was primarily due to the similarity among many of the
Battle of the Bulge examples used in demonstrating key aspects of the principle of war in
question. The react to contact drill scenarios were, for all intents and purposes, identical
across lessons.

Discussion

The results of this evaluation suggest that the prototype CBI Principles of War
module was effective in teaching doctrinal material, specifically the definitions of the
nine principles of war and supporting doctrinal information. As noted earlier, the
difference between the groups on the final quiz may be somewhat inflated since the items
used for the quiz were the same as those used for the self-assessment questions for each
lesson. Some subjects indicated that certain questions seemed familiar, but this may be
due to the similarity of item stems, which were virtually identical for certain sets of
questions, e.g., definition, doctrinal information. In no instance did the researchers get
any sense that the subjects recognized both the question and the correct response for that
question. This issue has been noted to the course designers and will be corrected for the
final principles of war module.

16



assistance from the researcher if they had difficulty with the navigation tools. However,
questions related to the lesson content or quiz items were discouraged.

The researchers emphasized to the subjects the importance of proceeding through
the lessons carefully and doing their best on the self-assessment questions and the 27-
item multiple-choice quiz at the end of the module. Observations from earlier piloting
suggested that most subjects completed the Principles of War module in about one-and-a-
half to two hours. Based on these observations, a three hour time block was allotted to
ensure completion of the lessons and the end-of-module quiz by all subjects. Upon
completion of the module and the quiz, subjects were debriefed and completed the
experimental version of the TCC Principles of War Module Evaluation Questionnaire.

For the control condition, subjects were instructed that they would take the end-
of-module quiz only. Control subjects were also assigned to their own computers and
work stations, and were observed by one of the researchers. Like the experimental
subjects, the control subjects began with the introduction screen and proceeded to the
menu. From the menu, they were instructed to select the end-of-module quiz. Control
subjects were allotted one hour to complete the quiz. Upon completion of the quiz,
subjects were debriefed and then completed the control version of the TCC Principles of
War Module Evaluation Questionnaire.

Results

Computer Experience

Analyses conducted on the Background/Computer Survey indicated that the
subjects were quite familiar with computers. Ninety-four percent of the subjects had
some experience working with computers. On the average, subjects had been working
with computers for approximately three years (35 months), range = 0 - 11 years. Sixty-
nine percent of the subjects had some formal computer training. The majority of the
subjects (81.3%) currently use computers in their jobs as part of their duties. Subjects, to
a large extent (81.3%), felt comfortable using computers. However, only thirty-seven
percent (37.5%) of the subjects said that they owned a personal computer.

End-of-Module Quiz Performance

Means and standard deviations were calculated for the quiz scores for subjects by
condition. The experimental group (n = 8) answered an average of 23.9 items correctly
out of 27 total items (88.9%), SD = 2.10, range 20 - 26. In contrast, the control group
(n = 8) answered an average of 13.1 out of 27 items correctly (48.1%), SD = 4.88, range
= 8-21. This difference was statistically significant, F (1, 14) = 32.73, p= .0001.
Despite the small sample size, these findings (i.e., the magnitude of the group difference
and the significant F value) suggest that the group difference was, in fact, reliable.

11



operational symbols, troop leading procedures, offensive operations, defensive
operations, and a brief overview on engineer and fire support operations.

Students are provided with various handouts and reading materials, primarily
chapters taken from Army field manuals (FMs). An open book, short answer/essay “pre-
test” is given to the students as a take home assignment prior to the start of the course.
With regard to course evaluation, three 20 minute quizzes are given at the beginning of
each class covering key points from the previous day’s instruction. Quizzes are primarily
short answer/essay. In addition, a one hour operational symbols quiz is administered
during the second day. The quiz consists of a mission scenario in which students are
asked to use the symbols to designate units and various actions on a map overlay.
Feedback is provided relatively quickly following each quiz. The final exam is, again,
primarily short answer/essay.

Significant hands-on time is devoted to practical exercises in the development and
presentation of operational orders (OPORDs). Additional time is allotted for constructing
map overlays. Students are encouraged to become actively involved in the topic
discussions. Large parts of the engineer and fire support sections are presented on video-
tape depicting these particular combat support units in action. This is accompanied by a
general overview of these units’ functions and the equipment that they use. The TEWT
entails having students walk the terrain representing the actual site on which the student’s
OPORD was based. Some students also brief their OPORD if they have not already done
this during the classroom practical OPORD exercise.

TCC: Course Redesign and Development

Based in part on ARI researchers’ analysis of course objectives and content, it
appeared that much of the TCC small group instruction could be eliminated and replaced
with a more efficient CD-ROM computer based instructional format.

A literature review was conducted to identify candidate instructional models to
guide the development of the new course. A key objective of the WARNET Pilot Team
was to successfully develop an instructional approach that ensured learning and high
levels of retention. With this in mind, the instructional model selected was based, in
large part, on the Personalized System of Instruction (PSI) [Buskist, Cush, &
DeGrandpre, 1991]. PSI is characterized by the following qualities: self-pacing, unit
mastery, lectures and demonstrations, written communication, and the use of proctors.
For a complete description of the PSI approach and supporting empirical documentation,
see Appendix A,

Revised PSI Model as Applied to the TCC

The modified version of the PSI model proposed for the computer-based/CD-
ROM section of the TCC has several key features. First, students must demonstrate high
levels of mastery of unit (module) content before they can move on to the next unit.



TCC Principles of War Module
Evaluation Questionnaire

1. Have you had any previous experience with the nine principles of war?
Yes
No

If yes, how much did this exposure help you in answering the quiz items?

2. How difficult were the quiz items for you?
Very Somewhat Somewhat Very
Difficult Difficult Easy Easy
3. Were the quiz items easy to understand?

Yes

No

If no, explain:

Please provide us with any additional comments that you feel may help us in
evaluating the effectiveness of this quiz.

B-2



Proctors. Proctors are selected from a pool of students who have taken a previous
section of the PSI course and have met several other criteria. The proctor is typically
chosen for his demonstrated mastery of the course content, for his understanding of the
special problems that confront the student as a beginner, and for his willingness to assist.
The use of proctors provides the personal or social aspect usually lost by having fewer
lectures. The proctor, as a peer who is intimately aware of the nuances of preparing for
and mastering unit quizzes, is seen as someone who can relate more effectively to the
student than the instructor, providing a personal touch to the learning process.

PSI Effectiveness: Outcome Research

Buskist, Cush, and DeGrandpre (1991) reviewed a number of studies which have
investigated or summarized the results of research aimed at comparing the relative
effectiveness of PSI with other pedagogical methods in facilitating knowledge acquisition
and enhancing overall academic performance. In general, specific studies have shown
that, compared to traditional lecture methods or other appropriate controls using the same
reading assignments, students in PSI courses have done better on: 1) multiple-choice
examinations (McMichael & Corey, 1969); 2) final examinations (Corey & McMichael,
1970); 3) essay examinations (Sheppard & MacDermot, 1970); and 4) examinations
designed specifically to test recall and application (Morris & Kimbrell, 1972). Even with
the number of test questions controlled or partially controlled for, PSI produced better
student performance.

Extensive reviews comparing PS] to traditional methods of instruction clearly
show the superiority of PSI. Taveggia (1976) analyzed the results of fourteen studies
which looked at course examinations across a variety of content areas including
introductory psychology, learning, cultural anthropology, chemistry, and electrical,
mechanical, and nuclear engineering. He found, without exception, that students did
better under PSI methods than under traditional ones.

In another review of 31 studies by Kulik (1976) comparing PSI to other teaching
methods across a wide range of subject areas, 81 percent of the cases showed that the
final exam performance for PSI students was statistically superior to those in traditional
courses. Kulik also found that retention and transfer were superior using PSI compared
to traditional methods of teaching.

Kulik, Kulik, and Cohen (1979) conducted a meta analysis of 75 comparative
studies pitting PSI against traditional methods of instruction. Among the many findings
that were reported, it was found that PSI examination scores were significantly higher
(by about eight points) than scores obtained from lecture based courses. Final course
grades were also significantly higher in the PSI courses. Using the traditional four point
scale, with 4 being superior and 1 passing, PSI courses averaged 3.09. In comparison,
the control classes averaged 2.31. Kulik, Kulik, and Cohen (1979) reported that the PSI
programs they studied raised final examination scores by about .50 standard deviations
over programs using conventional (non-PSI) means of instruction. They also found that

A-2



CONTENTS (Continued)

APPENDIX A. Personalized System of Instruction (PSI):
Overview and Empirical Documentation..........ssscssssessesseensceesseseeseessecnes

B. Background/Computer Experience Survey ........ssescsssssecsesseesseseeaneseeses
TCC Principles of War Module Evaluation Questionnaire
(Control Group) ......cessssssesssesesssesseessesseessssnsssesnssssesessnsessesseneeseeneesneanesseeses
TCC Principles of War Module Evaluation Questionnaire
(Experimental Group) ........sc:csssssessssessesssseeseesesesseesesssseesssensseensessseenseeesees

LIST OF TABLES

Table 1. Usefulness Ratings for the Principles of War Module by
Lesson Section ....csscsssessessessessessseesecesesssssessesssessessessssseesesseesessessvessensetsees
2. Relevance Ratings for the Principles of War Module by

Lesson Section ........scscesssssssssesssessesssecsecseesessessersssesessessseseesessetsessesssecseeaee

LIST OF FIGURES

Figure 1. TCC Principles of War Menu Screen.............0+

2. TCC Principles of War Learning Objective Screen.
. TCC Principles of War Battle of the Bulge Screen .
. TCC Principles of War React to Contact Screen.
. TCC Principles of War “Objective Dog” Screen.

URW

Page

A-l

B-1

13

14

= © © & oo



PRELIMINARY EVALUATION OF THE COMPUTER-BASED TACTICS CERTIFICATION
COURSE—PRINCIPLES OF WAR MODULE

EXECUTIVE SUMMARY

Research Requirements:

In 1995 the U.S. Army Infantry School (USAIS) at Fort Benning was selected by the
Deputy Chief of Staff for Training as the lead agency for implementing the WARNET Pilot
initiative. The role of the WARNET Pilot is to use distance learning and information technologies
to pilot the delivery of high-quality, standardized training to soldiers and units where and when they
need it. The WARNET Pilot Team was given the responsibility for developing selected Infantry-
specific courses. The U.S. Army Research Institute for the Behavioral and Social Sciences Infantry
Forces Research Unit at Fort Benning agreed to provide technical assistance for the course
development effort. This research describes a portion of the formative evaluative process used in
developing the computer-based Principles of War module for the Tactics Certification Course.

Procedure:

Sixteen subjects from USAIS participated in the research. Subjects were randomly assigned
to one of two groups. The experimental group received the computer-based instructional version of
the Principles of War module and an end-of-module quiz. Subjects assigned to the control
condition received only the end-of-module quiz. In addition to the quiz, subjects completed a
background/computer experience survey and a questionnaire assessing their opinions on selected
aspects of the Principles of War module.

Findings:

Subjects in the experimental group answered significantly more quiz items correctly
(88.9%) than did subjects in the control condition (48.1%). Ratings of selected aspects of the
module varied. Certain sections of the module clearly needed to be modified. Other sections
required only minor refinements. The overall ratings of the instructional value of the course were
positive.

Utilization of Findings:

The findings from this research were briefed to course developers, the WARNET Pilot
Team, and USAIS staff. The results of this feedback will be used to refine selected areas of the
module. The formative evaluation process followed in this research, and subsequent TCC modules
will help ensure that the distance learning technologies selected for the TCC are optimally
employed.

vii



The group difference may be somewhat inflated, however, and should be
interpreted cautiously. As was noted earlier, due to time constraints on the part of the
course designer, the end of module quiz items were identical to the self-assessment items
received at the end of each lesson. The impact that this item exposure may have had on
subject performance is unclear. Although they were not formally questioned, based on
the observations and remarks made during the end-of-module quiz, the subjects did not
seem to be aware of the connection between the self-assessment and quiz items.

Pearson product-moment correlations were computed between quiz scores and
computer background, age, status (military vs civilian), TCC qualification, and treatment
condition. The results showed only one correlation reached statistical significance. High
quiz scores were significantly correlated with subjects having received the module of
instruction, r(16) = -.84, p = .0001. Computer background, experience, level of comfort
working with computers, etc., were not significantly correlated with quiz performance.
Additional analyses (Chi-square) revealed no significant group differences for
educational level.

Further analysis of the data showed that the two subjects reporting prior TCC
qualification were assigned to the experimental group. However, their quiz scores were
not significantly higher than the scores obtained by the remaining subjects assigned to the
group. It appears, therefore, that prior TCC qualification did not artificially inflate the
experimental group scores.

Subjective Evaluations of the Principles of War Module

Experimental subjects were asked to rate the interest level, overall usefulness and
relevance of the module. Seventy-one percent (71.5%) of the experimental subjects felt
that the content of the module was very or somewhat interesting.

Usefulness of specific lesson sections. With regard to usefulness, all respondents
thought that the information presented in the module was either very useful (37.5%) or
somewhat useful (62.5%). Each lesson section was also rated for usefulness. Table 1
shows the breakdown of ratings by section.

12



evaluation, testing and refinement, followed by a more ambitious phase entailing
simultaneous development and refinement of new courses. The focus of the WARNET
pilot effort is directed toward the production and development of five short courses:
Tactics Certification Course (TCC), Teach, Assess, Counsel - Training and Orientation
Course, Instructor-Trainer Course, Infantry Mortar Leader Course, and the Bradley
Fighting Vehicle Leaders Course (WARNET Pilot Team TATSC Management Plan.
February, 1996).

The TCC was selected as the prototype course based on two criteria. The first
was that all Officer Candidate School (OCS) and Career Management Field (CMF) 11
course instructors are required to be TCC qualified. In order to establish the base of
instructors for the implementation of TASS in the seven regions designated by TRADOC
located throughout the U.S., TCC would, logically, have to be the first course. The
second rationale for selecting the TCC was based on time and experience. A reasonably
short course had to be selected to prove the technology and concept would work. This
would give USAIS the data base and experience upon which to structure future course
development. The TCC was selected because it has a 10 day resident phase Program of
Instruction (POI) that could be reduced to a shorter resident phase and a nonresident
phase using distance learning technology such as CD-ROM and CBI. The overall
concept for TATSC development is to create courseware that uses technology to gain
maximum learning for the student with a minimum of resident school attendance
(WARNET Pilot Team TATSC Management Plan, February, 1996).

ARI ’s Involvement with the WARNET Pilot Team

In November 1995, the ARI-Infantry Forces Research Unit (IFRU) was contacted
by the Office of the Special Assistant to the Commanding General, Army National Guard
to provide technical assistance for the WARNET Pilot Team course development effort.
Specifically, ARI- IFRU’s assistance was requested in the following areas: identification
and selection of an appropriate instructional model (or models) to guide course (TCC)
development; review of new course structure and format to ensure WARNET Pilot Team
course developers adhered to the major tenets of the instructional model(s); development
of an evaluation plan to compare the effectiveness of the new course vs traditional
method(s) of instruction; assistance in course evaluation; computation of all statistical
analyses and delivery of feedback from the evaluation to course developers, WARNET
Pilot Team, and USAIS-DOT staff.

TCC: Course Organization

In February and March 1996, two researchers from ARI-IFRU monitored parts of
two separate TCC classes. Classes consist of six to eleven students with one instructor.
The TCC, as presently configured, involves four days of lecture and small group
instruction, a one day tactical exercise without troops (TEWT), followed by a two hour
end-of-course exam on the sixth day. Content areas include Army operations,



following is an example of...” Subjects found these negative items very difficult to
“retranslate” or process. In many instances, the retranslation process clearly slowed the
subjects’ response time. At times this lead them to respond with the wrong answer.

Tamir (1993) found that while both modes of questions (i.e., positive and negative
phrasing) are valid, significant performance differences were found for items of high
cognitive value, i.e., items involving interpretation, or drawing conclusions. In Tamir’s
study students performed significantly better on the more cognitively challenging
questions that were written in the positive mode than those that were written in the
negative mode. For less difficult items, differences between positive and negative modes
of questions were negligible.

Marrelli (1995) stresses the use of the positive rather than the negative form in the
stem whenever possible. If a negative form of the stem is required, she recommends
capitalizing or underlying the negative term.

As implied earlier, the final version of the principles of war module will have quiz
questions that will be different in structure (wording) and format from the self-assessment
items. The screening quiz and the final quiz for those who opt not to test out of the
module will be identical - matching definitions, doctrinal information and examples to the
nine principles of war. The self-assessment items will be multiple-choice. For those
students who test out of only portions of the module or for those opting to start
immediately with the lessons but fail specific sections of the final exam, a different
testing format will be used for retesting these individuals. Specifically, after the students
repeat the specific lesson(s) they will be retested on the specific principle(s) of war
covered by the lesson(s) using a multiple-choice format.

Conclusion

The Principles of War module represents only one of five modules that will be
developed for the TCC. Each of these modules will go through the same (or a similar)
evaluative process as the Principles of War module described in this report. The results
of this evaluation indicated that the Principles of War module, as constructed, is effective
at teaching knowledge about selected concepts. Sections of the module could be
significantly improved, however.

Survey and observational data indicated that certain sections such as The Battle of
the Bulge and React to Contact drills, while potentially very useful, would require
extensive modifications. Other sections (Definition, Doctrinal Information) of the
Principles of War module were useful in promoting learning and/or provided relevant
information in support of the stated instructional goals. These sections require relatively
minor, if any, refinements. The Learning Objective section or screen can be easily
improved. The Review section, while seen as virtually worthless in its present format,
can be greatly improved. The modifications required in this section can be made quite

18



ri his
U.S. Army Research Institute
for the Behavioral and Social Sciences

Research Report 1714

Preliminary Evaluation of the
Computer-Based Tactics Certification
Course--Principles of War Module

Robert J. Pleban
Jamye B. Brown
and

Mike T. Martin
U.S. Army Research Institute

DTIC QUALITY EvSPECTED g

19980224 080

Approved for public release: distribution is unlimited.

July 1997



only on the selected lesson(s). Upon satisfactory mastery of the lesson(s), the student is
allowed to move to the next unit or module.

Development of Modules: Formative Evaluation

Using a formative evaluation plan adapted from Smith and Ragan (1993), the
principles of war module was carefully evaluated at key junctures during the
developmental process. A summary of the evaluation process is presented in the
following sections.

Goal review. The key point at this stage was to confirm that the goals of the
reconfigured TCC are representative of a real instructional need and are consistent with
USAIS/WARNET Pilot expectations. In a broad sense this issue was confirmed. One of
the major objectives of the WARNET Pilot initiative is to exploit distance learning and
information technologies to pilot the delivery of high quality, standardized training to
soldiers and units where and when they need it; thereby reducing the time and costs of
sending soldiers to the actual site of the course for instruction. The conversion of the
existing small group instruction/lecture-based TCC to a predominantly computer-based
instructional format represents a major step in addressing the instructional objectives of
WARNET Pilot.

Expert reviews. As draft versions of the Principles of War Module were
completed, course developers, ARI researchers, and an instructional design expert
reviewed course objectives, lesson review questions, and the end-of-module quiz to
insure key objectives were, in fact, being assessed. Checks were also made to see if
review (self-assessment) questions and the end of module quiz adequately sampled the
domain that the course objectives might cover. Screen design issues were also noted at
this time.

Before the converted TCC was used by students, subject matter experts and ARI
researchers examined the content of the instructional material for accuracy and
completeness. Checks were made to insure that: 1) course content was up to date; 2)
examples, practice exercises, and feedback were realistic and accurate; 3) the instruction
was appropriate for the target learners; and 4) the instructional strategies were, in fact,
consistent with principles of instructional theory.

Learner validation. During this phase, a prototype lesson (one of the principles of
war - mass) was administered to five Army non-commissioned officers representative of
the target population. Questions addressed during this time included: 1) Did the students
understand the instruction/textual material? 2) Do the learners know how to proceed
through the quizzes, exams, practical exercises? 3) Can they interpret the graphics on the
screen?

During this time the subjects were questioned about any problems they were
having and any misconceptions the subjects might have had were addressed. Checks

wn



Unit (module) quizzes are predominantly matching (The quiz for the prototype module
which was used in this evaluation was multiple-choice.) For the TCC, unit mastery
requires that the student answer all doctrinal definitions correctly. The exact number of
items and percentage required for mastery will vary, however, depending on the types of
questions, i.e., definition, knowledge, application, answered by the student.

Second, students receive immediate performance feedback consisting of a total
correct score (and percentage correct) and a score for each of the content areas for the
module (a total correct score and specific item feedback for each principle of war). Those
students whose scores fail to meet the selected criteria are directed back to the section(s)
in the lesson(s) where they had missed items and allowed to review relevant course
material. Following the review, students are quizzed (using a multiple-choice format) on
the appropriate section(s). Once the student has satisfactorily demonstrated mastery of
the sections or content areas in question, he or she is allowed to move on to the next unit
or module.

Third, interspersed within the unit are summary (self-assessment) questions
focusing the student’s attention on key points to be learned for each lesson.
Feedback (immediately) following the completion of the self-assessment questions
ensures that the student stays on track and is actively involved in the instructional
process. For the present module (Principles of War), three multiple-choice questions are
included at the end of the review section for each lesson. Students must answer all items
correctly before they can move on to the next lesson within the module.

No set deadlines for completing each section or quiz are provided, although a
general course deadline is established. For the most part, students can proceed at their
own pace.

Consistent with the self-pacing characteristic of PSI, an option is provided for the
student to test out of selected lessons or the entire module. The test is identical to the
final quiz described earlier which involves matching statements to the nine principles of
war. All doctrinal definitions must be answered correctly or the student must complete
the appropriate lesson(s) for the principle(s) of war in question. For the remaining items,
which tap the student’s doctrinal knowledge and ability to apply the doctrinal definitions,
the criterion for mastery will vary, depending on the types of questions correctly
answered by the student.

As was the case for the final quiz, students are provided with immediate
performance feedback consisting of a total correct score (and percentage correct). In
addition, students receive feedback for each area (principle of war) to include a total score
for the area in question and specific item feedback.

Students who fail to meet the criteria established for each principle of war must
take the lesson(s) for the specific principle(s) of war in question. Once the student
completes the assigned lesson(s), he or she is retested (using a multiple-choice format)



Appendix B
Background/Computer Experience Survey

In order to help us evaluate this course please provide some information about yourself and
your experience with computers. All responses will be kept anonymous.

Please Print

Id # (Last four digits of SSN):

Age,

Sex:___ Male ___ Female

Rank:

Component:____ Active Duty _____ National Guard/Reserve
MOS/CMF:

Years in service:

Highest Education Level (please check one and circle if necessary):
High School diploma or GED
Some College

College Degree (please circle): Associates Bachelors
Some Graduate School
Graduate Degree (please circle): Masters Doctorate

For the following please circle the response that applies to you and provide other
information when necessary.

1. Have you taken the Tactics Certification Course (TCC) before? Yes No Notsure
If Yes, about when? Month. Year.

2. Do you have experience working with computers? Yes No
If so, how long?

3. Have you ever had any training in the use of computers? Yes No
If so, what type of training?.

4. Doyouowna personal computer? Yes No
Ifso, how frequently do you use it and for what?.

5. Do you use a personal computer as part of your duties? Yes No

6. Do you feel comfortable using computers? Yes No



PSI produced less variation in achievement, higher student ratings, and fewer course
withdrawals and that these favorable results occurred across a variety content areas and
course settings.

A recent review by Fletcher (1992) of individualized systems of instruction
showed PSI to be one of the most effective systems employed based on the magnitude of
improvement in student achievement including final exams and end-of-course grades.
PSI was more effective than programmed instruction approaches, audio tutorial,
individual guided education and such computer-oriented approaches as the Individually
Prescribed Instruction/Adaptive Learning Environments Model (ALEM), Program for
Learning in Accordance with Needs (PLAN), optimized instruction, and intelligent
computer-assisted instruction. Only. the strands approach (a CAI application which
allows for individualization of pace, content, sequence, and style in the instruction of
basic component skills) resulted in more effective leaming. However, the data are sparse.
In one study mentioned, Flatter and Atkinson (1973) reported an improvement of .81
standard deviations in achievement for a strands application in beginning reading. In
another study, Ragosta, Holland, & Jamison (1982) reported an average improvement of
.26 standard deviations in achievement for strands applications in mathematics.

PSI: Component Analysis

In addition to assessing the effectiveness of the overall PSI model, researchers
have also looked at the relative contributions of the five PSI components in the learning
process. Buskist, Cush, and DeGrandpre (1991) reviewed the research in this area and
found that some general guidelines could be offered with regard to PSI component
effectiveness. First, courses involving the mastery requirement, immediate performance
feedback, and review units reliably produce high quality student performances. Second.
optional lectures, self-pacing, and the use of student proctors as peer tutors, do not, in
themselves, seem vital to student success in PSI-based courses. Finally, though self-
pacing may not be a critical feature of PSI-based courses, students do seem to appreciate
it. The key is to ensure students manage their course time wisely.



PRELIMINARY EVALUATION OF THE COMPUTER-BASED TACTICS
CERTIFICATION COURSE - PRINCIPLES OF WAR MODULE

Introduction

In December 1995 the TRADOC Deputy Chief of Staff for Training (DCST)
selected the U.S. Army Infantry School (USAIS) at Fort Benning, Georgia as the lead
agency for implementing the WARNET Pilot initiative. The role of WARNET Pilot is
to synchronize the development and delivery of new technologies in training. More
specifically, the intent is to use distance learning and information technologies to pilot the
delivery of high quality, standardized training to soldiers and units where and when they
need it.

WARNET Pilot Team

The WARNET pilot team was formed at the United States Army Infantry Center
(USAIC) under the Directorate of Training (DOT) - USAIS, with the support of the
National Guard Bureau (NGB) to manage multimedia distance learning (MMDL) and
distributive training development for USAIS. Guidance was given to the Infantry School
for development of Total Army Training System (TATS) courseware under the Total
Army School System (TASS) initiative, which was directed by TRADOC. The
WARNET pilot team, in conjunction with the Training Support Center (TSC) under DOT
was given the responsibility for developing Infantry-specific courses. This initiative was
based on specific guidance from HQ, TRADOC TASS OPLAN-2, effective 1 January
1995 (WARNET Pilot Team TATSC Management Plan, February, 1996).

WARNET Pilot Approach to Course Development

The WARNET pilot team and DOT-USAIS have initiated planning in the
following areas: multimedia distance learning, distributive training, and interactive
courseware. As currently envisioned, separate active component (AC) and reserve
component (RC) courseware will be phased out. In its place only Total Army Training
System Courses (TATSC) will remain for seamless training of AC and RC soldiers. To
insure seamless training, TATSC will incorporate various ‘state of the art technologies’,
such as computer-based instruction (CBI) and compact disk read-only-memory
(CD-ROM). These technologies provide the necessary flexibility to accommodate both
AC and RC training needs, scheduling demands, and logistics issues not possible through
approaches currently used, e.g., live classroom lecture (WARNET Pilot Team, 1996).

The WARNET pilot team will develop and coordinate the conversion of RC
courses to TATS courseware to support the TASS objective of establishing decentralized
training at multiple “teaching sites” available to all soldiers. The WARNET pilot
approach to course development involves, in part, the identification of selected infantry
courses for reconfiguration; development of a prototype course; course/courseware



With regard to the quiz given at the end of the module, thirty-seven percent
(37.5%) of the subjects thought the items were somewhat difficult and fifty percent felt
that they were somewhat easy. Eighty-seven percent (87.5%) of the experimental
subjects felt that the quiz adequately assessed their knowledge of the principles of war.
Overall ratings of the instructional value of the Principles of War module were positive.
The percentage breakdown was as follows: Excellent-37.5%, Good-12.5%, Fair-37.5%,
Poor-12.5%.

Additional Comments on Lesson Structure and Content

Comments made by the subjects as they went through the module and their open-
ended responses on the TCC Evaluation Questionnaire were collected and summarized.
Comments were categorized by lesson section or topic area and are presented in the
following sections.

Learning obiective. The learning objective section (screen) was seen as too
generic. The current approach, which involves the presentation of a generic screen for
each lesson referring to “this principle of war” was not helpful to the subjects, who
frequently had to ask what lesson (principle of war) they were in at the time.

Definition. The definition should not incorporate the word or principle that is to
be defined, e.g., FM 100-5 defines mass as “Mass the effects of overwhelming combat
power at the decisive place and time.” This was noted by several subjects for certain
principles of war such as mass and objective.

Doctrinal information. The presentation of doctrinal information was also
criticized by some subjects. They thought that the language and terms used for some of
the information was too complex, unfamiliar, or too complicated.

Battle of the Bulge. Examples were seen by the subjects as too similar to each
other. Subjects did not know which principle was being illustrated at times. Examples
for some principles of war did not clearly demonstrate the principle in question. The
subjects felt that more diversified samples, specific to the principle of war selected, were
needed. Subjects indicated that they were tired of seeing what they felt were basically the
same examples repeated over and over.

React to contact. The scenario used for the react to contact drill was the same for
each principle of war. In the present format, key aspects of the principle of war selected
could be gleaned by the student only from the post-scenario narration which attempted to
connect the scenario to the principle.

Review. Almost every subject skipped the review section on every lesson after

the first or second exposure. Subjects thought the review would be more of a summary
(e.g., a section providing bullet summary points). Its present format, telling students to

15



Tamir. P. (1993). Positive and negative multiple choice items: How different are they?
Studies in Education and Evaluation. 19. 311-325.

Taveggia, T. C. (1976). Personalized instruction: A sum
1967-1974. American Journal of Physics. 24. |

- of comparative research.

WARNE] Pilot Team (1996, February). Total Amv training svstem course
management pian. Fort Benning. GA: U.S. Army Infantry School.

Reproduced From +
Best Available Copy


Research Report 1714

Preliminary Evaluation of the Computer-Based Tactics
Certification Course--Principles of War Module

Robert J. Pleban
Jamye B. Brown
and

Mike T. Martin
U.S. Army Research Institute

Infantry Forces Research Unit
Scott E. Graham, Chief

U.S. Army Research Institute for the Behavioral and Social Sciences
5001 Eisenhower Avenue, Alexandria, Virginia 22333-5600

Office, Deputy Chief of Staff for Personnel

Department of the Army
July1997
umes ed
Army Project Number Training Systems and Education
20363007A793

Approved for public release; distribution is unlimited

il Preceding Pagd! Blank



Figure 2, TCC Principles of War Learning Objective Screen.



Assault Element
os

Touch the symbols to 3
identify =

Figure 5. TCC Principles of War “Objective Dog” Screen.

Design and Procedure

The study employed a randomized two-group, post-test only design. The
independent variable was instruction (CBI vs No CBI).

Subjects arrived at the Multimedia Training Development Division office in pairs.
Prior to their arrival, each pair of subjects was randomly assigned to either the
experimental (CBI) or control (No CBI) group. The researchers introduced themselves
and explained the purpose of the evaluation. Subjects were informed of their role in the
evaluation, what was expected of them, and completed the Background/Computer
Experience Survey.

Subjects in the experimental condition were told that they would go through the
entire Principles of War module, answer all self-assessment items following each lesson,
and take the 27-item multiple-choice quiz at the end of the module. Subjects were also
told that they would work by themselves at their own work station in separate rooms. All
instruction (CBI/CD-ROM) was presented on 486-compatible personal computers. The
module was loaded onto each computer prior to the arrival of the subjects. Each subject
began the Principles of War module on the introduction screen. One researcher was
assigned to each work area to observe the subject and note any problems that the subject
might have going through the module. The researchers tried to be as unobtrusive as
possible. Subjects were encouraged to report their perceptions of the module to the
researcher as they progressed through each of the nine lessons. They could request

10



REPORT DOCUMENTATION PAGE

1, REPORT DATE 2. REPORT TYPE 3. DATES COVERED (from. . . to)
1997. July Final February 1996-June 1997
4. TITLE AND SUBTITLE Sa. CONTRACT OR GRANT NUMBER

Preliminary Evaluation of the Computer-Based Tactics Certification 5b. PROGRAM ELEMENT NUMBER

Course—Principles of War Module 0603007A
6. AUTHOR(S) 5c. PROJECT NUMBER
A793
Robert J. Picban, Jamye B. Brown. and Mike T. Martin 5d. TASK NUMBER
2127
Se. WORK UNIT NUMBER
HO1
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT NUMBER

U.S. Army Research Institute for the Bchavioral and Social Sciences
Infantry Forces Research Unit

P.O. Box 52086

Fort Benning. GA 31995-2086

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. MONITOR ACRONYM

U.S. Army Research Institute for the Behavioral and Social Sciences ARI

5001 Eisenhower Avenue

Alexandria. VA 22333-3600 11. MONITOR REPORT NUMBER

Research Report 1714

12. DISTRIBUTION/AVAILABILITY STATEMENT
Approved for public release: distribution is unlimited.

13. SUPPLEMENTARY NOTES

14. ABSTRACT (Maximum 200 words):

This report describes a portion of the U.S. Army Research Institute for the Behavioral and Social Sciences Infantry Forces
Research Unit’s work in the formative evaluation of the computer-based Tactics Certification Course (TCC)—Principles of
War Module. Sixteen subjects from the U.S. Army Infantry School were randomly assigned to one of two groups. The
experimental group received the computer-based instructional version of the Principles of War module and an end-of-module
quiz. Subjects assigned to the control condition received only the end-of-module quiz. In addition to the quiz, subjects
completed a background/computer experience survey and a questionnaire assessing their opinions on selected aspects of the
Principles of War module. Subjects in the experimental group answered significantly more quiz items correctly (88.9%) than
did subjects in the control condition (48.1%). Ratings of selected aspects of the module varied. Certain sections of the module
clearly needed to be modified. Other sections required only minor refinements. The overall ratings of the instructional value
of the course were positive. The results from this research will be used to refine selected areas of the module.

15. SUBJECT TERMS
Multimedia distance learning Computer-based instruction (CBI) Compact disk read-only-memory (CD-ROM)
WARNET Pilot Formative cvaluation Tactics Certification Course (TCC)

SECURITY CLASSIFICATION OF 419. LIMITATION OF | 20. NUMBER 21. RESPONSIBLE PERSON

ABSTRACT OF PAGES (Name and Telephone Number)
16. REPORT 17. ABSTRACT | 18. THIS PAGE

Unclassified) Unclassified | Unclassified | Unlimited 36



easily. The Self-Assessment section, despite the negative wording of some questions,
was nevertheless seen as useful to the students in reinforcing key points in the lessons.

The results of the formative evaluation process described in this report will help

ensure that the distance learning technologies selected for the TCC are optimally
employed.

19



U.S. ARMY RESEARCH INSTITUTE
FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency Under the Jurisdiction
of the Deputy Chief of Staff for Personnel

EDGAR M. JOHNSON
Director

Technical review by

Thomas Brush, Auburn University
Robert Wisher, ARI

NOTICES

DISTRIBUTION: Primary distribution of this report has been made by ARI. Please address correspondence
concerning distribution of reports to: U.S. Army Research Institute for the Behavioral and Social Sciences,
ATTN: PERI-STP, 5001 Eisenhower Ave., Alexandria, Virginia 22333-5600

FINAL DISPOSITION: This report may be destroyed when it is no longer needed. Please do not retum it to
the U.S. Army Research Institute for the Behavioral and Social Sciences.

NOTE: The findings in this report are not to be construed as an official Department of the Army position,
unless so designated by other authorized documents.



TCC Principles of War Module
Evaluation Questionnaire

1 How interesting was the content of this module?

Very Somewhat Somewhat Very
Interesting Interesting Uninteresting Uninteresting
2. How useful to you was the information presented in this module?
Very Somewhat Somewhat Very

Useful Useful Useless Useless

How useful was each of the following sections for you? Rate each of the
following sections from 1 to 4, with 1 being very useful and 4 being very
useless.

Learning objective

Definitions (of specific principles of war)

Doctrinal information

Battle of the Bulge examples

React to Contact drills

Review

3. How much information presented in the module seemed irrelevant to the
instructional goals stated in the module?

Most information More information More information Most information

seemed irrelevant seemed irrelevant seemed relevant seemed relevant
to me. than relevant tome. than irrelevant to me. to me.

How much irrelevant information did each of the following sections
contain? Rate each of the following sections from 1 to 4, with 1 being mostly
irrelevant information and 4 being mostly relevant information.
Learning objective
Definitions (of specific principles of war)
Doctrinal information

Battle of the Bulge examples

React to Contact Drills

Review

4, How effective were the Battle of the Bulge examples in helping you to
understand the principles of war?

Very Somewhat Somewhat Very
Effective Effective Ineffective Ineffective



Appendix A

Personalized System of Instruction (PSI):
Overview and Empirical Documentation

PSI: An overview. According to Keller, who originated the PSI method, (cited in
Buskist, Cush, & DeGrandpre, 1991), PSI has five essential characteristics: self-pacing.
unit mastery, lectures and demonstrations, written communication, and the use of
proctors. The self-pacing feature enables students to take quizzes in sequence, at a pace
set by the students themselves. This feature is included to accommodate students and
adults with hectic schedules and work priorities. It also allows students to proceed at
their own individual rates based on their own unique study habits and differences in
academic ability. Finally, the self-pace feature gives the student the opportunity to
complete a course ahead of the scheduled completion date, which in many instances can
serve as a powerful reinforcer for students, leaving more time for other courses and
obligations.

Unit mastery. Students must master each unit of study before they are permitted
to proceed to the next unit. The criterion for mastery varies, depending on the course
content and objectives. Criterion ranges have been reported as low as 80 and as high as
95 percent correct. Units are small parcels of information, typically short assignments
based around a few central points. Unit quizzes are generally multiple-choice, fill-in-the
blank, and short answer. To help prepare students for quizzing, study questions which
are aimed at flushing out key concepts are inserted in the unit. An important component
of this principle is that students are permitted to take and retake, without penalty, unit
quizzes until they master the material covered in the unit. If students miss only one or
two questions on a quiz, they may defend or clarify their responses to the proctor. If
students show verbally that they do indeed understand the material, they are considered to
have mastered it, and may move ‘on to the next unit.

Lectures and demonstrations. In the PSI format, the lecture is viewed primarily as
a motivational tool, to be used mainly for its power to inspire students to study further.
Lectures included in PSI courses are typically short, perhaps 30 minutes, and occur only
about once a week. Attendance is not required as subsequent quizzes or tests do not
include material from it. The most important function of the lectures is to clarify or
synthesize textual information and to answer student questions.

Written communication. Almost all communication between a student and the
instructor or proctor is written. Students devote most of their time to reading or viewing
materials provided by the instructor. Quizzes cover only these materials and require that
the student respond by writing a short answer or choosing a multiple-choice item. The
key to this approach is the requirement that the student actively read, study, and respond
to questions over unit materials. To insure students are able to discriminate important
material from the unimportant, detailed behavioral objectives or study questions
(discussed earlier) are often incorporated in the unit.

A-1



were also made to insure that the language/vocabulary in the instruction was familiar to
the subject; that the intent of course objectives/questions were clear; and that the subject
understood the explanations/examples that were presented.

Detailed notes from the observations of the subjects’ behavior were taken.
Particular attention was paid to areas in the instruction in which students were unable to
proceed; comments on the lesson content; difficulties in navigation through the lesson;
areas of instruction that were reviewed or revisited; areas of instruction that were ignored;
and subjects’ performance on the self-assessment items. After completing the lesson, the
subjects were debriefed about their reactions to the instruction. specific computer-based
instructional issues, and key points emerging from the observations. The observations
and feedback were summarized and presented to the course developers for use in making
modifications in the instructional format.

Small group evaluation. Once revisions were made to the prototype TCC lesson
for the principle of war - mass, this corrected lesson was to be used as a template to
design lessons for the remaining eight principles of war. After all lessons were
completed, they were then reviewed by ARI researchers and an instructional design
expert. Feedback was summarized and presented to course developers.

The revised nine lesson TCC module was then administered to a sample of
soldiers representative of the target population. The objective of this phase of the
evaluation process was to determine if the selected instructional approach (modified PSI
using a CBI/CD-ROM format) resulted in significantly better performance on the end-of-
module quiz than no instruction at all. In addition, researchers were also interested in
obtaining the subjects’ perceptions of the instructional approach employed, course
content, and the instructional value of the Principles of War module. The results of this
small group evaluation are reported below.

Method

Subjects

Sixteen subjects were recruited from the United States Army Infantry School to
participate in this phase of the research. Eleven subjects were active duty non-
commissioned officers. Five subjects were Department of the Army civilians. The
average age of the subjects was 39.9 years, SD = 11.6 years, range = 24 - 58 years. All
subjects had completed some college course work. Two subjects were TCC qualified.

Materials

Background/Computer Experience Survey . This paper-and-pencil instrument
was composed of items tapping selected background variables including: age, sex, rank

military occupational specialty (MOS), years in service, and educational level, as well as



Reece Bic ly ; ¥
el ANY “if. v1 % Eady on 16 December 1944, German
api ky, 3 UB infantry. supported by tanks. attacked over
. see ohn the snow-covered ground of the Ardennes,
a Planned German Objectives 3 This sector was held by the American unit
‘Antwerp 2 Py , new to combat and battle weary soldiers.
Kf “2, These units could not have known they
Aw ty faced a breakthrough force of a German
ae a counteroffensive mhich Hitler had
f Cy. \f designed to cross the Meuse River, seize
prusseis f ? | Antwerp, and split the Alfed front. ‘The
russe! Si STIRS anther Battle of the Bulge had begun.
AR
sTUM
‘ We Se
st yinkyy
q
an Cellet »
An ake
ey, a *
‘i > Basfogne é
- ec [BREE Front Line (German Forces}

The Ardennes
tont Line (Allied Forces)

Figure 4. TCC Principles of War React to Contact Screen.