Research Note 85-47

Q Job Skills Education Program:

Semi-Annual Technical Report

I Robert K. Branson and Joan L. James

Florida State University

for

Contracting officer's Representative

Beatrice J. Fanr

Instructional Technology Systems Technical Area

Zita M. Simutis. Chief

Training Research Laboratory

Harold F. O'Neil, Jr., Director WC

C:) U. S. Army

LLJ Research Institute for the Behavioral and Social Sciences

L~. March 1985

Approved for public release; distribution unlimited.

S6 1 2 07
·•· 

THIS DOCUMENT IS BEST 

QUALITY AVAILABLE. THE COPY 

FURNISHED TO DTIC CONTAINED 

A SIGNIFICANT NUMBER OF 

PAGES WHICH DO NOT 

REPRODUCE LEGIBLY~ 
U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency under the Jurisdiction of the

* Deputy Chief of Staff for Personnel

L. NEALE COSBIY

EDGAR M. JOHNSON Colonel, IN

Technical Director Conimandcr

Accession For_

V N CkIS CPA&I

'.rTC TATR

d C

This resport, as submitted by the contr actor, has been cleared Ifcc release to Defeanse Technical Inf ormation Center (DTIC) to comply with regulatory requirements. It has been given no primary distribution other than to DTICe

Service (NTIS). The views. opiniens, and/or findings contained in this report are those of the authoris) and

shnd nill be iabonlthrug asaC orfcl optheri rofernesri AryNattooic.ona the ua dechnnlssnformeigaten

by other official documentation,

Nhudntb:osre sa fiilOateao h Am oiin oiy rdcsouls odsgae
UNCLASSIFIED e

ISECURITY CLASSIFICATION OF THIS PAGE (When 0.t. En..,d)

REPORT DOCUMENTATION PAGE BFRE INTUTI ON

1. REPORT NUMBER E PETSCATALOG NUMBER

ARI Research Note 85-47 7 G1rESINO MCPNT

4. TTLE and ubtile)S. TYPE or REPORT a PERIOD COVERED

JOB SKILLS EDUCATION PROGRAM: SEMI-ANNUAL Interim Report

TECHNCAL RPORTSeptember 1983-September 1984 6. PERFORMING OR1G. REPORT NUMBER

7. AUTHOR(s) D. CONTRACT ORt GRANT NUMBER(S)

*Robert K. Branson & Joan L. James, Florida

Universit

State UnvriyMDA903-82-C-0532

S. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT. PROJECT. TASK

Florida State UniversityARA&ORUNTUMES,.

Center for Educational Technology

Tallahassee, FL 32306 2Q263743A794

11. CONTROLLING OFFICE NAME AND ADDRESS I2. REPORT DATE

*U.S. Army Research Institute for the Behavioral March 1985

*and Social Sciences 13. NUMBER OF PAGES

5001 Eisenhower Avenue, Alexandria, VA 22333-5600 72

14. MONITORING AGENCY NAME & ADDRESS'D1 different from. Controlling Office) 15. SECURITY CLASS. (of CiaE report)

Unclassified

a. DECL ASSI FICATION/ DOWNGRADING SCHEDULE

* IS. DISTRIBUTION STATEMENT (of tis. Report)

Approved for public release; distribution unlimited

17. DISTRIBUTION STATEMENT (of the abstrt entered In Block 20. It diferent hoge Report)

I*. SUPPLEMENTARY NOTES

Beatrice J. Farr, contracting officer's representative. This report, together

with RNs 85-22, 85-37, 85-40, 85-41, 85-42, 85-43, 85-44, 85-45, and 85-46,

forms part of a series on the Army's Job Skills Education Program (JSEP).

19. KEY WORDS (Continue. on reverse side It necessary and Identify by block number)

CBI

Instructional development

* Formative evaluation

* Field trials

20. ABTR ACT IlCwlm e reverse o#b It neceeew awd Identify byblock num~ber)

* -. The Job Skills Education Program (JSEP) is designed to provide soldiers

* writh the prerequisite knowledge and skills required for successfully learning

* their Military occupational Specialties (MOS). When the JSEP is put into effect, it will replace the Army's current Basic Skills Education Program (BSEP)

with a sophisticated, computer-based system.

The second phase of the three-phase JSEP contract involved planning for

the implementation and development of computer-based instruction "(Continued)

DD ,' 1473 EDITION Of I NOV 65 IS OBSOLETE UCASFE

1SECURITY CLASSIFICATION OF THIS PAGE (When flat. Entered)
SECUR Y CLASSIFICATION OF THIS PAOE(Whs, Data Enome)

ARI Research Note 85-47

20. \(Continued)

(CBI), and a field trial of that instruction. The results of the field

trials and discussions of methods for management and development are contained in this report. (-

UNCLASSIFIED

SECURITY CLASSIFICATION OF THIS PAOE(Wh7In Date Entered)

*%-' ," ---

. . . * . . % . . .% . N -* ~ -- - . ... -* .- * **~~4 . ... v*.
i '

FOREWORD

The Job Skills Education Program (JSEP) is a multi-phase program begun in

Fiscal Year 1982, and designed to enhance enlisted career potential by %

improving soldier job performance. The sponsor, the Education Division, 6-*%'0

Office of the Deputy Chief of Staff for Personnel, expects JSEP to replace

the Army's current Basic Skills Education Program when it is implemented.

The JSEP program, being developed by Florida State University (FSU) will

result in a standardized curriculum for soldiers who demonstrate deficiencies

in the knowledge and skills required to successfully learn their Military ..

Occupational Specialty (MOS).

In accordance with current policy, JSEP will be an on-duty program. It will

also use a computer-based management system to facilitate an open entry/open

exit approach. At present, most of the lessons being developed will be

computer delivered; however, the plan calls for using existing materials, and

incorporating materials developed as part of other ARI efforts, whenever

appropriate.

A unique aspect of JSEP is that it builds upon a very detailed front-end

analysis of MOS Baseline Skills. The analysis covered tasks performed by

soldiers in the 94 highest density MOSs, in addition to Common Tasks (the

skills that all soldiers, regardless of their MOS, need to know). Although the

Army has over 300 MOSs, the 94 covered in the analysis represent about 80% of

all soldiers. Perhaps the most useful product developed for the analysis was

a taxonomy listing more than 200 prerequisite competencies.(P.C.) for these

MOSs. The competencies were derived from detailed reviews of Soldier Manuals,

and from extensive interviews with subject-matter experts at Army schools.

This effort produced a series of tests intended to diagnose deficiencies in

the P.C.s. Modified versions of these tests will be used in JSEP.

The JSEP program will include a front-end learning strategies module

designed to improve soldier skills in reading, studying, test taking, and

problem solving. The curriculum will consist of this strategies-training, plus

180 diagnostic review lessons, and 120 skill development lessons, which are

being developed for the PLATO and MicroTICCIT computer systems. The program is

being tried out at two TRADOC sites and two FORSCOM sites, prior to an Army-wide

phased implementation.

iii r

S.. . . .. . . . . . . . . . . . ... . . ... . . . . . .
, - - wviwZ. .I~~vLwL - ~,-V~~t~~'*, LL 

' SEMI-ANNUAL TECHNICAL REPORT

EXECUTIVE SUMMARY '*.

Requirement:

The solicitation requires that a semi-annual technical report be submitted

for Phase 11 to the Army Research Institute (ARI).

Procedure:

This report constitutes the official report of technical and develomental

processes undertaken in the first six months of Phase II of the Job Skills

Education Program (JSEP).

Findings:

Specific technical aspects are detailed for:

0 Project Management

o Curriculum Design

o Lesson Design and Programming ',

o Data Management and Student Routing

o Testing Procedures

0 Tryouts and Field Trials

0 Implications for Demonstration Year and Implementation

Use of Findings:

The processes reported here form the foundation for the Tasks for Phase II

and Phase I1.

F o

..................... ............... ... ...... .... .... - --. .....
SEMI-ANNUAL TECHNICAL REPORT

CONTENTS ,.

Page ,.

OVERVIEW............................................................ 1

Operational Problem ...................................................1

Research Objective .................................................... 1

A. PROJECT MANAGEMENT .................................................... 3

Computer Applications ................................................. 3

B. CURRICULUM DESIGN................................................. 4

Priority for Long Lessons ............................................. 4 b -"-'.

C. LESSON DESIGN AND PROGRAMMING .................................... 7

Lesson Design .................................................... 7

Learrier :tratieS"' .. ........................................ ". "-"

Hardware Choices ...................................................... E

Standard on Production of Lessons on MicroTICCIT ...................... 9

D. DATA MANAGEMENT AND STUDENT ROUTING ................................... 9

Comments about the JSEP Data Management System ........................ 9

Potential Routing Mechanisms .......................................... 10

E. TESTING PROCEDURES .................................................... 14

General Performance Measures ..................... .................... 14

Use of Job Performance Measures to Test Effect of JSEP .............. 17

G. TRYOUTS AND FIELD TRIALS ....................................... 17

The Population ........................................................ 17

Effectiveness of Lessons .............................................. 19

Student Response to Screen Formats ............................... 22

Results ............................................................... 22

Summary Conclusions ........................................... 27

H. DEMONSTRATION YEAR .................................................... 28 .

I. IMPLEMENTATION ............................................... 26

Statement of Work for JSEP Coordinator .............................. 30

vii

+........ ...... ........ A .. .- .".. . . ., . -..- . -.- .-.. .'...; ---" '-, ,.--------
APPENDIX A Summary of Priority Data..................... 33

j LIST OF TABLES

Table 1. Intercorrelations among predictor variables and performance

measures .. o ........... .* . .. ................... 1

2. Correlations of potential predictors of lessons pretest

scores..................................................... 12

3. General performance measures................................ 15

4. Sample background characteristics of*fort Rucker tryout ....... 28

15. Cumulative percent of soldiers scoring at percent level

or higher on pretest and posttest ........................... 19

6. Attitudes toward JSEP ...................................... 21

II
*

o 

•

* SE14I-ANNUAL TECHNICAL REPORT L

OVERVIEW

Operational Problem

It is not news that soldiers must be trained to do their jobs. They must

be trained so that each Army job is performed competently--regardless of

differences in ability and background in newly entering soldiers. To accept

less would cause many mission elements to fail. .

Moreover, many Army jobs are increasingly dependent upon the soldier's

ability to use high technology and the ability to learn new technology as it

develops. Soldiers, therefore, need more than training. They need enough

education to be able to learn subsequent jobs, to become eligible for

promotion, and ultimately, to provide leadership for tomorrow's Army.

The Job Skills Education Program (JSEP) is designed to provide soldiers

with job-related basic skills instruction that is prerequisite to learning

their skill level 1 and 2 job tasks during their first duty assignment. Based

on an extensive job analysis of 94 of the Military Occupational Specialties

(MOS) which contain the largest proportion of soldiers and tasks contained in

" the Soldier's Manual of Common Tasks, JSEP provides functional basic skills

instruction on MOS specific requirements.

* As it is corceptualize, the j3EP curriculum reLognizes Lnd.L Tne vast

majority of soldiers will have been exposed to similar basic skills

instruction before entering the Army. Many entering soldiers, however, will

not have learned those basic skills well enough, or will not remember what

they learned. To help soldiers learn better and remember more, JSEP

incorporates straightforward training in research-based learning strategies

that are directly aimed at improving learning and retention.

Research Objective

The goal of the US Army Job Skills Education Program (JSEP) is to design,

develop, and test a job-related computer based curriculum for possible Armywide adoption. The JSEP development project is concerned with four central

elements:

1. The computer based curriculum,

2. The soldier management plan,

3. The soldier testing program, and _

4. The supporting documents and reports.

The project is divided into three phases. Phase I undertook several tasks

which included lesson design, planning documents, and reports that recommend

specific courses of action. Many of the action plans spelled out in the Phase

1i?:
reports will be carried out in Phases 11 and 111; some will be carried out in

the years following Phase Ill.

Phase I contains tlefollowing tasks:

Task 1. Develop a conceptual definition for design

requirements and specifications

Task 2. Review existing basic skills curricula.

Task 3. Review job task analysis and select Military P Occupation Specialty (MOS) clustering schema.

Task 4. Develop JSEP implementation and management

plan.

Task 5. Adapt or develop design specifications

including instructional specification,

engineering requirements, and human factors

consideration.

Task 6. Develop evaluation standards.

Task 7. Develop JSEP evaluation plan.

Task 8. Develop cost benefit tradeoff analysis.

IasK i. tonouct preoictive cost ano training

effectiveness analysis. . -

Task 10. Develop standards and a plan for the civilian

academic community to award a high school

diploma based on completion of JSEP.

Task 11. Phase I Report.

Phase II requires the development of 300 hours of lessons, the soldier

management plan, and the preliminary tryout. Phase 11 includes the following

tasks:

Task 12. Select hardware; where appropriate develop

software; develop instructor training program

and courseware for 300 hours of instruction.

Task 13. Conduct preliminary tryout.

. Task 14. Phase II Report.

Phase Ill provides a continuation of the lesson development begun in Phase

II and requires execution of the studies and evaluation plans developed in

Phase . Phase Ill includes the following tasks:

Task 15. Develop software and courseware for an - additional 120 hours of instruction.

I .. 2 , ,
-T..%,-

Task 16. Conduct JSEP tryout at two US Army Forces k

Command (FORSCOM) sites and two US Army

Training and Doctrine Command (TRADOC) sites.

Task 17. Conduct evaluation and revision.

Task 18. Conduct cost/training effectiveness analysis. .

Task 19. Develop technology transfer plan.

Task 20. Document total process in final report.

The JSEP Semi-annual Technical Report is a collection of narratives which

cover the essential issues of Phase I and the first half of Phase 11 of the

contract. These issues have been grouped in nine topic areas. The number of

issues in each area varies. The topic areas are:

A. Project Management

B. Curriculum Design

C. Lesson Design and Programming

D. Data Management and Student Routing

E. Testing Procedures

G. Tryouts and Field Trials

H. Demonstration Year

I. Implementation

The narrative for each issue will describe the original plan, the current or

implemented plan, and the reason for any discrepancies or changes.

A. PROJECT MANAGEMENT

Computer Applications

Tracking and coordination of a major project is essential to timely

completion. Originally the critical path method and GANTT charting were

proposed as the main management tracking technology. While these methods have

proved useful, they have not proved sufficient. They do not provide the

detailed information necessary to take corrective action.

Each prerequisite competency must be tracked through its individual design

stages in order to get a comprehensive overview of the project. Also budget

projections and budget status are a critical issue with a large staff.

A nationwide search was conducted for computer software that was designed

for use in instructional development projects. Failure to find a suitable

3 -'1 

-- • . .. . . .~ -' - -. '' . • . . .• "..' . . . * . - -. . .. .*. .
" .' lackage led us to design our own system based on some of the available

*- business/management software. Our preliminary attempt was with an Apple

computer and the Visi-series. We used the Visidex program to track the

. individual PC specificatibns produced in Phase 1, but it was cumbersome.

*' VisiCalc did not seem to have enough power to handle our budget tracking

procedures. Our second, and more successful efforts were made with an IBM-PC

and the Lotus 1-2-3 program. With the addition of 128K memory, Lotus and the

IBM gave us the power to create two interacting spreadsheets which encumbered

and projected salaries and expenses. Lotus was also used to design a PC

tracking system. Lotus has a Macro programming function which allowed us to

update the PC tracking system and to print out various versions of" the data for

monthly reports and weekly monitoring with a minimum of reformatting.

The data base functions of Lotus have allowed us to make sequential

entries of purchase orders and charges for auxiliary functions and then sort

the lists by charge designation, which greatly facilitates spreadsheet updates.

We recommend that other large ISD projects explore similar options.

B. CURRICULUM DESIGN

Priority for Long Lessons

Setting PC Lesson Development Priorities

Developing detailed instruction for all PCs would result in far more than

tne approxnTea'ly '+u nours requireo unoer Tnis contracL. nnereas sue FC:

will require only about an hour to complete, some will take far more, such as

36f, Apply common rules of grammar, which will probably require mary hours of

instruction and practice. Short lessons will be developed for approximately

170 testable PCs. (See Table 7.) In an attempt to establish priorities for

long lesson development, we recommended a method for selecting those PCs to be

developed first.

We propose that the factors listed below be taken into account and lessons

tentatively rated high, medium, or low priority (see Appendix A):

o How many MOS require the PC?

o What is the density of the MOS which require the PC?

o Are there common tasks which require the PC?

o Are other PCs dependent on the PC?

o Are there compatible job-related existing materials that can

be integrated into JSEP for the PC?

o How did the TRADOC schools rate the PC?

o Is the PC related to the General Technical (GT) portion of the

ASVAB?

o FORSCOM Priority rating.

4

. .* ..
... :. .,... !

o Is it appropriate for high school credentlaling program?

Number of MOS. In order to answer the first question, the RCA MOS-PC

Matrices forTSEP I and BSEP II were consolidated. The number of MOS were

counted for each PC (see Table 8).

For example, for the 1 PC series except li, there were so many MOS for

each PC that there was too little additional information to be gained looking

at the rest of the questions. This PC was clearly needed. PC la through lh *'

were tentatively designated high priority. PC Ii has only 35 MOS and will be

classified medium priority.

MOS density. Most MOS were medium density, but there are two with very

high density, 11C with about 12,000 soldiers and 76Y with about 19,000.

Therefore, 18a is a medium priority. For 18b and 18c, all of the MOS have

relatively fewer soldiers, therefore 18b and 18c will have a lower priority.

Common tasks. PC series la - Ii (except ig) all have common tasks. PC la

through Ih are already high priority, so no reconsideration will be made. PC

ii, however, had previously been classified medium priority and would probably

be reclassified high priority.

Dependent PCs. Some PCs are prerequisite to other PCs. If a PC is

prerequisite to a high priority PC, it too, must become a high priority. PC

Ic, 1d, and le are prerequisite to If. If if is a high priority, all three,

Ic. id, and le, should become high priority.

TRADOC priority. TRADOC surveyed the proponent service schools to obtain

priorities for all JSEP MOS. The schools rated each PC for each MOS as:

A PC already taught in Advanced Individual Training (AIT) yK>

I Unanimous agreement priority one PC

2 Mixed responses mostly positive

3 Mixed responses mostly negative

The schools further rated each PC as to whether it was difficult for soldiers

who lacked the competency to learn their jobs. An A, 1, or high difficulty

rating increases the FSU priority rating. -.- ,U.

GT relationship. If a PC is GT related it automatically becomes high

prior-"ly. A ti. is G1 related if the PC matches the skills tested on the GT.

Existing materials. A PC could be reduced in FSU development priority if

JSEP compa i IC existing materials were available, If McFann-Gray's (MGA)

lesson "Introduction to Numbers" matched the indicator statements for PC la

then PC la would be taught using the McFann-Gray lesson. The MGA materials are ;-4

primarily drill and practice. They may be programmed on one or both of the

computer systems used to deliver JSEP.

Level of difficulty. RCA recommended developing instruction at two

difficulty levels. Since there is not yet any data on what would be difficult

for the target audience, FSU recommends a short lesson on each PC of the

approximately 170 PCs that Educational Testing Service (ETS) declared testable.

Long lessons.should then be developed for higher priority testable PCs.

5

"~~. .. .. '. .......... ......... .. .. ...........- .. _..... . ........-............ * ***
aX7.R -a

Rasin g l technical a titude area (GT) scores. One goal of BSEP,

BSEP 11, and - pr.o~rams sincelas-1-'ofPhase I is to increase the .. ,.,

chances that soldiers taking the Instruction could prepare themselves to be

eligible for reenlistment. When soldiers are eligible for reenlistment, based

on their test scores, the Army can choose from a much larger population those

to recommend for reenlistment based on their job performance and service

records.

Identifying soldiers who need to raise GT scores should be a part of the

JSEP management system. JSEP soldiers with GT scores within the range of

eligibility can go through the PCs that match the skills tested on the GT

(arithmetic reasoning, paragraph comprehension, and word knowledge).

The proposed long lesson priority is based on numbers of soldiers

requiring the lesson, the RCA analysis of ETS .Test results, the TRADOC study of

difficulty and priority, and the GT relatedness of the lesson.

0 The number of soldiers criteria gave heavy weight to the

lessons on common task related PCs. High density MOS

such as 11B or lessons required by more than 40 MOS added

weight to non-common task PCs.

0 RCAs analysis was used to identify those lessons on which

more than 50% of soldiers failed the ETS Test.

0 o The TRADOC study collected opinion data from instructors

and for each PC on job difficulty, for each MOS JSEP

priority, and whether the PC was already covered in the

school.

o Lessons related to GT improvement also received a higher

priority.

FSU recommends a decision based on a weighting of those four criteria with

the following factors in order of importance.

1. Common task PCs

2. ETS Test data

3. GT improvement

4. High number of soldiers impacted

Use of the RCA PC - MOS Matrix. The RCA PC-MOS matrix was divided into 2

parts.- E "and T-S SF 5e consolidated the data and came up with a JSEP

PC-MOS matrix. We planned to use the matrix to help decide lesson development

priorities. -

S

The matrix provides an excellent quick look at PC series to see how many

MOS require it. Although there are some bugs in the data tape, the information

has been very useful. We recommend a similar analysis for future curriculum

*: development efforts. A summary of the priorities is shown in Appendix A.

.- ,..

IIN

6 .

-.- - - -a•.:::%,§:. .- 
C. LESSON DESIGN AND PROGRAMMING

Lesson Design

One of the initial lesson design problems faced by the JSEP staff was

deciding upon the sequence in which lessons would be developed in the JSEP

curriculum. Our responsibility is to produce 420 hours of instruction, but

what would be the priority order in which lessons would be developed for the

RCA prerequisite competencies?

Various proposals for setting priorities were discussed and data from

various sources were reviewed. During the discussion of the various methods,

the observation was made that, for any particular PC, there are two broad categories of soldiers who, at first, would appear to require instruction on

the PC. The first category is composed of those soldiers who could not

correctly answer the test items for the PC, but with only a slight review would

recall prior learning and be able to perform quite well. A second broad

category of soldiers includes those who had never learned the skill and

probably lacked one or more important subordinate skills.

From this set of observations, the concept of short and long lessons

evolved. In general, the purpose was to first design very brief lessons (the

short lessons) which would quickly bring up to speed those soldiers who had

previously learned but forgotten the skills. Such lessons would take very

little time of those who had never learned the skill. The second set of

lessons (the long lessons) were to serve those soldiers who required in-depth

instruction in order to produce their first successful performance of the

skl ".

The lessons are designed as follows:

a. Short lessons. Almost all of the PC's can be considered to be

(in Gagne's terminology) intellectual skills. For the short lessons,

the skill is described, and an example of the application of the skill is

demonstrated. Then the learner is provided a series of representative

practice and feedback items. The feedback for incorrect responses is

sufficiently detailed to clarify misconceptions and bring student

performance for those who had previously mastered the skill, to an

acceptable level. The short lessons include direct instruction on only

the skill described in the PC, and not on any of the subordinate skills

associated with the PC.

b. Long lessons. The long lessons are intended for soldiers who didn't

pass the test for the short lesson on the PC and who probably lack one or

more of the relevant subordinate skills. A long lesson includes both a

number of subordinate skills and a fuller elaboration of the events of

instruction for each skill.

To develop a long lesson, the designer first does a hierarchical

analysis to identify potential subordinate skills. Then, for each skill

which is a potential problem for the soldier, a complete set of

instruction is provided which includes motivational pre-instructional

activities, a complete description of the skill and an appropriate range

of examples. An ample number of practice and feedback problems are

provided, along with alternate remedial instruction where applicable. The

7

'-':-:-':-'- • ~~~~~~~~~~~...... . .. . :. ... . -. ".". . ' *' .. -.-.-.. .. .... . ....... ................ ... :-.. . .- . .. . . .. . . . .
Itudent should master each successive subordinate skill before returning

to instruction on the PC itself. The intent is to provide sufficiently , extensive instruction, including appropriate practice and feedback, that

all soldiers will be able to perform the skill.

The testing strategies which will be used to route soldiers into and out

of the short and long lessons are still evolving. The basic principle is that

the soldier who mostly "knows" the skill but is diagnosed into the instruction

will exit rapidly after completing the short lesson, while the soldier who

really requires the instruction will receive it, in depth, in the long lessons.

Learner Strategies

During Phase I we conducted an extensive review of the literature on

various learner strategies that seemed to be well-established and would have a .-

potential value in the JSEP curriculum. Based bn this review and advice from

our consultants, Barbara McCombs, Donald Dansereau, and Richard Mayer, we

planned a six-module learner strategies package.

The rationale for our approach to strategy training is outlined in a paper

by Derry entitled Strategy Training: An Incidental Learning Model for CAI that

was presented in April 1 e annul meeting o[ the.American EducatInal

Research Association. We are teaching five types of learner strategies in

JSEP: (1) mood management techniques, (2) self-pacing methods, (3)

comprehension strategies, (4) memory strategies, and (5) problem-solving

techniques. Also, we plan to extend the problem-solving techniques to a

SPE Ipc~ CL1W41t 1 test-taki rig sk-1

Hardware Choices

There were many hardware options initially discussed and considered for

JSEP. Ultimately, the hardware options decided on were those which would

provide the most necessary capabilities across the largest number of lessons.

Hardware options adopted for MicroTICCIT include the standard MicroTICCIT

workstation, which includes light pen, detached keyboard, and screen printer.

Hardware options adopted for PLATO include the current VIKING terminals with

touch panel and detached keyboard, screen printer, and graphics tablet.

Hardware options which were considered but not adopted include videotape,

speech generation, sound generation, use of slides/film, realtime actual

equipment interface, graphic input, and use of optical character/mark sense

readers.

Some hardware options might still be desirable for inclusion in JSEP at a

later date. For example, we would like to include some sounds in JSEP lessons

for reinforcement and motivation. It would be desirable to include interesting

sounds in the form of beeps, buzzes, and sound effects (e.g. airplanes

crashing, truck starting, bomb exploding, etc.). At the present time these

capabilities are not practical for either the PLATO or MicroTICCIT systems.

""' "." "'-',";"-'-'-- '--'- -'-';................ '-.''-'-,.........-......-...'-..".........,..-.........,"...-

'
- Standard on Production of Lessons on MicroTICCIT

During most of Phase II production, we have used the TICCIT Authoring

Language (TAL). We made the decision to use TAL because it gave us the

following capabilities.

1. Freedom to write Interactive Rules.

* 2. Direct conversion to ADAPT. When ADAPT is released we will

convert all existing TAL programs and complete all production

in the new language.

3. Flexibility in lesson content and structure.

The alternative to TAL, the Authoring Procedure for TICCIT (APT), is a

highly structured authoring environment which presently prevents interactive,

rules, direct conversion with ADAPT programming code, and multiple lesson

formats.

ADAPT is designed to be The language for MicroTICCIT, and will be released

shortly. We will use ADAPT because it combines the efficiency in production of

APT, derived from its templated-formats, with the power and flexibility of TAL.

ADAPT also has new features that run on MicroTICCIT such as color palette, and

video.

Because our current production system does not have any MicroTICCIT

terminals we have not used ADAPT's new features. However, shortly after we

w-,tai6 1D;evr&".&M2 ;evera r rcTO, sc , , prou, -

system will match the delivery system in all important respects.

D. DATA MANAGEMENT AND STUDENT ROUTING

Comments About the JSEP Data Management System (DMS)

As of this writing, the DMS exists as an emergent set of assumptions and

interrelated design decisions. During the past year, our plans for the DMS

have had to be modified to accommodate the additional goal of GT improvement.

*" We have also recently re-examined our selection and routing criteria because of

the poor match between the RCA Locator and Diagnostic tests and our PC

objectives. No final decisions have been made regarding the new criteria.

The DMS will be the "control center" for JSEP and will collect demographic

data and lessons performance data and will use those data to provide routing

options to soldiers. The system will be flexible enough to account for the

needs of different sizes of Army bases. At large bases, the site managers and

' instructor will be allowed control over the DMS at specific input/output

points. At small bases, the DMS will be able to function without any

instructor involvement.

9

* . * '. -- ** . A t ;. 
*, ''Regardless of the special conditions at each site, we recommend that the ,.'.-

DMS operate roughly as follows:

1. The DMS will iollect demographic data about soldiers. This

will include scores on the GT. '

2. The system will perform a front-end analysis to determine

each soldiers' recommended program of study. All data to be

used in this analysis has not yet been identified.

3. Soldiers will obtain a copy of their program of study and

discuss it with their instructor.

4. They will then proceed through their program of study,

controlling their own pace and sequence of lessons. The

instructor will be involved as necessary. Soldiers will

report all completions to the instructor to update an offline record of their progress.

5. As soldiers proceed through their program of study, the DMS

will collect and make available to instructors performance

data ranging from test scores to time in each lesson.

Potential Routing Mechanisms

Four independent measures were explored as potential screening mechanisms

to neip soldiers icentity lessons in wnicn tney nave a nign IlKeinooo OT

requiring instruction. (See correlation matrix on Table 1.) These were the:

(1) JSEP Achievement Test used as a pretest,

(2) GT scores,

(3) RCA Locator Verbal, and

(4) RCA Locator Math.

These measures were correlated with pretest scores of the seventeen lessons

used in the tryout. It was hoped that one of these measures would demonstrate

consistent positive significant correlations across a majority of the lessons.

If this had been the case, then scores on the screening measure could

predict whether a soldier would likely require instruction in the lessons. The

results of the correlational analyses are presented in Table 2.

10

•....................................-...... ...... .... ... .......... , ...... ........ . . ..... . . .-.............. .....-.-.- , .............- . .. .,-,..,
Tabl e 1

Intercorrelations Among Predictor Variables and Performance Measures

1 2 3 4 5 6 7 8

1. Locator Verbal --- .06 -.32* .04 .03 -.03 -.10 -.03

2. Locator Math --- .02 .08 -.33* .02 .03 -.04

3. GT --- -.08 .03 •43* 37* .28*

4. JSEP Achievement Pretest --- -.11 -.33* -.23 -.05

5. JSEP Achievement Posttest --- .17 .17 .06

6. RCA Pretest --- .78* .37*

7. RCA Performance Posttest --- .16

8. CLOZE -- -

Sp < .05

.. '-
• Table 2

* Correlations of Potential Predictors of Lessons Pretest Scores

Lesson Pretests JSEP Achievement GT Locator Locator

Pretest Verbal Math

1. 30b Flowchart .01 .24 .19 .40*

(n = 39)

2. 32a Blocks in a Form -.13 .39* -.25 -.14

(n = 39)

3. 32b Forms -.16 -.02 .56* .19

(n = 37)

4. 32c Forms .00 -.41* -.23 -.39 (n = 17) .'-

5. 32d Forms -.02 .22 -.24 -.14

(n 28)

6. 32e Forms .00 -.03 -.14 .35*

(n = 29)

1 4a A .'l c . an ..-.. 33. 

Clauses

(n = 39)

8. 36c Capi tal ization -.12 .31 .04 -.15

(n = 29)

--p. ,'

9. 40a Hazards -.38* .12 .10 .18 Ir.

(n = 37)

10. 40b Preventive Measures -.25 -.03 -.12 -.20

(n = 39)

11. 40c Emergencies -.27* -.04 .05 .04

(n = 39)

12. Olg Rounding Numbers -.19 .35* .10 -.05

(n -39)

13. 5f Gages -.03 -.01 .36 -.13

(n : 17)

14. 26b Technical Vocabulary -.G8 .21 -.23 -.12

(n - 39)

15. 28a Tables and Charts .36* .01 .16 -.55-

(n 15)

12

~ .k .. *. -
- , . **-..* p **. .. . .. * -. 4 S -

Table 2 Continued

Lesson Pretests JSEP Achievement GT Locator Locator

Pretest Verbal Math

16. 28b Tables and Charts -.05 .33 .14 -.32 -. '..

(n = 18)

17. 28c Tables and Charts -.25 .15 .02 .08,:,

(n = 17)

Mean Correlation

across all lessons -.08 .08 .02 -.05

* p < .0 5

The results of the correlational analysis indicate the mean correlation

between the potential routing measures and the lessons were: JSEP Achievement

Pretest, M = -. 08; GT, M = .08; Locator Verbal, M = .02 and Locator Math, M =

-. 05. The correlations between predictor measures and lesson pretest measures

virtually cluscered around zero. We concluded that no single score from any of

these measures is likely tc cons _ tertly predict pnrforar 'ne across lesson

pretests. The potential use of any of these as screening devices appears

remote.

This lack of relationship between general measures and the more specific lesson pretests could be attributed to several sources:

1. The unreliability of the 10-15 item lesson tests could

attenuate the correlations to levels below significance in most lessons.

2. Since the distributions of scores of the pretests for many

of the lessons were so highly skewed toward the positive

side, correlations were consequently suppressed due to

non-linear relationships between variables.

3. Truncated ranges in the general ability measures would

reduce the potential for correlation with lesson pretests.

GT scores 80-99 represent the 3rd quantile in the

distribution of measures of general intelligence.

4. One would not expect the JSEP test, comprised of 3 items

randomly selected from 11 discrete knowledge units, to

predict mastery of a single knowledge unit. There is only

1/11 of the achievement test domain in common with any

single pretest.

13

............. . ,-. .-. .. '. . .. .,.,.,.. ,. .. . . . . ,',.- . . . ... . ., . - -. .--.- , '... ."., "." ,- -.
-'The Implications of these findings regarding the use of a single routing

mechanism may include the following:

a. Administer these general prediction measures in the Full Scale tryout

to validate the above results on a wider array of lessons, but do not

use them as either screening or routing mechanisms in the tryout.

b. Conduct an item analysis on the lesson posttests using another sample of soldiers to possibly improve the -tests items and hence their

reliabilities.

c. Explore the possibility of clustering lessons on the basis of

underlying cognitive ability factors and then using the general

measures to predict to these ability factors. These factors could be

identified possibly through factor analytic procedures or other

computer sorting methodologies. Again, recommendations pertaining to

the use of such procedures in the implementation phase would be made

from additional data gathered from the Full Scale Tryout.

E. TESTING PROCEDURES

General Performance Measures

Several general performance measures were administered in the Fort Rucker

Tryout to assess the transfer of specific objectives learned in the lessons to

more general cognitive abilities:

I. JSEP Achievement Test

2. RCA Performance Test

3. CLOZE Test

4. GT (ASVAB General Technical)

The JSEP Achievement Test and the RCA Performance Test were administered in a

pre - post format. The CLOZE Test was administered in only a posttest format

and the GT scores were simply acquired from soldier records. Data pertaining

to the instruments are presented in Table 3.

......

14 " " "

--- ."..•..... , .... , ...........-..•.....
Table 3 . ,.

General Performance Measures

Measure Pretest Posttest Effect size

Mean SD Mean SD

1. JSEP Achievement Test 28.6 3.4 33.3 2.6 1.38

(39 possible)

2. RCA Performance Test 48.7 11.8 49.1 13.5 .03 --

(79 possible)

3. CLOZE ---- 14.8 6.0 ----

(35 possible)

4. GT 91.2 4.9 - ---- -

The JSEP Achievement Test

The JSEP Achievement Test was developed by selecting at random three items

or tasks from the posttests of 11 lessons. Further development of the test

would render the instrument a sensitive, reliable, and valid proximal measure

RCA Performance Test

The items selected for the performance test were drawn from the RCA

Criterion Test. These items were generally consistent with the contenf-of the

E.P)esson but not always exactly so. In many instances the complexity of

the stimuli was greater in the Performance Test than in the items of the lesson

posttests. For example, the number of process/decision steps in the lesson on

flowcharting was 3-5 (as many as could be presented on a display screen) whereas the Performance Test contained 15-20 such steps. Items with electronic

circuitry and rounding numbers tasks were also more complex than material

presented in the computer-based lessons. Thus, there was an increment not only .-

in difficulty from the nature of the assessment task (recognition to recall) but also in terms of the complexity of the stimulus function.

Therefore, in order to construct a more sensitive and valid recall

measure, we should begin from our own JSEP Achievement Test as a point of

departure. From the pool of JSEP Achievement Test items we could perform the

following:

1. Select one JSEP Test item from each lesson comprising

the set of GT prep lessons and the set of MOS common task

lessons. Two Performance Tests would then be created, one for

MOS Common Tasks and one for the GT preparatory course.

15 4
2. The selection would be based on such considerations as

resemblance of the stimulus context to on-the-job situations

and the ease and objectivity of scoring the response.

CLOZE Test

The CLOZE procedure was used as a test because it is intended to measure

reading comprehension and in this case to assess the ability to read the

soldier's field manual. The CLOZE Test used in the preliminary tryout was

created by selecting a 300-word .pragraph with accompanying diagrams from the

Manual of Common Tasks. Every eighth word was omitted from the paragraph. In

all, thirty-five words were omitted from the paragraph. The content of the

passage concerned procedures for the proper aiming of a rifle.

The results of the trial test of the CLOZE. procedure indicated that the

group of soldiers attained a mean of 14.8 with a standard deviation of 6.0. The

scores were normally distributed about the mean. Only one soldier was unable

to begin the task.

While the CLOZE procedure is designed to measure a general ability to

comprehend written prose, it is questionable whether it could or would be an

appropriate outcome measure for the JSEP curriculum. The JSEP curriculum is

structured in terms of many narrowly defined skill and content units that can

be learned in 20 minutes to 1 hour. The question is raised as to whether

learning from specific learning units will be integrated and transfer to more

general abilities, such as comprehending a paragraph of prose in a field

manua."

Secondly, the nature of the passage that was selected was very different

from most of the written material typically found in the FM's. The more

familiar prose in the manual is in the form of stepwise lists of procedures,

diagrams, and flowcharts to conduct a given task.

For these reasons, it is probably unwise to use the CLOZE procedure as a "V

JSEP outcome measure to assess the ability to comprehend FMs.

The GT

One of the purposes of JSEP is to assist soldiers in achieving a better

performance on the GT (General Technical Subtest of the ASVAB). Soldiers were

selected for the Tryout on the basis of their GT scores--that is, they earned

GT scores between 80 and 99. Thus the GT score distribution of M = 91.2, SD =

4.9 differed from the Army wide distribution M = 100, SD = 20.0. Given the GT

score distribution of the expected JSEP population, in subsequent field trials

we will likely observe effects of JSEP instruction not only in terms of changes

in the mean, but also in terms of changes in the distribution of scores about

the mean.

I. 16

%•-~

* - , . . .*. .2 * , . % % ",*% -. ..* *-. " % *-* ' . . " - - . , °-.° . . -- , . "." " ." ... " - .. %*~ . " . " .

.- :.._: '-X .''.-'..''..'.-..L . .i' -'. ;L',_, -/..,, ." .",,'." ".- - ,'m" ." " ,""." " "" "',"- "" " " '" """"." " ,""".'.'. h,., " 1" .. ,X ,"
- .1

Use of Job Performance Measures to Test Effect of JSEP

Use of job performance measures was originally proposed for JSEP

- evaluation; however, there are certain technical difficulties.

JSEP could be evaluated in a longitudinal study provided that there were

enough implementation sites to have sufficient soldiers to study. A

longitudinal study of this magnitude could be implemented by Education

Division, ODCSPER, TRADOC, or FORSCOM. It would take three to four years.

SQT's and other measures, as yet not fully identified, could be used as the . -.

criterion variable for correlations.

G. TRYOUTS AND FIELD TRIALS

The JSEP Field Trials were conducted at Fort Rucker, 5-18 May 1984.

The Population

- By directive, we requested that soldiers be included in the tryout

* " population who met two conditions:

(1) they had earned GT (ASVAB General-Technical) scores between 80 and 99,

and

(2) for the most part they had been recently stationed at Fort Rucker.

The MOS represented in the tryout were 67 (N, V, Y), 76 (N, W, Y), 91B,

and 94B. Background data in terms of age, rank, sex race, time in the Army and

time on the post are presented in Table 4 below. Generally speaking, the 42

soldiers participating in the tryout had been in the Army less than 15 months,

at Fort Rucker less than 8 1/2 months and in their respective MOS 8 months.

Sixty-seven percent (67%) had ranks E3 or below. In order to achieve the

required number of participants for the tryout, the length of time soldiers had

been at Fort Rucker had to be extended beyond the desired 3 month limit. One

soldier was a master sergeant who had been in the Army more than 9 years.

Nevertheless, the GT parameters were not extended in either direction.

Over half (55%) of the population indicated they will seek reenlistment.

If this is the case, they will most likely require some assistance to raise

their GT scores in order to qualify for reenlistment. Thirty-eight percent

(38%) indicated they were not sure whether they wanted to reenlist. Perhaps

some of the doubt in this group could be attributed to lack of confidence in

their ability to meet desired minimum standards on the GT. Some had already

undertaken study to improve basic skills: 24% had taken a BSEP reading course,

10% a grammar course, and 29% a mathematics course. Ninety-three percent had

attained a high school diploma.

On the basis of background characteristics, Army career goals and

educational experiences, it is highly likely that most in the Fort Rucker

sample represent soldiers Army wide who would be interested in taking JSEP

lessons to improve basic skills, particularly for reenlistment. Therefore, we

conclude that most of the soldiers in the Fort Rucker tryout were

representative of potential JSEP populations.

17

l..-" -.A
LA

Table 4

Sample Background Characteristics of Fort Rucker Tryout (n=42)

Dimension Frequency Dimension Frequency

Age mean = 22.4 Time in present mean = 14.5

median = 20.7 job (months) median 8.0

SD = 4.4 SD :18.8

- range 40-18 range 2-84

Rank El (1) Time on po.s.t mean = 11.7

E2 (14) (months) median = 8.5

E3 (13) SD = 11.2

E4 (9) range : 1-42mo

E5+ (5)

Sex male 34 High school yes = 39(92.9%)

female 8 diploma no = 3(7.1%)

Race white 19 Highest grade 9th I

black 17 completed 10th I

hispanic 6 11th I

12tn 38

13th 1

MOS 67N, V, Y (11) Seek reenlistment? yes = 23(55%)

- 76N, W, Y (16) don't know = 16(38%)

91B (8) no = 3(7%)

94B ( 7)

Time in mean = 25.7 Any postsecondary yes = 4

Army median = 15.5 education? no = 38.

(months) SD = 25.2

range = 5-99+

Taken a BSEP yes = 10(24%) Had a previous yes = 7

reading no = 32 computer course no = 35 =

course?

Taken a BSEP yes = 4(10%) Had a previous Audio- yes = 11

grammar no : 38 visually oriented

course? course no =31

Taken a BSEP yes = 12(29%)

mathematics no = 30

Taken an ESL yes = 3(70%)

course no = 39

lb•

Fb''....
*, , Effectiveness of Lessons

The Fort Rucker tryout provided an opportunity to examine the quality of

lessons on an actual client population. Until the preliminary tryout, the

lessons had undergone only internal reviews and some one-on-one testing. -.

Because we were unable to conduct extensive formative development of the

lessons, important questions pervaded early development such as: Are the

lessons too difficult or too easy? Will they sustain sufficient interest and

attention? Are the screen displays attractive? How much can soldiers learn in

a short 20 minute recall lesson? How much can soldiers learn in a longer onehour lesson? The tryout offered an initial examination of such issues.

The difference between soldiers' pretest and posttest scores on the 17

lessons are presented in Table 5. Across all lessons there was an average

median gain score of 8.0%. There were two lessons in which there was a 20%

gain or higher, five in which there was a 10% gain or higher, five in which

there was a 5% gain or less. The long lessons (I hour of CAl) produced an

average of 14.1% gain while the short lessons (15 minutes including the test)

produced a 6.1% average median gain.

Table 5

Cumulative Percent of Soldiers Scoring at Percent Level or Higher

on Pretest and Posttest

Lesson 50 60 70 80 90 Median Median

Correct Gain

Long lessons

1.lg Rounding Pretest 74 49 31 26 8 60.3 +9.7

Numbers Posttest 80 62 52 35 25 71.0

(n=40)

2. 5f Gages Pretest 100 100 88 65 47 83.3 +11.7

(n=12) Posttest 100 100 92 83 75 95.0

3. 26b Technical Pretest 46 22 8 5 3 47.8 +31.5

Vocabulary Posttest 95 81 65 49 30 79.3

(n=37)

4. 36c Pretest 79 72 59 31 3 71.3 + 3.6

Capital i zati on Posttest 83 70 61 30 9 74.9

(n=23) -*

Short lessons

.T 8a lables Pretest 87 80 80 73 47 88.7 + 9.3

and Charts Posttest 92 92 92 75 67 97.0

(n=12)

6. 28b Tables Pretest 78 72 61 56 22 82.7 - 1.2

and Charts Posttest 88 88 71 65 18 81.5

(n=17)

19

Ig -'-.'
rrW.r rr r ,- - *v.--.*--. .- - -

Table 5 Continued

7. 28c Tables Pretest 94 65 47 18 18 67.0 +19.0

and Charts Posttest 86 86 79 57 21 86.0

(n-14) : .;

8. 30b Flow Pretest 94 79 64 46 5 78.4 + 9.0

Chart Posttest 90 87 82 67 32 87.4

(n=40) " A

9. 32a Locate Pretest 100 100 95 92 90 99.8 + 0.1

Blocks in a Posttest 100 100 100 100 95 99.9 ..

Form (n=39)

10. 32b Forms B Pretest 100 100 "97 89 84 97.6 + 1.2

(n=31) Posttest 100 100 100 100 93 98.8

11. 32c Forms C Pretest 88 88 88 82 76 97.8 - 0.9

(n=13) Posttest 92 85 77 77 77 97.9

12. 32d Forms D Pretest 86 79 71 54 11 75.8 + 4.7

(n=28) Posttest 100 96 82 71 32 80.5

13. 32e Forms E Pretest 100 93 90 69 52 88.8 - 5.8

(n=29) Posttest 100 100 86 59 41 83.0

14. 34a Major and Pretest 72 51 28 20 8 55.6 +17.7

Subordinate Posttest 82 74 69 46 38 73.3

Clauses

(n=39)

15. 40a Hazards Pretest 100 100 100 100 92 99.5 + 0.2

(n=30) Posttest 100 100 100 100 93 99.7

, 16. 40b Preventive Pretest 100 200 100 85 64 89.6 + 2.0

* Measures Posttest 100 100 95 90 72 91.6

-(n=39)

17. 40c Pretest 56 51 28 15 0 55.6 +24.0

Emergencies Posttest 95 92 73 65 32 79.6

(n=37)

The conclusions pertaining to the lesson performances are that the long

lessons appear to help individuals increase mastery of lesson objectives but

* that the effect of short lessons as "refreshers" varied considerably across

- lessons. The largest gains were in lessons that had low pretest scores and

dealt with concrete verbal or figurative learning (e.g., Technical VocabuVry,

Major and Subordinate Clauses, Emergencies). The more abstract the concepts or

the more a lesson stressed the application of rules, the less was the gain

(Capitalization and Rounding). Over half of the lessons (nine of seventeen)

* had median pretest scores exceeding 80" correct. Thus, there was a pronounced

20

* . . .-.
'ceiling effect on the amount of potential gain the lessons could demonstrate.

Finally, while the recognition mastery of the lessons was demonstrated and

sustained over several days, the amount of recall mastery was negligible. The

validity of the recall test however may be suspect, because the test items were

more intricate and complex than the test items in the lesson tests.

The third aspect of the evaluation concerned the attitudes soldiers had

toward the quality of the lesson, and the conditions under which JSEP would

best be administered. The responses of soldiers are presented in Table 6.

Table 6

Attitudes Toward JSEP

Item M SD

I = strongly agree

2 = agree

3 = uncertain

4 = disagree

5 = strongly disagree

Conditions for Learnin

1 wZ S . Z t ; -r. .

2. Using the computer for four hours at a time was 3.71 1.03

too long.

3. I think written assignments should be used along 3.80 1.04

. with the computer.

Soldier Impact -i

4. I think the lessons will help me to read and 1.98 .72

' understand the publications I use.

- 5. The skills I've learned in JSEP will help me to 2.76 .85

advance to a higher grade/rank in the Army.

Army Related Concerns

" 6. I think the instructor in the Education Center 4.29 .60

- should teach the lessons instead of the computer.

7. 1 would be willing to take more JSEP lessons on a 1.76 .69

computer if they were offered during on-duty hours.

• 8. 1 would be willing to take more JSEP lessons offered 3.14 1.24

on a computer if they were offered during off-duty

hours.

21

, .. .... ,,

-.. . ' '," . > " > ';. .-. ' .."' " "," '- '- ' -' " -. '. .. -. " " -""" -" " "" - . "".- -", . .. " ,*" '" " ,' ,' . .- ,- '. ",.
":': " Table 6, Continued

9. 1 think my unit commander would be willing to release 2.31 .87 "-" me from duty to take JSEP lessons.

10. I think JSEP should be included as an educational 1.62 .66

program offered by the Education Center.

We conclude that the soldiers enjoyed the lessons and felt they learned

from them. However, the majority of soldiers indicated they would not elect to

take the lessons off-duty. However, there was considerable variability among participants. The soldiers strongly preferred CBI to regular instructor

delivered lessons.

Student Response to Screen Formats -

The purpose of the JSEP General Questionnaire was to assess soldier

* attitudes about the JSEP program as a whole and how JSEP might suit their own

and the Army's goals. The subjects were 42 soldiers participating in the Fort

Rucker preliminary tryout with GT scores between 80 and.99. They were exposed

to a one-week, four hours per day simulation of the JSEP program. The

instrument was administered on the last day of their assigned duty time and the

soldiers were informed that they were not required to record their names on the

survey.

Results

Question 1.

" It was easy for me to learn to use the computer.

Strongly Strongly

' Agree Agree Uncertain Disagree Disagree

45% 50% 2& 0V 2%

Comment: The brief orientation procedures used to orient individuals to the

computer appeared to be fully adequate for 95% of the subjects. Appropriate sign-on procedures were demonstrated by the proctors, and the existing selfinstructional computer programs on MicroTICCIT and PLATO directed them from

there. Minimal individual coaching and encouragement was given by the

proctors.

22

* .* L

*- .* * *- . * . . . . -.. 

-- - - - - - - - - - . * . * . * t. - * * - . * * * . .* . ... *.* .-.
<Quehi on 2. *,,,.

Using the computer for four hours at a time was too long.

Strongly Strongly

Agree Agree Uncertain Disagree Disagree

. 0% 20% 12% 46% 22% -:..

Comment: Sixty-eight percent of soldiers disagreed 2the with the statement and

only 19% indicated four hours is too long. Thus, four hours was not considered

to be too long a time in which to receive computer-based instruction.

Question 3.

I think the instructor in the educational center should teach the

lessons instead of the computer.

Strongly Strongly

Agree Agree Uncertain Disagree Disagree L.'.'

0% 0% 7' 57% 36%

Comment: 930 of the soldie-s disa eed with this statement.

Question 4. (Bogus Question). .

* Question 5.

I think computer lessons should be used only to practice what we

learn from an instructor.

Strongly Strongly

Agree Agree Uncertain Disagree Disagree

0% 17% 14% 45% 24%

Comment: This was a question to help determine how a computer should be used

* in an instructional context. More specifically, wFether it should be

restricted to use as an adjunct to conventional instruction. Most soldiers

(69%) disagreed that it should be used exclusively as a complementary drill and

practice medium.

23

• . ..

~~~~~~~~... ,..........."............... ... ,...,.... ,..,.,.. :
: Question 6.

* I think written assignments should be used along with the

ccauter. -" 1

Strongly Strongly

Agree Agree Uncertain Disagree Disagree

10% 38% 14% 33% 5%

Comment: A question was raised by one of the trial observers that some

soldiers said they would like print materials to go along with the lessons.

Forty-eight percent of the OSEP subjects agreed with the statement while 38%

disagreed. Thus there is considerable ambivalence regarding the utility of .

such material. Another way of looking at the issue is that only 15% felt

strongly one way or the other.

Question 7.

The lesson about mood management helped me control my mood during

* JSEP lessons.

Strongly Strongly

Agree Agree Uncertain Disagree Disagree

Comment: The soldiers appeared to be mainly uncertain or not strongly

influenced by the mood management experience. The extent of ambivalence or

uncertainty might have been due to several factors: (1) the fact that the

lessons were administered in paper format, a medium soldiers did not seem to

prefer; (2) the lack of perceived relationship between the mood management

lesson and subsequent JSEP lessons; and (3) the absence, on the part of a

segment of soldiers, of a perceived need for such instruction.

Question 8.

I think I will use what I learned about mood management in other "_J;

situations.

Strongly Strongly

Agree Agree Uncertain Disagree Disagree

12% 62% 17% 10% 0%0.1

Comment: Seventy four percent of the soldiers agreed that the lesson on mood

management will help them in facets of their lives, outside of the realm of

instruction (see question 7). Thus the mood management lesson was perceived to

have more of a general value than in the mastering of lessons. The question is

raised, Is this the principal intent of the mood management lessons?

24

-' .--.. * .. . . . . . . . - * :-2.... 
Question 9. a'V.-'

The lessons will help oe in my nOS.

Strongly Strongly

Agree Agree Uncertain Disagree Disagree

7% 43% 21% 24% 50.

Comment: It must be remembered that soldiers were not assigned to the lessons

based on either their need for skill remediation or because these lessons were required for their MOS. For example, how much would cooks or pumpers and

gagers perceive a need for flowcharting, capitalization, or hazard prevention. Additionally, based on pretest scores of lesson , most soldiers were already at mastery level for most of the lessons.

Question 10.

Overall, the JSEP program was well managed.

Strongly StronglyAgree Agree Uncertain Disagree Disagree

38! . .4ET..

Comment: Ninety-three of the soldiers agreed the simulation was well

managed.

Question 11.

I would be willing to take JSEP lessons offered on a computer if

* they were offered during on-duty hours.

Strongly Strongly

Agree Agree Uncertain Disagree Disagree

38% 480 14% 0 0%

Comment: Eighty-six percent (86%) agreed they would be willing to take JSEP

courses on-duty. Thus soldiers appear to be very favorably disposed toward

receiving the kind of instruction encountered in the simulation.

Significantly, no one disagreed with the statement.

25

•-J. .

...................................-.."..-.-..-
Question 12.

I would be wi l ing to take more JSEP I essons offered on a computer

if they were offered during off-duty hours.

Strongly Strongly

Agree Agree Uncertain Disagree Disagree

7% 29% 26% 19% 19%

Comment: The responses indicate that soldiers are less favorably disposed to

taking JSEP on their own time. Nevertheless, only 38% disagreed with the

statement. This outcome must be interpreted taking into consideration that many ,'-,

of the soldiers saw no need for the instruction since they had already mastered

most of the material. Perhaps more would be favorably disposed to taking JSEP

on their own time if they perceived a pressing need for it, e.g., raising a GT

score to qualify for reelistment in the Army.

Question 13.

I think my unit commander would be willing to release me from duty

to take JSEP lessons.

Strongly Strongly

Agree Agree Uncertain Disagree Disagree

17% 43% 36% 2% 2% "

Comment: Sixty percent agreed with the statement and only 4% disagreed.

Consequently, most of these soldiers perceived there would be command support

for such a program and that the program will be worthwhile for the Army. This

item was also designed to allow soldiers to "project" on to their commanders

their real feelings about the worth of the experience. Thus, this item may be

one of the more critical appraisals of the JSEP experience.

Question 14.

I think JSEP lessons will help me read and understand the

publications I use.

Strongly Strongly

Agree Agree Uncertain Disagree Disagree

24% 57% 17% 2 0

Comment: Eighty-one percent of the soldiers felt the lessons would help them

to better understand written materials encountered in the Army.

26

.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
: Question 15.

The skills I've learned in JSEP will help me advance to a higher

grade/rank in the Anow.

Strongly Strongly

Agree Agree Uncertain Disagree Disagree

10% 12% 52% 17% Ole

Comments: Few (22%) soldiers saw the potential for JSEP to help them gain promotions. The reason for this may be similar to the possible explanation of

question 12 results.

Question 16.

I think JSEP should be included as an educational program offered . by the Education Center.

Strongly Strongly

Agree Agree Uncertain Disagree Disagree.

48% 43% 10% 0% 0%

-..- - - - - --- -

its potential inclusion in the Army's Educational offerings. Ninety-one

percent agreed with the statement.

Summary Conclusions

1. Soldiers had little difficulty learning to use the computer.

2. Four hours is not too long for a learning session on the computer.

3. Soldiers did not prefer instructor-taught lessons to the computer based

lessons.

4. The relationship between mood management lessons and helping to learn

lesson material was not apparent to the soldiers, but they did perceive

the relevance of mood management to other facets of their lives.

5. The soldiers saw the major utility of the lessons in helping them to read

and understand published material. However, they perceived less utility

for MOS job performance and even less for helping them acquire skills to

gain promotions. '.'

6. The soldiers very much enjoyed their experience during the trial and would

like to see JSEP become part of the Education Center's offerings.

27
H. DEMONSTRATION YEAR -'-. :...

The demonstration year is Phase II of the JSEP Implementatlon effort. In

March of 1984, representatives from Education Division, ODCSPER, ARI, and FSU

met to replan and reschedule JSEP implementation. At the beginning of the

development contract, September 1982, it was envisioned that Army-wide implementation would immediately follow curriculum development. During 1984 it

became apparent that the cyclical nature of contracting for learning center

staff, and the U.S. Army budget and procurement processes would provide a one

or two year gap between curriculum development and stage one of the phased Army- wide implementation. This provides an excellent opportunity to schedule

activities which will contribute to the eventual success of that

implementation. Three phases of activity were planned. The first phase will

be dissemination of the objectives and test questions for the curriculum.

Phase II will be a year-long demonstration at four sites. The final phase,

Phase 11, will be technical consulting to the actual implementation sites.

All three phases are concerned with transmitting information.

The demonstration year will provide an opportunity for Army personnel to

see a successful program in a relevant long-term setting. It will also provide

a setting for advanced training of personnel.

While this effort is not directly the concern of FSU under the present

JSEP contract, its existence has affected some of the decisions concerning this

contract, and FSU personnel are able to offer recommendations. One major

recommendation was that the demonstration sites be the same as the field trial

the equipment twice in a short period of time.

I. IMPLEMENTATION

It is intended that JSEP have a management system that could assign other

ACES components. This assignment might be to McFann-Gray mrTa-terTals, reading,

English as a second language, etc. The umbrella concept was the notion that

all programs be related.

Statement of Work for JSEP Coordinator

The job duties of the JSEP coordinator in learning center staff have been

defined in a statement of work. CBI instruction requires different

facilitation skills than one-to-one or small group tutoring. It was recognized

that the contracting cycle of the individual learning centers required advanced

information about the types of skills required of future staff.

With the time frames and the nature of CBI in mind, FSU staff and persons

familiar with staffing at the CBI facilities were asked to contribute lists uf

job duties. These lists were then distilled by eliminating duplicates.

The SOW includes all the job duties connected with JSEP; it does not

necessarily represent the duties of one person only. The staffing pattern and

numbers of staff will be dependent on the number of students per class, and the

skills available in the job market.

28

C, ° .

............................. "............"............................'.....................................................................,..........•... • '. - ,. . ',."%" .
We have carried out an extension of this activity. One of our staff

members has contacted the PLATO lab staff of the Army sites, interviewing them

about the conditions of their jobs. From these interviews we have developed . . guidelines for instructional materials for staff. "

Exhibit - Statement of Work

Present staff may or may not fit into the staffing profile for JSEP.

2'

°-"

.. . . . . .. . . . . . . . . . . . .
:STATEMENT OF WORK FOR JSEP COORDINATOR

"Backgr

1. Work experience in training or education with adults.

2. Work experience in or basic knowledge of CBI or CAI

instruction.

4. Good oral communication skills.

Pre-JSEP Training

1. Successful completion of training in: JSEP concepts and

formats, learner strategies, terminal operations, PLATO/TICCIT management. "

2. Participate in periodic conferences/workshops.

Computer Operation

1. Operate computer terminals; use terminals -to enter required

data; make necessary printouts; enter student

data/information; create and update permanent student record

2. Demonstrate system to interested visitors.

3. Work as trainer for new or replacement instructor(s).

Student Management

1. Receive students assigned to the JSEP program.

2. Record and report daily attendance of soldiers.

3. Assist students in using terminals, and in following JSEP

procedures.

4. Assist students in the use of Handheld Tutor.

30

- *.*** * K ... ..Ij . . . . -*. * *
. .3 7. o

5. Answer student questions concerning terminal operations, JSEP

content, policy, and logistics.

6. Monitor test taking to assure test security.

7. Operate the CAI system in student mode:

a. Sign on new student

b. Enter relevant entry and test data

c. Detect student problems with logic of instruction anc,

place student at correct location in instruction.

8. Administer tests and other instruction that may accompany

JSEP.

9. Manage JSEP students through the use of information provided

in the data management system.

10. Conduct student orientation to JSEP.

11. Accumulate, maintain, and distribute any supplementary

material (e.g., required printouts, charts for pacing module,

paper and pencil).

12. Operate any equipment used in JSEP such as tape recorders,

interactive videodiscs, etc.

13. Prepare tor handling computer down time and other

extraordinary situations.

14. Maintain any required records in addition to those kept by

the computer system.

15. Assist students in selecting appropriate lessons following

the data received from diagnostic testing and instructions -

from contractor and/or Education Center Director.

17. Provide accurate and motivating feedback to students.

18. Document student comments on the lessons and note criticisms

or problem areas.

Computer Maintenance

1. Conduct basic troubleshooting and maintenance of video

equipment, cassette recorders, videodiscs, computer terminals, and other media equipment. .-

2. Recognize system errors and problems, and document these.

31

" "'" -°.A
w-N

4' Instruction

1. Administer pencil and paper tests and interpret results

according to lesson specifications.

2. Prescribe appropriate lessons for student according to test

results.

3. Direct students to alternate/supplementary learning

materials. Perform some basic library and filing functions

in order to maintain and track these materials.

4. Administer learning strategy lessons according to

specifications.

5. Use a variety of motivational, anxiety reduction, mood

management, and problem-solving techniques (as introduced in -.-.

workshops).

6. Teach lessons or skills which cannot be presented or

evaluated on the computer.

7. Use reinforcement, self-monitoring, and behavioral

contracts; counsel students on the application of these

techniques to individual situations.

* 8. Use alternate instructional methods, including implementing

poiicies concerning failure and the repetition of lessons.

10. Administer (or assist) posttest.

11. Collect and interpret post-cycle data and provide Education

Center requested and/or contractor requested records.

32

................... ... .... * .• -" 
APPENDIX A

Summary of Priority Data

33
*. co - co - co

L V%

0 c 4WP% I

* W0-CIn N-

*~p Vi0

Z

* 0

* 3.-

-- 4-

C- 0 0 

N N. 4.

0 2z '0'DI -

a~ cc L

*~ ~ 0 c'' -'

o 0 g

C 06 0 0 U u a'A
-~C . -. .. . in % J . -J W%~- j.w .,~-.... ~ .

I- IC% - = CVC4C

v N 'F'F

aft

incN-OLn

0' 0 N

LiJ4)

- =-4=

-~ N .8I~

U-Ac

L. le

0 OF. 

06c4- 0 - 0

0~ ~ C~£

*~~~ 0..-u L

.a 3. vi 1.- *

.0 6.

C ou L 4- 0-0

Nn C-44
122

C- N- WNW

bk N0 to0

Z0

L L

* .4-

OL - N- N

06 0

N 4 0 '0

0'

8

0CII'VI 0'A 00 'D A a

* 2 *CL 

U- IQ 9:
c4 ON-

* 4-4

0 - 0L

0-9

Z 0

0 s.8N 1 ~

0 v)

* .4-

IN 0

I M 4 -4 C M

Im r

* ~0

4D CC ,

u-v;

6cr I-
-d v

_ ° . . *

-* CL

4G.-

U'_ %-

*,, , ....-.

Ch" %- w"

.N N ...-

0.

0 '0 :

i.

.. " b0 ". 

I.- - 0,. 0 0 ~

N N'

C

to x

W 00 C-N4

i.. . .-

z0.

400

N 0 0 A

L in 4-04

*-Cc

4.- -

00 1

w _0 C v N -

- "4"

w 060N N 
~4 C.* C4 - v 4

2. +-~

NN 4 C1 a C4 N C - n

.0

*100

* ~L~

0 to

4~44

00

AI

~LA

C L 

r

In A4 AI ~~~- I- -

in 0A i
4FNI

-W V -0C

(n o o0 eoC

10 N10

66 %

-r co q

0 0 -

U-4-

IZ ~ 0%-r

x > 4- 

0 ~ 0

+- 06

u c a.; 

C0 . 4A 

U~ ~~ 4. D- 
IL~

N JN

4-

.9- wk

* 0%

CL

10 (Ni -(

CC 44

0 .1c0 

.

0

Cc L-&

.n0 '00%I 

C C 'A 9:

c aa

-- 0- N L.-0

c0 06I L'0 r '

4)

Ao Q. - o(

(An
60 C L . < . . C.,

~ O°

CI IONl

CLL C.L '0 I- N

* 0

I , ,

I- '" .A-'.'- 

I-- In N4 V C4

110

CZ n

0r- N

0*-0

0 4n

(-)-

- Cc

-00 0 -L- -

WN

. c "n-" '

*.+0 V '0

0 NE. 0 u~b 

U-4-

U) C C

('A

Ic L

C 0 a C C

zo
L6. . . . . . . ~ t U * PI~ V~

-PA C...--.* in;.- -. *--

r4 C4 -

OL

10

.4

01

.9-4- 0 

0r 0 0C - -

-- 0 A4-I 4- V0 0 04

0 4-

0 IA 

-7-7
in -W in -r 

IL

4 N FIN N 0

II

4-4

0L

u .

L C> 

P. 2. 4- %P. N 

QC

i

-0 -l - r0-

C X

- 2 

o ot

.

006 01 I

-O vC c vi 0 4

* C CL0 0 4 00 i1 L. EL 3 0

A2 C0 6U4I I-L

V. 0 0 C0 Q0 -0 L,.o

C u - U0 0 4

0 4O )0 ' C

4 -

C4C0
-) *j ---- I ,-k71-)r . - X x - v ; 7'

4-

FI C-4

CL

0o m 

ao V 

CNPN n w

*qA.

'0 Go

U- 0

a C

4---

2 -A N

00MD

* 4-

10 4-

0. C c L

L - -

U .. VC .% 
o C4tno

.1.

0.

*0.

uQo W -

10

An co

0~~c P4'9.'.Q'

u S

0 . -

I 0. >0 0A'

00

4- 3.

0 c.0 I .

IA~4 IA- ..

L. ~L 0-U 40 0 L

0 0

N00 - 0 0o

0 0

lo0 0 UL

> L. 1 ID1

V3 .1 a 

14 -n. 3.7C

u InC U~U 0~0 4 3V
SL I W

inPNW

CL~~N

04- N- w

i-E

C4 In co-C

17 ~0

* O L

000

l~0 0~4 0 l

~0L'L L C

-- 41 Q.

I0 0 Iu-01
CLI

N~~ If PI

L'C

.tp - N N

u~OOZr 0

'0 0 0 0,0

W4-n

*n La.

0 A3

* 0

0 D6' 0

0 41 CL 

u 06

) .270 i

LOL
IL e

C..)L6 0

OL

L4

CrC

Ell

4-44

00

U-U

2 j... r-. 1

P11

0 0 -

* C 0

CU C

40 0 o

2- EC . 26 I E C C L.- in C V

.0 - .0 V- 0, in CL. 0 0

CCL 'aIL

CL Cc2.
[ -

Ze CL4

"a.

,". ..- '

0 *"~.0.

.I- .

* 0

* 3.

4-I

- In0 n

r we

-4--,.4-

aa

"oo c c

CA 4- C 

-4

,.0 0 - """

0 00

cc~~ P. .0

CD C4

0 0

"- " " -.

" " : " -" -.. .. . . i. "-:- 2-.'2:',"i- 0 "''' . '''' " ",0. 'V - , ' -,-,-.- - ,-.. :.-..., __ i;-"i -,."..' " " .? - V .. -2 1 i- - " -,"

OC 3mU
*1. w

V 4-

0)0

i~OD

0 z

WI0 0 Cox -0 -- 0 (4 'U' 9 

4- 44-

o + 0 o v9 x p. (m~1 0 4- 0 D

0. Z Ac

WI 0- L C F'

o- 0 0 gorC C

ZO 0 Lu 0 0 1. -

* 0 0 0---

0 06a

Cv M-1 C4-

CE
IL

cCLL

C -C4

ho CL

U. 0

0

~n- if 10

-3 - 0.

-1 1'0

- NC % - N

*C 01

0in 4-

0-aI-. 00) jt

'lix
An *.A7j .---

W In Pol

IM0

I.x

cm0 0

Li. X .

4--

fa ccr.q.-c

a 0 -

N -V 4-- i C

0. ILI V).L'

'. L +

0 a

an a a

-0 n S

rq C14I

&&
ITruw wr-r R -u 

kk

iH

CC

0 NO

0 0o

v 00C

4--

C4C
LL. 0

0a

ut n-I 3in

4-n

0 0F

00

0r 0 0

~~0 v0Gn o 04-

v v

CL

40 4v0 S

aa

4- Q6

L- 4-

con

C4 Ro
X.-.

L.

4.

L. b

Ll.

.21! co .5,...

a.D

.% W

4.

0n 'A'0I

04

0 u o

-. X.- Iu0-0 *r r.' OX~I '0

0 +.

.0S 0q 0 0 'S

ii4 N N1 4- C1 

0 0 C '0 - U

0

SNCL
-, -k -Or Ti 72. T- 7--- M77 .. 7.. IF 7- -T

NC

0a

L 1^ Ch -

CLL

0 j

VI- at 0a Li - - - .: 

4-

- 01-

4- 1
6.

L6C

+- 4m

Cc

= L . .

0 I0

wL 4-

* 6

L

40 X~00

aa

4-4-

oA 0 0o. o a

a1 4

1 4- 0 L. L aC C 6C I

I. '- - - -

o a 0l 0 0IvP

*FI
-. -7.** -t -N S - A - - - , -

* CL

.6-

-CL

IGo -- r

44

CNn

.0 In O

0 g

0 00

0 0g. c c

0 0L 

0 0C 06 Q

.C II- 0' In L 0

0 CLc4

1. CL Z S1

Oc 2+ a L

4- C

lop" CO C C.0

co4-

Ni8

FI Fn Fn~ O
- 7 -* I IT3

C4 C

N % N

V) 0

%00

(DL ~ o0 0

-100

'A N

W N 

0 %C

Z 0

X z0

0 c 0

c

*~ 0-Q ~ 4)4 

o ;: SC on

le 7 - 0 E.

4 OC cm C4

r c 10 I- -

CL 00 4

0 - T

EL. ~4- 0 I D 0

'AU V

4n E

p Co z -

L LU
.4-

30- to N-f

L6a

An 0

* 00 b' -

C L. .

N 0

og

0 00

4- 4- 't 4

4-~ 0 a L. 40 q

4; L

* 0--

4- N - in N

4 'A -

4l- .N60

aa t

re-L

cxz
4 94 l

CL

I--

0..

No No -N

a.Z

.,qrr, 0 0 N ,

I4-4

< - i. W- , on%

-w. C4% C4

gg .- 0

:-,

.Cn.

/N

rx 0

, a,. . ,,- - 0 i "'-" IAD 0 aA N

L a 

3 V

N N N- N

r.. Wi0

- * -a Cc

.L

'A 4 - 0 06

* o

..-..
* 44

0

co % %0W

0

ev W go %0 '0

IC 0 C4 In N N N4

VII

C4 (

u

04 CN NN

Z0

I" 0

00

U-c40 - 0

6 0 0 0 0

* 0.7

0 0

1A u-
4,.

ON.o

* 0

C* 

%

0'

0' L6~

U-UM

0 0 0

* 4-

0 in1. 1"

I.4-C

cr 0

s- id J.in L

0 - '

o 6 6

10 C 0oo 

.~u

0 - aa+

* 0

*.~~~~~~...*~~~~1 4.. *. * . . . * . . .* t.. 4. * * * *
LL I&.n

4.o

Z04 N ' F'4 F'.F

0 0 Ck - 4

.

10

0 40- 0r

(i r% 210

0~ .2.1 '. L. c'

0 u

CxL

*~~% N V' ' '

30

41C 4. 9

a a s. L.1 -

-0 v
14

Lai

10

41)0

0 c

L. X

0- 0

*~4 64

%04t