,.- echnical Report 613

The Prediction of

Training Device Effectiveness:

A Review of Army Models I

Daniel R. Tufano and Robert A.Evans

Simulation Systems Technical Area

Training Research Laboratory

LCl

0.. 11984

E

U. S. Army

Research Institute for the Behavioral and Social Sciences

4 April 1982 _.-

Approved for oublic release; distribution unlimited.

84 10 16 213
U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency under the Jurisdiction of the

Deputy Chief of Staff for Personnel

L. NEALE COSBY

JOSEPH ZEIDNER Colonel, IN

Technical Director Commander

Accession For

NTIS CPA&I

DTIC T..)

U~nannm >; ed [

Ju-';tji 'catiL'

By

Distr

AwVLKtLi/ty Codes

* Lis ~-xaz urd/orDist ! Special

NOTICES

DISTRIBUTION: Primary distribution of this report has been made by ARI.

Please address correspondence concerning distribution of reports to: U.S.

Army Research Institute for the Behavioral and Social Sciences, ATTN:

PERI-TST, 5001 Eisenhower Avenue, Alexandria, Virginia 22333.

FINAL DISPOSITION: This report may be destroyed when it Is no longer

needed. Please do not return It to the U.S. Army Research Institute for

the Behavioral and Social Sciences.

NOTE: The findings In this report are not to be construed as an official

Department of the Army position, unless so designated by other authorized

documents.
1. 1..I . W-;1<

UNCLASSIFIED

SECURITY CLASSIFICATION OF THIS PAGE (W7h.n 0.t. En,..d)

READ INSTRUCTIONS REPORT DOCUMENTATION PAGE BEFORE COMPLETING FORM

1. REPORT NUMBER 2.GVT ACCESSION NO. 3. RECIPIENT'S CATALOG NUMB3ER

Tech n ical1 Report 613-3

4. TITLE (end S~bttI.) S EO EOTAPRO OEE

The Prediction of Training Device Effectiveness:

A Review of Army Models

7. AIJTIOR(a) 6. CONTRACT OR GRANT NUMUER~o)

Daniel R. Tufano

Robert A. Evans

9. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT. PROJECT. TASK

US Army Research Institute for the Behavioral AREA A WORK UNIT NUMBERS

& Social Sciences (PERI-IE)

5001 Eisenhower Ave., Alexandria, VA 22333 2Q263744A795

It. CONTROLLING OFFICE NAME AND ADDRESS tZ. REPORT DATE

US Army Research Institute for the Behavioral April 1982

& Social Sciences (PERI-IE) 13. NUMBEftOF PAGES

5001 Eisenhower Ave., Alexandria, VA 22333 47

14. MONITORING AGENCY NAME &AOORESS(If different Iron, Controllini 011ice) IS. SECURITY CLASS. (of Ci. repoet)

UNCLASSIFIED

-- IS&. DECL ASSI FICATION/ OWN GRADING

SCHEDULE -

16. DISTRIBUTION~ STATEMENT (of tis Report)

Apprnved for public release; distribution unlimited

17. DISTRIBUTION STATEMENT (of the abstract entred li Block 20. If different from Report)

IS. SUPPLEMENTARY NOTES

19. KEY WORDS (Continue on revere, @(de If necessary and Identify by block number)

Training Device Analytic Model

Effectiveness TRAINVICE

Transfer of Training

Pr~edict ion

24- ABSTRACT' fl~att a ,.ejwe. e t uneceeeiy and Idetrit by block iuther)

AThis report is a critical review of the analytic models which have been

developed by the Army to predict training device effectiveness. In particular,

special consideration is given to a family of models known collectively as

TRAINVICE. Although they are all intended to provide an index of transfer

* potential, these four models differ in several important ways. The variables

N considered in the calculation of the indices are given different degrees of 4, emphasis (or weight) in each model. In fact, in some cases a variable is .

ignored completely; e.g., the instructional features of a training device.-*i'erb

DD 1 I41 ecot. or I Nov ssI is aSOLETE UNCLASSIFIED

2.SECUAITY CLAWSFICAT100t OF THIS PAGE (11h-n Date Entered)

% %~

** %1--
V UNCLASS[FIEDI

SECURITY CLASSIFICATION OF THIS PAGIE(Wh. Dot. Bate,.d)

lie prel~aratory. analyses employed to estimate the values of eachi variable change r great deal from model to model. Moreover, the combinatorial procedures used

tO calculate aIn index of effectiveness from the v.ral ausaevr ifr

ent in cacti model.

A major goal of this report is to present the principle points of difference among the various predictive models, and thus facilitate a comparative

assessment of their merit. The results of the analysis and subsequent validation will provide a basis for the selection/refinement of one or more of the

models for incorporation into a defined set of procedures for specifying training device effectivenessj

UNCLASSIFIED

LSECUARY CLASSIFICATION OF THIS PAGE~hon Date Rner.9

Z Z -z 
Technical Report 613

The Prediction of

Training Device Effectiveness:

A Review of Army Models

Daniel R. Tufano and Robert A. Evans

Submitted by:

Frank J. Harris, Chief

Simulation Systems Technical Area

Training Research Laboratory

Approved by:

Joseph Zeidner

Technical Director

U.S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES

5001 Eisenhower Avenue, Alexandria, Virginia 22333

Office, Deputy Chief of Staff for Personnel

Department of the Army

April 1982

*Army Project Number Training and Simulation 6 20263744ATOO

J- ~Approved for public release; distribution unllivtd.

* If ~& *
ARI Research Reports and Technical Reports are intended for sponsors of

R&D tasks and for other research and military agencies. Any findings ready

for implementation at the time of publication are presented in the last part

of the Brief. Upon completion of a major phase of the task, formal recommendations for official action normally are conveyed to appropriate military

agencies by briefing or Disposition Form.

a .

.

,-

a'.

¢,4.

i.a q V
FOREWORD

The Simulation Systems Technical Area of the Army Research Institute for

the Behavioral and Social Sciences (ARI) performs research and development in

the areas of training devices and simulators in the Army. Of special interest

is research concerning the evaluation of training device effectiveness.

Throughout the acquisition of a simulator or training device, training

effectiveness must be evaluated. Ideally, an empirical transfer of training

test would provide the data needed for an evaluation. However, when empirical

data cannot be obtained, training device effectiveness can only be estimated

using analytic methods.

This report provides a critical review of analytic methods recently

developed by the Army for the evaluation of training device effectiveness.

The results of this report have implications for training developers in PM

TRADE and TRADOC and for researchers in the field of training device effectiveness.

(JO EPH ZID R

TehnicalPirector

P17

v?

a.

I.

,'o.

V !
THE PREDICTION OF TRAINING DEVICE EFFECTIVENESS:

A REVIEW OF ARMY MODELS

EXECUTIVE SUMMARY

Requirement:

To review the analytic models and methods developed by the Army for the

prediction of training device effectiveness; and to recommend procedures for

the development, validation and application of improved models.

Procedure:

Four predictive models, known collectively as TRAINVICE, were compared in

terms of their implicit assumptions, analytic procedures, validity, and utility

for training-device acquisition.

FindingsDespite their common purpose, the four TRAINVICE models differ considerably in: the task, equipment, and personnel variables; and the mathematical

formulae used to calculate training effectiveness indices. The major limitation shared by all of the TRAINVICE models is that they yield overall indices

of effectiveness. The utility of such an index is strongly questioned. The

recommendation was made that a model be developed which would permit a more

detailed assessment of training device effectiveness. Ideally, such a model

would generate effectiveness indices for individual skills, and would provide

procedures for aggregating the skill indices into separate task indices. It

was concluded that separate skill and task indices would yield effectiveness

predictions of sufficient detail to be of use to the training developer in

the design, evaluation, and implementation of training devices.

Model application and development will require research to be done in

two areas: field validation of the TRAINVICE models in various task domains;

and longer range investigation of the models' underlying assumptions. The

latter area should include a refinement of the learning guidelines contained

in the models, and specification of behavioral criteria which are suitable

to analytic as well as empirical evaluation of training device effectiveness.

Utilization of Findings:

The review and recommendations will be of use to the training developer

wishing to use one of the existing TRAINVICE models as well as to the model

developer trying to improve the prediction of training effectiveness.

vii
THE PREDICTION OF TRAINING DEVICE EFFECTIVENESS:

A REVIEW OF ARMY MODELS

CONTENTS

Page

SECTION I: INTRODUCTION. ............................ 1

SECTION II: MODELS..............................4

TV-A......................................4

TV-B ................................... 11

TV-C ................................... 16

TV-D ................................... 21

SECTION III: SUM4MARY OF THE MODELS.....................27

Model Input. ... ................. ............ 27

Preparatory Analyses & Model Variables. .......... ....... 27

Model Output ... ................ ............ 30

Prescriptive Mode. .. ................. ......... 32

Separate Indices ... ................. ......... 33

SECTION IV: CONCLUSIONS. .. ................. ....... 34

REFERENCES. .. ................. .............. 37

APPENDIX. RATING SCALES ... ........... .................. 39

LIST OF TABLES

Tablel1. Model Input ... ................. ........ 27

2. Model Variables and Analyses .. ......... ......... 28

3. Model Equations. .......... ............... 31

ix
training device effectiveness. The need for such non-empirical evaluation

procedures has been a persistent concern of military training developers since

the I 9601s. In particular, the Army has recently developed a series of

models, known as TRAINVICE, which attempt to predict the degree to which

~training on a particular training device will transfer to performance on operational equipment. These models, which have evolved from a history of military training research, are the principle focus of this paper.

Jeantheau (1971) reported an attempt by the Navy at the "qualitative

assessment" of training device effectiveness. The forms and guidance included

in this document permit the cataloging of training device features and expert

opinions on those features. These procedures do not, however, result directly

in the evaluation of a particular training device. Rather, the method simply

provides a format for collecting and using information on training devices.

In a later effort, done for the Army, Caro (1970O) developed the Task

Commonality Analysis (TCA) method for the prediction of transfer of training from a Ievlce to operational equipment. The predictions were based on Realism

ratings of the stimulus (display) and response (control) properties of the

training device. In deducing which tasks would be trained well (i.e., high

transfer) and which would not, Caro adhered to Osgood's principles of transfer. He assumed that if both the stimuli and responses in the training situation were similar to those in the operational situation, then positive

transfer would result. Further, he assumed that if the stimuli were similar, but the associated responses were different, then negative transfer would

occur. Caro's choice is not surprising since these assumptions are ubiquitous

in the field of training evaluation and are well represented in the TRAINVICE

models which are discussed below.

trnfCaro's TCA method represented the state of the art when it was pub- er."lished. It provided the impetus and much of the groundwork for the developi ment of the TRAINVICE models. Although TCA is similar to the TRAINVICE models

in its goal and in some of its assumptions, it will not be treated more fully

here for the following reasons. The realism ratings were rudimentary (see

footnote 1) and were not based on clearly articulated criteria. The transfer

predictions consist of simple, qualitative statements about whether or not a

task will be trained well. Furthermore, the judgmental operations required to

generate the predictions have not been reduced to a formal algorithm. That is, there are no fixed procedures for transforming or combining data to arrive

at a clear prediction.

Raters were simply asked to judge whether or not a display or control

was "realistic". The realism score for a piece of equipment was the

percentage of raters who said tha that piece of training equipment was

"realistic".

*%-. - '...*o%o•-.*.-'.* .t' . . ' . ? . % . * . *.*#~ •. • ,~C C V y ~
It was not until 1 976 that the shortcomings of earlier approaches were

addressed. Between this time and 1co, the Army developed a family of predictive models known collectively as TRAINVICE. In their attempt to generate

quantitative predictions of effectiveness through formal procedures these four

models represent the most ambitious steps taken to date in the field of analytic evaluation. The level of sophistication and the potentially great utility of these models warrant a very close examination of the procedures, assumptions and validity of TRAINVICE.

The original method, developed in 1976, is referred to as TRAINVICE-A

(TV-A) in this report (Wheaton, Fingerman, Rose, and Leonard, 1976). In 1979,

the Honeywell Corporation modified TV-A as part of an effort to develop detailed guidance for user application (PM-TRADE, 1979). This modified approach

is referre d to as TRAINVICE-B (TV-B) in this report. Other modifications to

TV-A were developed by the U.S. Army Research Institute (Narva, 1979a, 1979b)

and are reported herein as TRAINVICE-C (TV-C). Finally, in an effort to

develop a user guidebook for applying TV-C, additional revisions were made.

(Swezey and Evans, l9O) This approach is referred to as TRAINVICE-D (TV-D)

in this report.

Although each model purports to provide an index of effectiveness, or

transfer of training potential for a device, these models differ in several

important ways. For example, the variables considered in the calculation of

the indices are given different degrees of emphasis or mathematical weight in

each model. The procedures used to estimate the values for each component

vary considerably from model to model. Moreover, the procedures used to

calculate an index of effectiveness from the variable values are also very

different in each model.

The TRAINVICE models do, however, share a common data collection

method. This method consists of a structured interrogation of a subject

matter expert. As such the models place a very high premium on the judgment

of an expert. The method focuses decision-making on a specific set of issues

for each task or part of a task. In the first of the TRAINVICE models, for

example, one of the issues considered is the similarity between the equipment

on a training device and that on the operational equipment to perform a particular subtask. This issue is further delineated into physical similarity

(appearance, location, etc.) and functional similarity (amount of information

flow between the human operator and the controls and displays). For each of

these, (i.e. physical and functional similarities) the expert assigns values

from a rating scale which ranges from 0 to 3. Guidance is provided by a description of the criteria associated with each value, (e.g., a "I" means

identical to operational equipment). This procedure continues until all

" equipment (i.e. displays/controls) associated with all sub-tasks have been

rated. An analogous rating procedure is performed for all variables in the

model pertaining to each subtask. In this manner, the subject matter expert

can estimate numerical values for each predictor variable considered by the

model (e.g., similarity, training techniques, task difficulty, etc.). These

estimated values for the variables are then entered into a general formula

which results in an overall figure of merit (index of training effectiveness)

for the training device in question.

The judgments of the subject matter expert and the index of effectiveness

rely on many assumptions, both theoretical and mathematical in nature. The

2
theoretical assumptions include: a) what is being predicted (e.g., a particular measure of transfer of training); and b) which task and equipment variables have the predictive power to generate such a measure of effectiveness.

The mathematical assumptions concern: a) the manner in which all the values

are combined (e.g., weighting strategies, etc); and b) the numerical properties of the rating scales used to estimate those values. As indicated

earlier, the four analytic models reviewed in this report differ considerably

in the assumptions made and in the forms in which the assumptions are manifested.

Section II of this report contains a detailed description of each model,

taken individually. Section III is a general summary and critique of all four

models, in which differences among the models are discussed in detail.

Finally, based upon the results of the critical review, future directions are

discussed in Section IV.

NA

~3
III

Models

a. TRAINVICE-A (TV-A)

Overview

The Wheaton, Fingerman, Rose, and Leonard (1976) approach, TV-A, is an

attempt to predict and evaluate training device effectiveness, specifically

transfer of skills from training to operational settings, by combining judgments about a variety of factors. Judgments are transformed into values related to the interactions among device design and use, trainee ability, and

training strategy. Effectiveness, therefore, is assumed to be a function of

the:

1. Transfer Potential: potential for transfer of training using a

particular device which is determined by the:

a. overlap or communality of the skills taught on a device and those

necessary to perform on the operational equipment, and

b. physical and functional similarity between a device and the opera- tional equipment

2. Learning Deficit: differences (i.e., deficits) between a trainee's

knowledge before training on a device and what must be known about the operational equipment, weighted by the difficulty of acquiring such knowledge,

and;

3. Training Techniques: appropriateness of training techniques or device

features incorporated into a device, and how well these features adhere to

accepted principles of learning.

The Wheaton, Fingerman, Rose, and Leonard (1976) model combines values of

judgments made for each of the above factors into an index reflecting the

effectiveness of a device.

Process

Generating forecasts about the effectiveness of a training device, according to Wheaton, Fingerman, Rose, and Leonard (1976) requires analyses of

the components: Transfer Potential, Learning Deficit, and Training Techniques for a training device. These components are subsumed under one of

the three categories in the structural and functional model of this training

device effectiveness approach. The process of analyzing each of the components (i.e., Transfer Potential, Learning Deficit, Training Technique) requires judgments2 to be made for five basic analyses of:

2 See Appendix for the rating scales.

4
SI. Task Communality

2. Physical Similarity

3. Functional Similarity

4. Learning Deficit

5. Training Technique

Values for these analyses are derived for a device under evaluation, then

where appropriate, compared to the operational equipment for which a device

was developed.

Inputs

Before performing the procedures by which values of the TV-A variables

are estimated, a user needs a list of training objectives and relevant task

analytic information for both an operational setting and a training device

being evaluated. TV-A requires that most analyses be conducted at the subtask level and some analyses at the level of the skills and knowledges which

comprise each subtask. A Subtask, according to Folley's (1964) definition is

an activity that is performed by one person and bounded by two events"

(Wheaton, Fingerman, Rose, and Leonard, 1976, p. 16). The value of each TV-A

variable is estimated for each subtask identified in the operational task

analysis.

Procedures

Task Communality Analysis (C)

Task Communality Analysis (C) assesses the overlap between training subtasks and those in the operational equipment. The value of C is determined

by comparing operational and training device task analyses with each other.

In this procedure, a training device is given a rating of "I" for each "i"

operational subtask it covers or "0" for those it fails to cover. Since a Ci

value of "0" decreases the sum in the numerator of the final prediction

formula, the task communality rating serves to penalize a training device for

each operational subtask not covered. A training device is not, however,

penalized for including subtasks which are not in the operational environment

(i.e., additional subtasks).

In the overall device effectiveness prediction formula, the sum of C

values for a device is compared to Ci values for subtasks on the operational

equipment. Since this comparison is made against the operational equipment,

Ci always "1" for the operational equipment.

Physical Similarity Analysis (PSA)

The Physical Sivilarity Analysis (PSA), and the Functional Similarity

Analysis (FSA), discuised below, combine to form the Similarity (S) component

in the predictive equation. The degree of similarity (Si) between a training

device and the operational equipment is the average of values assigned to the

fidelity variables: physical and functional similarity. The PSA allows for

judgments concerning the physical characteristics (i.e., appearance, size,

or
_%T -7 .1t 1.. -0. To . . . .-

location, etc.) of displays and controls used in training specific behavioral

performance on a device. The Functional Similarity Analysis is concerned with

the information processing activities of the human who is viewing the displays

and operating the controls.

The information required to perform the PSA is a list of all displays and

controls on the operational and training equipment relevant to each subtask.

The displays and controls corresponding to each subtask are given a rating by

judging how well the operational equipment is represented in a training

device: from "0" (not represented) to "3" (identical to operational equipment). These ratings are averaged across controls and displays and divided by

" to yield a physical similarity index ranging between "0" and "1".

Like the C analysis, the PSA for the operational equipment is "1_". The

rationale is that the operational equipment represents the maximum degree of

physical similarity. A PSA value of "1", therefore is assigned to all

displays and controls on the operational equipment corresponding to the

subtasks.

Functional Similarity Analysis (FSA)

Like the PSA, performing the Functional Similarity Analysis (FSA)

requires a list of operational subtasks and corresponding displays and

controls. A flow diagram for each subtask is then generated indicating the

type, amount, and direction of information to and from the operator for each

control and display. The amount of information (in "bits") is determined by

the number of stimulus (i.e. information transmitted from a display to an

operator) and response (i.e. information transmitted from an operator to a

control) states which displays or controls can assume. The remainder of this

analysis consists of rating differences between the amount of information in

an operation setting (Hos) and that in a training setting (HTS).3 For each

control and display, a training device is given a rating: from "0" (missing)

to "3" (identical: H - H ). Ratings for controls and displays are then

averaged and divided by "3P to give a functional similarity index for each

subtask which ranges between "0" and "1". The overall similarity index for

each subtask (Si) is the average of the Physical and Functional Similarity

Indices (P + F).

2

The FSA for each display and control on operational equipment, similar to

other analyses discussed thus far, is always a "1".

Learning Deficit Analysis (D)

The Learning Deficit(D) index, for every subtask, requires each skill and

knowledge be given two ratings (rating scales adapted from Demaree, 1961; see

The FSA analysis requires the rater to compare amounts of information in

log 2 units; a potentially difficult procedure for users who are unfamiliar with information theory. This shortcoming was addressed in TV-B's

revision of the Functional Similarity Analysis.
-%m . . - a - a.a ---- .....- *-*o-a . - - *- • .. . -b .o o

Appendix). The first rating, repertory scale (RS), assesses the degree to

which trainees are already proficient in the skills and knowledges to be

taught. A rating from "0" (no experience) to "4 (complete understanding) is

assigned to each skill and knowledge. An estimate is then made of the levels

of proficiency required of a trainee, for each skill and knowledge, in order

to perform a particular subtask to criterion. Accordingly, a criterion scale

(CS) value is assigned to each skill and knowledge: from "0" (no experience)

" to 4"(complete understanding). The criterion scale value minus the repertory scale value (CS-RS), then represents the learning deficit for each skill

and knowledge. The learning deficit index (LD) for each subtask is simply

* .the average of the learning deficit values of all skills and knowledges

involved:

S+K

LD £ CSi -RSi

i=1

# skills and knowledges

LD ranges between "0" and "4".

The Learning Deficit value for each subtask is then weighted by the

difficulty of training the skills and knowledges necessary for that subtask

(i.e., how hard it is to overcome the learning deficit). To do this, each

subtask is ranked according to the amount of time required to train that

subtask on the operational equipment (a rank of "1" for the easiest subtask;

higher ranks for subtasks requiring more training time). The learning deficit value for each subtask is multiplied by its rank, then divided by "4"

times the total number of subtasks. This procedure yields a weighted learning deficit value (Di) for each subtask which ranges between "0" and "I". A

" - D value is computed once for the operational subtasks because these values S. are applicable to both a training device and operational equipment.

Training Techniques Analysis (T)

In the Training Techniques (T) analysis a training device is rated on

how well it implements established learning principles. The first step is to

assign one or more task taxonomic labels, (after US Naval Training Device

Center, 1972), to each operational subtask, using the skills and knowledges

comprising each subtask. Associated with each of the thirteen task catego-

*O ries in the taxonomy are three sets of learning principles which are related

to stimulus, response, and feedback aspects of these tasks (after Willis and

Peterson, 1961; and Micheli, 1972). For each subtask, ratings are given on

." how well a training device implements each of the relevant learning princi-

- ples: "-3" (complete violation of principle); "0" principle not implemented

'-. or vio)ated); "_" (optimal implementation of principle). The lowest ratings

* given to learning principles in each category (i.e., stimulus, response, and

feedback) are then averaged, to yield a T score for each subtask. In order

to scale T down to between "0" and "1", "3" is added to the averaged score,

and the sum is divided by "6".

As Wheaton, Fingerman, Rose, and Leonard (1976) pointed out, the determination of T values is rather conservative since only the poorest implementation of training techniques on a device is considered. In the TV-A procedure, a training device does not get credit for having a few, especially good

instructional features.
The operational equipment is assumed to make optimal use of training

* techniques, therefore, T i = 1.

- Outputs

"" Indices

Each of the TV-A analyses can be calculated and collapsed across subtasks to derive a separate index purporting to assess Transfer Potential,

. Learning Deficit or Training Techniques. Calculation of these indices may

* serve as a diagnostic function to locate deficiencies or assets in a training

" device. For a detailed discussion on these indices, the reader is referred

• . to Research Memorandum 76-16 (Wheaton, Fingerman, Rose, and Leonard, 1976).

*Such a presentation is beyond the scope of this effort.

Overall Device Effectiveness Prediction

The developers of the TV-A model tried to predict the Gagne, Foster, and

Crowley (1948) measure of transfer:

C- E %"T T" = _ _

C

In this classic transfer of training paradigm, both C and E are measures of

practice (time, trials, errors) required on operational equipment, in order

to meet a performance criterion. C represents a control group, which practiced only on operational equipment. E represents an experimental group,

which practiced on a simulator or training device first, then transferred to

operational equipment. The question which this transfer equation attempts to

answer is: How much training time (i.e., on operational equipment) can be

saved by providing practice on a simulator?

T is, therefore, a measure of savings. It equals the amount of training

time on operational equipment saved by practicing on a simulator first (C-E),

as a proportion of training time required when operational equipment alone is

used (C). T varies between -co and +1. In theory, the closer T is to +1, the

greater the transfer of skills acquired with a simulator to operational equipment.

Wheaton, Fingerman, Rose, and Leonard (1976) attempted to predict T directly by estimating values for C and E, and substituting these into the

original transfer equation. In order to do this, it was assumed that training time (regardless of experimental conditions) is a function of: (1) how

well a training setting represents the operational (real world) situation,

both in terms of tasks covered in training and fidelity of the training setting; (2) the difficulty inherent in the tasks which must be learned to some

criterion; and (3) the appropriateness (or value) of the instructional techniques used to train the tasks. The first factor is represented in TV-A by

two variables, a coverage variable, C (task communality), and a similarity

variable, S (physical and functional similarity). The second is represented

by the learning difficulty variable, D, and the third by the training variable, T. As Wheaton, Fingerman, Rose, and Leonard (1976) stated, "The time,

trials, or errors to a criterion on subtask i is assumed to be a linear

function of Ci x Si x Di x Ti" (p. 48).

8
. . r...

fi* Since the training setting for a control group, however, is the operational equipment, it is clear that all operational subtasks are covered by

the equipment (Ci = I), and the physical and functional similarity is

identical for each subtask (Si = I). It is also assumed in a TV-A application that, when training takes place on operational equipment, the instructional techniques used are optimal (Ti = i). These assumptions mean that the

amount of practice required by a control group (C) is determined solely by

the difficulty of each subtask (Di ) summed over all subtasks:

N

E Di * i=l

In order to estimate E for the experimental group, the amount learned on a training device must be subtracted from the amount learned on operational

equipment. Since a training device is assumed not to be identical to the operational equipment, the values of the coverage (Ci) and similarity (Si) variables will not always be "i", and must be estimated by the procedures just

discussed. Likewise, the training techniques employed to teach each subtask

V. are assumed to be less than optimal when a training device is used, Ti must

also be estimated. The amount learned on operational equipment is N

-D--

Therefore E is assumed to be equivalent to: N N - Di -iZCi x Si x Di x Ti .

Given these estimated values of C and E, the predicted value of f is

calculated by the equation: N

t Ci x SxDi xTi

"!1 Di

Summary

The Wheaton, Fingerman, Rose, and Leonard (1976) model purports to generate a prediction of transfer of training potential for training devices

based on an analysis of both operational and training equipment. The model

aggregates values for a series of factors assumed to be related to a device's

effectiveness. The factors identified are task communality, similarity,

learning deficits of the trainees, difficulty of each task to be trained and

the training techniques incorporated into a device. The final evaluation

index or figure of merit is a value ranging from "0" to "1.0", with values

approaching "1" indicating greater transfer potential and, therefore, greater

effectiveness.

In reviewing the TV-A model it is important to note that the theoretical

assumptions and specific methodology were based on previous efforts (e.g.,

Wheaton, Rose, Fingerman, Korotkin, and Holding, 1974, 1976). Some of these

assumptions may be questioned, and one might consider some elements missing.

TV-A, however, represents one of the most systematic and complete methods for

V assessing device effectiveness. In fact, Wheaton, Fingerman, Rose, and

Leonard (1976) have themselves begun a critical assessment and have suggested

directions for future efforts. For example, they recommended consideration

9

*. .% .-
of some important external variables. These include the amount of training

and practice provided and user acceptance of a device. While these considerations are external to a device, they represent variables which can influence device effectiveness.

An additional device related variable that may be considered for inclusion in a model is what Wheaton, Fingerman, Rose, and Leonard (1976) call Environmental Fidelity Analysis (EFA). The EFA would potentially account for

special or adverse conditions which may affect performance. These conditions

may include extreme temperature, reduced visibility, etc. (Wheaton, Fingerman, Rose, and Leonard, 1976). It might be possible to obtain judgments or

estimates of degradation of performance, probability of occurrence and subtasks affected by such conditions. A future model may, for example, include

an assessment or estimate of how well a device prepares for such contingencies. The difficulty, of course, is that a device developer may not be able

to replicate such conditions, assuming they are known, and a researcher may

not be knowledgeable about human behavior under the same circumstances.

A future revision of TV-A might include a less laborious approach to the

Training Techniques Analysis. Wheaton, Fingerman, Rose, and Leonard (1976)

suggested that perhaps this analysis could be conducted at the subtask rather

than the skill/knowledge level. This possibility becomes more realistic when

*z considering recent evidence of the utility of such approaches as cluster

analysis in ranking job related tasks (see Boldovici, Boycan, Fingerman, and

Wheaton, 1979; Wheaton, Fingerman, and Boycan, 1978). From such analyses,

it should be possible to form clusters of skills/knowledges or subtasks which

. can generalize to entire tasks. Applications of a Ti analysis, therefore,

uuld be conducted on a restricted number of subtasks, and thus make analysis

easier.

The evaluation methodology presented in this section remains to be

validated both in terms of predictive ability and the constructs within the

method. As one reviews the literature in this area, this criticism applies

to other revisions of TV-A as well as to alternative approaches. It has become apparent, and will "e discussed in the last section of this paper, that

evaluations of the variL . approaches have been long overdue and represent a

situation that must be remedied.

The methodology discussed thus far is based on a variety of assumptions;

some of which are accepted while others may require further justification.

TV-A assumes a linear relationship, for example, between the component variables and transfer of training potential of a device. This assumption is

presently accepted particularly in the absence of any compelling reason to

do otherwise. Another assumption made is that equipment similarity (i.e.,

fidelity) is monotonically related to transfer, and, therefore, a valid predictor variable. This is also related to the assumption that operational

equipment represents an optimal training setting against which a device may

be compared. There is presently no evidence to support these notions. An

opposing perspective may assume that training devices are typically built

with instructional features which are not present when operational equipment

is used for instruction. In addition, training devices can be built to simulate the range of conditions a trainee may encounter on the job; this may not

be possible when using operational equipment.

10
Criticisms which are not unique to the TV-A model include the detailed

input requirements (e.g., task analytic data) and the premise that device

effectiveness seems limited to transfer of training. In addition, all approaches reviewed for this paper mathematically combine a number of variables into a final, overall index. These criticisms remain unresolved and

must be addressed in the near future.

TRAINVICE-B (TV-B)

. Overview

- . The TRAINVICE-B (TV-B) model assumes that a device is the appropriate

S"medium for training based on the media selection decision procedures specified in the Training Device Requirement Documents Guide (1979). Within the

media selection decision procedures, a training developer previously analyzed

*and organized tasks, skills and knowledges, and objective data formulating a

training device concept. The TV-B approach is purported to insure that established training requirements, incorporated into a device, were emphasized.

TV-B provides an approach to analyze and evaluate the effectiveness of a

training device, typically in comparison to alternative device concepts or

already existing devices. Applying the TV-B approach results in an effectiveness score for each alternative device concept, which is then used to

decide which concept should be developed further.

The TV-B approach, therefore, is embedded in a series of administrative

procedures designed to establish the need for a device, determine if a device

which potentially may serve a training function already exists and to evaluate either existing devices or device concepts in terms of effectiveness.

For example, in deciding whether a device is an appropriate training

medium, a developer would have already collected information regarding tasks,

task elements, and controls and displays. The application of TV-B, in effect, becomes a trade-off analysis, because a device is not expected to meet

all task training requirements. To the extent that a device does not address

all the requirements, a developer is provided with a methodology to assess

alternative concepts.

The TV-B methodology is similar to the TV-A approach. A rating of the

-. correspondence between the operational equipment and a training device is

combined with an index of the extent of tralaing required and ability level

of the trainees. The product of these values becomes the training device effectiveness index. In TV-B, however, when an existing device is compared to

a training concept or requirement, the effectiveness index may be adjusted

for providing additional training beyond that required. The assumption is

that training additional skills represents unnecessary costs which lead to a

loss of effectiveness.

Procedures

The TV-B methodology allows values to be assigned to components which

comprise two basic subdivisions: (1) device characteristics and (2) personnel

T>' , 11.

S., : , " "" . :.;/T?:. . . ; > , . .,,: ; : ,
and training requirements. These sub-divisions are further divided into the

following components:

Device Characteristics o Task Commonality

o Physical Similarity

-'- o Functional Similarity

Personnel & o Skills and Knowledges Requirements

Training Requirements o Task Training Difficulty

Values for these components are combined to form an index of training device

effectiveness.

The information required to perform a TV-B analysis includes the:

1) list of tasks and elements (i.e. sub-tasks) to be trained

(operational tasks);

2) tasks and task elements which can be trained with a particular

device. This list includes task elements covered by a training

device, which are not contained in the training objectives (i.e.

unique elements);

3) skills and knowledges required to meet the training objectives;

4) controls and displays used to perform the tasks in the

operational setting; and

5) controls and displays in the training device.

Task Commonality Analysis (TC)

The Task Commonality (TC) analysis in TV-B is different from the C analysis in TV-A. In TV-A, each subtask is given a"1" or a "0" depending on

whether it was covered by a training device. In TV-B, a TC-value is determined for each task, by rating whether or not task elements which require

training are covered on a device ("1i" covered, "0": not covered). The TC

value for a task is calculated by adding all task element ratings, and

* dividing this sum by a combination of the total number of required task

elements plus the number of task elements which do not require training but

U. ~are still covered by the training device (i.e. unique elements).

Physical Similarity Analysis (PS)

In the Physical Similarity (PS) analysis the controls and displays on a

training device and on the operational equipment are compared in terms of

their appearance, size, location, etc. The comparison is made only for device

characteristics which are directly involved in performing those task elements

which require training. Each control or display on a training device is rated

on the degree of physical similarity (i.e., fidelity) between it and the

corresponding control or display on the operational equipment. The rating

scale, used for this purpose, ranges from "0" (missing) to "3" (identical).

12

• • . • - • . ° • -. ° , . • ° • . . . . . . o . . . . . . .. . . . ..
The scale values and criteria for judgments are very similar to those in TVA. There are, however, changes in phrasing; e.g., "small noticeable differences" in place of the more traditional and, perhaps, technical "Just noticeable differences" (see Appendix).

In order to derive a Physical Similarity index for each task, the ratings

given to controls and displays on a device are totalled. This sum is then

divided by a combination of "3" times the total number of required controls

*and displays plus the number of "unique" controls and displays. The unique

pieces of equipment on a device are those used for task elements or skills

which are associated with the task in question, but do not require training.

Thus, the resulting index varies between "0" and "I", representing the

physical similarity adjusted for extra or "unique" equipment.

*Functional Similarity Analysis (FS)

The Functional Similarity (FS) analysis in TV-B, like that in TV-A

compares the controls and displays of a training device to those in the

operational equipment in terms of amount of information conveyed from or to

the human operator. Just as in the PS analysis, each of the "required" controls or displays relevant to a particular task receives a rating from "0" to

"3". The rating scale used, though similar to that in TV-A, includes less

technical language. A "2" on the TV-A scale, for example, means that the

amount of information in the operational and training settings are "within one

log unit of each other." The corresponding description in TV-B is "the

number of states in the training situation is less than half of the number of

states in the operational setting." The only time the two scales are equivalent, is when there is less information in a training setting. The log2 in

TV-A can also mean greater information in a training setting. This distinction, however, is consistent with the TV-B approach in adjusting for unique

skills.

In order to calculate the functional similiarity index for each task, the

ratings given to all controls and displays on a device are summed and the

total is divided by the number of required controls and displays plus the

unique ones. This results in an index ranging from "0" to "". The last

operation, (i.e. the inclusion of the unique displays and controls in the

denominator) is the cost adjustment for extra training device features.

Skills and Knowledges Requirements Analysis (SKR)

In TV-B, there are two separate preparatory analyses which correspond to

the Learning Deficit Analysis in TV-A. In TV-A, the Learning Deficit variable

represents an estimate of how much the trainees have to learn, weighted by the

amount of time it takes to train them to overcome a deficit, on the

operational equipment. The procedures involved in both TV-B and TV-A are

performed independently of the characteristics of the training device under

evaluation.

In Skills and Knowledges Requirements Analysis (SKR), each skill or

knowledge required to perform a task receives two ratings. The first rates

the level of proficiency trainees have before training. The second rates

13

... .. . ..
t,. ,'.

the proficiency level required after training. The rating scales used range

from "0" (no experience) to "4" (complete understanding) and are almost

identical to the Repertory and Criterion Scales used in TV-A (see Appendix.)

The difference in before and after proficiency levels is determined for each

skill or knowledge by subtraction. For each task, a SKR index is calculated

by taking the mean of the proficiency differences and scaling it down to

between "0" and "1" by dividing by 4.

Task Training Difficulty Analysis (TTD)

The TTD is quite different from the corresponding procedures in TV-A.

The first step in this analysis is to determine how much time would be

required to train the most difficult task element of all those in the training

objectives (i.e., across all tasks). Training time here means time to train on the operational equipment. A TTD index is derived for each task by rating

each required task element on how much time is needed to train it on the

operational equipment, relative to the training time required by the most

* -. difficult task element. The ratings are made using a scale which ranges from

1"0" (requires no training) to "4" (requires as much time to train as the most

time consuming task element) (see Appendix.) The index given each task is the

average of the difficulty ratings given each task element, scaled to between

110"I and "11".

Index of Training Device Effectiveness

The analyses just presented are used to calculate an overall index of

effectiveness for a training device or concept. The values for TC, PS, and FS

are summed and divided by 3. This value represents the degree of correspondence between a training device and the operational equipment. Next, the SKR

and TTD values, for each task, are added and divided by 2. This value represents the amount of training required. In order to calculate the Index of

Training Device Effectiveness, the value representing the degree of correspondence and the amount of training required are multiplied for each task.

These products are then summed with the final index obtained by dividing by

the amount of training required (i.e. SKR + TTD).

2 .:

The final index formula is: TC + PS + FS X SKR + TTD

i=3 2 T'

N 'SKR + TTD

i-El 1 " 2

2 )

The TV-B model attempts to adjust the final index by a correction factor

which reflects a loss of effectiveness due to unnecessary cost. This

adjustment factor is calculated as:

# of Required Tasks

(I of Required Tasks) + (0 of Tasks Unique)

~This factor accounts for capabilities in a device that are not required. The adjustment factor is applied by multiplying it to the final index. This

14
adjustment is assumed not to be required when assessing theoretical device

* - concepts, only existing devices.

Summary

TV-B is similar to TV-A in terms of many of the components which enter

into the overall training device index. Two major subdivisions comprise the

Honeywell approach. These include measures that assess the degree of correspondence between a device and operational equipment for which it was de-

* veloped. The degree of correspondence assessment is similar to TV-A in that

Task Commonality, Physical and Functional Similarity are determined. These

values are later combined with an index of the amount of training required

for a set of tasks; again, similar to TV-A.

A major distinction between TV-A and TV-B is that TV-B does not include

an assessment of the training techniques incorporated in a device. That is,

there is no measure of the appropriateness of the instructional features in

relation to accepted learning principles. Another difference is that while

TV-A adjusts the overall effectiveness index for failing to cover tasks on a

device, TV-B additionally penalizes a device for including additional instructional features beyond those required. The rationale of this latter adjustment is the assumption that a decrease in training effectiveness results

when unique or unnecessary skills are taught. The rationale continues into

cost considerations as well. That is, additional training in non-required

: skills costs more, and therefore is undesirable. These assumptions and related adjustments may be suspect and unwarranted. Without an assessment of

adherence to accepted instructional or learning guidelines there appears to

be little basis for such a penalization. In fact, there may be instances

where additional skills, beyond those required, may enhance overall transfer

of training and this may go completely unrecognized by an evaluator.

The TV-B approach, however, does emphasize the relationship of effectiveness with cost considerations more than the TV-A model. This is particularly relevant when the objective is to assess the total long-term training

cost in relation to effectiveness as the Guidebook indicates. Indeed

rarely does device development proceed without cost considerations in terms

of resources required for facilities, equipment, instructional material, personnel, students, supplies, etc.

4. Finally, TV-B, like TV-A, relies on a number of assumptions which include linearity and method of mathematical aggregation. These, along with

other issues pertaining to reliability and validity, are major concerns and

will be discussed further in a later section.

15
TRAINVICE-C (TV-C)

Overview

A revised version of the TV-A approach, referred to in this paper as

TRAINVICE-C (TV-C) was developed to increase the practicality and flexibility

of a device effectiveness model (Narva, 1979a; 1979b). TV-C attempted to

provide a means for answering three questions about a training device:

"what", "why", and "how".

The "what" question addresses what should be represented in a device.

Two judgments are required in the answer. The first refers to the requirement for an activity to be incorporated into a device. The second refers to

whether the device actually covers an activity.

The "why" question tries to uncover the reasons for including training

activities on a device. The two stages of this issue include training criticality, or the level of proficiency required at the conclusion of training,

and training difficulty, or how hard it is for a trainee to reach that proficiency level.

The "how" question pertains to the physical and functional characteristics of a training device. That is, TV-C assesses how well displays and

controls (i.e., physical characteristics) follow accepted instructional or

training guidelines, and the trainer's requirements. In addition, the "how"

refers to the extent functions of displays and controls (i.e., functional

characteristics) adhere to guidelines on instruction. Judgments are made for

every skill or knovledge required on a training device, with values corresponding to these judgments substituted in a formula designed to reflect the

percentage of maximum transfer which would be fostered by use of a particular

training device.

16
Procedures

Coverage Requirements Analysis (CR)

The first analysis performed in TV-C is the Coverage Requirements

Analysis (CR). The procedure consists of assigning a "1" or a "0" to each

skill or knowledge (from the operational task analysis), depending on whether

or not it should be covered by a device. In other words this analysis serves

to determine which skills and knowledges warrant training. This screening

process already existed in TV-A, as part of the Learning Deficit Analysis (a

CS rating of "O")(Narva, 1979a, 197gb). TV-C simply highlights this issue for

separate and initial attention. In either case, however, a high premium is

placed on the judgment of a training analyst.

Coverage Analysis (C)

The Coverage Analysis (C) compares the skills and knowledges in the

operational setting with those covered by a training device. Just as in the

Task Communality for TV-A and Commonality Analyses for TV-B, a C value of "1"

is assigned to each operational skill which is represented, a "0" when not

represented in the training setting. The only difference between the TV-C

approach to coverage and methods used in earlier versions of TRAINVICE is TVC, ratings are made for each skill, whereas in the others the rating is made

for each subtask.

Training Criticality Analysis (Ci )

All skills receiving a rating of "I" in both of the preceding analyses

are then subjected to the Training Criticality Analysis (Ci). Each skill or

knowledge is rated on the degree of proficiency which will be required after

training (not mission criticality). The scale used to make this rating is

almost identical to the Criterion Scale used in the Training Deficit Analysis

for TV-A (See Appendix). The only difference is that the "0" value was

dropped because a rating of zero proficiency has already been taken into

account by the Coverage Requirement Analysis. The values for the Ci variable

range from "11" to "4".

Training Difficulty Analysis (D)

In the Training Difficulty Analysis (D) each skill receives a rating,

* from "1" (minimal or none) to "4" (substantial), on the degree of difficulty

to learn that skill to required proficiency levels. Aside from the difficulty

inherent in a skill itself, a rater must also consider the proficiency level

of the trainees before training and that required after training. In essence,

this analysis greatly simplifies the TV-A procedures for deriving the weighted

learning deficit; especially the rank ordering of subtask difficulty.

17
Physical Characteristics Analysis (PC)

In considering the equipment on a training device, (i.e. the device characteristics), the TV-C approach is quite different from that in the earlier

versions of TRAINVICE (i.e., physical and functional similarity). As alternatives to equipment similarity ratings, the TV-C physical and functional

characteristics analyses represent attempts to have a training analyst assess

more directly "how" a device will train skills. In this sense, the device

characteristics analyses of TV-C resembles the training techniques analysis in

TV-A.

The Physical Characteristics Analysis (PC) addresses the appropriateness

of the physical equipment supporting the training of each skill. Each skill

associated with the controls and displays is translated into a generic haracteristic (e.g., Stimulus Capabilities: Visual Form - Visual Alphanumeric,

etc.). The generic characteristics recommended are those contained in the ISD

model (Braby, Henry, Parrish, and Swope, 1 975). Each of the generic characteristics of the cue or response related to a display or control is rated on

how well it follows available guidelines. The rating scale used ranges from

"0" ("not adequate") to "3" ("outstanding"). The physical characteristics

score, for each control or display, is the sum of the ratings given to each

relevant generic characteristic. Similarly, the physical characteristics

score for each skill is the sum of the scores given to each of its associated

controls and displays.

In order to assist in making the physical characteristics ratings, TV-C

refers a user to a series of learning guidelines (ISD). To use these guidelines, each skill must first be classified as belonging to one of ten behavioral categories (e.g. identifying symbols, detections, etc.). For each of the

behavioral categories there is an associated set of learning guidelines.

Narva (lgi ) cautions about the lack of specificity of the ISD guidelines.

These were originally intended to assist in the selection of instructional

media. For this reason, the user must be selective in the application of the

learning guidelines. Again, it must be emphasized that use of the guidelines

does not directly generate physical characteristics ratings, it merely alerts

the user to some of the general behavioral considerations associated with each

of the behavioral categories to which a skill might belong.

Functional Characteristics Analysis (FC)

The Functional Characteristics Analysis (FC) attempts to assess how the

physical characteristics of a training device are used. The first step in

this analysis is to place each skill in one of the ten behavioral categories

(as in the PC analysis). A user then refers to the set of ISD Learning

Guidelines associated with each behavioral category and selects those appropriate to the specific skill under consideration. Ratings are given to a skill

on how well each of the relevant guidelines are implemented or used in a

training device ("0", not adequate; to "3", outstanding). The FC value given

to each skill is the sum of the ratings made on each of the associated

guidelines.

18

", <. V .S.. .. ..- f. '.. .. ' ,JJ. -J -. '' ~~~ j '' jf ~ j f.
Index of Predicted Training Effectiveness

The calculation of the TV-C index of effectiveness was designed to

represent the percentnge of maximum transfer, the procedures to combine the

components consist of a ratio in which the various values given to a device

are combined in the numerator. The denominator is a combination of the

maximum possible ratings which could have been given. The TV-C formula is:

(CR x C x Ci x D x (PC + FC)).

(CR x c x Ci x D x (PCmax + FCmax))i

where:

CR Coverage Requirements Score

C Coverage Score

C i Training Criticality Score

D Training Difficulty Score

PC Physical Characteristics Score

FC Functional Characteristics Score

PCmax Maximum Possible Physical Characteristics Score

FCmax Maximum Possible Functional Characteristics Score

The form of the equation given in the first TV-C report (Narva, 1979a) shown

above, was modified slightly in a second report (Narva, 197gb) to the

following:

(CR x C xC x D x (PC + FC)).

(CR x C x 4x4 x (PC +FC

max mxi

The value of "4" substituted for the criticality (Ci) and difficulty (D)

variables in the denominator is simply the greatest value either of these

variables could have. Both of the above equations yield indices which range

between "0" and "1". A larger index value (i.e., closer to 1) presumably

indicates a greater potential for transfer to the operational equipment.

Summary and Critique

Like the other versions of TRAINVICE, TV-C attempts to assess the training

transfer potential of a training device by assigning values to a variety of

judgments about a device.

Essentially, there are three major subdivisions within TV-C, an input, a

training analysis, and a device characteristics analysis. The inputs include

the operational and training requirements which are derived from a task

analysis of each situation (i.e. operational and training). The training

19
analysis is an estimation of the required level of proficiency and difficulty

to arrive at that level for each trainee. Device characteristics analyses

include an evaluation of the physical and functional aspects of components

incorporated into a device as these adhere to accepted instructional or

learning priciples.

In two papers, ARI RM 79-6, 79-7, Narva (197ce, 1979b) outlined an

extensive modification of the original TRAINVICE predictive model. The

procedures described in these two papers are identical; only the calculation

of the index was modified in the second paper (i.e., Narva, 1979b). The most

striking difference between TV-C and the earlier models is the omission of an

equipment similiarity (fidelity) analysis. The training techniques analysis,

which had been dropped in TV-B was reintroduced in TV-C in the form of two

separate analyses (physical and functional characteristics analyses). TV-C

also contains a coverage requirement (or media selection) analysis, not

included in TV-A, or TV-B. The procedures and rating scales used in the

various preparatory analyses were almost completely changed in TV-C. Also,

the level at which these analyses are performed is at the individual skill

level, not subtask. Considerable changes were also made in the procedures

used to calculate an overall index of effectiveness.

TV-C included a Training Criticality and Training Difficulty analysis as

a weighting factor for required skills and knowledges. A skill or knowledge, therefore, which is required at a high level of proficiency, in addition to

J% being difficult to learn is assumed to have more significance than one requiring a lower proficiency and which is easier to learn. Given two devices under

evaluation, for example, one covers an important skill while the other does

not. The evaluation model was originally intended to penalize a device in

such a situation. TV-C fails to accomplish this. A "0" C value for a skill

causes both the numerator and denominator to go to "0" for a skill not

covered, although required. The result is as if that skill never existed. As

will be discussed shortly, TV-D corrected this situation.

The terminology of Training Criticality Analysis is somewhat misleading. The word Criticality seems to suggest the notion of importance, either

in the mission or training setting. As presented earlier, the Ci analysis

addresses the required level of proficiency for trainees, and has nothing

directly to do with criticality.

The criteria for a user to make judgments about each of the analyses

appears to be too vague. The scale for the D analysis, for example, is:

1 minimal or none

2 some

. 3 :much

4 = substantial

These descriptions of the rating scale may reduce the reliability of the

application. That is, because of a lack of specificity of definitions,

+- judgments by different users may vary according to individual interpretations. This possibility exists whenever scales of this type are used,

however, the more specific the criteria for assigning values the less likely

20
777

differences in interpretation can occur. This leads to a further restatement

d of the need to validate the methodology both in terms of construct and predictive validity. This issue will be discussed further in a latter section of

this paper.

TRAINVICE-D (TV-D)

Overview

In a project to develop a user's guidebook for TV-C, further revisions

were made to the evaluation model (see Swezey and Evans, 190; Evans and

Swezey, 1B). Despite their differences, TV-C and TV-D are almost identical

in the variables or model components considered and in the procedures used to

estimate these variables.

Two general uses of this model have been identified as predictive or

prescriptive applications. A predictive application is used when existing

training devices are available and a user wants to evaluate (or predict) their

effectiveness. In its prescriptive mode, the model is purported to assist

device developers in making design decisions in the early concept stages.

Components are applied either separately or in combination. When components

are combined, an overall index of device effectiveness is derived. The

overall index or separate components analyses are only of value when two or

more devices are under evaluation. In one sense this restriction is the

result of the overall index having no intrinsic or absolute meaning. In

another, each of the components can be used as a comparative assessment to

identify deficiencies in a device under evaluation.

The components of this model are:

o Coverage (C)

o Training Proficiency (P)

o Learning Difficulty (D)

o Physical Characteristics (PC)

o Maximum Possible Physical Characteristics (PCmax)

o Functional Characteristics (FC)

o Maximum Possible Functional Characteristics (FCmax)

The formula for executing TV-D is:N 'PC+FC (

NJi i PCmax m+ + FCm) FCmax ) C x P x D)

1 P x
..) j.. .' ,.. *4". , ; r .. .- , . . I . . •;. ,..... . .-. *. . *. . -. ...... . ... . - ,- ..- .,. ..

A device evaluation actually begins by determining whether training is

required on the skills and knowledges needed for performance on the operational equipment. Once the training skills or knowledges have been identified, a user then conducts analyses using the model components. These are

briefly described below.

Procedures

Because TV-D is a direct derivative of TV-C, it addresses the same

"What", "Why", "How" questions TV-C does. As will be discussed shortly, some

changes have been made.

Coverage Requirements Analysis (CR)

Although not formally used in the overall index formula, the Coverage

Requirements Analysis (CR) helps determine which skills or knowledges required

in the operational setting should be represented on a training device.

Working from a consolidated list of skills and knowledges, a user decides

whether each skill or knowledge should be covered by a training device. If

the decision is "yes", then a value of "1" is assigned to that skill; a "no"

decision receives a "0".

Coverage Analysis (C)

If a skill or knowledge is required, a user must then decide if that

skill or knowledge is actually represented. A Coverage Analysis (C) value of

"1" indicates that it is, while "0" indicates it is not. If the analysis is

conducted early in a device development phase then a required skill (i.e. CR

"1"), which was not originally covered in a device design (i.e. C = "0 ), can

be included. The effect of failing to cover a required skill is reflected in

a lower overall index for a particicular device.

Training Proficiency Analysis (P)

This component assigns a value corresponding to the degree of proficiency

which a trainee must attain for each skill or knowledge subsequent to training

* on a device. The Training Proficiency Analysis (P) is conducted on each skill

or knowledge which received a CR value of "1", even if one device in a comparison failed to cover (i.e., C "0") a particular skill or knowledge.

A four point (i.e., "1" to "4") rating scale is used to assign a P value,

where "1" corresponds to a level requiring limited knowledge. When expert

levels of knowledge are required, a P value of "14" is assigned. The P values

are then summed across all skills and knowledges.

Learning Difficulty Analysis (D)

6The Learning Difficulty Analysis (D) specifies the degree of learning

-'U difficulty associated with attaining a required skill or knowledge. Several

22

5-.

%. * 1. 9 % ~ *5.* \ ' 1 '~* ~ . ~* ..
factors have been identified which enter into a user's decision in assigning a

D value. These are the:

o level of skill/knowledge proficiency to be

attained by a trainee

o entry-level capabilities of a trainee

(i.e. pre-training on the skills or

knowledges.

o level of learning difficulty typically inherent

in a skill or knowledge

In making D judgments, a user assigns a value ranging from a low of "I" to a

high of "4". The higher a D value the more difficult a skill or knowl-edge is

to learn. Like the P analysis, D values are assigned only to skills or knowledges which have been determined to be required (i.e. CR=1) and then summed.

Physical Characteristics Analysis (PC)

This is the first of two analyses which are referred to as Device

Characteristics Analyses. In other words, attention is now focused on analyzing displays and controls on a device. The Physical Characteristics Analysis

(PC) assesses how well the physical characteristics of a device support

guidelines or principles of good instruction. A separate PC analysis is

conducted for each device under consideration.

In conducting a PC analysis, a user must first determine the type of

behavior that is required to accomplish a particular skill or knowledge. Each

skill or knowledge is assigned to a behavioral category which coresponds to

the type of performance required by a trainee. These behavioral descriptions

were adapted from the U.S. Army Interservice Procedures for Instructional

S•Systems Development (TRADOC Pam. 350-30, 1975). Next a user decides which

instructional practices are applicable for developing the type of behavior

associated with a skill or knowledge, which are listed under each behavioral

category. These instructional practices or guidelines represent a standard

against which each device will be evaluated. Because these guidelines correspond to skills or knowledges, they remain the same for each device under

evaluation.

A user then identifies the Generic Stimulus and Response Characteristics

for each display and control which correspond to particular skills and knowledges. That is, a user must identify the stimulus characteristics of displays and learner response modes. The list of possible stimulus characteristics (i.e. capabilities) and response modes are those presented by Braby,

Henry, Parrish and Swope (1975). The PC analysis concludes by assigning a

value or rating on how well each generic characteristic of a display or control supports the good instructional practices identified earlier. Values of

the PC analysis range from "0", extremely deficient in implementing the guidelines, to "i", implementation is highly proficient, for each skill or

knowledge. The total PC score then becomes the sum of the values assigned to

each skill or knowledge.

23

v..-'j. * * .,* ~ - 4- ., . 4 v "
%. .iMaximum Possible Physical Characteristics (PCmax)

skill The maximum possible physical characteristics (PCmax ) value for each

skill or knowledge is simply three times the number of applicable generic

stimulus and response characteristics.

Functional Characteristics Analysis (FC)

The second device characteristic analysis is the Functional Characteristics Analysis (FC). The FC analysis is similar to the PC Analysis in that it

assesses how well the functional elements of a training device follow guidelines for good instructional practice. Skills and knowledges are, again,

compared to the behavioral categories, and good instructional practices under

each category. These instructional guidelines are now identified solely for

functional and not physical characteristics. Again, these form a standard to

which the functional worth of displays and controls are compared. In

completing the FC Analysis, a user rates how well each display and control,

corresponding to a skill or knowledge, implements the functional guidelines

for good instructional practice. The scale used ranges from "0" extremely

deficient implementation to "3" highly proficient implementation of the

guidelines for each skill or knowledge. These values are summed for all

skills and knowledges under consideration (i.e. CR "1").

Maximum Possible Functional Characteristics (FC ) max

Like the PC ax' the maximum possible functional characteristics score is

three times the total number of applicable functional guidelines.

Index Calculation

The calcualtion of the final index is completed by simply substituting

the values of each analysis discussed above and carrying out the operations in

the formula: N PC + FC C x p x D) i= PCmax + FCmax)

ii xP D)

The resulting index will be a number between "0" and "1". As this value

approaches "1", the better training transfer capability of a device. The

* overall index, however, only has value when comparing two or more existing

. devices or device concepts.

Summary

While quite similar to previous approaches, some changes have been instituted. Training Proficiency Analysis (P) was formerly called Training

24

/.2

.h b .s.P•
Criticiality (Ci) in TV-C. The term "Criticality" was considered misleading,

perhaps suggesting importance of a skill, which was not the case.

The principle difference between the TV-D formula and that of TV-C is the

removal of the coverage variable from the denominator. In this way the credit

or penalty supposedly given to a device for covering or failing to cover a

particular skill is weighted by the Training Proficiency and the Learning

Difficulty scores. That is, an overall index of effectiveness would be

enhanced more for covering skills that require a high degree of proficiency

and are difficult to learn, than for covering relatively trivial skills.

Similarly, when a skill is not covered, the degree to which an overall effectivness index is decreased is weighted by the proficiency and difficulty

scores for a skill. In the TV-C formula, the credit given for coverage of a

skill was weighted by criticality and difficulty, however, lack of coverage

was not penalized at all. The presence of the coverage variable in both the

numerator and denominator would cause both to equal zero for an uncovered

skill. Thus, in TV-C, each skill not covered by a training device neither

*. contributes to nor takes away from an overall effectiveness index.

Four of the rating scales used in the preparatory analyses for TV-C were

modified in TV-D. These are the scales used in the Training Difficulty

Analysis, Training Proficiency Analysis ("Criticality" in TV-C), and Physical

and Functional Characteristics Analyses. In all cases, wording changes were

made in the attempt to provide more guidance to the user than had been

available in TV-C (see Appendix). There were no changes, however, in the

numerical properties of the scales.

The Physical and Functional Characteristics Analyses contain additional

changes in guidance given to users. The ten behavioral categories (from ISD)

used in TV-C were given expanded definitions accompanied by examples. TV-D

incorporated new learning guidelines, associated with each behavioral category, which were modifications of those already in the ISD. Moreover, each

learning guidline was identified with a "P", "F", or "P/F"; to indicate

whether a particular guideline was relevant to analyzing the physical characteristics, the functional characteristics, or both.

Several issues of concern have evolved regarding the application of TVD. Actually these issues appear equally valid for the earlier models as

well. The first is the manner in which values for the various components are

aggregated into a single index. The components of TV-D appear to form a

series of fractions, all based on separate criteria. These then become accumulated or summed in violation of basic rules for such addition. In other

words, there is no attempt to find a common denominator.

A second concern is that different guidelines on "good instructional

practices" are used for the PC and FC analyses. Further, the procedure for

designating the PC and FC values is cumbersome, both of these issues seem to

increase the possibility of poor reliability in assigning values.

By necessity, it seems that a long list of skills and knowledges are

required to apply TV-D. Once these are identified, a series of additions and

multiplications is required. Again, reliability seems to be vulnerable, if

for no other reason then arithmetic errors. In addition, a user must begin a

TV-D analysis with a consolidated list of skills and knowledges derived from

25
the list of all skills and knowledges required in the operational setting.

Construction of the consolidated list requires a user to eliminate from

consideration those skills and knowledges that are repeated on more than one

task or subtask. A TV-D index, therefore, is desired on only a selected

number of skills and knowledges, with no implication for a particular skill

being repeated. Perhaps a logical argument can be made that if a skill or

knowledge appears in more than one task, then that repetition should indicate

some degree of importance. Yet, in executing TV-D, all skills begin as equal

with only proficiency and difficulty as primary considerations or weighting

factors.

Another issue of concern is the reliance on TRADOC Pam. 350-30 as

providing "good instructional guidelines." These guidelines were developed

for training programs in general and not for training devices. This

application in a device effectiveness method is suspect.

,-%

, . .-..
III

Summary of the Models

Input

All four TRAINVICE models require task analytic and equipment information

as input. The models vary somewhat in the detail of the task information

required for input, as well as in the task taxonomic level at which variable

values are estimated (e.g., task-by-task) (see Table 1). There are two types

of equipment information required: physical (i.e. size, location, etc.)

characteristics of the controls and displays and functional chracteristics

(operation and use of the equipment). The four models are comparable in the

amount of detail required in the physical information. There are, however,

differences among the models in the level of resolution required in the

functional information. The two models which involve equipment similarity

analyses (TV-A and TV-B) require specification of the amount of information

(in bits) transmitted between the human operator and the controls and displays. The two models without similarity analyses (TV-C and TV-D) may need

more general accounts of the stimuli (or cues) supporting the behavior and the

types of responses required.

TABLE 1

Models Input Resolution Level of Analysis

TV-A Sub-task Sub-task

TV-B Task element; Skill Task

TV-C Skill Skill

TV-D Skill Skill

Preparatory Analyses & Model Variables

The four TRAINVICE models involve several general types of preparatory

analyses. Table 2 shows the commonalities among the models in terms of these

analyses. Each kind of analysis produces an estimate of a value for a particular variable.

In the coverage and communality analyses, a "I" or "0" is used primarily

to penalize for non-coverage of skills. Penalization issues are most relevant

to each model's equations, and will be discussed below.

The class of variables in Table 2 called "Learning" variables concern:

1) the amount of increase required in the proficiency levels of trainees, and

2) the amount of difficulty inherent in training each task. In TV-A both of

these are combined into a Weighted Learning Deficit score. In other words, an

estimate of incoming trainee skill level is subtracted from a criterion proficiency level. This difference is then weighted by the ranked difficulty of

training that particular skill. In TV-B, the difference in proficiency levels

(Skills and Knowledge Requirements) is estimated in a similar way to TV-A.

27
u u

>E

Ia w

>1 c, 1 Ia 1 i () n m .-4 C (

V U >, 0 U.

0W W :3 -4 a) l.4.4-4 4d. Co WOU *-4 > Co1U v,. >oU I I ca I VO44W CO " - c 41

04( w4.4 c 0I 4 4) vd4-~ X-Co> = Uc:3r_

u u)4C md Co 1 V0 a. J.r4 co rz. -

0-4

z -~co Co

ca a) -44 C-) u U U

.0 0 u I 1 u) 1 m4 CL -

0 I

co E-U 0) I 1 $4 co p .C 44t @3W (a.4 I tO -4 U) -4@ (L) dc

to $4 w01 0 w I H ( 1~- d-?4 (A CoL) Oi.00 4Q

tv P - >.43 CW > I C: "4~f >1II-OW -OcW -. 40AWc (

Cod 0) V -4 (L) 4 -H 4- 14 1-H U-4 -4 0 w 1d4-4 U 1.p 4 > cr >Cmo C-H-4co I Cd 44 CO m.Co~ d0os

0 a)0 Od $W $4.,"4 r .0c 3 r'.d

w).C C)Co tvC E-4 0CO 04U-v4to z.4-

U)4 to4 0)1

V .0 E4 4 z

* ~.0 1.

E- > ~U

E1 -4(3 MH (0-J1 ca Co -.4 ,-. 4 C-4 -aH .. 0O I 0 r-.

a) v-4 Co 0 o~ 01-.X i4 X *U@3 4 I C: A

U) (A~ "4 w1 u~ V.4C 0 - @DC V ( I Sd-.-44 I) -4C >, 0 or a,44 v4,4) 044...01 01 .4 I

C o o = 4 r- 0 O : - 4 0 ~ w 0 1 to d - A

E-4 u0 PW4 -HP4W H 2. w- I -4 AE4V -H

-H.0 1 0 4 1-1 0 -4 4$- U$co P64

U) >

E-4 Ic to 4 i

0 4W cc 4 VA 0 U $4- 43- 7V

0 C ;0% 0 0 C 0 0 t '44 Co du c

-4 U O VE44 @4W34 

cc $4 u A t

to$4 > a* C: >%r Co t0 

U4 CO 0 'C U 0-a4).J14

28

-~~~o 4 q -* %o- 
Training Difficulty is estimated more simply by a rating scale, instead of a

ranking procedure. The two values (proficiency requirements and training

difficulty) are averaged in the TV-B final equation. In both TV-C and TV-D,

the values of the proficiency and difficulty variables are estimated with

'''"- rating scales and are kept separate throughout subsequent calculations.

The equipment similarity and training techniques variables are the only

model components which are concerned with the features of a training device

and how well they will support training. As can be seen in Table 2, the

models vary widely in their emphasis, or lack of emphasis, on each of these

variables.

The one model which addresses both variables is TV-A. Here, equipment

similarity has two components: physical and functional similarity. Values

are assigned to each and are averaged for an overall Similarity score. In

order to derive the training techniques score, a user first categorizes each

subtask according to Braby's, et al. (1975) task taxonomy. The task category

then refers the user to a special set of learning principles for that category

(after Willis and Peterson, 1961; and Micheli, 1972). The principles concern

stimulus, response, and feedback aspects of equipment. A conservative estimate is made regarding the implementation of these principles by a device,

* which then generates a value for the Training Techniques variable.

In TV-B, training techniques are ignored, with an average of physical and

functional similarity scores as the only predictor variable. The analysis

used to generate the Similarlity score is almost identical to that in TV-A.

TV-C and TV-D abandon equipment similiarity as separate analyses. It is

hard to disagree with this because there is little literature supporting the

assumption of a general, monotonic relationship between equipment fidelity and

training effectiveness (a minimal criterion for the selection of any predictor

variable.) The traditional assumption of such a relationship has undoubtedly

been based on approaches to transfer of training such as Osgood's (1949). The

problem with such an assumption, in the context of training devices, is that

it must lead to the conclusion that the best device for training is the

operational equipment itself. Put differently, this approach assumes that the

cues necessary to maintain skilled performance, on the operational equipment,

,ji are sufficient and in fact optimal to support learning.

*8 The level of stimulation present on a training device however, may have

different effects on various kinds of learners. Skilled performers, for

example, have already learned to use to their advantage all the relevant cues

available in the operational environment. To a novice, however, the stimula-

,I"-. tion presented by the operational environment may be, in large part, noise;

(i.e., a source of distraction), therefore a hinderance to learning. Somei= times, it may be desireable, therefore, to reduce the number of cues available

(i.e., lower fidelity) during initial training. While in other situations, it

may be desireable to increase the amount of information presented in the

training environment in order to augment feedback and knowledge of results.

In yet other simulations, compressing the time frame of a task series may

enhance training.

While presently there may be insufficient knowledge regarding relationships between fidelity and training effectiveness to warrant its use as a

29

0+ ,

r.='
predictor variable (without qualifications), equipment similarity cannot be

ignored. In the solution adopted by TV-C and TV-D, fidelity is considered

only in the context of fairly specific task domains, not as an end in

itself. TV-C and TV-D adopted what is in essence an amplification of the

Training Techniques Analysis in TV-A. The modified analysis (in TV-C and TVD) directs a user to different sets of learning principles for different

skills. Using these principles, a user assesses how well the physical and

functional characteristics of a training device support training. The learning principles vary for each skill category. For some skills, the relevant

principles include guidelines concerned with some aspect of equipment similarity. For other kinds of skills, fidelity is de-emphasized. Realistic and

continuous feedback is recommended for tracking tasks, for example. Whereas

"equipment realism can be at a minimum" for procedural tasks.

Model Output

The values determined for the preparatory analyses are combined in a

specific computational formula for each model (in Table 3). Each formula is

used to generate an overall index of training effectiveness which ranges

between 0 and 1; the higher the index, the more effective a training device.

All of the equations used by the models have been designed to predict training

effectiveness, with overlap in the variables considered. The only mathematical property common to all of the formulae is the use of linear combinations. That is, the variables are combined in a simple multiplicative

fashion.

TV-A is the only model whose formula was based on the Gagne, et al.

S(1948) savings measure of transfer of training. The index of effectiveness

for a device is determined by the equipment similarity and training techniques

scores, weighted by the learning deficit score. The weighting strategy employed was the "weighted mean". The general form taken by a weighted mean

is: If each value x is associated with a weighting factor wi, where wi 0,

then wi is the total weight, and:

i=l n

Z wixi

~~i= 1wi . i=l

Note that the weights can not have negative values.

The equation used to generate the index in TV-B is not clearly related to

any particular transfer of training measure. The TV-B index is determined by

equipment similarity scores weighted by the required skills and knowledges and

task training difficulty scores. Together, these two variables cover information similar to that in the learning deficit score of TV-A. The manner in

which the weighting is accomplished in TV-B can only be considered a weighted

mean when there are no "unique tasks" trained by a device.

0

The index of TV-C was developed to reflect the percentage of maximum

transfer possible. The equation used to compute the index, therefore, is a

-p.

30

%•
Model -,(unttions~ for the Calculation of Overall Effectiveness Indices

N

Z, z C XS x Di xT i

TV-A Index - N

XD

N -number of subtasks

N(TC_+ PS + FNJK T

i-i 3 RT

TV-B Index- N Si+1r\X RT +UT

£ (_

N number of tasks

RT -number of required tasks

UT -number of "unique" tasks

C(CR xC xCi xD x (PC +-PC))

i-I

TV-C Index -N

£(CR xC x 4 x 4 (PCm t FCmax))i

* N n tumber of'skills

ifPC + FC

AF (LCx P x D) Fimax.

TV-D Index'- N

£(P XD)

% N - number of skills

N.B. See Table 2 for variable names.

31

.. . . . .
ratio of the variable values estimated for a particular training device,

divided by the maximum values which could be assigned to those variables (with

the exception of the coverage variables).

The only part of the TV-D formula which retains the above percentage is

the ratio of physical and functional characteristics scores, to their maximum

values. The rest of the equation has been revised, primarily for reasons

related to penalization of a device for non-coverage of particular skills.

In computing the overall figure of merit for a training device, a coverage penalty has been included in various ways in the four models. TV-A penalizes a device for not covering subtasks which require training. The penaliza- tion strategy used in TV-B lowers the index both for: 1) not covering tasks

requiring training, and 2) covering tasks which do not require training (i.e.,

unique tasks). The implementation of this penalty in TV-B is present in

almost all of this model's preparatory analyses, as well as being part of the

final equation. The reason given for the penalization of unique tasks was

that it would allow the TV-B index to reflect an unnecessary increase in cost,

while lowering training effectiveness. The major problem with this rationale

is the underlying assumption that all "extra" training features cost the same

amount and generally lower effectiveness; i.e., the penalty is equal for all

unique tasks. TV-B is the only model to use thi enalization strategy.

In TV-C, there is no penalization for non-coverage. If a skill is not

covered by a training device, zeroes are entered into the summations in both

the numerator and denominator of the final equation. That is, nothing is

contributed or taken away for skills not covered. The TV-D formula reintroduced the penalty for non-coverage. Moreover, the penalty for not covering a

particular skill is proportional to the "importance" of that skill (i.e.,

adjusted by the proficiency and difficulty variables). In other words, the

credit for coverage and the penalty for non-coverge are both weighted by the

same variables.

* Prescriptive Mode

In addition to its use in evaluating alternative training devices, an

analytic model (such as TRAINVICE) is also needed to provide guidance in the

specification of training device characteristics. That is, what is required

*is a prescriptive model as well as a predictive one. Whether or not both of

these functions can be performed by one of the TRAINVICE models (or any other

single model) remains to be seen. In all of the TRAINVICE publications, there

is only one strategy recommended for the use of a predictive model in the

prescriptive mode. This strategy is simply to perform the predictive procedures (ratings, etc.) with a device's design specifications as input. An

index of the device's potential training effectiveness (if built) is then

generated. If a prediction of poor transfer of training results, the device's

design can then be changed in an attempt to improve its effectiveness. The

new design can then be evaluated by generating a new prediction; and so on.

In other words, the model does not directly specify the most desirable training device characteristics. Rather, the model is used to give feedback on the

effectiveness of a proposed device; thus, providing indirect guidance in the

design process.

32
Wheaton, Fingerman, Rose, and Leonard (1976) caution us about such an

"" early (in the life cycle of a device) application of a predictive model.

They state that an early application would rely almost exclusively on the

Training Device Requirements document (TDR), and that the information in the

TDR would be of insufficient breadth and quality to allow performance of the

model's preparatory analyses. The only solution offered by Wheaton, Fingerman, Rose, and Leonard (1976) is the reformatting of the TDR.

In the absence of a major change in the TDR's scope and level of detail,

the question will remain: Can an analytic model demonstrate an acceptable

amount of predictive power when relying on rather unspecific task and equipment information? That is, can a predictive model work with low resolution

imput? If the answer to this question is negative, then the other question

which remains is: Can a truly (i.e., directly) prescriptive model be

developed?

Separate Indices

The overall index of effectiveness, generated by each of the models,

would clearly be of use when a choice must be made between two competing

training devices. The single figure of merit for each device provides the

decision maker with rather straightforward guidance; i.e., a "bottom line".

The utility of an overall index would, however, be minimal when decisions

must be made concerning: 1) training device design specifications and modifications (prescription); and 2) development of a program of instruction

which will complement the strengths and compensate for the weaknesses of a

training device (implementation). Either situation demands guidance which

is task, or perhaps skill, specific. In other words, what is needed is a

separate index of training effectiveness for each task (or skill). Whatever

the form that a separate index eventually takes, its development will contribute not only to the task specific questions of design and implementation, but also to the construction of a valid overall index.

33
IV

Conclusions

This report reviewed the TRAINVICE models for predicting training device

effectiveness. The models were presented as they were reported in the original documentation. It is hoped that we have remained faithful to the original

authors' intents.

TRAINVICE appears to be a promising method for analytically assessing

training device effectiveness during various stages of development. But

progress in developing and refining the methodology has been slow. Army

decision-makers need and can use a TRAINVICE approach now. Unfortunately, the

research community is not ready to field this methodology.

To meet this demand, ARI is conducting programmatic research to validate

and refine TRAINVICE methodology. As part of this research, a priori investigations of the mathematical sensitivity and distributional properties of the

models are planned. The core of these sensitivity/distribution tests will be

computer programs based on each of the TRAINVICE equations. The general

procedure to be followed will be the generation of index values, given systematic variation of component variable values.

Validation efforts will consist of comparisons of model predictions and

empirically obtained transfer of training data. Efforts are being made to

identify a variety of training devices and simulators which have recently been

(or will soon be) empirically evaluated. For each device, judgmental data

will be collected on the variables considered by each of the analytic

models. In this manner, an index of effectiveness can be generated using each

model, and all indices can be compared to the same set of empirical data.

In addition to actual field studies, laboratory research will be conducted to test the predictive power of model variables more systematically. The

experimental manipulation of these variables will consist of locating or

constructing devices which will conform to extreme as well as moderate variable values. It is hoped that the examination of devices which are markedly

different from each other, will permit the emergence of reliable effects. A

major problem which plagued prior validation efforts was that the devices

being compared were not significantly different in the amount of transfer

predicted or obtained. The planned approach will help to avoid merely

confirming a prediction of the null hypothesis.

An initial milestone is to develop a useable, although t.-im, version

of a model that may be routinely applied to training devices as they progress

through various stages in the acquisition cycle. While not expected to be

perfect, an evaluation approach which systematically assesses a device,

backed-up by guidance on its interpretation seems a possible reality in the

foreseeable future.

Since the application of any of the models reviewed here is a fairly

burdensome process, an associated milestone will be an automated (i.e.,

34

4X
*~~~~~ . . . . . . . . -.... * . 77 -

computer-based) implementation. The form the implementation is expected to

take is an interactive program which will: I) lead the user through the

model procedures and guidance relevant to each judgment; 2) maintain records

of all tasks, equipment, and judgmental rating information; and 3) perform

all calculations and generate hard copy of predictive indices. This strat-

. egy should permit the user to focus almost all of his or her time and energy

on making the judgments, which promise to be challenging in any analytic

model.

The results of training device evaluations, both analytic and empirical,

will ultimately be incorporated into a computer-based management information

system. As the data base contained in such a system grows, it will permit

training developers and researchers to track the history of individual training devices throughout their life cycles, from initial design to field utilization. Longitudinal training device data will, eventually, support the

continuous validation and refinement of both predictive and prescriptive

methods.

An investigation of current Army procedures followed in the writing of a

Training Device Requirements document (TDR) will also be performed to support

the development of prescriptive methods. As mentioned earlier, Wheaton,

Fingerman, Rose, and Leonard (1976) identified the shortcomings of the TDR as

the major limitation on an early, prescriptive application of analytic evaluation methods. Since the TDR investigation will address the ways in which

information is generated and used during the acquisition of a training device, this effort is expected to enhance Army utilization of device evaluation data, and to improve the overall quality of these data.

In reviewing the TRAINVICE models, it became apparent that there is also

a need for a thoroughgoing re-examination of the models' underlying assumptions about which characteristics of a training device will foster effective

training. In particular, this investigation must concern the applicability

of the various sets of Learning Guidelines to specific questions of device

evaluation. The Learning Guidelines used in the TRAINVICE models were originally intended to aid in media selection decisions. It is still unknown,

however, whether or not the same guidelines are of sufficient detail, or

validity, to be of use in the evaluation of the transfer potential of a particular training device. A second problem which needs to be addressed is the

assumption that each of the guidelines will promote transfer of training. In

some cases guidance appears to be directed primarily toward enhancing the

rate at which learning takes place, and in others, toward increasing skill

retention. Although rate of learning, retention, and transfer are all considered measures of "good training," they are not always similarly affected

by the same variables. For example, a variable which increases rate of

learning may have no effect on retention (Underwood, 1964).

Adequate definitions of each of the Learning Guidelines are needed. Such

a definition would consist minimally of an identification of the manipulable

parameters (i.e., independent variables) implied by each guideline, and the

specific effects of those parameters on rate of learning, retention, and

transfer of training. It is certain that the prediction of device effectiveness and the prescription of effective devices, will be greatly buttressed by

the guidance which results from this effort. First, an extensive review of

the research literature, both basic and applied, will be required to find

'35
sources supporting each guideline and to identify areas in which new empirical

research is needed. Once data have been collected the task of generating new

guidelines, and of incorporating them into device evaluation procedures will

remain. Clearly, the refinement of the Learning Guidelines must be considered

a long-term goal.

- To recapitulate briefly, our review of the TRAINVICE models has led us to

the following general conclusions. Despite their various limitations,the

TRAINVICE models are ambitious and promising methods for the analytic evaluation of training device effectiveness. The evident merits of these models

warrant a programmatic series of validation and, eventually, implementation

efforts. Any significant improvement in predictive or prescriptive methods

will require a long-term re-examination of the principles underlying training

device effectiveness.

The scope and amount of work outlined above is, admittedly, great.

However, the potential utility of analytic evaluation methods and the persistent need for them are at least as great.

-.

0.

36

.O

* . * 5S.* , i**.. .~.
RIV

V

References

Boldovici, J.A., Boycan G.G., Fingerman, P.W., Wheaton, G.R. M60A1AOS Tank

Gunnery Data Handbook. Alexandria, VA: U.S. Army Research Institute for

the Behavioral and Social Sciences Technical Report TR-79-A7, March, 1979.

Braby, R., Henry, J.M. Parrish, W.F., Swope, W.M. A Technique for Choosing

Cost-Effective Instructional Delivery Systems. Orlando, FL: Naval

Training Equipment Center TAEG Report No. 16, April, 1975.

Caro, P.W. Equipment-Device Task Commonality Analysis and Transfer of

Training. Human Resources Research Organization Technical Report TR-70-7,

June, 1910.

Demaree, R.G. Development of Training Equipment Planning Information. USAF

ASD Technical Report 61-544, October, 1961.

Evans, R.A. and Swezey, R.W. Development of a User's Guidebook I.- TRAINVICE

II. Alexandria VA: US Army Research Institute for the Behavioral and

Social Sciences, Research Note RN 81-28, 1981.

Folley, J.D., Jr. Development of an Improved Method of Task Analysis and

Beginnings of a Theory of Training. Valencia, PA: Applied Sciences

Associates, Inc. 1964.

Gagne, R.M., Foster, H., Crowley, M.E. The Measurement of Transfer of

Training. Psychological Bulletin, 1948, 45, 97-130.

Jeantheau, G.G. Handbook for Training Systems Evaluation. Navy Training

Device Center, NAVTRADEVCEN 66-C-0113-2, 1 971.

-t •Micheli, G.S. Analysis of the Transfer of Training Substitution, and

Fidelity of Transfer Equipment. Orlando, FL: Naval Training Equipment

Center TAEG Final Report, 1972.

I Narva, M.A. Formative Utilization of a Model for the Prediction of the

Effectiveness of Training Devices. Alexandria, VA: U.S. Army Research

Institute for the Behavioral and Social Sciences, Research Memorandum 79-

6, May, 197 a.

Narva, M.A. Development of a Systematic Methodology for the Application of

Judgmental Data to the Assessment of Training Device Concepts.

Alexandria, VA: U.S. Army Research Institute for the Behavioral and

Social Sciences, Research Memorandum 79-7, May, 1979b.

Osgood, C.E. The Similarity Paradox in Human Learning: A Resolution.

Psychological Review, 1949, 56, 132-143.

PM-TRADE. Training Device Requirements Documents Guide. Orlando, FL: Naval

Training Equipment Center, 5 January, 1919.

.37
Swezey, R.W., Evans, R.A. Guidebook for User's of TRAINVICE II. Alexandria,

VA: U.S. Army Research Institute for the Behavioral and Social Sciences,

May, 1 q0.

TRADOC Pamphlet 350-30: Interservice Procedures for Instructional Systems

Development, 1 975.

Underwood, J.J. Degree of Learning and the Measurement of Forgetting. Journal of Verbal Learning and Verbal Behavior, 1964, 3, 112-129.

U.S. Navy Training Device Center. Staff Study on Cost and Training

Effectiveness of Proposed Training Systems. Orlando, FL: Naval Training

Center TAEG Report 1, 1 912.

Willis, M.P., Peterson, R.O. Deriving Training Device Implications from

Learning Theory Principles. Volumn I: Guidelines for Training Device

Design, Development and Use. Port Washington, NY: U.S. Naval Training

Device Center, 1961.

Wheaton, G.R., Rose, A.M., Fingerman, P.W., Korotkin, A.L., Holding, D.H.

Evaluation of the Effectiveness of Training Devices: II Evaluation Plan

for the Preliminary Model. Second Interim Report. U.S. Army Research

Institute. Washington, D.C.: American Institutes for Research, 1974.

Wheaton, G.R., Fingerman, P.W., Rose, A.M., Leonard, R.J., Jr. Evaluation of

the Effectiveness of Training Devices; Elaboration and Application of the

Predictive Model. Alexandria, VA: U.S. Army Research Institute for the

Behavioral and Social Sciences, RM 76-16, July, 1976.

Wheaton, G.R., Fingerman, P.W., and Boycan, G.G. Development of a Model

Tank Gunnery Test. Alexandria, VA: U.S. Army Research Institute for

the Behavioral and Social Sciences, TR 78-A24, August, 1978.

Wheaton, G.R., Rose, A.M., Fingerman, P.W., Korotkin, A.L., Holding, D.H.

Evaluation of the Effectiveness of Training Devices: Literature Review

and Preliminary Model. Alexandria, VA: U.S. Army Research Institute

for the Behavioral and Social Sciences, RM 76-6, April, 1976.

38
43 0 0 C

w3 TO C.0 a.- w ~u 0.4.

0)CU4r0C "U I4' 430>.

43 4 .0 0 OAJ -LiC 4

-H '4 W41 -CO IV , -4 w3 $4i-i4: > to c3 3 0ca .4 -i w cc 4 u 43 4 j4) 1 -4 m : 41 :5 4-e-4433 d) ca a3 W3w .00 0i V*.0 4-I o. 0

a)0 41 4 4 C:C "aU -U OV$

43 4. 43~i 0. 4C C UC ~- W3 

41 4 0) 4) 14-4 - 0 34 43 40

43 0 431J 0 4 U) .0 u -4 4-P 43 4. 4334w wH r.4a4 r, ca0 co

r 3 4C W (1)-A4- 0 4JL4 w- .H &4 w4

43Ca 41 L (A U)J Wd Co 0 " U),-H w 0) i 0 w- r3 *- >% r. 41 W43 44 4J~ 0 C:COt)44 C:

r4 Ai 0 1 oc & ) c 4-i. p- 0d 0 CO 4U1- 4-J 43 0 0- -H-4 w *4 =U4 4 0 IdAj43.j- C:

o 43 43 ~ J- 1 4~414 U CUCdi. u-u U) ~ ~ ~ U ~ 0= C L O . $- - to p- -

443 ~ 43 -,4 434-04 ca3$4 a*00U:0 1 I

w 0(1 4- 4 C UCCO O0 ow.40 C43U -w CC 04 4 W 4 4 t--- r. -0 W W0 44 'Ua- 0

43 4 41 0.-44co43 03 rC1- 0 > w i CU 4

4a034 004 HA 43n4 . 4~ 0 U-4C0

U)a ~ Z- 44 C 43 Q).- -UU * . Ua

CU U) 0 4lC 0 43w4 u r.. C -4 434344

43 334i Qi04 0 .0C 0) 0-4 0 04i4-4

0aE .14 430 ~4)C~4 "q m -A V 0A0 % > Q 4 3 0 J i.- 4 A4 0j i 4 10-4 4 " > 41C

CU H u C q 434 Uu 1A .0-44.i 0) 0 00 m c

H-4 41i t4 -Li 4 1-4.,.0 ~ 0 41 :3.P. 0 00 .0 ).A :3 0 0 g 43 ) 3a) U) o a -H -Li -4 Cd -4 4) W U) 2U) .0 w 04 r.r 33 ) " 41J U) > -4 04.3 i 4 34C"3c 4) Ai1 4 . -IU CA w mUw -4 A u0 (U 4 .$ .4V C -4

=1a)V U a. .0mU1 w $4 - u 4 E-4 " W Q0 -

43 0 co-V aC 40C w 1 43 w"w0 40 w -0 0

$.) U) cc 0J.-.4 V- 0 ~ 444 0

0) o4 0 u - $4.0 4 A-iU0 cc $ 1C'0 U 4 043

u -H -H .- 440 0 A -C c- V H j c a wc 1 . rA U)4.1 CU-04 "J0 N -4 4 Ai $4 4 C: -4 C 1 I

43 > ~ 3.. 0 U-4 0 0 w ~4).0 w .co V4 0Cu 0 w > PL 44) CC CU 4 -04 "a 430*t 0- c3 a.0s 043) coUo ~ I. A J41 -H -I m 44 -4 4 04 0) 0

0 r. &I0u M 0 -HaXN m4

430).0 4) 34Q340 a 4 O c U 0 C) r4o ".3.r4 4) 400 t

.C: 0C ) 0 C: w .9 4) ~ -I4 0 0-4. 4 3r

4-i

A-4 0-4 01 C)rI-40

C >

*.0 -4 43

Ud ) r.i

. -44H U)

U)4 w3 I CUI-4i.H

0 43 043 -r4 U) cc 4d r. 0 A W-g -

C 0O-4 r.~-4

0. r-4 00 ( t 0

.0 cc- 439 u r

U)WV 34-4 .H4J41

(d 0 4 0-4 co3 -

39
.1 C1 C O 0-

CC *-4O.IC 1- 004 'L.,JC 0C m 4- co m a

$C U, 0 0 0. :3 - m4 tv C)1, V

0Od- 0u U 0 0uu)) -4>,10:3a wC:- 44C AU r= %6. U LW :30 o .CSU ) o r 00 cW 0 4 m ~ -CC- 0O~ 4-dU 4df-4 C, *0 OUCU >CW0CCc

faJ C0 al C 0 4 .- ,4 ~ C = U) U)G 00

.v 44 4-4 Cu 0C)~ ' 0 bd > 0 0d- 4 C 0 00 " 0 0 C - )- (4 )-4

0 V -4U I C) 4) U . UJ -04 U 4 " W-,4- 4j $4O cC E C9 U4C)C -,4 r.~ r_ U) C3 0' P~ 40 4.0 W Aj $4ca :3 4 0 :3 w 00 to 0 (f) 0 UU-4FA 1W

0 to-4 0 w CCw 0 AU 4.&-4- *U) .O U 24- r=- C. -A C r-4 C Li wC C~ Dt 'A C13 *C OU A- Jma Cu 3 c X > Ur) 0 4-

Cm U .6 ) C *-0 0. " 3OO$ . c0a

a )0 w~a w)i)) C r_ x. r4 0 -4 ) w) rzO .Cr~. = c r XU) - 4 0 Li C))CL. .~-d*

UUC)UUU0 0 C0 0 :3U~4 U>0 CC - -4 (n QUO

0. -AC dU) c w w0 0 0) 0 -4 .0 ( C , vN -44 0. u) -4 >U~ Aj C4 EC X -4 0 0. 1, 0 -4 r=-

-- H Co O 4-4 Li' 4-r-) M W C: $4

wE W) OO 41 CU a) Q) r-44 cf) 00 0U)0 Li 0 40) w 4-dU Li -4 C : C 4) "q

C) Cu4 0 ,-4 w) zC -. -4 -4

41r_0-4 03 00 4-i 0) U. a) :3

C) C0r - 4-i >, Ui) C) C)*-w c

> - 4 C SC C ) Uo U) =, 0~C "a~U

C C ) u 0 .1 C ) U ) U: C1 U ) P - 4 a t

Uo C 4) 0 "a (f) cd C) 0 co -40O CL)C)lCw0 4- 0 5 Ca r_ 54 i *a w-C cc C: -4 00 0 -H

0 $4-i C U) Cu ( C>4- w O !4 w 4 0V 4 ) 0 0 4 : 0 *4 0-lU 4-i r: e W"U

L)I ~ U 0i CC ) U)C) .JO0 -4 0 -4 to w C: U) A) aU -H 0 0~ U 2 4S-W

U) *-r4 C rU A C).- C4.1 ) 0- wC U1 0 0 .0 C) 0- VO)U toCC6) 44 -r

4 to t 4J.) -4 J U4 4 'r4Y"0 MOO0CI

00 4j &U 0 alC 0140 co D

C14 u * -4 0U to ) $4-d C 0) 0) ) Ai U Ca I-00 V Ai41co zca .C

SZ w- :1j 15 C: J$i4 u p- V,- A

Cu :3 U* '-4 O Me V0 co c Ca ) 0) 10 U 00 4- .d 00 Owa L UOd . a) *'0 W J4U (1) *,*4 S .0C Cu C C* > U) $4 4CC ) w)U0OC

toU Cu- V) -4 $4 *-4 41 V4I O "r4 p V '0 00 U P a CC: *.4 () ?A~ w ) C) 4) w (1) "A40 .V

GO C)00 0 r= U)C co (A 0 C) ' 0 H 0 w w 4 U)CC) w V -H 4 COO O "44 in-. *rI c.C = l . -4 0 v r4 I. WOUC") C)4E4U rA COO 0 0cc C i $4 M U)> 0U) O.

00

-r4

-r4 co 0 0 W

to 9: U.9a 4 4O

0-4 v0-

O) CO-Ha

to 4t w 

404
-r-4 0U Q) Q) Q) 0CC - .0 .0 U mI q

U) .,4 0. - w1 v.44. $4

0 w iU) ~ AJ U)C o O 0)4 .- W0-4 a -4 0 .OdO U(C0.u'Ca w V -0 CL 0 a)0 0 10 Q) 4 4j - x 0 U) v4 A-i,- 0 r.4C U -1,44i 14 t2 o 4C

u0 =CC 0.4.- 44 "C -144 0 a) * 4.' CLCCa 410 ) C ,0 V w &j In *,) > Ia~r 4 w0 ~4 Nd ()D 4 0CC r. 0C , 0 

.0 - -4 -44 01 Cf4 U) :) .~ " Q)~ -h -4a)co U0. :0-40 -4 ' - 0 >-~( =0 0 I $ .0 

)d.~ 0C rC- a~C . C -4 tCO - 0 0 JJ Ca -r4 0 . zC (C -,4 .0L0 V- 0C 4120 4.CC.-Cf 0 )-.:$0o CDC- - 41 oH -'-4 w-0 41 -- ) U) 0.0 ) n 0o4 *~4 0) 4.i 0.. a) C w'4)C $H.*4- CCC 0 ca 0 C 43) co C wCC -4 W0 -A4 aC 4 t 0- 0CC1 4-1 10 .0 =C 4C "a. 440: 0. r.-H >12. wC. 0CC ca 0 w 0C 0 r= L

-r - - H - 4 14a CL ~ w0 C00 oC 4-4 **-4 0 w -C ca w d a 0C MC 0a .0 1 4-4C r, CI0j0Cw Q -11 rz0 cl w C %&..4 co w a) r 0 ) $ H 00 -H co C 4 -H co CC-A a C 4 "aCCC-A 4 OJ.-4 *Q) 0 .H: Co 41'~C 44 41 W O 1 l .j 4k U )0 )000 C V CC0 O to ~Ca s w 44 14CCX0Ct r- -. 4 0 Q)rz4 = co )0 00 00 4J~ 0., mJa."-Li 41 -4 $4 4 0 a) 44 ' C41 .4 ~ -40 CL a caw C 3: CCo joA 0. 4C 00 4C-1 co 00a) 40w 0 44I_4 4.J 0 CC .caC a 0 1 0CC wC- 4) u .C a)-4C cCCCQ > L)-~ 0044 -4 a - (t -I -W 4CL .00 ld H~ CC -4 CCS-'4 0.0 0 "0 t.0~J Q 41 t 0 c -H 0 0; 3"a -H 4 41 coCr. r. CV4~ 00 4J Voo 0Q r 41 00 J r.CI. 0) a M C'oH CC4 'U0 0 C m 0 cC -H rCC U)E -H .- 4 U)lCL 41 (L)cCoC 0 . 0-4 .3c aN 414C to :I0 co 4C-1 a.1-4 4 C "A p d'CC'CCC > 4-1 CCc 0 140 r= " '0 2-4-4 0)vi-H ~ 0 0 $ 10= t '-N '0 4 4J~ 0 r.C)C 0. L a)OC 0 '4J44-C *Ho ) -4k w C tU - -H '0C " oCd z .I U 0 -H ~-4C 14 Ai 4'-4 (1 Cd v 0 CoC u 0 a) c0-4 44CC0 a) C) c Q) 0 Z w- 3-4 4 LO H Cd . d 4j s U5 u-C 0 0 a2- .00 a) C 9 C 0. -, 41J 0 0 C ~ C 0 .CC 4 Cc " co Or-4C ri ,- U 4 0- > -.H > .0 0Cto to -H V4 -4 'C r.4 .0HC -40 W0 )0 AC. Co 0 4 H 4 mCC m w CC CCco-H 1-ioC to C> ra 0-2C .C -4 >% 0 W4) a 0C ' 410 (D 10 aH 0. -H 0 $ Q. JC w CCO 4J 0.00 c 100 a)-"o -4 (A C> c 0U) " 0) ()cc 0C ZCC u x -4 - 4 'A1 r-ICco 14UQ q >4 r-4 ca. .24XC 

mCO. 0C2i 0e 0 00 0 0~0 .~c 00 0 W P. 04-~.. 0C '0Cw 0 412w c0 

-4 CCC 0CC t .0. -H c 0Ia. .000 45)-W 2da -. -14 W cn 44 ca CC: C/&j -H co cn C: 4.4 0o co

Q

03

-4

e CC 4-0CO

040

0 1-4 V

CO d 0A

1-4U

C41
V0 C4

T3 ' .- 4 0 -1 0 H

u) a)- V. > 1 0

.- 4 ccI 0,4 -4 z 0J-d)

0 0 0 - > 0)0- 4) "

D: Cn U4 4) V . -A cc0

4. 00 > '-4 4 Z .- o

oOU) -4 -0 -4 )4.

U) ") br 0.4f 0". wu

4. U r U)C- 0 A > at

t CL C Vq CO .. - 0

U-0. 44r -O .0~ *H14 31u

Z4-4 -A ) C: m U- ) 0) w

m-44-4 caW caw0)z od -

0 COA-4O - to0 41 ~Z d) 440

En CnC~- 1.4 091

.0 - 0 V4 zi00-4 0r-0 -

0 w 0 r0 ca 00 S 0

-4 1. 4,-Ir4 > 1 0o

j Q) 0 U4- $4 1 (nw(1 )> 00 W

(a. 4-. -4 4) 0 D 4-4 C: U4J C -

A- 4) "0. 4 C 0 to0 0$ -H4 H 

4 T

9:c d)4 0)10) Cfl.4 14 4 a~) 4 > E

u l Q L p) 0 4J 4.) Q S0Z0C

CoO. U) u) co 04 00 ca 0

co a. "o0 U) ) 40) 00 .- 0 w U)0 a. 0 :

,a C4o () 4-44 0 .-a) 4 4-4O vi f= W C:-H -4 )I

U)0) ca u . -4 d ) 0 U 4J - E.~ $-)

a)o co a) :j H - U)93 p 4

z - >- (aU Po)C- 14 > 01.'0aHQ)0 -4 

1- 0- -0 ) )0) (n P4 $4 r 0 14 Q U0 4" 4C

> o> -~ *-H a1. .4 0 "a ' r_0 ~-H

0 -4 CO D a40)E- p 4 .,4 0 H 41-4 *,4 )P. b$400-4

0)00) V. C. ~ 0 41 4Jco 0. 0 r 400WQ)-AdO

A .0 -H 00 -r4-i -4 Co 0 '-d 0 0 0-H4$ r

C: U 4) 0) 41 0 'VHJU1i04wU 4-1r4 00 4) u .000-H

0 4 Ai U > ) r. H 0 Z co ) C A) 0 wC w0) 4U

00 000d )0ca4 00:34-4d -H ) Co0A)0 0d-4 -4- 0) LdU' (V

'- -L X. 4 Q W44-4Z 0 C %V-1 41 w S ~ N ,j-1 0 - X

UU E-4 44 C0) U 4) 04) 0'Va) H410 4 4 )) -4 0

>V 10 -4 0) u 4) -4 0 > -H > .40 t~ 0.-0 u U

E- Q)0014p 0 CI W W U0 W 410,-H0 A0 0 N H 14:3 W rjd -A

0 0 00 P1ca00 *d- 010 p" )d-Hm GU S ~0-H U 4 > > -

00.,-H44 0 >0 :n 44 0. -H lo U -H a)0 t 4w.-40 d C0)a)-

*r- -- r= * 4 o V4 0 U M 41 0 0 W M.0 'TU

. 0 0 ) > Z Ob 0^b 000-I H-0 - 0) ) H> >.0 w 4-4 %D O0)

(n En -4 0a -d4-1 a) C C o o pV 0 V 9-4 140) o0 9: be00 en r

co 0Co.0 a .0C 4 UH *d-.0 .00. :$o 0 00) > 0 -H-0 C 0C

0) c 00 rn0) U 0 .-4 -4 $~ 4 w t C:.-H r4 0)> 4-Hd U- H

44 No -Iq0 .U44 C .0 o>>9 H4 0 4 U a)4-40A)0 (1) r4 CoU 000 -A 4 C

ca-H> - 0) a.0 W00 OM 40 0)4C w0$dO40 .- H -40 *4d4CZ$

Cot 0.-4 E4- .0 0 )04 E- 94 z 00 00u)>00)14 "t 4U 4 E-4

V4 -40 0- r- 0~- q M LnD r W % 0 4 c e

00 w0)400

r-4 0Co P

00 .0l Q) 0

ca -4 4J ca

r-4 ca C

0 0 .4 w0 d.0 -"40 O

140 00)H

U >- E-4 4

v-442

%0 N % %% %
'0 U) 0 '-.', ,4 0

0 0 ,.C:

C.)4- C4.J .) 0 a4 0

> co .1>0 '4 44-4

0 4J a) -4 4J 0

Q) .- 4 -40 0

.- 0 4 0 -0 c a

u 0 ) 2 '4 a

H a '0 .*-0 C) a)

r-o (D r- D w 

Cr0 .04- 0 -

u o .u .u 0 .-4 U a,

c. (A . H

QH.*H En a) 0.- "-0

41 C4 C 4) a)

. ,-q .4. ,- '-I ca 0 ) ,-

.' 0 a) , *H -H. - C -4 . )-E I .-0 , 0 .4. - 0 0 i *v4

-Z - 0 .- ,4 .H 0

>) 0 . 4 Q) . c-.4o $

E-4 "q r. a)4o 0 C-I v-4C

• -4 $4- JQi caC H .- 4 0. c

. i -4i CH 4. ca. 0 0

a) U (L ) . W ) mHa -H a)

ca )4 Ia) .-4 '0.4 CL I- -H 0 U) aQ) 0H 40J ow ) I 4a. '.4 -H . 0

C-4 -4-0 C .C 0

aH 0 0 a V -H - 44) A'0.4*H r -H ) -H -4 O - 00wr=

c. J-H 0CHa) C 4-i 4.)~ 0 A.4

0) 0r- 0- 4 0- - - J C n u U

-H4j- 4I 0C OC 0 -4.

H- 0.)H- C-H Cn .- 4)4- 00-4-

4.0 51 0o 41 . (nr H

.0 4.4 -- ) .- 0 O -

U) 4 0 0E-40 -

01

0*

-4 C

'H .4) >' E-4 4 0 CJ C

-43
01) 0 Q)

a Q) m w w 0 0 0( P.di u - O V -4 (L a a) (1 r-4 4 44u o ) w

.44 .0 J0 4-A.- 0 m 00O0 CL0 A- _>- 0) 

0 .,.4 -4 0.4-" 0) M.0 0) *-4 0)0Q) 0. .4d -4 M

e 0)j0 m r-4 0 .V v -*~O4- U 0

.14 Aj rz 0:-4.4.40 44 -4 C Q 400' . ( L Li 4 44 a) 00--4 U4 p ~ $4 4 *d U0 U a)c 0 C V).Q00 d00)0 -A -4 W C %.40) Li o) . -A -,4 ,4 414 4 A4 V.0C ~ 0 0c $ 0

0 -0 C4O J-0.4 -r w I x E 3- -4 -W 0 J 0) C ' a 'a 1 w) 10)f w r )OWU 4 .0 Li w 0 2 ) w.

0)- -4 0) U)J .40 AJ 0 W - 0a.c 0 0)C

to 0)1 Oejwu .00L.0:)0 z u o Q 4 0 CL.x 0 A-A ca rC 0)00.0 aw 0 u 0C0S 0 04-J -,4 J

0 0 u " w ". 0)0) - Ai 4)0a-) U 0 IA-4 a) : -4 >t 0 a 0 >1 m) 0 0 -> w) 0 .40 U)- w w m X0 *-4 -,4 C0 C0 Q0) CO 4w 0 41 Q) 4 0 -4 0 0 0 &J> -4w 000 "-4 C

0m 0) w- 0 0. w~-4 *d4 0 w 0..0 ~ 0) r0 wn). m

.4u 0cfl (0 U4 " 3- 0 -0 (2 U . b-141 O-.-4 -r 0 GJ 

W 0) .-- ,4 4j .- 4 0o00 c *- 44j ~ W W " Li v4 0 >

C) 41 0 wo n wu E -4 0010 C0 0. atoC 0 -4 0 0 Is c~ (V ))*H-w -4 0 to0 X - 1.0uA "

0o Li MJf $- > a 0 0O -A W-H -A C 0 -

0 '0 m a 0 0 4 44 S 0 0 C 09. 4 4 $ 4

4.0-1 ~-4 E20 = _ U .4 00) 0 $4 40) (*2

0) 0 a 0 cow0 p r 00) 0) 000 *-a -4f0.,4 003 W ) C'04 4 41 0 10- 0.0 0 -H 0 -4 Li AA 0) 0 W 14W () > w

> 4 A wc wC .0 4)0 -. u -4 4 0) .0) 0

0)0) 0 Q 44lOU1 ~U & 0 0 00 * '-40 > V) :3 .-dq 4- 3t =r co co r0 4 0 0..H

0) 0)0 0o0a) 4) psr-4 :1c 4 d0 2 4 o 0 4>

0 1-40 u w Q .00 0 0 )0c0a0c 0) $4..'a -H4 Ai

.0 >. cr :r 0 C )U4-4 U-4U) -4 0

.00) 44 0) 0) 0. 00 0)4 .0 p..~ p oua0

Q C) W 0 W 0 0 ).'0)W )>" w-Jif0 -'-$. 0

-A to 00) E-4 c C4I40 IU0.0 124 . :341 0.H.

0) 15 1-AUCO 0 0 ) . VO0 0j p E40) r)0)0 41 $4-4P..4I- .1-1 1 coi- a) .$. c *0-4U 0)02. 0 cc m 0 $4 1 0 

COO -4 0d-H 0 0)0 ".,4 0 .00) '-40. 000) *L4J

Q ) C: 'L -3 U C-4-H 4 0 $40)'l4 1 2d -H ,4 -4 .. D0 -4--A C 0u0) - 4$Q 0" s CO$4.X0$. - 0 23 0 0 C00) 0d r.0) ..- 40 41 $.40 -0c - -HU0)) 4) w d *H.04

S" -. 4 "A- 0 . 4 U 3 0) -H4 C w.). CU) 441 -4-4 0 w2.

c/ 0 $4 > w 0 " uC0 2) 00) C 0m24a 0 0 0 u 1-4 to0) a. r0 *44.0 0> -H 00a0 0"H

bo [-4~~0 V. 1-41 UM0 U En 4U0U 20 =1crU a D 4

0 0)

-. 0 c

-H-~ 0- 440cJ -

$40t 00 1x "V-x

cc4. 0040r

0 H4 4 P oV

0 r4 bo0

02 4.'

* -44
U) '

0 () ca ((3 Ji *l .H ccU 044 uHC 0 00 (A ((o 1-4 0 r_ r_ a 0 CI. co3) (3) 

0 H - Ja(0 -44 -4 0( uU -a0 U

44-J " Om C: .0 a) 1-4 4J Q 0

0 0 1 0H4 r 4-Uo 00 C -r-) a 4 41 m 4-O 4 iH -4 *0 .v-) U) >14 4-4rI 41 (n co4- 1-4H U)>. 

41C CU cc 0) rHH CU4 0 I.o0 r. WH a) ~ 41 44> Z~U 0 > -HA

00 0 ) Ia 0 En$4 041 a) aI

r_. C60 HH Q)a Uwo0 41i rb U4- 4-4 to0 &J n Ma-H U~U0 V:U m 0H 01 o~ S o (n1 r_)C 1 HS-i w HU. 4 a)- U) 0 Q 5-H >1 0r to a) r.C = Li44- > b0000 -4 co v44J 4-i(3 04- $-0$- p0 41Ju 4-LO CU S-a O 3 a) -JO ) U41-HO 0 u

4) Q) 4Ji-H 41i 0 0U) En j-L)-H 0 -r-,- -0 4CU.C (ni 0 ~ 41 (3) U) .uU)UC .0.r.

-LJ (3) 1.Hi a)4~ cc)U CU (n- >-US--

CU 41J- .4U U 0 41 U) -H aH to CU H 4-i -4-J 0 044 0)CO U)U)cn s-ICUi WflU) 41 r4 - *H.-4 -H W J -H 0 Ui t (AU -,4 U) 0 >N Q -4 Q) u 00 0 4J i 4

w) 41(3 '-14- 4.44- ca0 CU - 0 m -4 -4 0 = H - -O V ca4- cc 4- 0- CUO CL*-4 0CU r- . J 44 C:C

4-54- 4-i tol$W U) b003-HQ U) H40a) 00 0 0 0 H

(3) U)U 0 4)5- *H , Cu H - -H -Li " 4-4 .0 -HO u $.U)0-4 S-L Q j'co ~ 44 '4 to. m ~ 0 idcd b 0 0 >1 .0 -4 co bo UU4

04 (U 00 0a)4 S-I Sa-.-4 0U 41 Lo$40) SC 0aa CO ,- 4-JO) CU w a) H 0$:t-4s-.) 0f x U c boI- 4j 00 r. * -4 UU H5 0 00.0 4 U)CU*H p 0 .0 00-H P 0 *cr -4$-

0j 4-O U0 VU r-i A 00Cw r= 0-H .

rzUC 4-4 CU-Hi CU w)4U.A~ 0 4) r 014 )0 0 > ~ :4-i 0$ p0 CU 0 'o 0c t 0) 0U xC O W a)- 0"a(k)$ "0 44 4)4-J X .00E-4 44 ~ - C . "~. -Li co0. 4 E4E-44- 044 "q 4Ai. 4-i.aUw f-IC -H-~

H ) E-4 r.IC ,-i 0 c ) U)0 0 -4 000 -H ) *co a - -H- u-0: 11" 4 3co > r. " > ,- ca r.C CU CZ C4- 05-p -H a to-H CUW) *4i -4( *o *-- 0 -li V.U (3a ) p0 >> u0r $ )0 bo -4.00 Cd oQ ~-44 "(2 P LJ10 000 4)u -4 -4 (3) C m m CU C 2.- cU CUS-p C> U) )r.0) -0 0 ) 0.0 ) r4$. U) 41 U) '-4 4-H H *-4 -H4-j C 4J-r4 p C cuO 54 0$- OcUo0 -A H-44." U)Ur)& 4 U)t 0) 04J-r ) 0 0 4-4

Ct)UU 0-4 ,-U) w "-S- (a4-.4 a) w )(1c ( m4 V 0 4 4-) ) ) U) a )CJ 0) C *-I- in.OU C AJZ a C u 4-

-o4)

*-4 ( '4- Ca W-E-

-H n

cO~ 4

v Ai a -I-. Hca b

V- w0 $ 0

02H -H cc.4Q

W.4C 1H43 w5-) . -H r. .% w 5 % % V ~s-% ~ 5... . * % V
-4 -C c ~ U)

ca d) 0 CUw U)- 0 1 c a-4 c 4-1 n 'r4 4-J WI

-4 C 0)0 $ t)--4- 0) 0 -- 4-4 U00' $) "as- (1) 1-4 0 02.4 Li 4 U)*,-4~- 0 -,4 -4UZ L~lC WC

w.4j*- -4-.,4 *: . .r4 w 0. -r4 v J=0 

J- 0) m oJ -4±J- C .fO (A>10 4 41 e w *.4 M~* 0 U ) .- 4

41 . 0I 4-1 0 .4 -4 C *-,4 4CU

-4 00 a) -400 4LIW 

$4 ( U)-I 41 0 -4 CL. -4 2 CU **o- H. .00- CW.OC O-H 4-i d) 4-I 00 -Li 41 C -0 0 ) .U) 0 0 1-4 4.J a, to r 3 0000 w ) w -4 :3 C 0) ) 4-40 .00 U- C4 c 4. c 4 0

0 44 0.j (A-YX a 0. *44~ 0OOC L .01.4 .J LI0 tv 4-1 4 0 4-JO Ia4L d

r4 -H d W0 C= w.J wC C C c r2 0 o j C C). $4-) 020 w )0 4 C4 m 41 (L) U04 C w

44 4JL 00U 44IC)0 COl 0-4 1-4 C U) CUq ca4C 0 U4 04 4d 4 0 ..t

01) 41 a 0 0 C.-- 41IL~ I i V d 00 U) : 1) 0 0 co Wca "a

244 i-I * OOLI oC -4C.O -H020Do)= -op COO 4) C41 $4O Ai4 L 'V~I 04L &J - .0 4-444 0ll 414C m -'0. 2 CU 4OO 0 .0 44C 4J L 10 ca. -4 OCUw4OJ -44414 -LI 0Jco O a) C C *CUC o r.0.-C ~ :3 > 0 -40 ca CU04 p 4 L00 0O,-42

0 LIUoLI W 0 -JC CU 0 oC.- 0 00

.L 0IU Q) 0) $-4-H 0 0 CU 0) a)) caC

LI " CO -CU Ia. 41 -i.- a 0 &a -H ) :1 .0 V A.

0 caCC 0)HC 0 C4wV 0 0 > LI - 4) ca CU H u p 4Cd "0 r-Cw 41i C-r -HO 04 w -0 CU

4- )0 C.0 CCUC co C 4 44 -1 CU *.-C w C, **- tv coCUz U 0 -Hq mW 1 -d CC1 C) LI CW CU-4 -H a;Cda) CUrUC t wU4 $4 I0 U -H -4 .5d %4-4 uW -4C 0 a)0a0) r r >1 0) 0 r. 0 C4) 4to 4 CU0 .CC. Q) Cd 41 co a) r.4 ca 00

a) 'a >0 >.4 > . D o ca > U)H-4-44 > -4 41 10 r

10--4 4 c CU.) 14C CUUC.-4 " w Cw 44 caC Cor:-H $44-4 00 .0W >1 .C W-,440) w : C 1f>C . 0 004)9 CU 0 r-4 :1 0r4C '0 ".C cI -H 0)0. 0W4 C>1C '0)--4 w >C W 0 a)Uxco 0 41 CUCOm X) XC - -4 r-I " -0 co u4 1-1 *>CU 4 Cd =04w

00 00 1 0 W " 0)0 0 10 E 0 41$

z 0 0004En 4 U c O4 l.c Is o n.6 - 44 d 0 OL .

U)04 0H . W C O ) C UC C 0 0 L4

-4 L) E4 - 0 H 'D 0

w C %I 4co U

w0)

C) 0) Dt: 1 d)C - : 1 d -H '0 c 00IH a .

44 d)4

0. r--4.- M)

4.46

0.-!6
-'-*

:3 " 0

r00 415

.- 40 0 0

0 a

9. 0 -4 r0 4

0 41 cc

0 : 0 0)

v4.

w CIO - V)cc

02 00

co 1-4 0 0 W r4

$4l '-4 *.-4 0

00 -r 0

:1 > 4-1 -H $4 0 0 -ii) r

0 0- ca 0) 0 0 S 0'- 41- 4- 0i 1 

4 0- r.00 ,14 u i ca 0 3 0a m C

>~ ~~~ :3 r 1 10U 6 %- E-42 S o U 0 r. .w4 0

U) 4.j 4-i P-0 0

01c )r ) c 0 z0

*,4-4 a) P 4) 0 Q)- 4D0 4

0) $40 W 4 -44 4 ci 1

*~1 -d :)4 d 40--I 00

0) ca0 0 0

4-002 450 *-

1. co-4~

r4- A0 *0- -0

(d.r.- 0 004

0 -A 0

ca ~ S-o ~ 40 , 

04 02

0 $-s0 r. 544- v-U

fA *,-$4 U r *4 .- 4 - Cd co. 00 co

02IIAX. 000 0 0 0r
"A. -4 .-4 0

H~ 02i-

*~~~~1 -H 1'022

4.j U) 4j 0 to0 

C) W * .4 W). 

U ,4 $-i W) p A-i 14 0

4.C -j CL $- 0 0 U -4 c.4

* C) J-1 j r.Ai 4 co~

'A(1 ) 0 !d L

to ( U () lAi .4 -- .0 W4 &-j -

(n- .1 to4 01 U0 -4 -H W

r4 42 *4 CO 5a 7 c wA to 0 C 1

(2 -7i (14 (4 0 (12 C 74 q

A-iu *7 -40 U- 

-r4 U) .,4 U 4 'CO 0 (42 01 00 w1 4-4 L"i

to4 A-J -4 . V- 0-i 4 .0 -. 4

0. -. 4A-i r. U) '440

W- -H c) C 0 4j 72 4 4- '044

- 0I 0 0 w 0 4) V -0

H) .74 r4 $44 (D(

A. 0- 5-a 60 Ia -42 4) t0) 

C) .4 $- c i-i C: 3t44- 12

> .7$ . 4 - 4 0 U

z- C) j.i V) 0 00 ) u.4 

.n 0 0. 0. -1 0-AVI

> C 0 0 C0 (424

(12 A-i 4.) 0

:0 .7r7 ,4 *44 p- -74 4 . CL.

%44 0 012 r. V u 0

-% 00 ca4 to wC 0H0a 0 Q) -H IWC0

N120 a1 01-)4 124 " 0 Q .7 4 "4) 0

C)C4- a) ) 4-1 ) A-i r-I 0

-(1 -H7 1-4 -40 4) -4 w W 0 ) ) 010 o .-4

a) 3t 3: En DtU > > > I* w

0 0 0 0 co toC co 4o 70) -H 0

g1 02~~1 wo C:C to0 .0 4J 4-4 A-i

0)-1 . ~ 0C . 0C 0 '-o *-. -- -H *-H7 "0CC $4 "0 C 5-C wC bcc

ca .74 - -40 . ,-4 -4I -4 0 -4 1-4 C: , , *i U -r -H-I-7444 -I -.7 :3-4 : 44 0 to (1H7 ( 0) 0)74

"7442 co 74 70 V74(4o 0.74 0 $4 0 0 -T4 1 .0 u F r

0bd wi. -1Z pi~. -.4ApJ.54 (1) C C# C1 X-a w ::3 0-f

4-j

.00

* ,-4

.0 -I

-IC 0- 00 00 U

(4 )- bC4 r o HC 34

g 4 o 0 )C U) 0V4 00 020 )- N,~~12 M -04U)%74 7 0 -r# 41- >% H o0 10 4- (-4 002 -4 V -7r4 U 0

C > : o to .41.4 cc ot 4c

C 0 00 0) 4 V0 w'- r

0 bo r- ) cc0 .I

4 t 05 0> cc> 4 .fIcc a

>27 2.O 44A- r4 o-4

* 48

.P % 

-J -0 .,. o 0 *.
-'cl 1 -4

(a0 4) . CO

00. 4 '-(4V 

~~~CA 0 A O- f *

:,1 J .- :t-4 cc

x~ w 00 m0 -4 .

i j : 4~ 00 m.0 '4 0

0 W 0n 0

1. o 9. u. ~0 tn~'* 001-4C

O M -6- 0 20 .. 1. 0n

0< U)4 1. W : 0 .0

Ai 41 00 co> >0 UA - 2 O

i-i V.a 0 3o.> ~ ra r_~c L) fCO C-4 1. 0 >.

W e ) 4 4) 0L uc 93

v CL 00 r_0 p- 0 -4 a- 01-w ' 0 

S V 0 C. V)-r 40 w0)~ (A0 V0.0)CL C 0 ca

(n < "- C Wi LOO-4 r, U u D 4) a

0) w 1A ar4- )C0 U) 0 0

'V -. - cc~40 O 0) . 1 41 - n -4 C U f 0) m.2

(= C14I to40=0 a) Ad .W

-'1-4 ' 0)~4 E- 01-4.0. 4) 

V> 1S 00)' bO Wt0 z~O L0 -f- ,- 4I 4

0) 4 41 -H d -M f) 4-4 0~ 0 z -A * W > 00c 00c

u oc U0 00 'O0 'v S 4 00.,2 4 4.11r. J4 ALiw

u1~ U0k O,0 tj r_0.0 w2Cm0w."40- w.J w cc( (L)

.-4 f - Q.z ~ 0 W 00 >0 ' 40~ 0) -r r 3

> -40 L) 14 0>1 .,0 H 0 u 000 U P -4 U' 44

z 0. 001a0 0- *d0 00> 41- A0 =. '-4V04

,4 Do00 .- .0 c:ba:3 cot CO E 4 ~ w

0-r4 -4 0'C: 0 o c r v-4 "en4en 

>V. H .14 04.4 op.. w *v.20 to a - 4 V0E 

0 0 0)00> ,. 4 .4 A A OO-4 $4 -A d 4. W ~ v ' 0. 'V m.

4) > 60OOZ to 4J U 4 0 W 4j 0 1-40 0

V4 0) 14 0 r- d 4 CO44 0 4 U 4,4V-4. X 0 4)

6114 :t <0 .- 0) Ai -4 wd) W 0 l J 0.I" 4 0

U .0 oJ 00 o 1. 'V 0 o010) AJO00 .0 c -7. t

* 0 E-4 bo ciE-4 c C rU- 4 cz o44U) A4> E4.0 C) -.

V.4

00

00 .00

w 0) cc 

01.1 ,

4 MA

0 v40 cc4~ 0

cc- 9: -r4 w 4 w$

0 u14 *d ., 1.4

01 r-446

"4 Xv -4 1.1 

cc k to 00

4.449 Li
00 0 a*

C: CL 1 0-DVU

4 0) 44 0 Jf-

>, raN 0) C 03$ 

E4000-4 04 030 ( C: 0 w

E-C 0.- ow a: *.-1

to W-4 -4 014 m44JJ V- ca

. 0 :) ~ *C 030 

Cu *C0.. 4 (24 ~ V.C 0 3

0. u-4 COO 00 -. -4 (n c

wO -4) F- Q.. 00 CO 41 )

a .4- co :3 0 U) >0r 03 00 r_ r

-A m Cu " C ) )L '02 -U, 0-4 CO COQ)

0 000t c:2 C1 0: w3 U)

wu OU tI0'r -4- ca - w3 V, ~4

$4 C0 4s C)~ au$ = 0) a) -A

w C: -H -u j Cz :3 -o

Li -ul & $4C 0 -4 r3 -A =.- 03 W- v

C3 C-4 032 C -4 'A W w 4 w - 3 - u

0 <>0 a) w m CO4 0 *-4 $' CO cu

u) ,4j. J~54- 0 t", w -

0$to0 *0303 00 03 w 4- C 0) 0 03 .

0)Cu CC 4.J0 0 244 4-4 2, U i

> *.-4 , bo .- 0>~ .01 to 413

C44 tz0) aC L44 -HC M -COU C3 0 0 *- Cu

*410. C: 4 .400- toJ- W o 4-4 -A. u

Ai8 CO 0 CO W 0 =. C 'L

C ~ ~~~ 0 4)3tS-SC~ 03 Cd 3 C

0a 410C~~ coco 0 2 2 0 U 2 -

U -- 4 02- $-4 4-4 * 00 i 03 -.4 C:. U ~-4 0 0 -H4 -4 ,-4 aU a) w PL

d)- 03A 14 M . L . to 2 0

to' 4u- -4 24CO) 032 rUr 21 03 4-4

0 w mO~ 0 > .04 r_ - CH .- 4 54

.'44 0 0 > 4-4( CO i 03 0 m3 0) 03

4) .4C .. COO 00 2z 2 U

Z,..--- CL. "3 =. C4 Cu C

00 "4 03 ;0 -H >%- a) xr-4

0S 4 > 4 0 -0- S-0 41 0-4 v 41 03

0Cv20)-J 0.- C 2d Uo 0 C

03 0 -ACO U 0.O CuW Cu -) CuA3

C .-- U CO 0 ~ $W 3 m a

US44 401 C 54 U1 0 0

C4 CO u4-4 D 14 cC 0 0 :1 0 'V 0 C

Cu

V4

Cu C>

-H A0 030

>00a -'50 $4Cu CO U0c (0

$4.- cCU Cu"- CLO ) I

03.CCTo $4340C

02 w ~ I. a4 w0

** 0 49 0 go -40 a)- 41 COV

V-S tU u CU u P%" C U 4 U WU - Uco 9: 4 $4.u-A,.-4 m p s-A CU $ r

> to 0 tv Ao 0

0to

44( 60 #-

0 -4 40

50
0 4-4-H ..p4 

-4U000

A* 0

CC 0 0. 

E-4-4 r-4

CCd

co 00 " 0

0 U U44)

-r4 CC *.-44

U) ) co 0) 0 bC a. . 00

p. 0)H 0 44 -H 4-

"A -4 "- 4-4 u u-4l - Ii0 41 I 0) 0

U--" cc cd 4-

a WH ) a) 4-

0 41 H 0 0 -

-H~- U 0 ii 0 *

4-i AH cis 0

6z -4 C

0 >4 r 1w -4 0 jj

0 "4 - 0r -

cc4 cc 0 0 

w 0 0i -H 

u) >. 4 4i

C) C 4-iCU - C51
10 CO0C 41 PU

z M m C-4 ( 4

0o w- 0 0 CU () -, UH

u 1.0 .14 44. 1 V0 ',If0 

U, - CO r4-. cCO Ai () 4-H0 v4 >. "0C 4 V4 0 0 > 4 14: - 0) C Q) "4 .0$4 4 w -4.041Il-M

CU 0 0 . 0 .$.40 > coQ

>. 41 (014(1 CO0Q -4 0

0 U1i P cn 4-40 W 5w -. Co0 *.- 0E 01- 02004j - u U 0 0.u 0 0 w S Co *-r4 "D44 4- w Ai "A4-4,V4

0 -'-4 0 4-J 0- 1.'4 a)-4 0 -'4 :5 c .14 4-4 .c01w r-40.(-4 - : Ai 17 44 0~ 0 34-4 d t JC -0. w .-H

w" -4 0 Q) t ~ >N -,4l- (42 (1241

1- -4 -V 0) be w w 4-0. CO V 0 0 Ai 0) U .14 >1 W 0 10 w 0 0 w- m C ,UWu

W4 _* (2 p0 0 0) 0-.-A 00. 00)d)C

* .14 '-' 3)-i w w e -4 " -4 Q).0 -H (L > Q0 0 00Wm V oAi4 uu 0C .0 CV cu U4 r. 0 (1) W w OC.0 tJv m VQ Z 0) M~ -4 0.0 w. 41 01 w cc

cc W -'4 0 0 . 4 0 w C: 00. > 3 4- . > .a z u r_ 04 r. 0 u0'.qu 0 4-1W 4.5> -0) a U w.~ u 4-4

*'4.-.0-4 W- "4- 000 -4 0 0004

r-4.0r- 0) 0) c0 15 0U P-4 0 4 W 0) 41 4.5 I2 U 40 Q ~ .Co.~ LCU0 

0L Co Co 0 9:CU a 0 bCU .4 V0 0 bo 00wc 040 -4 r--, '4.0.

-4. -0 0 0 "a0 J1.1- 0 -4b .A0 H0 ,-. .-4 41-40 CC 004 4o4 44 -- 44 W W

00 1-' 0 0 "4 W $4 * - O.

>M CU 0>UU >" .G WO Cu >r0. C

W1 .0 .* 0K .0 C- (-4 = .4-40Q Z~ C CU W 000

0) % U 0 0 0A -4 0) w -daUO2=

-4 '00 0O 00-1 410-4- ;; CU .- '~ 0 -4 .4 v 0 F-4CCUf cc0- cc w- 0 

U- := C-4 W4~4 0 -41d W2 WU ~ - 0- 02 OM ~ 1-4 -'- .4OcUId 0- 0 01i-a w 0 4- 0c u 0)

W00 0 bd I .01 "~ .0)0O .0 o0o0 (vX0 a0

Co 0'.- z En 02 c20 L- C14 1-'4J 4 (0wC30- f 02- 0

$4 0d

00>

-4

.0 -4

>1 rn o

w (%c 0 Cda ~ f W 

r-ICU $.4 V4 >% 0 P

cO 04.h. 0 -4 93 f-4

>~Cca 0 4) c aG0 00c 1-H

4-40 a)0I C 0 -4 ii0 M
.4 0 .) 0 ) t

01- 0 ia 00 

.- 44 M w M-4 a) 4 -4 = 0 0

"40 co- 0 ~ U 1W

000.> ) c: (j0 0 .Cc.1 i2

> -r- >J . -a 0 :.10 0 e 4 4 c

C> Ad .q ' 4 4 . .1a A0 . wJC . CL

.44JJ-A >C 0 E ) UJ - CL

cS w.40 2-t M0 U -4UV

10 -44U C-4.'- M- U) W0. a1-ww. ( L.- u 0 Uo 0 )-

COO~ *.4 >j 0 Cl0-

0- r-C 00 ) 0-)- ca1- O0 M - C)

4-A u 4-1v 0 z co >0C

0LXCO -4 10) M 4 -A .0 x 0 50

OU) 0 O C U 0) 00f1 00

U C m w00 e0 Uj 000> >

.CC cu Q)~ u wJ4 " 0 50

"0 r-'- c Cn0 aC j

ul ~ Q.0OU CZ~C 0W~ Q)04-' ) o

*14s.4 CL -q c CO- 0 P OCl m c C> (0-4

0 Lj-0 0 ) E E V 4-4 z0 -

w 0. .1 a0 a)U 0140 P.A C 0-.

U000 a Zw U>0 E0 (1 0j '0 U A

Od Ai 00 0u> P.4 to0 CU ow

.,.4 0 to 0a *.-U..- Q..> -4 [O-0 In.r4-*

0J cOc cc 0( >0- 0C 3t.0

> "4 J0) CU $ -Cr4 A 0.I *.4 1 0) 0000 0 u r

-. ~~~~- C4 0.4 E4,400 $i.OJC -0 0- 1

A* 4 4 J0 0 - c 0IfJ m-~ O 0 si

E-0 co 0 a 0H' 4 W- nU to U4 o0 C

~ Li .J 04 "00 0 1-riJ4 " ',4 0J- wOC 0

0 U1 00 U5 .4 Li- tv 01 -,4 4r

C-) *.A0 CC -4A-. 0d> co cd 0004 U)W 4

4U C *1Z4 H 0 0CU 0 Q~ 44C U i0.0 O (l2f1

4- 14 ow *.-4 .- I1 0 5 C W >O

m0 ) .1)4-A cU . 0) >,4 0 0-4 01 00 00U

> *-0 z r. 4- ,4 4 4 COC> CW 4-i.- = 0 100 0

E-4 U $4.1 v .,IU 0-- C.) C000 u A V . .a 44 W.- O. LO $4 -4 1r 0 t0 0 bl

.0 Ua -r.4 0) 40 bU 130 -f-4 0OOC. .0 0..rV14 

tw W-4C -40 -r4-1 0'' 0- 4.

-r4 ~ -1 .- Uj- w U >E- 94 04-44 *,4b 0 C 00-r

O 44 0 40) 0 r4. 0) *14 00%0 .- 41U. -r4 CO :

cd u,4- U1 4J-IU'- 0 h .C O 4 >-4A 4a 4 00 -4 000 r

0. " 0 r4 u w )j U 0 W > 0 LI4 w a-40p 41

to-co4 4 -

00 4.ot o w4

0. w Q n z4 m,.- 0~ -4 4 E400 U - t x9 'Nw

000

-4 C4- 4-i -4c) qcn : n' -c

O 4 04-

-.>0 4. 

0.-4 00)4

% -r4 .0C4U

00

41

.100 000(A

9: 0 ".4 to Ai -v4

- U4- 0 uq- .4 4 >0 Uc 0

*4 r-4 C -A4C

04 u "-4 cc

4)

44 6000

0 -4 0Vr. * r4 - -4-"4

oe 14 c 0 cc

r44 $

53
cc t -1 4 r) E- I 4. 0

1.4 O . 00 ~ 4 EnZ 0 0 0) 0

-04 0 C0 J 0 * 0 Q) w C: -4 V. u~ (n U -u- 0: A 0

cc C: U- &J pi Ii 0 ON 1I 0- -41- m U 0 4CUc ~ 0 C0-4 00 4* 4 co 0

0 10 U-4 44~ r.i*40 g .440 4J 444 

.. 4 X 0 r4 .0 0 4 4- -Li -due en 0 co a 0>1k -4 V4 0 0 u 0V0>. 4 >% 0 0 V (A 4 0 *ca 0 0

1V.0 :3 w a) 0 mn 0 r ) " 0! W 4 t.Q 0 Q) r >1 1 4 U) en 0-r4 4 .-

41JO .0 41 C6 (n CD 40 Q) -A'

(A ~ o~ e4 U) -4 Q) =n (j) r_ r0 r. a U) , 4 E0~ a) 0 -H -4 Q) -r *-4 1-4 4-J 44CJ v.04-4 0 0)0W 0 1- ) 00

- -~.CO CU en oo0 Un a) " 0-H t 04 *,- o p~r o 0 a ,-H V- en *A 60 0o o .o0 04 '. 4 Q O en > 0)--4 0 ) 4)> '4-4

0 1 tv CU -L a)J O Ln : bo a.-H Ai 0 4-4

-i 0 *a m to0 rm - toca w. r- (30 ., W0..C sf.0 U)- e 0 >1 -4 0'% V 'a-4.-. '4-4 0

W 1-l-4 00 M -4 w co(fl a 00O14 0 00 0Ow 4) w 0* 10 1-4 0 44 -H4CL 04-40 0 0)0 co 0 0 :3 -r-40Q 0

u 4-4 -H 0J."C $4 1) >, 0- r4 . r4 0 , 0 41 > *

CAPeanir C~ - 0) 0~e1i- 4(D w0 * -r4 -r 0d 1 4-i4) 0O-H c X 0> >,4 w wn k wU- 4co 00j X 0 4 C: U-4 -0444 % *- 4 .0 4-- V)w c 4-i -4 00 Q) 0r

.-4 U 0) 000 *OO 41" .I04 0 ) CO 0 Ho 2

to 10~ a) bI 0 0) ~) 00 0) 44 r. 04 H M- r. 4U 0 Y.1UC UH r. r. Q)-10 4) C-) a

(U .vo $04 r. co0 >.X >~ )P1 0 - rq 0 0.1

% 01 > -c-4H u 44U0 w-- '44,-C >% >~4 A 01-4 0): en r-4H 44 -H 0)0 COH 4 0 CO coU CU4 - w 0 0H c-o 1o- 0 Vn m 00 0 -r -A n 00e Pe p a ca.d" 0 r

0 U .* 0004. en0-H,- 0 gno co w-0 010 0-rf -i- 03 014- E OO "4 "OOeUC )-t-i- C U)( 0 . r_ 0 00 4

H - 4 4r4 >, 0 "J -IE-4 u $ -4444. 41 41 L44- H4j a 0

0 r=0).U 1E4 X0 0 0 z- 00. 4) V C .,.4 1.O0VO0O44 ;. 0k me *r 4) a 00 4 0 -A40) 04-

>c w 400 ) - $4 0Jiu O-w4i 0 Pe~ 13 4O 0. 0 144w F_ Hca

E-4 z ) = r.ca 0 U) co 0 >% 0 r4 a)00 OX ~ 0 0 00 '4

, to 00 j 41i,4 0 44 W 0> '44 4 U 0 -4 - 0) W -4 Q)0 co C 0 -4 ca ) ".-1 CU 1-4 (a 0 00.m '44i 0 r . 100

000 u.00w >1-A -4 >..-A CL -- 0 0 44 LO 4J- -H

. 4-4M. z0 I U 134 0 r n 54 "H w.40 0> -H >1 4

0 U-4kv4. Z 0-d 0 ;0-Hr- P .04 0 00 r-4 1-40c

cc C0 z " r 0-4 -4 00 " 4- to- V4 44 4 -( -4 0)41

0 4b 0-o ".e4- o10-. 0 CU-,-4 ;Pl-r4 4 00 CO co 0r

en en:* 0 x .0 -H4u X u CU 0) -4 0) -H : -H R 00

"*HW -H- 00 04w4w co V-4014~ 1-4 = .V 44 V 4.9- 4 ig Ou n> pk u> 4J01- 4 44 Aio X cc 4 -4 0v0- 4 004 $4- r. l0(

encDc -rO O-- c0"- 0U c-4o NO wco C 0 0 -H -A s 1w1 co C-H Xp-

-4

- H ~-4 044 1-1 0

CU C>

00l w e 0

H.0 0o 0 0 44 0) 00 m0 A0

d CU -H 0 001-i C 01

m.0"0 ~ - E-4-aP

en4

V44 to U >% 41

A 0

4-40

-40 kOUV

F-4 cc 4 0) U

54
Ilk k%,.,C

4) x4 14"~ r_ 0

Q. tjV 0 0

co r-4 -000 -f4 c c

0 0L 4j 44 0 uW Li

- .~ ) W 0 ,4,4 r a 02

V0 Q~i ~ a W Q -4 0 j() (

ui w4:3wQ *j0 44:? C

&j0 w.j02La *"4 *4 0

ca t20m 0 a :3 -r 0 004 -~ 44 .- 0 -44 *r4

CU -4 U'- Li *'- W$

m w2 to V~ -40 0 V

0 u 0( M$ .0 cc wW, 5) U

r -4 0 1 a . a a 0 C>% 0 44

4Li ai C13 0 0 0 $4 Lij . 0

CL 'f W _,4 $4 ca U '-4 r-4 r-

-4$4 M r4 -i J j 0 14 W) 0 0 C:

w4 $ 0 '4-Z4000. W- c "a *4 a 0

V C 4 -1 0 0 cor Li -A. i > *..4

(n~ Li CUI - : c CUW 00 c00a ) Li

Q) u A, w02Li"4QW $4 > -,4 00 4J .4 co

d) M. C d x 0 4 -4 0.X- Li

W 4 0Q-a ~ u0 a L ((a00 9WV 

:5 .- 4 ca o0 Q)C co V.0 0 0 0

a) co V C A O 0 rz .1 0 -4

> -4o1.4c 0A -4.l 0*- 0..- .o 5 0 C

CU ca 0.0 s U Q)Z 04 4 0~

u .4 *r4'0 4) 0. W Ia .r0

En- 0 0 0 0 to-4. 0. "4 00 0 *$4

0 0-4 -HC > *) *0. r= -)4W 0 ~ -4

04 -4 HI 4 0244 0% -r4 WO 4c

.0 V ,4 w U0 0- 4-- 0 *.-4. -f4

z p0"a0 A >- 0U2 0.0 ,-4 0 *- w OAi

> %is$4.,V4 0 AUw -4 40u .- 00 0 '4.'

E-$4 W 040WO0 W "*m J. u4 000 0 ) 4-"

WO b Aj .C~ 0 C: 4 .- c "4 w .. 4 Q)0

to0 -40c 00w '44 * 0 O0. 0

u p 0) u r

4 0 14) CO A J " ,

.0 4.4~ 0 0Wa $4 k40 4)>1-A 0 >s Ai

0 co 44U-4 . .0H 4) C 0 r4 WC ,-4 C

0a A i w'~ L 04. e02.v 4 r4Lit t cc4 I CI

CA 02 wQ( o r-4 di 4 0 r4 r- 4

-Ha ,' -H C p =.a 4 a$ 'I.4 W U.4 $40S

0t 4 4 U > cVu> $40c b 0r 0 OW $4: 41 )

02i to Cr0 d 0.-4 0 c A4~4 $4 0 r4 A -4

Q)L W U) ~ &.-. 0)C ~Z W i ~.00 4 o A o0 0.2 E! w0.

(/2 0

-4

bo .0 w

C: -4 0

CU C>

1-40 to

4 4 -4 00 c - 0- 0$4 C: -l4 e-,

-W r0 -H X

-r4> (Co I $U4 -4 C.)

5.4 m 0) 0. W4

Wo 0)$4$

0 4'

0 0

44 th4453 H.; 0 9:

554