- — ______________ 

- -

~ 

- r~ ~i

~ i’~~~’ 

- .

Technical Paper 303 

~~~~~~~~~

TELEVISION AS STIMULUS INPUT IN

SYNTHETIC PERFORMANCE TESTING

John T. Cockrell

Q

C)

-J

C.3 ARI FIELD UNIT AT FORT KNOX, KENTUCKY

~

OV 16 1918 

~

ft

Li UL.~

L L ~

JU~~L

~

J

~

iri

U. S. Army

Research Institute for the Behavioral and Social Sciences

August 1978

Approved for public releese; distribution unlimited.

-

. ~

r.
U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency under the Jurisdiction of the

Deputy Chief of Staff for Personnel

WILLIAM L. HAUSER

JOSEPH ZEIDNER Colonel, US Army

Technical Director (Designate ) Com mander

NOTICES

DISTRIBUTION: Primary distribution of thu report has been made by ARI. Please address corresporiden

~

,

concerning distribution of reports to: U. S. Army Research Institute for the Behavioral and Social Scuances,

ATTN: PERI.P, 5001 Eisenhosse r Avenue, Alexandria . Virginia 22333~

FINAL DISPOSITION: This report may be destroyed when it is no longer needed. Please do not return it to

The U. S. Army Ruesich Instituta for the Bahsvioral end Social Sci.nces.

~~IL The findings in this report are not to be construed as en official Department of the Army position,

uniass so designated by other suthorizad dOcuments.

E

~ 

~

S

_______ ~~ tr ~~~4~~~~~J ~~~~~~~~~~~~~~~~~~~ 

- _____
— ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~ ~

:-

~

7_ ~~~~~~~~~~~~~~~~~~~

UNCLASSIFIED

SECURITY CLASSIFICATION OF THIS PAGE (WIiw Dot. Entered)

D~~~ e~nr I ~~sIss~~~vrA1~Ifl5 I DA~~~ READ INSTRUCTIONS

I’~~ruI~ I 5i5J..UM’ ~~I’I I 

~~ 5 5~~ F 535~ BEFORE COMPLETING FORM

~ RT ‘~~~~~° 2 ~~~~~ t tI5 ~~~~~~~~~~~~~~~~~ T 5 ..n’ .s ,...-.. MBER

~

j 1Zhnical/ap

~~~~3O3 1 i

~

i AR

~~ - T~ - .

~~~~~~~~~~~ ~~~ I TITLE (nod SubdUe) I—.J-rPE-

~~F .,EPwRT è PERIJO COVERED

• ( ‘

— TELEVISION AS ~~FIMULUS ~~ PUT IN ‘

~~~~~~~ SYNTHETIC ~~ RFORMANCE j

~ STING . 6. PERFORMING ORG. REPORT NUMBER

1. AUTHOR(.) 6. CONTRACT OR GRANT NUMBER(.)

(

\~ 

JJohn T./Cockrell

I. PERFORMING ORGANIZATION NAME AND ADDRESS ~~~ ‘. 50. PROGRAM ELEMENT. PROJECT . TASK

• . ,/ ,

~~~ ‘

,, AREA & WORK UNIT NUMBERS

U.S. Army Research Institute for the f r • Behavioral and Social Sciences 

~~ 

J 2Q763731A7 7p’

5001 Eisnehower Avenue, Alexandria, Va. 22333 “— °

~~

(.—— ———

• II. CONTROLLING OFFICE NAME AND ADDRESS ~~~~ — REPORT DATE

U.S. Army Training and Doctrine 

\

~‘ç// ~~~~~~~ -.

~

.F1— 7

~~~7 Command (TRADOC ) . N

Fort Monroe . Va 38 pages 14. MONITORING AGENCY NAME I AQO~

RE$5(ff diff erent ice 53. SECURITY CLASS. (ol tAle report)

-- 

J/1 ,~ Unclassified

-- ISa. DECLASSIcI CATION/DOWHGRAOING

56. DISTRIBUTION STATEMENT (of Ihi. Report)

Approved for public release ; distribution unlimited .

57. DISTRIBUTiON STATEMENT (of th. .b.lract .ni.r.d tn flock 20. ii differen t from Report)

IS. SUPPLEMENTARY NOTES

59. KEY WORDS (Canit nu. on reveres old. II n.ces.N1 end Id.nttfy by block number)

Simulated performance testing

Synthetic performance testing

Television testing

Job-skill testing

20. ASSTRA~~T (Continue on r.ver.. .id. U n.c... y end identify by block mesb.r)

-‘~ s part of an effort to provide more economical methods for job-skill

evaluation in the Enlisted Personnel Management System (EPMS), a study

• investigated the use of television as a means of providing test item

presentation . The television scene provided the job setting for each item • and then posed a question that required a real-time response from the

examinee . All items were job-connected and therefore represented simulated

skill items. The exaininee responded directly to the face of the television ,_

‘

/

screen by means of an electronic stylus or gun reticle . The test was ~~~~~~

DO ~~~~ 1473 EDItION OF S NOV 56 IS OBSOL ETE tTnclassified

~~

, •.

,

~,j ,, .

~~~ 

...

,

~~~~~~~~~~I(

CUR ITV AsSt :ICATI0M iwli PAGE (rinn oot . Enesr.d~

— ~~~~~~~~~~~~~~~~~~~~~~ •

~~~~~~~-~ -- — ~~~~ - ‘ 

, 

~~i~~~’
____

• Uncla ssified

• 

• SICUNITY C~~AUsFICATION OP THIS PAGI( ~~

Ren Dais IisIe.ed)

~

compared with a paper-and-pencil test which covered the same items and

.

~ hands-on oerformance test which covered many ~ f the same items. The

re~iu ] t

~; m d i cat ed that tel cvi Sjon te ~

;t j ~~q I vi rv ,tcc, t u t ,itil c to sold ier~

and feasible. The val i.iity of the prototyu r’ tu’ ~~t could not be determined

precisely because the criterion hands-on tcst was not usable .

Pu

I

~~j

F

l htcl.~sslfIed

• SECURITY CL AUIFICATION OF THIS PAOt l

~~~en bat. Entered)
T~~~ ~~~~~~~~~~~~~~~~~~~~~~ 

~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

•

—.. -- •--

~

—.•—

~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

TscIwiIc

~ 

Pa sr 303

• TELEVISION AS STIMULUS INPUT IN

SYNTHETiC PERFORMANCE TESTING

S 5 ~~~~~~~~~~~~~~~~~~~~~~~

• •

John T. Cockrell

• and

Donald M. Kristiansen, Work Unit Leader

ARI FIELD UNIT AT FORT KNOX , KENTUCKY 

. ._____

Submitted as complete and Approyed By: technically accurate, by:

Donald F. Haggard E. Ralph Ousek, Director

FI Id Unit Chief INDIVIDUAL TRAINING AND PERFORMANCE

RESEARCH LABORATORY

Joseph Zeidner

TECHNICAL DIRECTOR (DESIGNATE)

U.S. AR MY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES

5001 Eisenhowe r Avenue, Alexandria , Virginia 22333

Office , Deput~

t Chief of Staff for Personnel

Department of the Army

August 1978

Army Prol.ct Number Performance.Orlentad

20763731*770 lndlvidu*I Training

j ApprO~

Sd for public release; d~strlbutIon unlimited.

• — •
~

P1

~

ARI Research Reports and Technical Papers are intended for sponsors of

R&D tasks and other research and military agencies. Any findings ready for

implementation at the time of publication are presented in Ise latter part of

the Brief . Upon completion of a major phase of the task , formal recommen

• clations for official action normally are conveyed to appropriate military

agencies by brief m g or Disposition Form.

1⁄4~
______c - 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~ .,• ~~

, 

~~~~~~~~~~~~~~~~~~~~~~~~

FOREWORD

Research on improving performance both of qroups of soldiers functioning as a crew and of individual soldiers forms a major program at

the Army Research Institute for the Behavioral and Social Sciences (ARI) .

The Performance-Oriented Individual Skill Development and Evaluation

project is concerned with improving relevance , efficiency, and economy • of individua l enlisted training and evaluation . In a major move to

• imp rove the combat readiness of soldiers , the Army is implementinq the

Enlisted Personne l Management System (EPMS). This system requires

restructuring the individual training and testinq systems to make them

job relevant.

Evaluation of the individual soldiers for career progression in the

EPMS is based on criterion-reference performance testing of actual job

skills rather than generalized knowledge . In an effort to achieve more

economy in the large-scale testing required for the EPMS , a re search

program seeking to develop simulated performance tests has been m i -

tiated. One promising line of endeavor is the use of various audiovisual media to provide the stimulus input and job setting for the

skill items . The present publication reports the results of a study

investigating the use of television stimulus inputs in conjunction with

an electronic responding vehicle which requires real-time decisions and

responses.

This research was done at the ARI Field (m it at Fort Knox , Ky ., in

response to requirements of Army Project 2Q7637 31A77 0 and to special

requirements of the Training and Doctrine Command (TRA DOC) , Fort Monroe , Va.

ZEI

ical Director (Designate)
T~~~7 T ~ ~~~~~~~~~~~~~~~~~~~~~~~~~ ‘iP5~~~~ ~~~~ ~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~ ~~~~~

TELEV ISION AS STIMU LUS INPUT IN SYNTHET IC PERFO RMANCE TESTIN G

BRIEF

Requirement :

To investigate the validity and feasibility of using television

stimulus inputs in a synthetic performance test , and to determine if

such tests can replace hands-on performance tests. This research is • considered necessary because of the hi gh cost of hands-on tests and

tI

~~ need to develop a less expensive , reasonably valid substitute .

Procedure :

A synthetic performance test using television as the stimulus input

was developed and produced . The test was considered a performance

test because the items covered the actual tasks the examinees were

required to perform on the job . The test was administered to 70 soldiertrainees who had completed advanced training in the subject matter.

Score s made by these same trainees on a hands—on performance test

which had similar items were also obtained. The hands—on test was

administered routinely by the Army to all trainees at the end of the

advanced training. A parallel paper—and-pencil test was administered

to 64 soldier-trainees , and hands-on scores were also obtained for

these trainees.

Findings:

The results favored the feasibility of television testing . The

test was produced and administered without difficulty, and the exaininees

• had a very favorable attitude. The exanminees had no trouble understanding and responding to the items. The examinees judged the test

as “fair ” (impartial) in terms of testing them on important tasks they

should have mastered.

The validity of the results was inconclusive . The criterion scores

for the hands—on test were unsatisfactory in that most examinees made

F1 a perfect score. The correlation between the television and hands-on

tests was low—positive but nonsignificant. Comparison between the

• television and parallel paper-and-pencil tests also showed no overall

difference , although there were significant differences between many

items.
~ z

~~~~~ -~~ : ~~~~~~~~~ 5T2

~ ~~~~~~~~~~~~~~~~~~~~~

Utilization of Findings :

This study provides insufficient evidence to conclude that synthetic performance tests with television inputs can replace hands-on

performance tests. To determine more precisely whether television

t est iuq has promise requires the development of a more satisfactory

hands-on criterion test and a more thorough examination of those tasks

and response components that appear most amenable to television testing .

Ii

r5

~

kL_L___ • — 

•••- •S -- - ~
• -

~~

_______ 

.--“•, .—•-- ~• • -

~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

I -

T1:L.m V tS I~~N A~ ~

; 1 ’1MVIt ~S I NPLIT IN SYNTHETIC 1’!: - ’

~~.MANcl- T~:sI’1N~;

~

‘ N1’~

- NTS

P.iqe

1 ri ON •

• 1 ( ~k~~ro u;1d and ~~t t jonale 

~~~~~~~~~~

~

U ’1’H()I) 

Tel (~ViS ion ‘i’t ’ t 

Paper— md— Peitc 1 1 To n t 1

} i i n — ~~n P t r f orn w~~

o Te

~

;

’ 13

Response Equipment 13

~~x~imi ~~~~ 14

P r~ c ’duro 14

RESULTS 15

Pe~m ibi1i ty of Usino Te1e~’i~~ion in Testing 15

Acct ’pt .

~

i nct ’ of Television Test ui 1 I ~‘

Comp .ir i -

~c’n of Telev i on , Paper—and—Po ric i 1 , and

Hands—On Tests 17

CONCLUSIONS AND DISCUSSION 21

REFE REN CES 

H APPENDIX A. LIMITE D ~

;I1IDELINES FOR THE DEVELOPMENT OF

-

• 

• TELEVISION TESTS 27

B. EXAMPLES OF PAPER-AND-PENCIL ITEMS 31

C. EVALUATION OF THE TELESTRATOR EQUIPMENT 33

DISTRIBUTION 37

Li. ~~~~~• ~~~~. I
—- -

~~ 

-• — -- 

~

‘ 

~~~~~~~~~~~~~~~~~~~ 

—

~~~~

--- 

~~~~~~

-

•—--

~~

• 

~~~~~~~~~~~

LI SI’ OF 1’A F

~!~ S

1 . R’~.c ii t ion t t he 47 it em - ; on t Ii, • t I ov 1 Ofl t t’St . . 11

2. Schedu le ’ ~t il~i J is t it hut iou ~-‘ t exam noes 15

~~. Mean perce”it ~~i lOt ina~k Out the t elev t ton and paper—

auid— p o n c t I t e~; t 

11 4 . An iiv is o t Var t .111Cc -’ for t elev t t on and p

~tt’e r—an~l—

u 1 t ~~ ;t • IS

4- . ‘‘o~~n mo u ~~

e -’:it et rot ia~h’ I’v t me to levis i on and pai~e —

• .i:id — p ~~u tct 1 ~~t cups on the h.in d s— oui t est 

tl . ~~~‘t ~i .i1 I SOfl b~ t We’~ u i t elev t t~ rn arid ç st rnu — .luld — p e ’nc i I

it e ’m” pe rcellt 01 10 t at u •tii ~ied hv u Spon Sc ’ t vp o s.... 20

Corr4la t ion s t- ’e ’twt ’eut the t el, ’vli ;toui and ~~

.1pc ’ i — .1nd—

1 scores arid t he ~iauids _ on P°~ toiu ~anc o scor e S . . .‘ 

1

Ii ST OP PT

Ft oure ’ 1 . Con cept ion of st imul us t t do lit v and fe -ne; i h i l i t v

t t adeo t ~ 2

2. St imulus and 1eS}-’ouiSe ’ f i d el it y ~1imension~ 5

Re spou Se’ components for task Loading round i n i t o

main quni on an Mt ’C7t1 tank 

- - ____
________________ 

~~~~~~~~~~~~~ “‘~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ _

~~~-•-_w_ ~~~~~~~~~~~

L

TELEVISION AS STIMULUS INPUT IN SYNTHETIC PPRF()R!’~tNCE TEs’rIN(;

It INTRODUCTI ON

Over the past 10 years or so , the Army has tried to convert more of

its testing to the “h a nds—on ” performance mode , esnrnciall y at training

centers and at t

~rn beginning skill levels. Even more emphasis has been • • placed on performance test ini g in the last 2 or 3 years with the beqin—

ning of the S ki ll Qualification Testing (SQT) ~- ro qram. Performance

• testinq is highly desirable because of its hi gh face validity and hi qh j us e r acceptability ; howeve r , this type of testing is very costly , hard

t

~~ .,ta ndardi:~

e , and otterm not feasible

The alternative to hands-on performance testing has q en e rally been

the standard , group-administered , k nowledge -type , paper—and-pencil

test. Although relatively easy to produce and a d m i n i s ter , this t v n>e

of test is generally considered to have low validity and low user

acceptability.

Osborn (l~17O) has suggested tha t a compromise validity-feasihili~ y

tradeoff point might he reached by using synthetic performance tests.

According to Osborn , t h e ’ term “syn thetic performance test ’ re fers to

• any performance test that is less than a full hands-on test , but more

than the group-administered , knowledge-type , paper-and-pencil test.

Synthetic performance tests include all tests that use any type of

simulated inputs or responses. Part—task tests , in which only one or

a few response components of a task are measured , are also included

under synthetic performance tests. The synthetic performance test

is conceived as less costly than a hands-on test, but as a test that

still has reasonable validity and user acceptability.

To support the Army ’ s adoption of performance testing , t h e U.S.

Army Research Institute has initiated a broad-based research program

to investigate the possibilities of synthetic performance testinq as a

• cost-effective alternative to the usual hands-on procedures. The goal

of th is research is to develop a psychome tric base for both hands-on

and synthetic methods.

The research focus has been on the use of audiovisual media to provide the simulated stimulus input . The reasoning behind this focus Ic;

that audiovisual media stand midway in the stimulus fidelity range , and

at the same time , are at the medium to high end of the feasibility scale.

Thus , audiovisual media may represent a good f i d e l i t y— f e a s i b i l i t y tradeoff

1

Stimu lu s fidelity as used here refers to how closely the test stimulus

resemb les the real wor ld , and stimulus feasibility refers to how much

it costs to present the test stimulus in a testing situation (high

feasibility equals low cost).

1

‘H - • -• 

— - 

~

-

~~~ a- - .....‘ ..a .._ ’- —
‘ .

_ _ - •; _ - - ‘

~~~

_ 

_____________

SIIMULUS FIDELITY AND FEASIBILITY

LOW MEDIUM HIGH

STIMULUS STIMULUS STIMULUS

FIDELITY FIDELITY FIDELITY

A A A

PRINTED STILL SLIDES TELEVISION SIMPLE COMPLEX R EAL

WORDS PICTURES PLUS OR 3-D TRAINING EQU IPMENT

AUDIO MOTION MOCK-UPS SIMULATORS

PICTURES OR

SIMULATORS

H- - HIGH MEDIUM LOW

STIMULUS STIMULUS STUIULUS

FEASIBILITY FEASIBILITY FEASIBILITY

Figure 1. Conception of stimulus fidelity and feasibility tradeoff.

~~~I . I 2
- ~~~~ ~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~ • • • ~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~‘vr ~ jU~~

point insofar as stimulus input in; c ’oiic~

’rtied . El qure 1 shows a

t eilt of this ftdol ity—f easib i t ity t radeoff.

The’ overall research pi ~

;gram has the follow i ,m~i object i vi’s

1. To explore the pa r ameters of the v,ii ions ,iudiovisn.il nn’dl.a t-~ clot ormine the medi,~ ‘s appli cability to synthe t Ic pei formance t e s t i ng .

2. To explore various responding modes and ~~p~~ qçs ~j \. %~~5I5 that

call be m1s (’~l with audlovi su,il st • imulus input s.

To de termine whether t hone-’ r (’uI’onn(-’ eonqlomm (’nt of a t ask that- - 

can ho measured using and levi nu,a 1 media are suff i c1e~~t to yl el it an

accept .-ih 1 e measure of the om it I to t ,isk

4. To devel op a task classificat-ion ’*y.i t om that will i ’niahli ’ a

• synthetic performance test deve leper to determine by . arial y~ 115 1 t ue t .

-i ek

(a) when aud i ovisua I media should be used as the’ st mmm l us I nput- , (ht

• which medium is advi sabl e, and (c) which ron-e~~’nso components shou l ii

be measured .

Several experiments in thi n ; research program are now i ~‘i 0cO’O’

Using a number of different audiovisual media. This paper , wh i ch I’;

concern ed wi th television as the st imulus i npu t , presents I tu’ t i’

~~ u l I ‘. ot

li e first of the’se’ experiments.

This first experiment was 1 tmtt i’ei to nature and fe’cun, ’t or; the

feasibility of un; i ng television as the st imu lus i nput . A~ such It Wa ’;

concerned mostly with the fir st research obj ect ive—— appl I I l i t y of

media to best- leg——with some exploration hun t e; the ~ ‘corid .iisi t liii d

oblectives, responses to stimul i and test—task comt’.lt i ’~ot;’; .

R ? tional y

The impetus for this research stems from the Army ’s dc~’i’ ;toii t o

substitute the Skill Qualification Test 1mg (S~ 9’l progr.am for I he 1 ’urt cot

MOS testing program as a means of assessinin the jot’ ski ll s of e nlis ted

personne l . The SOT program itt Intended t i’ be b.-tsed on Iot’- - samplc t e n t

wherever prac I- t eal , a ~ ~~~~~~~ t ,.isted t o t he’ current MOS ~~~~ ‘o i • and —pene ’ I i

knowledge test

This eh.aneio wan brouqht .abeut pa i t t ally as a r

~

’’;u It of the u , ‘‘n r ’al ‘lm

of a numbe r of invest iqators (Enqel , .InI y 1 ‘

~~~ I’ m ;~~q ’ I , Oct ~‘hr i I ‘

~ ~

S

F.nge 1 ~~ Rehder , 1 ~ 75 Sh I ikey , ~

e’’ Iii y , SI; ii ki ’v •‘~ NI t’eW~ nili’r ,

who quest toned the validity of the MOS test foi t~~’t ’ ‘ik t II ,

~ssessment .

In 1966 the Army eonvene~1 a specIal beard of I nqul y (Brown 1

~~

at-d) I

survey the entire question of written MOS test ‘e h ’i assessing jo t s ‘nk i l I ~ and jot; knowledge . This boa i~l ri.;’omim’n.ieil that pe rfonna n~’e t i’s ;

‘nubstituted for w i lt I i’n tost~ wherever t;rai-t I elI I ’ . . Aimy

I lowing the pub it cat Ion of the f i,;~t t isis of t Is’ BEOWn t1⁄4’ai~t , t hi ’ Aim y — has made subs t ant Ia 1 ~

‘~~ ‘i i

~~’ nun in I nip I i’met;t 1Im ’ 1 I he rc’ceM~fli’s tat i

~~- ’ll Ce . •
______ — — 

‘

~~ I~~~~~ -,-‘s,r

~

,

~~~n.r ~

‘ 

T ____ - - ___________ - — ~~

the Tank Crewman Advanced Individual Training performance tests administered in the fo rm of a “county fair ,” with examinees moving from

test to test around the examination area , during and at the end of

each training cycle). However , due to high costs and difficul ty in

main taining standardiza tion , the performance test obviously is limi ted

in terms of making up a substantial part of each SQT test. This is

particularly true at the higher skill levels and for many hard-tomeasure tasks. Occhialini (1972), for example , presents evidence tha t

per formance tests are extremely difficult to prepare and administer ,

and are of questionable validity . Engel and Rehder (1970) review the

arguments against the use of performance tests for part or all of the

SQT battery . Their general conclusion is that the exclusive use of

performance tests in an SQT battery would be too costly and impractical.

-- 

• Reacting to the pros and cons of paper-and-pencil vs. performance

• tests , several researchers have proposed compromises. Engel and Rehdcr

( 1970) advocate a mixture-of-measurement technique in each SQT test ,

combining work samples , simulated tests , peer ratings , and paper-andpencil tests. They present evidence indicating that cognitive items

can be measured adequately by paper-and-pencil tests; that motormanipulative items require work sample or simulated tests; and that

peer ratings can be used to judge social , leadership, and overall

ability.

Osborn ’s (1970) approach is concerned with developing synthetic

tests that it is hoped will eliminate some of the impracticality of

administering performance tests, while reducing the verbal component

and improving the validity of paper-and-pencil tests. Osbo rn visualizes a continuum bounded on one extreme by paper-and-pencil knowledge

tests and on the other by job—sample skill tests. Within this

continuum, a number of synthetic tests more or less removed from each

extreme can be constructed. The continuum is conceived of as beinq

scaled in psychological units and varies along the dimensions of

stimulus fidelity and response fidelity (or a mixture of both) .

In any combat situation , the stimulus dimension would be a large

- ‘ complex composed of visual , auditory , tactile , kinesthetic , olfactory,

~~ pain , and stress inputs. The response dimension would be an equally

large complex of cognitive ,, motor-manipula tive , and perceptual outputs .

For the purposes of illustration , the stimulus and response fidelity

dimensions for armor crewmen might be conceptualized as shown in Figure

2. Osborn maintains , in an analysis similar to the one shown in Fi gure

2, that one must pull away from each extreme of the continuum to develop

synthetic tests that are both feasible and more valid than paper-an --

pencil tests.

An important aspect of Osborn ’s conception is hi~

1~ reasoning wi th

regard to part-task testing (Osborn & Ford , 1976). In this conception ,

each task is composed of a number of response components divided into

cognitive , perceptual , and motor behaviors. Figure 3 shows a task broken

--— -———

4 

-

~~
•1

I-

- - - I - - - —

• -

:0- ~~~~ I- • 1’I

- . •• • - U

- ~

_ • t. --

J. ~

-

~

:

‘

~

I

~ I’ ’— , -

~

I
—-~~~~~~~--~~~~~~~~~~ • --- ~~~~~~~~~~~~~~~~~~~ ~~ r: T ~

“‘

~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ = . ~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-j

Ilk AR

I kIRk I(l

~lA~

dL)

[__M~ctr’flIA~

L

~

I

~ 

> [~ A~~~~~wLtcH 

~~~~~~ 1

RECA U

I 11 I~t ’I I

(O (~

NI1 IV ~

[ P1C~~UP ~~~~~

[

~~~ 1AC1 ROUND 

~~ ~NTO CHAMBUJ ~~ ‘ I 

~

f sun

L~

ToP Mt~TO~ f M0~~R

I — 

RICALL RI cA l l - -

I SAFETY RULES ‘

SAl t 1Y RUtt\

~~~~~ -

Figure’ . Response’ components for task : li ’ ,sl i u sj

round i itt e’ main quo on an MisSP~ 1 tank

I,

_______________________________________________________ 

F ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~ •-- --

• ~ -- --r ~~ :~
down into response components. (This task ~~

n ; performe d by the l oader

- - on an M6OA1 tank.)

The reasoning behind part—task test i itci i S that it may r u s t be nt’ces—

sary to test every response component in a Isirt icuula r task in oudo r

to determine how well the whole t ask can be pe r formed . It may be

possible to get a good indication of whole—t ask pe rt on rn .uncu’ by mets ur req

only a few response components •r perhaps me’asuu Iu; ~i only one’ ci It ical

response component.

Part—task test- jug becomes crucial when .muu ~liov .sual stimulus inputs

a re ’ used because the nature of the medium ~u r ecl u ulos otst a i n;inq any

measurements on most motor—response’ components. In order to obtain

measurements on motor—responn;e components one needs to test on real

equipment or a hands—on simulator. S juice the measurable response

c ’ompo:;ents in audiovisua l s imu l 1 i t ion are 1 imi t ci to perceptual and

cognitive ones , it follows that t h ~ usefulness of audiovisual stimulus

inputs is dependent upon the validity of the part—task t ‘n t inq c ’on;c,’Ist

n ine oh je ctivi’ of tht’ research program is to check t h e part —t ask testing

concept by court ’ hit ing scor~’s made on part—tasks using audiovisual

nt jmu l us inputs with n ; 5 ort ’s made on the corre nu p orul i rig whole task tes ted

in the h anuls —on mode .

Use of Television in Testing. Television has been used in testing

primarily as a recording medium (Cockrell , 1974; h ays & Pulliam , l~~74)

A study by Shrive r (Shriver , Hayes , & Huthand , 1974) explored the

possibilities of using television as the stimulus input in a performaruce test. After developinq the test , Shriver concluded that television (lid not offer much p romise in terms of replacing hands—on tenting .

lie listed eight disadvantages of the television medium and decided to

,ebarulon the method and not attempt a systematic comparison between the

television test and hands-on performance tests. Some of the disadvantagk~s mentioned follow :

I. Television tests place t he subject in a passive role , watching

someone else perform and evaluating the correctness of t h e performance .

The re is ito reason to believe that success in thi s evaluation role’ will

insure success in the active role of performing the task.

2. Television violates i major ground rule of criterion-referenced

testing in tha~ it emphasizes process measurement rather than product

measurement.

3. Television costs are very high compare~

i to those of slides or

graphics because of th e’ large’ amount of equipment needed and the large

personnel time requirements.

Shriver ’s criticisms are informative , bu t they do not necessarily

settle the case . The nature of the medium does include some practical

difficulties both in producing the stimulus tapes and in administering

1 7

I-I
_ , ~~~~~~~~~~~~~~~~~~~ ‘- ‘,—,,-— ---——--—- --‘ .— ‘—‘~~

---‘--. ——--‘-. — —‘ .—-—.‘-- — 

-— ___________________________________

• - - - — —~~ - - -- 

~~~~~~~ - ~~~~~~~~ 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

the tests. However , these difficulties are minor compared to the complex

task of administering hands-on performance tests. If television can show

a useful correlation with job-sample tests and also show advantages over

written and other audiovisual tests, it may well he worth the extra cost.

Objectives

The primary objectives of the present experiment we re to appraise

some of the practical difficulties in using television as the stimulus

input and to make a rough comparison among television , paper-and-pencil ,

and hands-on performance tests. Th e secondary objective ~as to conduct

a checkout of a responding device (Te lestrator) designed to permit

examinees to respond directly to image s on a screen (s ee Appendix C).

Specifically, the objective s were as follows :

1. Determine the feasibility of using television in testing . The

items under consideration here were

a. Understandability of test items

b. Ease of responding

c. Time allotment for responding

d. D i f fic u l t i es and costs involved with administering television tests.

2. Determine the acceptability of television testing by exarn inees.

1 . Compare the results made on the television test with those made

on the paper—and—penc il and hands-on performenci’ tests.

4. Conduct a checkout of the Telestrator response device .

METHOD

The overall method consisted of ( 1) producing a television test for

a sample of tasks from the job field of t~ink crewman (lIE MOS) , (2) pro—

d tc tr ig a para l le l paper-and—pencil test covering the same i tems, and (3)

comparing the results made on these two test s wit h the results made on

an existing hands-on performance test that cove red man y of the same items .

The ~

ob field of tank crewman was :;e’ ls ’ctod because much prior

re search had been done in this field. A complete task analysis was

available , and a h ands-on performance test has been in use for the Tank

Crewman Advanced Individual Training course for 2 y ears. This existing

hands—on performance test was felt to be a good base against which to

compare the television and pap e r— a n d— ~s’nc i 1 tests.

8

______ - - -
• ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - _____________ ‘

~~~~~ “ 

-

~~~~~~~~~~~ ~~~~~~~7”’

Television Test

The first step in producing the television tape was to select the

cri tical tasks in consultation with military experts. The selection

cri teria were set by the military and included such considerations as

importance to fulf illing the mission , safety to the crewman , and safety

to the equipment. The critical tasks selected were quite similar to

the tasks covered in the Tank Crewman Advanced Individual Training

course . After the critical tasks were selec ted , they were ordered

according to skill level.

-

~~ For the final test , tasks were selected from skill levels 1 , 2, and

3~ 2 For the purposes of this experim ent , the tasks can be considered

to range from fairly easy to very difficult. Tasks were also selected

such that each of the four positions (driver , loader , gunner , and tank

commander ) was covered , and a few tasks pertained to the crew at large .

In consultation with the military, each task was broken down into

cognitive , perceptual, and motor components ; and each response component

was examined for its criticality to the task . Practical considerations

such as overall test running time , time to televise each item , number of - response components needed to cove r a particular task , and achieving

a balanced test (see Appendix A) eliminated many critical response

components. For each of the remaining critical response components a

television test item was conceived and a television shooting script was

written. Each item was televised in a crude fashion with a handheld

camera and a portable videotape recorder.

The raw footage was edited roughly into a prototype television test

by the addition of narration and titles. The prototype tape was intended

only as a model for a professional tape to be produced later and as a

vehicle to check technical accuracy and television feasibility. 3

Military experts checked the prototype tape for technical accuracy

and understandability. A revised television script incorporated

suggestions; a final television tape was produced using professional

television personne l , cameras , and editing facilities. The shooting and

editing of this final tape required approximatel y 30 calendar days (about

15 actual working days).

The final tape consisted of 47 test items plus 4 practice items and

had a running time of 53 minutes. The items ranged in running time from

1 There are five skill levels for each MOS ranging from skill level 1

(beginning ) to skill level 5 (most advanced).

2

Work on the preliminary television tape and the task selection required

to produce it were done by Human Resources Research Organization under

contract to the U.S. Army Research Institute .

9 

-
- -- 

~~~~~~~~~ ~~~~~~~~~~~~~ ~~~~~~~~~~~~

t

C C 000.4

2 ° 2-2 .2~ 8

4 (4 0 014 0 0000 00 0 0141414 0 0 140 0 0 0 .a ~~~~~0 0 09 .004 U .0 .0.0.00 .00 .0 140 U~~~ .0 U 0.0 0.0 0 0.0. 0. 0. U ,~~~44 U UUUO 0)) 0 0 0 4) 0 0 0 ~~~--o U -l U 0 - 4.4.44

~~~

14 

.1:_ _

~~ .

~ ~~~~~ .~~~~

~ 

• . 040.~~~. Q 

~ 2 ~~~~~~~ ~~~ . ~~~~~~~~~~~ .a .aO (a ..14140 .,0 .a U~~~~~ 0 O14 14 0 0 1 4 0 14 0 0000

~1 —I 14 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ U 1 4. .4 —4 5. 4) — 4) 4 14 *4 .4* 1 1)

u- 0 0 14 0000 00 001414140 ~‘ 1 4 0000 5.

~~~ I., Z x~~~~ x XX..4 *s I.I E 0: ~~~~~~~~~~~~~~ 14

4) 4 400 0 4444 044 440 4 )9.4 4 94444449 44 (4404 00 0>> 0 0 00 >0 00>0>> 0 > 0 0 0* 0 000

4) 44 44 .4’ ~~~~ “ 14140 4 144~~~M~~~~~ 4 14 ~~~~~~~~~~~ ~ 9 0 00 0140 C4. 0. 0.Q. * ‘4 ~ 0.0.01414. ’ 0. 0 0 . 4)0 . 1’ 0 4. 0. 0. 0. 44 0 00. 4 . ~~~~ 44990 44 44 0 04 4 4 4 .~ 0 4049 4 -

~ 444)00

C U U UI CC O U 000C U .400 0 1: U Ut) 0

44 14 040’ 0’ 0’ 144.1•4. 040’ 115.0’0’P’ C’ 14 0 ’ 1 4 0’4. U’ 0’ 1411141.

> 0 0 00 0 004 ( 0 0 0 0 0000 44 0 0 044 0 0 44( 0

- I- .110 UUU U 0.44. 4. 44. U 4.4.UUVU 04.0 4) 0 9. 44. 4.

.4 .4 >4 ‘N ‘N ‘N (‘4 ‘N IN ‘N ‘N ‘N 0.. —(

> 4 4 .4 14141. 1* 1414 1 4 11 14 54 14144.141411 (4 1414111414 U 144.1454

0440 0 0004) 4(0 0449990 09009 41 09)1 9

~ ~~ ~~~~~~~~~~~ t •~ ~~~~~~ ~ ~~~ 0. 0.000 ~~~~~~~~ ~ .

~ 0040)0 0

41’

44 44

4) 4.9(4~~~~~ 0 00)10 1)0 040441>0’

4-. 0 .4 .-. 4.141414 1411 004: 0

4. 00 4444 (4 0 0~~~’ t1 4444 .400441 44 41 44 * 404444 (4 .4 44.4

• C W00 ~~ ‘0’0’0 ~1 0 1 4 4) .) ‘0 0 • . I a0 ’ 0 1 V ‘V ‘0 ’0 ’ 0 ’0 ’ 0 V 0)>> 0 0.0. -’ 1111 5. 1. 000 U Ut. 014 14 11 5. 14 1414111.4. 5. 44410 4

0. 44 .1. 4) 000 0 -.4~~~~~~~~ 4 00 -. - .4 O 0 ’O 0 00000 0 4.1. 1.

41 140. (4 ~~ 9. 9. 0. 9. ~~~~~ ~~~~~~~~~~~~~ ~ I 111 13 1 004 .1

0

4-. 0

0 t9 * 944 44 4 44 4 444 44m 44444 44 4444440 44 4444(4 0

~ 

u_

~ ~ ~~~~~ ~ L -. 8

~ ,. ‘~

N 5 0 14

• 44 .4 45 441 44 44 44499 44(4 44994444 (5 49(444 (4 1 5 4 4 0*

4) 0 4) 4) 0 0 0 4) 44 4) 000000 4) 0000~~~ 0 440449

-

~ 44~~~~~~~~ *) 0. 9. 0. 94 9.9.94 9.94 94 494 94 94 9.94944... 14 940.94

8

4)

0.

84 0’

U C

44 --I

C

. 4 44

54 44 54

44 0 0. 14 1)

-~~ 4. C

. 4 1 4. 0 -.4 0 1.0 41 14 .4 44 1. --

1414 44. 0 0 .0 0 0 .0 0 0’ 414 0 0014 0.0 - 0’ 4. 5. 4)41 0’

0 0 0 *4 14 .0 .4 .4 tO 4 4 ) 0’ 0

0 4 . 4 4’ 44 0’0’V (4 .4 4)

Hi 

~ 

1111 -d U

~ 

i IU

~ 

Iii ~~~~~~~~ 2 1499 a ~ ~~ 

~~~ .!

~ ~~ .!~ 8 d. ‘

~ o .o .o~~ 8.o 00.4 0 00 00 .4 oo8’ 8’ I, i C • 0 014 0 0404040 U 0.0,1. 54 44 0.0.0.44 .0.0.44 4) 4) 44 .4

00 4 4.4.40 I 4.14 0 5 0 -4 0404(444 44 ‘0 C

-.4 -4 >. .4 .4 .4 0 00 -.440,0. 0. 14 54 04 -4 0’ 0. --4 -4 4. 0 .4

44 4 414 14 .0.0.04 4444 001400 4 04400 4) 44)500 ‘0 0 44 4

4) .4 0 4) 44 *4 4 9 0 ~

1 000009 U 0..4 ~~~ 04 I 1)5 .4 0 0 .0 5.

0 4444. 444444 0 94 . 4 . 4 4 4.-5 0 9440444 4(1 >4>400 I.

’ 0. 14

44 0 04 .4 ‘‘~4 ’N 44.0 ‘

004444 .0 1 4 . 4444.4 44 l’5 O1 ..4 .-( (4 >.

C 0. 0. 97 6. 1 (.6.6. ~~ ~~ X 4-I 4-I L(4W(4I~~~ 4.) 44. 4 ~~ 4 I

~ ~~~~~~~~~~~ II 1(4 0 0’

2 •. . . . . . . . . • 0 0 4 4

~~ D N~~~ 0(0 .4)N ’N~~~~~. 4 (O N ~~~ 0 (0.4N ‘N *1fl 49 1 44

0 ‘.4 N N ‘N C) >4 ‘N ‘N N (4

o .4 .4

44 44 ‘0 ti ll!!

H III H J

~ a

~

t i ___
-

~~ - ~~~~~~ -- ~~~~~ 

- 

~~~~~~~~ 

-. - ‘ :— ~-,---- ------

$11

0000

~

H §. ~~~~ 

U 8 8 U U

H 44 001010 00 I) 0 o 0 4 ) 0 0 0 .

~~ 4 4)1) .4 4(41 0.0.4.0. .0.0 0 ‘ .0 U .00 .0.0 ,. -0: 11 ‘4 ’4 .4 ’4 00 44 4) 0 0 04, 00 1’ 44’ 4) 0 CC C ~~~ 4) -- 1) 1) 4’ .40 4) 0 44 4- 4) 44 00 04) 4 4’ 41• 4 4 44 ‘0 —4 - 4

I 4.0. 9. 0. 9. 0. 4. 4. 1) 4 .4 . 4 . 1 . .49 1. 4. .4 14 ‘1 14 .14 .4 14 141. 14 -4

00 0 0 4) 4) 0 04 .) 04 1)0 444 ~ 1) 0000 0. 4) 4) ~~ *I .4 I. 11 ‘

.4 14 .4 4. .4 .4 .4

1. 0 0 00 00 14 4, .4 4 01. 00 10 4.5.1.14 X1XX IX 4(4 *1 1 444 14. IX I 61 4.44 4.

~ ~~

i•

~ 

ii 

~ 

.

~~~~~ U 

~ ~~U’ 440 00 4)01

0.9.0. ) 0 9. *1*) 9.0. *1 

.4 .4 .4 

‘ *1 

.4 44

0.0. ,‘ 1) 44 1 ’ . 4’ 4 ) Q

.4.44) ~ 0 0 0 0 . 09041 440 -‘ -. -. 00 -‘ -4 -4 4)0. 0. 0000 014 0 C C 0 0 CC - CCC 0 44 4) 1 .1 4 ( 414 1. 4. 0’ 0’ 0’ 0’ 44 14 t’ .0’ 0’ 040404045. - 4 >4) 4404)44 044 0 0 0 0 II) 040 0 000 44

1. 1. 0 0. 44. 9. U 0 4) U 9.44. (I t.) U 4)4)1.) 9. H 1j 1 ~

i 

~ 

I ~ 44

41. 444 .4 10 0404040’ 4)44 040’ 4 ( 0 0 .4 .4 0 0 00 514 CC 14 1,

1.0.4 9 9. 4 4 .4 44 44 (5 41 .1 .4 .410 44 40(4444

4)41044 4444 )) 4’ .-’ ‘0 10’ ‘0 ‘0 >44 1’4 V ‘0 ’ 0 V ’ 0 ’V ~~~~~~~~~~ L’U 14 I. 14 1. 44 0 4. 144.144.4.

• 04~~~~,’1~~ 9.9. 9.9.

—

~ ~ ~ 

1

~ 

:

~ 

:

~ 

~ 

:

~ ~ 

:

~ 

:1:~I

~ ~~~~~ ~~~~~ ~~

10 5-.

0’

C

C

4 II C 44

54 .0 44 4’ 4) 0’ 44 - ‘ .0 -— 1 4. 0’ ml) 0 444 4) -4 4 .0 14 -.4 0 0 440 0.0’ U 44) ‘.4 4j

.097- 4 -.4 0 4) C

1111 u

~ 1’ 1 j h ~ 04.4

!I! 0

~~~ 

111 4) 114)

. • . . . . . 

444 

~~ 

O~~ 4 (‘4 .4

I

~~~~~~~~~~~~~~~~~ o~~~~~~~ o~~~~~~ ~ ~~~~~~~~~~~~~~~~

~~~ V!1 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ P .0 0 ~

’ V”

I

--
~~~~~~~~~~~ —-.—.4- - -,->—------—-44•..4-—-- -—

~~

- ‘.-- -

~~~~~~~~~ -- -> .~~~~~~~~~~

35 seconds to 3 minutes with an average of approximately 60 seconds .

Ten seconds of the time for each item was allotted for the examinee ’s

response.

Table 1 provides a description of the final television tape . The

categorizing of response components into perceptual , cogn itive , or

motor types was somewhat intuitive . The intent was to show the predominant element of each response component and not to imply that

other elements were not present.

Of the 47 items stcwn in Table 1, only 37 were admi ni stered

to the examinees in the experiment and only 30 were scored . Most skill

leve l 3 items we re eliminated before the experiment upon the recommendation of the military staff at the Armor Center. These items were

considered too advanced for the examinees. After the start of the

- • experiment, several mili tary advisers recoum~

ended the elimination of

six more items , and one item was elimina ted due t o a poor television

p icture . These seven items were administered but not scored . The

footnotes in Table 1 give the reason for the elimination of any item

and also explain why certain items were not included on the hands-on

test.

A mo re specific description of each response type shown in Table 1

follows :

(1) Multiple choice . The examinee was required to select one

answe r from a list of three , four , or five alternatives. These alternatives were sometimes the same as those in the usual paper-andpencil test--namely, words on the screen--and sometimes consisted of

images on the screen.

(2) Error detection. The examinee was required to watch a procedure

being performed on the screen and to indicate the time and location of

an error, if one occurred , at the time it occurred. The examinee was

shown the procedure twice and responded On the second showing .

(3) Motor manipulation . The examinee placed a plastic gun reticle

- 4 (those reticles used with the main gun in the M6OA1 tank) on various

stationary arid moving targ.-ts as if preparing to fire the main gun . The

reticles were also used to simulate the adjustment of fire that would

be made if the first round missed the target . The motor-manipulation

response was supposed to be a crude simulation of the actual response

in aiming the main gun . However, the movements required were so far

down on the scale of response fidelity that the motor component appeared

not to be measured at all . Perhaps the reticle response was primarily

perceptual and cognitive .

4

12

. kA
F’ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

- ‘‘ 

~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~ ~~~~~~~ -: 

~~~ - 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ J>

Paper-and-Pencil Test

The paper-and-pencil test paralleled the television test on an

item—by—item basis. The stimulus input on this test was primarily

printed words , but some pictures and drawiiit~s were used on perceptual

items. Table 1 shows the stimulus input for each item.

As wi th the television test , only 37 of the paper—and—pencil test

items were administered and only 3d were scored . The items scored

were the same as those scored for the television test.

The paper—and—pencil items and the television items differed greatly

in the amoui—it of time allotted to respond to each item . The total time

- - limi t was the same for both tests; howeve r , examinees could allocate

the response time any way they chose on the paper-and-pencil toot hut

we re restricted to 10 seconds per item On the television test.

On the paper~ and~ penci1 test , examinees could change their answers ,

skip items and answer later , and review their answers ; on the television test , netie of this flexibility was permitted.

These differences between the two tests were r t am ed because each

medium lends itself mos t readily to the type of procedure used. Any

other pr ocedures or a common pre~~ dure foo both test would have required

much more control and thereby reduced administration feasibility.

Hands-On Performance Test

The hands-on test was one routinely administered to tan k crewmen

trainees as a final examination for the Advanced Individua l Traininq/

Armor course . This test was given in the form of a county fair with

8 stations and 30 performance measures. F.xaminees were graded on a

“go/no-go” basis for each performance measure . For each no-qo , examinees -

‘ 

were required to seek out remedial training and report back later for a

retest. If the retest was a no-qo the examinee had to report back the

nex t day , after further remedial training , for a second and final test.

For the purposes of the presen t experime n t , the score recorded for each — 

exantinee was the number of first-round no-go ’s. This was not a particularly good criterion because the number of no-go ’s was very small.

Response Equipment

A secondary objec tive of the st udy wa s to check ou t the televi sion

response equipment (Telestrator) . This equipment consists of a clear

plastic electronic tablet and associated recordino and programing

components. The electronic t a b let covet-s the television screen (the

tablet is approximately ~ inch away from the screen at the center of

the screen and approximately 1 inch away at the edues Ot the ot ’rt ’t’fl).

‘3
~~~~~~~~ “ ‘~~ TW ~

—.~~ --

~~

--- ~~~~~~~~~~~~~~~~~~~~ 

_

~

-‘.-.-— ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~ -w

The examinee looks through the tablet to view the test i tems . Responses

are made by touching the fact - of the tablet with an electronic stylus or

an electronic gun reticle at a particular time and location . Before the

test , the correct answers (time and location ) are programed on the television t ape . During the test , examinees art- credited with a correct answer if they touch the screen at the correct preprogramed time and Icca -

tion . Any other response by an examinee is counted as incorrect. Only

one answer is pe rmitted for each time , and the first answer——corr ect or

incorrect-—made during the 10-second response period is counted .

The response equipment was in prototype form and because of operational difficulties could not be used for the experiment. However , it

proved possible to test the operating concept of the equipment by p lacing

a human grade r behind each examinee and having this observer record on a

sheet of paper whether the examinee touched the correct location at the

correct time . This grading task was quite simple , and during a pilot run

— with eight examinees there were no difficulties in grading .

The television monitors were black and white and measured 15 inches

diagonally. The examinees sat approximately 2 feet from the sets at

self-regulated distances so that they could manipulate the response implements comfortably. Prior to the start of the experiment , it was decided to remove the electronic tablets from in front of the screens because of parallax problems . After the tablets were removed , the accuracy

of the responding and scoring improved to a very precise level.

The response implements consisted of a stylus used for all multi plechoice and error-detection items , and two plastic gun re tic les used for

motor-manipulation items . The stylus was simulated by using the eraser

end of an ordinary lead pencil. The two plastic gun reticles, the same

design as the M32 and M1O5D main gun reticles in the M6OA1 tank , were

manipulated by small wooden knobs glued to the plastic reticles.

Examinees

The examinees were tank crewme n who had jus t comple ted the Advanced

Individual Training/Armo r course . Altogether, 134 examinees assigned

from three different companies were tested . Examinees were drawn from

- 1 the companies by a selection process best described as haphazard rather

than random ; however , there is no reason to believe that selective bias

was present. As each group of examinees arrived for the experiment for

each session , the group was randomly assigned to the television or paperand—pencil test, Originally, 144 examinees were scheduled for the experiment , but 2 were lost due to scheduling problems and 8 were lost due

to scoring problems .

Procedure

Testing was conducted over a 5-day period in three morning and five

afternoon sessions . The actual schedule and distribution of examinees

are given in Table 2.

14 

—- ___
r ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~ ~~~~~~~~~~~~~~~~

—1

1

Tab le 2

- - ~

;

~ h~ 1iii It Id dt:1 t ritl ut IOn o~ !X IIIIi let -c

‘

~~st and __________

time of 1~~~~~~~~~~~~~~ 3 4 5 Tot~~1s

~~ re l evi sion n 1 e to Ii Ii 712

Morn i n o H 8 H — — —— 2.1

At t c rnooii H 8 8 11 Il 4t’

- - Pet - r— tnd—

petici 1 16 11 112 11 12

Morn tn-i 12 4 lcd —— — — 22

Afte rnoon 8 7 12 11 12

Pa ch q rou of subjects report eli at 1212012 ct I 300 and was ~

i i ven an

orientat ion session exp lainin g the piir ~-ose 0 t the ex~~- r jim- nt . All o t

the paper—and—pencil group was admini stereO the paper—and—pencil test

riqht aft e r an orientation . The television test was administered t o

four ex aminees at a time ; the rest of the television group was assigned

to a wai t mo room . Both the television and the pa~’er— an d—penct 1 tests

required approximately 1 hour to complete.

Approx imately 10 minutes of t ra m ninq we ic re~~u itO t o teach the

examinecs the method s for tesp ondi n-~ to the tel ev i si on items . ~

-i oot of

this t r a i n i ng was concentrated on the use of the pla st ic reticle . The

examinees were trained by having t hem respond to the four practice test

items . If any examinee had difficulty with the ret id es , such as choos—

nq the incorrect reticle or holding ret id es incorrect lv , the tape was

stopped and the four prac~ ice items t

~~e sent ed aqa in. In no case was it

necessary to present the pra ct Ico items more than twice .

- - RESULTS

Feasibility of Using_‘l~~levtsi on in Testing

The’ exam i nec s did not appe a-i - t o have any di f f icul t y in unde to t and ii

tie it ems . Al 1 of the content had been covered in the Advanced m div i d—

ual Training course , and the examinees had been tested on sim ilar items

several times. All of the it ems were also performance based and pose0

b est ions that occur normally in everyday operations.

-

~ 

- ~~~~~~~~~~~~~~~ -- ---~~

-

~ ~

_------ -

--- -

~~~~~~- ._ - .

~

f—~ .— ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- ~— ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~

The responding proceeded smoothly for most items . The examlnees

responded very quick ly on the easy Items (approximately 1—2 set -undo

with the stylus , 3—4 seconds with t h e ret id es) . On di ffi c ult it elm ;

the amount of response time at lot t tnt (iC) second;;) ot ill appeared ample

although there usually would be-’ a lot of hesitating over the answers .

On only a few i tens t he ex ami lun’s I a; led t u r(sslx)nd . When queried

,itter the comp letion of the test about the amoun t of response time ,

n~~st oxamineco indicate d that for the most part the response time W I;;

- - adt’quat e . A few exam ; nces said t hat more response t i uv shou d h.tvc

been allotted to some I tern;; .

The admin i ;;t rat ion ut t hi’ I i’levi 1; ion I cot Wa;; more I I me consuming

than that of I lit ’ ~~~~ r— .and—ponc ii I cot ts’ -au ;n’ of the nets! t u pro~ ide

preliminary t raining j r - i I lie ( - c )iret - f way I ri-spend and t hi - limit of

four ex ami nec;; i ‘ st’in; ion . Tidmin i ;; t i- a t ion coo Id be made more fea~ i b le

by in cri -’ a sinq the tiumbe r c- it tel e v i ; ;ion flK~n i toi ~~; , but it W oulti ot i ll hi’

advisable to tia v i - one fin ;! adinin j o t rat or for each fout i-x.imiin ’c~

; hec5iii ,

~t

of the exam nt-c ;; ‘ iti fami 1 i at i Ly w i t h liii’ - t

~~;~ cotiis ’ method . (‘ fli lt it - t’tI to

the adm i n 1st rat ion I ime for hands—ot; test ing , ti owevet , I t-It v is ion I c:; t i ll ’~ i;; much leo;; cost ly.

Acc~~~~ance of Ti- l evis ion Te stin~j

The react jolt of the ex ami neei; to t he t- e levi ; ion t i-s t .-ippi’a red t ii he

quite favo r able. I ’o;;I exam i n a t j ot; inte l-view;; I udi cateci t hat most cx—

anti ii i’co did tt.i I ly ri’ (er rod the telev I ~

; ton I en t t-o the hands—on I cot

and all exam i nec;; though t the t c lev i s ton t cot was fa i r . Even when

querI ed about t he test ‘ s being used as a bas i

~; for promot ion or ext t-a

pay , I lit - ex.ini i iu’c’s ;; t i l l t houqht it w.ii; (a I r . Son~

! c’xaminees Itri ’ —

let ic I t he hati ds—on mode o t t cot i tig , m i t no one i u- c fe rred t lie ii ,it s,i —

.iti d—pen c ii fl~)dC

Some re.ii-;un;; mentioned tifl~ pr&’ ft . ri j u g t lie t e levi s ion mode fol low :

1 . Scorinq is f.t i rer and not ~lepeisIcnt upon the whims of the test

admiri I strator .

2. Test ing is f a st i ’r and not- so drawn out .

In t t ’Ie v i s ion t cot inq no one I shout i nq at you and order tug

you around .

Some of tin ’ reasons for preferring t h i ’ hands—on niodi’ follow :

1 . There is more t ime to t i - i ink and t o respond .

2 . Test inq In more spread out and doesn ’ t come so I ast

Tt’ levi;; Ion hurt s I h~’ cyi~i;

- 

1t ~ —

________ — ___________
-~~~~ ~~~~~~~~ ~ - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - - -— - - - -. - - -

4. There is a chance to walk around between items .

The examinees also indicated that television testing would be

better than paper-and-pencil testing because the questions would be more understandable and require much less reading.

Com~arison of Television, Paper-and- pencil, and Hands-On Tests

The comparison between the mean percent error made on the television

test and that made on the paper-and-pencil test is shown in Table 3.

The means for the television and paper-and-pencil tests do not differ

to any grea t degree , indicating that the difficulty levels of the two

tests are fairly equal .

Table 3

Mean Percent Error Made on the Television and Paper-and-Pencil Tests

Test and Days

time of day 1 2 3 4 ‘3 Mean

Television

Morning 19.63 15.00 14.75 1(- .46

Afternoon 27.88 20 .38 28.00 26. 09 .‘7.09 27.54

ttnweighted mean 22.00

Paper-and-pencil

Morning 28.38 20.75 22. ~ 24.23

- Afternoon 27.13 24.71 19.5 20.90 27.38 26.05

ttnweighted mean 25.14

One interesting facet of the data is that afternoon television

examinees made many more errors than the morninq qroups . These results

are convincing because they are consistent across the first 3 days of

the experiment and because the afte rnoon means for Days 4 and 5 are

consistent with the other afternoon means . There does no t appear to be

any morning-afternoon effect for the paper-and-pencil test.

The analysis of variance using the unwe ighted means analysis for

unequal cell frequencies (Winer , 1962) is shown in Table 4. This analysis shows no difference between the television test and the paper-andH 17

- —- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
____ - -~~~ 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~ - ~~~~~~~~~~~~~~~~~~~

pencil test in terms of item difficulty. There was a significant

mornin g—aft ernoon effect , but the mOre- meaningful result is the

si gnificant mean square (MS) interaction . Analysis of this MS i i it i -raction reveals that the morninq-afternoon effect is concentrated on

the- te le-.risiun test and not on the paper-and-pencil test.

In order t.o check on whether the-’ afternoo n examin~’es may have beet;

less qualified than the morning exami nces , the first round no—go ’s

from the h ands—on t -st were analyze.!. These resul t ;; are shown in Table

5. Inspection of the means indicates l i t t le difference between the

television and paper-and-pencil groups , or between the morning and

-; afternoon groups . If anything , the afternoon qroup pe rformed slightly

better thai-i the morning group. An analysis of variance of these

results showed no significant diffe rence for any of the variables.

Tablo 4

Analysis of Variance for Television and Paper-and—Pencil Tests

~oet r.’c d f MS F p

TV vs. P i. P (method ) 1. 9.50 1.04 ns

Morning vs. afternoon (session) 1 124.57 1LI.0 ..01

Method x session 1. 58.66 6.40 - .02

‘ Within cell 130 9.16

Al though overall scores on the television and paper-and-pencil

tests did not di f fe r , there might be differences amonq the various

items . Accordingly , the items were grouped by response type (multiple

choice , error detection , and motor manipulation) and Ioq linear

Chi-square tests (Shaffer. 197~~

) were computed for each item . Table t ’

shows that there was a wide variation of difficulty among the items

ranging from 10 to 81% error. For the multiple choice-items , there

was little d ifference between the television and paper-and-pencil

versions. Only 1 of 13 items showed a significant difference . For

the error—detection items there was t substantial differett ’e, with

six out of nine items showing a si gnifican t difference .

18
-- ‘“

~~~~~ 

-“

~~~~ ‘~~ 

- - 

~~ - -~~ -~~~r: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

- -

Table ‘3

Mean Pc t een t Error Made’ by the ‘It-i cv i si out aii~I

Pape r _and_ Petuic ii 1 t oups on t in- iL-ti;.i; ; —~ ‘it Tv~ t

Test and Days

time’ of day 1 2 - 

4 - 

‘3 Mea n

Television

Mor ning ‘).0() t . . H ) 0~ 52 ———— ———— 8.16

Afternoon 0 • 00 0 • 00 I. . 52 ~

. . ‘3D 2. ~~

ltnw ci .thte-d mean 7.34

Paper—and—pencil

Morning 12.0() 1.1) 0 ~ . 0() —-— —— ————

Afternoon 7.52 ‘1 .72 ‘ .52 12.00 D.’3’ 8.4 8

I t owe ighted -ine ’diut

It i; i n t e r est ing to u i ot t’ that errors ot commission are note d ii ti —

cult to dt- t ect on t i - l evis ion ; whereas errors of omission and h o — e n - o r

items are more di f f icul t o detect Ott p a p er — att d—p cnci 1. Th ree ’ of I itteight motor—n~

inipulation items show some significant difference , .- nid

all three of these i tems show more difficulty for the television test .

The net result of this item d i f f i c u lt y analysis shows five’ items more’

d ift i cult on television tests and five items more difficult on paper—

and—pe- ’n e i I tests. This canceling effect is reflected in the overall

nonsiqni Ii k -ant d iffe rt-in ’e be tween the te 1ev i 51cm t -st and the pa p i-l--

and—pc- nc i I test

L 

The’ la st anal ys is , in Table 7 , shows the corre 1.-itions of the hand s—

on t ’ -st with t i-it - ’ paper—and—penci l test and the television test . Those

cor rel a t jout s ar , -’ also broken down for the morning and afternoon qroups .

There is a low positive correlation between the television and hands—ott

ti’ ot ii; I al so between the paper—and—penci l and hands —on t cots.

t — a si — p t -ti c t 1 cur re ’ lat ion is significantly di f f t ’ rent from ~e ro

however , t her t - s no s iqni ficant di fference between f lit’ t i -it-visio n

versus t it- hand~ —on and the paper-and—pencil versus hands—on

- - - -t re [at tons. Tu e hre ’ak dowtt for morn inq and afternoon groups sh ow;; t

-omewhat hi qile -r t’t)s it i v e~ c -i -ru- l at i on for t lie afternoon qroup and V*’I y

ii t le ~

- r p e-I.i t ion for the mu i n i no group. Once’ aq.i in , there i 5 no

-; igni fican t .1; t fert-nce let ween the television and paper—an d—pt-n c i i

correlation;; with the hands-on test .

* 

i-- i 

I 

~ 

i __ ____________________________
- i ~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~ ~~~~~~ — -

~~~~---_ ~~~~~~~~~~

Tab le (-‘

Comparison Retween Television and Paper-and-Pencil

Items Percent Error Arranged by Response Typo s

Deo[;otIse t YP’ and % error --‘

-

~ item number TV P I. P X ’

Multiple ci icv

1 1 0 ns

2 0 2 ns

4 41 3 1 1.43

5 21 14 .82

6 38 72 11.83*

7 47 57 ns

8 26 20 ns

‘1 16 12 ns

10 17 16 ns

ii 3 5 us

12 0 0 n o

17 1 3 ns

12 81 77 ns

Error detection

(commission)

3 51 28 7~ 47*

ii 11 11 no

14 40 12 1

~ .0’~ 37 46 56 1.42

Error detection

ij (omission)

16 19 52 14.88*

18 4 47 18. 00*

23 Il 4’3 11.26*

Error de tection ‘ (no error)

15 16 19 ns

-- 20 17 66 30.36*

Motor manipulation

(re t icles)

19 46 16 1. .24 *

21 51 17 16. 17*

24 4 5 ns

25 27 11

26 11 2 ns

27 14 i i ns

28 71 34 1.45

• 30 ____ 21 28 us

*p <. 01.

**p ‘..1O.

2(1
- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -

~~~~~~~~~~~~~~ __________ - ______

Tab1t~ 7

Correlations Between the Television and Paper-and-Pencil

Scores and the Hands-On Performance ~cor c- - -

Time of day

and tests correlated r p

Overall

Television vs. hands-on .24 ns difference is nonsiqnificant Paper-and-pencil vs. hands-on .33 c.01

- - Mo rnin-

~

Television vs. hands—on - .1)° ~~ difference is nonsignificant Paper—and-pencil vs. hands-on .16 no

Afternoon

-

i 

Television vs. hands—on .47 . .01 differe’nce ’ is nonsi- ’;t i fican t

Paper-and-pencil vs. hands-on .40 < .01

CONCLUSIONS AND DISCUSSION

The results from this research indicate that it is possible to produce

a synthetic test using television as the stimulus input . The e-xamiute t ’r -

can understand the problems , make prope r responses, and accept the test

as “fair ” for career evaluation .

The experience gleaned from the production and administration of thi s

prototype test indicates that television testing is mor e’ costly than

paper—and—pencil testing but far less costly than hands—on testing . The

production of the tape , from conception to final editing , required several

months and used the services of a substantial number of professional

people. Television tests are also somewhat inflexible , not only in the

difficulty in effecting changes in the test, but also in the t iming

decisions——the amounts of time to allot for posing each question and for

each response--that have to be made before the production of the test .

Television testing will sive a much more promising future if a

presen tation and response device can be designed which will permit the

exaviinee to advance to the next item as soon as the present one is

aniwered , to see t h e - ’ same i tem twice - , to change answers to a -it item , and

to review the entire test. Such a capability would permit the flexibility

of presenting multipart items , such as in t roubleshooting and would permit the presentation of multimedia items, such as using both television

and technical manuals in the same item.
- ___ , 

- - __________ - — -~~~~~~~ - — -- - - 

- 

- 

- 

- - 

- - ~~~~~~ -~~~~~~~~~~~~~~~~~~~~~~ --

~~ ~-~~~---~~~~~~~

The present experiment provides evidence that television testing

is highly acceptable to the examinees. Their predominant attitude was

that the test was little different from the hands—on tests in the Advanced Individual Training course , except that television was quicker

and less subject to scoring error . All of the scenes were quite familiar to the examinees, and the items were ones that they had been studying for 8-13 weeks.

Television used in the multiple—choice format appears to offer no

advantage over slide or paper-and-pencil formats. Before the experiment , it was felt that television would offer an advantage for those

items in which motion was an integral part of the stimulus . For exampie , Spangenburg (1973) has shown that watching a television display of

a procedure involving motion leads to more learning than watching a - 

- 

sequence of still shots. However , this advantage of motion proved to

be true for one motion item in the present research (item 6 , Table 6)

but not true for two items (5 and 7). Perhaps if more motion-type

items had been included in the multiple-choice category an advantage

might have been shown .

In the error-detection category there did appear to be a clear-cut

difference between television and paper-and-pencil items. Here the

fidelity of the stimulus did seem to play a role , and the en riched

stimulus of the teler .iion picture may have presented cues to the ex—

aminees. The two error-detection items that proved to be more difficult for television examinees (items 3 and 14 , Tables 1 and 6) we re

two of the first error—detect ion items to be presented. Since error

detection was an unfamiliar response for the examinees , this unfamiliari ty may have caused some diff icul ty . Th is same phenomenon can be

se€-rt in the motor-manipulation items which involve an even more unfamiliar response . Here the television exantinees had more difficulty

with the first few items than did the paper—and—pencil exantinees.

The correlations between the synthetic and the hands-on tests are

too low to warrant recommending the substitution of synthetic for handson tests. However , the correlat ions for the afternoon groups art’ high

enough to encourage further research . The hands-on criterion test used

in the present experimen t was somewhat unsatisfactory because of the

large number of perfect scores.

The drop in the scores on the television test for the-’ afternoon

group as compared to the morning group was intere -ting hut unexpected.

One possible explanation for the drop may be that the exantinees were

required to stare continuously at a fairly large television screen from

a very close distance for approximately 1 hour . A human being may be I -

able to tolerate this strain in the morning but by the afternoon accumulated fatigue plus a heavy Army lunch may have combined with the strain

• to produce a letdown .

22

A -

~~ -~~~ 

— ________
____________ ____ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The results of this study favored the feasibility of television

testing . The tests can be produced at a reasonable cost , can be

administered in a reasonable manner , are understandable by the examinees ,

and have high acceptability with the exaxninees.

The validity of the results was inconclusive . The criterion scores

for the hands-on test were unsatisfactory in that most hands-on examinees

made a perfect score . The correlation between the television and hands-on

tests was low positive but nonsignificant. Comparison between parallel

television and paper-and-pencil tests also showed no difference on an

overall basis , although there were significant differences between

many items. -

Evidence from thi s study is insufficien t to conclude tha t syn the tic

performance tests with television inputs can be used to replace hands-on

performance tests.

23

- — - -

~~~~~~~ -~~~~- - -~~ -—-~~~~~
-~~~~~~~~~~~~~~~~~~~~~~~ ---- --.-‘~~~~~ 

~~

- --— -

~~~~~~~J:~~~-- - . - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ r”~

REFERENCES

Cockrell , J. T. Evaluation of Minicourse 13: Role Playing in the Classroom. Far West Laboratory for Educational Research and Development,

1974.

F 

Department of the Army . Report of the Board of Inquiry of Army Logistics

System (Brown Board). Washington , D.C.: Author, May 1966.

Engel, J. D. An Approach to Standardizing Human Performance Assessment

(HumRRO Prof . Paper 26—70). Alexandria , Va.: Human Resources Research Organization , October 1970.

Enge l, J. D. Development of a Work Sample Criterion for General Vehicle

Mechanic (HumRRO Tech . Rep. 70-li). Alexandria , Va. : Human Resources Research Organization , July 1970.

Engel , J. D., & Rehder , R. J. A Comparison of Correlated .Job and Work

Sample Measures for General Vehicle Repairmen (HumRRO Tech . Rp.

70— 16) . Alexandria , Va.: Human Resources Research Organization ,

October 1970.

Hays , J., & Pulliam, R. Development and Evaluation of Video Systems for

Performance Testing and Student Monitoring (AFHRL TR—74—67). Brooks

Air Force Base , Tex. : Air Force Human Resources Laboratory , July

1974 .

Occhialini , N., Jr. Research Study : The Use of Qualification Tests as

Performance Tests for Snlisted Combat MOS Evaluation (Research Study

No. 162). Fort Benjamin Harrison , Indianapolis , md. : U.S. Army

Enlisted Evaluation Center , June 1972.

Osborn , W . C. An Approach to the Development of Synthetic Performance

Te~ ts for Enlisted Combat MOS Evaluation . (Paper for 12th Annual

Military Testing Association Conference , French Lick , m d . HumRRO

Prof . Paper 30-70). Alexandria , Va.: Human Resources Research

Organization , December 1970.

Osborn , W. C., & Ford , .1. P. Research on Methods of Synthetic Performance

Testing (HumRRO—FR-CD(L)-7 6-l). Alexandria , Va.: Human Resources

Research Organization , April 1976.

Shaffer , .1. P. Defining arid Testing Hypotheses in Multidimensional

Contingency Tables. Psychological Bulletin, 1972, 79, 127-141.

Sh irkey , E. C. Preliminary Validity Report of the MOS Evaluation Test

for Medical Specialist (Research Study No. 31). Fort Benjamin

Harrison , Indianapolis , md. : U.S. Army Enlisted Evaluation Center,

1965.

- 

-- - ~~~~~—

25 f4 wirn p

~~~~~~ig

A
Shriver , E L , Hayes , J F , & Hufhand , W R Evaluation Maintenance

Performance : A Video Approach to Syntbolic Testing of Electronics - Maintenance Tasks (AFHRL TR-74-57 (IV)). Brooks Air Force Base, - 

Tex.: Air Force Human Resources Laboratory , July 1974.

Spangenburg, R. W. The Motion Variable in Procedural Learning. AV

- 

Coninunications Review, Winter 1973, 21, No. 4.

Urry, V. W., Shirkey, E. C., & Nicewander, W. A. A Preliminary Validity

Report of the MOS Evaluation Test for Personnel Specialist MOS Code

716.1 (Research Study No. 21). Fort Benjamin Harrison , Indianapolis ,

- md. : U.S. Army Enlisted Evaluation Center , 1965.

Winer , B. J. Statistical Principles in Experimental Design. New York :

F McGraw—Hill, 1962.

~~

- 

26
- 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

-‘

~~~~~‘~~~ ~~~~~~r - --‘- -:~~ 

- - -

APPENDI X A

ITMI T E I ) ~UIDELINFS FOR THE DEVE LOPMEN T Oh ’ TELEVISION TESTS

Very little evidence is available as to the best way to present

test items on television . The only published research for military

tes ts is Shriver ’s (Shriver et al ., 1974) and as noted before , the

conclusions from this research were very neqative . Most decisions

made for the present test were based on paper-and-~encil-test ,

development , expert opinion, and experience . Very few hard-and-fast

guidelines can he offered because so many decisions depend upon the

f orma t chosen , the type of questions , and the amount ot ti~

me available.

An important limitinq factor in the develo~~ ent of television

tests is the amount of running time available for each rest item and

for the ca~ip1eto t t ’st. The maximum desirable time for a t e l evision test

such as the present one is ~O—e ~0 minutes for a number of reasons ,

including eyestrain , general fatigue , and administrative cost. One

advantage of television testinq over hands-on testing is the low administrat ive cost per examinee . The lonqer the television test , the loss

the advantage .

Although experience with television testing is too limited to offe r

much in the way of quidelines , it may he useful to describe the developmental st aqes and some of the difficulties encountered.

Prior to the development of the test it was decided to aim for a

50-t--d minute running t ime , to cover the MOS of tank crewmen at skill

leve ls 1, .

~~ , and 3 and the lob positions of driver , loader , ‘-iun ner , and

tank conunander. The test was to be a group test with individual TV

screens and the examinees were to respond to the i tems by touchino the

face of the television screen with a stylus or reticle.

K The first selection step was to ask various military training

departments (gunnery , automotive , and such) to submit a list ot~ cr it ical

tasks which should be tested . These departments submitted a total of 75

tasks. Because only a limited number of tasks could be used on the final

tape , the list had to be pared down considerably . Many tasks were eliminated in order to l-~

alance the test among skill levels and crew t—oritions.

For example, 40 of the tasks rececved from the departments were for skill

level 3 and only 5 of these tasks were on the final tape . Most remaining

excess tasks were eliminated simply by deciding to limit the test to tasks

associated with the actual operation of the tank . Critical areas such as

drug abuse, first aid , leadershi p, and tactical decisions , and complex

tasks , such as sketching an area map and tasks that required excessively

long television running times , were eliminated .

The fir st step in developing specific test items for each task was

to list all response components making up a task and decide whether each

response component was primarily cognitive , pe”ceptual, or motor. Each

coqn itive or perceptual re ise comjonent was then examined for criticality
w—. - 

~~~

_—

r - - - - - -.: :~~~~~

and f e a s i b i l it y for television testin q . Dependinq upon the number of

critical and feasible response components making up a task , a decision

was made to includ e one or more test items for the task (the number of

test items per task ranged from one to six as shown in Tdble 1). Primarily

cognitive response components were tested as error—dete tion and multi plechoice items . Primarily perceptual components were tested as motormanipulation and multi ple—choice items .

Some of the trial-and—error observations that can be made for each

response type are as toilows :

1. Multiple choice. Items of this sort are very simple to conceive and

develop and require very little test time (about 30—45 seconds) provided

choices are presented simu ltaneously such as on a four—way split screen

or four words on the screen. Presenting the choices serially creates

difficulties not only in terms of greatly increased running time but also

- 

- because the examinees often forget the first choice by the time they see

the last one. Either the choices have to be presented twice (responding

occur s on the second presentation ) or the examinees must respond “yes ” or

‘no” to each choice as it appears. Neither method is very satisfactory.

Rationally, presenting multiple—choice items on television does not

offe r too much advantage ove r a paper—and—pencil format except in terms

of reducing the need for reading and perhaps presentinq a more easily

understood item. For example , the motion and sounds associated with

television may be helpful in understanding the item .

2. Error detection. This response type has been criticized harshly by

Shriver (See Page 7), and there are other difficulties as well. One

ma -jor ditticulty is in producing the item (televising the procedure

accurately). If a no-error item is desired , it is necessary to find an

actor who can carry out the procedure without error. All too often ,

expert advisers cannot agree on the correct procedure . Many repetitions

of each scene have to be made before the experts and actors can reach

some sort of compromise , and even then there remain logical and inheren t

difficulties which cannot be resolved . For example , in televising items

for the load round into main gun task it was necessary to choose between

showing the action at normal speed or in slow motion . When the action

was shown at normal speed , no examinee could discriminate the crucial

element (hand position) and the item had no meaning. When the action was

shown in slow motion (so the crucial element could be seen) , examinees

criticized the slowness itself as an error.

Another major difficulty with error-detection items concerns very

slight devia tions from prescribed procedure which often escape the

scrutiny of expert advisers. Exceptionally well-skilled examinees may be

lured into pointing to the slight deviations as errors, while the less

• skilled neve r no tice the slight deviations and point to the ma jor intended

error . This was particularly true for the response format used in the

present. study , where the first response made by the examinee was scored

and all subsequent responses to the same item were ignored.

28
— ~~—~~~~~~~~~~~~~~~~ -—-~~ - - ~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~ ‘

~

—~~~~

--- - -

~~~~--— - ----P-, ~~~~ ~~~~ ~~~~~~

-

~~~~~~~~~

--- 

- - - - - -- - - - 

_ -

The error—detection response type was included to test the examinees ’

know ledue of incorrect as well as correct act ions . Many i n - ’orreI ~ actions

occur very infrequently but can he very serious when they do occur .

Per formino the correct action in a hands—on test does not necessari lv

indicate awareness of danger points , and examin~-es need t c be tested

directly as to awareness of incorrect actions . However , as Shriver

points out, watching someone else perform is very di ff e rent from d o ini

it yourself. Failing to notice an error may indicate lack of kiiowledie or

it may indicate inability to notice error in others.

The overall conclusion is that error detection is a doubtful response

type and more thought and research are needed prior to its acceptance as

a useful procedure .

3. Motor manipulation. This response type was rather specific to this

particular study and the response equipment being evaluated . In fact , one

strong selling point of the response equipment was its provision for

testing the motor—manipulation items . All test i tems under this particular

- - response type pertained to where the examinee should place the reticle on

the television screen when simulating firing the main gun under various

conditions . However , analysis reveals that this response type is not

really a test of motor ability , hut rather a test. of a combination of

perceptua l and cognitive abilities. The cognitive element was knowinq

the correct lead and elevation for each target and the perceptual element

was being able to discriminate the correct lead and elevation . There is

no evidence to i ndicate that the ability to mani~mlate a plastic r ot i cle

on a television screen has any correlation with the motor element i nvolved

in aiming an actual gun . On the plus side , this response t ype is more of

a recall item than a mul tiple—choice question and therefore should provide

a more exact measure of recall. On the minus side is the requirement

to learn a new response quickly (mani pulating plastic reticles). Incorrect

responses may be caused by lack of knowledge or perceptual ability, or

merely by failure to master the new response of manipulatin g plastic

reticles.

‘ This response type , like the error-detection response type , needs

much more thought and research prior to acceptance as a useful procedure .

Some commen ts on a few miscellaneous topics may also be useful:

1. Use of a time period to indicate error. One item (3, Tables 1

and 6) on the television test used the passage of time as the cue for

the examinec to note an err ’r. That is , the actor waited only 5 seconds

be fore con tinuing a procedure , where the prescribed procedure in the

tech nical manual ca l ls for a i.~

O-1S0-second wait. Some criticism has

indicated tha t this t ime—passage technique mi ght confuse ex aminees

because Americans have been conditioned through exposure to moti ei.

p ictures and televi sion to accept ~~y length time period shown on a

screen as the appropriate time . Item 3 did seem to confuse many examinees .

no

I
-‘ 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

~~~~~~~~~

2. Long items. Several items on the television tape had relatively

long running times (approximately 3 minutes) . Some critics claim that

including such long items may be unwise because coverage cf the total

subject ma tter is restricted at best , given a 50—60 minute time limit

for the test. Although a long item may not necessarily confuse the

examinees , it is noteworthy that 3 out of 4 long items retained in the

test did prove very difficult for the television exaininees and all 4

of the long items omitted from the study appeared to be confusing

during p ilot runs. Another reason for omitting long items is the

diff icul ty in getting an actor to perform a long sequence in letter—

perfect fashion . One very long item on the tape (evacuate injured

crewman) was never completely satisfactory . The final take was

accepted because the director became concerned with the safety of the

actor play ing the role of injured crewman.

3. Resolution. Unlike the human eye , television cannot capture

both a wide—angle view and good resolution at the same time . For

scenes that require good resolution it is a good idea to zoom in on a

scene and remain there . To attempt to show more than one closeup in

any one sequence tends to confuse the viewer.

4. Restricted view. Even with a wide-angle lens , television qives

a very restricted view and care must be taken to provide setting shots .

Precise judgme nts as to the placemen t of controls are di fficult to

make from a televi sion picture .

5. Poor depth perception?. Much depth perception is lost in a

television picture. Items that depend upon judgment of depth sh1-ul d

be omitted .

6. Closeup and motion. Any kind of motion in a closeup shot is

confusing. Necessary movements must be very slow and precise. However ,

it should be noted that slowness is perceived as an error b’~ many peoi le .

More research needs to be accumulated before a more precise set of

guidelines can be produced for television testing. Particularly needed

is development of a more adequate stimulus presentation and r~ sponserecording device . Also needed are researchers well grounded ~~i. the

capabilities and limitations of television and the use of television

cameras , lighting , and editing equipment. Television testinci offers

much potential but before this potential can reached , much preparatory

work remains to be done.

30 

-

~~~ 1
~ 

~~

- -- -

~~~~~~

.

~~~

—--

~~ 

—.--- - - ~~~~~~~~~~~~~~~~~~ ~~~ ~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~ ~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~

APPENDIX B

EXAMPLES OF PAPER-AND-PENCIL ITEMS

A. Multiple choice.

12. You are the driver of an M6OP.l tank . What response do you make to

the following ground guide signal given at ni ght with a flashlight?

-

~~~ a. Move backward . -

__________________ -

b. Start engine .

c. Stop engine .

d. Turn left.

B. Error Detection

6. You are the loader in an M6OA1 tank . The tank commander gives the

following fire command :

“GUNNER , BEEHIVE TIME , TROOPS , ONE SIX HUNDRED”

The firing switch has been checked and the breech is open . You do the

six steps in order:

- , 1 (1) Select a BEEHIVE round .

(2) Insert the round 2/3 of the way into the chamber.

(3) Push the round into the chamber with the heel of the right hand.

(4) Clear the path of the recoil.

(5) Turn the firing switch to FIRE .

(6) Announce “UP. ”

Did you do any thing wrong?

a. Step (2) is wrong.

b. Step (3) is wrong .

c. A step is missing.

d. All of the steps are correct.

31 H

I 
- .

~~-‘ ~~~=-,-

~ —

~~~~ ---—-~~~~~--I

. ______

- - — — ~. — - - — ~

. 

~—~~~~ -.-~~ —-r~ f~

_ -

~~z~~’~ 

~~~~~~~~~~~~~~~~~~ -

-ii

- ~~ C. ~

totor Man ipu iiation (Reticle Mani~~~~±~ ion)

- 

I - 

FOR THE NEXT FOUR PROBLEMS ENGAGE ALL STATIONARY TARGETS AT CENTER OF MASS

AN t) ASStIMP ALt . MOVING TA RGETS ARE TRAVELING AT 15 MPH .

if 2 1 . You are the gunner on an M6OAI tank . The tank commander gives the fire

command

— “GUNNER , SABOT , TAN K ”

Which of the following sight pictures would you take up usinq the periscope

- 

- 

retic1e~

a. b.

-

I I

d

~

If 

C 

-

~~~~~ 

- I

- - -

- - --- 

I -
_ _ _ ‘ -

- P

APPENDIX C

EVALUATION OF THE TELF.STRATOR EQUIPMEN~1

A

One of t he reasons the tclevision test was designed and produced

was to evaluate the Telestrator equipment (also known as the Telestar

equi pment). The nove l component of thi e equipment is an electronic

tablet which can be fitted over the face of a television screen. The

tablet will record the’ horizontal and vertical (XY) location when i t

is touched with an electronic contact point embedded in a stylus or

similar indicator (such as a qunsiqht reticle) . By the prope r use of

a u x i l iar y recording equ i pment it is possible to record the place’ and

time the screen is touched. The recording equ ipment includes a counter

which keeps a runn i ng total of the number of items , n umber ot answers

attempted , and number of cot-rect answer s

The compi e’te system includes the elect ronic tablet , a programing

unit , and several student units. The programming unit is used by t h e test

develope r to place electronically on the television t ape the XY

- coordinates for the correct answer for each test item and the t ime

period during which the equipment will accept this answer. The student

unit compares electronically the programed answer ai.d the examine e ’s

answe r and records the result

The student u n it provides three types of feedback to the examinee

for each test item. Immediate feedback is provided by a high-pitched

tone and a small red light that come s on for a correct answei versus

a low—pitched tone and no liqht for an incorrect answer. ~liqhtly

delayed feedback comes from a coun ter whi ch shows new totals of i tems

and correct answer:; at the end of the programed time period for

answering each problem .

A s to whe ther the Telestra tor equi pment has any merit or not , i t

is necessary to examine both :

1. The equi pment itself , as designed and produced , and

2. The overall testing strategy which includes (a) Individua l

responding , (b) Television stimulus, (c) Immediate f eedback , - I and Cd) Time limit on each response.

A . The Equipment

As with most newly designed equipment , the Telestrator contained

many bugs and never worked properly. However , it was possible to test

some aspects of the equipment by using htunat~ qt - aelt ’rs to record right

or wrong answers by the examinees according to the time and place ’ the

screen was touched . Several pilot tests were run with t h e ’ following

results.

4This summary of Telestrator operation was submitted previously to the

Training Support Division , TRA DOC .
-.1-_ I — -‘ 

~~ -:~ ‘

~

-

~~~

-

~

‘- —---~~~~~~ - — -

~~ —,.— ~~~~

U) Accuracy . There is a fundamental flaw in the Telestrator

design insofar as precise responding is concerned. The equi pment was

cl aimed to be accurate to 1/4 inch. However , due to parallax the

actual accuracy was more on the order of 1 or 2 inches. This grossness effectively eliminated the use of the reticle test items because

w ith any reasonable size reticle no discrimination was possible for

l eads or ranges. The grossness also eliminated many test items in

which the exaininee was required to discriminate among several tank

controls. The spacing between these controls as shown on the screen

wa s not gr~~ t enough to permit exact programing of the answers , and

one answer box would overlap another. The parallax results from

mountin a the electronic table t at some distance from the actual

television screen (due to curvature of the television screen the parallax

increases as one approaches the edge of the screen).

In order to continue the exper imen t and test the idea of respondi ng

to television , the electronic tablets were removed from the television

monitors and the examinees were required to touch the face of the

actual television screens . This completely eliminated all parallax and

permitted the use of reticic items and other precision responding.

(2) Video presentation . The electronic tablet is constructed in

such a manner that it blocks a 1-inch-wide area around the outer edqe of

the television screen. This is a serious limitation because it is

necessary to use a small-size monitor for such closeup work and this

outer 1 inch covers a substantial part of the available screen area .

B. Test ing~Strategy

Because of the device ’s failure to work properly and the poor design

of the electronic tablet , it was not possible to evaluate the testinq

strategy completely. However, by eliminating the parallax (removing

the electronic tablets) and using human graders to record responses ,

- ‘ it was possible to make a limited test of the strategy .

C l ) Responding to television . The examinees seemed to have a little

trouble understanding the test items , and responded very precisely. - I Three types of test items were used; Multiple choice , Error detection ,

• and Reticle manipulation .

No training other than instructions was required for learninq to

respond to the multiple-choice and error-detection items . Approximately

10 minutes were requ i ted for training on the reticle-manipulation items .

(2) Time to respond. Ten seconds were allowed for responding to

each item . The time limit was generous and most examinees responded to

most items in less than 5 seconds .

(3) Perception of test. The examinees perceived the tent as being

“fair ” and most actually preferred the television test over the hands—on

test.

34
_____________ - __________-~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

(4) Comparison with other tests. The television test was compared

to parallel paper-and-pencil and hands-on tests.

(a) Paper-and-penc il test. Overall there was little difference

between the mean scores on the television and the paper-and-pencil tests.

However , on an item-by-item basis there was considerable difference for

some items. On error detection , the television scores were much better ;

on reticle man ipula tion , the television scores were worse.

(b) Hands-on test. There was a low positive correlation between

the television test and the hands—on test . This correlation was less than

desirable.

The experiment was too limi ted to permit any conclusions at thi s time

wi th reference to the reliability and validity of the above results.

- -

- (5) Feedback . Because the examinees were tested four at a time and

because the Telestra tor equipment wa s not workin g , it was not possible

to provide immediate feedback after each item .

(6) Eye fatigue . The television test and the responding mode

required the examinees to stare continually at the television monitor .

There were many complai nts of eyestrain and there is some evidence that

the afternoon television examinees performed more poorly than the morning

television examinees.

Conclusions and Recommendations

(1) The Telestrator equipment as presently designed should he’ rt — i e ’

;-t ed

because of the parallax problem.

(2) The television method appears to offe r enough promise to warrant

the testing of other re sponse devices which do not have t lie’ p a ral 1 ix

problem .

(3) There are many unknowns in television testing and t he’ overall

testing strategy , and the research effort need’; to he ’ ~~~~~~~

such as :

(a) A more definitive comparison with hands-on t e st-; .

(b) Research on the “immedia te feedback ” idea .

Cc) Using alternative response device’s.

(d) Comparison with slide-tape devices.

(e) Further research on eye fatigue .

3’,
~~

‘ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~ —-~~~~~~~ -.

DISTRIBUT ION

ARI Distribution List

4 OASD (M&RA ) 2 HQUSACDEC , Ft Ord, ATFN: Library

2 HODA (DAMI-CSZI 1 HOUSACOEC, Ft Ord. ATTN: ATEC—EX --E—Hum Fac tors

1 HODA (DAPE PBR 2 USAEEC . Ft Benjamin Harrison. ATTN: Li brary - - 1 HODA (DAMA -AR I 1 USAPACDC , Ft Benjamin Harrison , ATTN: ATCP—HR

1 HODA (DAPE-HRE .PO) I (iSA Comm-Elect $ch, Ft Monmoutis, ATTN: ATSN—EA

1 HODA (SGRD-ID) 1 USAEC. Ft Monmouth , ATrN: AMSEL ,—CT—HDP

* 

1 HQDA (DAP ,iI-DOT-C) 1 USAEC. Ft Monmou th , ATIN: AMSEL—PA --P

1 HODA (DAPC-PMZ.A) I USAEC, Ft Monmouth. ATTN : AMSEL—S l ---CB

1 HQDA (DACH-PPZ-A) 1 USAEC. Ft Monmouth. ATTN: C, Fad Des Br

1 HQDA (DAPE-HRE) 1 USA Materials Sys Anal Agcy , Aberdeen , ATTN: AMXSY—P

HQDA (DAPE-MPO-C I 1 Edgewood Arsenal , Aberdeen , ATTN : SA REA— BL—H

1 HQDA (DAPE-DW) 1 USA Ord Ctr & Sch, Aberdeen, ATFN~ AT SL— TEM—C

1 HQDA (DAPE -HRL I 2 USA Hum Engr Lab. Aberdeen, ATTN: Library/Dir

1 HQDA (DAPE-CPSI 1 USA Combat Arms Tng Bd. Ft Banning, ATTN: Ad Supervisor - • 1 HODA (DAFD -MFA ) 1 USA Infantry Hum Rich unit , Ft Benning, ATTN: Chief

I HODA IDARD -ARS .P) 1 USA Infantry Bd, Ft Banning, ATTN: STEBC— TE— T

1 HODA IDAPC-PAS-Al 1 USASMA . Ft Bl iss, AUN: ATSS—LRC

1 HODA (DUSA-OR) 1 USA Air Oaf $ch. Ft Bliss , ATTN: ATSA—CTD--ME

1 HODA (DAMO-ROR ) 1 USA Air Def Sch, Ft Bliss. ATTN: Tech Lib

1 HODA (DASG ) 1 USA Air Oaf Bd , Ft Bl iss . ATIN: FILES

1 HQOA (DAb -Ph 1 USA Air Oaf Bd , Ft Bliss , ATTN : STEBO—PO

1 Chief , Consult Div (DA-OTSG), Ade lphi . MD 1 USA Cmd & General Stf College, Ft Leavenworth, ATTN: Lib

• 1 MiI Asst. Hum Res , ODDR&E , OAD (E &LS) 1 USA Cmd & General Stt College , Ft Leavenworth , ATTN: ATSW—SE— L

• 1 HO USARAL , APO Seattle , ATTN: ARA GP-A 1 USA Cmd & General Stf College , Ft Leavenworth , ATTN: Ed Advisor

I HO First Army , ATTN: AF IcA-OI-TI 1 USA Combined Arms Cmbt Des Act . Ft Leavenworth . ATTN: DepCdr

2 HO Fifth Army. Ft Sam Houston 1 USA Combined Arms Cmbt Des Act , Ft Leavenworth , ATTN: CCS

1 Dir , Army Sd Studies Ofc . ATTN: OAVCSA (DSP) 1 USA Combined Arms Cmbt Dev Act, Ft Leavenworth, ATFN: ATCASA

1 Of C Chief of Stf , Studies Ofc 1 USA Combined Arms Cmbt Day Act , Ft Leavenworth , ATrN: ATCACO—E

1 DCSPER , A11’N: CPS/OCP 1 USA Combined Arms Cmbt Des Act , Ft Leavenworth , ATTN: ATCACC—CI

1 The Army Lib , Pentagon , ATTN: RSB Chief 1 USAECOM. Night Vision Lab , Ft Belv oir • ATTN : AMSEL—N V—SD

1 The Army Lib . Pentagon, ATTN: ANRAL 3 USA Computer Sys Cmd , Ft Belvoir , A11N: Tech Library

1 Ofc , Asst Sect of the Army (R&D) 1 USAMERDC , Ft Belvoir , ATTN: STSFB—DQ

I Tech Support Ofc. OJCS 1 USA Eng Sch, Ft Belvoir , ATTN: Libra ry

1 USASA. Arlington, ATTN: IARD-T 1 USA Topographic Lab, Ft Balvoir , ATTN: ETL—TD—S

1 USA Rich Ofc . Durham , ATTN: Life Sciences Dir 1 USA Topographic Lab , Ft Belvoir , ATTN: STINFO Center

2 USARIEM , Natick . ATIN: SGRD-UE-CA I USA Topographic Lab , FtBelvoir . ATTN: ETL—G SL

I USATTC. Ft Clayton, ATTN: STETC-MO-A I USA Intelligence Ctr & Sch. Ft Huachuca ATFN: CTD—MS

1 USAIMA , Ft Bragg, ATTN: ATSU-CTD-OM 1 USA Intelligence Ctr & Sch, Ft Huachuca , ATTN: ATS—CTD—MS

1 USAIMA. Ft Bragg, ATFN: Marquat Lib 1 USA Intelligence Ctr & Sch , Ft Huachuca , ATTN: ATSI—TE

1 US WAC Ctr & Sch. Ft McClellan , ATTN: Lib I USA Intelligence Ctr & Sch, Ft Huachuca, ATTN: ATSI—TEX--GS

1 US WAC Ctr & Sch, Ft McClellan , ATrN: Tng Dir 1 USA Intelligence Ctr & Sch , Ft Huachuca. ATTN: ATSI—CTS—O R

1 USA Quartermaster Sch, Ft Lee . ATTN: ATSM-TE 1 USA Intelligence Ctr & Sch , Ft Huachuca , ATFN: ATSI—CTD—OT

1 Intelligence Materia l 0ev Ofc, EWL , Ft Holabird 1 USA Intellig ence Ctr & Sch. Ft Huachuc a, ATTN: ATSI—CTD—CS

1 USA SE Signal Sch, Ft Gordon, ATTN: ATSO-EA 1 USA Intelligence Ctr & Sch , Ft Huachuca , ATFN: OAS/SRD

1 USA Chaplain Ctr & Sd,. Ft Hamilton , ATTN: ATSC’TE-RD 1 USA Intelligence Ctr & Sch , Ft Huachuca , ATTN: ATSI—TEM

1 USATSCH. Ft Eustis , ATTN: Educ Advisor 1 USA Intelligence Ctr & Sch, Ft Huachuca, ATTN: Libra ry

1 USA War College , Carlisle Barracks . ATTN: Lib 1 CDR . HO Ft Huachuca , ATTN: Tech Ref Div

2 WRAI R , Neuropsych iatry Div 2 CDR . USA Electronic Prvg Grd , ATTh: STEEP—MT—S

1 DLI , SOA . Monterey 1 CDR , Project MASSTER , A rTN: Tech Info Center

1 USA Concept Anal Agcy , Bethesda , ATTN: MOCA-WGC 1 Hg MASSTER , USATRADOC . LNO

1 USA Concept Anal Agcy . Bethesda , ATTN: MOCA-MR I Research Institute , H O MASSTER , Ft Hood

I USA Concept Anal Agcy, Bethesda , ATTN : MOCA-JF I USA Recruiting Cmd, Ft Sherdian , ATIN: USARCPM—P

1 USA Art ic Test Ctr . APO Seattle . ATTN: STEAC-MO-ASL I Senior Army Ads., USAFAGOD/TAC , Elgi n AF Aus FId No. 9

1 USA Art ic Test Ctr , APO Seattle , ATTN: AMSTE-PL-T5 1 HO USARPAC . DCSPER , APO SF 96558, ATTN: GPPE—SE

1 USA Armament Cmd. Redstone Arsenal . ATTN : ATSK-TEM I Stimson Lib , Academy of Health Sci ences , Ft Sam Houston

1 USA Armament Crnd. Rock Island , ATTN: AMSAR -TDC 1 Marine Corps Inst., ATTN: Dean—MC I

1 FAA-NA FEC , At lantic City, ATTN: Libra ry 1 HQUSMC, Commandant , ATTN: Code MTMT 51

1 FAA-NAFEC , Atlantic City . ATm: Hum En~ Br I HQUSMC, Commandant , ATTN: Code MPI—20

1 FAA Aeronautical Ctr , Oklahoma City, ATTN: AAC-441) 2 USCG Academy, New London , ATTN: Admi ss ion

2 USA FId Arty Sch , Ft Sill , ATTN: Libra ry 2 USCG Academy, New London , ATTN: Library

1 USA Armor Sth , Ft Knox , ATTN: Library I USCG Traini ng Ctr , NY , ATTN: CO

1 USA Armor Sch, Ft Knox , ATTN: ATSB .DI-E 1 USCG Training Ctr , NY , ATTN: Educ Svc Ofc

1 USA Armor Sch , Ft Knox , ATTN: ATSB-DT-TP 1 USCG, Psycho l Res Br , DC, ATTN: GP 1/62

1 USA Armor Sch. Ft Knox , ATTN: ATSB-CD-AD I HO Mid—Rang e Br , MC Oat, Quantico , ATTN: P&S Div

37

,, ~ _L

~~

u_Jj_j —. .

~~~~ - 

•~--—-—~~~‘-

~‘ ~~ -

~~~~~ T~Jh~ j~~j - • -

_________________ 

•

~~

-4~~~~’J iUW 1j~ -..,

~. ‘ 

.

.

- 

• •- - - - --
I US N~~ine Corps Lii,sion Ofc . AMC. Alexand ria . ATFN: AMCGS—F 1 Oaf & Civil Inst of Enviro M.dicin. Canada •

I USATRADOC . Ft Monro e, ATTN: ATRO --ED I AIR CRE SS, Kstsalngton. ATFN: Info Sys Br

B USATRADOC. Ft Monroe. ATTN: ATP R-A D I Mills.., eloglek Tu nics., Cogihagen

I USATRADOC. Ft Monroe , ATTN: ATTS-EA I Military Attach.. French Embeasy, ATVN: Doc Sic

I USA Faeces Cmd, Ft McPherson. ATTN: Library I Medecin Chef, C E.R.P.A.—Arsen.l. ToulorslNaval France

2 USA Aveetien Test Sd, Ft Bucker , ATTN: STEBG-PO I Prin Salsalific On. Aggi Hum Engr Bath Dlv. Ministry

1 USA Agcy foe Aviation Safety, F I Bucker, ATTN: Library of Define.. New Delhi

1 USA Agcy for Avi ation Safety, Ft Bucker , ATTN : Educ Advisor I Purs Rech Ofc Library, AKA . Israel Defense Forces

I USA Aviation Sch . Ft Bucker , ATTN : P0 Drawer o I Minitteris van Difenal.. 000P/KL AId Sociaal

I HOUSA Aviation Sys Cmd. St Louis, ATI’N: ANSAV—ZDR Psychologliche Zalsen. Thu Hague. Netherlands

• 2 USA Avi ation Sys Teal Act , Edwards AFB . ATTN : SAVTE-T

1 USA Air Out Sets , Ft Bliss , ATTN: ATSA TEN

I USA A ir Motkhity Rich & Div Lab , MoReS FId, AT1N: SAVOL-AS

1 USA Aviation Sth. R.s Tog Mgr. Ft Bucker. ATTN: ATST—T-RTM

-• 1 USA Aviation Sets. CO. Ft Bucker, ATTN: ATST—D—A

I HO, DARCOM. Aleaandr’a. ATTN: AMXCD—TL

1 I

~Q, DARCOM . Alesandria , ATTN: COP

I US Military Academy, West Point, ATTN : Serials Unit - I I US Military Academy. West Point, ATTN: Ofc of Mitt Ldrshp

I US Military Acadensy. West Poe ATTN: MAOR

1 USA Standardization Gp, UK , FPO NY . ATTN : MASE-GC

I Ofc of Naval Rich, Arlington , ATTN : Code 462

3 Olc of Naval Rsch , Arlington , ATTN: Code 458

I Otc of Naval Rich, Arlingto n , ATTN : Cod. 450

1 Ofc of Naval Rich, Arlington, ATTN: Code 441

1 Nasal Aeros pc Med Ret Lab , Pensacola , A’fl’N: Acoua Sets Dlv

Naval Aerospc Med Rev Lab. Pensacola , ATTN: Code L51

- - I Naval Airospc Med Res Lab , Pensacola , ATTN: Code L5

1 Chief of NavPer s . ATTN: Per~-OR

I NAVAIRSTA , Norfolk . ATTN: Safety Ctr

I Nay Oceanographic . DC, ATTN: Code 6251 , Charts & Tech

1 Center of Naval Anal, ATIN: Doe Ctr

I NsvAirSysCoin, ATTN: A lR—5313C

1 Nay BuMad , ATTN: 713

I N.vNeficopterSubSoua 2, FPO SF 96601

1 AFHRL IFTI William AFB

I AFI-$RL ITTI Lowyy AFB

1 AFHRL (AS) WPAF8, OH

2 AFHRL (DOJZ) Brooks AFB

1 AFHRL IDOJN) Lsckland AFB

I HOUSAF IINYSDI

1 HOUSAF (DPXXA)

I AFVTG (P01 Randolph AFB

3 AMRL IHE) WPAFB , OH

2 AF Inst of Tech, WPAFB . OH, ATTN: ENE/S L

1 ATC IXPT0I R~ndolpti AFB

1 USAF ArroMed Lit,, Brooks A FB (StIL—4), ATTN: DOC SEC

1 AFOSR (NLI. Ariington

1 AF Log Cmd, McClellan AFB , ATTN: ALC/DPCRB

1 Air Force Acaderny,CO,ATTN:Dept of Bel Scn

5 NavPers & Dev Ctr.San Diego

2 Navy Med N .uropeychiatric Rich Unit, Sen Diego

1 Nay Electronic Lab , San Diego, ATTN: Ret Lab

1 Nes TrngCen. San Diego, ATTN: Code 0000—Llb

I NavPostGra Sch , Monterey, ATTN : Coda 5BAa - I I NavPoitGra$ch, Monterey, ATTN: Code 2124

I NavTrrsqEquipCtr , Orlando, ATTN: Tech Lib

1 US Dept of Labor. DC. ATTN : Manpower Admin

1 US Dept of Justice. DC, ATTN; Drug Enforce Admin

I Nat Bur of Standards, DC, ATTN : Computer Info SuctIon

I Net Clearing Housa for MH- - Info . Rockville

1 Denver Federal Cti. Lakewood. ATIN: R IM

12 Detente Documentation Canter

4 Dir Psych. Army Hg, RuaseIl Ofcs, Canberra

I Scientific Advsr, Nil Sd, Army Hg , Russe ll Ofes, Canberra

I Mit arid Air ArIactse, Au,tyian Embassy

I Cenlrr do Recherehe Di Facleurs. Humsine di Ia Defense

National., Bruss.l a

2 C.nedlan Joint Staff Washington

I C/Air Staff , Royal Canadian AF . ATTN: Pert Std Anal Sr

3 Chief , Canadian Oaf Rich Staff , A’FTN: C/CBDS(W)

4 BrItish Oct Staff , British Embassy, Washington

38

j