p./

Technical Report 428

A COMPARISON OF ALTERNATE FORMATS

FOR THE PORTRAYAL OF TERRAIN RELIEF

ON MILITARY MAPS

ii

John P. Farrell and Lawrenoe M. Potash

HUMAN FACTORS TECHNICAL AREA

DTIC

r MAR 1 3 1980

S

U.S. Army

Research Institute for the Behavioral and Social Sciences

November 1979

Approved for public release; distribution unlimited. a80 3 12 008
U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency under the Jurisdiction of the

Deputy Chief of Staft for Personnel

FRANKLIN A. HART

JOSEPH ZEIDNER Colonel, US Army

Technical Director Commander

NOTICES

DISTRIBUTION: Primary distribution of this report has been made by ARI. Pleae address corresporidence

concerning distribution of repouts to: U. S. Army Research Institute for the Behavioral and Social Sciences.

ATTN: PERI-TP, 5001 Eisenhoweor Avenue, Alexandria, Virginia 22333.

FINAL DISPOSITION: This report may be destroyed when it is no longer needed. Pleae do not return It to

the U. S. Army Research Institute for the Behavioral and Social Sciences.

NOEThe findings in this report are not to be construed as an official Department of the Army position,

unless so designated by other authorized documrents.
SECURITY CLASSIFICATION OF THIS PAGE (When Date Entered)

REPORT DOCUMENTATION PAGE BEFORE COMPLETING FR

1. REPORT NUMBER .GOTAESINO.3. RECIPIENT'S CATALOG NUMBER

Techica Reprt 23-12. OVTACCESIO No

4. TITLE (and Subtitle) S. TYPE OF REPORT & PERIOD COVERED

A. 9OMPARISON OF .4LTERNATE,.FORMATS FOR THE

PORTRAYAL OF TERRAIN RELIEF ON MILITARY MAPS 6 EFRIGOG EOTNME

7. AUTHOR(s) 9. CONTRACT OR GRANT NUMBER(*)

John P.[ Farrell A. Lawrence M. /Potash

9. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT. PROJECT. TASK ,-, AREA & WORK UNIT NUMBERS

§ US Army Research Institute for the Behavioral

and Social Sciences - 7 734A7

5nol EiRPnhowpr Avenue Algxandria, VA 22333 2__76374____. __3A774__

11. CONTROLLING OFFICE NAME AND ADDRESS I~ E OT E

Fort VAinern Topographic Laboratory 19 14. MONITORING AGENCY NAME & ADDRESS(if different from Controlling Office) 1S. SECURITY CLASS. (of this report-

- ( fUnclassified

-** 15aS. DECLASSI FICATION' DOWNGRADING SCHEDULE

16. DISTRIBUTION STATEMENT (of thie Report)

Approved for public release; distribution unlimited.

17. DISTRIBUTION STATEMENT (of the abstract entered In Block 20, If different from Report)

I8. SUPPLEMENTARY NOTES

19. KEY WORDS (Continue on reverse aide it necessary end identify by block number)

Terrain relief portrayal Map effectiveness

Map evaluation Relief shading

Map interpretation Layer tints

Human performance with naps

ft. ANSTRACT fCeaftku* si ,.vereeside it necewo..y md Ideattify by block number)

"Itwo experiments evaluated the effectiveness of relief shading and layer tints

as supplements to contour lines for solving eight types of military

map-use problems. The addition of layer tints was found to increase the

speed of solving most problems with no loss in accuracy. The addition of

shaded relief increased the time required to solve problems requiring

detailed information but may be valuable in providing a quick general

impression. Personnel experienced in using military maps solved map problems

D JANM743 OTOOFN'OISSLT Unclassified

SECURITY CLASSIFICATIIIN OF THIS PAGE (When Dot& Entereoj
SECURITY CLASSIFICATION OF THIS PAGEOle, Date Entemd)

20. more accurately but no faster, on the whole. Also, solving military

map use problems in the classroom appears to correlate with performance

in the field.

ii Unclassified

SECURITY CLASSIFICATION OF THIS PAGE(When Data Enterea,
Tecilcal Report 428

A COMPARISON OF ALTERNATE FORMATS

FOR THE PORTRAYAL OF TERRAIN RELIEF

ON MILITARY MAPS

John P. Farrell and Lawrence M. Potas --- C,-, -- r r

Raymond C. Sidorsky, Team Chief

, i :13" ..... . . .

Submitted byEdgar M. Johnson, Chief

HUMAN FACTORS TECHNICAL AREA

Approved by:

Milton S. Katz, Acting Director

ORGANIZATIONS AND SYSTEMS

RESEARCH LABORATORY

U.S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES

5001 Eisenhower Avenue, Alexandria, Virginia 22333

Office, Deputy Chief of Staff for Personnel

Department of the Army

November 1979

Army Project Number Information Display

20763743A774

Approved for public reiem; dilgrilbuton unlimilud.

i-Ki

malft
,/

ARI Research Reports and Technical Reports are intended for sponsors of

R&D tasks and for other research and military agencies. Any findings ready

for implementation at the time of publication are presented in the last part

of the Brief. Upon completion of a major phase of the task, formal recommendations for official action normally are conveyed to appropriate military

agencies by briefing or Disposition Form.

iv
FOREWORD

The Human Factors Technical Area of the Army Research Institute

(ARI) is concerned with the demands of the future battlefield for

increased man-machine complexity to acquire, transmit, process,

disseminate, and utilize information. The research is focused on the

interface problems and interactions within command and control centers

and is concerned with such areas as topographic products and procedures,

tactical symbology, information management, user-oriented systems, staff

operations and procedures, and sensor systems integration and

utilization.

One area of special interest is that of human factors problems in

the design of topographic products and procedures. This publication '1 evaluates the effectiveness for solving military map problems of supplementing current, standard relief formats with either shaded relief or

layer tints. Findings can aid in the design of cost-effective special

purpose Army maps and in the objective evaluation of cartographic

products.

Research on the design of topographic products and procedures is

conducted as an in-house effort augmented by contracts with organizations selected for their specialized capabilities and facilities. These

efforts are responsive to the general requirements of Army Project

2Q763743A774 and to special requirements of the USA Engineering

Topographic Laboratory. Special requirements are contained in Human

Research Need 79-230 "Topographic Product Design and Test Methodology."

' ,This research effort was made possible through the assistance of

Alex Pearson of the USA Engineering Topographic Laboratory and through

the cooperation of Combat Developments Experimentation Command, Fort

Ord, California.

SH ZEIDNER

.ecn ical Director

v
A COMPARISON OF ALTERNATE FORMATS FOR THE PORTRAYAL OF TERRAIN RELIEF ON

MILITARY MAPS

BRIEF

Requirement:

To evaluate techniques designed to enhance the display of terrain

information on military maps, and compare the effectiveness of these

- .techniques with standard U.S. Army contour line maps.

Procedure:

In two experiments, the standard U.S. Army contour line maps were

compared with innovative map formats consisting of contour lines supplemented with (a) relief shading, and (b) gradient (layer) tints. The

effectiveness of the maps was evaluated by measuring the accuracy and

speed with which topographic information was extracted from the map

products. A field trial was also conducted to compare the performance

criteria used in Experiments I and II with performance on map-use tasks

in the field, i.e., finding checkpoints and reporting target locations.

Findings:

The addition of layer tints to maps increases the speed with which

information can be extracted from maps with no loss in accuracy.

The addition of relief shading increases the amount of time

required to extract the topographic information needed in the map-use

tasks of most Army personnel, although relief shading has some value for

visualizing terrain.

Experienced personnel solve map tasks more accurately, but not

necessarily faster than less experienced personnel.

-4 Utilization of Findings:

Layer tints are a valuable supplement on special purpose maps for

fast and accurate solutions to terrain information problems, especially

if the problems involve elevation. Shaded relief does not aid in the

solution of map tasks that require detailed and specific terrain information.

vii
N 1

Measurement of the accuracy and speed of extraction of information from a map product was found to be a valuable technique for

evaluation of experimental map products and a valuable alternative to

user surveys and field tests.

- I

IPl

viii
A COMPARISON OF A.LTERN~ATE FORMATS FOR THE PORTRAYAL OF TERRAIN RELIEF ON

MILITARY MAPS

CONTENTS

Page

INTRODUCTION...............................1

Background............................1

The Evaluation of Military Maps.................4

Research Approach........................9

Research Objectives........................15

METHOD.................................15

Experiment I...........................15

Experiment II..........................16

* RESULTS................................16

Overview.................................16

Experiment t...........................17

Experiment 11..........................17

DISCUSSION.............................20

Overview...............................20

Effects of Map Format on Accuracy in Experiments I and 11.I 22

Effects of Map Format on Time Scores in Experiments I and 11 22

Effects of User Experience Level .. ....................... 23

Interactions Between Map Format and User Experience Level . 23

Implications for the Design of Maps...............23

SUMMARY..............................24

REFERENCES...............................25

APPENDIX A. Accuracy Effects in Experiment I............27

B. Median Time Effects in Experiment I. ......... 33

C. Accuracy Effects In Experiment 11I............39

D. Median Time Effects in Experiment 11 ......... 45

E. Field Validation of Eight Subtests of Information

Extraction from Map Products................51

ix
LIST OF TABLES

Page

Table 1. Experiment I: Summary of significant effects on map

format for mean accuracy scores ... ............ . 18

2. Experiment 1: Summary of significant effects on map

format for median time scores .... ............. . 18

3. Experiment I: Statistically significant effects from

user experience levels ..... ................ 19

4. Experiment 11: Summary of statistically significant

effects of map format for time scores .. ......... . 21

LIST OF FIGURES

Figure 1. Examples of three map formats; contour lines,

shaded relief, and layer tints ...... ........... 3

2. Sample landform identification problem ... ........ 6

3. Sample ridge-valley problem ...... ............. 7

4. Sample slope identification problem .... ......... 8

5. Sample high-low area problem ...... ............. 10

6. Sample spot elevation problem ...... ............ II

7. Sample vertical profile problem .. ........... . 12

8. Sample terrain visualization problem .. ......... . 13

9. Sample defilade problem .... ............... . 14

x
A COMPARISON OF ALTERNATE FORMATS FOR THE PORTRAYAL OF TERRAIN RELIEF ON

MILITARY MAPS

INTRODUCTION

Maps and map reading are essential components of most military

operations, and their role has become increasingly important with the

emphasis on combined arms tactics in the modern U.S. Army. Actions of

infantry, artillery, aviation, and armor must be closely coordinated and

precise positional information of the interacting units is required. To

facilitate accurate positioning it is necessary to have maps which

present terrain information in a form that is clear and unequivocal to

4 ithe user.

In an effort to make the terrain relief information on maps more

perceptually accessible to the user, the U.S. Army Engineer Topographic

Laboratory has been developing new map formats and has re',uested ARI

research support to assess the effectiveness of these experimental map

products. This report presents the results of experiments conducted for

this evaluation and briefly presents the research methodology which ARI

developed for that purpose. A more extensive discussion of the research

methodology developed for assessing relief map legibility is presented

by Potash, Farrell, and Jeffrey (1978).

Background

Military maps must present information about terrain, such as steepness, shape, and elevation of landforms, on a two-dimensional surface in

such a manner that it can be perceived quickly and accurately.

For approximately the last 50 years the means of depicting terrain

information on U.S. Army maps has been contour lines, or lines drawn

through all points of the same elevation on a given landform with the

interval between adjacent lines reflecting the vertical interval on the

land. Contour lines replaced other portrayal techniques because they

provide the most precise information about terrain. Unfortunately,

there are a number of difficulties with contour lines. Some of these

difficulties are cartographic: cliffs do not show up well; small terrain

features are not represented; and for rough terrain with extremes of

relief, such as that found in many parts of the western U.S., there

simply may not be enough space on the map sheet for the number of contour

lines required (Marsden, 1955). A more serious problem with contour

lines, however, is behavioral. Many map users have difficulty visualizing terrain when contour lines are the only guide, and it has been

found that many military map users report to hand application of

additional material to enhance their perception of the terrain portrayed

(Skop, 1958). In addition, in many military tasks, such as helicopter

navigiation at low altitudes, there is insufficient time to interpret

contour lines accurately.

-- L -
There are a number of alternatives and supplements to contour lines

as a means of portraying terrain relief. These include: (1) hachures,

or linear "hash" marks, to indicate the presence of slopes; (2) the

Tanaka method, a system of modified "ontours; (3) digital terrain information, or spot heights in numerical forms; (4) perspective drawing or

drawings to give the impression of space or depth; (5) relief shading or

the shading of slopes; and (6) layer or gradient tints which color code

different altitude zones. After conferring with U.S. Army Engineer

Topographic Laboratory personnel, it was concluded that for military map

problems it would be undesirable to sacrifice the precision of contour

lines, so supplements rather than alternative formats were considered.

The particular contour supplements to be evaluated were selected on the

basis of both previous research and the availability of cartographic

resources for generating materials. For example, the quality of hachures

are highly subject to the skill of the cartographic artist applying

*them, and easily degenerate into "hairy caterpillars" (Robinson and Sale,

. 1969, p. 175) when very high levels of expertise are not employed.

Similarly, the Tanaka method was not chosen for evaluation because the

cartographic techniques'required are not available on a widespread basis.

As a result of these considerations, shaded relief and layer or gradient

tints were chosen for evaluation as supplements to contour lines. In

Figure 1, examples of unenhanced contour lines and of contour lines

supplemented with shaded relief and with layer tints are shown.

In shaded relief, light is presumed to come from one direction, and

slopes on the incident side of a hill or given landform are unshaded,

while slopes on the opposite side are in the shade. The degree of

shading varies, with the steeper slopes being darker. In gradient or

layer tints, terrain is color coded to indicate bands of elevation.

Several gradations of a single hue may be used or these may be different

colors; for example, 500 to 1500 feet may be represented by yellow, 1500

to 2500 feet by tan, and all elevation above 2500 feet by brown. In

the current research three to four different colors were used, depending

on the range of elevation present in the specific map segments.

In related research on the value of layer tints Kempf and Poock

(1969) used 12 tints, varying in both intensity and color, to compare

contour lines versus contour lines plus layer tints. Their experiment

showed that the addition of layer tints improved performance on determining the altitude of given points, but had a detrimental effect on

determining grid coordinates. DeLucia (1972) also found that the

.4 addition of relief shading had a detrimental effect on the accessibility

of two-dimensional terrain information.

In another experiment, Phillips, DeLucia and Skelton (1975) compared

four types of maps: contour line, contour line supplemented with shaded

relief, layer tints, and digital or spot height maps. An interesting

innovation in this study was the use of two areas of the Atlantic sea

-2-
I -

r

Figure 1. Examples of Three Map Formats; Contour Lines, Shaded Relief,

and Layer Tints

-3-
floor to produce the maps and thereby control for previous experience

with the map or actual terrain as a contaminating variable. In their

results, layer tints proved best for judging relative height and

visualizing the landscape. Contour lines and contour lines supplemented

with relief shading proved to be roughly equal for all tasks. As

expected, digital or spot height maps were relatively good for height

judgment, but not for other tasks. Phillps et al. used layer tints without

contour lines, while in the current research, contour lines were supplemented by layer tints. There were also a number of procedural

differences.

The results of previous research suggested that both the maps supple-

* ., mented with shaded relief and those supplemented with layer tints would

be superior to unenhanced contour lines. It was also predicted that the

experimental maps would be less advantageous for the more experienced

map users who were more familiar with the standard contour line format

2than for the less experienced map users. The present research sought to

test both of these hypotheses.

The Evaluation of Military Maps

The primary goal of the research was to evaluate the effectiveness

of the alternate techniques for the portrayal of terrain relief. A

secondary goal of the research effort, however, was to develop a research

methodology for the evaluation of military maps. Until recently, alternate map formats were evaluated by either surveys of user preference or

by field tests. Both of these techniques have inherent problems. While

surveys are relatively inexpensive and also provide information on the

degree of user acceptance, user preference surveys have questionable

predictive validity for performance with maps unless the individuals

qtlstioned have actually used the map products under consideration

(Wheaton, Zavala, and Van Cott, 1967). One study found that the preference opinions of potential users were useless as predictors of performance under daylight conditions, and only in a night experiment were

subjective judgments of preference of any value in predicting user

performance (Hill, 1974). The positive results at night were most likely

due to the effect of image quality on perception at low levels of illumination. Under daylight conditions, however, judgment was affected by

biases due to the esthetic qualities of the map, personal preferences

due to previous experience, and other contaminating factors.

Field tests present other problems. Objective field testing is

expensive in terms of time and dollars, and many extraneous variables

which are difficult to control are present. In one study no significant

differences were found between map types because of the large number of

uncontrolled variables present in the field (MAPPRO I, 1973). If only

one terrain is used, the question arises about whether the results are

terrain specific or if generalizations can be made to other terrains.

-4-
Seasonal effects exist and must be controlled; for example, land navigation is generally easier in the late fall and winter when vegetation

does not obscure the shape of landforms or landmarks such as small

streams. A more serious contaminating variable is tracks made through

grass and other vegetation by the early players. These tracks persist

for a considerable length of time and affect the behavior of subsequent

players.

In view of these problems with the more traditional approaches to map

format evaluation, an approach was developed focused on the information

extracted from the map. Somewhat similar approaches were used by Kempf

and Poock (1969), who measured time and accuracy in determining elevation and grid coordinates, and by Phillips, DeLucia and Skelton (1975),

who required participants to extract 12 types of information from maps.

Both the approach of Kempf and Poock (1969) and that of Phillips (1975)

have some similarities to the current approach.

Eight types of relief information normally extracted from maps by

Army users were identified, based upon consultations with personnel at

the U.S. Army Engineer Topographic Laboratory and other representative

military users, and developed into subtests. Map segments representing

four types of terrain ranging from relatively flat to mountainous were

used. Answers were scored for both accuracy and time. Illustrations

of each of the eight terrain formation items are described below;

however, a more extensive description is provided in Potash, Farrell,

and Jeffrey (1978).

Eight Types of Terrain Information

1. Landform Identification. The participants are presented with map

segments with arrows overprinted, and shown sketches of various landforms,

such as hills, valleys, spurs, saddles, or depressions. The participants are then required to specify which of the representative landforms

is portrayed on the map at the tip of each arrow (Figure 2).

2. Ridge-Valley Identification. In this task the participants are

presented with map.segments that contain overprinted lines. The participants' task is to determine if a line is overprinted on a valley or a

ridge, or if it does not fall into either category (Figure 3).

3. Slope Identification. Slopes may be convex, concave or uniform. In

this task the participants are presented with map segments with arrows

overprinted. The participant is asked which type of slope is indicated

and if the arrow runs up or down hill (Figure 4).

-- 5K
1.hill

3. aPMn 4. ~l.

3.~

Figure 2. 

sketches. 

Sample Landf 

Overprinted 

arm Identification problem. Map segment and landform letters on map segments must be matched with appropriate landform.

6-
7Y

I-IF

~4 ~uer4 0042

45 r-3

Figure 3. Sample Ridge-Valley problem. Map segment with overprinted lines

which are used in ridge-valley identification.

-7-
I.

1. concave up 2. concave down

3. convex up 4. convex dowm

5. uniform up *uniform down

Figure 4. Sample Slope Identification problem. Slopes indicated by overprinted lines must be matched with an illustrated slope type.

-8-
4. Identification of High and Low Areas. In this task the participants

are presented with map segments divided into 64 grid squares, each of the

grid squares representing an area 1000 x 1000 meters. The participants

must specify the 4 highest and the 4 lowest grid squares (Figure 5).

5. Spot Elevation Problems. Map segments with several points indicated

by letters are presented to the participants, who are required to

determine the elevation of each point (Figure 6).

6. Vertical Profile Identification. In this task participants are presented with maps with overprinted lines which indicate slopes. The participant must match the slope indicated on the map with one of three

drawings of slope angles (Figure 7).

7. Terrain Visualization. Participants are shown a terrain sketch and

map segments with points indicated. The participants must visualize each

of the points indicated on the map and determine which best matches the

sketch (Figure 8).

8. Defilade or Intervisibility. Map segments with points indicated by

letters are presented to the participants, and the participants are asked

to determine if observers located at selected pairs of points are vieible

to each other (Figure 9).

Since the eight subtests directly measure the accuracy and speed

with which terrain information is extracted from maps, these subtests

are inherently valid. The basic criteria in performance with a map is

the extraction of terrain information. Tests which contain a representative sample of the behavior domain to be tested have content validity

(Anastasi, 1976, p. 96-102 and 133-134). The eight subtests represent

types of information that Army personnel must extract from maps. Thus

these eight subtests are criterion referenced and possess content

validity.

Unfortunately, content validity may not insure predictive validity.

As stated earlier, performance in the field with a map is a complex

multifaceted task, and predictive validity cannot be assumed. Therefore,

as part of the research program, performanc on the eight subtests was

compared with field performance on map use tasks.

Research Approach

In Experiment I a total of 48 Army officers and NCOs were tested for

the speed and accuracy with which they solved terrain information

problems in each of the eight subtests. While there was a considerable

range in the level of previous map experience of the participants, the

overall map-use experience of the group was somewhat higher than that of

the general U.S. Army population.

-9-
7..

toL14 15

17 22 3 23'

2.L.

33 '34' 36, se,

41 _A 4 . AV. f6 148'

-4Z( /1 A l: K.;I492 5537: 54 ! 56.'

51 e"5 [ 59 '6e0 62-6

contour iliterval 40 ft.

Figure 5. Sample High-Low Area problem. The four highest and four lowest

grids on overprinted map must be identified.

-10-
1V. V tJ 

-03

6A k 

-~ ~ *tr *kL~

Figre . ampi potEleaton rolem Elvaionofoveprite pono ma7emn 3 us dniid

..k .. - ... 7
B!

i-- A -I,

Figure 7. Sample Vertical Profile problem. Profile indicated by overprinted

line on map sement must be matched with a line drawing illustrating

vertical profiles.

12

A * 4b
> Pleyto Sch~ol -' .~.i.O (,

Brso

Your approximte angle of vision is enclosed by the dashed

line touchingtthe arrow t ip. The Imnpth of the thick solid

line touching the rrow tip represents the imaxi.am range for

touson.

SHOo~ sean ame

Figure 8. Sample Terrain Visualization problem. Drawing of scene mulst

he matched with overprinted point oil map segment.
II

60

59

N iU 1 ' N,

4¢755

Figure 9. Sample Defilade problem. Intervisibility between points

overprinted on map sement must be identified.

- 14-
Experiment II was essentially a replication of Experiment I, but the

overall map user experience level was lower. The participants in Experiment II generally had been in the Army less time and had less map experience than the participants in Experiment 1. The participants in

Experiment II were chosen so that results of the research effort could

be generalized to a much larger portion of the Army population. The

characteristics of the participants and the research procedure are

presented in greater detail in the Method section.

* I A field validation was also conducted to insure that results of the

measures used in Experiments I and II would generalize to performance in

the field. In this experiment only the standard contour line map was

evaluated and the performance of participants on the assessment test was

compared with performance on field criteria.

Research Objectives

(1) To evaluate two experimental map formats, using shaded relief

and layer tints as supplements to contour lines, and compare them to

standard U.S. Army maps.

(2) To assess the effects of user experience with each type of map,

and determine if alternate map formats are preferable with less experienced personnel.

(3) To determine the effectiveness of a method of evaluating map

products by measuring the speed and accuracy with which information is

extracted from the map.

METHOD

Experiment I

Participants. A total of 48 male Army officers and NCOs ranging in

rank from sergeant through major participated in this experiment.

Although the ,evel of experience in use of the standard Army contour

line map varied, the overall level of experience was somewhat higher

than the general Army population. Participants were obtained from the

Second Armored Division. This particular unit uses maps for land navigation more often than many other units. Approximately two-thirds of the

participants were Ranger qualified and had received advanced instruction

in map use and land navigation. Those participants who were Ranger

qualified and had used maps in exercises regularly within the last two

years were considered highly experienced; those participants who had

Ranger training but had not used maps in exercises within two years were considerd moderate in experience; and those who had not received Ranger

- 15 -
training and had not used maps in field exercises in two years were

considered low in experience. Fourteen participants were in the Low

category, 19 in the moderate, and 15 in the highly experienced category.

Performance Measures. A test containing eight subtests was created,

corresponding to the eight critical types of relief information discussed

earlier, to measure the extraction of information from each map format.

The eight types are: landform identification, ridge-valley identification,

identification of slopes, identification of high and low ground, determination of elevation of spots, vertical profile identification, terrain

visualization, and intervisibility or defilade. Each subtest was scored

on both speed and accuracy. Map segments representing 4000 meter by 4000

meter grid squares accompany questions, except for the "identification of

high-low area" problem which uses 8000-meter grid squares. The map

segments are taken from the standard scale or 1:50,000 map, and four

terrain contour intervals were used: 10, 20, 40, and 80 feet.

Procedure. Participants were tested in small groups of either six or

eight in a large room with widely spaced tables and one participant per

table. Participants were randomly assigned to seats, and the format

placed at a given seat was varied on a daily basis. Participants were

instructed to work as quickly and as accurately as possible. They were

instructed to take a rest break if they became fatigued, but to take the

break only at the end of a problem. Stopwatches were given to each

participant to determine the amount of time required to solve each problem,

and they were instructed to indicate the time required to solve each

problem with the answers on their answer sheet. Each participant worked

with only one problem at a time.

Experiment II

Participants. A total of 30 participants were used in Experiment 11.

The participants were obtained from initial entry officer candidate

students. Overall experience level was somewhat lower than that of the

general Armv population, although five participants did have extensive

prior experience.

Procedure. Ten different participants were tested each day in a

large room for three consecutive days. The map formats placed at the

seats were altered each day and seats were assigned to obtain the

required number of participants in each particular experimental condition.

RESULTS

Overview

In this section, results for the major variables, map format and user

experience, are summarized in terms of both accuracy and time scores for

both Experiments I and If (Tables 1 through 4). Appendixes A-D contain

- 16 -
K

the results in greater detail, with mean accuracy and median time scores

for each experience level as well as analysis of variance tables.

Results from the field validation experiment to assess the predictive

validity of the subtests are presented in Appendix E.

Experiment I

Since there was no overall performance score, separate two-way

analyses of variance (Map Format x Experience Level) were conducted for

each of the eight subtests for both accuracy and time scores. Therefore,

there are 16 two-way analyses of variance. Of the two major variables,

map format and user experience, map format is the more important.

Results on the map format variable are summarized for accuracy scores

in Table I and for time scores in Table 2. In Table 1, statistically

significant effects for map format can be seen on the landform identification, the ridge-valley identification and the spot elevation subtests.

On these three subtests, the map format using layer tints was superior

on two: ridge-valley identification and spot elevation. Inspection of

Table I shows that the accuracy scores on the contour line and layer

tint formats are relatively close to each other, and that on all eight

subtests, the map format using shaded relief was poorest.

Table 2 is a summary of the results using time required to solve the

problem as the scoring criterion. Inspection of Table 2 indicates that with

layer tint format the shortest time was required to solve the problem for

seven of the eight subtests, the only exception being terrain visualization. On only two of the subtests, however, was the effect statistically

significant: ridge-valley identification, and identification of high and

low areas.

The other major variable, user experience level, concerned two significant issues, (1) main effect due to experience and (2) interaction effects

between map format and experience levels. The results regarding experience are presented in Table 3.

Experience level of the user produced a significant main effect on

all eight subtests when accuracy scores were used as a criterion for performance. Using time scores as a criterion, however, produced only

three significant effects. There were no statistically significant interaction effects between experience level of user and map format in any of

the 16 two-way analyses of variance in Experiment 1.

Experiment Ii

In the data on Experiment II, within-cell variance was higher than

in Experiment I and fewer statistically significant effects were found.

- 17 -

" . .-- . - -"i," -' *.. --- ... ,,- " . .. L ..- , : .. , .... ..... .. . . ]. _.,L - A 'k
TABLE I

Experiment I: Summary of Significant Effects on Map Format

for Mean Accuracy Scores

Subtest Contour Lines Shaded Relief Layer Tints

1. Landform Ident 13.64 12.4 13.12*

2. Ridge-Valley Ident 23.86 20.67 25.19**

3. Slope Ident 12.14 10.28 11.69

4. Ident of High-Low Areas 47.6 46.8 48.44

5. Spot Elevation 23.14 16.56 22.56**

6. Vertical Profile 8.5 6.78 7.69

7. Visualization 9.5 8.34 9.13

8. Defilade 20.86 19.44 19.69

* p<. 0 5

** p<.Ol

TABLE 2

Experiment I: Summary of Significant Effects on Map Format

for Median Time Scores

Subtest Contour Lines Shaded Relief Layer Tints

1. Landform Ident 1.02 1.04 .76

2. Ridge-Valley Ident 2.36 2.37 1.71*

3. Slope Ident 1.34 1.36 1.19

4. Ident of High-Low Areas 4.23 3.13 2.51**

5. Spot Elevation 4.51 4.14 3.34

6. Vertical Profile 1.87 1.7 1.57

7. Visualization .88 .78 1.0

8. Defilade 2.68 2.46 1.9

*p<.05

**p<.0 1

- 18 -
Table 3

Experiment 1: Statistically Significant Effects From

User Experience Levels

Accuracy Scores Time Scores

Main Interaction Main Interaction

Subtest Effect With Format Effect With Format

1 1. Landform Ident .01 NS NS NS

2. Ridge-Valley Ident .01 NS NS NS

3. Slope Ident .05 NS .05 NS

4. Ident of High-Low Areas .01 NS NS NS

5. Spot Elevation .01 NS .05 NS

6. Vertical Profile .01 NS .05 NS

7. Visualization .05 NS NS NS

8. Defilade .01 NS NS NS

- 19 -
In the accuracy scores of Experiment II, no statistically significant

effects were found from either the map format or the user experience

level variables. Trends were similar to Experiment I in that the shaded

relief format produced the lowest mean score on seven of the eight subtests with the low experience groups, and the lowest mean scores on all

eight subtests in the higher experience group. Although these effects

were not statistically significant, such consistent trends should not be

ignored. These trends may be seen by inspecting tables in Appendix C.

Statistically significant effects were found in the time scores for

both the map format and user experience level variables. In map format,

the landform identification and the terrain visualization subtests produced significant effects. A summary of the effects obtained with time

scores in Experiment II is presented in Table 4.

For the low experience group, shaded relief was the poorest format,

except on the terrain visualization subtest. Shaded relief was somewhat

better for the higher experience group, as can be seen in greater detail

in Appendix D, Tables D-1 and D-2.

The experience variable produced significant effects on the landform

identification and the ridge-valley identification subtests in time

scores. In the landform identification time score results, a significant

interaction occurred between map format and experience (p<.01). In the

total of 31 tests for interactions between map format and experience in

Experiments I and 11 this was the only significant interaction.

DISCUSSION

Overview

In providing an interpretation of the data acquired in Experiments I

and II, three major issues must be considered: (1) effect of map format;

(2) effect of user experience; and (3) interaction effects between map

format and experience. Several aspects of each of these variables must

be considered. If a particular map format is superior for some subtests

but not for others, this will affect the generalizations that can be made

about that particular format. Effects of experience, and interactions

between map format and experience, must also be discussed. Interactions

are particularly important because a large number of significant interactions would indicate that a particular map format was more effective

at some experience levels than others, and this would also affect the

generalizations that could be made from the data. Two remaining issues

must be addressed in discussing the experimental results: (1) implications of the research for map design and (2) implications for evaluating

cartographic products. This last issue is discussed in Appendix E, where

the predictive validity of the eight subtests is addressed.

- 20 -
TABLE 4

* Experiment I: Summary of Significant Effects on Map Format

for Time Scores

Subtest Contour Lines Shaded Relief Layer Tints

1. Landform Ident 1.24 1.13 .95*

2. Ridge-Valley Ident 1.74 2.18 1.76

3. Slope Ident 1.01 1.46 1.13

4. Ident of High-Low Areas 2.48 2.83 2.31

5. Spot Elevationa Insufficient data on This Subtest

6. Vertical Profile 2.42 1.51 2.45

7. Visualization .73 .61 1.0*

8. Defilade 2.26 1.93 2.02

aThe spot elevation of 32 points must be determined on this subtest, and

due to the generally low experience level of the participants in this

experiment, many of them were unable to complete enough items to obtain

meaningful time scores on this subtest.

**p<. 2

i - 21
Effects of Map Format on Accuracy in Experiments I and II

In Experiments I and II the most important issue is the effect of map

format--whether or not the supplements of shaded relief or layer tints to

standard U.S. Army contour line maps result in superior performance. In

Experiment I, using accuracy scores as criteria, statistically significant

effects were found on three subtests: landform identification, ridge-valley

identification, and spot evaluation. On each of these subtests the difference was small between the map format using layer tints as a supplement and

the standard contour line map. Both of these formats, however, were

-i I superior to the format using shaded relief as a supplement. A general

trend in this direction is also evident on the remaining five subtests;

however, the effects did not reach statistically significant levels.

These general effects occurred for all experience levels in Experiment I.

Experiment II was rather similar to Experiment I but was conducted on

*" a population with a generally lower experience level. It was thought that

the relatively poor performance found with the shaded relief supplement

might have been due to the fact that participants in Experiment I were

somewhat more experienced in using maps than were the general U.S. Army

population. This greater experience with the standard U.S. Army contour

line map might have created a bias in favor of the standard map, so

participants with an experience level somewhat lower than that of the

general U.S. Army population were used in Experiment 11. However, using

accuracy scores as criteria, no statistically significant effects were

found on map format. Mean scores on the layer tint format were consistently higher (although not statistically significant) than on shaded

relief format on seven of the eight subtests in Experiment I. Such a

consistent trend should not be ignored. These results with accuracy

scores in Experiment I reflect the same trend found in Experiment I.

For most Army map use problems involving specific terrain relief information, layer tints appear superior to shaded relief as a supplement to

contour lines. This replication of the effect found in Experiment I with

a different overall experience level among the participants suggests

that the effect is a general one.

Effects of Map Format on Time Scores in Experiments I and II

In the time data for Experiments I and I, results were generally

similar to the accuracy data, but the map format using layer tints to

supplement contour lines was superior to both shaded relief or unenhanced

contour lines. In Experiment 1, significant effects occurred only in

ridge-valley identification and identification of high and low areas. In

both of these subtests, layer tints resulted in faster problem solving

time. In the subtests of Experiment I, layer tints required the shortest

solution time on at least six of the eight subtests regardless of the

experience group. In Experiment 11, the results using time data were

somewhat less clear. For the low experience groups, layer tints and

standard contour lines were better than shaded relief on landform identification, but for the higher experience group the standard contour line map

was poorest.

- 22 -
A statisticailly significant effect in the time scores favored shaded

relief on the visualization task. This effect cannot be considered a

sampling error or a fluke. Examination of the visualization subte-.ts

results in Experiment I (Tables I and 2) indicates a slight effect favoring

shaded relief when both time and accuracy are used as criteria. Once

again, a trend is evident. The visualization subtest differs from the

others in that the participant is required to visualize an area rather

than answer questions concerning precise detail as in the other seven

4subtests.

Effects of User Experience Level

User experience level had a more powerful effect on accuracy than on

time in Experiment 1. This effect was not replicated in Experiment II

because of the high-within cell variance on accuracy scores from the

relatively less experienced participants. The effect in Experiment I,

however, demonstrated that greater levels f experience resulted in more

accurate, but not necessarily faster, solutions of terrain relief problems.

* IAn important implication of this finding is that speed of solving map

tasks may be less sensitive than accuracy as a measure of human performance with maps.

Interactions Between Map Format and User Experience Level

In the combined results of Experiments I and HI, there were 31

statistical tests for interaction between the two main effects, map format

and user experience level. Only one statistically significant interaction occurred in the 31 tests. The lack of strong interaction effects

suggests that while the less experienced participants have lower performance levels than the more experienced, the same relative patterns occur

for different experience levels. This result suggests that special

purpose maps are not necessary for terrain problems with less experienced

participants.

Implications for the Design of Maps

These results have several implications for the design and evaluation of military and commercial maps. In general, the use of layer tints

to enhance contour lines made relief information the most perceptually

accessible to users. This was true for most of the measures which

involved extracting highly detailed information typical of Army map tasks.

The use of shaded relief increased the amount of time required to solve

problems with no increase in accuracy, except on the terrain visualization measure. This suggests that relief shading has some value in giving

a fast general impression of the terrain characteristics in a given area,

but not for questions involving more precise terrain information.

- 23 -
One important result is that when contour lines were supplemented

with shaded relief, which was intended to enhance the perception of

terrain relief, there was an increase in the time required to solve the

problems with no corresponding increase in accuracy for most of the subtests. This unexpected finding, consistent with the work of other

investigators such as Hill (1974), suggests that new cartographic

products require objective evaluation. It is likely that shaded relief

impeded performance because of a cluttering effect, and since such

effects often cannot be foreseen, empirical testing of new military map

products is advisable.

SUMMARY

The primary goal of this research effort was to evaluate two experimental map formats designed to enhance the perceptual accessibility of

terrain relief information. The two experimental map formats supplemented contour lines with either shaded relief or layer tints. The

first experiment involved participants who had relatively more experience

with maps than most U.S. Army personnel. To insure that the results of

research would generalize to the entire U.S. Army population, a group

with less experience was used in a second experiment.

For most of the eight types of terrain information that subjects were

required to extract from maps, the map format using layer tints to supplement contour lines resulted in faster problem solving with no loss in

accuracy. The use of shaded relief did not increase the map reading speed

and sometimes resulted in a decrease in accuracy of extracting the

specific terrain information typical of most 1.S. Army map use problems.

Shaded relief was of some value, however, for giving a general impression

of the topography of a given area. More experienced personnel solved map

problems more accurately but not necessarily faster than inexperienced

personnel. In 31 tests for interactions between map format and experience, only one significant interaction occurred. A map format that is

effective at one experience level is also effective at another, and

special purpose maps for map users with low experience do not appear

necessary.

A secondary goal of the research effort was to further develop the

research methodology for evaluating map formats. Traditional techniques

for map evaluation--user surveys and field tests--have limitations.

Therefore, a technique was developed which measures the speed and

accuracy with which information is extracted from a map product. This

technique was found to be a useful method for evaluating the effectiveness of experimental maps.

- 24 -
REFERENCES

Anastasi, A. Psychological Testing. New York: MacMillan Co., 1976.

DeLucia, A. The effects of shaded relief on map information accessibility. Cartographic Journal, 1972, 19, 14-18.

Hill, A.R. Cartographic performance: an evaluation of orthophoto maps.

Royal College of Art, Final Technical Report, 1974.

Kempf, R. and Poock, G. Some effects of layer tinting on maps.

Perceptual and Motor Skills, 1969, 29, 278-281.

MAPPRO II, Test of new and improved map products, USACDEC Experiment

72-15. Final Report, Phase I. Fort Ord: Combat Development Experimentation Command, 1973.

Marsden, L.E. Shortcomings of contour types. Surveying and Mapping,

1955, 2, 182-188.

Phillips, R., DeLucia, A. and Skelton, N. Objective tests of the

legibility of relief maps. The Cartographic Journal, 1975, 12, 39-46.

Potash, L.M., Farrell, J.P. and Jeffrey, T. A technique for assessing

map relief legibility. The Cartographic Journal, 1978, 15, 28-35.

Robinson, A.H. and Sale, R.D. Elements of Cartography. New York: John

Wiley and Sons, 1969.

Skop, J. Investigation of user requirements pertinent to relief

portrayal. Army Map Service, Corps of Engineers, Final Report, 1958.

Wheaton, G.R., Zavala, A. and Van Cott, H.P. Enhanced photomap

evaluation study. American Institutes for Research, Technical

Report AIR-F-15-4, 1967.

- 25 -
APPENDIX A

ACCURACY EFFECTS IN EXPERIMENT I

Table A-1

Mean Accuracy Scores in Experiment I, Low Experience Group

Subtest Contour Lines Shaded Relief Layer Tints

1.Landform Ident 11.75 10.25 12.5

q2. Ridge-Valley Ident 18.25 13.50 21.8

*3. Slope Ident 10.25 7.5 11.0

4. Ident. of High-Low Areas 44.0 40.5 47.5**

*5. Spot Elevation 20.75 8.75 19.3

6. Vertical Profile 6.5 4.75 5.83

7. Visualization 6.0 7.0 8.3*

8. Defilade 20.0 16.25 18.17**

*p(.05 level

P<O

TABLE A-2

Mean Accuracy Scores in Experiment I, Moderate Experience Group

Subtest Contour Lines Shaded Relief Layer Tints

1. Landform Ident 14.3 12.0 13.14

2. Ridge-Valley Ident 26.17 21.83 26.0

3. Slope Ident 13.0 10.3 11.29**

4. Ident of High-Low Areas 48.3 46.5 47.7

5. Spot Elevation 25.67 19.17 23.57

6. Vertical Profile 10.0 7.17 8.0

7. Visualization 10.10 8.67 9.43*

8. Defilade 20.3 20.17 20.0**

* p<. 0 5

**p<. 0 1

- 27-

TAB1 PACIK BLAA-2M flJM
TABLE A-3

Mean Accuracy Scores In Experiment I, High Experience Group

Subtest Contour Lines Shaded Relief Layer Tints

1. Landform Ident 14.5 13.88 14.33

2. Ridge-Valley Ident 26.0 23.39 30.67

3. Slope Ident 12.75 11.87 14.0*

4. Ident of High-Low Areas 50.0 50.3 52.0

5. Spot Elevation 21.75 18.5 26.67

6. Vertical Profile 8.25 7.5 10.67

7. Visualization 12.25 8.88 10.0*

8. Defilade 22.5 20.5 22.0**

* p<. 0 5

** p<.Ol

TABLE A-4

Analysis of Variance for Accuracy, Experiment I

Landform Identification

Source df MS F P<

Map Format 2 9.47 3.31 .05

Experience 2 27.71 9.67 .01

A x B 4 2.02 .70 NS

Within cells 39 2.86

Total 47

- 28 -

L L i A ,~~~i=i
TABLE A-5

Analysis of Variance for Accuracy, Experiment I

Ridge - Valley Identification

Source df MS F P<

Map Format 2 155.51 6.93 .01

Experience 2 320.03 14.26 .01

A X B 4 9.40 .42 NS

Within cells 39 22.45

Total 47

TABLE A-6

Analysis of Variance for Accuracy, Experiment I

Slope Identi fication

Source df MS F P<

Map Format 2 26.01 2.78 NS

Experience 2 44.36 4.74 .05

A X B 4 5.20 .56 NS

Within cells 39 9.36

Tot a 1 47

29 -

-Ji
TABLE A-7

Analysis of Variance for Accuracy, Experiment I

Identification of High and Low Areas

Source df MS F P<

Map Format 2 40.23 1.35 .NS

Experience 2 166.23 5.60 .01

A X B 4 14.66 .49 NS

Within cells 39 29.70

Total 47

TABLE A-8

Analysis of Variance for Accuracy, Experiment I

Slope Identification

Source df MS F P<

Map Format 2 273.12 13.65 .01

Experience 2 192.38 9.62 .01

A X B 4 35.83 1.79 NS

Within cells 39 20.0

Total 47

- 30 -
TABLE A-9

Analysis of Variance for Accuracy, Experiment I

Vertical Profile

Source df MS F P<

Map Format 2 14.68 1.95 NS

Experience 2 41.60 5.54 .01

A X B 4 6.37 .85 NS

Within cells 39 7.51

Total 47

TABLE A-10

Analysis of Variance for Accuracy, Experiment I

Terrain Visualization

Source df MS F P<

Map Format 2 6.58 .79 NS

Experience 2 40.73 4.87 .05

A X B 4 8.31 .99 NS

Within cells 39 8.36

Total 47

- 31 -
TABLE A-11

Analysis of Variance for Accuracy, Experiment I

Defilade

Source df MS F P<

Map Format 2 14.23 1.98 NS

Experience 2 45.72 6.35 .01

A X B 4 4.!3 .57 NS

Within cells 39 7.20

Total 47

TABLE A-12

Summary of Significant Accuracy

Effects in Experiment I i

Main Effects Interaction

Subtest Map Format Experience

I. Landform Ident NS .05 NS

2. Ridge-Valley Ident NS .01 NS

3. Slope Ident .01 .01 NS

4. Ident of High-Low Areas NS .01 NS

5. Spot Elevation NS .05 NS

6. Vertical Profile NS NS NS

7. Visualization .05 .01 NS

8. Defilade .01 .01 NS

- 32 -
APPENDIX B

MEDIAN TIME EFFECTS IN EXPERIMENT I

TABLE B-I

Median Time Scores in Experiment I, Low Experience Group

Subtest Contour Lines Shaded Relief Layer Tints

1. Landform Ident 1.42 .86 .92

2. Ridge-Valley Ident 3.01 2.16 1.96*

3. Slope Ident 1.72 1.72 1.29

4. Ident of High-Low Areas 4.78 2.67 2.9**

5. Spot Elevation 6.32 4.18 4.12

6. Vertical Profile 1.25 .77 1.38

7. Visualization .68 .86 1.2

8. Defilade 2.57 1.88 1.75

*p< .05 level

**p< .01

TABLE B-2

Median Time Scores in Experiment I, Moderate Experience Group

Subtest Contour Lines Shaded Relief Layer Tints

1. Landform Ident .84 1.32 .68

2. Ridge-Valley Ident 2.23 2.99 1.58*

3. Slope Ident 1.17 1.64 1.17

4. Ident of High-Low Areas 3.54 4.1 2.18**

5. Spot Elevation 3.59 5.1 2.99

6. Vertical Profile 2.13 2.0 1.4

7. Visualization .89 .81 .89

8. Defilade 2.49 3.08 1.86

*p< .05 level

**p< .01

- 33 -

k6 !
TABLE B-3

Median Time Scores in Experiment I. High Experience Group

Subtest Contour Lines Shaded Relief Layer Tints

1. Landform Ident .89 .92 .61

2. Ridge-Valley Ident 1.9 2.0 1.51*

3. Slope Ident 1.24 .98 1.03

4. Ident of High-Low Areas 4.72 2.63 2.48**

5. Spot Elevation 4.09 3.39 2.89

6. Vertical Profile 2.11 1.95 2.35

7. Visualization 1.08 .71 .87

8. Defilade 3.14 2.28 2.29

*p< .05 level

**p< .01

TABLE B-4

Analysis of Variance for Median Times, Experiment I,

Landform Identification

Source df MS F P<

Map Format 2 .44 2.37 NS

Experience 2 .24 1.29 NS

A X B 4 .35 1.86 NS

Within cells 39 .9

Total 47

- 34 -
TABLE B-5

Analysis of Variance for Median Times, Experiment I,

Ridge-Valley Identification

Source df MS F P<

Map Format 2 2.32 3.36 .05

Fxverience 2 1.28 1.86 NS

SA X B 4 .95 1.38 NS

Within cells 39 .69

Total 47

TABLE B-6

Analysis of Variance for Median Times, Experiment I,

Slope Identification

Source df MS F P<

Map Format 2 .31 1.29 NS

Fxperience 2 .89 3.66 .05

A X P 4 .22 .89 NS

Within cells 39 .24

Total 47

-35-
TABLE B-7

Analysis of Variance for Median Times, Experiment I,

High and Low Areas

Source df MS F P<

Map Format 2 12.61 10.01 .01

Experience 2 .15 .12 NS

A X B 4 3.13 2.49 NS

Within cells 39 1.26

* Total 47

TABLE B-8

Analysis of Variance for Median Times, Experiment 1,

Spot Elevation

Source df MS F P<

Map Format 2 6.73 3.20 NS

Experience 2 7.68 3.64 .05

A X B 4 4.20 1.99 NS

Within cells 39 2.11

Total 47

- 36 -
TABLE B-9

Analysis of Variance for Median Times, Experiment I,

Vertical Profile

Source df MS F P<

Map Format 2 .24 .26 NS

Experience 2 3.87 4.12 .05

A X B 4 .60 .63 NS

Within cells 39 .94

Total 47

TABLE B-10

Analysis of Variance for Median Times, Experiment 1,

Terrain Visualization

Source df MS F P<

Map Format 2 .14 .83 NS

Experience 2 .01 .06 NS

A X B 4 .19 1.15 NS

Within cells 39 .17

Total 47

- 37 -
TABLE B-11

Analysis of Variance for Median Times, Experiment I,

Defilade

Source df MS F P<

Map Format 2 2.09 1.91 NS

Experience 2 1.02 .93 NS

A X B 4 .93 .84 NS

Within cells 39 1.10

Total 47

TABLE B-12

Summary of Significant Time Effects in Experiment I

Main Effects Interaction

Subtest Map Format Experience

1. Ldndform Ident NS NS NS

2. Ridge-Valley Ident .05 NS NS

3. Slope Ident NS .05 NS

4. Ident of High-Low Areas .01 NS NS

5. Spot Elevation NS .05 NS

6. Vertical Profile NS .05 NS

7. Visualization NS NS NS

8. Defilade NS NS NS

-38-
APPENDIX C

ACCURACY EFFECTS IN EXPERIMENT 11

TABLE C-1

Mean Accuracy Scores in Experiment II, Low Experience Group

Subtest Contour Lines Shaded Relief Layer Tint I.

1. Landform Ident 12.44 12.12 13.25

2. Ridge-Valley Ident 23.3 20.5 25.25

3. Slope Ident 12.44 11.0 11.87

4. Ident of High-Low Areas 41.66 46.0 48.37

5. Spot Elevation 21.2 18.75 20.75

6. Vertical Profile 9.2 8.43 7.62

7. Visualization 8.3 6.67 9.12

8. Defilade 19.3 16.5 20.5

TABLE C-2

Mean Accuracy Scores in Experiment 1I, High Experience Group

Subtest Contour Lines Shaded Relief Layer Tint

1. Landform Ident 16.0 11.0 13.5

2. Ridge-Valley Ident 32.0 22.0 29.0

3. Slope Ident 14.0 8.0 15.5

4. Ident of High-Low Areas 49.0 43.0 53.5

5. Spot Elevation 25.0 16.5 24.0

6. Vertical Profile 9.0 6.0 12.5

7. Visualization 13.0 7.0 10.0

8. Defilade 22.0 20.0 20.5

- 39 -
TABLE c-3

Analysis of Variance for Accuracy Data Experiment 11,

Landform Identification

Source df MS F P<

Map Format 2 9.38 2.71 NS

Experience 1 3.04 .88 NS

A X B 2 7.35 2.12 NS

Within cells 24 3.46

Total 29

TABLE C-4

Analysis of Variance for Accuracy Data Experiment 11,

Ridge-Valley Identification

Source df MS F P<

Mao Format 2 64.36 1.81 NS

Experience 1 82.03 2.31 NS

A X B 2 17.07 .48 NS

Within cells 24 35.48

Total 29

- 40 -
TABLE C-5

Analysis of Variance for Accuracy DaLa Experiment I,

Slope Identification

Source df MS F p<

Map Format 2 26.77 3.19 NS

Experience 1 2.01 .24 NS

A X B 2 14.60 1.74 NS

Within cells 24 8.40

Total 29

TABLE C-6

Analysis of Variance for Accuracy Data Experiment 1I,

High-Low Area Identification

Source df MS F P<

Map Format 2 62.29 .82 NS

Fxperience 1 37.89 .50 NS

A X B 2 37.62 .49 NS

Within cells 24 76.18

Total 29

- 41 -
TABLE C-7

Analysis of Variance for Accuracy Data Experiment 11,

Spot Elevation

Source d f MS F P

Map Format 2 45.07 1.15 NS

Experience 1 9.67 .25 NS

A X B 2 14.16 .36 NS

Within cells 24 39.21

Total 29

TABLF C-8

F Analysis of Variance for Accuracv Data Fxperiment IL,

Vertical Profile ldentification

Source df MS F P

Ma omt2 10.61 1.15 NS

Experience 1 2.08 .23 NS

A X B 2 17.70 1.92 NS

Within cells 23 9.20

Total 28

-42-
TABLE C-9

Analysis of Variance for Accuracy Data Experiment II,

Terrain Visualization

Source df MS F P<

Map ?ormat 2 19.45 2.16 NS

Fxperience 1 14.36 1.59 NS

A X B 2 6.96 .77 NS

Within cplls 22 9.01

Total 27

TABLE C-1O

Analysis of Variance for Accuracy Data Experiment Ii,

Defilade

Source df MS F P<

Map Format 2 9.10 .75 NS

Fxperience 1 15.83 1.31 NS

A X B 2 4.17 .35 NS

Within cells 22 12.09

Total 27

- 43 -
TABLE C-11

Summary of Significant Accuracy Effects in Experiment IL

Main Effects Interaction

Subtest Map Format Experience

1 . Land form Ident NS NS NS

2. Ridge-Valley Ident NS NS N

3. Slope Ident NS NS NS

4. Ident of High-Low Areas NS NS NS

5. Spot Elevation NS NS NS

6. Vertical Profile NS NS NS

7. Visualization NS NS NS

8. Defilade NS NS N

-44-
APPENDIX D

MEDIAN TIME EFFECTS IN EXPERIMENT 11

TABLE D-1

Median Time Scores in Experiment I, Low Experience Group

Subtest Contour Lines Shaded Relief Layer Tint

1. Landform Ident .81 1.15 .83**

2. Ridge-Valley Ident 1.81 2.27 1.57

3. Slope Ident 1.06 1.58 1.10

4. Ident of High-Low Areas 2.44 3.13 1.99

5. Spot Elevationa Incomplete data on this subtest

6. Vertical Profile 2.39 1.16 2.69

7. Visualization .72 .68 .92**

8. Defilade 2.39 1.99 1.93

a Many of the participants were unable to complete enough items to obtain

meaningful time scores on this subtest.

*p<.05

**p<.Ol

TABLE D-2

Median Time Scores in Experiment I1, High Fxperience Group

Subtes t Contour ,Line, qhadedt Reli ef Laver Tint

1. Landform Ident 5.08 .01 l.,5*

2. Ridge-Valley Ident 1.08 1.80 ?.52

3. Slope Ident .59 .q4 1.25

4. Ident of Bigh-Low Areas 2.83 1.63 3.58

5. Spot Elevationa Incomplete data on this subtest

6. Vertical Profile 2.67 1.17 A 03

7. Visualization .87 .38 1.354*

8. Defilade 1.06 1.72 2.37

a Many of the participants were unable to complete enough items to obtain

meaningful time scores on this subtest.

**pe .01

- 45-
TABLE D-3

Analysis of Variance for Median Time, Experiment I1,

Landform Identification

Source df MS F P<

Map Format 2 5.69 71.83 .01

Experience 1 9.59 121.03 .01

A X B 2 7.05 89.05 .01

Within cells 24 .08

Total 29

TABLF D-4

Analysis of Variance for Median Time, Experiment LI,

Ridge-Vallev Identification

Source df MS F P<

Map Format 2 .60 .92 NS

Experience I .03 .04 .05

A X B 2 1.04 1.58 NS

Within cells 24 .65

Total 29

- 46 -
TABLE D-5

Analysis of Variance for Median Time, Experiment II,

Slope Identification

Source df MS F P<

Map Format 2 .27 1.21 NS

Fxperience 1 .39 1.72 NS

A X B 2 .22 .92 NS

Within cells 24 .23

Total 29

TABLE D-6

Analysis of Variance for Median Time, Experiment 11,

Hiph-Low Area Identification

Source df MS F P<

Map Format 2 .22 .15 NS

Experience 1 .10 .07 NS

A X R 2 3.07 2.18 NS

Within cells 24 1.41

Total 29

- 47 -
TABLE D-7

Analysis of Variance for Median Time, Experiment II,

Vertical Profile

Source df MS F P<

Map Format 2 4.95 2.20 NS

Experience 1 .61 .27 NS

A X B 2 1.01 .45 NS

Within cells 23 2.25

Total 28

TABLE D-8

Analysis of Variance for Median Time, Experiment II,

Terrain Visualization

Source df MS F P<

Map Format 2 .46 8.14 .01

Experience 1 .03 .62 NS

A X B 2 .17 2.99 NS

Within cells 22 .06

Total 27

48-
TABLE D-9

Analysis of Variance for Median Time, Experiment II,

Defilade

Source df MS F P<

* Map Format 2 .24 .30 NS

Exrerience 1 .56 .70 NS

A X B 2 .99 1.24 NS

Within cells 22 .80

Total 27

TABLE D-10

Summary of Significant Time Effects in Experiment 11

Main Effects Interaction

Subtest Map Format Experience

1. Landform Ident .01 .01 .01

2. Ridge-Valley Ident NS .05 NS

3. Slope Ident NS NS NS

4. Ident of High-Low Areas NS NS NS

5. Spot Elevation NS NS NS

6. Vertical Profile NS NS NS

7. Visualization .01 NS NS

8. Defilade NS NS NS

- 49 -
APPENDIX E

FIELD VALIDATION OF EIGHT SUBTESTS OF INFORMATION

EXTRACTION FROM MAP PRODUCTS

Introduction

A secondary purpose of this research effort was the development of

a research methodology for work in this problem area. Traditionally,

new products are evaluated by opinion surveys from user groups or by

field tests. There are problems with both of these methods. Surveys

are inexpensive and do provide data on user acceptance. However, if the

users have not actually used the map under consideration their opinions

have little predictive validity (Hill, 1974; Wheaton, Zavala, and Van

Cott, 1967). Field tests of map products also present problems. Field

tasks involving a map are complex and involve additional skills beyond

simply extracting information from a map. In addition a large number of

variables are present in field trials which are expensive and time

consuming to control. If only one terrain is used the results may be

terrain specific and may not generalize to other terrains. Seasonal

factors are involved, and results may not be the same at different

times of the year since land navigation is easier in the late fall and

winter when vegetation does not obscure terrain features. Many other

contaminating factors exist; for example, early participants make tracks

through vegetation which influence the behavior of later participants.

While field trials can produce valuable data when these factors are

controlled, it would be desirable to develop an alternate method for

assessing the effectiveness of new map formats.

After extensive consultations with cartographers at the U.S. Army

Engineer Topographic Laboratory it was determined that measuring the

speed and accuracy with which terrain information could be extracted from

a map would be a valuable addition to user surveys and field tests as a

method for evaluating cartographic products. Extensive discussions were

held with military map users at the U.S. Army Infantry School to determine

what types of information must be extracted from maps in Army tasks. It

was found that U.S. Army users extract eight types of information from

maps, and a subtest was developed for each type. These eight types are:

landform identification; ridge-valley identification; slope identification;

identification of high and low areas; spot elevation; vertical profile;

terrain visualization; and defilade or intervisibility. These are

described in greater detail in the Introduction to this report.

In the development of behavioral measures, validity is an issue that

must be addressed. Since the eight subtests are based on performance on

a map use task, the test is inherently valid in terms of construct

validity and content validity. If a test contains a representative sample

of the behavior domain to be tested, it has content validity (Anastasi,

1976, pp. 133-134).

51 -

IOBDUD ..... .
Predictive validity, however, must be demonstrated when a classroom

type of test, such as the eight subtests, is developed. The classroom

test must be compared to a criterion where performance is measured, and

correlation must be demonstrated. Since map use tasks in the field are

complex and contain skills other than map reading, it would be unreasonable

to expect extremely high correlations. However, performance in the

field on a map task should correlate with a field criterion. For

determining the predictive validity of these subtests, field tasks of

checkpoint location and target location were chosen.

Method

Participants. A total of 16 participants were used. These participants were a stratified random sample chosen to represent the varying

degrees of experience in the Army population at large. Participants

were obtained from installations other than where the experiment was

being conducted. No participants had prior experience on the terrain

used in the experiment.

Location. The field test portion of the experiment was conducted in

the Gabilan Valey area 'of Hunter-Liggett Military Reservation in California.

The U.S. Army Combat Development Experimentation Command (CDEC) was

conducting field research in this location, and the experiment was

supported by CDEC as a portion of their research effort. The area is

approximately 30 percent flat-to-rolling terrain and 70 percent rollingto-rough terrain. The floor of the Gabilan Valley averages 1200 feet

above sea level and is sparsely vegetated with oak trees and shrub. The

surrounding hills and mountains of the Coast Range are heavily vegetated

with oak, chaparral, and shrub. The area permits cross country vehicle

traffic in dry weather. Almost all of the rainfall in the region occurs

during the winter months, and the experiment took place in April and May

when conditions were dry.

Apparatus. Apparatus consisted of: 1/4 ton trucks; a radar-based

range measurement system (RMS) which used triangulation in conjunction

with a DEC-1061 computer system to determine accurate position location

information; a voice recording system; two-way radios; the map performance

measures, standard Army contour maps of the area, and various forms.

Procedure. The experiment included of both classroom and field tests.

After an introductory briefing participants were given the eight map use

subtests. These subtests were administered in a large room with widely

spaced tables in a single session. The field portion of the experiment

was conducted over the next five weeks.

52
Participants were given training in map use for land navigation and

safety before the field portion of the experiment. Numerous safety

problems are inherent with single participants navigating in relatively

isolated terrain: they may become lost, injured in accidents, etc.

Therefore, for safety, participants were tested singly using light 1/4

ton trucks with drivers. The participant acted as a navigator and

directed the driver. Drivers were trained to avoid giving cues and

restricted to speeds of less than 15 miles per hour.

In the field tests, participants were given two types of land

navigation tasks, checkpoint identification and target location. Four

routes were used with six checkpoints on each route and an observation

post at the end of each route. Thus, each participant had to locate 24

checkpoints. Routes were approximately 6 to 7 kilometers long and the

checkpoints varied from 1/2 to 2 kilometers apart. The order of the

routes for each participant was counterbalanced, and to avoid the

problem of tracks through vegetation becoming a cue for subsequent

participants, vehicles were driven through the area between trials.

Participants were issued maps, paper, and protractor. For each

trial or route they were given a map with a set of instructions and the

eight-digit coordinates of the start point, and six checkpoints. They

were directed to make a thorough map reconnaissance and plot their

planned routes. For the checkpoint identification task, they were taken

to a known start point and directed to find the six checkpoints and the

observation post. When the participants believed that they had arrived

at a checkpoint the position was noted. The exact location was determined

with the RMS. The participants were informed of their actual position

at each checkpoint so that they would always proceed from a known

position and previous errors would not be compounded in locating the next

checkpoint. Checkpoint location scores were determined by noting the

radial error or the direct line distance between reported location of

the checkpoint and the actual location. Direction of errors was not

recorded.

For the target location task, participants were directed to find an

observation post at the end of their checkpoint identification route.

If they were in error they were directed to the true location of the

last checkpoint. All participants were given their location in eightdigit coordinates, and allowed use of binoculars, compass, and protractor

as well as their map. The targets were 12-foot-square panels of wood

located at distances varying between 1000 and 6000 meters from the

observation post. Radial errors between the reported location and the

actual location of targets were calculated with the RMS system.

53

i.... ..... .
Two series of nine targets were used. Participants were allowed

to use any procedures they preferred to locate targets in the first

series. For the second series of targets, participants were directed

to use the procedures outlined in FM 21-26, the Army Field Manual on Map

Reading, to report target locations. Therefore, two sets of target

location scores were obtained, the first using unspecified procedures

and the second using specified procedures.

Results

On the three field tasks, the mean radial error between actual

location and reported location was determined. These scores were 4 correlated with the map subtest scores using Spearman-Rho correlations,

which are more appropriate than the Pearson Product Moment correlation

considering the number of participants involved. These correlations are

presented in Table E-1.

TABLE E-I

Correlations of Map Performance Task with Field Criteria

Subtest Target Location Target Location

Checkpoint Unspecified Specified

Location Procedure Procedure

I. Landform Ident -. 30 -. 10 -. 34

2. Ridge-Valley Ident -.14 -.18 -.27

3. Slope Ident -.32 -.13 -.20

4. Ident of High-Low Areas -.26 -.49* -.30

5. Spot Elevation -.28 -.47* -.22

6. Vertical Profile -.04 -.20 +.33

7. Visualization -.24 -.29 -.63**

8. Defilade -.31 +.005 -.31

*p< .05

**p< .01

54
In the first field task, checkpoint location, all eight correlations

are negative, which was the expected direction, since the field performance was measured by an error score. In the second field performance

task, target location with unspecified procedures, all correlations are

in the expected direction with the exception of defilade. Two correlations are significant at the .05 level: identification of high and low

areas, and spot elevation. In the third field task, target location

with specified procedures, all correlations were in the expected direction with the exception of the vertical profile. The correlation on

terrain visualization was significant at the .01 level.

Discussion

Compared to previous techniques for evaluating maps, the techniques

used in this research may provide a valuable alternative for assessing

the effectiveness of map products. The results of the field validation,

summarized in Table E-l, suggest that the subtests have predictive

validity. Of the 24 correlation coefficients obtained, all but 2 were

in the correct direction, which is negative because the field performance

score was an error score, correlated with the subtests.

Only 3 of the 24 correlations obtained were statistically significant

at conventional levels of confidence, .05 and .01. However, a number of

considerations must be kept in mind in interpreting the data. The

number of participants (16) was rather low, due to the expense of field

tests and logistical considerations; high correlations rarely occur with

a sample size of this magnitude. The field criteria also must be

considered. A large number of variables are always present in the

field, and many of these cannot be controlled. In addition, map use

tasks in the field are complex and a number of skills are involved in

addition to extracting information from a map product. In considering

the impact of these factors, the fact that 22 of 24 correlations were in

the correct direction and that three of the correlations were significant

supports the predictive validity of the subtests.

55 -
DISTRIBUTION

ARI Distribution List

4 OASD (M&RAI 2 HQUSACDEC, Ft Ord, ATTN: Library

2 HQODA (OAMI-CSZI I HOUSACOEC. Ft Ord, ATTN: ATEC-EX-E-Hum Factors

1 HODA (OAPE-PBR) 2 USAEEC, Ft Benjamin Harrison. ATTN: Library

I HODA (OAMA.AR) 1 USAPACOC, Ft Benjamin Harrison. ATTN: ATCP-HR

I HODA (DAPE-HRE.PO) 1 USA Comm-Elect Sch. Ft Monmouth. ATTN: ATSN-EA

1 HQDA JSGRO-ID) 1 USAEC. Ft Monmouth, ATTN: AMSEL-CT-HOP

1 HQDA (DAMI.DOT.C) I USAEC, Ft Monmouth, ATTN: AMSEL-PA-P

1 HODA (OAPC-PMZ-Al I USAEC. Ft Monmouth. ATTN: AMSEL-SI-CB

I HODA (DACH-PPZ-A) I USAEC, Ft Monmouth. ATrN: C, Fad DO Br

1 HODA (DAPE.HRE) 1 USA Materials Sys Anal Agey, Aberdeen, ATTN: AMXSY-P

1 HODA IOAPE.MPO-C) 1 Edgwood Arsenal. Aberdeen. ATITN: SAREA-SL-H

1 HODA (DAPE-OW) 1 USA Ord Ctr & Sch. Aberdeen, ATTN: ATSL-TEM-C

1 HODA (OAPE-HRL) 2 USA Hum Engr Lab, Aberdeen. ATTN: Libw/Dir

I HODA (DAPE-CPS) 1 USA Combat Arms Tng Sd. Ft Banning. ATrN: Ad Supervisor

SHQOA (OAFO-MFA) 1 USA Infantry Hum Rsch Unit, Ft Banning. ATTN: Osief

1 HODA (OARD-ARS-PI I USA Infantry 8d, Ft Banning, ATTN: STEBC-TE-T

1 HODA (DAPC.PAS-A) 1 USASMA, Ft Bliss. ATTN: ATSS-LRC

1 HOA (OUSA-OR) I USA Air Oaf Sch, Ft Bliss, ATTN: ATSA-CTD-ME

I HODA (OAMO-ROR) 1 USA Air Def Sch. Ft Bliss, ATTN: Tech Lib

1 HODA (DASG) 1 USA Air Oaf 8d, Ft Bliss. ATTN: FILES

1 HQDA (DA10-PI) USA Air Def Bd. Ft Bliss, ATTN: STEBO-PO

I Chief. Consult Div (DA-OTSG), Adelphi, MD 1 USA Cmd & General Stf College, Ft Leavenworth. ATTN: Lib

I Mil Asst. Hum Res, ODOR&E. OAD (E&LS) 1 USA Cmnd & General Stf College. Ft Leavenworth, ATTN: ATSW-SE-L

I HO USARAL. APO Seattle. ATTN: ARAGP-R 1 USA Cmd & General Stf College, Ft Leavenworth. ATTN: Ed Advisor

I HO First Army, ATTN: AFKA-OI-TI 1 USA Combined Arms Cmbt De Act. Ft Leavenworth. ATTN: DepCdr

2 HO Fifth Army, Ft Sam Houston I USA Combined Arms Cmbt De Act. Ft Leavenworth. ATN: CCS

I Dir. Army Stf Studies Ofe. ATTN: OAVCSA (OSP) 1 USA Combined Arms Cmbt Dev Act. Ft Leavenworth, ArrN: ATCASA

1 Ofe Chief of Stf. Studies Oft 1 USA Combined Arms Cmbt De Act. Ft Leavenworth. ATTN: ATCACO-E

I DCSPER. ATrN: CPS/OCP 1 USA Combined Arms Cmbt Des Act. Ft Leavenworth. ATTN: ATCACC-CI

1 The Army Lb. Pentagon. ATTN: RSB Chief 1 USAECOM, Night Vision Lab. Ft Belvoir, ATTN: AMSEL-NV-SD

1 The Army Lib. Pentagon. ATTN: ANRAL 3 USA Computer Sys Cmd. Ft Belvoir. ATTN: Tech Library

I Ofc. Asst Sect of 'he Army (R&D) I USAMEROC. Ft Belvoir. ATTN: STSFB-0O

1 Tech Suoport Oft. OJCS I USA Eng Sch. Ft Belvoir. ATTN: Library

I USASA, Arlington, ATTN: IARO-T 1 USA Toographic Lab, Ft Belvoir, ATTN: ETL-TD-S

I USA Rsch Oft. Durham. ATTN: Life Sciences Dir 1 USA Topographic Lab. Ft Selvoir. ATTN: STINFO Canter

2 USARIEM. Natick. ATTN: SGRD.UE-CA I USA Topographic Lab. Ft Belvoir. ATTN: ETL-GSL

1 USATTC. Ft Clayton, ATTN: STETC-MO-A 1 USA Intelligence Ctr & Sch. Ft Huechuca, ATTN: CTD-MS

1 USAIMA, Ft Bragg, ATTN: ATSU-CTD-OM I USA Intelligence Cit & Sch, Ft Huachuca, ATTN: ATS-CTD-MS

1 USAIMA, Ft Braqg, ATTN: Maertust Lb I USA Intelligence Cit & Scht. Ft Huachucs. ATTN: ATSI-TE

1 US WAC Ctr & Sch, Ft McClellan. ATTN: Lib I USA Intelligence Ctr & SCh. Ft Huachuca. ATTN: ATSI-TEX-GS

I US WAC Ctr & Sch. Ft McClellan, ATTN: Trg Dir I USA Intelligence Ctr & Sch, Ft Huachuca. ATTN: ATSI-CTS-OR

I USA Ouartermaster Scl, Ft Lee, ATTN: ATSM-TE I USA Intelligence Ctr & Sch. Ft Husichuca. ATTN: ATSI-CTD-DT

I Intelligence Material Oev Ofc, EWL. Ft Holabird 1 USA Intelligence Ctr & Sch. Ft Huachuca, ATTN: ATSI-CTD-CS

1 USA SE Signal Sth. Ft Gordon, ATTN: ATSO-EA I USA Intelligence Ctr & Sch, Ft Huachuca, ATTN: DAS/SRD

1 USA Chaplemn Ctr & Sch. Ft Hamilton ATTN: ATSC.TE-RO I USA Intelligence Ct & Sch. Ft Huachuca. ATTN: ATSI-TEM

I USATSCH. Ft Eustis. ATTN! Educ AdvisOr 1 USA Intelligence Ctr & Sch. Ft Huechuca. ATTN: Library

1 USA War College. Carlisle Barracks, ATTN: Lib I COR. HO Ft Huachuca. ATTN: Tech Ref Div

2 WRAIR. Neuropsychiatry, Div 2 CDR, USA Electronic Prvg Grd. ATTN: STEEP-MT-S

I DLI. SOA. Monterey 1 HO. TCATA. ATIN: Tech Library

I USA Concot Anal Agy, Bethesda, ATTN: MOCA-MR I HO. TCATA, ATTN: AT CAT-OP-O. Ft Hood

I USA Concept Anal Agcy. Bethesda. ATTN- MOCA-JF 1 USA Recruiting Cmd. Ft Sheridan. ATTN: USARCPM-P

I USA Arctic Tes Cir. APO Seattle. ATTN: STEAC-PL-MW 1 Senior Army Adv.. USAFAGOD/TAC. Elgin AF Aux FId No. 9

1 USA Arctic Test Ctr, APO Seattle. ATTN: AMSTE-PL-TS 1 HO. USARPAC. DCSPER. APO SF 96558. ATTN: GPPE.SE

I USA Armament Cmd. Redstone Arsenal. ATTN! ATSK.TEM 1 Stimson Lib. Academy of Health Sciences. Ft Sea Houston

I USA Armament Cmd. Rock Island. ATTN: AMSAR.TDC I Marine Corps Im.. ATTN: Dean-MCI

I FAA.NAFEC. Atlantic City, ATTN' Library 1 HO. USMC. Commandant. ATTN: Code MTMT

I FAA-NAFEC, Atlantic CrTy. ATTN: Human Engr Br I HO. USMC. Commandant. ATTN" Code MPI.20 2

I FAA Aeronautical CIt, Oklahoma C ty. ATTN: AAC440 2 USCG Academy. New London, ATTN: Admission

2 USA Fid Arty 5th. Ft Sill. ATTN Library 2 USCG Academv. New London. ATTN: Library

I USA Armor Scl. Ft Knox. ATTN: L,brarv I USCG Training Cir. NY, ATTN: CO

I USA Armor Sc.J. Ft Knox. ATTN: ATSB-I-E 1 USCG Training Ctr. NY. ATTN Educ Svc Ofe

I USA Armor Sch. Ft Knox. ATTN: ATS-OT.TP 1 USCG. Psycho Res Br, DC. ATTN: GP 1/62

I USA Armor Sch, Ft Knox. ATTN: ATSB-CDAD 1 HO Mid.Rae Sr. MC Det. Ouantico. ATTN: P&S Div

- 57 -
IUS Marine Corps Liaison Dtc. AMC. Alexandria, ATTN. AMCGS-F Z 7q sr. - Ev a necc:ve Ceraoa

I USATRADOC. Ft Monroe. ATTN. ATRO-ED AIR CIESS Kv-stigto- ATTN info Sys Or

6 USATRADOC. Ft Monroe, ATTN: ATFA-AD .1< rero, -0 015k- i'C, lnae

I USATRAOOC. Ft Monroe. ATTN: ArT-EA V' I lai" A!tacve cr, Enoass, AT TN oc Sec

I USA Forces Cmd, Ft McPherson. ATTN: Linr .r-c 'i :n E ;; P A -A'sev '>y. -i Nava' crnny

2 UiSA Aviation Test Bd. Ft Rucker. ATTN STEBO-PO ,, i' Sc" I i ',- Ao> H,' Engr P'a' D. M'n'sti,

1 USA Agcy for Aviation Safety, Ft Rucker. AT"TN* Lorary )t Diiese Ne. Civ

USA Agcy for Aviation Safety, Ft Pucker ATTN Edac Aovisor Rt's ;vc' Cfc L larara AK ,rae Defcn'e roizvs

' ..SA Aviation Sch, Ft Pucker, ATTN: PO Drawer 0 1 M,nister.s jan Cjerensi C0OP K Aid SOCiaa,

I MQIJ5A Aviation Sys Cmo, St Louis. ATTN AMSA\/-ZDR Psycnoioqisct'e Zavcn 7

0 r %a etf'c'a-i

2 USA Aviation Sys Test Act.. Eowaros AFB. ATTN: SAVTE-T

I USA Air Oct Scts. Ft Bliss, ATTN: ATSA TEM

I USA Air Mobility Rscn & 0ev Lao, Moffett Fid. ATTN SAVDL-AS

I JSA Aviation 5ch. Res Tng Mgt Ft Rucver. ATTN ATST.-T-RTM

I USA Aviation Scrn CO. Ft Racker ATTN ATST-O-A

1e D0 ARCOM. Alexandria. ATTN AMXCD-TL

I MO. DARCOM. Alexandria. ATTN. COP

I US Military Academy. West Point, ATTN: Serijis Unit

1US Military Academy, West Point. ATTN: Oic of MrIr Lorsno

1 IJSI Militare Academy, West Point. ATTN: MAQA

1 USA Standardization Go. UK. FP0 NY. ATTN4 MASE..GC

I Ofc of Naval ish Arlington, ATTN Code 452

3 Ofc: at Naval Rich, Ariington. ATTN Code 45a

1 Oft; of Navai Aschi. Arlington. ATTN Code 450)

IC~c at Naval Rich, Arlington. ATTN Code 441AcaSeDi

1NvlArs Med Res Lab. Pensacoia ATTN CuShDi

1 Navai Aerosoic Meo Res Lao, Pensacoia, ATTN Code L51

INavai Aerosoc Mex Rex Lao, Pensacola. ATTN Coon L-5

1 Chief of Navers. ATTN - Pers-OR

1N AVAI RSTA, Norflk. ATTN Safety Ct'

I 5Jav Oceanograohic. DC, AT'TN: Code 6251 Charts &t,0

1 Center of Naval Anal. ATTN Doc Ctr

I NavArSysCom. ATTN AIR-5313C

:Nav 6u'd. A?-N: 713

I NaxieicooterSubSgua 2. FP0 SF 96601

1 AFMRL (FT) Williami AFS

IAPHRL ITT) Lowry AFS

I 4FHRL iAS) WRPAPS. OH

2 AFHPL 0021 Brooks AFB

.AFHRL DOJNI Lackilano AFS

I -iOUSAF iINYSOI

ItOUSAF DPXXAi

IAFVTG PD)I Randolon AFB

3 AMR L M E) VVPAF. OH

2 AF Insr At Tecn, WPAF8, Of-. ATTN. ENESL

1 ATC (XFTDI Ranooon AFB

I USAF AeroMed Lb. Brooks AFB iSUL-4l. ATTN DOC SEC

1 AFOSR INLI, Arigtov

1AF Lzg Cord, McCIellan AF5. ATTN. ALC.DRCRB

1 A r Force Academy, CO. ATTN: Dept Of Bel Scn

5 NavIN's & 0ev Ctt, San Diego

2 Navy ',no Neuroosycfiiarric Rich Unit. San Dego

!Nay E'ectronic Lab San Diego. ATTN Res Lab

I Nay TrvgCen. San Diego, AT TN: Code 9000-L_ b

:NavPoxtGraScfr. Monterey ATTN. Code 5SAa

1 NavPostGraSch. Monterey. ATTN Code 2124

1 NavTrrrEouioCtr. Orlando. ATTN: Tech Lb

1 US Deot 0t Labor, DC. ATTN Manoowee Admin

IUS Deot of Justice, OC, ATTN: Drug Enforce Admin

INat Sar of Standards. CC. ATTN Computer Info Section

1Nat Clearing Mouse for MM -Info, qocxvi ie

IDenver Faceri Ctr Ldvewoad ATTrN iLMa

2 0efense Oocanientatonr Center

I ia.*:rAma Mo Russel IIfcs Cavoerny3 3

c antrvri. 'i3d Arrna C. lscC i~Ino, 

I' A, Aracnv Aostr an iiss

_I rrie -in, Rn Dr-. r Fss 4- av- !. j jnfntv

2caridian air1 Star, .Va3snnaon

SIC Ar Stat' Roai Canadian AF -%rTN 'rs Sto Anyi 9i

.. n(i> Dvt Ric, Staff ATTN : CROS;1NI

A 3. ;nt'e C ~at yf 2r 'y' ESntas, asnl yni

L 8