Tsc huIcaI Pup.r 286 

~7~

) [AD I

LE

~

EL

THE STRUCTURE OF 

~

COMBAT

INTELLIG ENCE RATINGS

Murray S. Miron and Samuel M. Patten

Syracuse U.iiversity Research Corporation

and

> Stanley M. Halpin

Army Research Institute for the Behavioral and Social Sciences

BATTLEFIELD INFORMATION SYSTEMS TECHNICAL AREA

~

iri

U. S. Army

Research Institute for the Behavioral and Social Sciences

S.pt.mb.r , 1978

Approvsd for public ru sh .; distribution unlimit.d.

7 

~ 

ii ‘3 i

~

) 0 1
U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCE S

A Field Operating A gency under the Jurisdiction of the

Deputy Chief of Staff for Personnel

WILLIAM 1. HAUSER

JOSEPH ZEIDNER Colonel , US Army

Technical Director (Designate) Commander

Research accomplished

under contract to the Department of the Army

Syracuse University Research Corporation

NOTICES

DISTRIBUTION: Primary distribution of this report his been made by ARI. P1., address corre spondence

concerning distribution of reports to: U. S. Army Research Institute for the Behavioral and Sociel Sciences. ATTN: PERI.P, 5001 Eisenho~~ r Avenue. Alexandria . Virginia 22333.

FINAL DISPOSITION: This report may be destroyed when it is no longer needed. Pleas, do not return it to

the U. S. Army Reiewch Institute for the Behavioral and Social Sciences.

IL The findings in this report are not to b construed as an official Depsrtment of the Army position, unless so deligneted by other tuthoriled documents.

—
.

~

-. .-

~~~~~~~~~~~~~~~ [

Unclassified SECURITY CLASSIFICATION OF THIS PAGE ~

W?i , 0.5. Ent.ra

~~

READ I?IITRUCTIONB REPORT DOCUM ENTA TION PAGE BEFORE COMPLETING FORM

I. REPORT NUMBER

Technical Paper 286 . 

~

2. GOVT Acc ESSIoN NO. 3. RE CIPIENTS CATALO G NUMSER

_____________________ — p ThE STRUC1IJRE OF COMBAT INTELLIGENCE RATINGS I

4. TITLE ( idSubtStl.) B. TYPE OF REPORT & PERIOD COVERED

= — Final Nov . j.

~~’2 - Jan 197L

~ 8. PERFORMiNG ORG. REPORT NUMSER

CONTRACT OR GRANT NUMSEP4’e)

___ ~~

‘

~

R’ 

~~~~~~ ~~~ ~Mu ray S

S. PERFORMING ORGANIZATION NAM E AND ADDRESS 10. PROGRAM ELEMENT. PROJECT . TASK

AREA C WORIC UNIT NUMBE RS

Syracuse University Research Corporation ______________

Merrill Lane . 

I~~~~ 621

~1A75~i7oo3 Syracuse. N.Y. 13210 ___________________________

II. CONTROLLING OFFICE NAME AND ADDRESS ,. 75.~ R~fa Ln~~~ U.S. Army Research Institute for the Behavioral / Sept ~~ ber , jg~~ and Social Sciences t., ~

IU- -UIR GF-~~~.

~~~ 5001 Eisenhower Avenue, Alexandria , VA 22333

14. MONITORING AGENCY NAME C AOORESS(I( dJU.tWS Ira., Coni,ollM4 OW..) II. SECURITY CLASS. (of SAl. raporf .l

Unclassified

5.. DECLA SSIFICAT ION/DOWNGRA DINO SCHEOULE

II. DISTRIBUTION STATEMENT (el SAle Report)

Approved for public release; distribution unlimited

(of A. b.ir.cl ,.tored In Sl.Ck 20, II 

~~

If.t.n t ha. Report)

IS. SUPPLEMENTARY NOTES

IS. KEY WORDS (Contlnu. an ,.,.,.. eli. IIn.c..eory and ld.ntily by block n.anb.r)

Tactical Intelligence, Combat Intelligence, Information Ratings , Data Quality ,

Information Quality, Content Analysis, Factor Analysis

\ ABSTRACT ~~~~~~~~~~~~ w,,.’e. U ._ ~~~~ _, lda.i Ifr by block ni Cor)

Previous research has indicated that ratings of tactical intelligence data on

the dilnensionb of accuracy and reliability are inadequate to convey the

analysts’ perception of the quality of the data. Forty messages derived from

28th Infantry Division files for the Battle of the Bulge were rated by 55

enlisted intelligence analysts on each of 50 scales, including the current

Accuracy/Reliability scales. These ratings were factor-analyzed to determine

the underlying Judgment structure. Three basic dimensions emerged, tentatively ,

w~~ FO w ~~~

~~ ,

~ 1413 ~~~~~~~~ ~~~ WOV SSIIOeBS~ ETE Unclassified

SE~~~~~TY Ct*WFICATIOR o. This PAGE ,-_

~~ . Dee. ~~~~~~~
p.- . —- ,“---

~~~

Unclassified

SECURITY CLASSIFICATION OF THIS PAOEMIan Data I.lese ~

20. (continued) .

labeled Accuracy , Relevance, and Directness. Both the current scales are

closely related to the Accuracy dimension. The stability of these three

dimensions remains to be evaluated . The implications of this judgment

structure for quality ratings of combat intelligence data merit further

exploration.~~~

— Settl

~’~ ~~

ctioI1 13

0

Unclassified

SECURITY CLASSIFICATION OF THIS PAGt(

~~ia. Data *nl.ra41
*

~1

Tic hul cal P~p.r 286

THE STRUCTURE OF COMBAT

INTELLIGENCE RATINGS

Murray S. Miron and Samuel M. Patten

Syracuse University Research Corporation

and

Stanley M. Halpin

Army Research Institute for the Behavioral and Social Sciences

BATTLEFIELD INFORMATION SYSTEMS TECHNICAL AREA

Edgar M. Johnson. Chief

Submitted by: Approved by: A. H. Birnbaum, Acting Director Joseph Zeidner ORGANIZATIONS & SYSTEMS TECHNICAL DIRECTOR (Designate) R ESEARCH LA BORATO RY

U.S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES

5001 Eisenho~er Avenue, Alexandria, Virginia 22333

Office, Deputy Chief of Staff for Personnel

Department of the Army

September, 1978

Army Project Number Intelligence Systems 20062101A754 Contract DAHC l9~

73-C.OO1B

Ø ~~~~~~ ~~~~~~~~~~ ~~~~~~~~~~ .,nlhnks,f

78 10 19 O1

~
r - . . - .-______ 

~~

-

.

ARI Research Reports and Technical Papers are intended for sponsors of

R&D tasks and other research and military agencies. Any findings ready for

implementation at the time of publication are presented in the latter part of

the Brief. Upon completion of a major phase of the task, formal recommendations for official action normally are conveyed to appropriate military

agencies by briefing or Disposition Form. ,

‘

I __ ____ 

___

~ 
--- - -_ .. _- —_-~~_-.-_ — ~-.~~._____ ____—-_- —--_-

FOREWORD

One of the primary concerns of the Battlefield Information Systems Technical Area of the

Army Research Institute for the Behavioral and Social Sciences (ARI) is with intelligence

systems-improved tactical intelligence information processing to aid command decisions. The

effective integration and utilization of fragments of intelligence data from many sources requires

that the intelligence analyst evaluate the quality of available data appropriately and communicate

the evaluation accurately to others. The research reported in the present Technical Paper

investigated analysts’ use of a variety of rating scales, including the standard 6-point Reliability

and Accuracy scales , and explored the basic judgmental dimensions on which analysts appeared to

evaluate the quality of data. Previous work has been reported in ARI Technical Paper 260,

“Subjective Interpretation of Reliability and Accuracy Scales for Evaluating Military Intelligence,”

and ARI Research Memorandum 74-14 . “An Inductive Taxonomy of Combat Intelligence Data.”

Research in this area is conducted as an in-house effort augmented by conracts with

organizations selected for their unique capabilities and facilities for research in intelligence

systems. The present research was conducted jointly by personnel of ARI and Syracuse University

Research Corporation, Syracuse, New York , under contract DAHC 19-73-C-0018. Work was done in

response to requirements of Army Project 20062lOlA754,”lntelfigence Information Processing,”

with the cooperation of the U. S. Army Intelligence Center and School, Fort Huachuca , AZ.

.

~

JOSE~H ZEIDNfI~ T~~Fi

~icaI Director (Designate ).

~ 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
THE STRUCTURE OF COMBAT INTELLIGENCE RATINGS

BRIEF

Requirement:

To explore and establish the concepts that underlie an intelligence analyst’s judgments of

intelligence informa tion quali ty, and to develop rating scales that permit accurate and uniformly

understandable ratings of information quality.

Combat in telligence is seldom derived in a simple, straightforward manner from information

available to an intelligence staff; it must be produced through careful analysis of many

fragmentary and frequently contradictory pieces of data. An indispensable aid for such analysis is

a knowledge of the quality of the data. Either the source of the data, or someone close to the

source, must evaluate the data to enable the analyst to focus on the implications of high quality

data and discount low quality data. The evaluation of the quality of intelligence data is currently

reported on two 6-point scales: a source reliability scale and an information accuracy scale, as

defined in Field Manual 30-5, “Combat Intelligence.” Evidence suggests that these scales are not

an effective vehicle for assignment or communication of quality ratings. As a consequence, the

ratings are frequently misused or omitted, and information of potential value to the analyst-a

realistic rating of information quality be someone close to the data source-has been lost.

Procedure:

Two groups of Army Intelligence personnel evaluated a set of intelligence reports, using 50

different ratings scales that included the current 6-point Reliability and Accuracy scales. One

group of 21 enlisted personnel had just completed the U. S. Army Intelligence School course for

intelligence analysts (96B), and the other group of 34 enlisted personnel were just starting the

same course. Most of the reports were from the files of the 28th Infantry Division for 10-15

December 1944 before the German Ardennes counteroffensive began (6 December 1944.

Participants’ responses were analyzed to determine the characteristics of the judgments made in

evaluating the reports and the relationships among the scales , and to derive a taxonomy of reports

based on the rating scale interrelations.

Findings:

Analysts did not successfully discriminate between information Accuracy and information

Reliability; both ratings are used to represent some idea of Accuracy . Furthermore, this underlying

concept of information Accuracy may be better represented by the 0-100 scale of Global Validity

than by either of the two standard scales. 

—.

— ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
I

Analysis of the participants’ ratings shows three basic factors or dimensions in the

psychological space which determined the way these intelligence messages were evaluated; that is,

the analysts (both trained and untrained) judged each message on three independent factors. The

primary factor deals with the ACCURACY or truthfulness of the message. In evaluating this aspect

of message quality, the analysts made use of the standard Accuracy and Reliability scales , the

Global validity scales , and bipolar scales such as True/False , Probable/Improbable, and

Acceptable/Unacceptable. The ratings which defined the second factor suggest that the analysts

held a concept of information RELEVPINCE or importance. In evaluating this aspect of message

quality, they used bipolar scales such as Heavy/Light, Many/Few , Large-scalefSmatt-scale, and

Risky/Routine. The third factor in message quality was tentatively identified as DIRECTNESS;

here the analysts used bipolar scales such as Interpreted/Uninterpreted, Imp lied/Unimplied, and

Understandable/Conf using.

Utilization of Findings:

The findings can be used to guide the development of scales which will more explicitly

communicate analyst’s evaluation of the quality of intelligence. Several modifications in

procedures are suggested by the obtained Accuracy/Relevance/Directness structure of ratings. In

particular, unambiguous scales to evaluate these aspects of information quality may be developed

to replace the current 6-point scales of Accuracy and Reliaoility. The present findings and the

usefulness of any new scales should be validated with a sample of more experienced intelligence

personnel and with a different scenario.
. -

~~~--~~~~~--

TIlE STRUCTURE OF COMBAT INTELLIGENCE RATINGS

CONTENTS

Page

INTRODUCTION 1

Current Ratings of Information Quality 1

Research Approach 2

Objectives of the Present Research 3

METHOD 3

Model for Data Collection 6

Data Elements 8

Taxonomic Analysis 9

Rating Scales 9

Subjects 11

Test Procedure 11

ANALYSIS 11

Data Reduction 11

Discriminant Function Analysis- -Subjec t Groups 13

Factor Structure of the 50 Basic Scales 17

Factor Structure of Intelligence Reports 25

Supplementary Analyses 31

SUMMARY AND CONCLUSIONS 35

APPENDIX A. Messages Rated By Subjec ts A-i

B. Report Factor Scores of the Scale Structure B-i

C. Analysis of Relationships Among Accuracy,

Reliability, and Other Scales C-i

D. Step Multiple Regression Analyses D-1

DISTRIBUTION LIST E-1

E

~

.. _ ____
LIST OF FIGURES

Page

1. Two-dimensional Taxonomic Data Matrix 6

2. Three-dimensional Taxonomic Data Matrix 7

~~~. Four-dimensional Taxonornic Data Matrix 7

1k. . Rating Scales Used in the Experiment 10

5. Data Analysis Procedures

B-i. Report At tribute Structure: Untrained Subjects B- 9

B-2. Report Attribute Structure: Trained Subjects B-b

B-3. Report Attribute Structure: Combined Subject Groups B-li

B-!l.. Factorial Dimensions and Octants of Judgmental Space B-12

C-i. Reliability Scale Relationships Over Messages:

Untrained Subjects C- 5

C-2 . Reliability Scale Relationships Over Messages:

Trained Subjects C- 6

C-3. Reliability Scale Relationships Over Messages:

Combined Subject Groups C- 7

C-4. Accuracy Scale Relationships Over Messages:

Untrained Subjects C- 8

C-S. Accuracy Scale Relationships Over Messages:

Trained Subjects C- 9

c-6. Accuracy Scale Relationships Over Messages:

Combined Subject Groups C-b

C-7. Global Scale Relationships Over Messages :

Untrained Subjects C-il

C-S. Global Scale Relationships Over Messages:

Trained Subjects C-12

C-9. Global Scale Relationships Over Messages:

Combined Subject Groups C-13

C-iO . Cluster Groupings of Reliability, Accuracy and

Global Ratings c-il.
r -

-.- --.

.

~~~~

LiST OF TABLES

Page

1. Judgmental Aspects of Intelligence Analysis 5

F 2. Characteristics of the Sample 12

3. Discriminants of Quality Judgments Between Trained

and Untrained Analysts i6

l.. Group Means and Standard Deviations of Combined Reports ,

by Rating Scale 19

5. Rating Scale Factor 3tructure : Trained Subjects 21

6. Rating Scale Factor Structure: Combined Subject

Groups 23

7. Rating Scale Factor Structure: Combined Subject

Groups 24

8. Group Means and Standard Deviations of Overall

Qualities, by Reports 27

9. Report Factor Structure: Untrained Subjects 28

10. Report Factor Structure: Trained Subjects 29

11. Report Factor Structure : Combined Subject Groups 30

D-i. Step Multiple Correlation Using Subjects as

Observations and Reliability as Criterion D- 5

D-2. Step Multiple Correlation Using Subjects as

Observations and Accuracy as Criterion D- 6

D-3. Step Multiple Correlation Using Subjects as

Observations and Global Ratings as Criterion D- 7

D-4. Step Multiple Correlations Using Reports as

Observations and Reliability as Criterion D- 8

D-5 Step Multiple Correlations Using Reports as

Observations and Accuracy as Criterion D- 9

D-6 Step Multiple Correlations Using Reports as

Observations and Global Ratings as Criterion D-1O 

~ -. - — .- ~~~-~
ARI TP 286

THE STRUCTURE OF COMBAT INTELLIGENCE RATINGS

I NTRODUCTION

Improvements in the technology and techniques of intelligence data

collection have imposed additional burdens on both the intelligence

analysts and the decisionmaker. Analysts must deal with the increased

volume of information . The decisionmaker is increasingly remote from

raw unprocessed information and therefore is forced to rely on others for

evaluating the quality of data. To assist users of intelligence data in

discriminating the valuable from the worthless, a system fot evaluating

the quality of intelligence data was developed during World War II.

The system, described in DA Field Manual 30—5, Combat Intelligence,

requires an examination of incoming informat

~

ton for pertinence and timeliness and provides for explicit ratings of source reliability and

information accuracy .

Considerable anecdotal evidence and some experimental evidence show

these procedures as inadequate . Present research examines the underlying

conceptual structure characterizing analysts ’ ratings of tactical

intelligence data. The research also examines the role of the current

Reliability and Accuracy ratings. An alternative approach to rating the

quality of intelligence data is suggested.

Current Ratings of Information Quality

Upon receipt of an intelligence report , the analyst must first

determine whether the information contained in the message or report is

pertinent to his unit, and whether it is timely or has been overtaken by

events. The analyst must also evaluate the reputation of the source or

reporting agency for submitting factual reports (reliability) as well as

the factual nature (accuracy) of the data being processed .

Evaluations of pertinence and timeliness presumably are reflected

in the analyst ’s decision to continue an examination of the data.

Accuracy and reliability evaluations result in explicit judgments on

two standard rating scales. These ratings are attached to the data;

other users of the data determine the quality and usefulness of the

data, at least in part, on the basis of these ratings.
n. . — —

~~~~~~

- 

~~~~~~~~~~~~~~~~~~~~~~~~~

-

~

-

~~~~~~~~

-----

ARI TP 286

Reliability is judged on the basis of a 6—letter system and Accuracy

is judged on the basis of a 6—number system, as follows :

Reliability of Source and/or Agency Accuracy of Information

A Completely reliable 1 — Confirmed by other sources

B — Usually reliable 2 = Probably true

C — Fairly reliable 3 — Possibly true

D Not usually reliable 4 — Doubtfully true

E — Unreliable 5 — Improbable

F — Reliability cannot be judged 6 — Truth cannot be judged

There are several problems with the present system of evaluation.

Analysts seldom agree or’. the precise meaning of the descriptive words

and terms . Possibly because of the confusion, ratings are frequently

omitted (Baker, McKendry, and Mace, 1968).*

In addition , although analysts are specifically instructed to rate

reliability and accuracy as separate items, individual analysts

consistently use either reliability or accuracy as the basic criterion

of message quality and correlate the other ratings with it (Meeland and

Rhyne, 1967; Sainet, 1975).**

As a result, ratings tend to be A—i , B—2, C—3, etc. Furthermore,

~

uost analysts use the Accuracy scale as their basic criterion and may

tend to inf late their ratings or not report items of low accuracy/

reliability. Nearly three—fourths of all reports are rated B—2 (Baker

et al., 1968; Samet, 1975).

Research Approach

The approach taken to uncover the psychological bases for quality

judgments was to ask a number of trained and untrained intelligence

analysts to make many rat ings of quality of each of 40 intelligence

messages. These ratings were examined for patterns that would

*Baker, J. D., McKendry, J. M. and Mace , D. J. Certitude Judgments in

an Operational Environment. ARI Technical Research Note 200.

November 1968. (AD 681 232)

**Meeland , T., and Rhyne , R. F. A Confidence Scale for Intelligence

Reports : An Application of Magnitude Estimation Scaling. Menlo Park,

Calif.: Stanford Research Institute Technical Note 4923—31.

June 1967.

Sainet , M. G. Subjective Interpretation of Reliability and Accuracy

Scales for Evaluating Military Intelligence. ARI Technical Paper

260. January 1975. (AD A003 260)

2 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ___
____ ___ ____ __-

~~ ___—---- ~~~~ .

ARI TP 286

indicate the underlying criteria being used by the analysts . An initial

conceptual analysis of the information evaluation task identified six

types of judgments that might be made: These judgments concerned (1)

data source , (2) message content, (3) situation , (4) action reported

considering the overall tactical situation, (5) inferences drawn from

the data, and (6) characteristics of the message itself. Forty—six

bipolar scales (e.g., Direct/Indirect ; Widespread/Local; Garbled/Clear)

were generated to represent these general concepts; these 46 scales, the

standard Reliability and Accuracy scales, and two 0—100 “Global

validity” scales were used by the analysts to evaluate the 40 messages .

The message set consisted of 33 messages from the files of the 28th

Infantry Division for the period 10—15 December 1944 (just before the

German Ardennes Counteroffensive of 16 December 1944) and an additional

7 fictitious and misleading messages generated for the experiment . The

raters were 21 “trained” enlisted personnel who had just completed the

U.S. Army Intelligence Center and School (USAICS) course for intelligence analysts (96B) and 34 “untrained” enlisted personnel who were just

entering the same course. Most of the participating personnel had had

little Army experience beyond Basic Training and , in the case of the

trained group , the USAICS course.

Ob ectives of the Present Research

The present experiment sought (1) to establish the underlying

conceptual structure which determines the way analysts evaluate the

quality of intelligence data, (2) to examine the relationship of current

Reliability/Accuracy ratings to the dimensions of the underlying

conceptual structure , and (3) to examine the utility of the conceptual

structure used to organize intelligence data. The question of the

amount and type of information needed to make an accurate assessment of

the quality of intelligence data was not addressed .

METHOD

The approach is based in techniques and procedures of numerical

t axonomy (Sneath and Sokal, 1973)*, which have been used widely in

biological analysis to identify natural grouping of species empirically .

Mere , these techniques identify natural groupings or f amilies of messages

and determine the characteristics considered important by observers in

evaluating the messages.

*Sneath, P. H., and Sokal, R. R. Numerical Taxonomy. San Francisco ;

V. H. Freeman & Co., 1973.

3

_____ 

~~------—- .~~
I

ARI TP 286

The intelligence analyst’s task was subdivided into basic judgments

about the :

(1) reliability of the data source (or agency) ;

(2) accuracy of the information;

(3) situation (its stability, and the magnitude of the action or

size of forces involved);

(4) action reported (feasibility of the action , and risks f or the

rater’s unit inherent in such action);

(5) inferences to be drawn ultimately from the irformatlon (the

value or utility of the information , and the nature of the inferences

which can be attributed to it); and

(6) characteristics of the message itself (clarity, timeliness, comprehensiveness).

Taken together, the aspects of incoming information define an a priori

conceptual framework that describes the data supplied to analysts.

These aspects also suggested specific bipolar measurement scales or

features, shown in Table 1.

The discriminatory power of the taxonomic characteristics, their

mutual independence , and their adequacy in representing the judgmental

process is, of course, subject to empirical test. These scale

dimensions do not exhaust the ways in which intelligence information

might be evaluated.

However, the aspects of intelligence evaluation chosen for measurement were judged to be sufficiently extensive to enable participants

to make a comprehensive evaluation of a selected set of tactical

messages while keeping the rating task within reasonable and practicable

bounds .

The 46 characteristics or qualities shown in Table 1 represent

judgments that depend on an interpretation of the content of the

messages. As such, they cannot be considered strictly as characteristics

of the stimulus. Instead , they represent in abstract terms the manner

in which the analysts make their judgments of message content.

Thus , the rating scales serve as the operational definition of the

conceptual framework of the structure of intelligence messages; the

empirical analyses serve to test this framework and thereby provide a

mere precise understanding of an analyst’s message ratings

.4
~ .—-- ~~~~~~~~ .

ARI TP 286

Table 1. Judgmental Aspects of Intelligence Analysis

Qualities Rated by

Basis for Judgment Elements of Judgment Bipolar Scales

Source/Agency Reliability Reliable-Variable

Dependable-Undependable

Truthful-Deceptive

Acceptable-Unacceptable

Faultless—Fau lty

Information Accuracy Accurate-Erroneous

True-False

Confirmed-Unconfirmed

Consis tent-Inconsistent

Substantiated-Contradicted

Situation Stability Stable-Unstable

Constant-Changing

Precarious-Imprecarious

Inert-Volatile

Active-Inactive

Magnitude Large Scale-Small Scale

Widespread-Local

Massive-Insignificant

Many- Few

Heavy-Light

Action Feasibility Feasible-Infeasible

Possible-impossible

Probable- Improbable

Likely-Unlikely

Expected-Unexpected

Risk Rou tine-Risky

Ordinary- Extraordinary

Safe-Dangerous

Secure-Insecure

Unhazardous-Hazardous

Inference Utility Useful-Useless

Relevant-Irrelevant

Pertinent-Extraneous

Analyzable-Unanalyzable

Appropriate- Inappropriate

Attribution Observed-Inferred

Unimplied-Implied

Uninterpreted- Interpreted

Fac tual-Theoretical

Direct-Indirect

Report Comprehension Specific-Vague

Clear-Garbled

Intelligible-Unintelligible

Understandable—Confusing

Timeliness Timely-Unti

~mly

Comprehensiveness Fragmented-Complete

S 

__ -

~~~~~~~ -.-~~~~-- ---~~~~~~~~ .-~~~~~~
ARI TP 286

Model for Data Collection

Ordinarily, in biology or linguistic semantics, the presence or

absence of a particular trait or characteristic is tabulated in a matrix

which takes the form of Figure 1 where the F i represents groupings of

characteristics, f1, and each datum is described by a pattern of fj

occurrences , +, or absences , — .

DATA ELEMENT S

2 3

f + — + — — F1 ~ — + + + —

f - + + - +

F 2 ~ + - + + +

4

Figure 1. Two-dimensional Taxonomic Data Matrix

In order to provide for greater refinement of measurement , the bipolar qualities have been scaled in seven steps indicating deg rees of intensity. Thus, the feature matrix becomes three—dimensional (Figure 

2).6
r —-— -

~

- —____

ARI TP 286

c

~

G

~~~~~~~~~~~~~~iiiiiii77

3 DATA ELEMENTS

~~~~~~~~~~ 34 5 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~

____

Fi gure 2. Three-dimensional Taxonomic Data Matrix

If we now add multiple observers as judges of each feature of each

datum, the matrix becomes four-dimensional. Collapsing the structure

over degrees of intensity (di) of quality scales, for the purpose of

illustration, produces the taxonomic data matrix shown in Figure 3 where

1 ~ dj ~ 7 represents the scalar degree of each set of bipolar qualities

for each datum (message).

S ~~~~~~

~~~~~~ ~~~~-

~~~TA ELEMENTS

2 3 4 5

~ 

d,

F

• I

F

4

Figure 3. Four-dimensional Taxonomic Data Matrix

7

t ___ 

-

~~~~~~~~~~~~ “----.— -~~—
ARI TP 286

The use of multiple observers in this model represents the repetition of the judgmental process to enhance the reliability of each

jud gment . This procedure allows for averaging of observation sampling

errors and permits the use of parametric indices of message similarity .

It is possible to construct separate observe r sampling divisions

within the total set of observers so as to assess the commonality of the

judgments across separate groups. Such sampling divisions were employed

in the present project.

Data Elements

The basic taxonomic units or data elements to be evaluated by the

analysts were 33 naturally occurring intelligence messages drawn from

reports of enemy activity during the period 10 December through 15

I)ecember 1944 along the Ardennes line in Europe. To these 33 messages, seven fictitious and misleading reports were added to determine whether

analysts would detect them. Thus, each subject would evaluate 40

messages.

These messages (Appendix A) depicted the buildup of the defending

German Army which eventually culminated in an aggressive , large scale

counteroffensive and breakthrough of the Allied lines on 16 December, known familiarly as the Battle of the Bulge. After the fact, it is

possible to deduce the significance of the intelligence messages .

At the time , however, the strength of the attack by a presumably

defeated , demoralized and hard—pressed enemy caught the Allies by

surprise. The messages vary from trivial sightings of horse—drawn

carts to reports of massive troop movements , POW interrogations, and

G2 summaries. They vary in length from short to long, diff er in degree

of completeness , and originated from several agencies and sources.

Although the situation is dated and involves a style of warfare

which differs sharply from the counterinsurgencies of more recent times,

the messages, unlike fictional scenarios, are real, have a documented

outcome , and are naturalistically valid.

B
-

ARI TP 286

Taxomonic Analysis

A clear definition of the data elements or “operational taxonomic

uni ts” of interest and a conceptual framework specifying which characteristics of the data elements should be measured provide the basic

elements for a taxonomic analysis. The present analyses focused on the

relationships among the characteristics of the messages .

Any consistent patterns in the use of the rating scales across

observers and across messages gave clues about the nature of the underly ing judgment space. Thus, ~f observers consistently rated as “probable ” and “likely” messages whi h they had also said were “possible ,”

then one dimension of their perceptual framework was some concept of the

likelihood of the event reported .

In a similar fashion , similarities among messages were examined .

Some messages were consistently rated as “likely,” others as “unlikely,”

some as “routine ,” others as “risky.” By grouping together messages

which received similar ratings , natural families of messages were

identified .

The same data were used to examine both the judgment space (the

grouping of characteristics) and the report space (the grouping of

messages). If our original conceptual framework was valid , then we would

expect that the grouping of messages would be maaningful to intelligence

analysts

However, complete validation of the framework would requite a revision

of the rating scales followed by data collection and examination of

the message groupings using the revised scales.

Rating Scales

The subjects rated each of the 40 messages on the 46 quality scales

chosen to represent the elements of judgn~ nt identified in the concept

of the rater’s task, as shown in Table 1. In addition , the traditional

Reliability and Accuracy rating scales and a Global or overall validity

scale were included in the rating task.

This latter scale, suggested by Samet (1975), was used twice, at the

beginning and end of the rating task for each message . Figure 4 shows

the scales as presented on the subjects’ response form ; the second

Global rating was on the reverse side of the form .

9
ARI TP 286 •

0 0

~~~~~

o 8 8

.1 1 .1 .1 I I .1 .1 .1 . 1 1 i I 1 1 1

H i J I1 ” I ”H .IHH 1i I1 .I J H

HHH ~~ I ) H I ) I HHH I ~~~~ I I I I I I I I 

~ 

I . 1 I .i I I 1 .1 . 1 • i I .

1 I .i I

I H I H I )) H IIl HHH l ~~~~?

I 1 I I I I 1 I ‘I I 1 I I

— 0 . ~

* ~ 8 8 8 ; ; ~ a $

~ E

5 0 0

5 • ~ 0.

x

w

0) — a

a • 8 5 - ~

- 0

~~~~~ 0 • .

~ —

~~~ I

J j . . I I .j . I I J . j J ..I J . II J.. I J . j ..I ..I .J~~

1 ) 111 .1 .1 1 11 .I l Il t ..I .i I I I 

~ 

I HHI H I HHHHHHH I ~~~~ ) H IIIIII H H IHIHH II

8 8

0 - Ii.

a a

10
-

~~~~~~~

ARI TP 286

Subjects

The sample employed in this project consisted of a school—trained

group, hereafter referred to as “trained ,” and one designated as “untrained.” Subjects were 59 enlisted personnel assigned to the course

for intelligence analysts (96B) at the U.S. Army Intelligence Center and

School (USAICS) , Fort Huachuca , Arizona .

The untrained group had just enrolled in the schoo1 and the school—

trained group had just comp leted the tra ining course . Of the 59 subjects

tested , the data from four were discarded on grounds of incompleteness or

obvious pattern checking behavior (e.g., rating all messages haphazardly).

The final complement of 55 subjects consisted of 34 untrained and 21

trained analysts (Table 2). Both groups had only minimal Army experience.

Test Procedure

Subjects were provided a 1:100,000 map of the area in which the

action took place , with troop deployments as of 10 December 1944 overprinted on the map .* An acetate overlay and grease pencil were provided

each subject for making notes and changing deployments as each intelligence report was examined.

In additiotA , the subjects were given a book of instructions on the

task, a book of answer forms, and the set of sequentially ordered intelligence reports covering the period from 10 December through 15 December.

Each group was tested in a session of approx~ .Qately three and one—

half hours. Two monitors were present during the sessions to give

instructions , answer questions , and check on progress throughout the

data collection .

ANALYSIS

Data Reduction

The data comprised 110,000 observations collected from the 55

subjects who rated each of the 40 reports on the 50 scales. The

resulting raw data matrix was examined from a number of viewpoints

(Figure 5). All data were transferred to punch cards , and the analyses

proceeded according to the five steps indicated in the figure.

*A USAICS Practical Exercise for nonresident instruction is based on the

same area. Prepared maps were thus available and provided by the

Tntelligence School.

11
-~~~~~~~~ ---- -

ARI TP 286

Table 2. Characteristics of the Sample

(n = 55)

Characteristics Trained Untrained

Total 21 34

Men 16 27

Women 5 7

Grade:

E1-E5 17 33

E6-E9 4 1

Length of Service:

i-6 months 13 28

7-12 months 2 3

13-18 months 1 0

19-24 months 0 0

25 + months 5 3

Previous Intelligence training:

School 0 0

On-the-Job training 0 3

-~~~

. 

T~~:::1.

~~~I~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
—.-------.- -..

~~-- — --- — —--,.

—-———---—-----..,-

~~

.-.,. -

~ - ,.- --.~~~~ -- . -~ ~~~~~~~~~~~~~~ .-.

ARI TP 286

The analyses compared the trained and untrained subjects using data

averaged over the separate reports to obtain mean scale ratings for each

subject , or averaged over scales to obtain mean report ratings for each

subject . Each type of analysis is interpreted in the following sections .

Only summary tables of these analyses are presented.*

These analyses were designed to answer the following questions , which

are keyed to the analysis steps of Figure 5:

(I) Is the pattern of quality ratings emp loyed by the trained and

untrained analysts discriminably different? In what ways , if any, does

the school training make a difference in the conceptual framework used?

(II) What is the nature of the conceptual framework underlying the

evaluation of intelligence reports? Is there an aggregate structure

common to trained and untrained subjects?

(III) What is the nature of the resulting report classification?

How were reports grouped by the subjects when the scale qualities are

analyzed? Is tha~ report structure meaningful? Does it imp ly measurement attributes for wh ich there is no current measurenent or training?

(IV) What are the determinants of the currently employed Reliability

and Accuracy judgments? Are these judgments independent across differing

reports and subject groups ? Can they be assessed by diffe rent and

perhaps purer methods?

(V) What is the predictive validity of the Accuracy and Reliability

ratings with respect to individual subjects? Is the validity of these

scale ratings influenced by subject training? Does the predictability

of the ratings differ with respect to subject or report variation .

Discr iminant Function Analysis--Subject Groups

The untrained group of students were included in the experiment to

provide data which would facilitate by contrast the interpretation of

results from the trained group . Given the two groups, their use of the

rating scales could be directly compared.

For this analysis, averages of each rating by all subjects were taken

over the 40 intelligence reports and examined to determine whether the

trained and untrained subjects could be discriminated by a linear

discriminant function based on the patterns of their respective uses of

the individual rating scales .

*Full data printouts of each analysis may be examined at ARI.

13 

-----—-.---— .-. ~~~~~— -- -- “. ~~~-,-

~~~~~~~~~ -.---
r . 

~~~~

- - - -. - -

~

. .

~~~~~ 

_____________

ARI TP 286

o o I•J 1.1 z Z

I— -

0- lix 0

~ I.’

0) ~~ WI t

- —

o 0

I I I I .0 _ I —

8

n ~ ~ V V

- U —

z

—1 -0 0 0

—

~~ 11 ~~~q

o 8 ~ I o t U, — V ~~~~ ~~~~~~~~ 0_ V

- —

0

~V I— o~~~ o-r ~~~~~ u _ U~~L__ U ,W WI

U, —

U~

ZO Ufl O ow z w

~~~ 

! k

~

~~ 

i

~

+;— ~~~ 

:‘~~~

!IJ<

’

~~~,

’

~ 

; ~

L ~

-[

~

L u

~

;

~ 

Ih;t~ J

~

i”:

~

HH 

~

!

~

L

~

- i-k. 

~~~ ~~~~

14
~

ARI TP 286

On the average, across all reports, if any given individual tended

to employ rating scales so as to produce a pattern of ratings like that

of other subjects in his designated group and different from that of

subjects in the contrasting group , discrimination would be achieved.

The test of discrimination is made on the basis of the a poeteriori

assignment of the subject to one or the other group on the basis of the

calculated regressions . The two groups discriminant case is equivalent

in all regards to a multiple regression analysis using a dummy

dichotomous variable as the criterion.

Table 3 details the 15 steps required to achieve 100% discrimination

between the two groups of subjects. Each step of the discriminant

analysis represents the addition of a new rating—scale variable to the

linear discriminant or regression equation.

The choice of scale variable at each successive step is made on the

basis of the maximum amount of discrimination which can be achieved

from the data supplied. Thus, at the first step , the average rating—scale

judgments of the two groups for the message quality PRE CARIOUS—IMPRECARIOUS

provided the greatest amount of discrimination between the two groups.

At this step , 21 of the 34 untrained subjects were correctly classified as members of that group and 18 of the 21 trained subjects were

correctly classified as members of the trained group . At step 2, the

next scale quality, ANALYZABLE—UNANALYZABLE , discriminatively moved

three additional subjects from misclassification as trained subjects to

the correct classification of untrained . However , five of the previously correctly classified trained subjects were lost to misclassification

as untrained . However , by the time the regression equation contained

the 15 rating scales shown in Table 3, all subjects had been correctly

classified .

On the basis of these variables, a consistent picture emerged of the

difference between these groups . The trained analysts judged the

reports with much less alarm and as more routine than did the untrained

analyst ., at the same time making finer judgments of the form of the

report . The trained analyst was better able to analyze the cont.utt and

import of the reports and make better jud~tments of threat potential and

accuracy . Additionally, there was much wider variation in the average

judgments of the accuracy and truthfulness of these reports among the

trained subjects, indicating a greater degree of discrimination.

15 

~~~~~~~~~~ -- -
ARI TP 286

C) H U) ~~~ 

‘

~~~ ~~ CU 

0 C ~~ CU LC\ .-

~~ 0 

~C) H r 1 C) H 0 

0 is •

4.) 

H is >~ I I I 

~1-10 i-I

Ce C

C

~ CU 7. 

“0 H 0 

~

0 N -

~~~ ~~ 

~C\ a)

U~ (\J -U- 0 -

~~ lf\ .r ~~ 1f\ ~~~- ~~ 

.

-

~ CC) 

z

~ 1 

S ‘0 C -,-~ ‘~~Ce Ce ~~~C 1i\ ’

~ O\C 

~t ~~~- 

H~~~~ - L N .

) ~~~

C) 

~~

. H ~~ $4

$4 ~ 0 C) N 

-

~ ._t CU _zJ 

\.O 10 \( 

UJ re\ 

-

4.) 

a) 

15 -

C ~ -zi- CU CU le\ CU Lr~~

c’J CU 

-~

~

- (U 

~ J -t (Ci o.

‘O 15

C ‘0 N 0\ 0 

k.O Th 1C\ 

~~

\ -f 

~0 J ’ \ 

~~ - H c-- 

N Ce a) 10 -Zt IC\ Lr~ zf — t 10 01 -zr ‘7 LC\ 10 

.: 0 ‘0

C a)

-o ..~ -.4 

C

a) .0 -.4

C CS

-.4 ~ C H Lr’\ N 10 N 0 N N 

CO CL’ IC) 

~J

’. If ’ N 

~1~’ 5) 

$4

CS C Ce 10, 1— 

~t 10 C) CC) N 

C- lf\ 

-

~ CC

-- 

If’ CC) 

~ \ .0 

1~

)

$4 a) . . . . . . - . . - 4.) Z IC) 0) (U 10 CU If’ CU CU CC\ Ur CU z~

- CU is 4-I 5 5 0 5

5) -.4 a) 11) E ‘0 -U C

a) is Ce

5)

‘0 

5 CS C

c5 ‘5) C a) a) ‘ C 5 4-)

00 4-1 Ce

‘0 0) $4 5 .-.4 4.) 

—

C a) S ril 4-I -~ 4.) 14.1

-.4 0 - ‘ H ‘0 Ce 0 Z ‘e H $4 a) S 1.4 1-4 i-I 15 U

0’ >‘ 

O~

~

-1 -1 rfj I-I 

C i—I ~ ‘.3 

~ 

z o

-. 0 0) 14-I ‘-4 Ce: H ‘S 4-) $4 0 N HZ El a)

H I ~ 

‘~

hD

~

HDIUH ‘0

- 

C I ~~4 .

~~ ~~~ 

.~4 CC) is Ce 

a)

a) S 5)

~- 4 5 

o

.

0 $1 (50 ) 

5 15 5)5) 0)

El 

‘-4 .0 Lf\ (C) a) “0 

PC) a) N CC) IC) 

~-t 

~D H CU CU H $4 . 

-

CS S pc~ ~~ 

CC) (Ci H (C) CU H CU 10 H 5 1-’ 

w US 150) 4-1

W.0 ‘-44

5 

.~4

‘0 14 (5 C 0. 15 a) H 04 10 ~~ (C) “0 ~

— aT) 

O-

~ 0 H 

(\J CC\ 0 .,4 1-I H H H H H H 

~ ‘.35

(5 .0

i6
___ __ __

ARI TP 286

There are implications for training in the differences between

groups . The trained and untrained groups were discriminably different

in their use of the rating scales , and the pattern of these differences

is consistent with that which should result from an effective training

course in intelligence analysis. However, as will be seen below , the

rating scales that were prominent in the discriminant function were

generally not those which are an important part of the subjects ’ judgment

space . These results alone do not provide a sufficient basis either

for the evaluation of the 96B training course or for changes in

instruction relative to the evaluation of intelligence data. However ,

the differences noted suggest areas where current training does have an

impact. Thus, the trained subjects were much more aware of the significance of messages (or the lack of significance), presumably through

their exposure to a number of practical exercises where they had to deal

with such messages in the context of tactical situations . However, this

awareness may reflect amount of experience more than amount of intelligence training per se. The trained subjects also were different

(although not necessarily better) in their use of the standard Accuracy

rating.

Factor Structure of the 50 Basic Scales

Average values for ratings by the trained , untrained, and combined

groups on each quality scale were obtained for each report. These means

factor , on the average , cannot be used to predict the size of its

ratings with respect to other factors . These means can be considered to

be the best overall assessment of each of the reports with respect to

each characteristic . The structure of the patterns of similar scale

usage derived from these averages provided an assessment of the implicit

conceptual space within which each of the subject groups evaluated the

reports , reflecting the resolving power of the analysts and the qualities

to which they are responsive . The conceptual structure can be compared

to the discrimination and bandwidth capabilities of a special purpose

radio receiver. Since the qualities were chosen to span the range of all

abstract potential qualities of intelligence reports , the resultant

structure can be said to indicate the selective tuning which characterizes the analyst receiver.

Principal components factor analysis was the specific technique

used to describe the underlying dimensions in the judgments . Subsequent

varimax rotation of the structure used in these analyses had the effect

of relocating the factor dimensions in the judgmental space so as to

spread the factor variance more evenly across the separate factors. This

technique also maximizes the within—factor variance and discards small,

specific , or error factors . Each separate dimension must conform to the

constraint ,, after rotation , that it be -perpendicular to every other

dimension . That is, the resultant factor space represents a cartesian

space of independent dimensions . Variation within one factor is independent of variation in every other factor; the judgmental ratings of a

variable with respect to one factor , on the average , cannot be used to

predict the size of its ratings with respect to other factors .

.

~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~
r 

- - __

ARI TP 286

Table 4 displays the mean values, for the three subject groups , of

the 50 qualities used as the variables in these analyses . Scales 1

through 46 vary from 1 through 7, the Reliability and Accuracy ratings

(scales 47 and 48) vary between 1 and 6, and the two Global ratings

(scales 49 and 50) vary from 0 through 1.00

Untrained Subjects. Table 5 displays the scale factor structure for

untrained subjects. The “factor coefficients” listed are the “loadings”

of the scales on their respective factors . The absolute values of the

loadings reflect the relative importance of the scales as representatives

of the underlying judgment dimension being used by the subjects .

All scales have some loading on all factors; listing of scales by

factors as in Table 5 involves a somewhat arbitrary cutoff at the point

where it appears that any additional scales, because of their

relatively low loadings on the factor, would contribute little to art

understanding of the underlying jud gment dimension represented by the

factor.

Some scales are “reflected” to be parallel with other scales on a

factor; thus a score of 1 on the False—True scale may be represented as

a 7 on the True—False scale, a score of 2 as a 5 on the reflected scale,

and so on. This procedure minimizes the number of minus signs,

simplifies interpretation, and in no way affects the meaning of the

results.

Note that the Global Validity scales have not been reflected ; a high

number on these scales represents a rating of high Global Validity,

while a low number on the traditional Accuracy and Reliability scales

represents high Accuracy or Reliability.

The four rotated factors of the structure in Table 5 account for

852 of the total report variance in ratings. Factor I accounts for

54% of the variance, consists of scales representing judgments of

accuracy, reliability , and truth , and is labeled ACCURACY .

Factor II accounts for 21% of the variance and involves the threat

potential and tactical relevance of reports. Factor II include.

aspects of the subjects’ evaluations of the repor ts dealing with the

magnitude and import of the developing tactical situation and is

labeled RELEVANCE.

Factor It is very similar to factors that have emerged in other

research involving quality judgments of action stimuli. Therein, this

factor represented a combination of the activity and potency of the

stimuli and was labeled Dynamism.

18

- -

~~~~~~~ - ---- - -. -~~ - --- --~~~~~~~ -—~~~~~~—~~~~~ - --
ARI TP 286

Table 14- . Group Means and Standard Deviations of

Combined Reports, by Rating Scale

(Page 1 of 2)

Untrained Trained Combined

Scale Mean SD Mean SD Mean SD

1 (Active) 2.84 0.68 2.58 0.67 2.74 0.63

2 (False) 5.45 0.57 5.71 0.83 5.55 0.63

3 (Untimely) 5.09 0.56 5.33 0.50 5.18 0.50

4 (Specific) 5.00 0.55 3.06 0.67 5.02 0.52

5 (Precarious) 5.51 0.37 4.05 0.29 3.72 0.29

6 (Extraordinary) 5.79 0.54 4.35 0.60 4.01 0.52

7 (Intelligible) 2.77 0.59 2.41 0.40 2.63 0.34

8 (Infeasible) 5.21 0.1i~3 ~.6i o.6i 5.36 0.44

9 (Factual) 2.98 0.46 2.67 0.64 2.86 0.48

10 (Unstable) 4 .72 0.50 14 .72 0.59 4.72 0.14-8

11 (Direct) 2.95 0.48 2.93 0.46 2.94 0.14-3

12 (Variable) 14 .67 0.54 5.12 0.88 4.814- 0.62

13 (Understandable) 2.77 O. L~14- 2.614- 0.53 2.72 0.41

14 (Likely) 2.67 0.14-8 2.43 0.66 2.58 0.50

15 (Massive) 3.28 0.70 5.25 0.58 3.26 0.62

i6 (Contradicted) 4.75 0.44 14- .6~ O. li-8 4.70 0.1i~

l

17 (Dependable) 2.~ 5 0.65 2.64 0.96 2.76 0.73

18 (Useless) 5.17 0.57 5.53 0.46 5.51 0.14-8

19 (Inert) 4.o8 o.~6 4.02 0.33 4.05 0.28

20 (Inappropriate) 5.00 0.144 5.23 0.43 5.09 0.58

2]. (Truthful) 3.79 o.6i 2.65 0.83 2.7~ 0.66

22 (Unhazardous) 4.74 0.47 14~.14-8 0.38 4.614 0.39

23 (Expect-sd) 5.91 0.37 5.52 0.46 3.76 0.37

24 (Uniinp lied) 4.li-8 0.29 4.37 0.34 4 .414- 0.25

25 (Heavy) 5.58 0.76 3.85 0.47 3.68 o.6i

26 (Possible) 2.14-2 0.li2 2.10 0.48 2.30 0.40

27 (Observed) 2.96 0.87 2.93 0.99 2.95 0.87

28 (Garbled) 5.30 o.4~ 5.24 0.49 5.28 0.14-3

19
— -.

-- ~ --—

ARI TP 236

Table 4

(Page 2 of 2)

Untrained Trained Combined

Scale Mean SD Mean SD Mean SD

29 (Faultless) 3.39 0.51 3.14-2 0.63 3.1i~

0 0.51

30 (Widespread) 5.87 0.69 14 .15 0.57 3.98 0.62

31 (Accurate) 2.86 0.57 2.69 0.74 2.80 0.60

52 (Uninterrupted) 4.37 0.37 4.31 0.41

~ 4.35 0.28

33 (Relevant) 2.87 0.49 2.44 0.51 2.71 0.14-6

34 (Insecure) 4.34 0.43 4 .63 0.50 4.14- 5 0.41

55 (Analyzable) 2 .75 0.149 2.26 0.42 2.56 0.42

36 (Risky) 5.33 0.56 ~~~ 0.46 3.53 0.48

37 ( Fragmented) 4 .4- 2 0.44 14.55 0.48 4.47 0.40

38 (Acceptable) 2.78 0.52 2.64 0.66 2.73 0.53

59 (Many) 5.58 0.73 5.

~~3 0.72 3.68 0.~

’)

40 (Confirmed) 3.)i o.66 3.68 0.93 3.49 0.73

4-i (Changing) 3 . ?9 0.31 3.76 0.36 3.814- 0.27

42 (Large Scale) 5.57 0.87 3.87 0.81 3.68 0.82

14-3 (Improbable) 5.13 0.4-7 5.43 0.66 5.25 0.49

44 (Extraneous) 4.58 0.33 5.31i- 0.46 4.87 0.314-

45 (Consistent) 3.24 0.14-6 3.15 0.14-1) 3.12 0.14-i

4-6 (Dangerous) 3.28 0.50 3.514- 0.39 3.38 0.14-2

47 (Reliability) 2.53 0.57 2.48 1.52 2.51 0.89

48 (Accuracy) 2.47 0.50 2.70 1.09 2.56 0.69

49 (Global-i) 0.74 0.08 0.80 0.14 0.77 0.10

50 (Global-2) 0.75 0.11 0.81 0.14 0.77 0.11

~~~~~~~~ ihe I.b.I for the bipolar scales indicet u the iow end 0$ the 1-7 scale.

20

~
- - - - -- -~~~~~ - - -

ARI TP 286

— I’

1J

U

-

-4(5 U I. ,

IJ

I ~

:

111 r14

‘-I L&J 9 (5

U # Z u .

I H~~~~~~~~~~~~~~~~~~~~~ 

I ~ . il 

~~~~~~~~~~~~~~~~~~~~ U I c I —

(

~ ~~~~~~~~~~~~ 

S I

III

I I I -

~

!I1I

~1iIIiIj ~

J

~

Fii

21

—---4
~~~~~--.— -

~~~~~~ - -~~~~~~~~~~ -- - -=---

_

~~~---- - -

ARI TP 286

Factor III accounts for 7% of the variance and seems to reflect a

judgment about applicability of message content, a determination whether

the content could be used directly or required drawing some inference.

This factor was tentatively labeled DIRECTNESS. Factor IV is a relatively

unimportant specific factor, accounting for only 3% of the variance, and

will be omitted in subsequent discussion .

Trained Subjects. Table 6 presents the rotated structure for the

trained subjects. The structure accounts for 73% of the total report

rating variance. The distribution of the variance across the four

factors is more uniform than that found for untrained subjects , implying

that the trained subjects were using finer discriminations in their

judgments of the attributes of the 40 reports.

Whereas the untrained subjects shoved a strong tendency to confine

their evaluations to judgments of the accuracy of the reports, the

trained subjects found other discriminable qualities on which to base

their judgments. As before, the most crucial judgmental dimension was

ACCURACY ; RELEVANCE slipped down in importance from second to a close

third.

The third factor extracted is best labeled DIRECTNESS because of the

scales contributing to it (Expected , Ordinary , Feasible) , although for

trained subjects it includes INTELLIGIBILITY AND USEFULNESS as correlated attributes . Nonetheless , the factor is essentially similar to

that found for the untrained subjects.

Factor IV does not provide the basis for straightforward interpretation. Although the inference—contributing determinant of IMPLIED has

the greatest influence on this factor, the miscellaneous nature of other

qualities such as specificity and fragmentation makes it difficult to

assign an appropriate label. Since it contributes only 6% to the total

structure and is heterogeneous in nature, it was dropped from

consideration.

Combined Subject Groups. Table 7 presents the composite structure

for the total subject sample. Since the separate group structures are

highly similar in factor content and prominence , this structure reflects

the cotmnon perceptual space within which analysts in general can be

expected to perform.

Synthesis of scales contribu ting to the fac tors conf irmed ACCURACY

as an appropriate label for Factor I, RELEVANCE for Factor II , and

DIRECTNESS for Factor III. A fourth factor contributing only 3% to the

variance was not considered further.

22 

~~~~~~~~~ ——-j - - - -—-~~~~ -~~~----—-- ----- - - —-- ---
- - - - — -- —-- --— — —-~~~~~~~~-- ----- --~~

—.- - - --

ARI TP 286

2

S — 2

-

I >

I ’

€ 2

U

4)

4) -

In

~~~~

(5

1.4

(-4 

~~t 2 -

1

~~~~ - ~

~~ 

I 

~ 

I I -

.

~~0 I

U I ~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~

1! Ii li ii

4)

I

(-4

2

I 

~ I !

i

~~

i I u i1i i j ~

!I

~

23

—-— --

~~

-- ---- - -- ~~~~~ - -—- -------- - - -—— - --— - - -—-- —
-

~~~~~- - - - - -~

- - - --

ARt TP 286

~

D tg~ in I—.

U) U

p.

o

1.1

0 :

~~- s I s • S

~~~W c ~ -

U ~ I c

~ .21 ~

- O- - I

2 ~ E

U) U

• ~ ~~~~~~~~ -

.

~

o

C-

, -

~~

1 ’ ..

4) ~

- S Ii U IIJ0

uJ $ llJ < - ii S (I) Z • • ~

~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~

~ 

-

~~ ~ I I ~ I I S

0 

~~~~~~~~~ ~~~~~ .

C ~~~~~~~

I a 

~ 

J

.1-i S I

I 1 - I

~

iii !i !

~

i

~

i I ~ I I I ‘ IJ.2 I —

L ~ 

I 

~ 

ii 

~ 

h 

~ ~ 

I i

24 ~

---

~~

- - -- — - ---- - -

~~~~~~~~~~
ARI TP 286

The dimensions of the factor structure represent indenendent aspects

of the judgmental process involved in intelligence analysis . They are

the critical underlying determinants of the judgment space in which the

analyst makes his discriminations among the message he is asked to

evaluate .

Neither the Reliability nor the Accuracy rating scales as currently

employed adequately captures the dimensionality of this space . Both

these rating procedures assess only the first axis of that space——

ACCURACY; cther scales are more representative of the underlying ACCURACY

dimension . Considerably greater discriminating variability in the

reports is left entirely unassessed by current measures .

Although the judgment space for the two groups of subjects is nearly

identical in composition , this similarity does not imply that the two

groups evaluated the reports identically . It is the structural space

within which those evaluations were made that is the same .

Thus, both groups considered important the information qualities of

accuracy , relevance , and directness . Specific evaluations of the

quality of a given message by the two groups are , however , discriminably

different , as has been established .

The situation is analogous to that of two observers who can see the

same visual stimulus but who differ as to the significance or interpretation of that stimulus. No matter how much training an observer is

given , he cannot make accurate discriminations among stimulus objects

to which his perceptual makeup is insensitive .

The significance of these findings concerning the scale factor

structure is that the analyst comes to his task with an implicit

perceptual structure . To be maximally effective , training instructors

must understand and capitalize upon that implicit structure . Findings

of the present analysis strongly suggest that this structure is not

modified by the Army Intelligence School training procedures .

In fact , what is known of such implicit perceptual structures

indicates that it would be extreme ly difficult to modify that structure .

What is required is that the training use the underlying conceptual

structure to best advantage .

Factor Structure of Intelligence Report s

For the analyses of the structure of the report space , the mean

subject data matrices of the preceding analyses were transposed so that

the pattern of intercorrelations among reports could be examined with

respect to scale means. Analyses of the report groupings , as

determined by the analysts’ judgments , reflect the degree to which the

subjects’ judgment space served to differentiate the intelligence

report8.

25 

----- -- - —-- - - —.- - - - - -- .—----

~~~~~ -- -- - - _ _
- --

ARI TP 286

The analyses represent a form of internal validity test of the scale

structure obtained. If this grouping of reports , this taxonomY, is

judged meaningful, significance of the scale structure from which derived

would be confirmed. The report structure was included for this purpose.

Table 8 shows the report means for the three subject groups across

all scales . All the me ans are substantially equivalent , and the variance

about those means is substantially higher than that found for the scale

means of Table 4. The explanation is that these analyses have taken

averages over what is already known to be a factorially diverse and

independent set of scales . The averages displayed reflect only the

average positional biases .

Lack of discritninability in the report averages , however , does not

indicate sameness among the reports with respect to individual scale

ratings to be factored. On the contrary , the high variances about these

means indicate considerable differences in scale ratings from report to

report .

Analyses have a distinct mathematical dependency on those previously

done. The report structures necessarily cannot contain greater

discriminability than emerges from analyses of the scale structure .

Technically, that is, the rank of the report structure matrix is

determined by the rank of the judgment space , which has been found not

to exceed 3. The reports grouped together on the factors described

below are combined in terms of a common variance on ACCURACY , RELEVANCE

and DIRECTNESS. Later displays make this picture clearer by modeling

the report distribution within the perceptual space.

Tables 9 through 11 detail the report groupings by factors that

emerge from these analyses. Structures for the three groups of untrained,

trained , and combined subjects are quite similar, and discussion is

focused on the combined structures shown in Table 11.

However , some differences between the groups were noted : The

trained subjects used only a two—dimensional structure to group the

messages. The report structure for the trained subjects In Table 10

shows that all messages deriving from friendly and reliable sources have

high loadings on Factor I, and all messages based on POW, civilian , and

deserter sources have high loadings on Factor II.

Considering the fact that ACCURACY is the most prominent feature of

the subjects’ judgment space , and considering that Accuracy and Reliabi].— ity are intertwined , it is not surprising that the subjects’ primary

distinction between reports depends on the source, i.e.-, reliability and

accuracy of the reports . ITowever, it is somewhat puzzling to find no

systematic grouping of reports on the basis of RELEVANCE or DIRECTNESS .

The structure for the combined subjects shows a three—factor space which

is apparently due to one discrimination on the basis of ACCURACY and a

second on the basi of RELEVANCE .

26
-- ___ ______ -

ARI TP 286

a\ ~

(\ C4~

\ ~

0 -m o ~

o a- -

~~ -~~ N- CO \0 0 c~ ~~ 4- 0 N- H C’\ ~~ -a- ~ \ -a- K~~ H -a- CO H OJ H C— ai Oj G\ N\ tr\

ID 0 HO H H H H H H H OH H H OH H OH H S

>: ~

‘\ r4\ ~~ r(\ CC) CC) N~\ ~

(

~

\ IC) CC) tC\ r(\ CC) CC) CC) CC) CC) CC) CC)

U) -

~ a- a- U\ N- CC) 0 0 N- LI) CC) C- LI) a- PC) N- LI) C— 0 CY\ 4-’ 0 -~~‘ CC) 0 LI) 

~

a- O~i ‘

-0 -a- 0 ‘-0 0) CC) CC\ H C’) CL.) CC) 0 4- CC) U) .

o U 0 H H H H H H H H H OH H H OH H H H H 0.

U “.4

CS S LI) N- C~

J 0 ‘.0 N- -

~~ 0 LI) If) CC) LI) CL..) a) a) ‘-0 a- 0) 4- H I- ’ CS CC ‘-0 0) ‘.0 if) ‘.0 -a- N- N- 4- N- LI) LI) (I) N- LI) LI) ‘

~

0 LI) LI)

ID .

Z IC) p4~\ PC) CC) ~~ N~\ C(~

) t4~

) p4W) ~4~\ C4~

\ CC) ~

4~\ pC) CC) re~ CC) pc-s

CA

U

0.) CC) ~

C) II) N- N- H CC) 0.) PC) 4- 0 ~~ H ‘.0 0) 0) LI) ~~ IC)

0 CC) 0 1

~ a- CC) ~

C) - - p~~~ ~~ H ~~ ‘H N ~~ H C) pC) p()

— ~~ U) .

CS 4) 0 H OH H H H H H H OH H H OH H OH H S 0

~ “-4

I. S LI) a- LI) ‘C) H ‘ 1.) -C 0 0 CO 0) H CU N- 0 CC CI.) CL..) CC- 0) -a- iS N- ‘-0 CC Li” -

~

a- —a- -

~ lr’~ N- 4- N- LI) a- CO ‘

-r tr’~ -a- PC) 4- CS 5 .

~ Z CC) CC’ PC) CC ‘Th tC) CC) IC) CC) tC) rC) NTh CC) CC) IC) CC) CC) CC) CC) CC) U

‘4-4

o 1

~ U) 0 H CU CC) 4- LI) CO ~ - CC C) H 0.) rC) -a- IC\ CO N- CO 0) 0 5 0. C)) C)) Cu C’.) C’.) 0.) C)) CL..) (

~IJ NTh CL~

) PC) PC) pC~ pC) rC) CC) CC) CC) a- o 2)

-‘-4

1

~

IS

CO CO -

~ 

0) CC N- CO ~~ 0) CU 4- H CU 0) 0 0 0 C).) ~~ ~~ ‘-s Cl 0 CC ~

C) rC) H CC) 0 ~

C) 0 CI) ~C) - a- C’ii CU ‘U U)

‘U U H H OH H OH H H H H H H H H OH H H H S CS -‘-4

5 ‘C) IC) ‘.0 m ~ ‘ C) LI) N- CC) CU 4- CX) CC) CC) ‘-C H 0 H 0 0 CS 0 CS - , ‘

-0 N- C C )

~~~ N ‘CC ‘-0 ‘- -0 II) If) LI) I) ~

-O ~

i) N- LI) 4- CO N- I-’ U U U) ~~ CC) PC) CC) CC) piTh C) rC) CC) CC) CC) C-Th CC) CC CC) PC CC) CC) pC) CC) PC)

‘U

CS 

o -

~ -a- N- H LI) If) CC) ‘0 H H N- ‘ CO H 0 LI) Q 0) CC) U) 0 -

~~ - C’ CU LI) 0-) 0 N- ~~

\ II\ 4- 4- -a- r~~ ‘0 C).) 0 ‘C) LI) 0 CC) S U) .

CS ‘U H H H H H H H H H H H H H H -C H H H H H U 4)

z S

‘-4 0. (55 ~~ C)

~~~~ ’ -

~~ H lI) (\j -$ ~

C )CU H CC C’.) (7’

~~ CO CIC 0 CI.) CU N- 5 (5 LI) ‘CC -

‘ C N- Ct ‘.0 ‘-0 CO LI) If) LI) LI) LI) CC) N- ‘.0 II) N- N- o .

I.. Z CC) PC) PC) CC) PC) CC) PC) Pr) Cr) pC~ CC) PC) PC) CC) CC) CC) PC) PC) CC) PC)

‘.0 0 CC) LI) N- 0 ()J 0 CC) LI) H C— CC ~~ N- ()J .-a- 0 0 ‘U 0 CU -a- CC) H 000 H C).) C C0 CU ‘

~~~ H 0)CC) LI) CC) 4) 4) U) -

— (5 H H OH H H H H H H H H OH 00 H H H H .

0 .4

(-4 5 C~J CC C)) 0 C-i Cs) N- ~ \ PC) CU ‘-0 \0 PC) P4) II) N- -a- CC) p C) ‘

-0 4) CC) ‘CC a) C- - ‘.0 C)) CO ‘.0 ‘.0 LI) II) LI) LI) N- CC) ‘.0 4- pe\ LI) ‘

~ ) (5 4) .

~ Z PC) PC) PC) PC CC) P4Th IC) PC) CC) PC) PC) CC) IC) CC) PC) IC) CC) p4) IC) PC)

‘4

‘a

0 H CL.) ~~‘ -a- L~

”

~ ‘

~~~ N- CC) 0) 0 H Cs.) CC) 4- LI) ‘.0 ~

— cc) a- p. H H H H H H H H ~

-1 H .2)

27
- - ---- - -

~ - ---~~- - ——----- - - ---—----- -‘-- -- -- - - -- — -—‘---- - - - -- —------------~~~;-

-— -‘ 

—I,,

ARI TP ~

U U .2 .2 .2 3

j ! ; 

~ 

!

~ 

11 

~ ~ 

I II

‘U4) C) I’ L à

-

~~ C

.2 2 .2 — 3 .2

C) I

Cd,

• .. I.- U U 1

~ U

5 3 $ 3 4 4-i

o

3 F ‘a

Z ~~~~~- ( - 2 C/) C C ~ !

~

‘a ) ~ t i, - o ~ ~ ‘

~~~ 4 ) ~~

~

~ 

i

~

!

~ ~ ~

~

1i H 

~ ~

! II 

~

i ~~~~~~ :

‘

‘ 

~ ~

h !

~ ffl ~

! 4

4 .2 2 .2 .2 $ 2 .2 .2 .2 3 2; .2 F 3 4 .2 f F

~ U Ill ii U I! I it 

~ 

t

~ 

J j

~ 

Ii I I JI U ‘H U !~ ! II1

~ ~

n ~

4 i

~ ~ -~~~~~ 

~ ii ~‘ hi ~ t’ It ’ j I 

~ ~ ~ 

H

~ 

i

~ ~ 

!

~ ~i’ t I 

~ 

hI Jf ~ ifi ~

i

~

i 

~

il ul ;li ~

i 1it 11 iiii 

~ ~ 

ill ~

li lih J

~ ~ 

il

~

i .2 .2 3 .2 3 * .2 4 .2 .2 a a

28

____ 

- - -

~~~~~~~~~~~~~ - - - -
ARITP 286

Table 10. Report Factor Structure : Trained Subjects

TRAINEOSUBJECTS

F..Ia, 1 I96%I F..Io, lII?b~ I

Me.g. P4...*a. ~~~~~~~~~~~ F..l.. Co.4lm~~ t U. p Nooib.. 16.1.00 P1.4.. ~~~~~~ Co.tt.n.oI

7 401 161 (3m .0~

.20,I0 I.o.u — los no.4, 01 96 6 (00 4, I

n’ 0’. 90011k da aI10~ by, (3.-nan (50, 96

EiJ,06naS., .4.0 n.e. to. .5.001 Amy “-0.04 10 C.lngna aid Vs., .2.004.41 I’S

0,100 1 a .th *.

~~~ note the 9km.

S MI oo,U 294. SD.. p0_nI ~~~•I dy ~

.th .97

5.00.04 1,0161 on,. snm,o 001 I I ,oon4. .4 16 40, nI 0.. ,.po.~ .0_ny 0.00,9. by, 600 ,n.o U)

s.Id9.Y 100. ,ai.n.d ,o 212m 9(3 D.o 00040 p.o .0 ln.th.m.ong

.0,00k O’0104 m o d .0_nV 900d

37 A., P.,~~ ,gcoons..nm -am.i, .nW,y 0.0.. 96

0100.09 .o..ltl 0,l On ..dm9. aid ~~~~~~~~ 21 1(211, nI flagI I94e,1 0090 ,10,. Hg 269. PG D., 93

m.dn.ll.ni ya.dl -L 7,,.. I.. , nov04 dos. 10 hosI I_n.

30 (124. tot R.g. So.0mo-t s-mop .0_ny h.._no~ 56 3) V Co.~. ..po,ll POW ..y, Hq 352d A,IiI0.y 9091 93

00161 S~~ le~ do..og Sos., OP 1.1,0,0 , p.oSSy ow 0eIt.n .

(4 290, Do A.lp 1a00.,,.o.my ,.n,id .0.0,11 CP 96 35 44. nt Do. lOW T*n ..po.10 Gain.., .

~ .L, e, 7$

224,1, PA 9. p0 0.1 op -- LownbOoI .10011 00940000 0

m.,6.16I,o~ ,.

~

d. .1 2 16oa~~

.. ~

04 op

24 112$. InS Hog. .po.t. 00105 wS des Marl 94 o.tto to. no 1,0.10

09001

23 1 (00’ InC Rags ‘.90,,~ .n_ny .00109. 0000 m.

~~mI 75

(0 1104. InS H ag, .opo. o 1.004 .o.o.y n$to 90 900 P000lOd I0 00ml ,o,nad 16.490 5.110

0,000,5.00*01,

34 44, 1,1 1).. S.nnm. ’y POW , rn_n, 6.00. d.n,.0.~

0 14

9 1964. InS Itoil OP p.n,o.,oCd ‘Iporl .4.10(9 99 -o *0, 00 000 90,(o(9 ‘0040 to. Inq

los. lop. Took.

5 29th nt 0.. Pt I.ai•, 01,0(11 cnds..fl 11001*0 5* 4 1100, 101 .290, 1900110 G.nnan p0001 on h.nMy 96 10-600 000010’* .10 (0001(3.10(01044,1.,.

o0. of ,00,001d 10 000 004 0 .0.01100 ,009. ,nn ., bnn~ 101

and .s.Il

79 VIII Gap, Soan~

na.-y . POW ~._no 1169, Par.., D.n 96

II A.. On. 50.016, 0.101. ~

4lod 40 — 11001,600009 .56 ,non~

n9.ooll Ia Tn... none, 01*011 Me,d. &tb.anq

26 790, lot Don P90 1._n 10000 001,0’ 100101 60,0,6. 4* 32 2511. tho,uon So.n.n.y no..ny wtII.,y haplo, .95 0,009 oSmalI011000. . III00, SO 000p., and *0 -

*9. 9.010. Ol In (song mOn-sq .9010.0.0 * ,noo,n, we., (roe, 6.ibo~

30 III ‘.gl. . 290. laP Di, ,-,.5.aI. oO,.a..d a..ln 94 I 1069. InS 94$ lOporl. 101.1 * .p..h.. Pompony IT

01010101 OnSI .0*0 (US 0000*,,. rV HOOt 100Sf O.ndo.t .

33 (069, lot Don sommay I.I06sng poltl 109.0 1 95

aiat..M 0,010. mo..owni,. 9. .. ..we,y 9S1004~ r atnd. hs..b.ni ‘fl I-tI 4. SOSOv ballS-on

‘.

00.-I.

2 (19$. lOP ,Ip.1 I. I16o0.n 90001 on tnnnthy aId. .96

h ymn, I

~c~ou01 100*0 loIn Sn .0604 06.0

25 112* 1019.9.00 p...onnS 1,_nd 1I,.n a_ny .94

040016.~

19 (000. nI Do Soooo.,y onoll fl,, foOd donn9 .63

*.ky.,a. 0_OS’S SO_nV 100(01 S0010t. gIS,1.0

.010( pottol..

20 112$. InS ItogI OP p.0.0001 M.d Iwo ai.ny .013

9.04.9.49.. depI9.,

22 1109,161 94$ 0940(5 PIn ~9 5600!

~~~,dSl 93

b51 1 0744 -09(0

04 49, In I 0.. (a_nb 0,_nv oS,0n16 ~~“011 92

opp~0l0 I) tot 9.90 dnoog a_.Ig. 06...., .

90_n

21 260, 0., A.101.y 1.01.0naiO ,o.0. ~~~n 001y .63

p.th 0104$ w.* no.1_n aba *0.019 a n0-.U

.2_n o~~~00at 00-I

30 56. *0_nd 0.. Ma_n_n _n.n ne~~~i p.s.s.5* 43

~~~~~ 10050400 004 Mm a5.,.tI ’r $. lpd 60o

29
ARI TP .

~~u

8 9 2 0 0 2 8 8 8 (3

00 n

t - 

~

~ 1i ~ ~ ~

-. ~~~ 

~ ~ o - 01 

~ ~~~~~ ~ ~~~

~ !‘~ ~ ~

P !; !J ~ 

!

~ ~

i i

~ ~

j ~~~ ~~~

:

2 

;

~ 

:

~ 

0 0 * 0

p

~~

~ ~~ h hj

lid ~ t 

~~ 

Ut ~ 

I! ~

~ 

ch 

~ ~ 

£~ P 22 ~~~& ~~~~

~Lg

~

h ~li - ~~ ~~ 0 J O 2 5 0 5 0

b-I

‘-I

(3 (3 (3 01 9 8 ( 3 (3 8 ~~~9 9 2 2

~

0

I-a

I

11 11 11 11 ‘1 HI I ~ 

LI ! 11 il1 tI

~ ‘In U ~

i Ii F0 

~ 

1

~ ~ 

1 

~ I II

~

1 p 

~ ~ ~ !

~

l Ii 

~ 

H ii ~ 

Ii ~~~~ ‘

‘ 

~ ~

tjJ u ‘

~

J ~1 I I i ti ;ll th ~

I 

~ 

1111 8 5 (3 I 8 (3 ~

9 0 (3 92 8 *

30
— - - 

~

II

ARt TP 286

The three report factors for the combined subject groups (Table 11)

account for 78% of the total scale variance in the ratings of these

reports. Factor III accounts for more of this total variance than Factor

II. The display reverses the normal ordering of factors from greater to

least total variance to indicate the order of the unrotated factors . This is, Factor II was more prominent than Factor III in the unrotated

structure but accounted for less variance than Factor III after rotation .

The summarized reports shown in Table 11 reveal that the three factors

have relatively clear identifications . Factor I groups those reports

containing UNUSUAL, LARGE SCALE or THREATENING ACTIVITY . Factor II is

largely determined by DESERTER and CIVILIAN REPORTS . Factor III groups

reports of PATROLS, TRANSPORT and SMALL ARMS ACTIVITY . As a taxonomy of

these reports , the structure is reasonably coherent and significant .

Supplementary Analyses

Several additional ways of examining the data help clarify relationships among the rating scales and among the reports in the message set.

These analyses and their conclusions are discu8sed briefly here and

presented in more detail in Appendixes B—I ).

As previously established, the subjects used three basic factors or

concepts in judging the 40 ms~ssages——ACCURACY , RELEVANCE , AN)) DIRECTNESS.

The subjects’ approach to the message ratings led to two or three clusters

of related messages , messages which were given not only a similar pattern

of ratings but which had definite similarities of message source and

content .

Another way to describe relationships among the messages as rated by

the subjects is to assign factor scores to the messages and to locate

the messages in the three—dimensional space defined by the rating factors.

The factor score for a given report was obtained by taking a weighted sum

of all the ratings on that report , the weights being proportional to the

loadings on the factor.

Thus, in computing the ACCURACY factor score for a report , the ratings

given on the True—False scale, the Reliable—Variable scale , etc., were

given high weights . Once the location of each report in the judgment

space had been determined , it was possible to examine, fo r example, the

types of messages identified by high ACCURACY , low RELEVANCE , and low

DIRECTNESS.

A detailed examination of the report factor scores , and figures

showing the geometric relationships among reports , is presented in

Appendix B. The ACCURACY factor clearly played a major role in the

subjects’ evaluation of the reports; the RELEVANCE end DIRECTNESS factors

were of lesser value.

31 

- ------—--------- - - - - -----

~~- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -
__ - 0—~~~~—-—— -- -

~~~-~~~~------~~~~~~~~~~~~~~~~~~ ~~~~~~~ -.

ARI TP 286

This result was not surprising considering the relative percentages

of rating variance accounted for by the three factors : ACCURACY is

clearly the dominant factor , and reports would be expected to group

principally on the basis of their ACCURACY factor scores . However, the

composite factor scores do provide meaningful groupings of the messages.

If single rating scales are developed to represent these factors , ratings

on the new scales may be valuable for structuring or organizing data to

facilitate its use.

Appendix C presents a more detailed analysis of the ACCURACY dimension and of the relationships among the ratings on this dimension and the

traditional Accuracy and Reliability ratings .

From examination of the inter-rating correlation in Appendix C , it

is clear that the Accuracy and Reliability ratings are highly related ,

despite the doctrinal definition of these ratings as independent .

Apparently , for the 40 messages used in the experiment , Accuracy and

Reliability were indeed highly related . That is, all accurate reports

might have come from reliable sources , and subjects may have shown good

judgment in tying the two closely together.

However , examination of the messages (Appendix A) and discussion of

some messages (Appendix B) show instances where unreliable sources

produced accurate information and reliable sources provided inaccurate

information . This fact , combined with the data in Appendix C, strongly

suggests that subjects were using the Reliability rating inappropriately

by tying it too closely to their judgment of Accuracy .

In addition , it appears that the Global Validity ratings may provide

a more straightforward representation of subjects ’ estimates of information accuracy than the standard Accuracy rating does .

The multiple regression analyses in Appendix I) support this

conclusion but suggest that the trained subjects indeed may have been

differentiating between Accuracy and Reliability . Trained subjects

apparently based their judgments of Accuracy in pert on the Reliability

of the source, but they did not seem to judge Reliability on the basis

of the Accuracy of the information in a given message.

The supplementary analyses, then, confirmed the finding of a

dominant concept of information ACCURACY . However , relationships of the

standard ratings of Accuracy and Reliability to this concept and to each

other remain somewhat unclear. It is apparent that these standard

ratings do not capture the full meaning of the ACCURACY dimension as

derived from the subject data. To explore more fully the interrelation—

ships among the various ratings , the following steps need to be taken:

1. Replicate the present experiment using a different sample of

analysts with more extensive military experience and Intelligence

training.

32

______ __ 

~~~~~~~ - - -

~~~~~~~~~~ --- -—- --— - - - 
r

~~ ~~~~~~ -

~~~----~~--- --—--~~~~~~~~~~ -~~~~~-- -—--~~-- — -—~~- - - -

~~~~~~- - ~~00 -.- --—~~~- - - -

ARI TP 286

2. Replicate the present experiment using a different sample of

messages .

3. Develop rating scales specifically designed to capture the

meaning of the principal judgment factors identified in the present

experiment.

The data from these experiments should verify the stability and

generality of judgment factors appropriate for the evaluation of combat

intelligence data. In addition , some insight would be obtained about the

potential usefulness of new rating scales specifically designed to tap

these factors .

SUMMARY AND CONCLUSIONS

The present experiment was an exploration of the structure of combat

intelligence data as seen through the eyes of intelligence analysts. A

selection of 40 messages was used as representative of intelligence

information received at division level.

Ratings were requested from 34 enlisted personnel just beginning an

intelligence analyst course at the U.S. Army Intelligence Center and

School. Mi additional 21 enlisted personnel, just finishing the course ,

were asked for similar ratings. Members of both groups were relatively

inexperienced in intelligence analysis . Attributes of the reports to be

rated included specific qualities of content as well as the traditional

Accuracy and Reliability ratings and ratings of the overall validity or

credibility of the report .

The purpose was twofold , (1) to ascertain the underlying conceptual

structure employed by the subjects in evaluating typical messages

containing intelligence information , and (2) to assess the role of the

traditionally employed reliability and accuracy ratings prescribed for

the intelligence analyst.

With respect to the conceptual structure , it was found that analysts

had an implicit judgmental structure within which they evaluated the

attributes , qualities , or characteristics of the reports upon which they

were called to pass judgment .

The dimensions of this judgmental space, determined by multivariate

analyses, were Accuracy , Relevance , and Directness . In combination ,

these independent dimensions accounted for more than 82% of the total

variance of the combined evaluations of the repor t sample as made by the

55 analysts (Accuracy, 57%; Relevance, 19%; Directness , 6% ).

33 

— - - ----- --- —~~~~~~ - -- -

~~~~ - - - - ~~~~~~~ - - ‘ ---0-- — --- ---——- - ~~~ -— ---
____ ___ __— ~

_

~ o.~~~~~~~ .. — —— •-- ———— — ___ ____ _ _0.__ - —— — -—

ARI TP 286

With regard to the traditionally used Accuracy and Reliability scales , the evidence of prior studies was verified that the scales are, in fact , not independent . Findings also indicated that the judgments of Accuracy

as traditionally measured were significantly contaminated by apparently

unfounded judgments of source reliability, particularly on the part of

the trained analysts.

For the untrained subjects, however, the two measurements were

completely interchanged. In contrast, a specially devised Global scale

of report validity was found less contaminated and to have generally

better measurement validity for report Accuracy.

These findings imply that scales for evaluating quality of intelligence data could be derived to provide a more effective vehicle for

rating of data quality and transmission of ratings .

Scales more closely paralleling the analysts’ conceptual framework

would permit a more natural judgment process, and the ratings given

would more accurately reflect the rater ’s feelings about the data being

rated . The use of 0—100 rather than 1-6 scales might also contribute to

more effective communication between the person rating the information

and the analyst or decisionmaker utilizing the information .

Before development of new scales and procedures, however , the results

obtained in the present experiment need to be validated by (1) replicating the study using a sample of trained intelligence officers with

combat experience and (2) repeating the study using a different message

Set and associated scenario.

34 
ARt TP 286

APPENDIX A

MESSAGES RATED BY SUBJECTS

BATTLE of the BULGE

- - 28th Infan try Division

Selected Messages for 10—15 December 1944

Alignment of Units in VIII Corps

from North to South

106 Infantry Division with 4th Cavalry

Regiment attached on North Flank

‘Rth Infantry Division

ll~th Infantry Regiment

110th Infantry Regiment

109th Infantry Regiment

9th Armored Division (60th Armored Infantry)

Battalion of Combat Command A on line

4th Infantry Division

A-l 

~~

-_——-—-~~~~~~~~ ~~~
___ 

____ _ --- ~~~-- -- -- -

ARt TP 286

Message Hr 1

10 December 1944 28th Infantry Division

1000 hours

The 3d Battalion , 109th Infantry Regiment reports that a deserter

captured in the vicinity of coordinates 9241 at 0540 hours is from the

1st Battalion , 915th Regiment , 352d Volksgrenadier Division. The

prisoner of war stated his company is located in the vicinity of BIES))ORF

(9743).

Message Hr 2

10 December 1944 28th Infantry Division

1500 hours

3d Battalion , 110th Infantry Regiment reports it observed an 8—man

enemy patrol in the vicinity of coordinates 851581 at 1115 hours . Enemy

withdrew eastward and crossed river in a rubber boat which was found in

the vicinity of 854584.

Message Nr 3

10 December 1944 28th Infantry Division

1200 hours

3d Battalion, 110th Infantry Regiment reports that between 0900—1130

hours it sighted 15 horse—drawn vehicles and 1 staff car between coordinates 890570 and 897562.

Message Nr 1

11 December 1944 28th Infantry Division

1500 hours

3d Battalion, 110th Infantry Regiment reports seeing a 6—man German

patrol walking towards the Our River in the vicinity of coordinates

871541 at 1100 hours. Observers did not fire , but followed the enemy

patrol. Tracks led to the river in the vicinity of coordinates 875543.

They could not determine whether the enemy patrol used a boat to re—cross

the river or waded. They kept watch in the area until 1300 hours, but

did not see any more activity . The current in the river was swift and

the water was level with the banks. Observers returned to their unit at

1400 hours.
— -

ARI TP 286

Message Nr 2

11 December 1944 28th Infantry Division

1200 hours

28th Infantry Division Interrogation of Prisoner of War Team reports

reliable civilians state that as of 10 December:

1. NIEDERGECKLER (9753) and GEICHLINGEN (9451) were full of

German soldiers .

2. Many mines were being laid in the vicinity of BAULER (9052).

Message Nr 3

11 December 1944 28th Infantry Division

1200 hours

106th Infantry Division reports a deserter who surrendered 9 December

says he is from Headquarters 15th Army which recently moved from the

Rotterdam area to a location in the Cologne—Bonn—Dueren area. Rumor at

his Headquarters is that Field Marshal von Rundstedt has ordered

withdrawal to the East bank of the Rhine because shortage of reserves

will make defense west of the Rhine no longer feasible after the first

of the year .

Message Nr 4

11 December 1944 28th Infantry Division

2400 hours

4th Infantry Division reports enemy rifle and pistol fire in northern

part of sector (north of ECHTERNACH) at 1400 hours. About 50 rounds

each of 80 mm and 120 mm mortar fire were received during the day .

Message Nr 5

11 December 1944 28th Infantry Division

2400 hours

Summary of Enemy activity along 28th Infantry Division front for 11

December.

1. All units reported a quiet day .

2. Scattered rifle , pistol and mortar fire were received all along

the front.

3. 11 rounds of light artillery fire were also received .

A4 

- 

j ~~~~~~~- --—~~~- - ~~~~~~~--~~~~~~~- - ~~~~~~--~~~~~~- - - - --- -~~~~~-- —- ~~~ --~~
ARI TP 286

Message Nr 1

12 December 1944 28th Infantry Division

1000 hours

3d Battalion, 109th Infantry Regiment reports observation post

personnel sighted three(3) TIGER Tanks in the vicinity of Road Junction

at coordinates 952458 south of NIEDERSGEGEN at 0905 hours .

Message Nr 2

12 December 1944 28th Infantry Division

1200 hours

110th Infantry Regiment reports.

1. Four(4) horse—drawn vehicles observed moving southeast in the

vicinity of 88875 at 1019 hours .

2. One(1) half_track*, several horse—drawn vehicles carrying troops

and one(l) horse—drawn artillery piece (weapon) observed moving southwest in the vicinity 930602 at 1100 hours .

*V~~icle with wheels in front and tank—like tracks in back .

Message Hr 3

12 December 1944 28th Infantry Division

1400 hours

VIII Corps G2 Air reports pilots of heavy bombers returning from mission

sighted a 40—truck convoy moving southeast from WAXWEILER (0167) to

BITBURG (1454) at 1100 hours .

Message Hr 4

12 December 1944 28th Infantry Division

2400 hours

112th Infantry Regiment reports :

1. Occasional rifle and pistol fire and flares all along front

during hours of darkness .

2. Sound of several vehicles heard in the vicinity of ROSCHEID

(8970) at 1950 hours.

3. Single enemy planes flew over regimental area at 2015 and 2113

hours.

A-S 

—~~~~~~~~~~~~~~ --- - -

~~~~~~~~~ - - ~~--—~~~~- - - ___ ~~~~~~~~ - - - ---- --~ - --.---- --~~ - - -- - - - - - —-- --- -—
--- -

~~~~~~~~~~~~~~ -~~~~~~~~~~~~~ -

ARI TP 286

Message Nr 5

12 December 1944

2000 hours

VIII Corps relayed following message from First Army : “Train movements

indicate that the buildup of enemy forces on the western slope of the

MOSELLE Valley continues . The Gross Deutschland Division* has again

been reported in the area by a prisoner of war . A conservative estimate

would place at least two(2) Volksgrenadier Divisions and one(l) Panzer

Division in the enemy ’s rear area opposite VIII Corps . These units

maintaining have not used their radios .”

*A Panzer (armored) division.

Message Nr 1

13 December 1944 28th Infantry Division

0400 hours

28th Division Artillery reports that at 0300 hours the 229th Field

Artillery Battalion detected a two(2) man enemy patrol in the vicinity

of its CP. The patrol withdrew when fired on.

Message Nr 2

13 December 1944 28th Infantry Division

1550 hours

3d Battalion , 110th Infantry Regiment reports that Patrol Hr. 6 saw

little activity in the vicinity of ROTH (915482) at 2000 hours on 12

December.

Message Nr 3

13 December 1944 28th Infantry Division

2100 hours

4th Infantry Division reports that a deserter from the 2d Battalion,

316th Regiment, 212th Volkagrenadier Division stated :

a. The 230th Regiment of the Division was on line for about two

weeks but is now in reserve.

A-6

~

- - - - -
ARI TP 286

b. App roximately 600 men on special types of duty in rear areas were

called back to the 316th and 423rd regiments of the 212th Volkegrenadier

Division by a secret order on 11 December. They were told to return to

their regiments immediately because they were needed for an upcoming

attack against the thinly held American line . These men for the most

part had been attending noncommissioned officer schools .

c. Morale of German Troops in his division is good to very good .

Message Nr 4

13 December 1944 28th Infantry Division

2400 hours

112th Infantry Regiment reports that troops along the front heard sounds

of enemy vehicles during the evening.

1. Tracked vehicles in the vicinity of coordinates 9272 at 2130

hours .

2. Trucks , motorcycles and possibly tanks in town of ROSCHEID(8970)

at 2200 hours .

3. Motors in the vicinity of coordinates 874693 from 2215—2245 hours.

Message Hr 5

13 December 1944 28th Infantry Division

2400 hours

106th Infantry Division Summary stated :

1. Indiscriminate rifle and pistol firing along entire front during

hours of darkness .

2. Unusual motor activity in the vicinity of SCHEII) (064963) and

BERG(908750) in the past 24 hours suggests presence of new troops in the

area.

3. Aggressive enemy patrols displayed increased activity along

entire front during past 24 hours.

Message Nr 6

13 December 1974 28th Infantry Division

2400 hours

VIII Corps Intelligence Summary reports that a message forwarded from

First Army at 2000 hours states that a prisoner of war says the 116th

Panzer Division has moved south to the Trier(2l29) area . Its rear

echelon including division headquarters left MLJENCREN—GLADBACK on 6

December. First Army notes this report seems to confirm a southward

movement earlier reported on the 116th Panzer Division.

A-7

___ — - ------ ~~~~ - - - ----~~ -

———-—
- - -

ARI TP 286

Message Hr 1

14 December 1944 28th Infantry Division

0600 hours

112th Infantry Regiment reports that observation post personnel heard

two (2) trucks on Hill at coordinates 878698*. Time sounds were heard

was 0405 hours.

*SE of ROSCHEID (8970) and opposite the boundary of the 2d and 3d

Battalion 112th Infantry .

Message Hr 2

14 December 1944 28th Infantry Division

0630 hours

2d Battalion , 112th Infantry Regiment reports wounded POW captured near

LIELER (838705) at 132345 hours while on patrol states he heard the

headquarters of the 26th Volksgrenadier Division moved up to ESCHFELD

(902692)—ROSCREID (8970) area on 5 December.

Message Nt 3

14 December 1944 28th Infantry Division

1000 hours

110th Infantry Regiment reports that vehicular movements have been heard

in the vicinity of coordinates 854680 at 0745, 0830 , and 0910 hours . Direction and volume undetermined .

Message Nr 4

14 December 1944 28th Infantry Division

1015 hours

1st Battalion , 110th Infantry Regiment reports that an enemy deserter

picked up at 132300 hours in TINTESMUEHLE , a small village at coordinates

850673 at the bend in the river East of HEINERSCHEID (821675) and SE of

KALBORN (840682) says he was born in STRASBOURG.* He stated that the

machine gun covering the ruins of the bridge at coordinates 850672 was

moved on 7 December to a better location 100 meters from its former

position where it can fire directly down the center line of the old

bridge.

*On west back of the Rhine, Southeast of the VIII Corps sector.

A-8

k. —— - - 
ART TP 286

Message Nr 5

14 December 1944 28th Infantry Division

1400 hours

112th Infantry Regiment reports that vehicular movements were heard again

behi nd hill at coordinates 878698 and vicinity 873687 at 1300 hour8 .

Message Nr 6

14 December 1944 28th Infantry Division

1200 hours

3d Battalion , 110th Infantry Regiment reported that a large formation of

Enemy Infantry was heard in GEMUEND (871557) at 102200 Dec~

ember.

Suspects assembly area for raid into 110th sector. Requests Corps

Artillery bring all possible fire on the area as soon as possible.

Message Nr 7

14 December 1944 28th Infantry Division

1600 hours

The 28th Division Interrogation of Prisoner of War Team reported that a

woman who crossed into Germany on 10 December was arrested and taken to

BITBTJRG(l353) . She escaped and walked back into Luxembourg where she

reported to police this morning . She is vouched for by the Luxemb ourg

police as thoroughly reliable . Her statement follows: “I saw many

horsedrawn vehl:les, pontoons , small boats , and other river—crossing

equipment coming from the direction of BITBIJRG and moving west throuç~

h

GEICHLINGEN (9451). In BITBtJRG , I overhead some military personnel

saying it had taken three weeks to get there from Italy . There were also

some troops in the town with gray uniforms with black collar patches .*

I also saw many artillery pieces , both horse—drawn and carried on

trucks .” -

~~

*Uniform of SS(Elite Nazi) troops .

Message Nr 8

14 December 1944 28th Infantry Division

1930 hours

A-9

_______ 

---~~~~ - --~~~~~~~~~~~~~~ ~~~ -- - -—---- -4
ARI TP 286

VIII Corps G2 Air reported that an Air Force visual reconnaissance

mission this afternoon made the following observations : 1. Train consisting of 50 small freight cars was moving south in

the vicinity of coordinates 2050. 2. 20 more small freight cars were on siding in the same general

area.

3. Considerable activity in the marshalling yard at TRIER(2129).

Message Nr 9

14 December 1944 28th Infantry Division

2000 hours

112th Infantry Regiment reports that observation post personnel heard

three(3) vehicles headed west at 1840 hours . It sounded as if they

were pulling heavy loads up a hill in the vicinity of coordinates 880696.

Message Nr 10

14 December 1944 28th Infantry Division

2130 hours

3d Battalion , 110th Infantry Regiment reports that a patrol found a dead German in the vicinity of coordinates 847594 at 2000 hours. Only item

of military value found was a map stamped 2d Company , 78th Infantry Regiment.* The map was DASBURG , Sheet 5902. Man appeared to have died

about 24 hours ago from a bullet wound .

*Thjs would indicate the map came from the 1st Battalion , 78th Infantry Regiment, 26th Volksgrenadier Division .

Message Nr 11

14 December 1944 28th Infantry Division

2400 hours

112th Infantry Summary stated that there was unusual intermittent

harassing small arms fire from 140001 hours until dawn. It started

again at 1810 hours . At 2000 hours there was a marked increase in

volume . At least one burst of automatic pistol or machine gun fire

was reported at different points along the front and extending to the

south of the 112th sector, at an average rate of one burst every five

minutes . This activity was still continuing at 142400 hours .

A- 10 
~

- -.. ~~- -

~~~ -~~

ARI TP 286

Message Mr 12

14 December 1944 28th Infantry Division

0600 hours

V Corps G2 Periodic Report for 13 December had a significant item under “Enemy Dispositions .” It stated that a prisoner of war from the 2d

Company, 942d Infantry Regiment , 353d Volksgrenadier Division who was • captured in the V Corps sector reported that the former Command Post

of the 353d Artillery Regiment while opposite the VIII Corps was in the

vicinity of BETTINGEN (0450). He stated that it is customary for

relieving units to occupy the same positions as tho8e of the relieved

unit. This points to the possibility that the same Command Post is now

occupied by the headquarter of the 352d Artillery Regiment.

Message Nr 13

14 December 1944 28th Infantry Division

2400 hours

28th Division Artillery Summary :

1. During the past week the enemy has kept his artillery well back

from the front .

2. No enemy firing has been reported at night.

3. No counterbattery* fire has been reported .

4. Enemy artillery activity for 14 December:

a. 11 rounds , reportedly all 75mm , were fired into the

sector of the 112th Infantry Regiment .

b. Approximately six(6) rocket projectiles believed fired from

near coordinates 950440 heard and seen at 1940 hours .

*Firing at our artillery

Message Mr 14

14 December 1944 28th Infantry Division

2400 hours

106th Infantry Division Summary :

1. Listening post personnel reported heavy increase in motor

vehicle movement particularly vic ORMONT(0892), SCHEID(0696), and

BRANDSCHEID(9881) before dawn and after dark . They heard sounds

suggesting unloading of troops .

2. Harassing fires very light.

3. At least three(3) Enemy patrols reported in 106th sector.

A-Il

--
ARI TP 286

4. Two(2) Prisoner of War, from 3d Company, 295th Regiment , 18th

Volksgrenadier Division report 1st Battalion , 295th Regiment in the

ORMONT area.

Message Hr 15

14 December 1944 28th Infantry Division

2400 hours

4th Infantry Division Summary stated that vehicular traffic was reported

on the 12th Infantry Regiment’s front during the evening . Otherwise it

was a quiet day .

(Note : 12th Infantry occupies the northern regimental sector of the 4th

Infantry Division from ECHTERNACH north to the 9th Armored Division’s

sector.)

Message Mr 1

15 December 1944 28th Infantry Division

0600 hours

4th Infantry Division Interrogation of Prisoner of War Team report states

that a German national from SAARBRUECKEN* was picked up while visiting

relatives vicinity Luxembourg City on 6 December. He claims he is an

employee of the German Railroad System. He further states that on

30 November the marshalling yards in ZWEIBRUECKEN* were tied up for six

hours when two carloads of explosives collided and exploded .

*Opposite Third Army zone to the southeast.

Message Hr 2

15 December 1944 28th Infantry Division

2400 hours

109 th , 110th, and 112th Infantry Regiments Summaries all indicate marked

increase of small arms harassing fire between 0400 and 0800 hours,

followed by a -very quiet afternoon and evening.

A- 12
- ~ - --

ART TP 286

Message Nr 3

15 December 1944 28th Infantry Division

2400 hours

28th Division Artillery Summary :

1. Observation Post 101 (964414) observers reported at 1040 hours

that enemy in front of them were equipped with overcoats for the first

time.

2. During the day when guards were changed, enemy soldiers double—

timed to and from their posts and did quite a bit of saluting not

observed before.

3. Vicinity 963436 and 958435 enemy did a lot of moving to and from

pillboxes .

4. There was also quite a bit of foot traffic vicinity 965439 all

day.

Message Nr 4

15 December 1944 28th Infantry Division

2400 hours

4th Infantry Division Summary states that two(2) POW from the 44th

Fortress Mechine Gun Battalion were captured at 995196 at 1830 hours . They said they were told two days ago that a Panzer Division and two(2)

Infantry Divisions were in the rear of their area and that those units

would soon launch an offensive of major proportions .

Message Mr 5

15 December 1944 28th Infantry Division

2400 hours

9th Armored Division Summary states :

1. Enemy outpost personnel were more in evidence than usual during

the day and appeared more confident .

2. Abrupt change of routine of personnel on the other side of the

river strongly suggests new troops may have arrived on the front.

A- 13 

~~~-- -- - -- ------- -- .--~~~~~~~~~ --~~~~~-.- ~~~~~~ - ~~~~ -- - - - - -
ARI TP 286

Message Mr 6

15 December 1944 28th Infantry Division

2400 hours

106th Infantry Division Summary:

1. Conside rable vehicular activity was heard in the vicinity of

coordinates 996806 at 0045 hours and all along the front from 1800

hours to the time of this report.

2. An enemy patrol consisting of 34 men was reported in ALLMIJTHUN

(040948) during the early morning .

3. Enemy observation planes were active over throughout the division sector early this evening .

4. A prisone r of war from the 2d Battalion 295th Regiment , 18th

Volksgrenadier Division who was captured this afternoon says his unit

arrived in this area on the afternoon of 14 December relieving an

unknown unit which was to be pulled out for an attack along the entire

front between the 17th and Christmas. POW had heard tha t SS* troops

were to be used and that they would employ infiltration tactics. He

further stated that in the area of his unit there were large generators

and LeO large sea rchlig ht reflectors .

*Elite Nazi

A- 14

I 

______________ ____________ 

~~~~~~~~~ — -~~~~~~~~~ - - 
-

~-— —-,-- -

~~~~~-- ----—-- ~~~~~~~~ 

--- 

- •

ART TP 286

APPENDIX B

REPORT FACTOR SCORES OF THE SCALE STRUCTURE

One way to depict the taxonomy resulting from the scale structure

obtained , as applied to these reports , is to display the factor scores

of the 40 reports in the calculated judgmental space .

Such displays have been produced by creating three dimensional

models (Figures Bl through B3). For each model , the three axes of

ACCURACY , RELEVANCE and DIRECTNESS form a coordinate system for

displaying the location of each report in the subjects ’ judgment space .

Locations of each report in this factor space are based on the linear

equation of the weighted combinations of each of the scale ratings , i.e.,

the factor scores of the reports calculated from their weighted scale

averages . For the purposes of the display , the factor scores have been

scaled relative to their maxima in each dimension.

Although this produces some distortion , the distortion is constant

in all dimensions. Thus , relativized , the report locations are merely

spread Out more than they would be if not so scaled , but their positions

relative to each other are as found in the analyses .

For purposes of the following discussion , the three dimensional

(continuous) structure of the scale factor space for combined subjects

is cut into eight discrete compartments (or octants). The reports

falling into each of the compartments of the space are then analyzed.

The three factorial dimensions of the judgmental space depicted by

Figure B3 for combined subjects can be envisioned as shown in Figure B4.

In Figure B4a, for example, the three dimensions are shown as intersecting axes with each dimension extending from the origin or 0—point

one unit measurement in both positive and negative directions . The

Accuracy and Relevance axes define a horizontal plane. Accuracy and

Directness dimensions define a vertical plane as do the Relevance and

Directness axes . If each of the dimensions of the structure is considered to have a positive (+) and a negative (—) end or pole , the octants

formed by the various combinations of the poles of the three dimensions

can be defined , in order , by the

+ + + poles of the Acct~racy , Relevance , and Directness factors

(Octant 1), by the + + — (Octant 2), + — + (Octant 3), + — —

(Octan t 4), — + + (Oct ant 5), — + — (Octant 6), — — + (Octant 7)

and — — — (Octant 8). The order of poles is Accuracy , Relevance,

and Directness .

The three planes intersect along lines passing through the point of

origin of the axes . Octant 1 Ia formed by the positive axes of all

B-i

— ---- --•- —-- —.•-
- • - -

MU TP 286

three dimensions , as shown in Figure B4b , i.e., it has cubic form and

lies in the upper far right hand corner of the judgmental space .

Octant 2 lies directly below Octant 1 and forms the lower far right

hand compartment of the judgmental space , as shown in Figure R4c. In

a similar manner, Octants 3—8 may be envisioned from their polar

definitions . Figure B4d shows Figure B3 subdivided into octants but

without the report positions being depicted .

There is high face (content) validity for the classification of each

of the reports contained within each octant as described below . The

reports falling within each octant are enumerated and their collective

identity summarized. In addition , a typical report is cited to provide an

appreciation for the type of report falling within that part of the

judgmental space identified by the octant specified.

Octant 1: + Accuracy + Relevance + Directness. Reports 10, 18, 19, 24

26 , 28 and 40 have values greater than zero on each of the three scalar

factors . The common characteristic of these reports is enemy activity

seen or heard by U. S. military forces , local friendly observers, and

POWs.

The reports came from subordinate units of the 28th Infantry

Division , including the Interrogation of Prisoner of War Team ; adjacent

divisions, and from local police units friendly to the Allied cause.

The activity consisted of movements of supplies , troops , artillery , and

river crossing equipment by trucks and horse—drawn vehicles ; small arms

firing and enemy reconnaissance in the form of patrols and observation

aircraft.

The troop movements varied in size from patrols of small numbers of

men to the entry into the area opposite the VilIth Corps of a whole

Panzer (Tank) division and the appearance of elite SS Troops . POW

reports indicated preparations for an enemy attack, and those of

friendly units described enemy actions as unusual and aggressive .

Report 40 (coordinate values of .17, .50 and .21) is among the

highest rated reports in this octant . The text of the report was as

follows :

15 December 1944 28th Infantry Division

24 hours

106th Infantry Division Summary:

1. Considerable vehicular activity was heard in the vicinity of

coordinates 996806 at 0045 hours and all along the front from 1800

hours to the time of this report.

2. An enemy patrol consisting of 34 men was reported in ALLMUTHIIN

(040948) during the early morning.

B-2 

~~~~~~~~~~~~~~~ -~~~~~~~~~~ - -- - -~~~-- -~~ - —~~~~~ - - - ---—- -- - . --~~~~~~~~ ----~~~~~~~~ __
________________________ -~~~~~- • - -~~~~-

~~~~~~~~ -

- - - 

-‘U’

ARI TP 286

3. Enemy observation planes were active overhead throughout the

division sector early this evening.

4. A prisoner of war from the 2d Battalion 295th Regiment , 18th

Volksgrenadier Division who was captured this afternoon says his unit

arrived in this area on the afternoon of 14 December relieving an unknown

unit which was to be pulled out for an attack along the entire front

between the 17th and Christmas . POW had heard that SS troops were to be

used and that they would employ infiltration tactics . He further stated

that in the area of his unit there were large generators and 40 large

searchlight reflectors .

Octant 2: + Accuracy + Relevance — Directness. Reports 11, 17, 27,

30, 33, 37 and 39 are located in this octant.

The common element of these reports is enemy changes all along the

FEBA of the VII Corps from the 106th Infantry Division on the North of

the 28th Infantry Division through the 28th and continuing into the 9th

Armored Division on the South of the 28th, together with sizeable truck

convoy and train movements carrying supplies and possib le troops from

the North into the enemy rear areas opposite the VIII Corps.

Four of the seven reports in this octant came from units other than

the 28th Infantry Division . There was one each from the 106th Division

and 9th Armored Division and two from VIII Corps G2.

Specifics other than truck convoy and train movements included

increased small arms fire during hours of darkness and increased motor

vehicle movements , both opposite the northern part of the VIII Corps

sector , and obvious changes in enemy front line units such as increased

activity, better discipline , better clothing for winter weather , greater

self—confidence , and a different schedule of daily operations .

The text of Report 37, as representative of the octant , was as

follows:

15 December 1944 28th Infantry Division

2400 Hours

28th Division Artillery Summary :

1. Observation Post 101 (964414) observers reported at 1040 hours

the enemy in front of them were equipped with overcoats for the first • time.

2. During the day when guards were changed , enemy soldiers double—

timed to and from their posts and did quite a bit of saluting not

observed before .

3. Vicinity 936436 and 958435 enemy did a lot of moving to and from

piliboxes .

B—3 

- -—•--•- - .—- - —--- --- ---a
- ~~~~~~~~~~~~~~~~~~~

ARI TP 286

4. There was also quite a bit of foot traffic vicinity 965439 all

day .

Octant 3: + Accuracy — Relevance + Directness. Reports 2, 7, 9, and 20

are in this octant. These reports in the 3d Octant are characterized by

friendly front line troops seeing or hearing minor enemy activity such

as small enemy patrols , small numbers of tanks and trucks and small

arms, mortar and artillery fire .

Most analysts apparently failed to note that Report 9 is a false

report , it being impossible to see the location at which the tanks were

reported from the observation post whose personnel made the report .

In addition , the analysts failed to appreciate the fact that the

appearance of TIGER tanks close to the front was unusual, and if they

believed the report true they should have perceived a relevant threat

since the opposing Volksgrenadier Divisions were not normally equipped

with any tanks, much less the heavy TIGERS .

Report 2 is representative :

10 December 1944 28th Infantry Division

1500 Hours

3d Battalion , 110th Infantry Regiment reports it observed an 8—man

enemy patrol in the vicinity of coordinates 851581 at 1115 hours.

Enemy withdrew eastward and crossed river in a rubber boat which was

found in the vicinity of 854584.

Octant 4: + Accuracy — Relevance — Directness. Reports 4, 8, 12 , 14,

29 , 32, 34 , and 36 are located in this octant. The commonality of these

reports is that the analyst—subjects regarded them as accurate but of

such minor nature along the 28th and 4th Division fronts as not to

constitute a relevant threat .

The activity consisted of small patrols which were not aggressive,

light small arms and artillery fire , occasional aerial reconnaissance ,

one dead German with a map, and a report that enemy artillery had been

kept well back from the front during the week and had not even been

firing at usual targets .

Subjects apparently analyzed the reports in much the same way as the

actual ones did in December 1944.

They missed the point that the dead Ge rman soldier (Report 29) had

a map of the area in which he was found but that his unit was last

reported much farther north , which should have led thee to question

whether his unit had moved south , which, in turn, could have been

associated with other southward movements farther back in the enemy

rear areas reported by aer ial observers .

B—4

~

---- - - - - - - -
ARI TP 286 -

In addition , the subjects were apparently taken in by the artillery

positions and relatively small amount of enemy artillery fire in the

past week (Report 32).

This is understandable since it fitted the general Allied impression

of the time that the Germans were in a defensive posture and short of

ammunition . However , in this case it was part of German deception built

on a past reality and our continuing belief in it abetted by German

efforts to preserve the impression .

The text of Report 14 is typical :

13 December 1944 28th Infantry Division

0400 Hours

28th Division Artillery reports that at 0300 hours the 229th Field

Artillery Battalion detected a two (2) man enemy patrol in the vicinity

of its CP. The patrol withdrew when fired on.

Octant 5: — Accuracy + Relevance + Directness. Reports 5, 16, 21, 31,

and 38 are located in this octant . Analysts correctly evaluated the

relevant and dangerous nature of these reports , if true : increased

numbers of troops in towns near the front lines, recall of men from rear

area schools to front line units for an attack ; forward moves of a

division CP; units shifting along the front , and three new reserve

divisions in the enemy rear area.

However, they placed no credence in POW reports , nor even in those

of civilians characterized by local police as reliable. Additionally,

many may have recognized Report 21 as really false. If they checked

their maps and knew their tactics they realized that the place reported

as the new location of the 26th VG Division Command Post was entirely

too close to the FEBA.

The text of Report 16 is representative :

13 December 1944 28th infantry Division

2100 Hours

4th Infantry Division reports that a deserter from the 2d Battalion,

• 316th Regiment, 212th Volkegrenadier Division stated :

a. The 320th Regiment of the Division was on line for about two

• weeks but is now in reserve . -

B—5 
______________________ - - - - -

- - -~~~~

ARI TP 286 -

b. Approximately 600 men on special types of duty in rear areas

were called back to the 316th and 423rd Regiments of the 212th

Volksgrenadier Division by a secret order on 11 December. They

were told to return to their regiments immediately because they

were needed for an upcoming attack against the thinly held

American line. These men for the most part had been attending

non—commissioned officer schools .

c. Morale of German troops in his division is good to very good .

Octant 6: — Accuracy + Relevance — Directness. Reports 6, 13, and 22

are within this octant . The rationale for this grouping is hard to

explain . Two reports (#6 and #13) are of major importance. The third

report (#22) is rather minor and seems out of place.

Move of an Army Headquarters southward (Report #6) is significant

even though the withdrawal across the Rhine appears unlikely fo r the

reasons stated. The relevant threat was recognized if the report were

true . The POW was probably not considered reliable , and rightly the

withdrawal across the Rhine was not believed.

Report #13 also represents a relevant threat since it indicates a

buildup of at least three divisions opposite VIII Corps and First Army.

However, at least part of the report originated with a POW, and from the

position on the west bank of the Moselle, the three divisions could be

used against the north flank of the Third Army to the south as well as

against First Army .

Apparently , for that reason , the subjects did not believe the

report accurate or infer much from it.

On the other hand , Report #22 is a straightforward front line

report of hearing enemy vehicles three times in the early morning ,

although conditions were such that the direction or size of the movement

could not be determined . One would have expected such a report to fall

into Octants 3 ( + — + ) or 4 ( + — — ) rather than into Octant 6

( - + - ).

The text of Report 6 which is a false message added to the data set

was:

11 December 1944 28th Infantry Division

1200 Hours

106th Infantry Division reports a deserter who surrendered 9 December

says he is from Headquarters 15th Army which recently moved from the

Rotterdam area to a location in the Cologne—Bonn—Dueren area. Rumor at

his Headquarters is that Field Marshal von Rundstedt has ordered with—

drawal to the East bank of the Rhine because shortage of reserves will

make defense west of the Rhine no longer feasible after the first of the

year.

B—6 

~~~~~~~~~~ --- —~~~~~~ -~~~~~~ - -~~~~~~ -- ~- -~~~~----~~~~~~~~-~~~ -- --- -- -~~~~~ - -~~~~~~~~~ -~~~~~~~~~~~
Octant 7: — Accuracy — Relevance + Directness. Reports 1 and 23 are

located in this octant. Both of these reports originated with deserters , whom the subjects apparently do not trust, and the information reported

consists of minor locations of only local interest.

Report #23 is false because the deserter says he was born in

Strasbourg which is in the Franco—German border area and all such soldiers

had been screened out of the German front line units during November and

December 1944 to prevent just such desertions .

The test of Report 23 was as follows :

14 December 1944 28th Infantry Division

1015 Hours

1st Battalion , 110th Infantry Regiment reports that an enemy deserter

picked up at 132300 hours in TINTESMUEHLE, a small village at coordinates

850673 at the bend in the river east of HEINERSCHEID (821675) and SE of

KALBORN (840682) says he was born in STRASBOIJRG .* He stated that the

machine gun covering the ruins of the bridge at coordinates 850672 was

moved on 7 December to a better location 100 meters from its former

position where it can fire directly down the c’~nter line of the old

bridge .

Octant 8: — Accuracy — Relevance — Directness. Reports 3, 15, and 35

are located in this octant. In content , all of these messages have information which is of minor importance or irrelevant to the 28th Division .

In two Out of three cases , it is one day to 15 days late.

Report #15 is false, since a patrol from the 3d Battalion , 110th

Infantry Regiment would not have been that far south without an explanation having been given , and Report #35 is false since it is unlikely

that the Germans would have let a railroad employee visit relatives

behind Allied lines just before the counteroffensive .

It appears that the subjects correctly analyzed these reports except

for #3, with respect to Accuracy . They had no apparent basis for calling

the report from the 3d Battalion , 110th Infantry Regiment inaccurate.

The text of Report 35 , still another fak e message , was as follows :

15 December 1944 28th Infantry Division

0600 Hours

4th Infantry Division Interrogation of Prisoner of War Team report

states that a German national from SAARBRUECKEN** was picked up while

*On West Bank of the Rhine, Southeast of the VII Corps sector.

**Opposjte Third Army Zone to the Southeast.

B-i

____ —- - -
—

ARI TP 286

visiting relatives vicinity Luxembourg City on 6 December. He claims he

is an employee of the German Railroad System. He further states that on

30 November the marshalling yards in ZWEIBRUECKEN* were tied up for six

hours when two carloads of explosives collided and exploded .

To this point , it has been established that the judgmental space

employed by the subjects in evaluating the sample of intelligence reports

is coherent and produces groupings of the reports which reflect the

nature of the content of the judgmental spaces .

I

Although the examination of the reports in each of the octants of

the combined judgment space may have raised some questions about the

value of the RELEVANCE and DIRECTNESS dimensions, it is clear that the

ACCURACY dimension plays a major role in the subjects’ evaluation of the

reports.

*Opposite Third Army Zone to the Southeast.

I

-

t

B—8 

~~ --~~~~~-— ——-- --~~~-_-. - ------~ -.“--- • --------
~

. - _ -- ——- -

~~~~~

- 

-

~~~----

ARI TP 286

• ~

1

~

S e

S

I

S

S —.

~ 0

C 0 

~~~ In

U

SS~ N I )D$U(J

V

* C

________________ 4 C C

U

a

a

.•

•

~ .0

0

0

‘I

th

w

U.

~~

- 9
_______ ~

_ - -

~- - -~~~~~—--- 

ARI TP ~~~~

0

0

%‘ 

—

S ~ —•.

;

~

- - , - -

* - 

-• 

•,

.

_ .

~ ‘-

- 

S - 

~

‘ -~~

C ~~ , I; 

- - - . -

— 01. - - 0 0

.

• • * - a —

U

~~

1 U,

‘~

,

0 C

U

_________ 0

/

I. ,

C —

0

0.

N

S

C

U8-10 
~

_

~~- - . -

ARI TP ~~~~~

_-

~ 

__*.*

~ 

~~~~~~~~~~~~~~~~

:%.

~~~~

. 1

2I

~~ i__• ~~~~~~~~~~~~~~~~~ 

- -

~~~~~T_ ~~_

— 

-

~ - 

J_

/ 

I

B-fl

___________ _____ - - - —-—-4
ARI TP 286

-, 

,

~

/

/

/

/

~

/ 

0 *CC000CV 

—l 

,

/

/

/

/

/

~

/

/~

/ 0 aC..

000cv

F*Ci0 100 OMINSION$ 00 4JOGMINTAL V*CI OCT*NT I 0

~ AIOO~ INTA L ~~AC(

~

/!

~~T/EE~~~~ _ 

OCT*~ T 000 ~~0GM0NT*L VACI / 

- 

/

/ 

~ / ml oct*~ii 00 tNt F*CT000 L b

~~

tis.o.m 00 ~J00Ntoflot ~~~~

Figure B•4. Factorial Dimensions and Octants of Judgmental Space
— _ ~ - —- --- -—--——--

~~~~~~~ - --- -- -—-—--- —--— - .

ARI TP 286

APPENDIX C

AN ALYSIS OF RELATIONSHIPS AMONG ACCURACY , RELIABILITY , AND OTHER SCALE S

The polar coordinate plots of Figures C—i through C—9 in which the

correlations of the Reliability (Scale 47), Accuracy (Scale 48)*, and

combined Global ratings (Scale 51) with all other report characteristics

are depi ct ed , are designed to display the patterns of zero—order

relationships for these particular scales .

Ea ch 5—degree radius of the plots , starting at zero degrees (center, top) through 2

~5 degrees , represents , in order , the 50 scales plus a

derived 51st scale which is the average of the two separate global

ratings (Nos. 49 and 50) . The center of the plot represents either

the Reliability tScale 47), the Accuracy (Scale 48), or Combined

(average) Global (Scale 51) ratings .

The length of the polar vector outward from the scale used as the

basis of comparison expresses the degree of relationship to that base

scale as indexed by the Pearson Product Moment correlation coefficient .

The farther the end of the scale vector from the center of the figure ,

the greater the relationship between the scales .

The plots display only correlations accounting for more than half of

the correlating variance . No vectors have been drawn for scales whose

correlation with the base scale is less than an absolute value of .70.

This value was arbitrarily chosen so as to select only those scales with

the greatest predictive relationships .

The correlations displayed in these plots have been derived by using

subject averages as entries and thus reflect the degree of relationship

between each scale with respect to the similarity of dissimilarity in the

ratings of the 40 reports.

Thus, a high correlation between two scales indicated that the

average subject of his particular group tended to rate the reports as

being similar with respect to those qualities .

For example, if Reports 1, 6, 10, and 30 were rated by the average

subject as being very accurate and these same reports were also rated

as being highly reliable , then the correlation between Accuracy and

Reliability would be high.

*See Figure 4 for the scales corresponding to the numbered identifications .

C—’

—---- —-- - - --- - - -

~~ -~~~~~~~~ ----- --- ------- --- ___________
~- _ _~~~~~~~~~~~~~~~~~~ ----

ARI TP 286

Reliability Relationships. Figures C— i through C—3 display the

relationships between the Reliability (Scale 47) and all other scales .

Inspection of the figures reveals that scales rating True—False

(Scale 2), Dependable—Undependable (Scale 17), Truthful—Deceptive (Scale

21), Accurate—Erroneous (Scale 31), and Accuracy (Scale 48)* all have

correlations in excess of .90 for the untrained subject group .

The trained subjects produced highest correlations with reliability

for the scales rating True—False (Scale 2), Dependable—Undependable

(Scale 17), Accuracy (Scale 48), Global—l (Scale 49), Global—2 (Scale

50), For the combined subjects , Reliability was most closely as8ociated

with the ratings of Dependable—Undependable and Accuracy .

Accuracy Relationships. Figures C—4 through C—6 display the relationships between Accuracy (Scale 48) ratings and all other scalar qualities .

Inspection of the figures indicates that Accuracy correlates beyond

.90 with True (Scale 2) (opposites henceforth dropped), Dependable

(Scale 17), Truthful (Scale 21), Faultless (Scale 29), Acceptable

(Scale 38), Reliability (Scale 47), and all Global scales (49, 50 and

51).

The trained subjects also disniaved highest relationships between

Accuracy and the above scales ~~~~~~~ Dependable , Faultless (Scale 29),

and Acceptable (Scale 38).

The combined groups produced highest relationships between Accuracy

and True , Dependable , Truthful , Faultless (Scale 29), Accurate (Scale

31) , Confirmed (Scale 40) , Reliability, and all Global ratings (Scales

49 , 50 , and 51).

Global Relationships. Figures C—7 through C— 9 display the Combined

G1o~

,al (Scale 51) relationships . The untrained subject group rated

True (Scale 2), Accuracy (Scale 48), Global—l (Scale 49) and Global—2

(Scale 50) as most closely associated with the Combined Global variable .

Obviously the high correlations between the separate and combined

global ratings are considered as instrument relationships and should be

discounted .

The trained subject group correlations indicate highest relationships

between the Combined Global score and True , Truthful , Accurate , Reliability , Accuracy and the separate Global ratings .

*Note that the intelligence ratings as traditionally employed are

referred to as Accuracy and Reliability . The quality scales, on the

other hand , are referred to as Accurate—Erroneous and Reliable—Variable.

See Figure 4.

C—2
—I’

ARI TP 286

The combined subject groups display highest relationships with True, Dependable, Truthful, Faultless, Accurate , Confirmed , Accuracy , and the

separate Globals .

Cluster Relationships. Figure C—b displays the cluster groupings

around the Accuracy , Reliability and Global variables of the scalar

qualities for each of the subject groupings .

For the purposes of this display , that scale having the highest

correlation with each of the prima ry variables is identified by the arrow

from it to the next variable .

The correlation matrix is then searched for that variable which has

the highest relationship with this second variable and so on until one

of the subsequent variables has its highest correlation with one of the

previously identified variables already displayed .

Such clusters can be interpreted as the most conservative ordering

of the scale relationships since, at each successive step , only the

highest correlation with each variable is considered.

All correlations of these clusters excepting that for the untrained

subjects with respect to the correlation between Scales 51 (Combined

Global) and 49 (Global—l) (its value being .91), are in fact , in excess

of .94.

These analyses of the zero—order relationships indicate quite clearly

that the Reliability and Accuracy ratings are highly correlated in both

the trained and untrained analyst groups . For the total subject group ,

the correlation is in excess of .95.

Reliability is being assessed by these subjects as if it were a

measure of the truthfulness , acceptability and accuracy of the intelligence message. The training of these subjects does not change the

pattern significantly .

Also , with respect to the traditionally employed scales Reliability

and Accuracy , Accuracy is the better single measure. The judged

Reliability of the message is being determined by the judged Accuracy

rather than vice versa.

To some estent , this result was inherent in the test given the

subjects . Accuracy is a one—time evaluation of each report , but

Reliability of source or agency , if determined correctly, can be built

up only over a period of time as the analyst works with the reports of

each source or agency and develops an appreciation of their reliability

from opportunities which arise to check on the information as reported .

This fact in itself indicates how dependent Reliability is on the

average Accuracy of reports from any given source or agency submitted

over a period of time .

C—3

__ ~~~ - - _ - - -

~~ - --------~~~ - - — -- - ~~ - _ _ - -- - - ---—-.
ARt TP 286

In the test situation, the subjects were initially forced to infer

Reliability from other qualities of each report or to make assumptions

about it based on the nature of the source or agency , e.g., patrols,

subordinate units G2s and S2s, the Divisional IPW tears, G2s of adjacent

divisions or higher headquarters, Air Corps pilots, friendly and enemy

civilians (except in those cases in which local police made an evalua—

tion), friendly local police, POIJa and deserters .

However, despite this difficulty, findings of this study correspond

to those of earlier studies of the traditional Accuracy and Reliability

Scales by indicating a high degree of inter—correlation in their use

(see Samet, 1975).

Accuracy ratings are apparently well assessed by the Global scale

ratings. Significantly , the Global Scale rating appears to be a purer

measure of the accuracy of the messages than is the standard Accuracy

measurement .

Examination of Figure C—1O indicates that the cluster chains for the

Global ratings do not extend beyond their own sub—components.

Although Reliability does correlate with the average Global ratings

in the combined , trained and untrained subject samples, the A—F Reliability ratings are not as highly correlated with the Global ratings as

they are with the 1—6 Accuracy judgments .

The conclusion is straightforward. The Reliability ratings, as taken

from this sample of subjeats with this set of messages, were nearly useless and the accuracy ratings are more purely and thus better measured

by the Global rating.

Considering that fewer instructions are required for use of the

Global scale compared to those required for use of the Accuracy scale,

the use of Global ratings instead of the present Reliability and Accuracy

ratings might improve the quality of the initial ratings of information

accuracy as well as subsequent understanding of the meaning of the

ratings (Same t, 1975).

C—4

— - - ~_

~1~~~
- ~~~~~~~~ .

ARI TP 286

w

000 I-

~

I- 

~ S -a .. ‘a. 0

~~~~~~~

~~ 

~~~~~~~~~~~~~~~~~~ 

~~~~~~~ ~~~~~ —g 

~~ 

~~

~~ ~~~~~~ \ \ 

~ 

L I I / ~~~ ~

‘- $ ~

b. ‘a ~ • — \ ‘ ~~~~~~~~~~~~~~~~~~~~~ / / / g

‘k~

. ~~~ \ \ .—k~~ \ I I 7

~ -~ ,? / / “

~ei , y/ \ *~~ \ 

~ 

( / 7

~~~/ /

~ \ ~~~~ 

.

“ N ‘

:K’

~ 

\ \ 1 / / / /

~ ‘ . / ~ ~ 4,J ~

x~

/ \ \\ \_ -

~

—f—-!---J I, / X ~~~~

7

3’

~~~~dS I ~

~fl8.i .1 ______________ O3131dI’

~

O3

____ ___ I

! ~~18VJ4333y

/ I U,

0.

Do 

I

/

5’ 

~~~~

3

~~~ . ‘~

3. ,

d

~

.

IC,

4

.2’ U UC~ 5 

- -
ARI TP 2/(

~

.00

- 

_

~~ 

— — —g

\ \ I / ~~~~~~~~~~~~~~

, ~~~~ 

%~~~ ~‘ ‘ / / ~~~

~~~~~~ 2’ 

\ \

~~ _ _ _~~~~~~

-

-

~~~~~~~ ~~~~~~~ /

/ ~

!

~~ 

N~~

\ 

- \ \ 

~ 

J

~ 

/ /

/~/ 

~

3flH ~

L ~

. L._....~~ ~~~~~~~~~~~~~~~~~~~~ ~~~~— 1

~~~~~~~~~~~~~~~~~~~~~~~~

I

//~ S 319Vid333y

-, - / I

-

_--

_iii_ _ _ 

/ 

,

I

e

3i

~~

1

~~~

03

/ 

~~~ 

~~~~~~~~~

a U

t

U

C-6

~ 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - -
r - - - - - - -

ARI TP 286

-J

a

‘ 

‘

~~~ —e—!—g---.

~

~~ % ~~~~~~ \ ~

i ~~~~~ ~~4f -? ~~ 

~~~ ~ 

\ I I I / S~~~~O 

~

$ ~~ . ~~. \ \ 1

~

—_-- —L / ,

~ 

..

~ ~~

%

~ /

2’ 

\ \

~~\~

-\ /

~~

/

~~

/ / S~ ~~

)

.

. \ —\ \ I I / tN. / /

5\

N 

~~ 

\ \

~~ 

/ / / /\/ \

/ 

~~~~~~ ~~~~~~~ ~ 

J / //// 7 ~~~~ ~~~~ ~

_______ ~ a3La1av~

o3 ~

I 3

~~ V-Ld333y

~~~~~~~~~~~~~~~~~~~~~~~~~ 5 0.

~ 

*l

~~~,3

/ •0 15

,

5

~ 44~ 4f4

,

_: ~

f

<~ .!

\.

U

I,

UC-7 

-~~~~~~~~~~~~~~~~~~ ~~~~ - - ~~ -

~~~~~~~~~~~~ -——~~~~~~~~~~~~~~~~~~~~
— - - -

ARI TP 286

1

2

’

4 Z

~ ‘. ~~ d

~, ‘

~~~ ~~~_~~—!—!—~~~~~~~~~~~ t ~

L - ‘ !

~~~~~~ I ~~~~ ~~ 0

a)) ~

a ~~~~~ \ / ~~~~~~ ~~~~ ‘ I I ~~~~~ ~

p ~~

•

.

~ . 

~~ \ \ ‘ / / / g o

~ e ~~ ‘ \ \ \ __—1--- ~~~~~~~ I I \ C~ ~ ~ ‘

~

A. - \ \ .

-

~ ~ I

~~~~—/-~ I / g s

.

3~ ~~~~ -V ,~ \ ~~~-\ I I /

~~~~ ,

~, D —/ / \ * ~ ~ , 

~~ \ ~— \ \ t I / t—~ / 5 9.,

, 4 - N \ K\ \ I / / N /

~ 

I N/ \ \ \ I i I / / ,/ \ 7 ~ ,sP~ I 

~~ 

/x \ \Vc~1T Tr—k/,/ ic ~

—\ / ~\ %~5~

JP~

$

31 / I3

~~_. c

I I

5 ANAL ”

~~~ .,~

~fl*iz 1

., OJ131d~~OD

/ I

/ 

3lSV1d3a~ y

5 0

I d O

* 0.

Do / g

0

~ ,

~~~~ ‘

t a’ U UC-8 

— -- - - - - -~ - - - -- - -- - - ---- - --- 
______

ARI TP 286

-a 4 a 0

.00 a

.~ a a-

•8. I I ~~~~~~~~ ~) ~~~~~~~~~~~~~~~~ \ 1 I ~~~~~~~~~~~ ‘

~~

-a)) *~ ~~~~ \ 

~ 

/ / i—. ~

b. ~ ;

~~ \ \ - I / / ~~

Z:~~

’ \ 

~~~ 

~~~

/‘ N /S\ \ ~

\ 

~ 

/ / / ,

~

\ ~~/ S~ Cl,

\ a, /

~~ 55Cti~~ I—

I U,

I ~f l~l

~

LJ.

I ____________ ___________ 

I

/

~ 

S

II I o

U,

I 0.

Do 

5,

*~

I 1

~

03 

~ ~

‘4 ‘4~ •~

, ~~ ‘

0 , ‘5’. 0 _ ,J ~ ~

_ _~

5

~~ e~ ‘

~

3 d

~

C:,

a)

U.

C-9

- - - - ~~—-----~~~--~ 
ARI TP 286

000 I-

~~~~ 

‘

~~~ _~~

— s—2—g~~~~~~ (a.

00), i

~ 

%~~!— 2ç \ / ~~~ /

~4/

, ) ‘ \ \ 1 1 / / ~ ° ,,2

a) ..‘ \ \ \_—t —f---

~ 

/ / 7\ (

~

~~ \ \ ~~~~ t I 7—. I / ~ 4-’

- \ \— \ I I / ~ — / / \ (a C.,

\ 7\ \ % I I 7~~ . / 5 . N \‘ \ \ 1 I I / N/ 7 “ .0

~~~‘ I N / \ \\ ~

\ I 1 / / 

~ 

/ \ V ~~~~ /

~ 

\ \\)

~

kf ~t

~~ /// 2c -~~~\

L

~ ~~ 

N~~~~~~ \\\\ ~

/ / ~~~~~~~~~~~ )

~

( ~ ~~~~ ~~~~~~~~ 

llJ///z2fl ~—~ 

j

~—~~ ~~~~~~ L

~~~~i ~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~ \ ~

I ~~~~~~~~~~ ~~~~~~~~~~

\

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

J ~ OaI3ldPiOO a)

I I

/1 3l9V1d33~~y

‘

-0/I 0

/

~77t ~ D o I // “

I g

/ ~~

—I

~~ I / / I

~~t— 2 / / / ,

~~~, a,

/ / I ~ •Ip•

a,

~~~~~ / I U “4~, ‘

/— __ ~~~~ 2 I I 4 / ~~~~~~~ 1 ‘ 44. 4 Cl)

~~ /

~~~ 

A’

~~ ~~~~~ __ _ / I —_L_ I - — ~~ ‘

~~.

4 a,

U.

C -10

—---

--—--‘------~

-- -- — - 
------ ----

ARI TP 286

€00 a.. a 

-J

.00 

a a- o a-. a

) 

4 oa t a-~~~~~

a U, ~~ _

~‘:~~~c- \ 

~~~~ \ \ \ 

/ 

~~ 

/ 5 0

1k 4

4. 

~~~~~ 

/ 

~~~~~~~ ~\ ~ /

yz~~~~ ~~ 

I 

Q~ j31d~~

O0 __________ - ~~

319V~~~~~~~

/

1; 

~ ~~~~~~~~~~ ~~

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

~~~ II 

.

—L ~~~ I 

/

/ 

0

Do 

/ 

2

S

S ~~~~~ a,

,5~~4+~

4

~4

44 -

~~

~0 

/ • 

—a,

0 ~~~

-

*3

C-”
___ ---- - -- -

~~~ ---- --- -

ARI TP 286

000

-a U, -a 4 a o ‘a. z

UI a-. o. a -r a a-.

Ut) -

~~

~‘ ~~~-2~~~~ ?~~

-g--_ _

~, -

~~ - — . I I ‘------

\ / / Z-~~ ~~

(

~ \

~ 

~~

/ /“

~

1i2

~~~~_- 

7 ~~~~~~~~~~~ IL/

~~~~~

//Z

~~~~ ~~~~~~

I

- I 

~~~~~~~~ 

I/ ~/ ~

-l_~~~ 

\ ~~~~ ~~~ i 

~~~~~~~~~~~~ 

4 E9__—~~1~

I —~~~~~~ ~~~~~~~~ II \\\‘ ‘

~~~~ ~~~~~~~~~~~~~ / —5

~~~~~

— 

~~ 7 / / ~ L4 4—k~\\ N ‘.J ~

-—-

~~~,‘ I 0

~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~ /// / \ \\\\\~~~~~~~~~~~~~~~~~~~~~~~~ 

~~~~~~~~~~~~~~~~~~~~~~ 

°

~~‘rn~ .~~~

\ ~~~~~~ c /y D oJ/ / \ \\\\>K N

~~~~ 

/

~~ 

X / /

~~~~- ~~~~~\ \“>< ~~~~~~‘

\ -

, 

\ / / I I ~~~~~\\ ~~~~~~/ ‘—. / 1

~ a,

\ V X / II 1 ~ \\\Y N~~~~ ~~~.

AD a, \/ / °—.J / I I \ \ \X \ “‘ 4 ‘%

\ / /

~~~/ / \ ~~~ \ \\ ,* % 2 ‘- / / 7—~ I I ~—I ‘ V V \ 5 + 6

~

,i, a, “-K / ~ ~ ‘ ~~~~~~~~~~~~~ -\ “ ‘ 

“ “

~~‘

0 / / I I ~~~\ \‘ ‘ ‘NJ I I I ~~~ \\_ $ f ~, ‘7 - ——— ~ ----__ _ / I I -

~

U

*3 a,

5 .2’ U.

—

~

------ --- —-- - 
____

ARI TP 289

0 w aM 0— a-4 5 — -a ) ~ 4 a .0O

a aM $ in ~ . a

Ut) 6

,~ ‘

~~ ..—2— ! --5 .- (a,

~. (

~ ~

o 7 —g -. ., 0 ~~ 2 ..

~~~ I I ~~~ — 4” ‘~ ‘& .,— - \ I / / — . -~~ q. U,

‘a) ‘4 -~~~ ~~ ~~“ 0.

a / g

~~ o ~

f Lr, 4 ,

,- .. ~ ~~

.. 2 “

~ ‘)“ CD ~~ 4 / \ ( a~• 4.. i

~ ~~~ ‘ — 5 C.) %, 4 / . -çt 7 5 44, \ p~

cp

C/)

0

/

~13IIS I \

~ ~ c’-~’

~ 4 ’3V*U.L I \ ~~ZMt.E U

— ______ 

~ 

Qaifl dSO3

A 

378VA~

i3

~30.

‘

-0/I S

/71- 7 035VIiI~,

~~~ a

II Doi I / I c 7—J I 7 •

0

/ I__ _ _—4_ ____ — / II -ì 2 / I I / a,

... 

“ 4

~~ a,

•

,

~~

,<

‘ 

~~~~~~~~

~~~~~~~ ‘ . *s o 4. / —CD

0)

C-)

a a, UI I.-

C-13

— -

~

-- - -~~~~~~~~~- - - - --~~~~~~~~~~ - --- --- - - --~~~~ ---— - - -

~~- -- -~~---- ----- ------
fl--- - —-- --- -—-------- ----- --

~~ 

- - - 

~~~

. -‘

ARI Ti’ 286

~ E0.0A ~ ILITV UNTAAIP4EO RILIAIILITY COMBINED . TRAINED

ACCURACY UNTRAINED ACCURACY COMBINED. TRAINED

OLOSAL COMBINED - IPdTRAINE 0 OLOSAL TRAINED

Figure C-b . Cluster Groupings of Reliability, Accuracy and Global Ratings

C-14

________________ ____ -— -

~~-- - - ----- 
-—— ----~~~—~~~~~~~~~~~~~~~~ - ---~

--—--—- ~ — - -

ARI TP 286

APPENDIX D

STEP MULTIPLE REGRESSION ANALYSES

The following discussion presents still another way of assessing the

judgmental characteristics of the Accuracy and Reliability ratings . In

what follows , we shall be concerned with the degree and quality of the

Accuracy and Reliability ratings as predicted by their multiple relationships with the other scales employed .

In these analyses , we shall consider the patterns of ratings which

mutually predict the two forms of ratings as criteria with respect to

both subject and report averages .

Analyses Employing Subjects as Observations. As indicated in the

statistical procedures of Figure 3 (Analyses IV : A through C), these

analyses were made by averaging over the individual reports for each

subject ’s scale judgments .

These mean report observations on each subject and scale were

multiply intercorrelated over the subject scores against the successively isolated criterion variables of Reliability (Scale 47), Accuracy

(Scale 48), and Combined Global (Scale 5l)* ratings for each of the

separately considered subject groupings .

These combinations produced a total of nine separate scale analyses ;

three subject groupings by three criterion variables . The summary

results of each of these multiple regression analyses are displayed in

Tables Dl through D3.

All displays record the order in which each new variable was added to

the criterion predictive set and the resultant multiple correlation of

the total predictive set after each step . Variables are added to the

equation at each step so as to maximize the multiple R.

The first step of each display records that zero—order relationship

between the criterion and the single predictor variable with greatest

correlation. Subsequent steps are multiple correlations between the

criterion and the cumulative set of predictors.

When subjects are used as observations for the correlations among

scale quailties , the interpretation of these multiple Re is different

from that whqn reports are used as observations , as discussed below.

*In all of the following regression analyses, the two separate Global

ratings have been combined into a single variable by averaging their

separate values. The new variable is numbered Scale 51 (see also

discuøeion above), and Scales 49 and 50 corresponding to Global—l and

Globa l-2 have been dropped .

D—1 

-------—-—--——~~~~
__ -~~~~ - -

ARI TP 286

Here , the correlations reflect the degree to which each subject, without regard to report ratings, tends to use the scale attributes in

the same manner. We have to ask “Is there a consistent pattern of scale

usage across differing subjects?” and “Do differing subjects use the

scale attributes similarly or diff erently?”

The answer to these questions is expressed by the maximum value of

the multiple R achieved at the final step of the analysis. If subjects

tend to have report—averaged ratings on one scale which multiply covary

with other scales, the multiple R will be high .

An inspection of the results indicates that the maximum multiple R

achieved after entering 15 variables is uniformly equal to or greater

than .90 for all analyses.

That means that something more than 80% of the total subject rating

variance in the criterion variables can be accounted for by on 15 of the

scalar qualities . It remains to determine only the particular scalar

qualities in each analysis which are determining that variance.

With Reliability as criterion variable and considering only the

combined subject sample, more than half the obtained subject variability

in the reliability ratings is accounted for by the Accuracy and Global

scales. The remaining variance of the Relisbility ratings is accounted

for by the scales :

Appropriate (20), Accurate (31), Possible (26), Ac tive (1) , Hazardous

(22), Timely (3) , Dependable (17) , Expected (23), Extraordinary (6),

Widespread (30), Heavy (25), Truthful (21), and Massive (15)* in that

order , in short, by the determinants of Threat Potential, Accuracy ,

Expectedness and Dependability.

With Accuracy as the criterion variable, more than 80% of the total

variance of the combined subject sample is accounted for by the qualities

of Reliability, Acceptableness , Accuracy , Activeness, and Probableness

(Table D2).

The scales , in order, are Reliability (47), Accep table (38) ,

Conf irmed (40) , Ac curate (31) , Faultless (29), Massive (15) Useful (18),

Active (1), Probable (43), Poss ible (26) , True (2), Dependable (17),

Extraordinary (6), Direct (11) and Relevant (33).

one pole of the bipolar pairs is used and it has been reflected

to common direction (see Figure 4).

D-2

L ___________ —---- _ __
AB.I Ti’ 286

With the average Global rating (Scale 51) as the criterion variable, more than 80% of the total variance of the combined subject sample is

accounted for by the characteristics of Accuracy , Intelligibility , Massiveness, Consistency, and Usefulness (Table D—3).

The scales, in order , are: True (2), Intelligible (7), Reliability (47), Probable (43), Direct (11) , Clear (28) , Interp reted (32), Wide—

spread (30), Extraordinary (6), Consistent (45), Useful (18) , Massive

(15), Precarious (5), Faultless (29), and Truthful (21).

Analyses Employing Reports as Observations. For the purposes of

these analyses each report was treated as a separate observation by

taking means across the appropriate subject groupings on each of the

rating scales (see Tables D—4 through D—6).

Such analyses assess the multiple correlations of the scale qualities

with respect to the separately considered reports. Thus, patterns of

inter—relationships indicate the degree to which the characteristics of

the reports tend to covary .

These analyses are in a sense more conservative explanations of the

scalar structure than the previous subject analyses . Only the report

variance in scale usage need be accounted for instead of the larger

subject variance.

Reports which are judged by the averaged subject sample to have high

ratings on more than one attribute will show high correlation with those

other attributes .

The multiple Rs of these analyses are uniformly higher than in the

previous analyses . The 15 variables for each of the criterion variables

across the separate and combined subject groupings account for either

all (R — 1.00) or very nearly all (R .99) of the report variance

obtained with respect to these attributes .

In point of fact , the zero—order correlations of the first step of

each of these analyses contribute more than 81% of the criterion

variance alone.

For the trained , untrained, and combined subject groupings , the

single best predictor of the reliability ratings is Accuracy (Scale 48)

(Table D-.4). Dependable (Scale 17) and Consistent (Scale 45) improve the

determination for the combined subjects, with Dependable being the

second highest determinant in the separate groupings.

All subject groupings indicate that Reliability (Scale 47) and

Global (Scales 49 and 50 or 51) ratings are the two highest determinants

of the Accuracy ratings when considered with respect to report variance .

D-3 
ARI TP 286

The single best predictor of the Combined Global rating is Accuracy

(Scale 48) for all subject groupings, with percent of accounted variance

in excess of 81%.

In summary of these multiple Regression Analyses , it can be said that

they are consistent with, confirm , and extend the earlier conclusions

regarding the traditionally employed scales of Accuracy and Reliability .

It can be seen that the Global rating is a purer assessment of what

is being measured by the Accuracy rating; Accuracy judgments being

heavily influenced by the subjects ’ Reliability assessments. Reliability, on the other hand, is influenced by other attributes .

The Reliability assessments of the reports employed is significantly

based on an assessment of the internal consistency of the messages and

the likelihood of their content .

With respect to report variance, Accuracy and Reliability ratings are

each predicted by the other , for both trained and untrained subjects.

For the Reliability ratings, the single best predictor (R .94) is

Accuracy in both groups of subjects .

Accuracy is also predicted by Reliability in both subject groups with

R — .94. On the other hand , the multiple predictions of Reliability and

Accuracy analyzed with respect to subject differences indicate that

Reliability judgments are predicted only by the Accuracy ratings given

by the untrained subjects and not by the trained subjects .

For the trained subjects , the individual ratings of Accuracy can be

predicted from their ratings of Reliability , while Reliability ratings

are not predicted by the Accuracy ratings of the trained subjects.

Because of the nature of the simulation of battlefield conditions , or

for that matter any simulation , the subjects could not have had any

direct data on which to base their Reliability assessments. Nonetheless,

two paradoxical findings emerge .

First , the school—trained subjects ’ judgments of report Accuracy can

be predicted with high precision from their judgments of the Reliability

of the sources and agencies , second , the assessments of Reliability, as

made by all subjects, indicated that on the average the reports were

judged to be highly Reliable (see Table 4).

It appears clear that the two scales are at least partly contaminated

by one or both of their other measurements and are susceptible to

misinterpretation and misuse.

D -4
‘I

ARI TP 286

Table D-l. Step Multiple Correlations Using Subjects

as Observations with Reliability as Criterion

RELIABILI TY (~~7): SUBJECT MEANS

Subject Sample

Combined Trained Untrained

Scale R Scale R Scale R

1 48 (A ccuracy) .64 1 (Active) .47 48- (Accuracy) .82

2 51 (Combined .71 43 (Probable) .61 31 (Accurate) .84

Global)

3 20 (Appropriate) .74 31 (Accurate) .73 3 (Timely) .88

4 31 (Accurate) .78 15 (Massive) .81 23 (Expected) .90

5 26 (Possible) .80 35 (Analyzable) .85 21 (Truthful) .91

6 1 (Active) .82 36 (Routine) .89 18 (Useful) .92

7 22 (Hazardous) .84 24 (Implied) .93 37 (Completed) .93

8 3 (Timely) .85. 41 (Constant) .95 16 (Substantiated) .94

9 17 (Dependable) .86 12 (Variable) .96 26 (Possible) .94

10 23 (Expected) .88 5 (Precarious) .97 46 (Dangerous ) .95

11 6 (Extra— .89 11 (Direct) .98 11 (Direct) .96

ordinary)

12 30 (Widespread) .90 28 (Clear) .99 51 (Combined .96

Global)

13 25 (Heavy) .91 21 (Truthful) 1.00 7 (Intelligible) .97

14 21 (Truthful) .93 8 (Feasible) 1.00 17 (Dependable) .97

15 15 (Massive) .93 29 (Faultless) 1.00 29 (Faultless) .98

D- 5 

- -
-- —_ _ ——~——-—_ —--

ARI Ti’ 286

Table D-2. Step Multiple Correlations Using Subjects

as Observations with Accuracy as Criterion

ACCURACY (48): SUBJECT MEANS

Subject Sample

Combined Trained - Untrained

Scale R Scale R_ Scale - R

1 47 (Re liability) .64 29 (Faultless) .58 47 (Reliability) .82

2 38 (Acceptable) .74 47 (Reliability) .66 3 (Timely) .88

3 40 (Confirmed) .76 1 (Active) .73 40 (Confirmed) .89

4 31 (Accurate) .79 31 (Accurate) .79 15 (Massive ) .91

5 29 (Faultless) - .81 18 (Useful) .89 7 (Intelligible) .92

6 15 (Massive) .82 22 (Hazardous) .93 21 (Truthful ) .93

7 18 (Useful) .83 38 (Acceptable) .94 35 (Analyzable) .94

8 1 (Active) .84 43 (Prob able) .98 46 (Safe) .95

9 43 (Probable) .85 23 (Expected) .98 22 (Hazardous ) .95

10 26 (Possible) .86 20 (Appropriate) .99 13 (Unders tandable) .96

11 2 (True) .87 15 (Massive) .99 14 (Likely) .96

1.2 17 (Dependable) .88 13 (Understand— 1.00 33 (Relevant) .96

- able)

13 6 (Extra— .89 41 (Constant) 1.00 37 (Completed) .97

ordinary)

14 11 (Direct) .89 51 (Combined 1.00 51 (Combined .97

Global) Global)

15 33 (Relevant) .90 44 (Pertinent) 1.00 5 (Precarious) .98

—~~~~~~~~~~~~~~~ I__

~~~~~~~
ARI TP 286

Table D-3 . Step Multiple Correlations Using Subjects as

Observations with Combined Globa l Rati ngs as Criterion

COMBINED GLOBAL 51 (49-5 0) : SUBJE CT MEAN S

____ _______

Subject Sample

Combined Trained - Untrained

Scale R Scale R Scale R

1 2 (True .72 2 (True) .74 21 (Truthful .71

2 7 (Intelligible) .76 32 (Interpreted) .81 43 (Probable) .77

3 47 (Reliability) .78 8 (Feasible) .84 6 (Ordinary) .80

4 43 (Probable) .80 9 (Factual) .86 47 (Reliability) .83

5 11 (Direct) .82 43 (Probable) .89 37 (Comp leted) .86

6 28 (Clear) .83 37 (Completed) .92 1 (Active) .87

7 32 (Interpreted) .84 29 (Faultless) .94 39 (Many) .88

8 30 (Widespread) .85 16 (Substan— .97 11 (Direct) .89

t ia ted)

9 6 (Extra— .87 38 (Acceptable) .98 15 (Massive) .90

ordinary)

10 45 (Consistent) .87 13 (Understand— .99 24 (Implied) .92

- 

able)

11 18 (Useful) .88 17 (Dependable) .99 3 (Timely) .93

12 15 (Massive) .89 10 (Stable) .99 28 (Clear) .94

13 5 (Precarious ) .90 12 (Reliable) 1.00 34 (Secure) .94

14 29 (Faultless) .90 30 (Widespread) 1.00 42 (Large Scale) .95

15 21 (Truthful) .91 31 (Accurate) 1.00 13 (Understandable) .96

D-7

-- --- ~~~~~~~~---.-~~- - - --— —---~~~~~~~~~~~rn- ---- ---- --- -

~~~ ----~~~~~~———-——-~~~~~~~~~~
r

~ 

____ -

ARI TP 286

Table D- : . Step Multip le Correlations Using Reports as

Observations with Reliability as Criterion

RELIABILITY (47): REPORT MEANS

Subject Sample

Comb ined Trained Untrained

~~~~ __~~c~~e R - — Scale R Scale R

1 48 (Accuracy) .95 48 (Accuracy) .96 48 (Accuracy) .94

2 45 (Consistent) .96 17 (Dependable) .97 17 (Dependable) .95

3 17 (Dependable) .98 7 (Intelligible).98 23 (Expected) .96

4 14 (Likely) .98 22 (Hazardous) .99 51 (Combined .97

Global)

5 16 (Substan — .9 9 14 (Likely) .99 44 (Pertinent) .97

tinted)

6 26 (Possible) .99 27 (Observed) .99 10 (Stable) .97

7 23 (Expec t-cd) .99 9 (Factual) .99 14 (Likely) .98

8 38 (Acceptable) .99 19 (Volatile) .99 31 (Accurate) .98

9 51 (Combined .99 43 (Probable) .99 32 (Interpreted) .98

global)

10 30 (Widespread) .99 11 (Direct) .99 34 (Secure) .98

Il 28 (Clear) .99 38 (Acceptable) 1.00 3 (Timely) .99

12 7 (Intelligible) .99 31 (Accurate) 1.00 6 (Ordinary) .99

13 22 (Hazardous) .99 40 (Confirmed) 1.00 41 (Constant) .99

14 19 (Volatile) .99 8 (Feasible) 1.00 9 (Factual) .99

15 13 (Understand— .99 20 (Appropriate )1 .O0 37 (Completed) .99

abl e) ______________________ _________________________

D-8
______________________________________________________ —

ARI TP 2~

-L

Table D-5. Step Multip le Correlations Using Reports as

Observations with Accuracy as Criterion

ACCURACY (4~ ): REPORT MEANS

Subject Sample

Comb ined Trained UntraIned

Step Scale ____ 

Scale R Scale ____ 

R

1 51 (Combined .96 51 (Combined .98 47 (Reliability) .94

Global) Global)

2 47 (Re liability) .99 47 (Reliability) .98 51 (Combined .97

Global)

3 34 (Secure) .99 36 (Routine) .98 34 (Secure) .98

4 22 (Hazardous) .99 30 (Widespread) .99 23 (Expected) .98

• 5 3 (Timely) .99 11 (Direct) .99 12 (Reliable) .98

6 40 (Confirmed) .99 14 (Likely) .99 32 (Interpreted) .98

7 23 (Expected) .99 ~O (Appropriate) .99 27 (Observed) .99

8 12 (Re1iabl~~) .99 34 (Secure) .99 31 (Accurate) .99

9 43 (Probable) 1.00 8 (Feasible) .99 2 (True) .99

10 5 (Precarious) 1.00 19 (Volatile) .99 1 (Active) .99

11 7 (Intelligiblc )l.O0 46 (Safe) 1.00 30 (Widespread) .99

12 28 (Clear) 1.00 27 (Observed) 1.00 3 (Time ly) .99

13 35 (Analyzable) 1.00 16 (Substan— 1.00 43 (Probable) .99 - tiated)

14 17 (Dependable) 1.00 45 (Consistent) 1.00 42 (Large Scale) .99

15 9 (Factual) 1.00 10 (Stable) 1.00 35 (Analyzable) .99

D- 9

-- — —-— —- -- —.- .—--- - ~1
__________________________ 

________ -

ARI TP 286

Table D-6. Step Multiple Correlations Using Reports as

Obse rvations wi th Combined Global Ratings

as Criterion

COMBINED GLOBAL 51 (49-50): REPORT MEANS

Subject Sample *

Combined Trained • 

Untrained

Step Scale R Scale R Scale R

1 48 (Accuracy) .96 48 (Accuracy) .98 48 (Accuracy) .93

2 5 (Precarious) .97 2 (True) .99 6 (Ordinary ) .95

3 47 (Reliability) .98 13 (Understand— .99 14 (Likely) .95 - able)

4 2 (True) • .98 40 (Confirmed) .99 20 (Appropriate) .96

5 16 (Substantiated).98 19 (Volatile) .99 8 (Feasible) .97

6 39 (Many) .99 46 (Safe) .99 12 (Reliable) .97

7 8 (Feasible) .99 20 (Appropriate) .99 27 (Observed) .98

8 3 (Timely) .99 18 (Useful) .99 32 (Interpreted) .98

9 6 (Ext ra— .99 4 (Specific) .99 43 (Probable) .98

ordinary)

10 34 (Secure) .99 15 (Massive) .99 2 (True) .98

11 5*.(PrecariouS) .99 30 (Widespread) .99 44 (Pertinent ) .98

12 36 (Routine) .99 11 (Direct) .99 28 (Clear) .98

13 41 (Constant) .99 27 (Observed) .99 1 (Active) .99

14 32 (Int erpreted) 1.00 12 (Reliable) 1.00 36 (Routine) .99

15 4 (Specific ) 1.00 13* (Under— 1.00 26 (Possible) .99

standable) __________________________

*Variable dropped from predictor set. Such variables have all of their

reliable variance accounted for by the preceding predictors .

D-lO

-

~

. ~~—• -—----- ---- --- - - ---- — • --— • - -- - --- - -• - —---- --- •--- ----- -•-- - -

~~~-— - -—-
- -~~~~~~~~~ --~~~~~~~~~~ - - - ~~ - - -•- - -. -- -~~~

ARI TP 286

DISTRIBUTION

AR) Distribution List

4 OASD (M& RA ) 2 HOUSACDEC . Ft Ord, ATTN: Library

2 HODA (DAMI- cSZ) 1 HOUSACOEC. Ft Ord , ATTN: ATEC—EX—E— Hum Factors

I HODA (DAPE-PBR 2 USAEEC. Ft Benjamin Harrison, ATTN: Library

1 HODA (DAMA -AR ) 1 USAPACDC , Ft Benjamin Harrison, ATIN: ATOP—HR

1 HQDA (DAPE-HRE-Po) I USA Comm—Elect Sch, Ft Monmouth, ATTN : ATSN—EA

I HQDA (SORDID) 1 USAEC, Ft Monmouth, ATTN: AMSEL—CT—HDP

I HODA IDAMt-DOT-C) 1 USAEC. Ft Monmoutps , A1’TN: AMSE L—PA—P

1 HODA (DAPC-PMZ-A) 1 USAEC. Ft Monmouth, ATTh: AMSEL—SI--CB

1 HQDA (DACH-PPZ-A) 1 USAEC, Ft Monmouth . ATI’N: C, Fad 0ev Br

1 HQDA (DAPE-HRE) 1 USA Materials Syt Anal Agcy, Aberc een. ATTN: AMXSY—P

1 HQDA (DAPE-MPO-C) 1 Edgewood Arsenal, Aberdeen, AT1’N: SAREA—B L—H

1 HODA IDAPE-OW) 1 USA Ord Ctr & Sets. Aberdeen, ATTN : ATSL—TEM—C

1 HODA (DAPE-HAL ) 2 USA Hum Engr Lab, Aberdeen, ATTN: Library/Dir

1 HODA (DAPE-CP$) 1 USA Combat Arms Tng Bd, Ft Benning, ATFN: Ad Supervisor

1 HODA (DAFD-MFA) 1 USA Infantry Hum Rich Unit, Ft Banning. ATTN: Chief

1 HODA )DARD-ARS-P) 1 USA Infantry Bd, Ft Benning, ATTN: STEBC—TE —T

1 HODA (DAPC-PAS-A) 1 USASMA , Ft Bliss , A11’N: ATSS—LRC

1 HODA (DUSA-OR) 1 USA Air Def Sch. Ft Bliss , ATI’N: ATSA—CTD- -ME

1 HODA (DAMO-RQR) 1 USA Air Oef Sch, Ft B$iu, ATTN: Tech Lib

I HODA (DASG) 1 USA Air Oaf Bd , Ft Bun , A’T’TN: FILES

I HODA (DAIO.Pl) 1 USA Air Oaf Sd, Ft Bliss, ATTN: STEBD—~

’O

1 Chief , Consult Div (DA-OTSt3) . Adelphi, MD I USA Cmd & General Stf College, Ft Leavenworth, A1TN: Lib

1 Mt Asat. Hum Res, 000R&E , OAD (E&LS I 1 USA Cmd & General Sti College , Ft Leavenworth, ATFN : ATSW—SE—L

I HO USARAL , APO Seattle, AT1’N: ARAGP-R 1 USA Cmd & General Stf College. Ft Leavenworth, ATTN: Ed Advisor

1 HO First Army, ATTN: AFKA-O l-Tl 1 USA Combined Arms Cmbt Des Act , Ft Leavenworth , ATTN: DepCdr

2 HO Fifth Army, Ft Sam Houston I USA Comb ined Arms Cmbt Des Act . Ft Leavenworth, A’fl’ N: CCS

1 Dir, Army Stf Studies Ofc , ATTN: OAVCSA (DSP) 1 USA Combined Arms Cmbt Des Act, Ft Leavenworth, A’fl’N: ATC*.SA

1 Ofc Chief of Sit, Studies Ofe 1 USA Combined Arms Cmbt Des Act, Ft Leavenworth, ATTN: ATCACO—E

1 DCSPER . ATTN: CPS/OCP 1 USA Combined Arms Cmbt Des Act, Ft Leavenworth, ATTN: ATCACC—CI

1 The Army Lib, Pentagon. ATTN: RSB Chief 1 USAECOM . Night Vision Lab , Ft Belvoir , ATTN: AMSEL—NV—SO

1 The Army Lib, Pentagon, ATTN: ANRAL 3 USA Computer Sys Cmd, Ft Belvoir , ATTN: Tech Library

I Ofc , Asat Sect of the Army IR&D) 1 USAMEROC . Ft Belvoir, ATTN: STSFB—DO

I Tects Support Otc, OJCS 1 USA Eng Sch. Ft Belvoir, AT1’N: Libra ry

1 USASA, Arlington, ATTN: IARD-T 1 USA Topographic Lab , Ft Belvoir , ATTN: ETL -TO—S

1 USA Rscts Ofc, Durham, ATTN : Life Sciences Dir 1 USA Topographic Lab , Ft Belvoir . ATTN: STINFO Center

2 USARIEM . Natick . ATTN: SGRD-UE-CA 1 USA Topographic Lab, Ft Belvoir , ATTN : ETL—GS L

1 USATTC , Ft Clayton, ATTN: STETC-MO.A I USA Intelligence Ctr & Sets , Ft Huacisuca , A’fl ’N: Cr0—MS

I USAIMA, Ft Bragg, AT1N~ ATSU~

CTD~

OM 1 USA Intelligence Ce & ScIs, Ft Huachuca, ATTN: ATS—CTD-MS

I USAIMA. ~ t Bragg. ATTN: Marquat Lib I USA Intelligence Ctr & Sest. Ft Huachuca. ATTN : ATSI—TE

1 US WAC Cii & Sets . Ft McClellan, ATTN Lib 1 USA Intelligence Ctr & Sets, Ft Huechuca, ATFN: ATSI—TEX--GS

I US WAC Ctr & Sets , Ft McClellan, ATIN: Tog Dir USA Intelligence Ctr & Sets , Ft Hu.chuca. Aim: ATSI—CTS--OR

1 USA Quanermaster Sets , Ft Lee. ATTN: ATSM-TE 1 USA Intelligence Ctr & Sets , Ft Huachuca. ATTN: ATSI—CTD—DT

1 Intelligence Material D.~ Ofc, EWL , Ft Holsbird 1 USA Intelligence Ctr & Sets, Ft Huachuca , ATIN: ATSI—CTD —CS

1 USA SE Signal Sets , Ft Gordon, ATTN: ATSO-EA 1 USA Intelligence Ctr & Sets , Ft Huac(suca, ATTN: DAS/SRD

1 USA Chaplain Cit & Sets. Ft Hamilt on, ATTN : ATSC.TE-RO 1 USA Intelligence Ctr & Sch, Ft Huactsuca , Ai1’N: ATSI—TEM

I USATSCH, Ft Eustis. ATTN: Educ Advisor I USA Intelligence Ce & Sets , Ft Huachuca, ATTN: Library

I (iSA War College, Carlisle Barracks , AT1’N : t

~b I CDR , HO Ft Huachuca, ATTN : Tech Ref Div

2 W RA IR , Neuropsycfsiatry Div 2 CDII. USA Electronic Prvg Grd, AT TN: STEEP—MT—S

1 DLI . SOA . Monterey I CDII, Project MASSTER , ATTN: Tech Info Center

I USA Concept Anal Agey. Bethesda, ATTN : MOCA-WGC I Hg MASSTE R , USATRADOC. (MO

1 USA Concept Anal Agcy, Bethesda, ATTh: MOCA-MR I Reiearch Inititut., HO MASSTER . Ft Hood

I USA Concept Anal Aqey, Bethesda, ATTN: MOCA-JF I USA Recruiting Cmd, Ft Sherdian, ATTN : USARCPM—P

I USA Artic Test Ctr, APO Seattle , AT’TN: STEAC-MO~

ASI I Senior Army Ada,, USAFA000PTAC . Elgin AF Aux FM No. 9

I USA Artic Test Ctr, APO Seattl e, ATTN : AMSTE.PL.TS I HO USARPAC. OCSPER , APO SF 96558. ATTN : GPPE—SE

I USA Arnsasyw,t Cmd. Rediton, Arsenal, ATTN: ATSK-TEM I Stimson Lib, Academy of Health Sciences. Ft Sam Houston

I USA Armament Cmd, Rock ltland, ATTN: AMSAR-TOC 1 Marine Corps Inst., A’rTN: Dean-MCI

I FAA.NAFEC, Atlantic City , A TT’N: Library I HOUSMC . Commandant, ATIN: Cods MTMT SI

1 FAA.NAFEC, Atlantic City. AiiM: Hum Enge Br I HOUSMC. Commandant, AT’TN: Code MPI—20

1 FAA Aeronautical Civ. Oklahoma City. AIiM: AAC.440 2 USCG Acederny, New London, AIIM: A&n isaion

2 USA FM Arty Sets, Ft Sill, ATTh: Library 2 USCO Academy, New London, ATTN: Library

1 USA Asni or Sets , Ft Knox , ATTN: Library I USCG Training Ctr , NY , ATTN: CO

I USA Armor Sets , Ft Knox , ATTN: ATSB.Dl-E I USCG Training Cii, NY, AIiM: Educ Sic Ofe

1 USA Annoy Sets . Ft Knox , AIIM: ATSB-OT-1’P I USCG. Psychol Res Br, DC, ATTN: GP IF52

1 USA Armor Seti. Ft Knox, ATTN: ATSB-CD.A0 1 HO MM—Rang. Br, MC Di . Quantico, AIIM: PBS Div

E-l

— - --- - - - _

---~~~~—•.- _ —-~~~~~~~~
r -

~~

ARI TP 286

I US M.~n. Corps Usision Ole, ANC, Alsuandria, AiIM: AMCGS—F I Def & CIvsl Inst of Envint Msdlcln., Canada

I USATRADOC, Ft Monro., AIiM: ATR O-ED AIR CRESS. Kensington, AiiM: Info Sys Sr

6 USATRADOC . Ft Mama., ATI’N: ATPR-AD 1 Mllitascpsylcologlik h ansel., Capstsagsn

I USATRADOC. Ft Monroe, ATTh : ATTS-EA I Military fitt.dw, French Embassy, AITN: Doc Sec

1 USA Forces Crnd, Ft Mofhsrion, ATIM: Library I Medecin Chef, C.E.R?,A, —Ars.ftat , Toulon/Naval France

2 USA Aviation Test Sd, Ft Rsickst , ATTN STEBG—PQ I Prin Scientific Off , AppI Hum Engr Rids Div . Ministry

1 USA Agoy lot Aviation Safety, Ft Ruckus . ATTN: Library of D.f.ns*. NeW Delhi

1 USA Aory tot Aviation Safety, Ft Ruckus , AIIM: Educ Advi sor I Puss RsCh Otc Library, AKA, lsr*eI Defense Fosa.s

1 USA Aviati on 5th, Ft Puck.,, ATIM: P0 Drawer 0 1 Min hsteris wan Defensie. 000P/KL Afd Socluel

1 HOUSA Avi ation Syt Cmd, St Louis, AIIM: AMSAV-ZDR Psythologiache Zaken, The Hague, Netherlands

2 USA Aviation Sys Test Act , £dwv ds AFB, ATTN: SAVTE—T

1 USA Air D.t 5th , Ft IWe, A IM : ATSA TEN

I USA Air Mobility Rids & Dew Lab, Moff.t t Fld, ATTM: S*VDL-AS

I USA Aviation Sds. Rae Tag Mgt, Ft Nudist, AiIM: ATST—T- RIM

I USA Aviation Sth, CO. Ft Ruckus, ATTN: ATST—O—A

1 110, USAMC, Aiexan&ia, AIIM: AMXCD—IL

I HQ, USAMC. Alexandria, ATTN: COR

1 US Militar y Academy , West Point. A’fl ’N: S.

~

i.l, Unit

1 US Military Academy , West Point , ATTP4: Ofc of NiH t,drshp

1 US Military Academy, WetS Point, AiiM: MAOR

I USA Standardization Gp, UK. FPO NY, AIiM: MASE-.GC

1 Of c of Naval Rsds . ArlIngton, AIiM: Code 452

3 Ofc of Naval Rids, Aiilngt on, ATrN Cod, 458

I Oft of Naval Rids, Arlington, AIIM: Cods 450

1 Ofc of Naval Rids , Arlington, ATTN: Cods 441

I Naval Aerospc Med Ret Lab, Pensacola, AIIM: Acaus 5th Dlv

I Naval A.rospc M.d Rae Lab, Pensacola, AiIM: Cods L51

1 Naval A.rospc Mad Ret Lab, P,nsacela. ATTN: Cods 16

1 Chi,f of NusPers,A T’TN:Pers.OR

I NAVAIRSTA, Norfolk. ATTN: Safety Cu

I New Oceanographic, DC, ATrN: Coda 6251. Chstts & Tech

I Center of Navil Anal, AIIM: Dcc Ctr

I NavAirSysCom. AIIM: AIR-5313C

I Nay SeiNed, ATTN: 713

I NavHsllcoptsrSubSqua 2, FPO SF 96601

I AFHRL lET) Willi am AFB

1 AFURL (II)Lowsy AFS

1 AFHRL lAS) WPAFB, OH

2 AFHRL (DOJZ) Brooks AFB

1 AFHR L (DOJN) Licklarid AFB

I HOUSAF (INYSO)

1 HOUSAF (OPXXA)

I AFVTG (NO) Randolph AFB

3 AMRL (HE) WPAFB, OH

2 AF Inst of T.ch, WPAFB. OH, ATTN: ENE/SL

I AIC IXPTO) Randolph AFB

1 USAF AeroMed Ub, Brooks AEB (SUL—4). AIIM: DOC SEC

I AFOSR (ML). Arlington

I AF log Cn,d, McClellan AFB, AIIM: ALCIDPCRB

I Au Force Academy, CO. AIIM: Dept of 6.1 Scn

5 .

~

Psrs & Dev CIr,Sen Olago

7 P4 .u

~ %leh NeuropsycNa trlc Rich tJsut , Sass Diego

New ,,tronlc Lab, San Diego. AT1’N: Ret Lab

ti et TsngCsn, San Diego, AiiM: Coda 9000’- Lib

I Must . stGsa$ds , es..,5,, AiiM: Code SSAa

I NuwpostGreScls. Man*’.y, AIIM: Cods 2124

I NavTrngEquipCtr, Orlando, AIIM: Tech Ub

I US Dept of Labor, DC, ATIM: Manpower Admin

I US Dept of luttita, DC, AIIM: Dn,g Enforce Admln

I Nat Stir of Stande,th DC, AiiM: Computer Info Section

I Net Clearing (b ias foe’ NH—Info, Mockvllll

I Dunwer Federal Cas, Ldi. ,.4, AIIM: SIN

12 Defense Document.don CsnIsr

4 DV Psyds, Anssy Hq, NuusN Oft., Canberra

1 Salentlf It Mew, tell Sd, Army Ha. 

Nuu.N Ofcs, Canberr e

1 Mel and Al, Altadte, Assvlen Embaasy

I Cant,. de R.di.r0i Des Facteura, $umein. da I. Defense

National., Sevudi

2 Cenidan Joint SteP ~~~s1..g5isn

I CMV Staff , Royal CanadIan AF, ATm: Pen Sad Anal 5’

3 ChIef, Canadian Del Rids Staff , AiiM: C~

CRO$(W)

4 Su ites Del Stilt, Srttlsli Embassy, W.shll.4aUes

~~~~~~~~ -——~~~~-