O2LEVEL/

cc/,: f., / 3.

REVIEW OF PEER EVALUATION RESEARCH ,"

0'/..' -+....

"Ronld G,)Downey I Paul J Duffy\7

' • ii .. ..+ _ .......... ..... .. ............. .

PERSONNEL AND MANPOWER TECHNICAL AREA

'4M3 /D C C= on

DEC 4, 1978

=-B_

UW S. Army . .....

Research Institute for the BRhavioral and Socwial Sciences

/1 oovted fo8 Public d u,,

+!,+x, .]% ++,+•Appro~veU tor 0ub11¢ reloae,; dlltrlbuiton unhlmiilld,. '
DISCLAIMER NOTICE

THIS DOCUMENT IS BEST

QUALITY AVAILABLE. THE COPY

FURNISHIED TO DTIC CONTAINED

A SIGNIFICANT NUMBER OF

PAGES WHICH DO NOT

REPRODUCE LEGIBLY.
U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency under the Jurisdiction of the

Deputy Chief of Staff for Personnel

WILLIAM L. H~AUSER

JOSEPH ZElDNKR Coonl US' rmy

Tectinici D~irector Cormmander

Of STA I UVTI ON: Primary dilsttibuflorl of this report has beeninsidre by AA I. Please address centretpondenct,

oreeAf1igrrr I1stributhin of reports to: U. &. Armyn Research Institute for the IshavioraI and Social Sciences,

ATTN: ~O1 300i~' tiisrrilhowet AvIoAue. Aisexndi..1, Virginia 22333

.FINAL flISPQSITION This report mair he destroyed whom i. s no longer neetled. Pllsew do rict return it to

the U. S. Army Plesearch Institute for t14 Behaiovral end Sairia Sciences.

QIXI The findings in this report aer not to be construed as ant official Oepeitmenfl of the Army position,

unless so desigrneted by other @uthotizetl documents.
SECURITY~ CLASSIFICATION OFYI AC(fti, Daie' EM',tfed)

REPORT DOCMAENTATION PAGE READ INSTRUCTIONS

16 REPtORT NOMER2 GOVT ACCESSION NO. 3. R9CIP1ENTrS CATALF0 NUMBER

Technical Paper 342 _______________

4. TITLE (sevelIubotiol) 1. TYPE Or REPORT A PERIOD COVERED

REVIEW UiP PEER EVALUATION RESEARCH

S. PERPFORMINO ORG. REPORT NUMBER

"r AUVHOR(,) . CONTRACT ORk ORANT"UM191t.a)

Ronald G. Downey and Paul J. Duffy

6 FOMING I JeNISA ''ACTPI NAk ADRS ICO. PRoOGAM HLEM,1N 1',PROJjLCT, TASK( Ary rS ~eercn % tntT_ e lor the 13ehavioral1 AREA 4 WORK UNIT NUMBES t

and Social Sciences (PERI-IL)

5001 Eisenhower Avenue, Alexandria, Va. 22333 2Q16271.7A766

1I. Ct',NTROLO.NO OFFPICE NAME AND ADORES' It. REPORT DATE

Avwy Deputy Chief of Staff for Personnel October 1978

Washington, DC 20310 18. NUMNER OF PAGES%

28

ý4. MAONITORING AGENCY NAME & a'51111i FSi7teTrni frm Contro7ins Ollie* Ill. SECURITY CLASS. (of Ohio report)

Unclassified

It. W-Ai's5"I" CATIONIDOWNGRADIN

Is. oisTRiouTION STATEMEN (of hisReort)

Approved for public release; distribution unlimited.

17. DINT1111UUTION STATEMENT (of the abstract onfored in Block 20, It different froin Report)

IS. SUPPLEMENTARY NOTES

III. K(EY WORDS (Curgilnue an rsvers. oida. If Roe~oAry aid Identify by block number)

Peer evaluation

Poor rating

Vtalidity

Ruoliabil ity

Situational fitetors

10. ASTINACr ('Con~tinue an rovers@ sisl. It niecomoory and identify by black& n~wvhr)

!Peer ovaluation recsarch was reviewed from the three major perspectives

of validity studies, methodology, and situational factors. Most of the research progra~ms were conducted in the course of developing procedures for

evaluating training groups (eqg., in Officer Candidate School, U.S. Military

Academy, and Ranger course) . Substantial concur-rent and predictive validity qenerally was found, with correlation coefficients in the .30 to .50 range. Different evaluation methods (rating, ranking, nominations, and combinations

DI 1473 FOFTION OF' I NOV 69 IS ONSOLETE Unclassified
Unclassified

SECURITY CLASSIFICATION OF THIS PAOE(Wo Dole Xltvred

\20.

\of these techniques) did not differ substantially in either reliability

or validity. Evaluation methods did, however, differ in acceptability

and feasibility. Situational factors have documented or potential effects

on the evaluation process that developers and users of peer evaluations

should be aware of. Although many issues surrounding peer evaluations

remain unresolved, evidence suggests that these issues can be resolved,

and that they do not detract from the conclusion that peer evaluations

are a powerful tool in discriminating complex human bohavior..ý

NTIS V, POlm

S0

Unclassified

SICURIlY CLASSIFICATION FTI AIII~tnDt noo
Technical Paper 342

REVIEW OF PEER EVALUATION RESEARCH

Ronald 0. Downey and Paul J, Duffy

PERSONNEL AND MANPOWER TECHNICAL AREA

Submitted as complete and Approved By:

technically ac&urate, by: Ralph R, Center E. Ralph Dusek, Director

Technioci Ares Chief PERSONNEL AND TRAINING

RESEARCH LABORATORY

Joseph Zeidner

rECHNICAL DIRECTOR

U.S, ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES

5001 Eisenhower Avenue, Alexan•dria, Virginia 22333

Office, Deputy Chief of Staff for Personnel

Department of tho Army

October 1E78

Army Project Number Off our Careers

201U717A700
ARI Research Reports and Technical Papers are intended for sponsors of

R&D tasks and other research and military agencies. Any findings ready for

implementation at the time of publication are presented in the latter part of

the Brief. Upon completion of a major phase of the task, formal recommendatlons for official action normally are conveyed to appropriate military

agencies by briefing or Disposition Form.

"4
FOREWORD

This research, carried out within the Personnel Accession and Utilization Technical Area of the Army Research Institute (ARI), includes a

representative review of previbous findings, both w4thin th.,', Amy and

otherwise, on the validity and reliability of peer fe,.•ations. The

research also reviews several situational or contextual, factore that

should be considered in conducting peer evaluations.

This research is an in-house effort and is responsive to Army Project 2QI62717A766 and to special requirements of the Office of Deputy Chief

of Staff for Personnel.

Technical Director

Q,

.. .. .. . .
REVIEW OF PEER EVALUATION RESEARCH

BRIEF

Requirement:

To review previous findings on the validity and reliability of peer

evaluations as well as various situational moderators.

Procedure:

Peer evaluation research was reviewed from the four major perspectives of evaluation process, methodology, situational factors, and validity studies.

Findings:

Studies investigating the structure and nature of the peer evaluation process have generally found fairly clear factor structure across

widely varying samples. There is some evidence that the structure may

be as much in the nature of the rater as the ratee. A review of findings

from research that utilized different methods indicated little evidence

for substantial differences, in either reliability or validity, among

techniques. Further, a review of the documented and potential effects

of situational factors impacting on the evaluation process indicated

that users of peer evaluation should be aware of these issues in designing programs. Research generally has found substantial concurrent and

predictive validity, with correlations in the .30 to .50 range, but with

most studies limited to training groups.

Utilization of Findings:

Several issues surrounding peer evaluations remain unresolved; however, evidence suqgests that these issues can be resolved, and that peer

evaluat.ionsi are a powerful tool in discriminating complex human behavior.
REVIEW OF PEER EVALUATION RESEARCH

CONTENTS

Page

INTRODUCTION ................ ............................. 1

VALIDITY OF PEER EVALUATIONS ......... ...................... 1

METHODOLOGICAL ISSUES ................... ...................... 6

Metric and Distribution ................. .................... 7

Basis of Comparison .................. ...................... 8

Reliability ............... ..........................

Acceptability ........... ......................... 10

Feasibility .................... .......................... 11

SITUATIONAL FACTORS ................. ...................... ... 11

Group Size ..................... ....................... ... 12

Informal Group Structures ........... ................. ... 12

Demographic Characteristics ........... ............... .... 13

Group Boundaries ............... ...................... .... 14

Hierarchical Characteristics .......... .............. .... 15

Friendship... ............... ....................... .... 16

Length of Association ............. .................... .... 16

Type of Interaction ............. ..................... .... 17

SUMMARY ..................... ............................. ... 18

REFERENCES ....................... ........................... 21

DISTRIBUTION ..................... .......................... 27

LIST OF TABLES

Table 1. Some rupresentative studies on the validity

of peer evaluations .......... ..................... 3

LIST al' FTIGUIiS

Fiqure 1. Score distributions for reliable and

unreliable evaluations ........ ................... 9
REVIEW OF PEER EVALUATION RESEARCH

INTRODUCTION

When confronted with the prospect of drawing order out of complex human behavior in the equally complex world of work, much traditional

behavioral science research has been marked by two primary characteristics. First, heavy reliance has been placed upon human evaluations of

other human beings. Second, this evaluative information has been typically gathered from a limited observational viewpoint, that of a superior

toward a subordinate. The technique presented in this paper does not

deviate from the first of tht-',e characteristics; it does rely on human

evaluation of other human beings. However, it goes beyond the second

characteristic by gathering evaluative information from the perspective

of an individual's peers. For purposes of this paper, peers are operationally defined thus: (a) they have some common purpose or frame of

reference (e.g., members of the same work group), and (b) generally

speaking, they lack a formally recognized authority relationship between

them. Although the term "peer rating" is most commonly applied to this

technique, the present paper uses the more generic term "evaluation,"

reserving the term "rating" for one particular technique.

A source of much confusion in peer evaluation research has been a

lack of clarity between the technique and the dimension or characteristic evaluated. Although previous work reviewed here substantially supports use of peer evaluation as a technique, issues surrounding the

particular dimensions evaluated are not discussed in this review.

This paper contains throe relatively complementary sections. First,

a representative selection of typical validity research is reviewed,

along with a brief history of the use of peer evaluations. The second

section discusses various methodological issues underlying the peer evaluation technique, and the third section prusents several situational or

contextual factors that ;an affect a peer evaluation effort.

VALIDITY OF T'EI•ER EVALUATIONS

The history of the. pler evaluation technique can be traced from the

seminal work of Moreno (1934) and the development of the sociogram technique. However, the history of tho technique as it is dealt with here

is more conveniently traced to several efforts conducted during and after

World War 1*' (see, for exmunple Clarke, 1946; U.S. Army Research InstitUte, 1943; Wherry, 1945). One ur Lhe earliest investigations published

in the protossional literature i•i that. by Williams and Leavitt (1947)

|......6 & ........... ..........
Since that time, peer evaluations have been used for two primary purposes. The first of these purposes is evaluative in the criterion sensei

The concern is in judqing the extent or adequacy of some individual characteristic (e.g., leadership effectiveness, job performance). The second

purpose is evaluative in the sense of gaining information with which to

predict some future outcome (individual potential, motivation to work,

etc.). Both purposes have guided the efforts in research as well as

operational settings, although typically only one purpose has been the

focus in any given situation.

Table 1 summarizes the results and major characteristics of a representative sampling of studies which report validity information for peer evaluations. This overview is intentionally not exhaustive, since several

other more specialized reviews are available elsewhere'(e.g., Gibb, 1969;

Hollander, 1954a; Boulger & Coleman, 1964; & Nadal, 1968). Lindzey and

Byrne (1968) have also presented an excellent review of the use of social

choice methodology of which peer evaluations are one type.

There are several noteworthy features in Table 1. First, the magnitude of the validity coefficients is generally strong in both concurrent

and predictive studies. Peer evaluations have shown rather strong predictive ability even for periods up to 5 years (Hollander, 1965). Furthermore, in those studies that included measures in addition to peer

evaluations, the peer evaluations tended to have the highest concurrent

or predictive validity.

Also, the majority of the evidence for the value of peer evaluations

has been gathered in a training situation, particularly in the military

environment. In fact, only two of the studies in Table 1 (Weitz, 1958;

Downey, Medland, & Yates, 1976) used a sample from other than a training

or educational environment. With a few exceptions, most evidence has

been gained from peoplo relatively low in the hierarchy of their organizational setting.

A third major feature of Table I is the variety of dimensions that

peers have been required to evaluate and the variety of criteria with

which peer evaluations have been related. The peer evaluation dimensions have included leadership potential, personality traits, and supervisory skill, to name but a few.

2
O¶ O~ NN r- r- %D0 0 '0 ' co0 4.4 Hi in vA" q 0 .0 -0 %0U) '. r4 -4 H ULM '0 %D N

4 1 ' - 41 44 it 41 41 41 41 41 01

4N441 M 1 41 4c 41 A1 4

00 c w 0m O '.0 !-

(nN N .

00

0 4.4

z 0

0 to it'U'U

u Ai 0 u ro I 0 0.4 "1 ~~ ., 4.4 0 4 , 01

Li4 41. (1 4-1 4144 '

'-44

"P4 P4 PA4) 41 41

"#4 0.

0 0 .0

"-4 ., 4

4) 00 44 q 4

4.4

0 04 (

.4 41 H -H u 0

4) '" U1 114' 'a)4

01 (In 4 0 00 4H

.0 -Itf

0 '1 La H) t

V) 4) 4J .a

.4.j >.,0 )'

0 .ci~ H .4 d0
C OC~ lA n-,- ., 00 N Iq ve

'4J N(N 14 O M M M fl. r q .0 N M m U

.0 .0 -M m ~ '' - - - - - '

41 41 44 4444 41 41444 44 44 44

k H C N -1 N LA N" co 10' N (N 0~'

0 in

0 0) U) -W (1 4)u 4

U4 0 :Hl"'

u H0 ,I 0 0j 00 s v0 W 4 4j ~ 41. :3.4 U 4

4 .4( r4. 4J

NW * 0 ~ 0 0

c.~4 ~ ~ LI 4 H4 P4 ~ .

0P N 04 

U)) u 4.) U 4 M 0 U) t 44. 144.4

41 1.4 04

0l LO N4 0 (At 04 u U) n Ki 0) H th4

H ) ,4) H H u .l r) 0- 0l )~ M- 02 0- 4.IH1 H, 0) 10 0 H Q)

0) to &4 m~ H

A0 4 E-4 0

H) E.1 U) C)- 14H )

.4.)j N) 4)0N .,'4 y u(

P .P4 fd 4) r 4 '.4 .,N '4.4 -4 -4 44. 44. ) "44 Pb q 44. 4.4- M) 0 U "0 0 4-J U) 44 iU U)V A. t

.40 41) 4.d 4 4)4 N 4

,4 -H - u Fq. .,4 4 *. q P4 N4 M P- u 2 0t HO rd HO

t; 

LA

l)LA C) r.f Uf)

o~~' 0L A A I.- ,4 C)

Nn (4 0 Cn -4N

'.4' NEH £-W 0 4.J 4j()41 (nC.-4 It.

0 0 H- a) H a . , I
V' HeI . - 0 0 4. co ~ m m H- Hj *l ý

p~ 40 m'~

) - * . M . C *

LMme v t * k0 10

0~ 0

U~~ 0 4~

4.) 1

ON 44 ) A w 0w to 0

H 4J~q 4

or 0- fa8 04 0-A

0 4 04 U) 4)(

to P4 41

4j (1) 04 (D Q0 Ii 0 0 U U) 0U U0

44 (1 4.4 V 44 44I w 4.4 '4-4 d 4-4 44 0 U) .4J0 0 0 0 . -

0 d) I a

I 4 U) 4jU 4.) I u)

4.)~4 NO N .0 'o4J 

040~N co a) 4.) 

000'

(a CY C, .-4--) 4.) - Mn 14' -H inH ý 

2'0 -. m Q) U-ý4

(n El .- d -4 LU U . ( 4.i U -' #A

... .N ...... 
Attempts to implement peer evaluation programs have produced an

impressive array of findings. However, several limitations also appear.

For instance, there is only minimal evidence of the validity of peer

evaluations among individuals at organizationally higher levels. There

is also a limited, but growing, amount of evidence of the utility of peer evaluations in other than the training environment. In addition, in

studies that use peer evaluations as a predictor of a concurrent or future criterion, virtually all the validity evidence is of a bivariate

variety. Although a number of studies demonstratod that peer evaluations are often the best single predictor from among several predictors,

no research was found that attempted to determine what other predictors

might account for unique variance along with peer evaluations. An exception to this preoccupation with the bivariato paradigm is occasionally found in assessment centzc methodology. Mackinnon (1975) has elsewhere presented a comprehensive review of assessment centers, but even

in assessment centers with a wealth of information available, the

differential validity of peer evaluations has not always been adequately

addressed.

METHODOLOGICAL ISSUES

Peer evaluations have been performed by means of four primary techniques: ratings, rankings, full nominations, and high nominations. The

general paradigm of the rating technique calls for a group member to provide a rating of the relative amount or degree oZ the dimension under

consideration possessed by every other group member. The ranking procedure simply requires each group member to rank-order all other group

members from high to low (or some other relevant continuum) on the dimension under consideration. The full nomination technique requires that

each group member choose a specified number or proportion of the group

as being either high, medium, or low on a given dimension. The minor

variation of this technique in which nominations of the middle are not

required is also referred to as full nominations. However, the case in

which only high nominaticrto are elicited is reserved as a discriminably

different technique, for reasons to be elaborated upon it, later Portions

of the paper.

Several variations based on combinations of these basic techniques

are forced distribution rankings, or combinations of rankings with ratings. General scoring algorithms for the four primary techniques follow.

N

Rank'

-"...•k 00 S(:(.) L'M - ... X . .
Full Nominations:

i(rL) + Z(2r ) + .(3r

Score L M H

N

High Nominations:

Score r

N

where rRt 0 rating,

rk - ranking,

r w low nomination,

rM mid (or no) nomination,

rH high nomination,

N - number giving an evaluation, and

N J total nurber in the group.

All these techn!ques pr.duce ocores with means independent of group size, with the exception of the ranking formula, in whivh case adjustment

must be made for group sizes greater than 100. The standard deviation of

the various scores is a fui.4tion of the reliability (consistency) of each

group's evwluations; Gordon (1969) and Willingham (1959) dual with general issues related to reliability. Also, for a group usinq either a

ranking or nomination technique, the average score is determined; the

average score using the rating technique is free to vary.

Metric and l)inLributiur.

The metric and distributional properties of associate ev'aluations

are directly related to the particular technique employed. With respect

to scaling propprties, the rankinyu and both nomination procedures produce 3n ordinal scale (Stevens, 1951). The ratinqs from an evaluator

are the most nearly equal interval data, although here also it can be

argued that those art, merely An ordinal scale. The scaling properties

of t-he summatod scores from the various techniques approximate interval

data as the nLumber in the evaluation group increases.

7
The four common procedures will generally produce diLferent distributions, examples of which are displayed in Figure 1. Given the relatively free response mode, ratings will often produce negatively skewed

distributions larq'ely because group norms tend to inflate any evaluative

procedure. The ranking procedure, if it were perfectly reliable, would

produce a rectangular distribution with one person at each rank. Generally, less than perfectly reliable rank scores will tend to be normally

distributed, with very unreliable scores producing a more leptokurtic

curve, and a perfectly unreliable procedure producing a point distribution with everyone roceivJnq an average rank equal to tho middle ran).

Full nomination scores produce a distribution which, if perft.Qtly reliable, is trimodal, with on,- group receiving all high nominations, another

group all low nominntions, and thu remainder middle nominations or none

at all. High nominations pv-.,uuue a bimodal distribution (not shown in

Figure 1).

Basis of o ison

Scores resulting from the four primary techniques vary along another

important dimension--the evaluative procuss evokood in the evaluator upon

which judgments are made. Drucker (1957) initially pointed out the duality of focus with which peer evaluations can be executed: whetheo the

frame of reference or standard upon which the evaluations are made is internal or extei'nal to the group. In one case, the evaluator compares

the particular individual againgt a frame of reference external to the

group and assigns the individual to a category. In the second case, the

evaluator compares the particular individual against a frame of refer- I

ence internal to the group and makes a judgment of mnre or less, and

assigns the individual to the appropriate category. The external process

can be used only with the rating procedure. The internal process can

also be used with ratings, with rankings and nominations, it is required.

The internal procusýi, in general, requircs a moderate number of individuals in the group (|nore than fivo) . The direct implir~atlcn oC this distinction is that the exter:nal frame of roference allows both comparison

between individuals across peer groups and the compar-ison of peer groups.

The intornal process does not allow comparison between individuals across

peer groups unless the assumption is accepted that the groups are equal

on the particular ability, trait, or behavior.

A corollary of this implication is that population norms can be

developed only through the use of a rating procedure and an external

frame of referencu, aciain unless qrou•;, equality is assumed or assured.

The reliabilit:y of associate evaluations has generally been determined by one of two methods, cutimation ot internal consistency or testretest correlation. Both methods art analogous to the same procedures

in clahsuical te,;t theory (Lot-d & Novick, 1968)
LUU

* I

I Iu

I* Score I f!

g II •l

unreliable evaluations.

...... .. -- ..... ... .. .
The internal consistency of peer evaluations is the degree to which

members of a peer group agree with one another when observing an individual in a similar situation and at the same time. Using the multiplechoice test paradigm, the evaluators are comparable to test items and

those who are being evaluated are comparable to persons taking the test.

Although Gordon (1969) has recommended the use of the alpha coefficient

for estimating the internal consistency or reliability of peer evaluations, the most common procedure has been a split-half (or group) eatimate. The split-half estimate is made by randomly assigning peer group

members to one of two groups, computing scores in each group for all

group members, and then correlating the scores for each ratee from each

group (mee Hollander, 1957, & Downey, 1974). The correlation coefficient is then adjusted for total group size using the Spearman-Brown

formula (Gulliksen, 1950). If small groups are used, a random split

may not be possible, and Bome technique for averaging the intercorrelationa between evaluators could be used (Gulliksen, 1950). A

The test-retest method of estimating reliability requires that

group members evaluate each other at two different times. Scores from

the two different evaluations are then correlated. Examples of this

type of estimate are given in Hollander (1957) and Downey (1974, 1976).

Perhaps the most rigorous examination of reliability was done by Gordon

and Medland (1965), in which they varied both time of administration and

group doing the evaluations and found reliability coefficients in the

Research has generally demonstrated the reliability of peer evaluations to be in the .70 to .90 range, regardless of the type of reliability estimate employed. Research comparing the various evaluative methodologies is rare but has generally supported the view that all four methods

are quite similar, with perhaps a slight advantage to ratings (Suci,

Vallance, & Glickman, 1954; Downey, 19741 Hammer, 1963). Even the use

of a paired comparison procedure does not significantly improve reliability (Bolton, 1971).

Acceptability

A major factor in the success or failure of any peer evaluation

procedure, whether for operational or research purposes, is the degree

to which participants accept the purpose of the evaluations. Acceptability is generally studied as a specific issue of the particular program under investigation rather than comparative analyses of acceptability across techniques or situations. There in therefore little formal

evidence of differences between techniques in this respect, but inferences cani ho' rawni from the particular qualities of the technique.

S10
A major factor in the acceptability of a technique is the degree of

perceived difficulty. From this point of view, both the rating and ranking of large numbers of individuals (more than 20) can be time-consuming

and makes for difficult discriminations, particularly among group members

who are more or less average on the particular dimension. On the other

hand, the nomination procedure allows the individual to place a large number of people in a desired category and does not require such difficult discriminations.

The rating procedure is quite acceptable to the raters where the

rated group is small and cohesive. The full nomination technique is acceptable to the nominators for moderate-size to large groups in which

not all individuals are well known to one another. The high nomination

technique is even more acceptablo because it does not require an individual to make negative evaluations.

Another determinant of the degree of acceptability is the degree to ,:

which group members are knowledgeable about the evaluation procedure,

process, background, and use. Downey (1975) found that acceptability

improved as a function of an educational program. Two different considerations were notedt (a) the degree to which peer evaluations were

felt to be valuable and accurate estimates and (b) the degree to which

the evaluations were acceptable for particular uses. Downey also found

that a person's peer evaluation score and degree of acceptance of the

peer evaluation process were positively correlated; larger correlations

were found in the group who knew less about the peer evaluation process.

Feasibility

Closely linked with the concept of acceptability is feasibility,

or costs associated with the implementation and execution of a particular peer evaluation system. The major costs associated with a peer evaluation system are (a) preparation of evaluation materials, (b) administration time, and (c) scoring cost. Prior to the advent of automatic

data processing procedures, the costs associated with use of any peer

evaluation system in large groups or on a large scale were prohibitive.

Merely in terms of bits of information collected, it can be seen that

the number of evaluations is typically equal to n (n - 1) where n is the

number in the group. Thus, peer evaluation systems are relatively costly

efforts, which typically require more than minimal sophistication with

data processing procedures. Unfortunately, little systematic information

on cost iN available.

SITUATIONAL FACTOWS

In addition to the methodological concerns of the various techniques,

soveral situational or contextual factors can affect a peer evaluation

system, often without regard to the specific technique under discussion.

These factors include group size, informal group structures, demographic

I I" I! I I! " !.....i~--------------------! ......
r

characteristics, group boundaries, hierarchical characteristics, friendships, length of association, and types of interaction.

Group size

Very flw attempts have been made to stuldy the independent effects

of group size. More often than not, what evidence there is has been

reported as a byproduct in research directed ulsawhere. For example,

Downey, Modland, and Yates (1976) used a peeor nomination technique with

groups of Army colonels in 14 career groups that varied in size from 22

to 321. Reliability coafficionts varied from .63 to .94 and the rank

order coefficiant between group size and reliability was .03. Downey

(1976), in a sample of Army Ps.',-rm, compared peer ratings collected

within squads ( n 1 10) with peer nominations collected on the same man within platoo: (• % X, 40). Coefficients between the two scores were in

the .60's. ýiO'" r, platoon scores were both more reliable and more predictive of job performance.

As mentioned previously, from the standpoint of feasibility both

ratings and rankings would seem to be most appropriate for relatively

small group sizes (approximately a dozen), whereas the nomination technique is virtually mandatory for large groups (more than 50). From the

standpoint of empirical results, it appears that small groups may produce

somewhat unreliabla scores, with reduced validity. Alternatively, although it is rational to believe that there is an optimal upper size

peer group, scant evidence exists to support this view.

informal 6roup Struvt~ire,;

Within any forma.lly dvfined group, there may exist one ox more informal subgroups defined by some sort of mutual self-interest. The issue

then arises as to the ofCeou thveu;o informal subgroups may have on a peer

evaluation procedure condtimted in the Wtoal group.

The worst uase would uo one in which two equal-sized informal -jubgroups existed within a total group, and each qroup member was exclusively in one nubqroup or the other. In nuch a situation, one or both

subgroups might makr, Oh-cir ovaluationu solo.ly on the basis of subgroup

membership, i.u., on a banis othor thon the one intended. The not effect of such behavior is to attenuate Lho validity of the peer evaluation procedure; attenuatioai is most pronounced when both subgroups engage

in such behavior, The effect dlminishui Lif one of the groups does, In

fact, provtdo (,val.uationH over the whol, group on the dimension intended.

The eff(,ct al,';o dimlni,•ýihs as informal ,ubqroup size decreases or as the

number of !ibqjroupnt:r , ie

.2
Zn terms of technique, the effect of subgroup behavior is pronounced

if ratings or rankings are used. Resultant scores are most likely to be

negatively skewed. The use of full nominations will tend to produce scores

with decreased variance, and high nominations will produce the worst case

with a drastic reduction in variance. An important point when using nominations is that the use of too many nominations relative to total group

site may increase the effect of subgroup behavior (see Downey, 1974).

It is clear that subgroups of sufficient size can have an effect

upon the final scores. The problem is the incidence of such effects and

whether there exists a mechanism for detecting them. If the evaluation

process ix part of an ongoing process, the simplest procedure for checking

for these problems is the repetitive production of reliability indices

as part of the procedure for rýoducing peer scores. If the reliability

coefficients were to drop below .60, it would probably indicate a prob- lem, and care should be taken in use of the evaluations. Alternatively,

a two-way analysis of variance design, one factor being the type of

raters and the other factor being the same type of ratees could be used.

If a significant interaction were found, then a strong case could be made

for considering the peer scores as at least partially the result of group

membership.

Demographic Characteristics

The use of peer evaluations with their reliance upon fallible human

observers immediately raises the possibility of racial and sexual bias

on the part of evaluators. This concern is especially crucial in view

of recent problems associated with demonstrating the absence of bias in

employment selection and classification measures as well as in criterion

measures.

The evidence concerning racial bias in peer evaluations is mixed and

inconclusive. In a study dealing with Air Force recruits, Cox and

Krumboltm (1958) found that subjects were rated higher by members of

their own race, but the affect varied across groups, and there was substantial agreement on rank order across races (r - .76). They concluded

that any bias was far from complete and suggested that prior acquaintanceship of group members might account for the differences. In a similar

study in the Army, deJung and Kaplan (1962) found similar resultst RatIngs differed as a function of the rater's race. However, an analysis

of covarianco adjustinq for a combinod interest and math score showed

that whites did not give higher adjusted scores to whites or blacks,

but that blacks qave higher adjusted scores to blacks. Results were

interpreted in torms of asniqnment of hiqhor scores to close acquaintanceH--a result had most impact upon blacks ratinq blacks (because of

the smaller qroup size).

13
In a more recent study in an industrial training context, Schmidt

and Johnson (1971) used a forced-choice rating distribution in groups

made up of approximately equal numbers of blacks and whites. No differences due to race were found.

The evidence suggests that peer evaluations can be subject to racial

bias, but the effect is perhaps more strongly related to the interaction

between friendship or acquaintanceship and the particular evaluation

method used than to the fact of race itself. The presence of substantial correlation between the rank orderings from each race indicates

that the ordering was not much affected by race, But the use of ratings

allows evaluators to assign unrelated scores to individuals whom they

consider special in some way.

In terms of sexual bias, Mohr and Downey (1977) recently reported

results from a small sample of Army officers, in which females scored

lower than males on evaluations received from both males and females.

If bias occurred, it was on the part of both groups. An interesting

finding was that females' self-ratings were not related to either male

or female evaluations, but males' self-ratings were related to these

evaluations.

This admittedly small number of studies appears to indicate that

differences based upon race and sex can occur, but does not make clear

whether these differences are attributable to race or sex group differences, to interaction patterns (e.g., friendships), to the specific

methodology, or to some combinations of these factors. It would certainly be safe to say that researchers should be sensitive to the potential for such bias.

Group Boundaries

The discussion of peer evaluations has proceeded to this point as

if it were clear just what is meant by a peer or associate group. Most

researcher3 report their procedures in sufficient detail to show the

general characteristics of the groups in the study. However, given the

variety of overlapping and highor order groups in most real-life settings,

the issue becomes that of defining some basic guidelines for selecting

the appropriate ratinq group. It is clear that the selection of the

evaluative group can lie ffected by such factors as length and type of

interaction, formal n•rqani'ational structure, informal group structure,

friendship attternts, and, of course, the particular dimension being

evaluated.

Thore a•re fow emliricaL firndincys to guide selection of the peer

group. Rather, culdeloinos must be best guesses based on partial information |'rom relatcd daLa.

14
In a 1976 study, Downey found that platoon evaluations produced

more reliable and slightly more valid scores than did squad evaluations, but the differences were potentially confounded by differences in method

and group size. Gordon and Medland's 1965 study, in which individuals

were evaluated at two different times by totally different groups, indicated a high degree of stability across the two evaluations. Even the method used to compute reliability indices, random splits of the primary

group, supported the notion that group composition can be drastically altered without giving rise to major problems in the reliability and

validity of scores.

Hierarchical Characteristics

A concept related to that of qroup boundaries is that of hierarchies.

Suppose one were to perform a peer evaluation procedure in a traditionally hierarchical organization. If work groups at the subordinate level are

chosen as the peer groups, what effect does inclusion of their immediate

superiors have on the resulting evaluations? Conventional wisdom tends

to hold that inclusion of such individuals can contaminate the procedure,

and therefore they should be excluded from the worker peer groups and included in a peer group of first-level supervisors.

Again, results bearing upon hierarchical inclusion are mixed. Research by Levi, Torrance, and Pletts (1958) indicated no effects from

including the formal leader in the peer evaluation process. Research

by Downey in 1975, in which the leaders of small combat units were included in the peer nomination process, indicated that the leaders spanned the full range of peer evaluation scores. There was a positive relationship between formal position and peer evaluation scores of leadership

potential (as there should be, if the original selection procedure for

leaders had any validity). These data were experimental, and the introduction of an operational system might change the result.

A rational solution to the boundary/hierarchical problem should be

guided by the following suTgestions:

1. The group selected should be larqe enough to overcome problems

ansociated with primary qroups.

2. The group1 should not be so larqe as to include subgroups who

may be relat.tvwly unknown t-o each other or may be competing for

similar rc!-ources and rewards.

3. rho funct ion of the riroup solt.ctod should be reasonahly related

th the dimmension to be vvallatoted; e .. , if evaluation of leader-

,hip in a work setting iL dusired, a work group and not a social

qroiup should be selected.

....................... . . .. ...... ..... ......... .... .............
Friendshipl

I-,rJcnidithip has bean a major research issue in the history of peer

uvalu~tionm. According to folklorrn, peer evaluations are the product

of friendship or popularity and are therefore not valid indications of

the dimension under consideration. The impact of this bit of folklore

has been that, with the exception of simple validity studies, this in

probably the single most researched question associated with peer

evaluations.

Wherry and Fryer (1949) were the first to address the issue of

friendship in peer ratings. They reported that although there was a

moderate degree of relationship between friendship and a leadership ceiterion, the major portion o, the predicted criterion variance was indapendent of friendship. They concluded that peer evaluations of leadership are not popularity contests. Studies by Gibb (1950) and Horrocks

and Wear (1953) in college samples supported Wherry and Fryer's findings.

Borgatta (1954) also reported that leadership and popularity evaluations

were related, but he failed to draw any conclusions. Several other investigations have documented a moderate degree of relationship between

friendship and peer evaluations of leadership (Hollander, 19561 Hollander

C Webb, 1955S Theordorson, 1957).

Downey (1974) presented evidence that the use of full nominations

(with small numbers of high and low nominations required) reduced the

correlation between friendship and leadership evaluations compared with

forced distribution ratings.

It neems that when an evaluator is faced with the task of evaluating several people, some of whom he or she considers friends, the evaluator will tend to select a friend rather than another person considered

to be of equal, or at least indistinguishable, merit. Therefore, the

variance associated with friendship may be a source of systematic error

primarily in the middle of the dintribution. This systematic error variance will increase in 1arge groups, in which some members are relatively

unknown to each other or the interaction patterns are not fully established for all members.

However, in spito of the imprcssive array of research findings as

to the minimal effect of friendship, the "popularity contest" issue remains the arqument most consistently offered against the use of peer

evaluatiotns in an operational settinq.

Lc'r.igt~ho A~~li s oI i ion

Wh•n po'v'r vvalu1otions art, consiiderod for use in any situation, an

important question iN how long group mumbers must be associated with

each other bufore thvy ean provide reliable and valid evaluations. This

issue iH often ratmod in th• cont~uxt of transient tralninq groups.

lb
Research fairly consistently finds that peers can make reliable and

valid evaluations after a relatively short period of time--typically

3 to 6 weeks (Hollander, 1957).

Subsidiary to the overall issue is the mffect of including a new

group member in an intact group. Mayfield (1975) has suggested that in

such a situation there may be reason to suspect that a longer period of acquaintanceship is necessary for sufficient integration into the group.

A more generalized way of approaching the question is to determine which

person is known or not well known to other members of the group. Evidance has shown that an individual not well known to other members of the

group will typically be evaluated as near the middle of the distribution

of peer evaluation scores within the group (Downey, 1974).

In terms of technique, a nomination procedure is most likely to docrease the error variance assoclated with acquaintanceship; ratings or rankings tend to capitalize on the error variance ant show a greater dogree of relationship with acquaintanceship.

Type of interaction

Although peer evaluations have been used and reported over a span of more than 25 years, they have been applied in rather limited situations. Most of the research has been conducted with junior personnel in

a military training context such as Officer Candidate School (0CS), A

recent effort to use a peer nomination process in a senior Army officer

promotion system produced supportive results (Downey, Medland, a Yates,

1976). Outside the military, Weitz (1958) and subsequently Mayfield (19701 1975) have worked in industry with insurance salesmen.

Freeberg (1969) reported a project in which peer evaluations were

more highly related to a performance criterion when the interaction between peers was relevant to the dimension being evaluated. Bayroff and

Machlin (1950) found that leadership evaluations could be made in an

academic environment an" were highly related to evaluations made after

exposure to a situation where leadership was displayed. Lewin, Dubno,

and Akuia (1971) indicated that video tapes supplied sufficient information for reliable evaluations and that these evaluations were highly related to evaluations from group members.

Until more extensive research is conducted in broader organiza- tional contexts with a wider selection of subjoct populations, the generility of the peer evaluation process is larqely a matter of conjecture. However, it would be safe to assume that peer evaluations of a

variety of conmplex human bohaviora can be rendered reliably after

uxposuro of tho peors to each other irn nituations that require the

individual to Intr,act oither with tho environment or with others in

rolovant sttuiLiCons. Further, tho validity of the evaluations will be

a futiction of the denrqo to which the particular hehaviorn are relevant

17
to the dimonsion under study. Hollander (1956) found that reliable

evaluations were given after 1 hour of discussion between peers in a

naval OCS class, but the scores had only moderate relationship with

evaluations obtained 3 weeks later, and were even less predictive of

eventual job performance. This convergence of views by peers after a

short period of exposure is probably a function of similar psychological

maps of behavior on the part of peers, and the preliminary evaluations

are subject to revision based upon further information. There seems

to be little advantage in using one evaluative technique over another,

so long as the technique does not require the evaluator to make finer

discriminations than are possible, based on the type of Interaction

and the amount of information that can be gathered o rom the interaction.

SUMMARY

Researchers have used the peer evaluation technique both as a criterion of complex human behavior and as an index of future potential. The particular dimension measured has varied considerably. The validity

research summarized presents an impressive array of finding@ with correlation coefficients in the .30 to .50 range either in a concurrent or

a predictive situation. Research on extending the generality of the peer

evaluation procedure to a more diverse sampling of peer group types,

particularly nontraining groups, has been limited.

The four major techniques have also demonstrated important similarities and differences in their psychometric properties. For example,

only ratings can produce comparable scores across different groups without extensive assumptions. Researah results indicate little differences

in measurement reliability between techniques. The limited findings also

indicate that, in general, ratings and rankings are less acceptable than

either of the nomination techniques.

In view of the documented and likely effects of various situational

factors on the evaluation process, it is important that the researcher

be aware of potential problems in the use of peer evaluations. No direct

relationship was found between group size and the reliability or validity

of the evaluations, but it can be assumed that very small or very large

groups will produce less reliable and less valid scores. Group structure and demographic characteristics were found to be sources of potential difficulties. With respect to the popular issues of friendship,

acquaintanceship, and type of personal interaction, there is little

evidence that these have a major impact on the validity of the scores.

Indications are that all techniques are relatively impervious to a variety of situational factors, the nomination technique being perhaps the

most versatile.

1!
One possible adjustment in future work with this technique is to

begin referring to it as associate evaluation rather than peer *valuation. The term peer evaluation, or more commonly peer rating, has acquired overtones of meaning and often has a negative connotation among those requirud to perform the evaluations. Moreover, the more generalized rubric "associate evaluation" conceptually embraces more individuals;

the distinction should not be merely semantic.

In brief, peer evaluations, or associate evaluations, have been

shown to be fruitful tools in both research and application. Several

issues regarding their uve remain to be resolved, but there is sufficient evidence to suggest that these issues can be resolved, and that

they do not detract from the conclusion that associate evaluations are

a very powerful tool for discriminating complex human behavior.

?

J

19
REFERENCES

Mix-, Y., Kovarsky, Y., & Sharan, S. Peer Nominations as a Predictor

of Multistage Promotions in a Ramified Organization. Journal of

Applied Psychology, 1970, 54, 462-469. 1i

Bayroff, A. G., a Machlin, C. T. Dovelopment of Criteria of Leadership

in ROTC. Unpublished manuscript, 1950. (Available from R. G.

Downey, Kansas State University.)

Berkshire, J. R., a Nelson, P. D. Leadorithig Peer Ratings Related to

Subvugnt, Proficiency in Training and in the Fleeat (Special Report

No. 58-20). Pensacola, Fla.: Naval School of Aviation Medicine, 1958. !

1to 1olton, W. L. An Application of Constant Sum Paired Comparison Technique

thesis, Poor Ratng University atan of AryROCN.rCm. Tennessee at Knoxville, Unpblished 1971. master's

Borgatta, E. F. Analysis of Social Interaction and Sociometric Percsption. Sociometry, 1954, 17, 7-32.

Boulger, J. R., & Coleman, J. Research Findings with Peer Ratings

(Research Note 8). Washington, D.C.: Division of Research, Peace

Corps, 1964.

Butler, R. P. Correlates of Officer Performance (Research Report 74-021).

West Point, N.Y., U.S. Military Academy, 1974.

Clarke, J. M. Picking the 9,000. Infantry Journal, 1946, 59, 7-13.

Cox, J. A., & Krumboltz, J. D. Racial Sias in Peer Ratings of Basic

Airmen. Sociometry, 1958, 21, 292-299.

deJuhg, J. E., & Kaplan, H. Some Differential Effects of Race of Rater

and Rates an Early Peer Ratings of Combat Aptitude. Journal of

Applied Psychology, 1962, 46, 370-374.

Doll, R. E. Officer Peer Ratings as a Predictor of Failure to Complete

Flight Traininq. Aero9pace Medicine, 1963, 34, 130-131.

Downey, R. G. Associate Ratings and Senior Service School Selection.

ARI Rcsenrch Memorandum 73-4, 1973.

Downoy, R. C. Associat(, Hlvuationsi Nominations vs. Ratings. ART

Technical Paper 253, 1974.

Downey, R. G. A~sociato Evaluations: Improving Field Acceptance.

ARI Research Memorandum 75-5, 1975.

21

MCEDINO PAGE BLANK-NOT PILM.D
Downoy, R. G. Utilixation of Associate Nominations in a Training Environmenti Ranger Course. ARI Research Problem Review 76-8, I.

October 1976.

Downey, R. G., Medland, F. F., G Yates, L. G. Evaluation of a Peer

Rating System for Predicting Subsequent Promotion of Senior

Military Officers. Journal df Applied Psychology, 1976, 61,'

206-209.

Drucker, A. J. Predicting Leadership Ratings in the United States

Army. Educational and Psychological Measurement, 1957, 17,

240-263.

Freeberg, N. E. Relevance (.9 Rater-Rates Acquaintance in the Validity

and Reliability of Ratings. Journal of Applied Psyohology, 19569,

53, 518-524.

Gibb, C. A. The Sociometry of Leadership in Temporary Groups. Sociometry, 1950, 13, 226-243.

Gibb, C. A. Leadership. In G, Lindzey & S. Aronson (Edo.), Handbook

of Social Psychology (2nd ed., Vol. 4). Reading, Mass.--ddCronWesley, 1969, 205-282.

Gordon, L. V. Istimating the Reliability of Peer Ratings. Educational

and Psychological Measurement, 1969, 29, 305-313.

Gordon, L. V., 4 Medland, F. F. The Cross-Group Stability of Peer

Ratings of Leadership Potential. Personnel Psycihal , 1965, I8,

173-177.

Gulliksen, H. Theory of Mental Tests. New Yorki John Wiley and Sons,

1950.

Hagqerty, H. R. Status Rp I On Research for the U.S. Military Ac.adMy.

ARX Technical Research Report 1133, 1963.

Hammer, C. H. A Si• liedTechnicjue for Evaluatinq Basic Trainees on

6eadorship Potential. ARI Research Memorandum 63-10, 1963.

Hollat1dor, 1. P. Buddy Hntinqsz Military Research and Indu.:trial ImpliScations. PToronnell Pycholoqy, 1954, 7, 385-393. (a)

1loilander, I-;. P, Peor Nominatioti on Leadership as a Predictor of the

Pias-Fail Cr:iterion in Nava'. Ail Traininq. JTournal of Applied i'i syh..?l~o.,; , 195r4, 3-, .150-353. (b) . . ... ied• -

Hlollandor, E. P. Tho lri.'ondsh:ip Factor in Peer Nominatlions. Personnel

Pnyihnt~oqy, 1956, .9, 435-447.

22

.,-- . . . - __ - . ___ __ n 
Hollander, E. P. The Reliability of Peer Nominations Under Various

Conditions of Administration. Journal of Applied Psychology,

1957, 41, 85-90.

Hlollander, H. P. Validity of Peer Nominations in Predicting a Distant

Performance Criterion. Journal of Applied Psychology, 1965, 49, S434-438.

Hollander, M. P., & Webb, W. B. Leadership, Followership, and Friendshipi An Analysis of Peer Nominations. The Journal o1 Abnormal

and Social Paychology, 1955, 50, 163-167.

Horrocks, J. E., & Wear, B. A. An Analysis of Interpersonal Choice

Relationships of Colleqe Students. The Journal of Social Psycholo , 1953, 38, 87-98.

Klieger, W. A., deJung, J. E., & Dubuisson, A. U. Peer Ratings as

Predictors of Disciplinary Problems. ARI Technical Research

Note 124, 1962.

Kraut, A. I. Prediction of Managerial Success by Peer and TraininqStaff Ratings. Journal of Applied Psychology, 197S, 60, 14-19.

Kubany, A. J. time of Sociometric Peor Nominations in Medical Education.

Journal of Applied Psychology, 1957, 41, 389-394.

Levi, M., Torrance, B. P., & Pletts, G. 0. Sociometric Studies of Combat Air Crews in Survival Training. Sociometry, 1958, 21, 304-328.

Lewin, A., Dubno, P., & Akula, W. Face-to-Face Interaction in the PeerNomination Process. Journal of Applied Psychology, 1971, 55,

495-497.

Lindfey, G., & Byrne, D. Measurement of Social Choice and Interpersonal

Attractiveness. In G. Lindzey & E. Aronson (Hds.), Handbook of

Social Psychology (2nd ed., Vol..2). Reading, Mass.t AddisonWesley, 1968.

Lord, I". M., & Noviuk, M. R. Statistical Theories of Mental Test Scores.

Rvadiim, M.nas. t Addison-Wasley, 1968.

MacKinnon, D. W. Atn overview of Asesosment Centers (CCL Technical Report

No. 1) . Groensboro, N.C. i Ceiitir for Creative Leadership. May

1975.

Mayfi.•ld, 1-,. C. Management Selection. Buddy Nominations Revisited.

1'vroonnOelPayqhology, 1970, 23_, 377-391.

23

Ill• • ..... ,• '• .i•;•. •• i• .. '•", • ';i•2• ;i;;"i:•' •"•272'°•" ., ........ •3 •":'•'" 1• •°"'•" •i "•:'" ;•?•~l ........ 2 :"-•'• -'" . .........-.-. .. .....
Mayfield, E. C. Peer Nominations in the Life Insurance Industry. In

R. G. Downey & F. F. Medland (Chair), Peer Ratings: Beyond Validation Studies. Symposium presented at the meeting of the American

Psychological Association, Chicago, 1975.

Mohr, E. S., & Downey, R. G. Are Women Peers? Journal of Occupational

Psychology, 1977, 50, 53-57.

Moreno, J. L. Who Shall Survive? Nervous and Mental Disease Monograph,

1934 (No. 58).

Nadal, R. A. A Review of Peer Rating Studies (Reoearch Report No. 68-8).

West Point, N.Y.i Office of Military Psychology and Leadership,

U.S. Military Academy, 1468.

Paterson, P. E., Lane, N. E., & Ambler, R. K. Carefulness Peer Ratings

as a Predictor of Succeas in Naval Aviation Training (Special

Report 66-1). Pensacola, Fla.t U.S. Naval Aerospace Medical

Institute, U.S. Naval Aviation Medical Center, 1966.

Ricciuti, H. N. Ratings of Leadership Potential at the U.S. Naval

Academy and Subsequent officer Performance. Journal of Applied

Psychol2oy, 1955, 39, 194-199.

Roadman, H. E. An Industrial Use of Peer Ratings. Journal of Applied

Psychology, 1964, 48, 211-214.

Schmidt, F. L., & Johnson, R. H. Effect of Race on Peer Ratings in an

Industrial Situation. Journal of Applied Psychology, 137l, 57,

237-241.

Smith, G. M. Usefulness of Peer Ratings of Personality in Educational

Research. Educational and Psychological Measurement, 1967, 27,

967-984.

Stevens, S. S. Mathema.ics, Measuroment, and Psychophysics. in S. S.

Stevens (Ed.), Handbook of Experimental Ltycqhoj=. New York:

Wiley, 1951, Ch. 1.

Suci, G. J., Vallanco, T. R., & Glickman, A. S. An Analysis of Peer

Ratiný;s (Tochnical Bulletin No. 54-9). Newport, R.T. . Bureau

of Naval Personnel, 1954.

Theordorson, G. A. Thei Rvlationship Between Leadership and Popularity

Roles ii• Smull Groups, American Sociological Review, 1957, 1R,

56-67.

24
rt

Tupes, E. C. Relationships Between Behavior Trait Ratings by Peers and

Later Officer Performance of USAF officer Candidate School Graduates

(Research Report AFPTRC-TN-57-125). Lackland Air Force Base, Tex.t

Personnel Laboratory, Air Force Personnel and Training Research

Center, 1957.

U.S. Army Research Institute for the Behavioral and Social Sciences.

Seloction of Leaderst The Statun of the Measurement of Leadership.

ARI Report No. 444, April 1943.

Waters, L. K., & WaterN, C. W. Peer Nominations as Predictorm of ShortTerm Salos Porformance. Journal of Appllod Psychology, 1970, 54, 42-44.

Weitz, J. Selecting Supervisors with Peer Ratings. Personnel Psychology,,

1958, ýLj, 25-35.

Wherry, Robert J, Validation of a Program for Selection of officers for

Retention in the Peacetime Army. ARI Research Report 704, July 1945.

Wherry, R. J., & Fryer, D. H. Buddy Ratingst Popularity Contest or

Leadership Criteria? Personnel Psychology, 1949, 2, 147-159.

Wiggins, N., Blackburn, M., & Hackman, J. R. Prediction of First-Year

Graduate Success in Psychology| Peer Ratinqs. The Journal of Educational Research, 1969, 63, 81-85.

Williams, S. 8., & Leavitt, H. J. Group Opinion as a Predictor of

Military Leadership. Journal of Consulting Psychology, 1947, 11,

283-291.

Willinqham, W. W. A Note on Peer Nominations as a Predictor of Success

in Naval Flight Training (Research Report). Pensacola, Fla., U.S.

Naval School of Avisition Medicine, 1958.

Willingham, W. W. Estimatinq the Internal Consistency of Mutual Pear

Nominations. Psycholo ical Reports, 1959, 5, 163-167.

25
DISTOISUTION

AN$ Olat~btom List

4 011111 tMARA 1 M&OUAMO, Pt OATrt~i Llibrar

U IA ISM M .01410a 1 HOI4OUAOOC, Ft Ord, ATINi ATIC-11)4-11-44mt Puame

i NOCA iOAP4409111 S USAUIC, Ft lonlamIR Nortiso, ATTN: Ulbroy I MOGA 1OAMA.AN) I UBAPACOC, Pt 5ajmnsIn Harwtlsen. A'i~h ATOP-HPI

I HODA 10IC 451RSPOI I USA Comm-lilnt San, Ft Monmouth, ATTN t ATII-I1A

I 1400A 10111111-0 1 USAIC, Ft Mennmuti, ATTNi AMIBL-OT -HOP

*I NOCA ICAMIODOT-C) I USAVO, Pt Monmnouth. ATYN! AMSEL-PA--P

I NOGA IDAPCPMA) I USAIG. Pt Monmmith, AlTYNt AMSOL -11-411

1 I4WA IDAM-PPE.Al I UBA9C, Ft Monmiouth, ATTrN; C, Pool Doy It

I 1400A lowP-wool I USA Materilst bi Anal AgGY, AleNseen, AITNI AMXIY-P

I 6400A IODAIMP"4 I Nopee AMamei, Abordeen, AlTTNi SARIA-SL-H

I 401A IDAPU-OW tUSA Ota Ott of Sal, Abaiemlon, A!TNi £751-TIM-C

I 6CA 1141111WS.MNI I USA H~um RW Lotb, Absndnon, AM-~ Llbka'v/it~

I 1400A ISDDU.CPSI108 I USA treanhst Am. Too Sol, Ft bonnihm Al-tNt Ad Suiplarviee I WODA 4000.1110A) I USA 111(10411 "Win AMR Unit. Pt SIng,"R AM-N Ohle

I NODA 40ANO6ARU'1P) i UOA Infantry Od, Pt Wonnt% AM-N 4713TISCI-T

IN0A IDAPO-PAM I USAI44MA. Ft SIII% AlTN: ATS-LRC

N oDA IDUSA'O111 11 USA Air Of Sa@N Pt Sl1fm, ATTN: ATIIA-MT-MI

I GIOWA EAMOSOKI40 I USA Alt 0et SUK ht Vsa. Al-TN Yealt Lib

I NOSIA 4011111113 1 UUA Alt Oaf Id.Pt 11aaAINi PILES

INCAODA4A*II I USA Alt Cet III, Ft Sila. AltYNt 8711SO-PO

I OWt, C~iul Oly 40"sCII AdalpMd, MO I USA Omad b Geneal Itt Cellp, Ft Luoerivrwenhi, AM-~ LAW

I WMAwl M w 01". aO, 00M , CAD (IlWI I USA C&d A Getaral 1tW Gallop. Pt LBBwfwait, AlTT~t ATIW404-

I NO0 USAAL. APO SaMlls, Al-TNi ARAOP.N11 I VSA Ciii b Oeneal Itt Cel~p, Pt Loovenlo*, ATM Wi A00 tw

I NO PlMAoyew Arm,: APma4I I WSA embhined Arma Cum Oan Alt Pt Leanvflltw, AlTN:t Sap4 I NO Pill Amw, Pt Ism Names I USA CamMogdAlW4Ma MbeO A, Pt Lao~wenesoAl-~IN 00

I Ott, myW St ieOLON , A WiM CAVOSA101O111 I USA CuenedAno~Ne ultw At, Pt Lwawue*.ATTN aAICAI

I ofenf " 9 Setg, 911idee4 1l UI A Owambonu Aim 0mM Oaty Am. Pt Umvwensllt Al-tWt ATOMS-1

11 OUPIVA.AIM: Oum I USA Oemblie~ Amni Ce Sao At^ Pt Lowuselvlol Al-M ATWAG-Ci

I The kemw M, hoowe, Arm,: Nis owle I USAICOM11, %*a~ Viele Lab, Pt 111eflt.d AM-~ AMUL-NV-IS

I Whe buy LAN %POelP& AIIM ANNAL 3 USA Compior 111e Cuid, Pt Ukwe,*. Al-Ttd ?ab1e

Ite Ok AN be ofO IRVIV no I USAMISOC, Pt Salved AIT IM: llP-SO

I TOOalt W 01111 0.1011 1O IIUA rIe n, Pt t 1et1e"p, ArIM: Liberom

I U1110110 4ullopen AIM: lAND? I USA Tefepavolole Lek, Pt olvet, Al-TNi NIL-79.4

I WIA km Ole, 000%ae Al-Ti Ub sanea Ott 1 USA Vepwophia Lek Pt Salve, Al-TNi 01411POmGsse

I WANIS01M, Naatsa. Al-Nt gINO0-VICA I USA TepepploeLb. P* t Belit.d Al-n, STL-M

I USATTO, Pt OleyAu, AIM: ETIVCMC.A I USA IMM11spoa (hp 6 Salt, Pt Ntsaaltuuls AII: @10-MS

I USAIMA, Pt SInas ATYNt ATSU4fOC I LOA Iinalipem O A gon, Pt Muadwee, AM-~ ATI-Ml-M

UI IMAO PtfStp Al-TM M~aa LIW I USA lnW1WtiAn Ott b $el, Pt Humltuw, Al-TN: AiTM-7S

I US WA ON & M., Pt Me0ietlN. AIIM: LbW I USA Intellilpot Ott & Bel, Pt OWuaelueo. AlTMt ATM-TENX-Ga

I US WAG iOn6 , Pt Miteli, AIIM: Thg Dr I USA Inmiiipna Ott b $0. Pt Nuwaltuea, AlTM: A71111-41`11-00

W UA bertmuaamo Ii, 0t Loe, Al-TN ATIP' 'T I USA lntedilpnao Ott At On, PtI Mussaltuge, AIIMI AlTM-ClD-S-T

I lNeIpmUmMesolol Don Ot, OWL..A Pt ~tas, I USA 100ts140t *Or of Sal, Pt itugaltues, AIIM, A71SI1-CT-0

I USAIN SWgne Oil, 111 Gerdu, Al-M: AYIIOA I USA InlifieOe Car 1118, Pt HMuahoss, ATW4, 045450

I USA Doeulot Co A Salf, Ft Mamlftan, AM-N ATSOTE.N0 I USA tntahllpie Car b Belt Pt Itumliua, ATTN : Alli1-TUM

I WATSON, P1 Swinie, AIM:Wu IA. Nwi I WIA bnWilflgan Our bn P6 t lliuashuf, ATIM: LW"er

I USA WIP Weieiliam G arit a Iw h" AT~i: Wit I OCOR, MaOf Pt M se. ATY~i Yeash Ret Div

I WRAIN, Mmje"Vidlew" Dtt I CON. UBA E1eevitieO Prug Ord. Al-TN SIUP-T.S

I OLI, IDA. Miuwmt" I CON, Ptolst MAMMAil, ArIM: Yiehiie I antoEm

I USA Caneose "i AM., 9ehead., AT~iM MOCA.WW I He MAWISIIN AINA@ROC, LNO I WA 011101u10 Anal May, 1110101e11, ATTNM: MOCAUNl I NeAaalt dlIothtiute itO MANI AU. Pt Need

I USA QWWp A"a Agey, 11110fioi, AYM: MQA.JP I W6A Awsielf ON.S Ptilhe~lm, AW-Ni USARCPM-P

I USA Aolli fat Cit, APO Iaeuta, ATTN, sTE~ACMAgL, I 60nlW Army MAw.. UAPA@OD/TAC, 14nh AP AuK Pid Noae

I UNSAttbdsTatCit. APO SItia, AlTN: AMSTI-PLAV I NO USARPAC, 0011011111A APO IPF516 AIM, VM-S41

I USA Al IWOuit C~wi, RaiWei Amate, ArIM: RIENII1M I 11111mooi Lilt Aassomy of badilt hi , P- t Sam Wmawe I USA hiommsit Cmii Rel. %AlmiW, AYYN: AMApjA.FUC I MerhW Daa Insmt., A771M: COem-MCI

IPAA.NAPU,. Atfasli CIty, ANiM I w"ae I W&OIJMC, Ovemntitdeot, ATTN, Cem m¶my SI

I PAANNAPIC, Asimnle iGOV, AIIM; Hum Ing So I wauSMCOWimm., ~eATTN i ColaoMPI -10

I PAR Reosmmideis 441. Chialtea CSty, AITiM AA0.40 1 USCO Aeadsmv. Now Lwttlor Al-tNi Aduoiooia I A Fia Ati 111an, Pi Bill, AMII Lihiaty I W40O Aesodeuy, Naew Leondon, AIIM: Willewy IWA 4~iner On Pt alne.s, AM-N Library I USCO Toesinin Ott, NY, ATTMN CO USA AftoSM. OPI Me"e, Al-Ill; AYIS.Oit1 UM WOO % fte.tigO NY, ATTN; S &doift Olt

IUWA burn0A Of, Pt Knee, AWiIM ATO-OlYP I WOO,. Pwyete Nis It. 00, AIIM: OP iM

I WA Amw Ii.Of nt um, Al-M AY.O-AS i MC mid-Imop it, MO But, &iumnlee, AT"N, PWm 01v

27 . -A __ .i.~ PAO
11 

6 

U 

USATfiADOC, 

IMTO S leA"DC.CoPe 

Pt 

tLlM 

MONOeg, 

lonw OfoeAM 

ArT Nh 

C.A-TR1# 

ATP1O.-AD 

endns~ , Arm s~ iamoo$ - I AIR 01 SCM K n110e e q Eitvlr AMu InSlew, km

I IaATMADOC, Pit Mapped, ATmNI ATP I AIRO UUp Momisepese , a1Y~ Inf le k -IAD I MiIr ftlo Pow IIITh ws , A~ O Npetsa

IL A Awy tor Aviation Saftey. Ft Muetto, A~iN Iding Advlwe IUSA Aviawlos khs Pit flusker, AflN, 

I OmflteOa Oft LIbwap, AMA. loaM Osfeng Form

IWOUS AVIWNS"I~ y Ceed, It Lessle, A77h: 

PC btesu 

AMSAV-ZDN 

0 1 Miset wow W alpOsol~eb OOCPPK MAlMMI hMNSWIW4~e ighMl Thil Nip., Nsggpysd

I USA Alt 0oE she, Pt bligs, ATI~i ATSA TIM IUSA Alt Miagldiky RION A Deal Lob. MWfOt Pld, AflNt IAVOI.-AS

iUSA Aviat~w Ide. Roel ?ng Met, III fluelta, ATTNi ATVS?-ATMf

I 

IUSA Aviation Ikh, 00. Pit Muckor, AI¶'Ni AII-D-A D, OANCOMs, Aoitafemnyla, A'flN: AMIICD-iL I NO, OARGOM, Aleeomidlnie, ATTN: CDR I US Military Aledeenye, ftast Polntt. ATTN: Seirials Unith 11 US Miillteg' Ado"m, West Point, ArTN: Dive uf Milt LeWtshp I US M61tary Avoladnty, Wait Poinit, ArflP: MAO H I USA Ulwendorideaalsn doUfi, FPO NY, A e.'d: "All -CC

I Oft of Now Raflesh Arlington, ATTN Codei 412 0(6 e of Neval Plesh, Ahlioglnse, ATTN: Codei 4M IOf.st 

1OfN of Nval Remi, Atllnejtosi, ATTN: Cods 400

INeWa of Aeretooi Navalflesh, Mod Athieewe,,n Rle AflNi Cocle 441

I Neal Asetsggs Med Moot Lob, ftewamola, ArrNt Aseu, Sat Div Lillie Peonseesite ATTN, Ced Lai I Noela Aeraloso Mod flee Lob, Pofesse, ATTNs Cede LI

INAVAINS'TA, Norflik, ATTN i letely Cot I

I No, Onmineeelohs. 00, ATT'N! Ced* Silt, Chars 6 Ted

f APHtIIL IOTýWZillieee APB

I APHALI0W)Lowy APB

t NOUSAP (A111N AIO W

I H *APH (OF[M AI booAl I APVG IWAI fLonolipsti APB

5 AMAL 1140) WPAPA, ON

I AP inste uf Teedi. WPAF I, OH4, ATTN: INUM/L I AC 1XIOW)l~ flevesiOl API I USAP AprrMeftl Liu, Swoostu API (SUL -4), ATTN: DOC SIC t APOSM INtl. Affington I AP Log Crieil, Msf.lIetloi APB, ATTN- A L CIOPCAI I Ast Powee Aeoiloeenv, CO. AWN: DOw 'I Sol a".

7 NewV M14t Nsusailefys~aerotl Alvint Unit, Lion 04"g INew ftlweosseui Lobt I, OrDlaiep, A??N: Rlai Lob tNov 'rei~isC. Ams Diego, A11Ne Cride ACO0.Llh I lif"Aelstors, Mar0"illsy, ATTIN: Codea 664a I Ne~esus~,Mortelity, ATTN! COOe 2124

I NvtiselqeejcrOilorlfe, AI1N: Teels Libi I US Odaius Ws I".uu 00, ATTN Manileave, Adgeels I Us 000s elf beetles, OC, ATTN. Drug, C fllsito Adm"Ir I Not Alet ofi Stedds, LUf AT rNi Gorreuetor Iliac hImion t Not Clccatifglt How 10l1 MI -I -l1r, flhsekvilli. I Mille Pedeseal C0e LIWhwld, ATTN: it M of coleses.l5~ecWttI C44100a

4 Oil Poerch, Armyv No, flemel Ofo., Cenlserre I 1100411tic 4dm,, Mil Eld, Amin Hal, Nussel Ofet, Cmuer& I Metol *5 Ali Afteuhit, Auivosn Winlsatif t 04e1evse e 11c f lesIeut fle Peralliftir, 144se1orilej dois Oak""~

Naleseloew, Hlcur-le

V CrA1cee1sle .olieet staff Waseehsil,,e I CIA"e 118i, Milleyi Cendoiane AP-, ATYN: Port Steil Aito It # Vfthit. Gowealfil olef 111s,4, 91511 A TTN! COICflSfl 4 fiicemth lhs 4tlsed, Diteolgie knifiemey, Wothinqiton