DeNisi, A.S., Cafferty, T.P., & Meglino, B.M. (1984). A cognitive view of the performance
appraisal process: A mdel and research propositions. Organizational Behavior and Human
performance, 33, 360-396.

Dweck, C.S., & Leggett, E.L. (1988). A social-cognitive approach to motivation and personality.
Psychological Review, 95, 256-273.

Edwards, m.R., & Ewen, A.J. (1996). 360-degree feedback: The powerful new nodel for
employee assessment and performance improvement. New York: American Management

Association.

Greller, M.M., & Herold, D.M. (1975). Sources of feedback: A preliminary investigaiton.
Organizational Behavior and Human Performance, 13, 244-256/

Hanser, L.M., & Muchinsky, P.M. (1978). Work as an information environment. Organizational
Behavior and Human Performance, 21, 47-60.

Harris, M.M., & Schaubroeck, J. (1988). A meta-analysis of serlf-supervisor, self-peer, and peer-
supervisor ratings. Personnel Psychology, 41, 43-62.

Hazucha, J., Hezlett, S., & Schneider, R. (1993). The impact of 360-degree feedback on
management skils development. Human Resource Management, 32, 325-351.

Hegarty, W.H. (1974). Using subordinate ratings to elicit behavioral changes in supervisors.
Journal of Applied Psychology, 59, 764-766.

Heneman, H. (1974). Comparisons of self and superior ratings of management performance.
Journal of Applied Psychology, 59, 638-642.

Holzbach, R. Rater bias is performance ratings: Superior, self-, and peer ratings. Journal of
Applied Psychology, 63, 579-588.

Horner, D.H. (1995). Leader development and why it remains important. Military Review, (July-
August), 76-87.

Kaplan, R. (1993). 360-degree feedback PLUS: Boosting the power of co-worker ratings for
executives. Human Resource Management, 32, 299-314.

Landy, F.J., & Farr, J.L. (1980). Performance rating. Psychological Bulletin, 87, 72-107.

Latham, G.P., & Wexley, K.N. (1994). Increasing productivity through performance appraisal.
Reading, MA: Addison-Wesley.

London, M., & Beatty, R. (1993). 360-degree feedback as a competitive advantage. Human
Resource Management, 32, 353-372.

14



For most organizations, the objectives of the multirater process include:

developing an individual to his/her full potential

aligning individual and team behavior with organizational vision and values
focusing on competency based rewards

providing fair and accurate performance measures

supporting a commitment to continued learning

reinforcing and communicating other organizational initiatives

stimulating an organization’s employees to improve particular skills and essential
competencies

e enhancing employees’ awareness of strengths and developmental need

Features of an Effective multirater Instrument and Process

The advantages of a multirater assessment process will only be realized if a number of critical

issues are addressed. The following section is a

discussion of those issues. Note that the particular
emphasis during development and implementation
may vary somewhat depending on the organization’s
stated objectives . The use of 360 assessments as
part of an employee evaluation process, for example,
will require particular attention to a match between
assessment items and job performance standards.
However, the following concerns must all be
addressed to some extent in any multirater

Effective Implementation depends on:
+ A relevant assessment instrument
* Involvement of target audience
+ Ensuring confidentiality
* Training for participants

* Continuing review of the process

assessment implementation.

+ Follow-up support for self development

1. A relevant assessment instrument: developing competency items
Often the most difficult task in a multirater feedback project is to identify an

organization’s core competencies, and * Do the competencies match the organization's
how they can be communicated in a vision and values?
survey. According to Edwards and + Is the competency set complete? ,
Ewen (1996), the best way to select + Is there a focus on critical behaviors and skills

wi fata y ° that differentiate the organization in the
competencies for the survey is to

marketplace?
organize a focus group whose job is to + Is the organization’s common language used?
answer the question: “ What are the vis simple language that all members of the
critical competencies our organization els similar content grouped together, such as
will need in the future to sustain our competencies associated with teamwork?
competitive advantage?”. Within the + Is the competency set simple and short?
Army, there are doctrinal leadership
competencies which can be used as a
guide to develop the items of a leadership assessment instrument. Examining established doctrine
is an excellent place to start when determining important skills and behaviors that are of value to
an organization. As a general rule, competencies should be written clearly, in the active voice,
and be brief. Competencies should be defined with illustrative behaviors. Recently, the Army



FOREWORD

The United States Army is concerned with continuous self-development of all soldiers.
Self-development often begins with an accurate assessment of strengths and weaknesses. This
literature review was conducted to examine the potential of multirater assessments, also known as
360-degree evaluations, for systematic use by the Army. Possible utilization of multirater
assessments includes applications which would promote accurate self-assessment, encourage
developmental growth in areas relevant to leadership, and reinforce organizational values. A
growing literature on the use of multirater assessments in commercial organizations is reviewed
and discussed in this report.

This review was completed by the ARI Research Unit at Fort Leavenworth in conjunction
with the Leadership Research and Assessment Division of the Center for Army Leadership
(CAL), and the results were briefed to the Director, CAL, February 1997.

ZITA M. SIMUTIS
Technical Director



London, M., & Wohlers, A.J. (1991). Agreement between subordinates & self-ratings in upward
feedback. Personnel Psychology, 44, 375-390.

London, M., Wohlers, A.J., & Gallagher, P. (1990). Upward feedback surveys: A source of
feedback to guide management development. Journal of Management Development, 9, 17-
31.

Mabe, P., & West, S. (1982). Validity of self-evaluation of ability: A review and meta-analysis.
Journal of Applied Psychology, 67, 280-296.

McCall, M.W., & Lombardo, M. (1983). Off the track: Why and how successful executives get
derailed (Technical Report No. 21). Greensboro, NC: Center for Creative Leadership.

McCauley, C., & Lombardo, M. (1990). Benchmarks: An instrument for diagnosing managerial
strengths and weaknesses. In K.E. Clark & M.B. Clark (Eds.), Measures of leadership
(pp. 535-545). West Orange, NJ: Leadership Library of America.

McEvoy, G.M., & Buller, P.F. (1987). User acceptance of peer appraisals in an industrial setting.
Personnel Psychology. 40, 785-797.

Mead, G.H. (1934). Mind, self and society. Chicago: University of Chicago Press.

Nilsen, D., & Campbell, D. (1993). Self-observer rating discrpancies: Once an overrater, always
an overrater? Human Resource Management, 32, 265-281.

Schneier, C.E., & Beatty, R.W. (1978). The influence of role prescriptions on the performance
appraisal process. Academy of Managment Journal, 21, 129-135.

Shrauger, J.S., & Schoeneman, J. (1979). Symbolic interactionist view of self-concept: Through
the looking glass darkley. Psychological Bulletin, 86, 549-573.

Smither, J., London, M., Vasilopoulos, N., Reilly, R., Millsap, R., & Salvemini, N. (1995). An
examination of the effects of an upward feedback program over time. Personnel
Psychology, 48, 1-34.

Thornton, G.C. (1980). Psychometric properties of self-appraisals of job performance. Personnel
Psychology, 33, 263-271.

tuckman, B., & Oliver, W. (1968). Effectiveness of feedback to teachers as a function of source.
Journal of Educational Psychology, 59, 297-301.

Van Velsor, E., Ruderman, M., & Phillips, A. (1991). Enhancing self-objectivity and performance
on the job: The developmental impact of feedback. Greensboro, NC: Center for Creative
Leadership.

15



performance was moderate or low improved their performance over the 6-month period after
having received feedback, more than the group of managers who received no individualized
feedback. This (quasi-experimental) study was limited in that the researchers could not randomly
assign managers to the two conditions. Thus, while multirater feedback programs seem promising,
a true experimental examination of the impact on performance is still warranted.

Implications for Use by the U.S. Army

The Leader Development program of the U.S. Army is supported by the three pillars
prescribed by doctrine: institutional training, operational assignments, and self-development.
While the starting place for any self-development plan must be an accurate assessment of relevant
strengths and weaknesses, there is currently little to guide the self-development portion of this
process. Much of the research reviewed in this paper highlights the importance of others’
perspectives in forming an accurate self-assessment. The multirater feedback process may be the
tool the Army has been searching for to create permanent changes in its soldier's performance
through increased self-awareness of strengths and weaknesses. However, there are some aspects
of multirater feedback systems which the Army needs to further develop.

In general, if the Army wants a better instrument to work with, more investigation of the
psychometric properties that work best with a multirater survey is necessary. In particular, are
wider scales necessary so that feedback appears in a manner that is more motivating? Do
behavioral anchors on survey questions add to the quality of feedback and interpretability? How
can we balance the need to thoroughly cover all competencies with the need to keep the survey
length reasonable, especially when individuals may be completing surveys for more than one
individual? How should feedback be graphically displayed so that it is most easily interpreted?

Future research efforts should attempt to determine how the organizational structure of the
Army (compared to corporate structures) influences the utilization and acceptance of feedback.
How can individuals in the Army be supported during the follow-through period of remediation?
Strategies and interventions, which could enhance the effectiveness of the 360 process, should be
developed to meet the specific needs of the Army.

Finally, it would be beneficial to determine empirically whether multirater assessments create

permanent increases in self-awareness and self-development which could contribute substantially
to meeting the long term goals of an organization such as the Army.

11



recognizes the complexity of management and the value of input from different sources (Becker &
Klimoski, 1989). Additionally, multiple ratings allow testing of consistency of leader behaviors
and reliability of the information gathered.

Peer Appraisal.

Lateral feedback has been shown to be reliable (peers are consistent with one another), valid
(highest rated people eventually receive the promotions), and perceived as highly credible to those
who receive the feedback. Nonetheless, in smaller organizations, there may be a shortage of peers
with sufficient contact to provide quality feedback. Generally, while peers tend to provide honest
feedback (especially when the feedback is for developmental purposes only) there is always the
fear that a peer may provide false negative information in an attempt to derail a target’s career, or
to promote their own career. While feedback from a single peer has its shortcomings, input from
multiple sources makes it easier to spot false negative feedback.

Upward Feedback.

Research indicates that upward feedback (subordinate evaluations) is a better predictor for
performance improvement than self and supervisor assessment (Wilson, O’Hare & Shipper,
1990). Subordinate assessments provide important details on how supervisors achieve results,
and on the reasons why communication failures occur (Berman & Hellweg, 1988). Subordinate
assessments have also been associated with improved teacher performances (Tuckman & Oliver,
1968), and with eliciting modest changes in managerial behavior (Hegarty, 1974; Van Velsor,
Ruderman, and Philips, 1991). Subordinates are in an excellent position to view and evaluate
leadership behaviors. Although subordinates may provide information of slightly higher quality
than supervisors or peers, organizational cultures with a highly structured hierarchical system may
find the use of subordinate reports incompatible with organizational values. Additionally, there
may be fears that subordinates may “gang up” against a particular ratee, or that the ratee may
retaliate against a source of negative feedback. Therefore, anonymity and other safeguards are
essential (Edwards & Ewen, 1996).

According to Bernardin, Dahmus & Redmon (1993), managers and supervisors have several
concerns regarding upward (subordinate) feedback. Such concerns include: 1) supervisors may
focus on pleasing subordinates in an effort to get higher appraisals, 2) the authority of the
manager could be undermined by the pressure of upward appraisal and the implications of low
evaluations for the manager’s status within the organization, 3) subordinates lack the ability,
aptitude, training or necessary job information to provide valid ratings, 4) subordinates may be
reluctant to be candid about their bosses for fear of repercussions, or they may inflate ratings in
order to gain points with the manager, 5) employees who are being pushed the hardest by their
supervisors may rate those supervisors more harshly, 6) managers may also be uncertain about
how to interpret subordinate appraisal relative to ratings from other sources , such as their boss.
Most of these concerns can be dealt with by instituting safeguards that ensure confidentiality, and
with employee training in giving and receiving feedback.



MULTIRATER ASSESSMENT PROCESS - A LITERATURE REVIEW

CONTENTS
Page

INTRODUCTION ......sesssssssssssssesssssscssecssccnessssecsnessussssssssvessssssscsssessscsssssssssscssuceusseusenssenceneeenresse 1
EVOLUTION OF THE MULTIRATER FEEDBACK CONCEPT......sssssssssssssseessscsssesseseensesseese 1
TRADITIONAL FEEDBACK METHODG..........csssssssssssssessssesssssscssessesssssssessssensesusenseenseaeesneesee 2

Supervisor Appraisal.

Peer Appraisal...

Upward Feedback..

Self-Assessment
BENEFITS AND OBJECTIVES OF MULTIRATER FEEDBACK PROCESS .........sssssssssseneene 6
FEATURES OF AN EFFECTIVE MULTIRATER INSTRUMENT AND PROCESS............0000 7

Developing Competency Items..
Involvement of the Target Audience
Ensuring Confidentiality
Training for Participants
Continuous Review of Process and Instrument..
Follow-up

ENHANCING THE EFFECTIVENESS OF A MULTIRATER FEEDBACK SYSTEM.

IMPACT OF MULTIRATER FEEDBACK PROGRAMS .....ssssesssssssesssssssesssecssecseessecnensseneenses 10

IMPLICATIONS FOR USE BY THE U.S. ARMY...

REFERENCES .....ssssssscossessssscsssecsssecssnessssessssscssssecssssesssssccssnscsesscsssuccesvesesnessssecssscsessessseessnessses 13


MULTIRATER ASSESSMENT PROCESS - A LITERATURE REVIEW

EXECUTIVE SUMMARY

Research Requirement:

To provide background information for the multirater survey project promulgated by the
Deputy Chief of Staff for Personnel.

Procedure:

Major databases were searched for articles and reports on the use and effectiveness of
multirater feedback evaluation systems. These reports and articles were reviewed and analyzed,
and major findings and issues are reported.

Findings:

A review of the current literature on multirater feedback evaluations indicates that this
process could increase self-awareness, thereby encouraging self-development. However, more
empirical data are necessary to conclude that multirater feedback has a lasting and beneficial
impact on behavior and performance.

Utilization of Findings:
The literature review will be used by ARI, the Center for Army Leadership, and other

organizations exploring the possible utility of multirater assessment as a tool for self-development
within the United States Army.

vii



References

Ashford, S. J. (1989). Self-assessments in organizations: A literature review and integrative
model. Research in Organizational Behavior, 11, 133-174.

Atwater, L., Roush, P., & Fischthal, A. (1995). The influence of upward feedback on self and
follower ratings of leadership. Personnel Psychology, 48, 35-59.

Atwater, L., & Yammarino, F. (1992). Does self-other agreement on leadership perceptions
moderate the validity of leadership and performance predictions? Personnel Psychology,
45, 141-163.

Bandura, A. (1997). Social learning theory. Englewood Cliffs, NJ: Prentic Hall.

Bandura, A. (1982) The self and mechanisms of agency. Psychological perspectives on the self (J.
Suls, Ed.). Hillsdale, NJ: Erlbaum Associates.

Bass, B., & Yammarino, F. (1991). Congruence of self and other’s leadership ratings of naval
officers for understanding successful performance. Applied Psychology: An International
Review, 40, 437-454.

Becker, T.E., & Klimoski, R.J. (1989). A field study of the relationship between the
organizational feedback environment and performance. Personnel Psychology, 42. 343-
358.

Bernardin, H.J., Dahmus, S., & Redmon, G. (1993). Attitudes of first line supervisors toward
subordinate appraisals. Human Resource Management, 32, 315-324.

Bernardin, J. H., & Beatty, R. W. (1984). Performance appraisal: Assessing human behavior at
work. Boston: Kent.

Bernardin, J.H., & Beatty, R. W. (1987). Can subordinate appraisals enhance managerial
productivity? Sloan Management Review, 28, 63-73.

Borman. W.C. (1974). The rating of individuals in organizations: an alternative approach.
Organizational Behavior and Human Performance, 12, 105-124.

Carver, C.S., & Scheier, M-F. (1981). Attention and self-regulation: A control theory approach to
human behavior. New York: Springer-Verlag.

Cooley, C.H. (1902). Human nature and the social order, New York: Scribner.

Cummings, L.L., & Schwab, D.P. (1973). Performance in organizations. Glenview, IL: Scott
Foresman.

13



Generally, one is evaluated on the particular competencies which are highly valued within his
or her organization and which impact individual performance (e.g. communication skills,
creativity). One potential outcome of a multirater evaluation is that organizational values, which
guide behavior and decision-making at all levels, can be disseminated and reinforced throughout
the organization. A second and equally important potential consequence of this process is that
individual employees can gain an increased self-awareness of the competencies related to
organizational values. With an increased self-awareness, employees are empowered to develop the
skills and competencies that are important to the organization, and which help to guarantee a
successful career track.

Currently, multirater feedback systems are an amalgamation of several traditional feedback
sources. Thus, it is instructive to examine the relevant empirical literature regarding those sources.
The following section reviews supervisor appraisal feedback, peer review feedback, upward
feedback research and self-assessment feedback .

Traditional Feedback Methods

Supervisor Appraisal

Research has shown that supervisor-only feedback tends to be inflated, time-consuming, and
does not differentiate levels of performance nor motivate employees to improve (Edwards &
Ewen, 1996). Supervisor appraisals and multirater feedback share common problems, such as
questionable validity and rating bias (e.g., response consistency, leniency, halo and stereotyping)
(Borman, 1974). However, supervisor assessments have additional problems, including
favoritism, politics, and varying degrees of care when making assessments. No one enjoys being
the bearer of bad news and that includes supervisors, who sometimes skirt issues related to
employee weaknesses and poor performance. Generally, supervisors avoid doing evaluations for
as long as they can, and eventually provide only flat, non-specific information (Edwards & Ewen,
1996).

Also, supervisors may have insufficient opportunity or motivation to observe employee
performance. Schneier & Beatty (1978) found that supervisors may draw inferences about one’s
ability to lead or manage based on the productivity of that person’s work unit, as opposed to
actually observing that person’s individual behavior. While productivity of the work unit indicates
to the supervisor what is getting done, it does not indicate how things are getting done. How
things are being done may, or may not, mesh with the vision and values of the organization as a
whole.

Traditional performance appraisal and the multirater feedback process differ in two important
ways. First, performance appraisals are conducted primarily for evaluation and have,
organizational consequences (e.g., pay raises, promotions, or transfers). A multirater assessment
is usually conducted to provide guidance for employee development and performance
improvement. Secondly, supervisory ratings provide only one source of evaluation data, whereas
multirater feedback provides data from multiple sources. Therefore, it is unlikely that a single
person can have undue influence in the feedback the employee receives. Multirater feedback



Technical Report 1076

Multirater Assessment Process—
A Literature Review

Angela Karrasch
Kansas State University
Consortium Research Fellows Program

Stanley M. Halpin
U.S. Army Research Institute

S. Delane Keene
U.S. Army Research Institute

November 1997 iq Q 8047028

DTIC QUALITY INSPECTED &

A United States Army Research Institute
\* for the Behavioral and Social Sciences

Approved for public release; distribution is unlimited.


Thornton, 1980; and Wohlers & London, 1989). Furthermore, a literature review conducted by
Mabe and West (1982) indicates that the average correlation between self-ratings and the
accuracy criteria was small. Additionally, Shrauger and Schoeneman (1979) reviewed 50 studies
on the congruencies between self and others’ evaluations and found that about one-half of the
studies showed no correlation at all, and the majority of the studies showed either significant but
low correlations or ambiguous results. Additionally, their research showed modest to strong
correlations between individuals’ perceptions of themselves and the way they assumed others saw
them. Finally, a significant correlation was found between individuals’ views of how others saw
them and how others actually saw them.

Why does self-assessment inaccuracy occur?

According to Ashford (1989) self -assessment inaccuracy may occur because of three common
difficulties individuals experience during the self-assessment process. First individuals must be
able to obtain the information necessary to assess their performance accurately and they must
interpret this information honestly. Additionally, they must balance the desire to understand what
they should be doing and how well they have done with the costs to the ego that they suffer by
hearing negative feedback. Furthermore, individuals must obtain standard feedback information
within a social environment in which information seeking may have particularly negative symbolic
meaning as a sign of weakness or insecurity.

Accurately rating one’s self is not an easy task. However, supplementing self-assessments
with feedback from others should increase self-awareness of strengths and weaknesses.
Consistent with the symbolic interactionist perspective, research by Van Velsor et. al. (1991),
indicates that one month after attending a management development program and receiving initial
feedback, about 80% of managers modified their self-assessment in the expected direction on one
or more scales after feedback from others.

Implications of inaccurate self-assessments

At least two studies (Bass & Yammarino, 1991; and Atwater & Yammarino, 1992) indicate
that inaccurate self-raters tended to be poorer performers than people who rated themselves as
others rated them. According to Bass and Yammarino (1991), individuals with inaccurate self-
assessments will misdiagnose their strengths and weaknesses. Both inflated and deflated self-
assessments can adversely affect job-relevant decisions. Inflated self-assessments are associated
with career derailment (McCall & Lombardo, 1983). Research indicates that derailment may
result because those with inflated self-assessments have self-aspirations and expectations which
exceed others’ perceptions of their capabilities. Conversely, those who under-estimate their
abilities and skills will tend to set a low aspiration level and will underachieve (Bandura, 1982).
According to Dweck and Leggett (1988), the effects of deflated self-assessment may be mediated
by how individuals view a task - as either a performance goal or as a learning goal. Individuals
with performance goals seek favorable judgments of their competence; whereas, individuals with
learning goals seek to increase their competence. Individuals with low self-assessments and
learning goals are more likely to put effort into a task than the individuals with low self-
assessment and performance goals.



Self Assessment.

In order to succeed in an organization, individuals must continuously self-assess their
performance, skills and potential (Ashford, 1989). Several studies indicate the self to be the most
available and trustworthy source of feedback (Greller & Herold, 1975; Hanser & Muchinsky,
1978). Simply participating in a self-evaluation makes one increasingly self-aware, more
accepting of feedback from others, and more committed to goal setting (Ashford, 1989).
Additionally, self-ratings can promote personal development, improve communication between
supervisors and subordinates, and clarify differences of opinion between supervisors and other
managers (Cummings & Schwab, 1973).

Carver & Scheier (1981) suggest that control theory provides the framework for
understanding the self-regulation processes. This theory proposes that individuals survive by a
continual process of matching their behavior to a goal or standard. Goals are obtained when
individuals accomplish the following tasks: 1) setting standards for their behavior, 2) detecting
discrepancies between their behavior and those standards (self-assessment), and 3) enacting
behaviors to reduce these discrepancies.

Ashford (1989) adds two self-assessment tasks to this list for the individual within an
organization. Specifically, individuals must be able to assess whether the standards they have
chosen to guide their behavior actually enable goal attainment. Secondly, individuals must
develop proficiency in seeing and assessing their behavior in a manner consistent with how others
perceive and evaluate it. Effective self-regulation in organizations requires attention to issues of
validity of one’s self-assessment with respect to the assessments of others (Ashford, 1989).

Another perspective, taken by symbolic interactionists, argues the relevance of others’
influence in the assessment of self (Cooley, 1902; Mead, 1934). The basic tenet of their theory is
that one develops a self-concept and makes self-assessments based on his/her beliefs about how
others perceive and evaluate them. Accordingly, one is better able to predict and control social
interactions if they are able to see themselves as others see them (Mead, 1934).

Understanding and anticipating others’ assessments is crucial for an individual working within
an organization. Given that only the individual is fully aware of any situational constraints on
his/her behavior, self-assessment could be more accurate than the subjective ratings of others, but
it is the other’s assessments which have important consequences for the individual. It should be
noted that most research regards other’s ratings as the objective criterion (i.e., one is an accurate
self-rater only if his/her ratings match the ratings given by others).

Most research on self-assessment indicates that compared to ratings by others, self-
assessments are often inflated, unreliable, invalid, biased and inaccurate (Yammarino &
Atwater,1993). Van Velsor, Ruderman, and Young (1991) reported that only 10% of managers
studied saw themselves as others saw them. They also found that overrating self seems to be the
most common profile. Prior research indicates that regardless of the dimension being measured or
the sources of other ratings, the average self-ratings tend to be more favorable than other’s ratings
( Ashford, 1989; Harris & Schaubroeck, 1988; Landy & Farr, 1980, Mabe & West, 1982;



MULTIRATER ASSESSMENT PROCESS - A LITERATURE REVIEW
Introduction

The United States Army faces dramatic organizational challenges brought about by global
political changes, increased operating tempo, constrained resources and undefined missions
(Horner, 1995). Similar challenges in the corporate world have emphasized the need for quick
and efficient adaptation, more teamwork, more accountability, and improved performance at less
cost. In response to quickly changing needs, many organizations are turning to an employee
evaluation and feedback process which helps to direct employee professional development while
promoting the organizational culture most conducive to meeting organizational needs. This
evaluation process is the multirater feedback process, which involves receiving performance
feedback from multiple perspectives (i.e., from subordinates, peers, self, supervisors and possibly
customers).

The purpose of this paper is to report research findings relevant to the multirater feedback
concept and to discuss the implications of this body of research for professional development of
Army leaders. The evolution of the multirater process will be reviewed. The multirater method
will be compared to more traditional methods of assessment, and benefits and objectives will be
discussed. Finally, recommendations and caveats for implementation within the Army
professional development system will be discussed.

Evolution of the Multirater Feedback Concept

The multirater feedback process, sometimes referred to as a 360-degree assessment, is rooted
in both the developmental and the performance appraisal fields. The use of performance
appraisals began in the late 1800’s as production processes became mechanized during the
Industrial Age. At first, supervisors used trait checklists for employee performance appraisals
These eventually evolved into various scales and behavioral descriptions which were designed to
improve the quality of such appraisals. During the 1960’ s and 70’s the most prevalent form of
performance appraisal was a comparison of individual results with organizational goals and
strategies (i.e., management by objectives). The use of supervisor feedback for the explicit
purpose of employee personal and career development did not become popular until the late
1980’s. Up until this point, supervisor feedback was mainly perceived as evaluative, not
developmental (Edwards & Ewen, 1996).

The total quality management (TQM) initiative is most commonly credited with originating the
use of multirater feedback for developmental appraisals. This initiative took organizational
surveys and extended them into division level surveys, then department level and finally service
representative level, (i.e., customer service) surveys. Now multirater feedback from subordinates,
peers, self, superiors and customers is used by organizations that have become concerned with
career development, diversity management, fair reward decisions, accurate and valid performance
measures, and legal protection for decisions relating to personnel.



Van Velsor, E., Ruderman, M., & Young, D. (1991). Enhancing self-objectivity and performance
on the job: The role of upward feedback. Paper presented at the Society for Industrial &
Organizational Psychology, St. Louis, Missouri.

Van Velsor, E., Taylor, S., & Leslie, J. (1993). An examination of the relationships among self-
perception accuracy, self-awareness, gender and leader effectivenss. Human Resource

Management, 32, 249-263.

Wilson, C.L., O’Hare, D., & Shipper, F. (1990). Task cycle theory: The processes of influence. In
Clark, K.E., & Clark, M.B. (Eds.), Measures of leadership. West orange, NJ: Leadership
Library of America.

Wohlers, A.J., & London, M. (1989). Ratings of managerial characteristics: Evaluaion difficulty,
coworker agreement, and self-awareness. Personnel Psychology, 42, 235-261.

Yammarino, F., & Atwater, L. (1993). Understanding self-perception accuracy: Imlications for
human resource management. Human Resource Management, 32, 231-247.

16


Technical Report 1076

Multirater Assessment Process—A Literature Review

Angela Karrasch
Kansas State University
Consortium Research Fellows Program

Stanley M. Halpin
U.S. Army Research Institute

S. Delane Keene
U.S. Army Research Institute

Fort Leavenworth Research Unit
Stanley M. Halpin, Chief

U.S. Army Research Institute for the Behavioral and Social Sciences
5001 Eisenhower Avenue, Alexandria, Virginia 22333-5600

November 1997

Amny Project Number Manpower and Personnel
20363007A792

Approved for public release; distribution is unlimited.

Preceding Page Blank
iti



Research Institute developed the Leader Azimuth Check which includes items based directly on
new leadership doctrine. For instance, the new doctrine indicates that planning and organizing is
of critical importance to the Army. These competencies are defined with statements such as:

“develops effective plans to achieve organizational goals”, “anticipates how different plans will
look when executed”, and “sets clear priorities”.

Finally, considering that individuals generally receive feedback only on the competencies
specified by the instrument, a multirater assessment instrument can determine and potentially
restrict the feedback received. Even in cases where free-text input is allowed, written evaluative
comments will tend to address the competencies contained in the instrument. For example, if
there are no items concerning an individual’s integrity, then no feedback on this topic will be
generated for the individual receiving feedback.

2. Involvement of the target audience

In order to + A cross-functional set of employees are surveyed
maximize the relevance and + Functional users review the survey
utility of a multirater + The survey is pretested to assess readability,

completeness, and clarity

instrument, all involved « The survey is refined based on user experience

members should participate
in its development. Those
being evaluated by the standards they set may have especially insightful input. Additionally, the
entire 360 process should involve all members of the organization including top executives. The
organization should rely on research-based protocol for collecting and scoring data. The selection
teams (i.e., the group of people providing feedback) should consist of at least four respondents in
addition to the immediate supervisor and self. Finally, an understandable process that is
communicated clearly to all is a critical feature to the success of a multirater assessment program.

3. Ensuring confidentiality

During the data collection process, + A source outside of the organization
anonymity is an important safeguard ensuring scores the assessment and
accuracy of responses. Anonymity is utilized so completes the feedback report
that the source of an individual’s feedback is not + Feedback is provided only when
identifiable in any manner by that individual. If there is more than one respondent

respondents do not feel confident that their responses will remain confidential, they may provide
uniformly inflated evaluations, or they may refuse to participate at all. Results from a follow up
survey, after a multirater feedback procedure was implemented, indicated that 24% of
respondents would have rated their bosses differently if the ratings had not been anonymous
(London et. al., 1990), The end result of not protecting anonymity is that those who provide the
feedback do not trust the process, and those who receive the feedback do not view the feedback
as credible (Edwards & Ewen, 1996).



Enhancing the Effectiveness of a Multirater Feedback System

Organizations wishing to boost the effectiveness of a multirater feedback program have several
options which may include: continuously reviewing the feedback, providing feedback in the form
of written responses to open ended questions, helping recipients of feedback develop an
understanding of what is motivating their behaviors, and examining an employee’s personal and
career history (Kaplan, 1993).

If an organization decides to utilize one of these interventions, caution must be practiced to
protect the employee. Organizations must choose the intervention most conducive to their
climate and structure. While delving into an employee’s personal and career history may seem a
viable option in one organization, it may seem intrusive in another.

Impact of Multirater Feedback Programs

Multirater feedback programs are based on the assumptions that (a) multirater performance
evaluations provide different perspectives from which to detect discrepancies between how we
view ourselves and how others view us, (b) awareness of discrepancies enhances self-awareness,
and (c) enhanced self-awareness facilitates performance. However, despite the growing
popularity of multirater feedback programs, there is very little research testing these assumptions.

In one study, Hazucha, Hezlett & Schneider (1993) attempted to evaluate the impact of
multirater feedback as a management development intervention. They had managers complete a
management skills profile to measure skills and to give feedback. Two years later, they conducted
a follow-up with 48 of those managers. They reported that managers’ skills increased following
multirater feedback, and that self-other agreement was greater two years after the initial multirater
process. Additionally, they reported that management skills were related to later advancement.
They concluded that multirater feedback is an effective management development tool. Hazucha
et. al. (1993) also administered a second survey and found that the development activities which
were most strongly related to skill development after multirater feedback were 1) reviewing plans
and progress at least quarterly, 2) obtaining input into development plans from co-workers, and 3)
receiving coaching and feedback.

London and Wohlers (1991) conducted an examination of the effects of an upward feedback
intervention on profile agreement (which is the agreement between self-ratings and averaged other
ratings). Profile agreement was fairly low across the sample, however it improved over a one-
year period of time. These results indicate that feedback from others raises awareness of
strengths and weaknesses.

While these two studies are a step in the right direction, the absence of a control or
comparison group makes it difficult to determine whether the performance increases occurred as a
result of receiving feedback. An additional study by Smither, London, Vasilopoulos, Reilly,
Millsap, and Salvemini (1995), examined the impact of implementing an upward feedback
program by comparing managers who received individualized feedback to managers who did not.
These researchers found that, based on subordinates’ ratings, managers whose initial level of

10



REPORT DOCUMENTATION PAGE

1. REPORT DATE (dd-mm-yy) 2. REPORT TYPE 3. DATES COVERED (from. . . to)
1997, November Final July 1996-November 1997
4, TITLE AND SUBTITLE Sa CONTRACT OR GRANT NUMBER
Multirater Assessment Process--A Literature Review
3b. PROGRAM ELEMENT NUMBER
0603007A
6. AUTHOR(S) 5c. PROJECT NUMBER
A792
Angela Karrasch (Kansas State University), Stanley M, Halpin (ARI), | 4. TASK NUMBER
and S. Delane Keene (ARI) 1122
Se, WORK UNIT NUMBER
HO1
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT NUMBER
U.S. Army Research Institute for the Behaviorial and Social Sciences
Fort Leavenworth Research Unit
Building 90, McClellan Avenue
Ft. Leavenworth, KS 66027
9, SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10, MONITOR ACRONYM
U.S. Army Research Institute for the Behavioral and Social Sciences
ATIN: TAPC-ARI-RK ARI
5001 ne meine 11, MONITOR REPORT NUMBER
an 3!
Alexandria, 600 Technical Report 1076

142. DISTRIBUTION/AVAILABILITY STATEMENT
Approved for public release; distribution is unlimited.

13, SUPPLEMENTARY NOTES
Prepared in conjunction with the Leadership Research and Assessment Division of the Center for Army Leadership (CAL).

14, ABSTRACT (Maximum 200 words):

This report is a review of the literature associated with multirater assessments. An analysis of this literature indicates that
multirater assessment systems may be a useful tool for promoting accurate self-assessment of strengths and weaknesses among
Army personnel. Furthermore, multirater assessments encourage developmental growth in areas relevant to leadership and
reinforce organizational values.

15. SUBJECT TERMS
Multirater Feedback 360 Degree Assessment

19. LIMITATION OF j 20. NUMBER 21. RESPONSIBLE PERSON

ABSTRACT OF PAGES (Name and Telephone Number)
16. REPORT 17, ABSTRACT | 18. THIS PAGE

Unclassified | Unclassified | Unclassified | Unlimited 22

‘Standard Form 298



U.S. ARMY RESEARCH INSTITUTE
FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Directorate of the U.S. Total Army Personnel Command

EDGAR M. JOHNSON
Director
SS

Technical review by

Robert E. Solick
Douglas K. Spiegel

NOTICES

DISTRIBUTION: Primary distribution of this report has been made by ARI. Please address
correspondence concerning distribution of reports to: U.S. Army Research Institute for the
Behavioral and Social Sciences, ATTN: TAPC-ARI-PO, 5001 Eisenhower Ave., Alexandria,
Virginia 22333-5600.

FINAL DISPOSITION: This report may be destroyed when it is no longer needed. Please do not
return it to the U.S. Army Research Institute for the Behavioral and Social Sciences.

NOTE: The findings in this report are not be construed as an official Department of the Army
position, policy, or decision, unless so designated by other authorized documents.



Another variable possibly mediating the relationship between deflated self-assessment and
performance is task understanding. Underestimation of one’s understanding of the task may lead
to excessive information-seeking, which takes away from the time to complete the task.
Overestimation of task understanding may lead to a failure to perform the task correctly.

Implications of accurate self-assessments

Accurate self-assessment is associated with numerous benefits for the individual and the
organization. For instance, Bandura (1977), proposed that self-assessment of aptitude and skills
influence an individual’s decisions about how to allocate his/her efforts, how intensely to work,
and how long to persist in the face of challenges.

Individuals with accurate self-assessments are in a better position to take corrective actions,
such as increasing effort or changing task strategies when needed. If individuals believe they are
doing fine, when in fact, they are not, they are unable to make the necessary corrections to their
behavior (Nilsen & Campbell, 1993).

McCauley and Lombardo (1990) found that self-awareness was positively related to superiors’
assessments of promotability - the more accurate the self-assessment, the more likely an individual
would be to be promoted. According to Bass and Yammarino (1991) naval officers with more
accurate self-perception attained higher ranks and were rated as more promotable by their
superiors

A process like three hundred sixty degree assessment, which is thought to enhance self-
assessment accuracy, offers an improved alternative. Although any of the traditional feedback
methods have associated benefits, most people find the combined feedback from their supervisors,
peers, and subordinates highly motivating. Individuals can discount feedback from a supervisor
(or any one source), but hearing the same constructive criticism from peers and subordinates with
whom he/she has personally worked is more likely to motivate improvement.

Benefits and Objectives of the Multirater Feedback Process

A properly implemented multirater assessment enhances employee development and provides
an organizational culture that defines itself as: more empowering, team oriented, productive,
participative, and as offering equal opportunity for it’s members to succeed and advance.
Researchers have documented the numerous advantages of multiple rater systems, including the
following: enhanced ability to observe and measure various job facets (Borman, 1974), greater
reliability, fairness, and ratee acceptance (Latham & Wexley, 1994), enhanced communications
and performance ( London, Wohlers & Gallagher, 1990; Bernardin & Beatty, 1987), more input
for merit evaluation and compensation adjustment (McEvoy & Buller,1987) and improved
defensibility of the performance appraisal program from a legal standpoint (Bernardin & Beatty,
1984).



4. Training for participants
Training in providing and receiving feedback is critical to the success of a multirater

feedback program. Besides the
obvious value of having
organizational members accurately
completing forms, there are added
benefits of training. Training can
highlight the advantages of a 360,
thereby instilling confidence in the
process. Moreover, training sets

+ An introduction to the concept of multirater feedback is
provided to all employees

+ An explanation for why the organization is adopting the
process is provided

* The various steps and safeguards are clarified

+ Employees are trained in how to complete the feedback
surveys, how to interpret their results, how to accept
feedback constructively and in how to follow up with
personal plans of action

users at ease with the process.

5. Continuing review of process and instrument

Research indicates that the process for conducting a 360 degree assessment must be
formalized to ensure fairness and accuracy. Informal systems are easy and fast to implement,
however they fail to provide sufficient safeguards to ensure fairness and are more likely to
multiply rather than reduce error in the system. (For a detailed discussion of safeguards that can

be practiced with a 360, see Edwards & Ewen, = The process provides sufficient safeguards

1996). The dangers of an informal system
involve reported false information that may
appear credible and which may inappropriately

+ An appeals process is included (especially if
the 360 is used for performance appraisal
versus developmental purposes only)

* The process is formal

impact an employee’s self-perception and/or
career. Furthermore, an informal process with

its associated problems may negatively
influence one’s willingness to participate in a fair and formalized, or structured, 360 feedback
system at a later date (Edwards & Ewen, 1996).

6. Follow-up

Follow-up is a critical factor in the
professional development of members of an
organization. The real work begins when
individuals receive their feedback. The
multirater feedback process can provide
individuals with the information they need to
set personal goals and standards, to improve
their skills and keep their career on track. Once feedback is received and properly interpreted, the
multirater feedback system is still not completed. There must be follow-through. Follow-through
means more than creating objectives and action steps based on feedback. According to Kaplan
(1993), follow-through should be viewed as a project, and as such, it should be managed actively.
There are various methods for managing the process and thereby boosting the effectiveness of the
multirater assessment methodology.

+ Individuals are provided with the
information needed to set personal
action plans

* Individuals are encouraged/coached as
they follow through with their personal
action plans

+ Individuals are held accountable for
executing their personal action plans