What Do Self and Peer Ratings Really Measure

Using Kenny’s (1994) social relations model (SRM), data from 29 organizational teams who completed developmental performance ratings of themselves and other team members were analyzed. The current study (a) estimated the amount of variance associated with the ratee, rater, and relationship components, (b) projected dependability estimates under different measurement conditions, and (c) explored the social and relational processes associated with group dynamics and functioning. Results indicated that the relationship component accounted for significantly more variance than the ratee component, which in turn accounted for significantly more variance than the rater component. In general, results also indicated dyadic and generalized reciprocity for some performance dimensions. Further, self-ratings were related to how one rates, and is rated by, others. Social Relations Model 3 What do Self and Peer Ratings Really Measure? Rapid changes in how organizations are structured and function have forced human resource practitioners to adjust their practices to meet the demands of today’s organizations. Current trends indicate that organizations are becoming flatter and more decentralized (Murphy & Cleveland, 1995) and that many organizations have shifted from individually-based work to team-based work (Fedor, Bettenhausen, & Davis, 1999; Reilly & McGourty, 1998). Associated with these changes have been changes in how employee performance is measured, developed, and evaluated. In particular, many organizations now use self and peer ratings as part of their performance management systems (Church & Bracken, 1997). For team based-work, peer and self-ratings may be the only sources capable of providing relevant information regarding group processes and individual contributions (Fedor et al., 1999). Peers may be an especially valuable source of performance information because peers generally work closely together, interact frequently, and have the opportunity to observe both task and interpersonal behaviors (Murphy & Cleveland, 1995). This proximity and frequency of interaction is predicted to increase rating accuracy (e.g., Wherry & Bartlett, 1982). However, it is precisely this proximity and frequency of interaction that also raises concerns over the use of peer ratings. Those skeptical of using peer ratings suggest that such ratings will be contaminated with friendship biases (Doll & Longo, 1962; Landy & Farr, 1983) and that peers may be unwilling to differentiate among one another in an attempt to maintain harmony within the group (Murphy & Cleveland, 1995). Ironically, although these frequent and close interactions have been argued to both improve and hinder the quality of peer ratings, such interactions have been largely neglected in the extant research (Hennen, 1997). Social Relations Model 4 Early research on performance ratings focused on rating scale format in an attempt to increase the psychometric properties of the scales. Landy and Farr (1980) called for a moratorium on rating scale research and instead encouraged researchers to consider the cognitive processes of the raters. In response, research quickly shifted to studying cognitive processes, but similar to the research on rating scale format, this cognitive stream of research has been criticized on the grounds of only making minor contributions (Murphy & Cleveland, 1995) and of not advancing organizational practices (Banks & Murphy, 1985). As Murphy and Cleveland (1995) note, what appears to be missing in these earlier approaches, and what may explain their limited utility, is a consideration of social and contextual factors that influence performance ratings. Specifically, previous models have neglected to recognize that appraisals take place in a dynamic context (Murphy & Cleveland, 1995) in which rater and ratee interactions and relationships are important considerations. As Vance, Winne, and Wright (1983) note: “Reproduceable level effects attributable to raters or ratees only partially explain ratings. A dyad-based analysis capable of representing rater-ratee interactions may be necessary to fully explicate the process” (p. 619). As called for by Vance et al. (1983), the current study focuses not only on rater and ratee effects, but also on rater-ratee interactions and relationships. In doing so, the current study addresses three issues raised by Murphy and DeShon (2000a) in need of future research to more fully understand what performance ratings represent. Specifically, the first objective is to use Kenny’s (1994) Social Relations Model (SRM) to investigate self and peer ratings. Unlike classical test theory which only decomposes variance into true and error variance, SRM analyses permit target (ratee), perceiver (rater), and relationship (rater by ratee interaction) variance components to be estimated separately. As noted by Murphy and DeShon (2000a), a major Social Relations Model 5 limitation of past research with respect to variance partitioning has been its inability to disentangle the perceiver effect from the relationship effect. Previous research (e.g., Greguras, Robie, Schleicher, & Goff, 2003), for example, has observed that a substantial amount of variance could be attributed to some function of the rater but was unable to identify if this variance was primarily attributable to the rater main effect or rater by ratee interaction effect. Because the current study is able to separate these effects, it is able to more precisely identify and estimate the amount of variance attributable to each of these variance components. The second primary objective of the current study is to use the estimated variance components to project dependability (reliability) estimates under various measurement conditions. As Murphy and DeShon (2000a) note: “. it would be interesting to know the effect of including or excluding the rater variance in the error term consistent with the distinction between absolute and relative error in generalizability theory. This information could be used to inform the researcher about the exchangeability of raters in the rating process or used to determine the number of raters needed to achieve a particular level of generalizability for a particular research question” (p. 893). Previous studies were largely unable to provide these estimates because of their inability to separate rater from relationship effects. The third primary objective of the current study is to merge and extend the organizational science literature on self and peer ratings with the social psychological literature on interpersonal perception. As discussed below, the SRM analyses estimate variance components attributable to the rater, the ratee, and their dyadic relationships. These variance components can then be correlated with one another and with self-ratings to provide information about the social and cognitive processes involved in the rating process. As Murphy and DeShon (2000a) note, the nested designs of the vast majority of studies on performance ratings make it virtually impossible Social Relations Model 6 to sort out rater effects from relationship effects, and therefore, it is largely unknown if, and how, these effects might be correlated with one another. The current study overcomes this limitation by using a round-robin design to estimate and correlate the different variance components. We see this as a major strength of the paper as such analyses will allow for a greater understanding of self and peer ratings and rating processes. An overview of the SRM is presented next, followed by a discussion of its usefulness within the context of studying self and peer performance ratings. Social Relations Model The SRM developed out of the person-perception literature and provides both a theoretical basis and a statistical tool to assess the interdependencies among ratings (Kenny, 1994; Warner, Kenny, & Soto, 1979). When individuals evaluate the performance of their peers, their ratings likely are dependent on one another’s ratings. For example, in a team setting, Jim’s evaluations of Eric likely are related to Eric’s evaluations of Jim. As Hennen (1997) notes, previous research on peer evaluations has ignored this nonindependence of data. By ignoring these interdependencies, results from typical statistical methods may be distorted (for a discussion, see Kenny & Judd, 1986), and importantly, meaningful information about the interdependencies among peers is lost. It is this very interdependency that makes the study of teams interesting and that is the focus of the current study. After all, if team members were not interdependent, members would merely represent a collection of individuals (Marcus, 1998). One advantage of the SRM is that it does not require an assumption of independence, but rather, the nonindependence among peers is of interest and is analyzed (Kenny, 1994; Marcus & Kashy, 1995). Social Relations Model 7 The SRM is a special case of generalizability theory (Cronbach, Glaser, Rajaratnam, & Nanda, 1972) and decomposes perceptual variance on a construct into target effects, perceiver effects, and relationship effects. Target effects are considered true score variance and reflect a target’s tendency to elicit similar ratings from all raters. Perceiver effects are rater main effects and reflect a rater’s tendency to rate all persons similarly (e.g., leniently). Relationship effects are factors unique to specific dyads. To estimate target, perceiver, and relationship effects using the SRM, multiple perceivers must rate multiple targets. This requirement generally is met by employing a round-robin data collection design in which each member of a group rates, and is rated by, every other member. The round-robin design is distinct from more typical generalizability theory (GT) models in which multiple raters evaluate a given target but the given target does not evaluate those raters (Cronbach et al., 1972). The variance partitioning is conducted on e