Perceptions of Organizational Politics: A Demonstration of the Reliability Generalization Technique

Although validity evidence in its many forms (e.g., content, construct, criterion-related) is an admittedly large part of the psychometric pie, reliability is also a critical piece (Nunnally and Bernstein, 1994). As stated by many psychometricians, "A test may be reliable without being valid, but it cannot be valid without being reliable" (e.g., Aiken and Groth-Marnat, 2006: 97). Therefore, reliability is a necessary condition for validity. There is widespread acknowledgement (Gronlund and Linn, 1990; Henson, 2001; Thompson, 1994; Vacha-Haase, 1998) that reliability is actually a feature of the scores obtained in a specific sample from the use of an instrument, and not a feature of the instrument itself. Specifically, reliability values (such as Cronbach's alpha) obtained using the same instrument may vary from sample to sample. In fact, the APA Task Force on Statistical Inference declared that "reliability is a property of the scores on a test for a particular [italics added] population of examinees" (Wilkinson and APA Task Force on Statistical Inference, 1999: 596). Since researchers have come to realize that "perfect reliability is only a handy fiction" (Nunnally, 1967: 218), a certain amount of measurement error is expected in the social and behavioral sciences. However, poor reliability can result in attenuated construct relationships, such as the zero correlation described by Reinhardt (1996) for scores that are perfectly unreliable on at least one variable. Other detrimental effects of poor reliability of scores include reduced statistical power (Onwuegbuzie and Daniel, 2004). Because reliability is such a central issue in empirical research (e.g., Nunnally and Bernstein, 1994), many (e.g., Henson and Thompson, 2002; Vacha-Haase, 1998; Vacha-Haase et al., 2002) have argued for an increased focus on the topic of reliability in the form of reliability generalization (RG) studies. The purposes of this study were, first and foremost, to demonstrate the technique of reliability generalization so that other management researchers might use it, and secondly to provide researchers investigating perceptions of organizational politics with information that allows them to understand: (1) the typical reliability of scores on the Perceptions of Organizational Politics Scale (POPS) so they can place their own reliability measures in context, (2) the amount of variability in reliability coefficients so they can understand the robustness of reliability estimates, and (3) the sample and/or test elements that systematically contribute to variations in reliability so they can understand how generalizable measures of political perceptions are across diverse samples. Politics are sometimes described as dysfunctional aspects of organization life whereby employees compete for scarce resources usually at the expense of each other (Kacmar and Baron, 1999). The perception of organization-level politics implies that organizational members view life in the organization as tainted by behavior rooted in self-interest (Ferris et al., 1989). Thus, the measurement of phenomena like politics in the workplace is of potential value to organizational intervention efforts and organizational development. To accomplish these goals, this study examines factors affecting variance in score reliability across studies that employ measures of perceptions of organizational politics, using reliability generalization, a meta-analytic framework described by Henson and Thompson (2002), Vacha-Haase (1998), and Vacha-Haase et al. (2002). RELIABILITY GENERALIZATION Reliability generalization as an analytic tool was first described by Kennedy and Turnage (1991) as an extension of validity generalization (Hunter and Schmidt, 1990; Schmidt and Hunter, 1977; Schmidt et al., 1985). However, the technical sophistication of reliability generalization was greatly expanded upon by Vacha-Haase (1998) as she described reliability generalization as an analytic method of characterizing mean measurement error variance and the sources of variability in scores. …