Representative Bureaucracy: An Estimation of the Reliability and Validity of the Nachmias-Rosenbloom MV Index

During the past ten years, the number of studies using the Measure of Variation (MV) index to assess minority incorporation (or representation) in the federal work force has increased. The MV index has been used primarily to gauge the extent to which employment in federal bureaucracies is representative of U.S. minority groups (Meier, 1993). Along with assessing the degree to which the federal bureaucracy is representative of U.S. ethnic and racial minorities, the MV index has been used to predict minority incorporation (Kellough, 1990; Kim, 1993). However, according to some researchers, the MV index lacks an adequate degree of reliability under certain conditions (Kim, 1993). This assertion questions whether the MV index is appropriate for measuring representative bureaucracy and whether the results of recent empirical work are statistically valid. Although researchers have questioned the reliability of the MV index, little published empirical evidence exists that indicates the degree to which the MV index is reliable, valid, and invariant to ordering (or sequence) effects and data transformations. The degree to which the MV index is reliable and valid is important because of its measurement and statistical implications. From a measurement standpoint, the MV index's degree of reliability affects the extent to which it is valid and the extent to which accurate inferences can be made about the representation scores. Nitko (1983) states that a necessary condition for a score to be valid is that it have an acceptable degree of reliability. Nitko also says that when scores are unreliable they represent large measurement errors rather than "true" observations. From a statistical perspective, the degree to which the MV index is reliable effects the magnitude of the standard errors of estimates. According to Cook and Campbell (1979; 43), "measures of low reliability.cannot be depended upon to register true changes" because unreliable measures tend to inflate standard errors of estimates. In turn, the magnitude to which the standard errors of estimates are unstable effects the degree to which correct inferences can be made. Thus, if the MV index lacks an adequate degree of measurement reliability and validity, the statistical results of recent empirical work in the area of representative bureaucracy are tenuous at best. Empirical work on inequality indices indicates that an index must possess several characteristics if it is to be used with confidence. According to Allison (1978), representation (or inequality) indices should be scale invariant and the degree of inequality should decrease as differences between groups (or individuals) become smaller. The representation indices should also be reliable and valid. Selvanathan and Rao (1994) state that the utility of an index depends upon the level of confidence or degree of reliability that can be attached to the value of the index. Selvanathan and Rao (1994) believe that changes in the value of the index should be reflected by a measure of reliability associated with the index. Finally, Selvanathan and Rao (1994) observe that indices that claim to measure the same phenomena should yield similar and consistent results. Put differently, the values of the representation indices should correlate positively (or negatively) with each other as the phenomena under investigation changes. Mathematicians and statisticians have developed several tests to determine whether representation (or inequality) indices are reliable and invariant to ordering differences and transformations. Fisher (1922) proposed and developed the following tests for selecting an appropriate index: 1. Test of Proportionality - if all the current quantities (or elements) are multiplied by a positive constant c then the new index should be equal to the old index multiplied by c. 2. Element Reversal Test - any change in the order in which the elements are ordered should not alter the value of the index. …