A Monte Carlo Comparison of Ten Item Discrimination Indices.

At least eight studies have appeared in the literature comparing discrimination indices commonly used for item analysis. Some of these have followed an analytical approach (Adams, 1960a; Bowers, 1972), while others have based their comparisons and subsequent conclusions on empirical data (Adams, 1960b; Aleamoni & Spencer, 1969; Engelhart, 1965; Guilford & Lacey, 1947; Hales, 1972; Oosterhof, 1976). Four of these studies (Aleamoni & Spencer, Engelhart, Hales, and Oosterhof) concluded that the indices they compared were virtually indistinguishable in terms of item selection efficiency, while the remaining four found differences large enough to recommend against the use of certain indices. Two factors seem to have contributed to a lack of consensus in this area of research: (1) researchers have used different methods for comparing indices, and (2) item sets used in different studies probably have not been comparable in terms of item precision levels and difficulty levels. With regard to the first factor, we believe the most appropriate method of evaluation is one which compares indices on the basis of validities for tests constructed by using those indices. This contrasts with the method used by some researchers which employs factor analyses of the inter-index correlation matrix. The major deficiency of the factor analytic approach is its subjectivity. Discrimination index coefficients are correlated with a "discrimination factor," and these correlations are used as the basis for comparisons. We find it difficult to assess the importance of any resulting differences, since we fail to see the exact relationship between these correlations and test validities achieved by using the corresponding index for the item analysis. Concerning the second factor above, we believe that any study should include a wide variety of item sets to insure fairly extensive generalizability of results. None of the studies mentioned above meet this requirement. Oosterhof, for example, subjected 50 verbal analogy test items to item analysis, selecting his items from a widely used published test (DAT). We have strong reservations concerning the validity of conclusions based on item analyses that use preanalyzed and prescaled items. However, our greater concern focuses on the problems which seem inherent in using any one set of items to represent the diverse sets that are generally analyzed by researchers (a problem acknowledged by Oosterhof). A Monte Carlo approach was selected as the best method for satisfying the requirements stated above, since test validities can be estimated with a high degree of accuracy using a larger computer generated validation sample. In addition, characteristics of item sets (precision, difficulty, etc.) and other item analysis factors can be easily manipulated, allowing study across a wide variety of item analysis situations. This in turn allows fairly extensive generalizability of results. In this study 10 indices were compared: biserial, point biserial, rank biserial, phi based on the median split, the Camp approximation to the tetrachoric (Castellan, 1965), Ivens' index (Ivens, 1971), the index proposed by Brennan (1972), and three examples of the simple discrimination index, based on the upper and lower scoring 1/2, 1/3, and i/lo on the total test.