Sample Selection in Models of Academic Performance

I. INTRODUCTION Virtually all graduate programs in the United States require applicants to submit scores on standardized tests, such as the Graduate Record Examination (GRE) or the Graduate Management Admissions Test (GMAT). The usefulness of these tests as predictors of graduate student performance, however, has never been firmly established. In fact, formal statistical investigations (validation studies) typically find that standardized tests have surprisingly little predictive content. Hanson and Harrell (1985) report a negative correlation between GMAT scores and subsequent compensation, Hansen (1971) reports very little correlation between GRE scores and graduate grade point averages (GPAs) in economics, and Sternberg and Williams (1997) find GRE scores of limited use in predicting performance in psychology. Even a recent study by the GRE Board of the Educational Testing Service (1998) shows relatively small correlations between first-year graduate GPA and general GRE scores. Why do admissions committees pay so much attention to these scores when formal evidence suggests that they have little or no predictive content? In this article, we suggest that a form of sample selection bias makes interpretation of standard validation studies problematic. As Darlington (1998) and Cornell (1998) point out, validation studies use data on students who have matriculated at a particular institution. The admission procedure and acceptance decisions make these data a censored sample from the population of prospective students. Institutions occasionally admit students with atypically low test scores, but only if these students present other, countervailing evidence of high ability. Students with atypically high test scores occasionally will choose to enroll, but only if countervailing evidence of low ability precluded their admission to higher-ranking institutions. Given that both high and low scorers exhibit some other, countervailing evidence on ability, it is not surprising that standardized test scores by themselves are not strong predictors of performance. This article provides a formal model of how the selection process influences the results of standard validation studies. We find that in a competitive market for students, optimal behavior on the part of admissions committees and on the part of applicants drives the simple correlation between test scores and performance toward zero, regardless of the relationship in the population of prospective students. The key features of the model are that institutions select students (and students self-select) based on more information than test scores alone and that a competitive market for students results in institutions enrolling students of relatively homogenous predicted ability. The sample selection process generates a negative correlation between test scores and other observable information, which, as the market becomes more competitive, drives the simple correlation between test scores and performance toward zero. II. A SIMPLE ANALYTIC MODEL Assume that a simple scalar variable, y, measures scholastic ability in the population of potential students to a particular institution. This ability variable will be revealed perfectly (as performance) if the student matriculates but cannot be observed before matriculation. The admissions committee at this and other institutions must make acceptance and rejection decisions based on observable variables that may be correlated with this performance variable. The committee observes characteristics, [x.sub.1] and [x.sub.2], which, for concreteness, we interpret as the score on a standardized test and an index of undergraduate performance, respectively. Assuming joint normality of three variables, y, [x.sub.1], and [x.sub.2], in the population of prospective students, expected ability (and performance) will be linear in the two observable variables, (1) E(y | 1, [x.sub.1], [x.sub.2]) = [mu] + [x. …