Comparison of the Incremental Validity of the Old and New MCAT.

The predictive and incremental validity of -both the Old and New Medical College Admission Test (MCAT) was examined and compared with a sample of over 300 medical students. Results of zero order and incremental validity coefficients, as well as predictipn models resulting from all possible subsets regression analyses using Mallow's C criterion, were subjected to cross-validation analyses by randomly dividing two medical school classes into screening and calibration samples. Results supported the incremental validity,of both the_Old and New MCAT. Coefficients were generally larger for the New than for the Old MCAT. Prediction models of NBME Part I and II performance, comprised of the New Biology and Chemistry subtests and the Old Science and General Information subtests were crOss-validated. Prediction models of clinical evaluation clerkship performance were equivocal. 41, Incremental Validity 1 A number of studies recently have compai.ed the ability of the Qld and the New MCAT to predict stu-dent achievement in medical school. Since the New MCAT was first used in 1978, most studies have focused on the cognitive outcomes of student performance in medical school during the basic science curriculum. As students admitted on the basis of their New MCAT Scores, along with other admission criteria, progress through the medical curriculum, the contribution of the new MCAT in predicting other outcome measures such as clinical performance is beginning to be studied (e.g. Carline, Cullen & Scott, 1982; Carline et al, 1983; Hull, Calhoun & Maxim, 1981). Findings presented in the Association of American Medical Colleges' New Medical College Admission Test Interpretive Manual (1977), as well as the results from local analyses, indicate that the test has strong predictive validity for cognitive outcom8 during the first and second years in medical school: McGuire (1980) investigated the relationship of the New.MCAT to the criterion of class standing at the end of ttie freshman year. Results of a correlation analysis revealed that all of the New MCAT subscales except for Skills Analysis: Reading correlated significantly with class standing (2. .001). Similar results were also found for undergraduate GPA and undergraduate science GPA. Among the group of predictors used to create a revised admissions prediction index, maximum predictability was achieved by undergraduate GPA and the New MCAT Science Problems score, which is calculated from problem-solving items involved in the biology, chemistry and physics components. As Jones and Thomae-Forgues (1981) point out, the predictive ability of the New MCAT in relation to performance in the basic medical sciences is not surprising in view of the heavy emphasis placed in development on the science and medicine-related content relevance of test items. Performance in medical school has long been considered to be related to science achievement; therefore, those who succeed in the first science-based academic quarter have little trouble completing their medical studies (Cullen et al, 1980). Furthermore, Dawson-Saunders and Doolen (1981) and Jones and Thomae-Forgues (1981) discussed the New MCAT's potential value as a predictor of Incremental Validity 2 clinical performance. Due to the increased emphasis on interpretation and problem solving in the new format, they suggested that the new MCAT may result in measures which are more closelyassociated with the information gathering, evaluation, and utilization skiffs required during the clinical experierkce. Hull et al (1981) found validity coefficients of the New MCAT subtests with NBME Part I scores (basic science) to be consistently larger than the validity coefficients of undergraduate grades with NBME I. Car line et al (1983) also found the New MCAT subtests to be superior to grades in validity coefficients for NBME Part II scores (clinical science). In reviewing the success of the New MCAT in predicting medical school performance, Jones and Thomas-Forgues (1982) noted several patterns in the America Medicl College Application Service data set. Among these p terns were the MCAT's ability to predict NBME Part I performance better than undergraduate college grades and that "predictions of medical school course grade performance based on MCAT scores and undergraduate college grades are better than those based on either one alone" (p.6). Friedman and his collegues (1980, 1981) have used the method of incremental validity (Sechrest, 1963) to illustrate the utility of both New and Old MCAT scores in iMproving the amount of variance accounted for in measures of medical student performance beyond that accounted for by other preadmission measures. They found that the explained variance in examination performance during the first two years was proportionately greater when the New MCAT was used in place of the Old MCAT in a stepwise regression analysis after all other admissions variables 'were included. Their analyses also revealed that the New MCAT's incremental predictive power was higher for nationally standardized outcome measures as compared to that for locally prepared achievement tests. When NBME Part I examination scores on the Microbiology and Anatomy subtests were used as criterion measures, the most valuable New MCAT predictors were the Biology, Science Incremental Validity 3 Problems, and Skills Quantitative assessment subtestsi Erdmann (1980) characterized the re s of the "first round" of MCAT studies as encouraging, and called