Variability in Pretest-Posttest Correlation Coefficients by Student Achievement Level. Washington, DC: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences

State assessments are increasingly used as outcome measures for education evaluations. The scaling of state assessments produces variability in measurement error, with the conditional standard error of measurement increasing as average student ability moves toward the tails of the achievement distribution. This report examines the variability in pretest-posttest correlation coefficients of state assessment data for samples of low-performing, average-performing, and proficient students to illustrate how sample characteristics (including the measurement error of observed scores) affect pretest-posttest correlation coefficients. As an application, this report highlights how statistical power can be attenuated when correlation coefficients vary according to sample characteristics. Population achievement data from four states and two large districts in both English/Language Arts and Mathematics for three recent years are examined. The results confirm that pretest-posttest correlation coefficients are smaller for samples of low performers, reducing statistical power for impact studies. We also find substantial variation across state assessments. These findings suggest that it may be useful to assess the pretest-posttest correlation coefficients of state assessments for an intervention’s target population during the planning phase of a study. NCEE 2011-4033 U.S. DEPARTMENT OF EDUCATION This report was prepared for the National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences (IES), under Contract ED-04-CO-0112/0006. Disclaimer The Institute of Education Sciences at the U.S. Department of Education contracted with Mathematica Policy Research to develop a report documenting the variability in the pretestposttest correlation coefficients estimated from samples of students who had taken state proficiency assessments and showing how attenuation in pretest-posttest correlation coefficients could affect statistical power in education experiments targeted at low-performing students. The views expressed in this report are those of the authors, and they do not necessarily represent the opinions and positions of the Institute of Education Sciences or the U.S. Department of Education. U.S. Department of Education Arne Duncan Secretary Institute of Education Sciences John Q. Easton Director National Center for Education Evaluation and Regional Assistance Rebecca A. Maynard Commissioner September 2011 This report is in the public domain. Although permission to reprint this publication is not necessary, the citation should be the following: Cole, Russell, Joshua Haimson, Irma Perez-Johnson, and Henry May. “Variability in PretestPosttest Correlation Coefficients by Student Achievement Level. ” NCEE Reference Report 2011-4033. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education, 2011. This report is available on the IES website at http://ncee.ed.gov. Alternate Formats Upon request, this report is available in alternate formats such as Braille, large print, audiotape, or computer diskette. For more information, please contact the Department’s Alternate Format Center at 202-260-9895 or 202-205-8113.