NAEP Validity Studies Panel Responses to the Reanalysis of TUDA Mathematics Scores
Author: Hughes, Gerunda
Date: 2019-10-00
Type: Reports - Evaluative
Subjects: National Competency Tests,Test Validity,Academic Standards,State Standards,National Standards,Urban Schools,Accountability,School Districts,Standardized Tests,Alignment (Education),College Readiness,Career Readiness,Grade 4,Grade 8,Mathematics Achievement,Scores,At Risk Students,Weighted Scores

During the past decade, the NAEP Validity Studies (NVS) Panel has been monitoring, studying, and commenting on potential issues with the validity of the National Assessment of Educational Progress (NAEP) arising from changes that have been brought about by the adoption of rigorous state college and career readiness standards, such as the Common Core State Standards (CCSS). NAEP is meant to be reflective of the entirety of what is taught in the United States, and the many changes to standards in the past 10 years have led to questions about the extent to which NAEP continues to meet this objective. The NVS Panel has conducted several studies to investigate this issue and has found some variations in the alignment between state and NAEP standards across different NAEP grades and subjects. Given the prominent attention that the NAEP results receive, states and participating districts are especially interested in learning how well those results relate to what is taught and tested in schools. States and districts have informally posited that, if the alignment between NAEP frameworks and their own content standards were closer, then their NAEP scores might be higher. Stated another way, there are concerns that NAEP may be underreporting the actual abilities of their students--and trends in achievement--because of some degree of misalignment. When the 2017 NAEP Mathematics TUDA (Trial Urban District Assessment) results were reported and it appeared that student performance trends on NAEP were not similar to student performance trends on the state assessments that were aligned to college and career ready standards, several leaders in the affected TUDAs called for what amounted to a &quot;recount.&quot; The results of student performance on the state assessments from 2013 to 2017 were showing more positive trends than the results of student performance on NAEP during the same period. In the reanalysis discussed here (Dogan, 2019; reproduced in the appendix), the study author noted that most of the negative NAEP trends observed in districts in recent years (i.e., 2015 to 2019) coincide with major changes in states' learning standards and assessments, raising the legitimate question: Can these trends be a function of the differences between NAEP assessment content and states' transition to new college and career readiness learning standards, such as the Common Core State Standards (CCSS), and the corresponding shift in the content of state assessments to be aligned to these newer standards? It was determined that this issue of mismatch trends or the misalignment of results should be examined for at least two reasons: (a) Urban districts are held accountable for students' performance on state assessments that are aligned to state-mandated content standards; however, (b) urban districts are not held directly accountable for performance on NAEP, which historically was regarded by many as the standard against which the adequacy of state assessments is judged. This report serves as a response from the NVS Panel to the analysis conducted by Dogan (2019). Selected panel members were asked to provide comment, and their responses have been edited together into this report. The conclusion of the report is that the secondary analysis done by Dogan for the NAEP TUDA scores is important and worthy of further exploration as part of ongoing efforts to monitor the validity of NAEP. [This report was commissioned by the NAEP Validity Studies Panel.]