Study of the Feasibility of a NAEP Mathematics Accessible Block Alternative
Author: DeStefano, Lizanne
Date: 2013-08-00
Type: Reports - Research
Subjects: National Competency Tests,Measures (Individuals),Disabilities,English Language Learners,Testing Accommodations,Test Format,Achievement Gains,Test Construction,Error of Measurement,Test Items,Difficulty Level,Mathematics Tests,Test Validity

This paper describes one of the first efforts by the National Assessment of Educational Progress (NAEP) to improve measurement at the lower end of the distribution, including measurement for students with disabilities (SD) and English language learners (ELLs). One way to improve measurement at the lower end is to introduce one or more &quot;accessible&quot; blocks into the NAEP Balanced Incomplete Block Design (BIB). Accessible blocks are defined as blocks that are aligned within the NAEP content frameworks, but designed to provide more information about the abilities and skills of students at the lower end of the distribution. The process of creating the NAEP accessible blocks described in this document began in February 2007 with efforts to define what constitutes an accessible block and to design a process to develop mathematics-accessible blocks that are representative of the NAEP content frameworks. Panels of mathematics content experts and item writers were convened to identify major themes and dimensions that affected item difficulty in mathematics and to develop general strategies for reducing difficulty without compromising content and construct validity. This process culminated in the creation of the Item &quot;Modification Guidelines and Item Modification Procedures&quot;. Using the &quot;Item Modification Guidelines and Item Modification Procedures&quot;, a sample of Grade 4 and Grade 8 item blocks were modified to create accessible blocks. This investigation of NAEP accessible blocks served as a proof of concept study in two important ways. First, the creation, application, and expert review of the Item &quot;Modification Guidelines and Item Modification Procedures&quot; illustrated that it was possible to develop standard procedures for creating items that were less difficult but still adhered to the content framework. Second, the results from the field test of accessible and operational NAEP blocks indicated that it is indeed feasible to construct accessible blocks that are scalable with the main NAEP assessment and that improve measurement precision at the lower end of the NAEP performance continuum. The primary study findings were as follows: (1) Across all groups and subgroups, there were substantial and similar average gains in the percentage correct by block for the accessible blocks compared with the source blocks; (2) There were consistent declines in the number of students omitting items and significant reductions in the percentage of students not reaching items for the accessible blocks compared with the source blocks; (3) All accessible items were scalable, and modified items had similar average discrimination and guessing characteristics (a and c parameter estimates) as the source items, while there were significant reductions in item difficulty (b parameter estimates); (4) For the lowest performing students, the conditional standard error of measurement was significantly lower for students completing two accessible blocks than for those completing two source blocks; and (5) Test information functions for books comprised of two accessible blocks were appropriately targeted to the lower end of the performance continuum. The following are appended: (1) Item Modification Guidelines and Item Modification Procedures; (2) 2007 Expert Panel Members; (3) 2009 Expert Panel Members; and (4) Item Rating Scale.