The Development and Technical Adequacy of Seventh-Grade Reading Comprehension Measures in a Progress Monitoring Assessment System. Technical Report #1102.

This technical report describes the process of development and piloting of reading comprehension measures that are appropriate for seventh-grade students as part of an online progress screening and monitoring assessment system, http://easycbm.com. Each measure consists of an original fictional story of approximately 1,600 to 1,900 words with 20 multiple choice questions assessing students’ literal, inferential, and evaluative comprehension skills. Three answer options are provided for each question: correct, near correct, and far correct. All measures were piloted with seventh-grade students whose teachers responded to the pilot invitation on the official easyCBM website. The data were analyzed using one parameter Item Response Theory (IRT) modeling. Complete results of the pilot testing are presented and discussed. Grade 7 Comprehension – Page 1 The Development and Technical Adequacy of Seventh-Grade Reading Comprehension Measure in a Progress Monitoring Assessment System In this technical report, we describe the process of development and piloting of reading comprehension measures as part of an online screening and progress monitoring assessment system developed in 2006 for use with students in Kindergarten through eighth grade (Alonzo, Tindal, Ulmer, & Glasgow, 2006). We begin with a brief overview of the two conceptual frameworks underlying the assessment system: progress monitoring and developmental theories of reading. We then provide context for how the comprehension measures fit into the full assessment system. Additional technical reports provide similar information about measures of early literacy (Alonzo & Tindal, 2007a) and fluency (Alonzo & Tindal, 2007b). In addition, the following reports present the results of technical adequacy studies of the measures developed for use in the primary grades (Lai et al., 2010; Jamgochian et al., 2010) and with upper elementary and secondary students (Sáez et al., 2010). Conceptual Framework: Progress Monitoring and Literacy Assessment Early work related to curriculum-based measurement (CBM) led by Deno and Mirkin at the University of Minnesota (1977) was instrumental in promoting the use of short, easilyadministered assessments to provide educators with information about student skill development useful for instructional planning. In the three decades since, such progress monitoring probes as they have come to be called have increased in popularity, and they are now a regular part of many schools’ educational programs (Alonzo, Tindal, & Ketterlin-Geller, 2006). However, CBMs – even those widely used across the United States – often lack the psychometric properties expected of modern technically-adequate assessments. Although the precision of instrument development has advanced tremendously in the past 30 years with the advent of more Grade 7 Comprehension – Page 2 sophisticated statistical techniques for analyzing tests on an item by item basis rather than relying exclusively on comparisons of means and standard deviations to evaluate comparability of alternate forms, the world of CBMs has not always kept pace with these statistical advances. A key feature of assessments designed for progress monitoring is that alternate forms must be as equivalent as possible to allow meaningful interpretation of student performance data across time. Without such cross-form equivalence, changes in scores from one testing session to the next are difficult to attribute to changes in student skill or knowledge. Improvements in student scores may, in fact, be an artifact of the second form of the assessment being easier than the form that was administered first. The advent of more sophisticated data analysis techniques (such as the Rasch modeling used in this study) have made it possible to increase the precision with which we develop and evaluate the quality of assessment tools. In this technical report, we document the development of a progress monitoring assessment in reading, designed for use with students in Kindergarten through Grade 8, focusing here on the reading comprehension assessments specifically intended for use with seventh-grade students. This assessment system was developed to be used by elementary and middle school educators interested in monitoring the progress their students make in the area of reading skill acquisition. Reading is a fluid construct, shifting over time from a focus on discrete skills necessary for working with language in both written and spoken forms, to those more complex combinations of skills associated with decoding, and finally to comprehension—a construct in which all prior literacy skills are called upon in the act of reading. Reading assessment typically follows this general progression as well (Reading First, 2006). Assessments of emerging literacy skills evaluate student mastery of the alphabetic principle. These tests measure students’ ability to correctly identify and/or produce letters and the sounds associated with them. They measure Grade 7 Comprehension – Page 3 students’ ability to manipulate individual phonemes (sound units) within words, when, for example, students are asked to blend a list of phonemes into a word, segment a word into its corresponding phonemes, or identify the sounds which begin or end a word (Ritchey & Speece, 2006). As student reading skill progresses, it is necessary to use different reading measures to be able to continue to track the progress students are making as developing readers. Oral reading fluency, which measures a combination of students’ sight vocabulary and their ability to decode novel words rapidly and accurately, is consistently identified in the literature as one of the best predictors of student reading comprehension in the early grades (Graves, Plasencia-Peinado, Deno, & Johnson, 2005; Hasbrouck & Tindal, 2005). Eventually, however, the information provided by measures of oral reading fluency plateaus. Readers attain a fluency threshold that enables them to attend to comprehension rather than decoding (Ehri, 1991, 2005). Once this threshold has been reached, fluency is no longer sensitive to increases in reading comprehension. At this point, one must turn to measures designed to assess comprehension more directly. Although this technical report provides information specifically related to the comprehension measures developed for use in our Progress Monitoring assessment system, we provide an overview of the complete system so readers can understand how the comprehension measures fit into the system as a whole. The Measures that Comprise Our Complete Assessment System Based on previous empirical studies of early literacy assessment (see, for example, the report from the National Reading Panel), we developed two measures of alphabetic principle (Letter Names and Letter Sounds), one measure of phonological awareness (Phoneme Segmenting), three measures of fluency (Word Reading Fluency, Sentence Reading Fluency, and Grade 7 Comprehension – Page 4 Passage Reading Fluency), and one measure of comprehension (Multiple Choice Reading Comprehension). Concurrent with the publication of this technical report, we are completing the development of additional measures of vocabulary and non-fiction reading. A variety of technical reports (available on the brtprojects.org website) provide detailed documentation of the development and technical adequacy of the other measures available in the system for other grades. When one is interested in monitoring the progress students are making in attaining specific skills, it is important to have sufficient measures to sample student performance frequently. Thus, our goal is to create 20 alternate forms of each measure in our assessment system at each grade level where the measure is designed to be used, retaining three of these forms for use as benchmarking assessments while the remaining 17 are designated for progress monitoring. Because these alternate forms are designed to be used to track student progress over time, it is essential that all forms of a particular measure in a given grade level be both sensitive to showing growth in a discrete skill area over short periods of time (2-4 weeks of instruction) and comparable in difficulty. These two equally important needs inform all parts of our measurement development effort: the construction of the technical specifications for each of the measures, the design of the studies used to gather data on item and test functioning, the analytic approaches we use to interpret the results of the pilot studies, and subsequent revision of the measures. In all cases, we seek approaches that provide us with enough information to evaluate the sensitivity of the individual measures to detect small differences in student performance and the comparability of the different forms of each measure to allow for meaningful interpretation of growth over time. Grade 7 Comprehension – Page 5 In the section that follows, we describe the methods we used to construct, pilot, and analyze the performance of the reading comprehension measures in terms of reliability and validity for use in a progress monitoring assessment system. Methods We selected the format of the comprehension measures based on prior empirical work with local school districts (Alonzo & Tindal, 2004a, 2004b, 2004c). In this work, teachers had expressed their desire for tests that closely resembled the types of readings students regularly encountered in their classes. At the same time, concerns about increasing the reliability, ease of use, and cost-effectiveness of our measures prompted us to use multiple choice question types rather than open ended question types in our comprehension measures. Accordingly, we developed the MC Comprehension Tests in a two-step process. First, we wrote the stories that were used as the basis for each test. Then, we wrote the test items associated with each story. We embedded quality control and content review processes in both these steps