# Presentation: 360230

## Research on Making Large-Scale Reading Assessments More Accessible for Students with Disabilities June 7, 2007

- Deborah Dillon, Jamal Abedi,
- Cara Cahalan Laitusis, and Linda Cook

- Discussants:  Geneva Haertel and Gerald Tindal

**Notes:**
This title of this session is research on making large-scale reading assessment more accessible for students with disabilities. 

The panel members are me (replacing Martha Thurlow, who was unable to make it because of her husband’s ill health), Jamal Abedi from the University of California, Davis, Cara Cahalan Laitusis of ETS, and Linda Cook of ETS.

Also with us today are two discussants – Geneva Haertel of SRI, and Jerry Tindal of the University of Oregon. 

We plan to leave time for questions at the end. 

## National Accessible Reading Assessment Projects

- Designing Accessible Reading Assessments (DARA)
- Partnership for Accessible Reading Assessment (PARA)
- Technology Assisted Reading Assessment (TARA)

**Notes:**
Three projects in the National Accessible Reading Projects group – have agreed to work together. All are focused on developing accessible large-scale reading assessments for students with disabilities – note that “large-scale” means those tests that are used for state assessments, for example.

Today, will hear from two of the projects – PARA and DARA.

## NARAP Projects’ Goals

- Develop a definition of reading proficiency
- Research the assessment of reading proficiency
- Develop research-based principles and guidelines making large-scale reading assessments more accessible for students who have disabilities that affect reading
- Develop and field trial a prototype reading assessment

**Notes:**
Refer to brochure that highlights each project – and that notes the common goals of the projects. Point out that we are all working now on goal 2, and that today we will highlight some of out goal 2 work and the implications of that work.

## Partnership for Accessible Reading Assessments (PARA)

- Collaboration of National Center on Educational Outcomes and U of MN Department of Curriculum and Instruction, CRESST, U of CA Davis, and Westat
- Focus on all disabilities that impact reading, particularly:
- Learning disabilities
- Speech or language impairments
- Mental retardation
- Deafness or hard of hearing

## Designing Accessible Reading Assessments (DARA)

- Educational Testing Service (ETS)
- Focuses on students with learning disabilities
- Focuses on component approach to assessing reading skills. Primary focus are:
- Word Recognition
- Reading Fluency
- Vocabulary Knowledge
- Comprehension

## Technology Assisted Reading Assessment (TARA)

- ETS, NCEO and Center for Applied Special Technology (CAST)
- Focus on students with visual impairments
- Focus on:
- Examining the performance of operational ELA tests for students with visual impairments
- Development of prototype Technology Assisted Reading Assessment
- Inclusion of VI students in NARAP field test

**Notes:**
TARA project was funded in July by IES through the special education assessment for accountability grant and is a collaboration between;

## Partnership for Accessible Reading Assessment (PARA)Deborah Dillon and Jamal Abedi

##     PARA Research            Program Assumptions

**Assumption #1:** We do not know everything about what goes into accessible reading assessment yet.
**Assumption #2:** Preliminary research must inform design of accessible assessment.
**Assumption #3:** Both preliminary and experimental research will inform development of Principles and Guidelines for future assessments.

## Three Major Research                  Question Areas

**What characteristics of current ****assessment practices hinder**** accessibility?**
**What ****characteristics of students**** require more accessible assessment of reading?**
**What characteristics would ****accessible assessment**** have?**

**Notes:**
ASSESSMENT PRACTICES THAT HINDER – Indicate that Jamal will talk about analyses that have been conducted to examine the characteristics of test items, including DIF and DDF studies that have been conducted.

CHARACTERISTICS OF STUDENTS – indicate that we are looking at a variety of student characteristics, and that this work is underway. 

CHARACTERISTICS OF ACCESSIBLE ASSESSMENTS – two main studies are underway here. Jamal will talk about one of them – the Segmenting Study. I will give a brief description of another study that we be underway soon – the Motivation Study 

## Motivation Study

**Purpose:** To examine whether improving the motivational characteristics of a large-scale reading assessment increases its accessibility for students with disabilities, and in so doing provides a more valid assessment of these students’ reading proficiency due to their increased engagement.

**Motivation Study**

##   Research Questions

- 1. Does the option of exercising choice in the selection of reading comprehension passages, which is hypothesized to improve student motivation and engagement on a large-scale assessment, produce significantly higher measured reading comprehension for all students?
- 2. Is there a significant difference in reading scores of students with disabilities versus general education students on large-scale reading assessments?
- 3. Is there a significant difference in student performance on text type (literary-_fiction_ versus informational-_exposition_ passages) on large-scale reading assessments?

##               Research Questions

- 4. Is there an interaction effect between choice, type of text, and type of student?
- 5. Is there a correlation between students’ general motivation to read (e.g., as measured by the MRQ-Motivation to Read Questionnaire) and their performance on a large-scale reading assessment?   Are participants who are more motivated to read (as measured by the MRQ), more likely to benefit from the choice option on a large scale reading assessment?

## Design: Calibration Study

- Prior to the motivation study, 32 reading passages (16 each for grades 4 and 8) with 10 items each, will be calibrated to empirically determine the comparability of the passages
- Calibration requires the placement of all passages & questions on a common IRT-based equal-interval measurement scale, the development of passage scoring tables, & the use of a mechanism for equating scores across passages
- Item fit analysis will be employed to determine which items will be retained or eliminated
- Students will rate passages to indicate interest and difficulty
- Participants include 400 students in grades 3-5 and 400 students in grades 7-9 from intact classrooms, including students with and without disabilities

##  Design: Motivation Study

- Participants include 280 students who are fluent in English—140 students from 4th grade and 140 from 8th grade, including targeted samples of students representing a range of disability groups
- The motivation assessment includes 2 literary-fiction and 2 informational-expository passages for both grade 4 & grade 8; each passage will be followed by 5-6 multiple choice items; the assessment is untimed & completed on a computer-based platform
- Interviews will be conducted after testing with subsets of students from the control and experimental groups at both grade levels

##   Attending to Issues of Motivation

- General motivation will be measured prior to the test to obtain information on students’ feelings about “self as reader” (e.g., Motivation for Reading Questionnaire-MRQ).
- Situated motivation will be measured using questions woven into the test booklets for the choice and no choice conditions (placed after the comprehension items); specific questions will tap
- students’ perceptions of the texts they read
-    (e.g., difficulty; interest), and
- students’ sense of self-efficacy in reading and completing the items following the passage (the task).

## Design &Analysis: Motivation Study

- A counterbalanced stratified random assignment design will be used with experimental *choice* (C) groups that select reading passages for the assessment (“design your own assessment”) and control *no choice* (NC) groups that do not select passages
- The dependent measure is comprehension performance; the factors include choice condition (choice/ no choice), disability status (youth with disabilities/ youth without disabilities) and text type (literary-fiction/informational-exposition)

## Design &Analysis: Motivation Study

- The design is a split-plot design with two between-subjects factors (A = passage choice and B = disability status), one within-subjects factor (C = text type), one blocking variable (S = subject), and one covariate (X = motivation as assessed on the MRQ) at the between-subject level; A, B, C, and X are fixed effects, and S is a random effect
- Analysis of variance will be used to evaluate various effects; correlations of students’ performance on the comprehension test & responses on the MRQ and situated motivation questions will be calculated