# Presentation: 378451

## Research on Making Large Scale Assessments More Accessible  for Students with Disabilities

- Institute of Education Sciences Research Conference
- June 7-8, 2007
- Cara Cahalan Laitusis and Linda Cook
- Educational Testing Service

**Notes:**
Good afternoon.  Cara has just given you some background about the NARAP and the DARA projects and she has described some of the work we have done to evaluate the effect of a read-aloud test change on the Gates McGinitie reading test, using a differential boost paradigm.  What I’m going to talk about next are two studies that we carried out to further explore the impact of a read aloud test change on the GMRT using the same data that was collected for the differential boost study.  The two studies I will talk about are a differential item functioning study and a factor analysis study.

## Designing Accessible Reading Assessments (DARA)

- Educational Testing Service (ETS)
- Focus on students with learning disabilities
- Focus on component approach to assessing reading skills. Primary areas are:
- Word Recognition
- Reading Fluency
- Vocabulary Knowledge
- Comprehension

## Today’s Presentation

- Who is reading aloud NCLB reading assessments and why?
- Review of research examining the impact of reading aloud modifications on the validity of test scores.
- Summary of findings and implications for developing an accessible reading assessment

## Who is Reading the Reading Test?

- States are divided into four positions
- California and New Jersey:  Reading involves decoding (visual or tactile) so read aloud is not allowed
- Wisconsin:  Read aloud results in test of another construct (listening comprehension) and test was not designed for this purpose so read aloud is not allowed
- Kentucky and Delaware: Reading is comprehension of text (visual, tactile, or audio) so read aloud is allowed
- Texas and Maine: Word level decoding is not the primary construct so some read aloud is permitted (e.g., reading test questions but not passages)

**Notes:**
Wisconsin has a similar policy but their reasoning is that  . . . 

Texas and Maine has reasoning that is similar to California and New Jersey

## Prevalence of Read Aloud

- Many students use read aloud on tests of reading even when it is not allowed under their state’s policy
- In 2006 nearly 5000 4th graders in California used Read Aloud on the STAR English Language Arts assessment even though the accommodation is “not allowed” and invalidates the student’s score for NCLB reporting.
- When read aloud is permitted the use of read aloud can be very large (up to 70% of students with disabilities) leading some states to change their policy or list specific caveats in 2006.

## Read Aloud Policy Changes in 2006

- Texas changed policy to permit read aloud (e.g., reading aloud proper nouns, test questions, and answer choices but not passages) on some reading tests for dyslexic students only
- Massachusetts changed policy to allow screen reading software, but specifies the accommodation for students with disabilities who are unable to read, not those performing below grade level
- Missouri changed policy to discontinue allowing read aloud due to large number of students using the accommodation (70% of students with disabilities)

## Research Questions

- Do students with reading-based learning disabilities receive differential performance gains from read aloud or would all poor readers benefit from having the test read aloud?
- Do tests and test items taken with and without read aloud perform the same psychometrically?
- same factor structure
- no evidence of differential item performance
- equivalent predictions of reading comprehension

## DARA Research on Read Aloud

- Experimentally Designed Study
- Differential Boost from Read Aloud
- Predictive Validity of Scores from Read Aloud
- Psychometric Studies
- Differential Item Functioning
- Differential Distractor Analysis
- Factor Analysis
- Think Aloud Studies

**Notes:**
Over the last year and a half we have conducted several studies examining the impact of read aloud including
READ SLIDE
Today Linda and I will present the results of 3 of these studies

## Differential Boost Description

- Framework for studying impact of changes in testing conditions
- Change in testing condition may be viewed as an accommodation if students with disabilities receive a significant increase in scores compared to students without disabilities
- Most researchers refer to differential boost as an interaction between group and change in testing condition

**Notes:**
Differential Boost is a framework for studying the impact of changes in testing conditions; that is, test accommodations or modifications on the validity of test scores obtained by examinees with disabilities.  
I should  mention at this point that the term accommodation is typically reserved for testing changes that do not affect the construct the test measures.  The term modification is typically used with testing changes that are thought to impact the test construct.
The basic premise for a differential boost study is that a change in testing condition, may be viewed as an accommodation if students with disabilities who take the test without the testing change and then take a parallel form of the test with the testing change show a significant gain in test score when compared to students without disabilities who took the test under the same conditions, that is, the students with disabilities show a differential boost in test scores.  
Most researchers refer specifically to an interaction between group (examinees with or without disabilities) and the change in testing condition as a necessary condition for the change to be considered an accommodation. 

## Data Collected

- 2 Reading Comprehension Tests
- GMRT Forms S & T
- Audio Form (also had extra time and answering in test booklet)
- Standard Form (extra time and answering in test booklet)
- 2 Fluency Measures
- WJ Reading Fluency and TOSWRF
- 2 Decoding Measures  (4th grade only)
- WJ Letter Word ID  and WJ Word Attack
- Student Survey
- Teacher Survey

## Sample

- 1181 4th grade students
- 527 students with reading based learning disabilities (RLD)
- 654 students without a disability (NLD)
- 855 8th Graders
- 394 RLD
- 461 NLD

## Differential Boost Data Collection Design

| Group | Session 1 |  | Session 2 |  |
| --- | --- | --- | --- | --- |
|  | Form | Accommodation/
Modification | Form | Accommodation/
Modification |
| 1 | S | Standard | T | Audio |
| 2 | S | Audio | T | Standard |
| 3 | T | Standard | S | Audio |
| 4 | T | Audio | S | Standard |

**Notes:**
The slide you are looking at now shows the data collection design that was used for the differential boost study 
We randomly assigned each school in the study to receive one of two sessions first, either the standard condition which had extra time only or the audio condition that consisted of the read-aloud modification delivered by CD and extra time.
Students within schools were then randomly assigned to whether they would receive form T or form S first. Each student in the study took both an audio and a standard version of the test.
When we analyzed the data, we looked for both an order affect (audio given first versus standard given first) and a form affect (Form T versus Form S).  We found neither affect for the fourth grade data which allowed us to collapse across order and form so that we could compare the score differences we were interested in (audio minus standard) for students with and without disabilities. 

## Grade 4

|  | RLD (n=527) |  |   | NLD (n=654) |  |
| --- | --- | --- | --- | --- | --- |
|  | M | SD |   | M | SD |
| Standard | 457 | 32 |   | 497 | 38 |
| Audio | 477 | 30 |   | 502 | 33 |
| Boost | 20 | 29 |   | 5 | 24 |
| Fluency | 473 | 21 |   | 500 | 25 |

**Notes:**
As you can see from this slide the average score gain (or boost) for the RLD sample was 20 points and the average score gain for the NLD sample was 5 points.  

In addition the average score on both the standard and audio versions for the RLD sample were lower than the average scores on either the audio or standard version for the NLD sample.  

In addition the fluency scores for the RLD sample were significantly lower (about 1 standard deviation) that those for the NLD sample

## Grade 8

|   | RLD (n=376) |  |   | NLD (n=471) |  |
| --- | --- | --- | --- | --- | --- |
|  | M | SD |   | M | SD |
| Standard | 511 | 28 |   | 553 | 33 |
| Audio | 521 | 27 |   | 555 | 31 |
| Boost | 10 | 23 |   | 2 | 21 |
| Fluency | 514 | 34 |   | 560 | 42 |

**Notes:**
As you can see from these means the gap in scores between students with and without RLD is consistent but the gain in score from the audio accommodation is decreasing and is almost insignificant for the NLD group.

## Differential Boost Analyses

- RM-ANOVA
- Dependent Variable: Boost
- Independent Variables:
- Experimental Group (form/accommodation order)
- LD Status (RLD or NLD)
- RM-ANOVA
- Controlling for ceiling effect by removing top performers
- (_>_ 45 of 48 items correct on standard)
- RM-ANCOVA
- Covariate: WJ Reading Fluency

## Results of the Differential Boost Analyses

- Students with reading-based learning disabilities had a significantly larger boost from audio (read aloud) accommodations than students without disabilities
- Controlling for other factors (e.g., reading fluency, decoding, or ceiling effects) does not change these findings

## Predictive Validity Analyses

- Regression analyses examining which test scores capture the most variance in teachers rating of reading comprehension
- Tested 4 models:
- Standard
- Standard + Fluency
- Audio
- Audio + Fluency
- Examined which assessment model should be used to assess reading comprehension by grade and disability group
- Several limitations to these analyses
- Ratings were done in October
- Definition of reading comprehension may have varied between teachers
- Even with these limitations this study is the first study to examine the predictive validity of audio (read aloud) test scores

## Results from Regression Analyses

- Standard score alone is a good measure of reading comprehension (as defined by teachers) for NLD students and an adequate measure for RLD students
****=.607 for grade 4 NLD and .525 for grade 8 NLD
****=.460 for grade 4 NLD and .405 for grade 8 NLD
- Audio score does not appear to predict teachers ratings of reading comprehension better (or as well) than the standard score for RLD students
- Audio score with fluency (and standard with fluency) captured more variance in teachers ratings than standard score alone for Grade 4 RLD

**Notes:**
=.460 for grade 4 RLD and ..405 for grade 8 RLD

## Purpose of Differential Item Functioning and Factor Analyses

- Determine if the test is measuring the same underlying construct (s) for the following comparisons:
- Students without disabilities who take the test with and without a read aloud test change
- Students with disabilities who take the test with and without a read aloud test change

**Notes:**
The purpose of both the differential item functioning analysis and the factor analysis of the GMRT data that I’m going to talk about this afternoon was to support and expand on the findings of the differential boost study that Cara just discussed by evaluating whether the test was measuring the same underlying construct or constructs for students without disabilities and students with learning disabilities who took the test with and without a read aloud testing change.  
We already had some evidence, based on the differential boost study, that the read aloud, or audio, testing change might be considered an accommodation rather than a modification. That is, a test change that did not impact the construct (s) that the test was measuring. We were seeking further confirmation of this finding by using the same data used for the differential boost study, but two different methodological approaches, differential item functioning and factor analysis.  

## Differential Item Functioning Analyses

- Comparisons of item responses for four groups
- Reading based learning disability (RLD), no accommodation (Standard)
- Reading based learning disability (RLD),  audio modification (Audio)
- No disability (NLD), no accommodation (Standard)
- No disability (NLD), audio modification (Audio)

**Notes:**
As I just mentioned, we used data from the administration of the GMRT to NJ fourth and eighth grade students, for both the differential item functioning (DIF) analysis and the factor analysis.  The slide you are looking at now shows the groups that we used for both of these analyses.
[Read from the slide]
The first analysis I’m going to talk about is the DIF analysis and, because of the limited amount of time I have this afternoon, I’m only going to talk about the Form S grade 4 results.

## Summary Statistics for Grade 4 Form S DIF and Factor Analysis Samples 

**Notes:**
The slide you are looking at now shows the raw-score summary statistics for the Form S grade 4 samples that we used for both the DIF and factor analysis studies.  
I should mention that the possible range of raw scores on Form S is 0 to 48.  
The slide shows the comparisons that were used for both of the analyses. For the DIF study, Comparison 1 used students with learning disabilities (RLD) who took the test under standard conditions as the reference group and students with disabilities who took the test with a read aloud test change as the focal group.  For Comparison 2, the DIF study used students without disabilities (NLD) who took the test under standard conditions as the reference group and students without disabilities who took the test with a read aloud test change as the focal group.  I’ll explain what is meant by reference and focal group in a minute.

## Differential Item Functioning (DIF)

- Test takers matched on proficiency level
- Used total test score as matching criterion
- Reference Groups
- Students with and without disabilities who took test under standard conditions
- Focal Groups
- Students with and without disabilities who took test with read aloud test change

**Notes:**
As many of you know, differential item functioning occurs when test-takers who belong to different groups, who have been matched on  proficiency level, do not perform in the same way on a particular item.  The groups that are compared for a DIF analyses are referred to as the reference group and the focal group. The reference group performance is the standard for the DIF comparison and the focal group is the group whose item performance you are interested in studying.  For this analysis, the reference groups and the focal groups were matched on proficiency level using the total score on the reading comprehension test.
As I just mentioned, we had two reference groups for the study, students without disabilities who took the test under standard conditions and students with disabilities who took the test under standard conditions.  We also had two focal groups;  students without disabilities who took the test with a read aloud test change and students with disabilities who took the test with a read aloud test change.

## Differential Item Functioning (DIF)Analyses

- Used Mantel-Haenszel procedure with total score as criterion
- Mantel-Haenszel categorization
- A—negligible DIF
- B—slight to moderate DIF
- C—moderate to large DIF
- Direction of DIF Flags
- Negative favors reference group
- Positive favors focal group

**Notes:**
The procedure that we used to examine DIF for the comparisons I just described was the Mantel-Haenzel procedure with, as I mentioned previously, the reading comprehension total score used as the matching criterion for the analyses. 
For those of you who are not familiar with the MH procedure, the procedure is used to classify items as A, B, or C DIF.  Items with A DIF have negligible DIF, those with B DIF show slight to moderate amounts of DIF and those with C DIF show significant levels of DIF.  Items that show negative DIF favor the reference group and those that show positive DIF favor the focal group.

## Comparison 1: RLD  Standard (reference group) vs. RLD Audio (focal group)

**Notes:**

The slide you are looking at now, shows the results of Comparison 1. 
[go over layout of slide]
For this comparison we examined differential item functioning for the RLD group that took the test with an audio, read aloud, test change, compared to the RLD group who took the test without this test change.  The two groups differed considerably in their average scores on the test, almost .6 standard deviations.  
The DIF results do not show what one might expect,  which is a consistent pattern of DIF favoring students who took the test with the read aloud modification.  The results show only B level DIF and the direction of the DIF is mixed Sixteen out of 48 items showed DIF; 9 items favored the group who took the test without the modification and 7 items favored the group who took the test with the modification. 

## Comparison 2: NLD Standard (reference group) vs. NLD Audio (focal group)

**Notes:**
The slide you are looking at now shows the results of Comparison 2. For this comparison we examined differential item functioning for the NLD group that took the test with an audio test change compared to the NLD group who took the test without this test change.  The results are quite similar to the results shown on the last slide that were for students with reading disabilities. 
The results for the non-disabled students showed only B level DIF and the direction of the DIF was mixed, with a total of 10 out of 48 items displaying DIF; 6 items favored students who took the assessment without the read aloud modification and 4 items favored the group who took the test with the modification. 
The results shown on this slide and the slide you just looked at, indicate that the test, as a whole, is measuring basically the same underlying construct or constructs for the two comparisons that we carried out: (1) students without disabilities who took the test with and without the read aloud test change and, (2) students with disabilities who took the test with and without the read aloud test change.  These results support the findings of the differential boost study that indicated that a read aloud test change might possibly be considered an accommodation rather than a modification.

## Factor Analysis of GMRT Data

- Analyzed Form S (4th and 8th grade levels) and Form T (4th and 8th grade levels)
- Investigated the impact of a read aloud test change for students with and without disabilities who took the test with and without the test change
- Does the GMRT measure the same underlying construct (s) for the study groups?

**Notes:**
So, the next step was to examine the same data set that we used for the differential boost study and for the DIF analysis using factor analysis procedures.  
Again, because my time is limited today, I’m only going to discuss the results of the factor analysis of Form S given to grade 4.  This is the same exact sample used for the DIF study that I just discussed.
The purpose of the factor analysis was similar to the purpose for the DIF analysis and this was to add support to the findings of the differential boost study by examining the internal structure of the GMRT to evaluate whether or not the GMRT measures the same underlying constructs (factors) for students without disabilities who take the test with and without the read aloud test change and for students with disabilities who take the GMRT with and without the read aloud test change.  

## Factor Analysis of GMRT Data

- Single group exploratory factor analysis of four groups
- RLD who took the test under standard conditions
- RLD who took the test with an audio test change
- NLD who took the test under standard conditions
- NLD who took the test with an audio test change
- Confirmatory factor analysis
- Single-group to establish single factor
- Multi-group to establish base line model
- Multi-group comparisons to establish measurement invariance
- Comparison 1—RLD standard vs. RLD read aloud
- Comparison 2—NLD standard vs. NLD read aloud

**Notes:**
We began our study of the factor structure of Form S of the grade 4 GMRT by carrying out both exploratory and confirmatory single group factor analyses for the same four groups of students used for both the  differential boost and DIF study.
We followed the single group analyses with multi-group confirmatory factor analyses that we used to establish the measurement invariance of the GMRT given to students with disabilities who took the test with and without the read aloud test change (Comparison 1 from the DIF study) and for the test given to students without disabilities who took the test with and without the read aloud test change (Comparison 2 from the DIF study). 

## Summary of Multi-group CFA for RLD Standard vs. RLD Audio

**Notes:**
The information in the table you are looking at now shows the results of the multi-group confirmatory factor analyses (the CFAs) that were carried out to test measurement invariance for the GMRT taken by the RLD standard and RLD audio groups.  The indices and critical values that we used to determine whether or not the CFA models fit the data were: 1) a ratio of the Chi-square value divided by the degrees of freedom less than the critical value of 3; 2) the Root Mean Square Errors of Approximation (RMSEA) less than .05; 3) a Goodness of Fit Index (GFI) greater than .95; 4) a Comparative Fit Index (CFI) greater than .95 and, 5) a change in the CFI less than .02.  
The results show that as the model becomes more restricted the fit indices are still well within the range of critical values used to determine fit. 
Consequently, the results of the analyses can be interpreted as indicating the test is measuring the same single construct for the RLD standard and RLD audio groups.
I don’t have time to show you the results of the multi-group CFAs that were carried out for the NLD standard and audio groups, but these results were very similar to the results for the RLD group. The results of both analyses provide further support for the results of the differential boost study that indicated that a read aloud test change might be considered an accommodation rather than a modification.

## Summary

- Purpose of the studies was to determine if the test measures the same underlying construct (s) when taken with and without a read aloud test change
- Results of DIF and factor analysis studies indicate that test measures the same underlying construct for the RLD group, or for the NLD group when taken with and without a read aloud test change

**Notes:**
So, let me summarize what I have talked about this afternoon.  The purpose of the studies I’ve described today was to investigate the impact of a read aloud test change on a reading test taken by students with a reading based learning disability.  We attempted to determine if the test was measuring the same underlying constructs for the groups used in the differential boost study who took the GMRT with and without a read aloud test change.  We used factor analysis and DIF analysis to investigate the impact of the read aloud test change on the GMRT scores.  The results of the factor analysis and DIF studies indicated that the GMRT was measuring the same single construct for two comparisons: students without a disability who took the test with and without a read aloud test change, and students with a disability who took the test with and without a read aloud test change. The differential item functioning analyses, the factor analyses and the differential boost results taken together indicate that a read aloud test change used for this test may be considered an accommodation rather than a modification.  

## Additional Research Studies

**Pitoniak, M., Cook, L., Cline, F., & Cahalan-Laitusis, C. *****Using differential Item Functioning to Investigate the Impact of Accommodations on the Scores of Students with Disabilities on English-Language Arts Assessments.***** **
**Cook, L. Eignor, D., Sawaki, Y., Steinberg, J., & Cline, F. *****Using factor analysis to investigate the impact of accommodations on the scores of students with disabilities on English-language arts assessments.*****  **
**Bolt, S. E., & Diao, Q. *****Reading-aloud a reading test: Examining reading sub-skill performance.***
**Koretz, D. *****The assessment of students with disabilities in Kentucky.***
**Koretz, D., & Hamilton, L. *****Assessing students with disabilities in Kentucky: the effects of accommodations, format, and subject.***

**Notes:**
At this point, I think it is important to mention that the results of the studies that I have just described are not inconsistent with those found in earlier studies that were carried out using data collected in an operational setting as compared to the experimental setting used to collect the GMRT data.  Pitoniak and her colleagues found very little differential item functioning for a fourth and eighth grade English language arts test given to students with and without accommodations including a read aloud test change.  Cook and her colleagues,  working with the same data used in the Pitoniak study, were able to demonstrate that the factor structures were the same for tests given to students with and without disabilities under accommodated and standard conditions.  
In addition, Bolt and Diao examined whether reading items functioned differently for students with disabilities in grades 4, 5, and 7 when the students received a read aloud test change.  They found a low incidence of DIF for students taking the test with a read aloud testing change.  Other researchers, such as Lewis, Green, and Miller, Koretz,  and Koretz and Hamilton have found evidence of DIF with a read aloud test change but the DIF sometimes favored students without disabilities taking the test without the test change.  So there appears  to be a body of evidence accumulating that indicates a read aloud test change may be considered as an accommodation for reading tests under some circumstances, however, we do believe that additional research is needed on the impact of a read aloud accommodation on scores obtained on large scale assessments.  

## Research Answers

- Students with reading-based learning disabilities DO receive differential performance gains from read aloud.
- These gains can not be completely explained by poor fluency ability or ceiling effects
- Reading tests and test items taken with and without read aloud:
- Have the same factor structure
- Have no evidence of differential item performance
- But do not predict reading comprehension in the same way for students with and without disabilities

## Implications for Accessible Reading Assessments

- Based on regression findings assessments should test access skills (fluency and decoding) separately from comprehension for RLD students
- Based on DIF and factor analyses, a reading test given with an audio test change appears to measure the same constructs as a reading test given under standard conditions
- Audio score alone may be an easier but not a better measure of ‘reading comprehension’ (compared to standard score) for RLD students