Results of a Cognitive Interview Study of Immediate Feedback and Revision Opportunities for Students with Disabilities in Large Scale Assessments. Synthesis Report 92
Author: Johnstone, Christopher
Date: 2013-12-00
Type: Reports - Research
Subjects: Disabilities,Alternative Assessment,Feedback (Response),Error Correction,Test Items,Interviews,Student Attitudes,Multiple Choice Tests,Grade 8,Mathematics Tests,Mathematics Achievement,Testing Accommodations

Validly assessing students with disabilities has been a challenge for decades but is increasingly vital to educational policy and practice in the current era of accountability. Numerous technological and policy developments have occurred in the past several years with the emergence and decline of various forms of alternate assessments. This study was part of a larger research project originally designed to investigate an approach to Alternate Assessments with Modified Achievement Standards that provides immediate feedback and revision opportunities for students when they answer test items. After answering an item, students would receive feedback about whether the answer was correct or incorrect and would be given additional chances to correct their answer (for partial credit). This particular study employed cognitive interviews to determine whether: (a) Feedback and Revision had a qualitative impact on student interaction with the assessment (i.e., students' interview responses suggest that the feedback and revision feature had an impact on those students' success on the test), and (b) there were qualitative differences between groups of students with and without disabilities (i.e., students' interview responses suggest that the utility and effects of feedback and revision differ for these two groups). After students had completed the test items, we asked the students post-task questions about their perceptions of the assessment and its various conditions. Results indicated that the vast majority of students appreciated getting a second or third chance when their answer was incorrect (although four students commented that having opportunities to revise responses was &quot;unfair&quot;). Test and interview results suggest that feedback and revision were helpful if students had a sense of how to complete the item but were not useful for students who were demonstrating low mathematical proficiency. The latter group of students became frustrated with the test's &quot;incorrect&quot; feedback and simply guessed as quickly as possible when they were uncertain of how to attack an item. In terms of feedback and revision data at the item level, most students were able to answer items correctly on two attempts. When third attempts were taken, students were primarily in a guessing mode, most likely because they did not understand the item's content. Overall, students guessed far more often on multiple choice than constructed response (open-ended) items. Overall, students also had more correct answers on multiple choice than open-ended items, despite similar content on both types of tests. The following are appended: (1) Round 1: Number of Correct Answers Out of All Attempts; (2) Round 2: Number of Correct Answers Out of All Attempts; (3) Round 3: Number of Correct Answers Out of All Attempts; (4) Round 1: Student Perceptions of Feedback and Revision; (5) Round 2: Student Perceptions of Feedback and Revision; (6) Round 3: Student Perceptions of Feedback and Revision; (7) Round 1: Student Guessing; (8) Round 2: Student Guessing; (9) Round 3: Student Guessing; (10) Round 1: Strategy and Guessing Pathways (SC, AUC Conditions); (11) Round 2: Strategy and Guessing Pathways (SC, AUC Conditions); and (12) Round 3: Strategy and Guessing Pathways (SC, AUC Conditions).