# A Case Study to Explore a UDL Evaluation Framework Based on MOOCs
## 4 Results
### RQ2. What Are the Perceptions of UDL as an Evaluation Framework for Untrained Raters?
This suggests that for 20 of the 31 checkpoints, students did not find it difficult to evaluate. Students also did not see ambiguity for 13 of the 31 checkpoints. There is a distinction between how difficult a task is and how accurate a student is at the task. Just because something is hard does not necessarily mean that it was performed incorrectly. Further insight is gained in that some of the checkpoints were both identified as difficult to evaluate and considered redundant with other checkpoints. This suggests that there is room to improve the language around the checkpoints for evaluation to \begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**Group** & **Across Principles** & **Checkpoints** & **Description of Redundancy** \\ \hline Group 1 & No & 2.1 Clarify vocabulary and symbols 2.2 Clarify syntax and structure & The evaluation of the use of language regarding the existence of a glossary of terms (2.1) and maintaining the same terminology (2.2) \\ \hline Group 2 & No & 6.2 Support planning and strategy development 6.4 Enhance capacity for monitoring progress & Monitoring progress and strategy development facilitating reflection (6.2) and progress (6.4), both in the quizzes \\ \hline Group 3 & Yes & 1.2 Offer alternatives for auditory information 1.3 Offer alternatives for visual information 2.5 Illustrate through multiple media 5.1 Use multiple media 5.2 Use various tools for construction and composition & Use of alternative formats. The inclusion of different questions formats such as captions and transcripts (1.2), audio descriptions (1.3), images, text, video, or graphics (2.5), the use of external tools (5.1), and external links (5.2) adds confusion \\ \hline Group 4 & Yes & 4.1 Vary the methods for response and navigation 7.1 Optimise individual choice and autonomy & Time limit (4.1) is related to options for physical action, allowing extra time to achieve a task, (7.1) indicates options for recruiting interest allowing the needed time to participate in discussions \\ \hline Group 5 & Yes & 5.3 Build fluencies with graduated levels of support for practice and performance 8.1 Height salience of goals and objectives 8.3 Foster collaboration and community 8.4 Increase mastery-oriented feedback & The interaction in the system and between peers for reflection. Including facilitators (5.3) space to formulate and share learning objectives (8.1), with other partners (8.3) and again facilitators (8.4) \\ \hline Group 6 & Yes & 6.1 Guide appropriate goal setting 7.3 Minimise threats and distractions 9.2 Facilitate personal coping skills and strategies & Information about learning objectives, activities (6.1), the use of a calendar (7.3), and spaces to discuss difficulties encountered (9.2) \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Checkpoints identified as redundant by students. reduce the ambiguity for students. There must be certain redundancy and overlap within the framework too; the key characteristic is that many of the checkpoints reported as redundant belong to different guidelines and even principles. Perceptions of students are that UDL is useful and has benefits, but that using the checklist is not straightforward and training and experience for its application are needed. While there is some ambiguity and some areas that are difficult to evaluate the fact that students identify this as beneficial suggests this evaluation framework should be iterated on and improved to better support student evaluations.