# A Case Study to Explore a UDL Evaluation Framework Based on MOOCs
## 4 Results
### RQ2. What Are the Perceptions of UDL as an Evaluation Framework for Untrained Raters?
Students have the perception that WCAG is included to some extent in some of the UDL guidelines, specifically when using the new version of WCAG (2.1) since the new criteria are oriented to accessibility on multiple devices. However, WCAG is designed to correct technical aspects, whereas UDL is for the design and evaluation of pedagogical aspects:  
_WCAG 2.1 are more oriented to the correction based on the staging of the content, and to the variety of tools and the good use of them, without presenting errors in their implementation, to facilitate user access. UDL is a methodology that values more conceptually the mechanisms that promote learning and make it more open to a greater number of people._ (ST18)  
Difficulty to evaluate. Students identified several checkpoints as difficult to evaluate (see Table 4), indicating the overlap between checkpoints during the evaluation. Students report how difficult some checkpoints are formulated to evaluate without being strongly engaged with the MOOC considering aspects such as the learning design, assessment, or communication, which include checklists assessing the role of facilitators and interaction with other students and aspects related to learning outcomes and adaptation of the content:  
_The checkpoints where it is assessed whether the proposed activities agree with what it is desired to learn are difficult to assess since it depends on each of the students. It is the same case of the level of difficulty of the MOOC activities, the feedback in the tests and the existence of questions that help reflection._ (ST14)  
**Redundancy. ** Regarding redundancy, students report that several checkpoints ask about similar concepts, in some cases, redundancy is within the principle such as in Groups 1 and 2 (see Table 5) for an evaluation of the use of language and monitoring progress. An example includes: \begin{table}
\begin{tabular}{l l} \hline
**Principles** & **Checkpoints** \\ \hline \multirow{6}{*}{Provide multiple means of Engagement (7, 8, 9)} & 7.2 Optimise relevance, value, and authenticity \\  & 8.1 Heighten salience of goals and objectives \\  & 8.2 Vary demands and resources to optimise challenge \\  & 8.4 Increase mastery-oriented feedback \\  & 9.1 Promote expectations and beliefs that optimise motivation \\  & 9.2 Facilitate personal coping skills and strategies \\ \hline \multirow{6}{*}{Provide Multiple Means of Representation (1, 2, 3)} & 1.1 Offer ways of customising the display of information \\  & 2.3 Support decoding of text, mathematical notation, and \\ \cline{1-1}  & symbols \\ \cline{1-1}  & 3.4 Maximise transfer and generalisation \\ \hline \multirow{6}{*}{Provide Multiple Means of Action and Expression (4, 5, 6)} & 5.3 Build fluencies with graduated levels of support for practice \\ \cline{1-1}  & and performance \\ \cline{1-1}  & 6.2 Support planning and strategy development \\ \hline \end{tabular}
\end{table} 
Table 4: Checkpoints identified as difficult to evaluate by students. _The checklists about discussing with students what you want to learn are redundant. In the case of the existence of a social network or external tool, the MOOC already has enough tools to be able to work with it._ (ST7)  
However, other identified redundancies exist across multiple principles, which make it more difficult to simplify the evaluation framework and show the possible overlaps in UDL within guidelines belonging to different principles. An example quote that shows redundancy across checklists and guidelines is:  
_The different questions about the language could be unified since they are redundant. The questions about which tools are used within the MOOC are also repetitive. Finally, a couple of times we are asked about the content, formats, and structures of the MOOC._ (ST8)  
To answer RQ2, for the 31 checkpoints, 11 were identified as difficult to evaluate (see Table 4), and 18 were associated with a redundant group (see Table 5). At the intersection of difficult-to-evaluate and redundancy, there are 5 checkpoints (i.e., 5.3, 6.2, 8.1, 8.4, and 9.2).