# A Case Study to Explore a UDL Evaluation Framework Based on MOOCs
## 4 Results
### RQ2. What Are the Perceptions of UDL as an Evaluation Framework for Untrained Raters? Table 3 details the thematic analysis including codes and quantification of the student's responses to the questions included in the second task of the exercise (question 1 (Q1) is divided between advantages and disadvantages). Advantages. We could anticipate that participants would see the value of designing up front for student variability. Furthermore, common beliefs about UDL were expected to appear in student responses. Those expectations were confirmed because the predominant categories of EDI, Expectations and Motivations, Learning Design, and Alternative Formats are features UDL implementation seeks to accomplish. An example of a response that illustrates the alignment of student-perceived advantages and the UDL framework is as follows:  
_UDL optimises learning so that in a group where we find students of different levels and abilities, we can teach everyone equally without excluding them. Facilitates access to study material, offering access in more than one format. In this way, it also promotes motivation among students and their participation._ (ST13)  
Disadvantages. As we had no clear expectations of how untrained raters would interpret and use the checklist, the disadvantages help establish what work remains in operationalising UDL as an evaluation framework. Students identified the checklist as difficult to implement, complex, and time-consuming. Students also indicated a lack of familiarity with the UDL framework. An example quote that illustrates the challenges is as follows: \begin{table}
\begin{tabular}{l l} \hline \hline
**Principles** & **Checkpoints** \\ \hline Provide multiple means of Engagement (7, 8, 9) & 8.2 Vary demands and resources to optimise challenge \\ \hline  & 1.1 Offer ways of customising the display of information \\ Provide Multiple Means of Representation (1, 2, 3) & 2.3 Support decoding of text, mathematical notation, and symbols \\  & 2.4 Promote understanding across languages \\  & 3.4 Maximise transfer and generalization \\ \hline Provide Multiple Means of Action and Expression (4, 5, 6) & 5.1 Use multiple media for communication \\ \hline  & \\ \end{tabular}
\end{table} 
Table 2: Checkpoints with a slight agreement. \begin{table}
\begin{tabular}{l l} \hline \hline
**Question** & **Codes** \\ \hline  & \(\bullet\)  Advantages: Equity, Diversity, and Inclusion (5), Expectations and motivations (3), \\ Advantages and disadvantages (Q1) &  Learning design and assessment (3), Alternative formats (2), Language (1) \\  & \(\bullet\)  Disadvantages: Difficult to implement (2), Complex (2), Expensive (1), Unfamiliar (1) \\ \hline Comparison (Q2) &  Universal Design (4), Accessibility (3), Expectations and motivations (1), Usability (1), \\  & Personalisation (1) \\ \hline Difficulty to evaluate (Q3) & Overlap between checkpoints (3), Communication (2), Learning design and assessment (2), Alternative formats (1), Personalisation (1), Self-regulation (1) \\ \hline Redundancy (Q4) & Alternative formats (3), Communication (2), Learning design and assessment (2), \\  & Language (2) Personalisation (2), Time limit (1) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Codes derived from students’ perceptions of UDL. There may always be a student who cannot use the created product; therefore, it is necessary to design strategies and curricula that are inclusive for as many students as possible. Despite this, some students will need individualised support and attention. And despite everything, the main disadvantage that UIDL brings is the large investment that must be made in educational centres and the little interest on the part of public and private institutions to carry it out._ (ST9)  
Comparison. Students were asked to compare the use of WCAG and UDL. While students in the sample are familiar with WCAG, UDL was new to them. Students understand WCAG as a set of guidelines for web accessibility but lack the pedagogical perspective included in UDL.