# A Case Study to Explore a UDL Evaluation Framework Based on MOOCs
## 4 Results  
### RQ1. To What Extent Did Untrained Raters Agree When Using the UDL Evaluation Framework? The results of the interaction of undergraduates with YourMOOC4 all have been divided first by checkpoints, then by guidelines and principles, in each of the following figures. The mean, standard deviation (SD), and Kappa (K) are shown on the diverging stacked bar charts. K Interpretation is 0.0-0.20 slight agreement; 0.21-0.40 fair agreement; 0.41-0.60 moderate agreement; 0.61-0.80 substantial agreement; and 0.81-1.0 almost perfect agreement [26]. Two Kappa values have been calculated, K1 includes the five Likert values, while K2 is reduced to three options (disagreement, neutral, and agreement). Fair agreement values are presented with a * while moderate, substantial, and perfect agreements are shown with a + to facilitate the visibility of the results. Results are complemented by a sample of quotes from the open-ended questions during the evaluation using YourMOOC4 all. In the case of \"_provide multiple means of engagement_\" (Figure 3), in the MOOC, students identify they can participate in the discussions or activities and that the responses from the facilitators are positive and oriented to help (checkpoints 8.3 and 8.4), for example:  
_There is a forum where you can contact your classmates and thus release stress and continue learning thanks to their help. The tests contain great feedback on what was taught, but do not identify its level of difficulty. As a help, there is only one glossary, with certain terms and the forum for the \"team\" to answer your questions._ (ST8)  
Students agree MOOC is designed to allow motivation and activities to match with the learning outcomes, with information for optimising individual choice and options for self-regulation (9.1 and 9.3):  
_The course is designed to effectively motivate the student. Its structure does not only seek purely theoretical content but plays with various options to achieve a key motivation so that students can develop their activities, ask their questions and progress in the content in an even fun way._ (ST7)  
Concerning \"_provide multiple means of representation_\" (Figure 4) students are positive about videos containing captions and transcripts, the use of the language consistent terminology and having a logical sequential ordering of tasks (1.2 and 2.1):  
Figure 3: “Provide multiple means of engagement” evaluation results. Note: standard deviation (SD), Kappa for 5 values (K1); Kappa for 3 values (K2); fair agreement *; moderate, substantial, and perfect agreements +. While the MOOC and its platform are failing to adapt to the environment, modify the information and personalise the learning experience (1.1 and 3.4):  
_The content seems to me to be presented concisely. At all times you see the content index, which lets you know where you are going and not disconnect from the conceptual map of the course. The \"weak\" points of the MOOC are, for example, that it does not allow for modification of the visualisation of the content._ (ST23)  
Regarding the principle _\"provide multiple means of action and expression\"_ (Figure 5, students are not confident with the use of social networks or external tools available (5.1):  
_According to the scope of the expression in the MOOC, the only point I find regarding the proposed evaluation is the non-existence of the possibility of communication through social networks (or at least I have not been able to find it anywhere in it)._ (ST27)  
Students understand the MOOC is supporting the process of reflection, the availability of information, and the capacity for monitoring progress (6.4):  
Figure 4: “Provide multiple means of representation” evaluation results. Note: standard deviation (SD), Kappa for 5 values (K1); Kappa for 3 values (K2); fair agreement *; moderate, substantial, and perfect agreements +. Figure 5: “Provide multiple means of action and expression” evaluation results.