# A Case Study to Explore a UDL Evaluation Framework Based on MOOCs
## 4 Results
### RQ1. To What Extent Did Untrained Raters Agree When Using the UDL Evaluation Framework?
Note: standard deviation (SD), Kappa for 5 values (K1); Kappa for 3 values (K2); fair agreement *; moderate, substantial, and perfect agreements +. _I found the course progress screen very interesting; it is very useful since it allows you to better control the time you have to finish it._ (ST 6)  
The results in terms of principles and guidelines have been described in detail above, but can also be analysed in aggregate form. The evaluation is generally positive and shows fair and moderate agreements (Figures 6 and 7), being the worst-rated guideline, and least agreed, _\"expression and communication\"_ (5). To answer RQ1, [PERSON] kappa values were computed for both K1 using the five Likert values in the questions, and K2 which reduced the evaluation to three options (disagreement, neutral, and agreement). For the 31 checkpoints using K1 scores there were: 11 slight, 17 fair, and 2 moderate agreements. In addition, the 31 checkpoints using K2 scores were: 6 slight, 5 fair, 17 moderate, and 2 substantial agreements. These results indicate that while the agreement for some items was achieved (i.e., 2.2. and 4.2), for other items, the responses among raters were variable (i.e., 1.1, 2.3, 2.4, 5.1, 5.3, and 8.2). The lower levels of agreement can be interpreted either as variable insights into limitations of course design or could be an indication of different interpretations of the evaluation tasks. It is important to recall that UDL aims to design up front to consider the variability of students [5]. In that sense, in our research questions, the focus is on interpreting the results from the perspective that variable ratings represent the variability of students (in RQ2 we examine the potential for different interpretations of the evaluation framework). The notion of designing with consideration for human variability is that the design decisions that are necessary for some students are beneficial for all students. From this perspective, all areas where students disagree are potential opportunities for improvement in course design. The relationship between disagreement and agreement evaluations provides a potential prioritisation mechanism to address design concerns. Across all checkpoints, the results indicated there were 14 out of 31 checkpoints where at least one student disagreed, indicating the course did not implement the UDL checkpoint (see Figures 2-5). Of the 14 checkpoints with disagreement evaluations, 11 of those had slight agreement ratings using K1 scores (i.e., 1.1, 2.3, 2.4, 3.4, 5.1, 5.2, 7.2, 7.3, 8.2, 9.2, and 9.3) whereas 6 had slight agreement ratings using K2 scores (i.e., 1.1, 2.3, 2.4, 5.1, 5.3, and 8.2). The strengths and limitations of prioritising course improvements using agreement statistics of course evaluations are bound to the frequency of disagreement [28]. Prioritising the six K2 slight agreement checkpoints would encompass all checkpoints with at least 10% of students with disagree evaluations. While it would help improve the overall evaluation for many students, it might not identify issues of critical importance that were identified by small numbers of students. It would be important to reconcile prioritisation by considering  
Figure 6: Guidelines evaluation results. Note: standard deviation (SD), Kappa for 5 values (K1); Kappa for 3 values (K2); fair agreement *; moderate, substantial, and perfect agreements +. Figure 7: Principles evaluation results. Note: standard deviation (SD), Kappa for 5 values (K1); Kappa for 3 values (K2); fair agreement *; moderate, substantial, and perfect agreements +. which groups of students would benefit from the revisions. Minority groups of students may be also in the minority in terms of their UDL evaluations. As can be seen in Table 2, with the strengths and limitations of prioritising course improvement using agreement statistics in mind, these results suggest focusing course improvement on six checkpoints where there was slight agreement using the K2 calculation.