# Recommending Learning Objects with Arguments and Explanations
## 4 Evaluation
### Results  
The experiments evaluated the _effectiveness_ of C-ERS to provide recommendations that suit the students' profile and learning objectives. The advantages of the argumentation-based recommendation over other approaches in this domain, such as content-based, collaborative filtering, and knowledge-based, was previously demonstrated in our previous work [4]. Therefore, we do not intend to evaluate the whole learning process of students, as proposed, for instance, in [PERSON]'s model [40], which tries to evaluate the learning process from reaction, to learning, to behavior, and to organisational performance. Here, we just focus on [PERSON]'s level 1, evaluating the reaction of the students to the learning objects proposed by the system. As shown in Figure 3, for all recommendation processes, we computed the average of original ratings that students provided to the learning objects that the system recommended to them in two separated groups, the average rating of the learning objects that were provided in recommendation processes were explanations were shown ('With Argument' in the figure), and the average rating to those learning objects that were provided in recommendation processes where the system did not show any explanation (just the object, 'Without Argument' in the figure). In addition, 25 th and 75 th percentiles are marked by the boundaries of the boxes, and the average ratings by dots. Moreover, Figure 4 shows exactly how often each rating was chosen. As we can see in the figures, students provided higher ratings to those LOs that presented explanations, which demonstrates the  
Figure 3: Average ratings provided by the students without and with arguments and their associated explanations. Figure 2: Schema followed by the described experiment. advantages of using explanations to offer effective recommendations. Furthermore, the number of highly rated objects (3 to 4) increased when they came with explanations. We consider as a baseline the case in which students do not receive explanations attached to the recommendation (without argument), since the hypothesis to be demonstrated is that the ratings assigned to 'justified' recommendations are higher, in other words that students are more satisfied when they understand the reason why the system provides them with a specific recommendation. On the data shown in Figure 3, the tests of statistical significance show a \(p\)-value of 0.0046 and \(g\) of Hedges of 1.27 (effect size significant for samples with different variances \(-1.21\) without arguments; 1.64 with arguments), which demonstrate the significance of the results obtained. In Figures 5 and 6, experiments also evaluated the average ratings provided by the students for each priority ordering to show the explanations, and how often each rating was chosen for each priority ordering. As depicted in the figures, all priority orderings seem to have a similar performance, with a slight increase in the average rating and in the quantity of LOs that got better ratings for P3 (first show arguments that suit the usage history of the users). Figure 4: Histogram with the rating comparison between students without and with arguments. Figure 5: Average ratings provided by the students for each priority ordering to show the explanations. To evaluate the _persuasive power_ of C-ERS, the initial rating provided by the student to LOs and the final one after the 'conversation' between the system and the student were compared. Figure 7 shows the average percentage of decreased, improved, and unchanged ratings throughout the conversational recommendation process (from the 1 st iteration to the end). As illustrated in the figure, a wide percentage of ratings were improved by the interchange of explanations between the C-ERS and the student, which demonstrates the persuasive power of explanations to improve the opinion of students about the LOs recommended.