# Recommending Learning Objects with Arguments and Explanations
## 4 Evaluation
### Methodology  
For the evaluation tests of our C-ERS, a prototype was implemented that uses 75 LOs from FROAC (Federation of Learning Objects Repositories of Colombia). These LOs include learning materials from computer science such as database, systems, algorithms, programming, etc. The LOs are also provided with metadata that represents information about the title, description, keywords, type, language, format, and interactive level. All learning objects were in Spanish and 20% of them also had an English version. The tests were performed by a group of 50 bachelor degree students in Computer Science of the Universidad Nacional de Colombia. The students signed an informed consent to participate. During the registration, the system requested from these students information about their topics of interest (interest keywords) and level of education (years as bachelor degree student). They also completed a test to determine their appropiate learning style. Thus, the system was able to register the next features for each student:  
* Personal data: ID, name, surname, sex, date of birth, nationality, city of residence, address, language, phone, and mail. * Student's educational preferences:
* Interactivity level: high human-computer interaction LOs (preferred by nine students), medium human-computer interaction LOs (preferred by 38 students), or LOs that focus on presentation of content (preferred by three students). * Preferred language: all students mother tongue was Spanish, and thus all preferred LOs in Spanish. * Preferred format: _jpeg_ was selected by nine students, _mp4_ by eight students, _pdf_ by 31 students, and 'other formats' by two students. * Learning Style: to model the learning style of each student, it was followed the VARK ([[http://vark-learn.com/](http://vark-learn.com/)]([http://vark-learn.com/](http://vark-learn.com/))) model. The model classified 25 as visual students, 6 as auditory, 12 as reader, and 7 as kinesthetic. * History of uses: for each LO ranked by the student, the system stores its ID, the rating assigned, and the date of use. After the registration, the evaluation tests were performed as follows. Each student participated in 10 recommendation processes, where the system recommended LOs that match his/her profile for a specific search (a list of specific keywords). As pointed out in Section 3.2, the system could follow four different argumentation strategies, which result in different orderings to show arguments (see Figure 1 for details on which arguments succeed to others). In the recommendation processes, the system was settled to follow the following priority orderings:  
* Recommendation processes 1-2: P1 (show content-based arguments first)
* Recommendation processes 3-4, P2 (show collaborative arguments first)
* Recommendation processes 5-6, P3 (show knowledge-based arguments first)
* Recommendation processes 7-8, P2 (show general arguments first)* Recommendation processes 9-10, random  
Among these recommendation processes, the system computed the best top five recommendations (those with the largest number of generated arguments) and provided the student with the best of the list (the one with the more arguments generated). Then, the student had to provide an initial rating for the LO recommended. Furthermore, for 5 out of these 10 recommendation processes (even recommendation processes, i.e., 2, 4, 6, 8, 10), the system showed an explanation for the LO recommended. When no explanation was provided (odd recommendtion processes), the system's interface allowed the student to 'Accept' the recommendation, 'Reject' it, or 'Ask for a justification'. When an explanation was provided, the student could 'Accept' the LO, 'Reject' the LO, or else, 'Reject' the explanation (and hence, the underlying argument). When the student _accepted or rejected the recommendation_ at the fist step of the recommendation process, he/she was able to run the next recommendation process of the evaluation tests.