## 3 Semantic Educational Recommender Systems
### Designing the Recommendations with TORMES Methodology  
When it comes to involve educators in the recommendations design process, there are two key issues to consider: (1) the elicitation of educational oriented recommendations; and (2) the evaluation of the recommender (and the recommendations designed) for its educational purpose. To this end, a methodology has been proposed named TORMES, which stands for Tutor-Oriented Recommendation modeling for Educational Systems [34]. TORMES is based on the ISO standard 9241-210 and aims to involve educators in the process of designing educationally oriented recommendations through user centered design methods, which are complemented with data mining analysis. In order to elicit educational oriented recommendations, and following the ISO standard 9241-210, the following four activities are to be considered:  
1. description of the context of use through individual interviews with educators, which is complemented with data mining analysis from past courses;
2. requirements specification through the scenario based approach that is used to extract knowledge from the educators and allows identifying recommendation opportunities that can be provided in the scenario outlined to improve the learning experience;
3.creation of design solutions in terms of semantically described recommendations, where educators are involved in validating and refining the initial set of recommendations elicited from the scenarios in the previous activity through focus groups; and
4.evaluation of the recommendations designed against the requirements by rating their relevance and classifying them in terms of their scope. Through this step-wise process, the SERS approach is able to deal with the particularities of the educational domain. Moreover, the semantic descriptions of the recommendations can provide explanations to the learners about the recommendations offered. This provision of explanations has already been acknowledged in the literature as beneficial for user experience with the recommender [35]. To evaluate the recommender system and the recommendations designed for it, four dimensions have been proposed. These dimensions are: (i) integration of the SERS into the e-learning system; (ii) impact of the recommendations on the users (both learners and educators); (iii) value of recommender systems' quality properties such as utility, serendipity, coverage, _etc._; and (iv) impact of the recommendations delivery on learners' interactions. A large scale experience has been carried out with 377 learners to design educational oriented recommendations and evaluate them with the aforementioned dimensions in an e-learning system, which results from integrating the SERS into Willow. Willow is an adaptive computer assisted assessment system that is able to automatically process short learner's answers or essays written in natural language and provide feedback on the responses given [36]. Some results have already been reported elsewhere. For instance, in [37] we have reported the evaluation of the impact of this integration in the learning process, showing that a positive statistical impact was detected on indicators related to the engagement in the course, the learning effectiveness and efficiency, and the knowledge acquisition. We also analyzed the impact on the user experience in terms of the consistency usability principle, showing that the recommendations kept the high perception of the users regarding usability and satisfaction. However, it should be investigated if the recommendations do have an impact on users' perception when the usability of the original system is low. In this experience, the results showed high degree of usability due to the dialogue metaphor in which the interaction is based in Willow. Hence, the participants benefited from the additional support provided by the recommendations, since an improvement is perceived on several types of indicators (_i.e._, engagement, learning efficiency, learning effectiveness and knowledge acquisition).