## 4 Discussion
In spite of this fact, most evaluations of TEL recommender systems reported in the literature have focused only on the accuracy of the recommendation algorithms [40]. In this study, the usefulness of the OER recommendations was evaluated in terms of both relevance and quality. Therefore, the recommendations considered to be more useful were those that recommended high-quality OER that were found relevant by the user. If only relevance had been considered for measuring the usefulness of the generated recommendations, recommendations of OER with low pedagogical quality (e.g., for having inaccurate content or an inappropriate level of detail) would have been considered useful in those cases in which participants, after a quick review, considered that the OER could suit their interests. Obviously, these recommendations should not be considered to be useful (or at least not as useful as high-quality OER recommendations) because there is little benefit from providing users with OER with poor quality, and this can even be counterproductive in some cases. Quality assurance is of major importance for OER repositories since teachers need some guarantee of quality before incorporating OER into their lesson plans and low-quality OER can put learners at risk of being misinformed or wasting time. Therefore, there is little doubt that pedagogical quality is a key feature of OER that these repositories should consider and evaluate. Notwithstanding, this feature is usually ignored by OER recommender systems, even for those recommender systems using content-based techniques, for which it should be easy to incorporate OER quality measures into the recommendation process since they are able to generate recommendations based on item features and ratings. This work proves that OER repositories can benefit from evaluating the pedagogical quality of their OER by incorporating the obtained quality measures into a recommender system. In spite of these benefits, it is clear that there are important barriers preventing the use of quality-based recommendation approaches in OER repositories, with the most critical barrier being the difficulty to effectively collect pedagogical quality scores for all OER. In those OER repositories, such as MERLOT or ViSH, that systematically evaluate the quality of the published OER following a community-based mechanism, evaluations are conducted by volunteer reviewers or users, and hence no quality scores are generated for a significant percentage of OER. Although some measures have been proposed to face this barrier, such as assigning a temporary score to non-evaluated OER [53], automatically evaluating quality using intrinsic features [70], or estimating the OER pedagogical quality based on user interactions [71], this issue remains as an open challenge for OER repositories that requires further research. In the dataset used for this user study, all OER were evaluated by at least two experienced reviewers and, based on these evaluations, quality scores were calculated for all of them. Therefore, in other scenarios where no reliable pedagogical quality data are available for all OER in the dataset, the performance of the quality-based recommendation approaches is expected to be worse. Nonetheless,in the worst scenario where no data related to pedagogical quality would be available at all, the hybrid approach analyzed in this study would still be able to generate OER recommendations as useful as the ones generated by the traditional content-based approach.