## Materials and Methods
### User Study  
This user study was conducted online, and therefore participants could complete it by themselves using their own computers through a web browser. The only guidance received by the participantswas a text document with the necessary instructions to participate in the user study. All participants performed the following steps:  
1. Participants accessed an OER repository and create a user account. Concretely, participants accessed ViSH [43], an OER repository publicly available at [[http://vishub.org](http://vishub.org)]([http://vishub.org](http://vishub.org)), which is enriched with additional features such as authoring tools, an audience response system, recommendations, and a social network.
2. Participants accessed an online evaluation tool provided through the ViSH web portal. This tool was developed specifically for this user study.
3. Once participants had accessed the online evaluation tool, they were asked to provide some basic information, including demographic data (age, gender, language, and occupation) and their previous experience using OER repositories.
4. Participants chose a topic of their interest (e.g., engineering, biology, or history) among different options. It should be clarified that this topic was used in the user study only to select an OER in the next step (it was not taken into account by any of the recommendation approaches).
5. In this step, participants were told to put themselves in the situation that they are searching for learning resources in an OER repository (such as ViSH) and that, finally, they find one in which they are interested and, after clicking on a link, they navigate to a new page where they can see it. An OER of ViSH tagged with the topic chosen by the participants in the previous step was randomly selected and shown to the participants in order to act as the particular OER that each participant found in their fictitious search. Below this OER, a list of recommendations with 20 randomly sorted OER were presented to each participant. Finally, participants were required to rate, given the described situation, the relevance of each of the recommended OER using a five-point Likert scale from 5 (very relevant to me) to 1 (not at all relevant to me).  
The lists of OER recommendations shown to the users in the last step of the user study were built by concatenating four distinct lists of five OER, each of them generated by a system using a different recommendation approach. Each of these four recommendation approaches is described in detail in the Section 2.4. Therefore, in order to build each list shown to the participants, a total of 20 OER recommendations were generated: five by a traditional content-based recommender system, five according to a quality-based non-personalized approach, five by a hybrid recommender system that combines content-based and quality-based recommendation techniques, and five chosen at random. It should be highlighted that after concatenating the different lists, all OER were shuffled and hence it was not possible for participants to know which OER was recommended by which system. In fact, participants did not even know that the recommendations had been generated by different systems. Furthermore, by using this research design, position bias was prevented. In the case of overlap (i.e., when the same OER was recommended by two or more systems), duplicates were removed, and therefore no OER were shown to a participant more than once. Therefore, in these cases, it should be taken into account that, although 20 OER recommendations were generated, the OER list shown to the participants was actually comprised of less than 20 recommendations.