## Materials and Methods
### Usefulness Evaluation of the OER Recommendations
On the contrary, the minimum utility score of a set of lists of recommendations calculated through the normalized R-Score metric is 0 and is obtained when all users rate all the recommended items with ratings that are equal or lower than the threshold value d. In this user study, the parameter \(\mathsf{S_{MAX}}\) was set to 5 since participants rated the relevance of the OER using a 5-point scale, the threshold value d was set to 1, and the half-life parameter \(\alpha\) was set to 3. In addition to calculating overall measures of relevance, the normalized R-Score metric was used, in this user study, for obtaining an overall measure of pedagogical quality of the OER recommended for each of the different recommendation approaches. These overall quality measures were calculated in the same way as the overall relevance measures, but with the following two changes in Equation (5): \(\mathsf{S_{i}}\) was the quality score of the OER in the i-th position of the list (calculated according to the LORI evaluations) instead of the participant rating, and \(\mathsf{S_{MAX}}\) was set to 10 since it is the maximum value the quality score of an OER could be. As for calculating the relevance, the threshold value d was set to 1 and the half-life parameter \(\alpha\) was set to 3. When calculating overall quality measures with the normalized R-Score metric, on the one hand, the maximum utility score of a set of lists of recommendations (i.e., a value of 1) is reached when all recommended OER have a quality score of 10, which only occurs if all reviewers rate all those OER with the maximum score in all LORI criteria. On the other hand, the minimum utility score of a set of lists of recommendations (i.e., a value of zero) is obtained when all recommended OER have a quality score lower or equal than 1, the value of the parameter d, which in this case acts as a quality threshold instead of as a participant rating threshold. In summary, the following data were obtained from the results of the user study by using the normalized R-Score metric:  
* A relevance score for each recommendation approach and participant of the user study on a 0-1 scale indicating how relevant for the participant were the recommendations generated by the approach. * An overall relevance score for each recommendation approach on a 0-1 scale indicating, on average, how relevant were the recommendations generated by the approach for all participants. * A quality score for each recommendation approach and participant of the user study on a 0-1 scale indicating how good in terms of pedagogical quality were the OER recommended to the participant according to the approach. * An overall quality score for each recommendation approach on a 0-1 scale indicating, on average, how good in terms of pedagogical quality were the OER recommended by the approach to all the participants of the user study. Thereby, the usefulness of the OER recommendations generated by each of the four recommendation approaches was evaluated in terms of both relevance and pedagogical quality. This evaluation compared these recommendation approaches based on the relevance of their recommendations and the quality of the recommended OER.