Year: 2021

Abstract:
Modern hybrid recommendation systems require a sufficient amount of data. However, several internet privacy issues make users skeptical about sharing their personal information with online service providers. This work introduces various novel methods utilizing the baseline estimate to learn user interests from their interactions. Subsequently, extracted user feature vectors are implemented to estimate the user-item correlations, providing an additional fine-tuning factor for neighborhood-based collaborative filtering systems. Comprehensive experiments show that utilizing the user-item similarity can boost the accuracy of hybrid neighborhood-based systems by at least 2.11% while minimizing the need for tracking users’ digital footprints.

text: Modern hybrid recommendation systems require a sufficient amount of data. However, several internet privacy issues make users skeptical about sharing their personal information with online service providers. This work introduces various novel methods utilizing the baseline estimate to learn user interests from their interactions. Subsequently, extracted user feature vectors are implemented to estimate the user-item correlations, providing an additional fine-tuning factor for neighborhood-based collaborative filtering systems. Comprehensive experiments show that utilizing the user-item similarity can boost the accuracy of hybrid neighborhood-based systems by at least 2.11% while minimizing the need for tracking users’ digital footprints.

heading: 1 INTRODUCTION
text: The continuously accelerated growth of communication technology and data storage in the past decades has benefited customers with an enormous amount of online multimedia content such as movies, music, news, and articles, creating billion-dollar industries. Following this evolution, recommendation systems (RSs) have been widely developed to automatically help users filter redundant information and suggest only suitable products that fit their needs. Such systems are used in a variety of domains and have become a part of our daily online experience (Ricci et al., 2015).
RSs are commonly classified into three main types (Adomavicius & Tuzhilin, 2005): the contentbased technique, the collaborative filtering technique, and the hybrid technique. The content-based approach, as in Lang (1995); Pazzani & Billsus (1997); Lops et al. (2011); Narducci et al. (2016), learns to recommend items that are similar to the ones that a user liked based on items’ features. The main weakness of this approach is the lack of available and reliable metadata associated with item (Mooney & Roy, 2000). Meanwhile, the collaborative filtering (CF) approach does not require product information but only relies on users’ interaction history which can be either explicit or implicit feedback. CF systems can be divided into two major categories: i) neighborhood-based models which suggest items that are most similar to the item that a user is interested in (Herlocker et al., 2000; Tintarev & Masthoff, 2007), and ii) matrix factorization models which could explore the latent factors connecting items to users in order to make accurate recommendations (Ricci et al., 2015; Koren et al., 2009; Koren, 2008). However, it is often the case that there is not enough transaction data to make accurate recommendations for a new user or item. To tackle this so-called cold-start problem, hybrid methods are proposed by combining auxiliary information into CF models (Singh & Gordon, 2008; Agarwal & Chen, 2009; Wang & Blei, 2011; Li et al., 2011; Rendle, 2010).
In the interest of the hybrid approach and its advantages, our study attempts to improve typical neighborhood-based RSs utilizing available content-related knowledge. The main contributions of this work are summarized as follows:
• Introducing new methods to represent user preference via combining user’s interaction data and item’s content-based information, which helps to estimate the similarity between a user and an item.
• Integrating the user-item similarity degree into the baseline estimate of neighborhood-based RSs to provide more precise recommendations, surpassing competitive hybrid models.
The remainder of this paper is organized as follows. Section 2 reviews the basic knowledge on neighborhood-based CF systems, including hybrid models. Detail descriptions of our proposed methods are presented in Section 4. Section 5 gives experimental results and in-depth analysis. At last, we conclude this study in Section 6.

heading: 2 PRELIMINARIES
text: In this paper, u, v denote users and i, j denote items. rui denotes the preference by user u for item i, also known as the rating, where high values indicate strong preference, and all the (u, i) pairs are stored in the set K = {(u, i)|rui is known}. Meanwhile, R(u) denotes the set of all items rated by user u. In rating prediction task, the objective is to predict unknown rating r̂ui where user u has not rated item i yet.
Popular neighborhood-based CF techniques for the rating prediction task and an existing hybrid variant are briefly reviewed as follows.

heading: 2.1 NEIGHBORHOOD-BASED MODELS
text: The neighborhood-based approach is one of the most popular techniques in CF, which is only based on the similarity between users or items to give recommendations. There are currently two methods for implementing neighborhood-based CF models: i) user-oriented (or user-user) model which predicts a user’s preference based on similar users, and ii) item-oriented (or item-item) model which finds similar items to the item a user liked and recommends these items to her. Of the two methods, the latter introduced by Sarwar et al. (2001) has become dominant. This is due to the fact that the number of users in real-life systems is often orders of magnitude bigger than of items, which makes the user-oriented model inefficient. Furthermore, the item-oriented model is capable of providing a rational explanation for recommendations (Ricci et al., 2015). Therefore, our implementations in this work adopt the item-item approach as the base model.
The fundamental of neighborhood-based models is similarity measure. By computing the similarity degree sij between all pairs of items i and j using popular similarity measures such as Cosine similarity (Cos) or Pearson Correlation Coefficients (PCC), we can identify the set of k neighbors Sk(i, u) which consists of k most similar items to i rated by user u. Then, r̂ui can be predicted as a weighted average of the ratings of similar items:
r̂ui =
∑ j∈Sk(i,u)
sijruj∑ j∈Sk(i,u) sij (1)
Even though Equation (1) can capture the user-item interactions, much of the observed ratings are due to the bias effects associated with either users or items, independently of their interactions. In detail, some items usually receive higher ratings than others, and some users tend to give higher ratings than others. kNNBaseline model proposed by Koren (2010) adjusts the above formula through a baseline estimate which accounts for the user and item effects as follows.
r̂kNNBaselineui = bui +
∑ j∈Sk(i;u)
sij (ruj − buj)∑ j∈Sk(i,u) sij (2)
where bui = µ + bu + bi denotes the baseline estimate, µ denotes the mean of overall ratings, bu and bi correspond to the bias of user u and item i, respectively, which can be trained using popular optimization algorithms such as Stochastic Gradient Descent (SGD) or Alternating Least Squares (ALS).

heading: 2.2 INTEGRATING CONTENT-BASED INFORMATION INTO NEIGHBORHOOD-BASED MODELS
text: A number of problems regarding kNN models using similarity measure on the rating information were noticed by Duong et al. (2019). The first problem is the sparsity of the rating matrix, which
might yield an inaccurate similarity score between two items that share only a few common users. Secondly, filtering common users who rated both items to calculate the similarity score is a timeconsuming task due to a large number of users. To address these problems, a novel similarity measure was proposed using item content-based information. Assuming that each item i is characterized by a feature vector qi = {qi1, qi2, ..., qif} ∈ Rf where f is the number of features, which is stored in matrix Q ∈ Rn×f . The value of each element encodes how strong an item exhibits particular properties. The similarity score sij between movies i and j is calculated as follows.
sCos content
ij = ∑f k=1 qikqjk√∑f
k=1 q 2 ik √∑f k=1 q 2 jk
(3)
or
sPCC content
ij = ∑f k=1(qik − q̄i)(qjk − q̄j)√∑f
k=1(qik − q̄i)2 √∑f k=1(qjk − q̄j)2 (4)
where q̄i and q̄j are the mean of feature vectors qi and qj , respectively. Experiments showed that the item-oriented CF models using Coscontent and PCCcontent provide equivalent accuracy to the stateof-the-art CF models using rating information whilst performing at least 2 times faster. Hereafter, kNNBaseline model using one of these similarity measures is referred to as kNNContent.

heading: 3 EXPERIMENTAL SETTING

heading: 3.1 MOVIELENS DATASET AND EVALUATION CRITERIA
text: In this work, the MovieLens 20M dataset is used as a benchmark. This is a widely used dataset in the study of RSs which originally contains 20,000,263 ratings and 465,564 tag applications across 27,278 movies. The ratings are float values ranging from 0.5 to 5.0 with a step of 0.5. Tag Genome data, which is computed on user-contributed content including tags, ratings, and textual reviews, is firstly introduced in this version of MovieLens dataset (Harper & Konstan, 2016). To utilize this kind of data, a cleaning process is applied to the dataset. Specifically, the movies without genome tags are excluded from the dataset. Then, only movies and users with at least 20 ratings are kept. Table 1 summarizes the result of the cleaning stage.
The preprocessed dataset is split into 2 distinct parts: 80% as the training set and the remaining 20% as the testing set. To evaluate the performance of the proposed models, three commonly used indicators in the field of rating prediction are used: RMSE (Root Mean Squared Error) and MAE (Mean Absolute Error) for accuracy evaluation where smaller values indicate better performance, and Time [s] for timing evaluation. Here, RMSE and MAE are defined as follows.
RMSE = √ ∑
u,i∈TestSet (r̂ui − rui)2 /|TestSet| (5)
MAE = ∑
u,i∈TestSet |r̂ui − rui| /|TestSet| (6)
where |TestSet| is the size of the testing set. The total duration of the model’s learning process on the training set and predicting all samples in the testing set is measured as Time [s]. All experiments are carried out using Google Colaboratory with 25GB RAM and no GPU.

heading: 3.2 BASELINE MODELS
text: In this paper, several popular models are selected as baselines to evaluate the proposed methods. Firstly, we implement two competitive neighborhood-based models including kNNBaseline (Koren, 2010) and kNNContent (Duong et al., 2019). Besides, SVD (Funk, 2006) and SVD++ (Koren, 2008), two well-known representatives of matrix factorization technique, are also experimented due to their superior accuracy and flexible scalability with extremely sparse data (Funk, 2006).
Error rates and the time to make predictions are measured for comparison. The optimal hyperparameters for each baseline model are carefully chosen using 5-fold cross-validation. In particular, the error rates of the neighborhood-based models are calculated with the neighborhood size k ∈ {10, 15, 20, 25, 30, 35, 40}. Due to better performance compared to Cos and Coscontent, PCC and PCCcontent are chosen as the similarity measures in kNNBaseline and kNNContent models. SVD and SVD++ models are trained using 40 hidden factors with 100 iterations and the step size of 0.002.

heading: 4 PROPOSED SYSTEMS
text: So far, kNNBaseline models have successfully applied the item-item similarity exploiting rating information and the available metadata representing item features provided by users. In contrast, the knowledge about user-user correlation finds it difficult to be deployed in practical applications due to its modest performance and high memory requirement (Ricci et al., 2015). Besides, to our best knowledge, the interest of a user in individual characteristics of an item also lacks careful consideration, which is a major problem restricting the growth of RSs. One of the main reasons is that the user-item correlation is commonly defined as a similarity degree between a user’s interest in individual item features and an item feature vector, which requires a customer to provide her personal preferences as much as possible for accurate recommendations. In reality, it is impractical due to a variety of data privacy concerns (Jeckmans et al., 2013).
This study first tackles this problem by introducing various novel methods to represent a user preference in the form of a vector, utilizing her past interactions with items and the feature vectors of those items. We then propose a modification to the baseline estimate of kNNBaseline model and its variants by integrating the user-item similarity score, which boosts the precision of the conventional kNNBaseline model.

heading: 4.1 ESTIMATING USER INTERESTS FOR USER-ITEM SIMILARITY MEASURE
text: In RSs, there are two main sources of information: the interaction records (such as transactions history, ratings, ...) and the item content information (item catalog, movie genres, or the Tag Genome data in the MovieLens 20M dataset). User personal data, however, is not stored or included in publicly available datasets due to the risk of exposing user identities. Therefore, in most datasets for research, there is rarely any data or statistic that directly specifies user interest in each item feature. In this section, we present 3 different methods to characterize a user’s interest based on the ratings and metadata of the movies she watched.
The most straightforward approach to estimate a user interest is via a weighted average of the feature vectors qi of items that she rated as follows.
pnormu =
∑ i∈R(u) rnormui · qi
|R(u)| (7)
where rnormui is the rating of user u for item i which has been normalized to the range of [0, 1]. As a result, the normalized feature vector pnormu of user u has the same dimension and range of element values as an item vector qi. More importantly, each user is currently described in an explainable way: elements with higher values indicate that the user has a greater preference for the corresponding item attributes and vice versa.
Although this method helps to create a simple shortcut to understand user preferences, all users are treated in the same way. Specifically, users’ ratings are all normalized using the minimum and maximum values of the system’s rating scale. Whereas, in practice, different users have a variety
tendencies of rating an item according to their characters. For example, easy-going people often rate movies a little higher than they really feel, and conversely, strict users often give lower scores than the others. That means if two users have conflicting views after watching a movie but accept to give a 3-star rating for that movie, for example, then the system will implicitly assume they have the same weight of opinion. This fact leads to the researches taking into account the user and item biases, which have a considerable impact on kNNBaseline and biased SVD models (Koren et al., 2009; Koren, 2010). Therefore, a modification of Equation (7) incorporating the effect of biases is proposed as follows.
pbiasedu =
∑ i∈R(u)
zui · qi∑ i∈R(u) |zui| (8)
where the residual rating zui = rui − bui denotes how much extra rating a user u gives more or less than her expectation to a movie i. In more detail, this formula applies the residual ratings as weighting factors to the corresponding item feature vectors, which helps to eliminate the restrictions of rnormui . The resulting biased user feature vector p biased u now has its elements in the value range of [−1,+1], where −1 / +1 indicates that she totally hates / loves the respective item attribute, and 0 is neutral preference. It is expected that pbiasedu could measure the user interest in each item attribute more precisely than its normalized version pnormu .
However, both of the above methods treat all items equally in profiling a user interest. For example, consider user Janet and two movies “Titanic” and “Mad Max”. The scores of “Titanic” and “Mad Max” for the romantic genre are 0.90 and 0.05, respectively, which means “Titanic” is a romantic movie while “Mad Max” has almost no romantic scene. Assume that Janet’s normalized ratings for these movies are r̃Janet,Titanic = 0.7 and r̃Janet,Mad Max = 0.72, which are almost identical. Thus, the romantic genre score of Janet calculated by Equation (7) is quite low: (0.7 × 0.9 + 0.72 × 0.05)/2 = 0.333. The fact that “Mad Max” has no romantic element does not mean that Janet doesn’t like romantic movies. Equation (8) also encounters the same problem. This might lead to misunderstanding the character of a user in some cases.
This problem can be solved by alleviating the influence of low score features whilst primarily focusing on features with high values. Accordingly, the simplest method is to use the scores themselves as the weights in parallel with normalized ratings to estimate user feature vectors. Specifically for the above example, the score of Janet for the romantic genre is equal to (0.7×0.9×0.9)+(0.72×0.05×0.05)(0.7×0.9)+(0.72×0.05) = 0.854, which is much more reasonable than measuring the affection of a user for a kind of genre based on items that are not relevant to that genre. The biased feature vector of user u weighted by item feature vector pw−biasedu can be formulated as follows.
pw−biasedu =
∑ i∈R(u)
zui · q2i∑ i∈R(u) |zui| · qi (9)
From the user feature vectors calculated using one of the above methods, it is noticeable that each value describing user interests has comparatively the same meaning as the corresponding value in the item feature vector. Therefore, the strength of the relevance between a user and an item can be evaluated using common similarity measures such as Cos or PCC (Section 2.2), which eventually calculates a user-item similarity matrix. In the next section, we demonstrate the effectiveness of these vector representations by integrating user-item similarity into the popular kNNBaseline model and its variants.

heading: 4.2 INTEGRATING THE USER-ITEM CORRELATIONS INTO THE BASELINE ESTIMATE
text: In item-oriented kNNBaseline, the baseline estimate takes the main role of predicting the coarse ratings while the analogy between items serves as a fine-tuning term to improve the accuracy of the final predicted ratings. Furthermore, it also means that the more precise the baseline estimate is to the targeted rating, the better the kNNBaseline models get in terms of prediction accuracy (Duong Tan et al., 2020). However, we realize that a conventional baseline estimate only considers
the biases of users and items separately, ignoring the user-item correlations, which might lead to a rudimentary evaluation approach.
For example, an RS needs to estimate the ratings of user James to two movies “Titanic” and “Mad Max”. Assuming that the average rating, µ, is 3.7 stars. Furthermore, “Titanic” is better than an ordinary movie, so it tends to be rated 0.5 stars above the average. Meanwhile, James is a critical user, who usually rates 0.3 stars lower than a moderate user. Thus, the baseline estimate of “Titanic” rated by James would be 3.9 stars (= 3.7 − 0.3 + 0.5). On the other hand, “Mad Max” tends to be rated 0.6 stars higher than the mean rating; hence, the baseline estimate of James for “Mad Max” would be 4.0 stars (= 3.7 − 0.3 + 0.6). However, from James’s past interactions with other movies, the system estimates James’s interests using one of the methods described in Section 4.1 and discovers that a romantic and drama movie like “Titanic” seems to be very suitable for James while his personality is contradictory compared to an action and thriller movie like “Mad Max”. Consequently, the above predicted ratings of James now turn out to be rather irrational.
As illustrated in Figure 1, there is an approximate-linear relationship between the user-item correlations and the residual ratings: the more interested a user is in a movie (i.e., the larger user-item similarity score), the higher she tends to rate that movie. In order to take the analogy between user and item into account, we propose a revised version of the baseline estimate by integrating the user-item similarity score as follows.
bui = µ+ bu + bi + ω × sui (10) where sui is the similarity degree between user u and item i, and ω serves as the weight to adjust the contribution of the user-item correlation term to fit the rating information.
By introducing ω, the least squares problem of the enhanced baseline estimate term is now updated to the following function.
b∗u, b ∗ i , ω ∗ = argmin bu,bi,ω ∑ u,i∈K (rui − (µ+ bu + bi + ωsui))2
+ λ ∑ u b2u + ∑ i b2i + ∑ u,i∈K ω2  (11)
In this paper, two common optimization techniques, namely SGD and ALS, are experimented to solve this problem. An SGD optimizer minimizes the sum of the squared errors in Equation (11) using the following update rule.
bu ← bu + α(eui − λ.bu) bi ← bi + α(eui − λ.bi) ω ← ω + α(eui.sui − λ.ω)
(12)
where eui = rui − r̂ui is the predicting error, α is the learning rate, and λ is L2 regularization term. Different from SGD, the ALS technique decouples the calculation of one parameter from the others (Koren, 2010). In one iteration, the ALS process can be described as follows. First, for each item i, the optimizer fixes the bu’s and ω to solve for the bi’s.
bi =
∑ u|(u,i)∈K rui − µ− bu − ωsui
λi + |{u|(u, i) ∈ K}| (13)
Then, for each user u, the optimizer fixes the bi’s and ω to solve for the bu’s.
bu =
∑ i|(u,i)∈K rui − µ− bi − ωsui
λu + |{i|(u, i) ∈ K}| (14)
Finally, the optimizer fixes both the bu’s and the bi’s to solve for ω.
ω =
∑ u,i∈K sui(rui − µ− bu − bi)
λω + |K| (15)
Here, the regularization terms λi, λu, and λω are the shrinkage and vary due to the number of the ratings that affect each parameter. Therefore, each parameter of bu’s, bi’s, and ω needs a distinct value of λ, which can be determined by cross-validation. By applying a learnable weighting factor ω to the user-item similarity term, the new kNNBaseline model is capable of exploiting auxiliary information to achieve more precise predictions.

heading: 5 PERFORMANCE EVALUATION
text: To assess the new methods of characterizing user preferences and the proposed baseline estimate in Section 4, Tag Genome in the MovieLens 20M dataset is used to construct a movie feature vector: qi = {gi1, gi2, ..., gik, ...} where gik is the genome score of genome tag kth. In our experiments, pnormu , p biased u , and p w-biased u are first integrated into the traditional baseline estimate to find the optimal technique of profiling user interest in terms of predicting accuracy. The enhanced baseline estimate is then implemented into several neighborhood-based models to comprehensively evaluate its impact on the final rating prediction.

heading: 5.1 ACCURACY OF THE BASELINE ESTIMATE UTILIZING THE USER-ITEM CORRELATION
text: The enhanced baseline estimates are learned using both optimization algorithms SGD and ALS for comparison. For SGD, the baseline are trained using the learning rate α = 0.005 and the regularization λ = 0.02. For ALS, typical values for λu and λi in the MovieLens dataset are 15 and 10, respectively (Hug, 2020). However, the number of training points in set K is much larger than the number of appearances of each user or item, which completely differs the value of λω from λu and λi. Therefore, a grid search is performed on λω , which finds that λω = −9, 500, 000 is the best choice.
Table 2 shows that utilizing the user-item correlation helps to improve the accuracy of the traditional baseline estimate at the price of increased complexity. Empirical results also prove the superior of pw-biasedu over its counterparts for both similarity measures being used. Specifically, calculating the user-item similarity with PCC achieves the coarse rating prediction with 6.46% lower RMSE and 6.71% lower MAE but takes approximately 3.6 times as much time as the original baseline estimate (optimized via ALS). A noteworthy point here is that ALS achieves consistently lower error rates than SGD for all cases at the expense of requiring an additional hyperparameter tuning process (and thus a further computational complexity). However, this trade-off is acceptable at this stage because the absolute time to determine the baseline estimate compared to the total time to make the final prediction is negligible. Hence, ALS is selected as the optimizer for the proposed baseline estimate hereafter.

heading: 5.2 PERFORMANCE OF THE UNIFIED NEIGHBORHOOD-BASED SYSTEM
text: Finally, the advanced baseline estimates are integrated into kNNBaseline and kNNContent models to refine the ultimate rating predictions. For calculating the item-item similarity in kNNBaseline model, two common measures Cos and PCC are examined. The same goes for kNNContent model, where Coscontent and PCCcontent are both implemented for comparison.
As shown in Figure 2, the outperformance of the modified baseline estimates over their original makes a significant improvement in predicting accuracy: the newly proposed neighborhood-based models totally surpass their initial versions for all cases. It is noticeable that even though incorporating sui calculated by PCC is clearly better than Cos when using pnormu , the difference between these two similarity measures gets much smaller in the case of pbiasedu and almost disappears with p w-biased u . This is because the user feature vector generated by Equation (8) or Equation (9) has the original ratings subtracted by the baseline estimate, which makes the mean of the resulting vector come close to 0. Therefore, applying Cos or PCC to the approximately zero-mean vectors produces nearly identical results. In the following experiments, pw-biasedu and PCC are thus opted for calculating the user-item similarity for best accuracy.
Table 3 shows a comparison between the neighborhood-based models incorporating the user-item correlations and several common CF ones. The most accurate model, kNNContent with sui, gains:
• 4.80% lower RMSE and 4.88% lower MAE than original kNNBaseline.
• 2.11% lower RMSE and 2.03% lower MAE than original kNNContent.
• 2.56% lower RMSE and 2.91% lower MAE than SVD.
• 2.22% lower RMSE and 2.10% lower MAE than SVD++.
These improvements in predicting accuracy are achieved at the expense of the additional complexity. However, in practice evaluating the user-item similarity matrix from fixed-length vectors could be performed in parallel with a low computational cost. Hence, we consider that this trade-off is worth it in real-life applications.

heading: 6 CONCLUSION
text: In this paper, we first introduced various techniques to characterize user preferences utilizing both rating data and item content information. The new user representations not only help to understand user interests in each item attribute but also make it possible to measure the user-item correlations. An innovative method was then proposed to adjust the baseline estimate of kNNBaseline model that takes the user-item similarity into account. Thereby, the resulting hybrid models achieve at least 2.11% lower RMSE and 2.03% MAE compared to their neighborhood-based counterparts. This leads to the conclusion that neighborhood-based RSs could be greatly improved by integrating both the item-item and user-item correlations in the predicting model.

Review summaries:
This paper proposed several ideas to improve neighborhood-based recommendation systems by characterizing user preferences using both rating data and item content information. The 2 major proposed ideas are 1) weight item feature by feature score, 2) incorporate user-item similarity score in item score prediction. Experiments over the MovieLens dataset show that the proposed approach can outperform the baselines.

In this paper, the authors study the recommendation systems. They first propose three new methods to characterize user preferences utilizing both rating data and item content information. The first method takes the weighted average of the feature vectors of items, the second one incorporates the effect of biases in the first one, and the third method puts weight on the biased feature vector of users using the item feature vector.  Next, they revise the baseline estimate by integrating the user-item similarity. They further integrate these methods into two baseline models, namely kNNBaseline and kNNContent. Experimental results suggest that these improvements lead to an increase in accuracy (2%-6%) with the expense of the additional complexity. 

This paper incorporates user-item similarity score to item-based CF recommender systems.  The user-item similarity score is calculated by Pearson Correlation Coefficient or Cosine Similarity between user features and item features, where item features are tags, and user features are a weighted sum of (user's rated) items' features (tags). The calculated score are used as part of the baseline estimate of the item-based CF method.

This paper studies hybrid recommendation methods that specifically combines neighborhood-based techniques with content-based user/item feature techniques. It proposes several ways to estimate a user-item correlation matrix and integrates it into neighborhood-based methods as an additional fine-tuning factor. The paper compares proposed methods with several baselines and evaluates RMSE and MAE on MovieLens20M.