Title: RETHINKING GENERALIZED MATRIX FACTORIZATION FOR RECOMMENDATION: THE IMPORTANCE OF MULTI-HOT ENCODING

Abstract: Learning good representations of users and items is crucially important to recommendation with implicit feedback. Matrix factorization is the basic idea to derive the representations of users and items by decomposing the given interaction matrix. However, existing matrix factorization based approaches share the limitation in that the interaction between user embedding and item embedding is only weakly enforced by fitting the given individual rating value, which may lose potentially useful information. In this paper, we propose a novel Augmented Generalized Matrix Factorization (AGMF) approach that is able to incorporate the historical interaction information of users and items for learning effective representations of users and items. Despite the simplicity of our proposed approach, extensive experiments on four public implicit feedback datasets demonstrate that our approach outperforms state-of-the-art counterparts. Furthermore, the ablation study demonstrates that by using multi-hot encoding to enrich user embedding and item embedding for Generalized Matrix Factorization, better performance, faster convergence, and lower training loss can be achieved.

1. INTRODUCTION:

In the era of big data, we are seriously confronted with the problem of information overload. Recommender systems play an important role in dealing with such issue, thereby having been widely deployed by social media, E-commerce platforms, and so on. Among the techniques used in recommender systems, collaborative filtering (Sarwar et al., 2001; Hu et al., 2008; Su & Khoshgoftaar, 2009; He et al., 2017; Wang et al., 2019) is the dominant one that leverages user-item interaction data to predict user preference. Among various collaborative filtering methods, Matrix Factorization (MF) is the most popular approach that has inspired a large number of variations (Koren, 2008; Rendle et al., 2009; He et al., 2017; Xue et al., 2017) . MF aims to project users and items into a shared latent space, and each user or item could be represented by a vector composed by latent features. In this way, the user-item interaction score could be recovered by the inner product of the two latent vectors. Most of the existing extensions of MF normally focus on the modeling perspective (Wang et al., 2015; 2018) and the learning perspective (Xue et al., 2017; He et al., 2017) . For example, BPR-MF (Rendle et al., 2009) learns user embedding and item embedding from implicit feedback by optimizing a Bayesian pairwise ranking objective function. NeuMF (He et al., 2017) learns compact embeddings by fusing the outputs from different models. DeepMF (Xue et al., 2017) employs deep neural networks to learn nonlinear interactions of users and items.

Although these approaches have achieved great success, they still cannot resolve the inherent limitation of MF. Specifically, apart from the interaction by inner product, there are no explicit relationships between user embedding and item embedding. In other words, the connection between user embedding and item embedding is only weakly enforced by fitting the given individual rating value. However, in real-world scenarios, user embedding and item embedding may be interpreted as some high-level descriptions or properties of user and item, which are supposed to have some explicit connections. For example, a user likes some item, probably because the user and the item share some similar high-level descriptions or properties. Which means, the latent features of a user could be potentially enriched by taking into account the latent features of the user's interacted items, since these interacted items could expose the latent features of the user to some degree. Similarly, the latent features of an item may also be enriched by the latent features of the item's interacted users. However, most of the existing approaches regrettably ignore such useful information. An exception is the SVD++ model (Koren, 2008) , which provides each user embedding with additional latent features of items that the user has interacted with. Despite the effectiveness of the SVD++ model, it suffers from two major problems. First, it only enriches user embedding, and ignores the fact that item embedding could also be enriched by the latent features of users that the item has interacted with. Second, the latent features of the interacted items are averagely integrated without discrimination, while each user normally has different preferences on different items. Note that there are also other approaches that regularize or enrich user embedding and item embedding by exploiting supplementary information, such as social relations (Ma et al., 2011; Guo et al., 2017 ) and text reviews (Chen et al., 2018) . However, in this paper, we do not assume there is any supplementary information, and only focus on the data with implicit feedback. Motivated by the above observations, this paper proposes a novel Augmented Generalized Matrix Factorization (AGMF) approach for learning from implicit feedback. Different from existing approaches, AGMF aims to enrich both user embedding and item embedding by applying multi-hot encoding with the attention mechanism on historical items and users. In this way, user embedding and item embedding are explicitly related to each other, which could further improve the performance of the Generalized Matrix Factorization (GMF) model. Extensive experimental results clearly demonstrate our contributions.

2. PRELIMINARIES:

In this section, we first present the problem statement and then briefly introduce the basic MF model, the GMF model, and the SVD++ model.

2.1. PROBLEM STATEMENT:

Let U = {1, 2, • • • , U } be the set of U users, and I = {1, 2, • • • , I} be the set of I items. We define the given user-item interaction matrix Y = [y ui ] U ×I from implicit feedback data as: y ui = 1 if the interaction (u, i) is observed, otherwise y ui = 0. It is worth noting that y ui = 1 indicates that there is an observed interaction between user u and item i, while it does not necessarily mean that u likes i. In addition, y ui = 0 does not necessarily mean that u does not like i, and it is possible that u is not aware of i. Such setting could inevitably bring additional challenges for learning from implicit feedback, since it may provide misleading information about user's preference. The goal of recommendation with implicit feedback is to predict the values of the unobserved entries in Y, which can be further used to rank the items.

2.2. MATRIX FACTORIZATION:

MF is a basic latent factor model, which aims to characterize each user and item by a real-valued vector of latent features (Koren et al., 2009) . Let p u and q i be the latent vectors for user u and item i, respectively. MF tries to give an estimation ŷui of y ui by the inner product of p u and q i :

ŷui = p u q i = K k=1 p uk q ik (1)

where K is the dimension of the latent vectors. As can be seen, the latent features could be considered as linearly combined in MF. Hence MF can be regarded as a linear model with respect to latent features. This linear property of MF restricts its performance to some degree. As a result, there are an increasing number of approaches (Xue et al., 2017; He et al., 2017; Zhang et al., 2018) proposed to alleviate this problem, by learning a nonlinear interaction function using deep neural networks.

2.3. GENERALIZED MATRIX FACTORIZATION:

Generalized Matrix Factorization (GMF) (He et al., 2017 ) is a simple nonlinear generalization of MF, which makes a prediction ŷui of y ui as follows:

EQUATION

where denotes the element-wise product of vectors, h is a weight vector, and σ(•) is an activation function. To show that MF is a special case of GMF, we can simply set h = 1 where 1 is the vector with all elements equal to 1. In this way, apart from the activation function, the MF model is exactly recovered by GMF, since p u q i = 1 (p u q i ).

2.4. SVD++:

SVD++ extends MF by leveraging both explicit ratings and implicit feedback to make prediction:

EQUATION

where N(u) denotes the set that stores all the items for which u has provided implicit feedback, and c j is a latent vector of item j for implicit feedback, while p u and q i are free user-specific and itemspecific latent vectors specially learned for explicit ratings. The only difference between SVD++ and MF lies in that p u is enriched by

|N(u)| -1 2 j∈Nu c j .

3. THE PROPOSED APPROACH:

Figure 1 illustrates the framework of our AGMF model. It is worthy noting that for the input layer, unlike most of the existing approaches (Koren et al., 2009; He et al., 2018 ) that only employ onehot encoding on the target user's ID (denoted by u) and the target item's ID (denoted by i), we additionally apply multi-hot encoding on user u's interacted items, and item i's interacted users.

In this way, potentially useful information is incorporated, which could enrich the embedding of u and i. Note that this part is the core design of our proposed AGMF model. By enriching the one-hot encoding with multi-hot encoding, the historical interactions between users and items are exploited, therefore our AGMF model with multi-hot encoding achieves superior performance to the GMF model with only one-hot encoding. We argue that this design is simple but more advantageous in threefold. First, it is an augmented version of the GMF model that only takes into account the one-hot embedding of the target user and the target item. Second, it encodes more information than the GMF model, by considering numerous historical interactions of users and items. Such natural information is valuable and should not be ignored for recommendation with only implicit feedback. Third, different from most matrix factorization based recommendation models that only relates user embedding and item embedding by fitting the given individual value, our AGMF model joint learns user embedding and item embedding in a more explicit way.

In order to avoid the conflict that user u may overly concentrate on item i if the target item is i, we will exclude i from N(u) (denoted by N(u)\{i}) when predicting ŷui . Similarly, we will also exclude u from N(i) (denoted by N(i)\{u}) when predicting ŷui . In what follows, we detail elaborate the design of our AGMF model layer by layer.

3.1. INPUT AND EMBEDDING LAYER:

Given the target user u, the target item i, user u's interacted items N(u), and item i's interacted users N(i), we not only apply one-hot encoding on u and i, but also apply multi-hot encoding on N(u) and N(i). In this way, u and i are projected to latent feature vectors p u ∈ R K and q i ∈ R K . Similarly, for each historical item j ∈ N(u)\{i} and each historical user k ∈ N(i)\{u}, we can obtain

{q j ∈ R K |j ∈ N(u)\{i}} and {p k ∈ R K |k ∈ N(i)\{u}}.

3.2. PAIRWISE INTERACTION LAYER:

Following the interaction way used in GMF, we also apply the widely-used element-wise product (He et al., 2017; Cheng et al., 2018; Xue et al., 2018) to model the interactions of u and N(u) as well as i and N(i), Generally, interaction ways such as p u + q j , p u -q j , or any other function that integrates two vectors into a single vector, can also be used. Here, we choose element-wise product because it generalizes inner product to vector space, which could retrain the signal of inner product to a great extent. 

… 1 0 0 0 … 1 0 1 0 … 1 0 1 … 1 0 0 ai(2,1) ai(2,3) multi-hot encoding of N(u)

target item i's one-hot encoding multi-hot encoding of N(i)

Embeding Layer:

Pooling Layer

Pairwise Interaction Layer:

Prediction Layer

  ⊙ ⊙ ⊙ 1 q 2 q 3 q 3 p 2 p 1 p 2 p 3 q 2 q 2 q 1 p 3 p ⊙ 2 p 1 q 2 p 2 q    u f    i f softmax softmax user attention model item attention model

Figure 1 : The framework of our AGMF model.

3.3. POOLING LAYER:

Since there are multiple historical items of user u and multiple historical users of item i, how to extract useful information from these generated latent vectors is crucially important. In reality, the historical items of the target user u normally make different contributions to u on decision of the target item i. The same situation holds for the target item i while interacting with the target user u. Therefore, we perform a weighted sum on the latent vectors obtained from the pairwise interaction layer, i.e.,

p u = j∈Nu\{i} a u (u, j)q j , q i = k∈Ni\{u} a i (i, k)p k (4)

where a u (u, j) denotes the attention weight that the target user u on its interacted item j, and a i (i, k) is the attention weight that the target item i on its interacted user k. Note that a u (u, i) = a i (i, u) does not necessarily hold, as you are my best friend, while I may not be your best friend. p u and q i are the supplementary latent vectors generated by the pooling layer, which will be used to enrich p u and q i . In this paper, we define a u (u, j) (∀j ∈ N u \{i}) and a i (i, k) (∀k ∈ N i \{u}) as the softmax normalization of the interaction scores between users and items:

EQUATION

where

f u (•) (f i (•))

is the user (item) attention model that takes the user-item interaction vector as an input, and outputs the corresponding interaction score. In this paper, we define f u (•) and f i (•) as:

EQUATION

) where h u and h i are the weight vectors of the user attention model and the item attention model, respectively. Note that unlike existing approaches that normally take multi-layer neural networks as the attention model, we only use a single-layer perceptron. In this way, our proposed attention model is exactly a standard GMF model. Our experimental results show that such simple GMF model can achieve satisfactory performance, with keeping simple and efficient. While deeper structures could potentially achieve better performance, we leave the exploration of deeper structures for attention modeling in future work.

3.4. PREDICTION LAYER:

With the supplementary latent vectors p u and q i for p u and q i , inspired by SVD++, we represent the latent vector of user u by p u + p u , and represent the latent vector of item i by q i + q i . Then we reuse the GMF model as the prediction model, and the predicted interaction score ŷui is given by: ŷui = ((p u + p u ) (q i + q i )))

where h is the weight vector of the prediction model. We empirically use the sigmoid function σ(x) = 1/(1 + exp(-x)) as the activation function throughout this paper.

3.5. LOSS FUNCTION:

Since this paper focuses on learning from implicit feedback data, the output ŷui of our AGMF model is constrained in the range of [0, 1], which could provide a probability explanation. In such setting, the commonly used Binary Cross Entropy (BCE) loss could be employed:

EQUATION

where O + denotes the observed interactions and O -denotes the set of negative instances that could be sampled from unobserved interactions. In this paper, for each training epoch, we randomly sample four negative instances per positive instance.

4. EXPERIMENTS:

In this section, we conduct extensive experiments to demonstrate that our AGMF model outperforms state-of-the-art counterparts. In addition, we provide ablation study to clearly demonstrate the importance of multi-hot encoding for generalized matrix factorization.

4.1. EXPERIMENTAL SETTINGS:



4.1.1. DATASETS:

We conduct experiments on four publicly available datasets: MovieLens 1M (ML-1M)1 , Yelp2 , Amazon Movies and Tv (Movies&Tv)3 , and Amazon CDs and Vinyl (CDs&Vinyl). For ML-1M, we directly use the original dataset downloaded from the MovieLens website. Since the high sparsity of the original dataset makes it much difficult to evaluate recommendation approaches, we follow the common practice (Rendle et al., 2009; He et al., 2016) to process the other three datasets. For the Yelp dataset, we filter out the users and items with less than 10 interactions (He et al., 2016) . For Movies&Tv and CDs&Vinyl, we filter out the users that have less than 10 interactions. As this paper focuses on the data with implicit feedback, we mask all the data with explicit feedback to have only implicit feedback by marking each entry 0 or 1, which indicates whether the user has interacted the item. The main characteristics of these datasets are provided in Table 1 . We compare AGMF with the following state-of-the-art approaches:

• SVD++ (Koren, 2008) It merges the latent factor model and the neighborhood model by enriching the user latent feature with the interacted items' latent features.

• BPR-MF (Rendle et al., 2009) It trains the basic MF model by optimizing the Bayesian personalized ranking loss.

• FISM (Kabbur et al., 2013) It is an item-based approach, which factorizes the similarity matrix into two low-rank matrices.

• MLP (He et al., 2017) It learns the interactions between users and items by multi-layer perceptron.

• GMF (He et al., 2017) It generalizes the basic MF model to a non-linear setting.

• NeuMF-p (He et al., 2017) NeuMF is a combination of MLP and GMF, and its pretraining version is called NeuMF-p. In this paper, we compare with NeuMF-p, as NeuMF-p provides better performance than NeuMF without pretraining (He et al., 2017) .

• ConvNCF (He et al., 2018) It employs a convolutional neural network to learn high-order interactions based on the interaction map generated by the outer product of user embedding and item embedding.

4.1.3. TRAINING DETAILS:

We randomly holdout 1 training interaction for each user as the development set to tune hyperparameters suggested by respective literatures. Unless otherwise specified, for all the algorithms, the learning rate is chosen from [5e -5 , 1e -4 , 5e -4 , 1e -3 , 5e -3 ], the embedding size K is chosen from [16, 32, 64, 128] , the regularization parameter (that controls the model complexity) is chosen from [1e -5 , 5e -6 , 1e -5 , 5e -5 ], and the batch size is set to 256. For MLP and NeuMF-p that have multiple fully connected layers, we follow the tower structure of neural networks (He et al., 2017) , and tune the number of hidden layers from 1 to 3. For ConvNCF 4 , we follow the configuration and architectures proposed in (He et al., 2018) . All the models are trained until convergence or the default maximum number of epochs (by respective literature) is reached.

For our proposed AGMF model, no neural network is adopted, hence we do not need to tune the network structure. We initialize the weight vectors by the Xavier initialization (Jia et al., 2014) , and initialize the embedding vectors using a uniform distribution from 0 to 1.

For training AGMF, we employ the Adaptive Moment Estimation (Adam) (Kingma & Ba, 2014) , which adapts the learning rate for each parameter by performing small updates for frequent parameters and large updates for infrequent parameters. We implement AGMF using PyTorch 5 , and the source code as well as the used datasets are released. We fix the embedding size at 128, since we found that a larger embedding size always performs better. Note that the number of interacted users or items may be very large, to mitigate this issue, we truncate the list of interacted users and items such that the latent representation of each user/item is enriched by the latent vectors of at most 50 latest interacted items/users.

4.2. EXPERIMENTAL RESULTS:

In this paper, we adopt the widely used leave-one-out evaluation method (Rendle et al., 2009; He et al., 2016; Bayer et al., 2017) to compare AGMF with other approaches. Specifically, for each dataset, we holdout the latest interaction of each user as the test positive examples, and randomly select 99 items that the user has not interacted with as the test negative examples. In this way, all the algorithms make ranking predictions for each user based on these 100 user-item interactions.

To evaluate the ranking performance, we adopt two widely used evaluation criteria, including Hit Ratio (HR) and Normalized Discount Cumulative Gain (NDCG). HR@k is a recall-based metric Table 2 : HR@5 and NDCG@5 comparisons of different approaches. The best results are highlighted.

ML-1M:

Yelp Movie&Tv CDs&Vinyl HR@5 NDCG@5 HR@5 NDCG@5 HR@5 NDCG@5 HR@5 NDCG@5 BPR-MF ML-1M Yelp Movie&Tv CDs&Vinyl HR@10 NDCG@10 HR@10 NDCG@10 HR@10 NDCG@10 HR@10 NDCG@10 BPR-MF that measures whether the testing item is on the top-k list, and NDCG@k assigns higher scores to the items with higher positions within the top-k list (He et al., 2018) .

4.2.1. PERFORMANCE COMPARISON:

Table 2 and Table 3 show the top-k performance of all the algorithms when k = 5 and k = 10, respectively. From the two tables, we can observe that:

• AGMF achieves the best performance (the highest HR and NDCG scores) on the four datasets.

• Although AGMF is a simple extension of GMF, it still outperforms the complex state-ofthe-art approaches NeuMF-p and ConvNCF.

• Compared with GMF, AGMF achieves significantly better performance. Such success owes to multi-hot encoding with the attention mechanism, which provides enriched information for user embedding and item embedding.

4.2.2. ABLATION STUDY:

As aforementioned, the GMF model makes prediction by ŷui = σ(h (p u q i )), while our AGMF model makes prediction by ŷui = σ(h ((p u + p u ) (q i + q i ))). Clearly, without the supplementary latent vectors p u and q i , AGMF reduces to GMF.

Table 2 and Table 3 have clearly showed that AGMF significantly outperforms GMF. While the used GMF model for performance evaluation is provided by (He et al., 2017) , which is implemented by Keras, and uses a different initialization strategy. Hence it may be slightly different from AGMF without p u and q i . For fair comparison and pure ablation study, we conduct experiments using the codes of AGMF, to compare the performance of AGMF and AGMF without the supplementary latent vectors p u and q i . While with a slight abuse of naming, in this ablation study, we still name AGMF without p u and q i as GMF. Figure 2 reports the comparison results of AGMF and GMF on all the datasets. It can be seen that AGMF always achieves better performance than GMF, in terms of both evaluation metrics. Furthermore, we also report the comparison results of AGMF and GMF in each training epoch in Figure 3 . We can observe that:

• AGMF consistently outperforms GMF in each training epoch.

• AGMF converges faster than GMF.

• AGMF achieves lower training loss than GMF.

By integrating historical interactions into user embedding and item embedding, the above observations are revealed by this paper for the first time. Therefore, the importance of multi-hot encoding for generalized matrix factorization is clearly demonstrated. Moreover, these observations may bring new inspirations about how to properly integrate the one-hot encoding and multi-hot encoding for effectively improving the recommendation performance.

5. CONCLUSION:

Learning good representations of users and items is crucially important to recommendation with implicit feedback. In this paper, we propose a novel Augmented Generalized Matrix Factorization (AGMF) model for learning from implicit feedback data. Extensive experimental results demonstrate that our proposed approach outperforms state-of-the-art counterparts. Besides, our ablation study clearly demonstrates the importance of multi-hot encoding for Generalized Matrix Factorization. As user-item interaction relationships are vitally important for learning effective user embedding and item embedding, hence in future work, we will investigate if there exist better user-item interaction relationships that can be exploited to improve the recommendation performance.

:

https://grouplens.org/datasets/movielens/

https://www.yelp.com/dataset/challenge

https://www.amazon.com/

https://github.com/duxy-me/ConvNCF

https://pytorch.org/

Figures and Tables:

Figure 2: Comparison results of AGMF and GMF on all the used datasets.

Figure 3: Comparison results of AGMF and GMF in each training epoch.

Characteristics of the used datasets.

Table content: <table><tr><td>Dataset</td><td>ML-1M</td><td>Yelp</td><td>Movies&amp;Tv</td><td>CDs&amp;Vinyl</td></tr><tr><td>Number of users</td><td>6040</td><td>25,677</td><td>40,928</td><td>26,876</td></tr><tr><td>Number of Items</td><td>3706</td><td>25,815</td><td>51,509</td><td>66,820</td></tr><tr><td>Number of interactions</td><td>1,000,209</td><td>698,506</td><td>1,163,413</td><td>770,188</td></tr><tr><td>Rating density</td><td>0.04468</td><td>0.00105</td><td>0.00055</td><td>0.00043</td></tr></table>

HR@10 and NDCG@10 comparisons of different approaches. The best results are highlighted.

Table content: <table><tr><td/><td>0.496</td><td>0.344</td><td>0.700</td><td>0.526</td><td>0.633</td><td>0.479</td><td>0.671</td><td>0.523</td></tr><tr><td>SVD++</td><td>0.541</td><td>0.376</td><td>0.703</td><td>0.528</td><td>0.631</td><td>0.480</td><td>0.666</td><td>0.520</td></tr><tr><td>FISM</td><td>0.528</td><td>0.372</td><td>0.691</td><td>0.511</td><td>0.583</td><td>0.452</td><td>0.592</td><td>0.457</td></tr><tr><td>MLP</td><td>0.526</td><td>0.362</td><td>0.671</td><td>0.498</td><td>0.570</td><td>0.425</td><td>0.588</td><td>0.445</td></tr><tr><td>GMF</td><td>0.540</td><td>0.372</td><td>0.676</td><td>0.507</td><td>0.569</td><td>0.427</td><td>0.620</td><td>0.481</td></tr><tr><td>NeuMF</td><td>0.548</td><td>0.381</td><td>0.695</td><td>0.521</td><td>0.596</td><td>0.453</td><td>0.629</td><td>0.491</td></tr><tr><td colspan="2">ConvNCF 0.549</td><td>0.391</td><td>0.708</td><td>0.532</td><td>0.634</td><td>0.484</td><td>0.673</td><td>0.525</td></tr><tr><td>AGMF</td><td>0.561</td><td>0.393</td><td>0.716</td><td>0.547</td><td>0.644</td><td>0.497</td><td>0.685</td><td>0.542</td></tr></table>

Peer review (iclr2020):
The work presents a matrix factorization (MF) framework for enforcing the effect of historical data when learning user preferences in collaborative filtering settings. The key idea is to augment vector representations of users and items with additional terms related to other interactions from the observation history. The standard inner product between entities is then replaced with its generalized non-linear counterpart. The authors use a non-linear AGMF model that incorporates both user and item profiles in a symmetric way and utilizes smart parametrization to learn the corresponding weights of all involved observations. Even though I generally like the presented approach, I believe the work misses some key points and should be rejected. Most importantly, the conceptual model design has a limited novelty and is basically reinvented. Likewise, the experiments part lacks comparison with appropriate baselines, which raises concerns on actual performance of the proposed approach. Main argument: The authors claim that most of the existing approaches do not take into account additional interactions data and that the only model that provides such functionality is SVD++. This is not true. First of all, the SVD++ model itself was not the first one to propose such a design. Similar augmentation was implemented in NSVD model by [Paterek 2007] (a user vector was replaced by a combination of all items in user profile). Even though it was simpler than SVD++ it served as an inspiration for later models. You could simply write the NSVD model in the form SQQ^T, where Q encodes item embeddings and SQ represents user embeddings (no separate user space was learned). Matrix S serves as the design matrix which is essentially the one-hot encoding (can be “multi-hot” using the authors’ term). The SVD++ model can be written in similar way as XPQ^T, where X is a new design matrix, which encodes all user interactions and allows to augment standard scalar product with additional terms. The model can be extended further to XP(YQ)^T with the addition of item-side design matrix Y, which now resembles the AGMF approach proposed in the paper at review. Such a formulation is used in a number of models, e.g., SVDFeature by [Chen et al. 2011] or LightFM by [Kula 2015]. Even though they are typically used for hybrid systems (to incorporate side information about users and items), they are perfectly suitable for the considered here task as well. Probably, one of the earliest published work that utilizes such a symmetric design is MatchBox by Microsoft [Stern, Herbrich, Graepel 2009] (they use the notion of user and item traits). Finally, the most general framework for the task is presented by the Factorization Machines (FM) proposed by [Rendle 2010], which is widely used in industry as well as in various recsys challenges. This model has many implementations and should be considered as the default baseline for this kind of work. Excluding models like FM or at least LightFM from comparison makes the work incomplete, especially considering that the authors emphasize the importance of taking interactions history into account by comparing AGMF with a simpler GMF model. Regarding the source code, it’s nice that it has more or less complete state in a sense that’s it’s runnable with almost no modification. I was able to run it on my GTX 1060; however the training was pretty slow. Note, that even simple and fast SVD-based model can be tuned to provide a decent quality of recommendations, see EigenRec model (for example, compared to other popular models in the RecWalk paper by [Nikolakopoulos and Karypis 2019]). I used the scaling trick from EigenRec (see, e.g., https://www.eigentheories.com/blog/to-svd-or-not-to-svd/ for more details on how to do scaling). I was able to get HR=0.713 and NDCG=0.446 on the Movielens dataset with this modified SVD model. Computing of the optimal result (with rank 128) takes only 1.2 seconds on my 4-years old laptop. The comparable level of HR and NDCG was achieved by AGMF only after approximately 3.5 hours of training (each epoch was taking 6-7 minutes and it required around 30 epochs). In fact, the entire hyper-parameter search time for the SVD-based model took less than 4 minutes, which was dominated by the evaluation time not the model computation time (for any set of hyper parameters you need to compute SVD only once for the highest rank value, as the model with lower rank values can be obtained by a simple rank truncation procedure). This raises the concern in an overall practicality of the AGMF approach. The best score is only 2.5% higher than that of SVD and takes more than 8 hours to get (on GTX 1060) vs. 1.2 seconds for SVD (computed on 4-core CPU). Recommender systems is a very applied field. Hence, the complexity analysis as well as computation time comparison should also be part of the work. Potential scalability issues should be explicitly explained and not left for the readers to discover. Things to improve the paper that did not impact the score: 1) Have you tried to work with the version of NeuCF that is modified the same way as AGMF? NeuCF provides the general framework that also allows to incorporate interactions history into inner product computations via “feature vectors”. According to the authors, NeCF > GMF, which means that potentially NeuCF with extended user and item description, similarly to AGMF, could also perform competitively. I believe, it would make the work better from the perspective of fair comparison. It’s important to note, however, that there’s a certain evidence that recent neural network-based approaches do not actually outperform simpler linear models, see work by [Dacrema, Cremonesi, Jannach 2019] on “A Worrying Analysis of Recent Neural Recommendation Approaches”. For example, NeuCF was not better than SLIM and in some cases even worse than KNN-based models. It would be better to add at least some of this baselines into the comparison. 2) Accordingly, I’d like to see the source code for entire experiments showing how experiment is conducted, how baseline models are tuned, etc. The source code attached for the review only contains basic training for one dataset and only for AGMF model. 3) Recommender systems data is noisy. It’s hard to make a reliable comparison of models without an analysis of statistical significance of different results. I would suggest to employ a different evaluation strategy with user-based k-fold cross validation (each fold consisting of 1/k fraction of users). This would allow to report confidence intervals for your results and demonstrate reliability of your conclusions. 4) I believe the term “multi-hot” encoding is misleading. FM also uses the same representation; however it’s still called “one-hot”. References: Paterek, Arkadiusz. *Improving regularized singular value decomposition for collaborative filtering.* In Proceedings of KDD cup and workshop, vol. 2007, pp. 5-8. 2007. Chen, Tianqi, Weinan Zhang, Qiuxia Lu, Kailong Chen, Zhao Zheng, and Yong Yu. *SVDFeature: a toolkit for feature-based collaborative filtering.* Journal of Machine Learning Research 13, no. Dec (2012): 3619-3622. Kula, Maciej. *Metadata embeddings for user and item cold-start recommendations.* arXiv preprint arXiv:1507.08439 (2015). Stern, David H., Ralf Herbrich, and Thore Graepel. *Matchbox: large scale online bayesian recommendations.* In Proceedings of the 18th international conference on World wide web, pp. 111-120. ACM, 2009. Rendle, Steffen. *Factorization machines.* In 2010 IEEE International Conference on Data Mining, pp. 995-1000. IEEE, 2010. Nikolakopoulos, Athanasios N., and George Karypis. *Recwalk: Nearly uncoupled random walks for top-n recommendation.* In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pp. 150-158. ACM, 2019. Dacrema, Maurizio Ferrari, Paolo Cremonesi, and Dietmar Jannach. *Are we really making much progress? A worrying analysis of recent neural recommendation approaches.* In Proceedings of the 13th ACM Conference on Recommender Systems, pp. 101-109. ACM, 2019.