Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################

Summary:

This work proposed an inductive recommendation framework on user-item relation graphs. Such a framework relies on the user-item relations without the requirement of side-information and perceives certain flexibility in terms of the parametrization for user/item representations. The authors also provided theoretical analysis to highlight some mathematical insights out of this framework. The proposed method is evaluated on three real-world datasets and compared with several baselines.

Overall I find the work was well-reasoned and executed in a relatively good shape, thus recommending acceptance.


##########################################################################

Strength:
- Relevant topic to the ICRL community and could have potential impact in real-world applications
- The proposed method is well reasoned and technically sound
- Experiments are executed in a decent shape

##########################################################################

Weakness:
- Motivations behind its technical contributions can be further sharpened; comparisons to previous related studies on the inductive graph learning domain can be further improved
- Some gaps between the current experiment setup and real-world recommendation senarios

##########################################################################

Detailed Comments:

I'll address the above potential weakness in details here.

I personally find a bit difficult to digest the motivations of this work and how it differentiated from previous inductive graph learning work until diving into its detailed parametrizations. Fig. 1 and its descriptions are helpful in terms of illustrating the inductive setting, but not quite informative in terms of concrete contributions of this work conceptually.
My takeaway from the proposed framework is, the attentive pooling method falls into the aggregator family of inductive graph learning, despite that the aggregation and sampling scheme are performed on user side globally instead of on the user-item local neighborhoods. In this regard, it may also be helpful to highlight the (mathematical) difference between this work and existing inductive graph learning (e.g. pinSage) after eq.5/6.

Although the experimentations are executed in a good shape, there are still some gaps between the current setup and real-world recommendation requirements. 
- The proposed method is largely evaluated on the rating prediction setting, AUC is reported on the amazon dataset but no Top-K ranking metrics are performed during the experiments. It is acceptable given these metrics are consistent with the optimization objective, however, the notable gap between pointwise prediction setting and the real-world online top-K ranking setting needs to be called out.
- Another concern about the current evaluation protocol is, it enforces the temporal dynamics on the user side and assumes item representations remains the same - again it is consistent with the proposed method (i.e., Q remains the same) thus expected to favor it. The question is, whether these assumptions are consistent with real-world senarios. As far as I know, both movieLens and Amazon datasets have associated timestamps, what the real temporal dynamics here and what would be the warm/cold item/user distribution look like if splitting data chronologically?
	

Minor Concerns:
	- Annotations in  Figure 4 can be further enlarged for visibility

review_title: Interesting work with some caveats

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:  
The work proposes a relational learning scheme that extends standard collaborative filtering approach and aims at improving recommendations quality for new users. The proposed method extracts relations from historical rating data within a preselected subset of support users and utilizes this knowledge to better represent newly introduced (query) users.

Reasons for score:  
The paper misses comparison with a large set of competitive techniques based on incremental learning approach and provides no justification for this omission. Evaluation methodology seem to have test data leak, which may be the major source of performance gains instead of the model itself.

Pros:  
The authors present an interesting view on relational learning problem within the collaborative filtering setting. Generating recommendations online in an instant fashion for both known and new users is indeed a very relevant problem of a great practical importance. The proposed relational learning-based modification of standard collaborative filtering schemes is described clearly and incorporates novel ideas. The authors also prove two theorems, which further support feasibility of their approach and provide some hints on the expected behavior. 

Cons:  
Both in the abstract and in the text, the authors state that standard CF techniques are unable to deal with new users without the need to retrain the entire model. This statement is wrong. For example, in the case of warm start scenario with matrix factorization techniques one can easily update the model specifically for a newly introduced user by performing a few update steps using only ratings of this user. In the case of SGD this would be just a few "half-steps" of gradient descent with fixed matrix of item embeddings. This can be generalized to neural-networks based approaches. Similarly, for the ALS algorithm (e.g., [1]) it would require "half-step" of solving a linear system w.r.t. new user embedding (see eq. 4 in [1]), which can be performed efficiently and in fact is one of the standard approaches in many production systems.
Furthermore, this can be even reduced to an analytical solution in the case of PureSVD approach [2]: it only requires learning item embeddings, and the user embeddings are simply represented as a weighted sum of the embeddings of items they interacted with (see eq. 6 in [2]). It's similar to the way the d_u variable is defined in this paper in eq. 4. Finally, a natural generalization of the latter representation would be an autoencoder, e.g. MultVAE [3] or RecVAE [4], which naturally resolves the warm-start scenario without the need for any modifications as there's no need for a separate user representation.
Therefore, in order to make comparison complete I would suggest to include some incremental learning techniques as well as autoencoder solutions and clearly demonstrate how the proposed method compares to them in terms of recommendation quality, flexibility, and computational efficiency.

The second major point here is the evaluation methodology. In eq. 4, matrices W_q, W_k represent trainable parameters. So how are they trained? Equation 7 explicitly states that training is done on historical data from query users (matrix R_2) and no other data splits are present.  If that's the case, than there's a test data leak: historical rating data of query users is used to train parameters of the model and then the same users are used to evaluate the performance of the model. It doesn't correspond to the warm start scenario. The weights W_q, W_k must be fixed first (after they were trained) and then used to generate representations of users that were never shown to the model before with eq. 6. Hence, there should be at least two disjoint subsets of query users: one for validation and another one for actual testing of the model. Unfortunately, I couldn't find any hints on such a splitting neither in the main text, nor in the Appendix, which makes me believe there's no such splitting. Avoiding test data leaks is absolutely crucial for a fair comparison.

Minor comments:  
The problem of generating recommendations is not the same as the problem of rating prediction/matrix completion. It's important to keep this distinction in mind. The standard task for recommender systems is generating an ordered list of relevant items. The quality of this cannot be measured with RMSE. In fact, there's a strong evidence that models that perform well in terms of RMSE metric may not be good at all in terms of more appropriate metrics like precision, recall, nDCG, MAP, MRR, etc. Please, consider adding more appropriate metrics into the work.
Also note that AUC is not the best choice for that matter as it makes no distinction between proper ranking of irrelevant items and proper ranking of relevant items. Considering that the majority of items in recommendations are typically irrelevant, high AUC scores may not reliably represent actual performance of algorithms in their ability to generate lists of relevant items.

References:  
[1] Hu, Yifan, Yehuda Koren, and Chris Volinsky. "Collaborative filtering for implicit feedback datasets." In 2008 Eighth IEEE International Conference on Data Mining, pp. 263-272. Ieee, 2008.  
[2] Cremonesi, Paolo, Yehuda Koren, and Roberto Turrin. "Performance of recommender algorithms on top-n recommendation tasks." In Proceedings of the fourth ACM conference on Recommender systems, pp. 39-46. ACM, 2010.  
[3] Liang, Dawen, Rahul G. Krishnan, Matthew D. Hoffman, and Tony Jebara. "Variational autoencoders for collaborative filtering." In Proceedings of the 2018 World Wide Web Conference, pp. 689-698. 2018.  
[4] Shenbin, Ilya, Anton Alekseev, Elena Tutubalina, Valentin Malykh, and Sergey I. Nikolenko. "RecVAE: A New Variational Autoencoder for Top-N Recommendations with Implicit Feedback." In Proceedings of the 13th International Conference on Web Search and Data Mining, pp. 528-536. 2020.  
review_title: vote to reject

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Quick summary
This work explores a popular problem, i.e., collaborative filtering, in an inductive setting, which is very important for real-world recommender systems. To address the challenges in the inductive settings, i.e., learning accurate representations for users who do not occur in the training data, the authors propose to construct a relational graph between users in the training data and new users based on a standard matrix factorization model and then use an attentive message passing framework to inductively compute user-specific representations. Besides, the authors prove the expressive and generalization capabilities of the proposed framework. Extensive experiments are conducted to demonstrate the effectiveness of the proposed framework both in transductive and inductive settings, as well as the scalability.

### Clarity
The presentation of the paper is good.

### Originality
Generally speaking, the inductive collaborative filtering is of limited novelty, while the technical solution is novel and solid, especially the part in constructing a relational graph between support users and query users.

### Pros
1. The technical solution is interesting and solid, with a clear presentation in the paper.
2. The proofs of Theorem 1&2 are interesting, which theoretically shows the expressiveness and generalization abilities of the proposed model.
3. The experiments are extensive, most of which are convincing.
4. The presentation of the paper is good.

### Cons
1. The major concern is that the novelty of inductive collaborative filtering with GNN is limited since Zhang & Chen 2020 [1] proposed the IGMC framework, which has done a comprehensive study on the inductive CF problem. Though the authors point out the difference between the proposed IRCF and Zhang's work, they do not give adequate materials to support their arguments. For example, the mentioned disadvantage of IGMC is that *the subgraphs in IGMC are ignorant of user and items indices*, however, from the perspective of the author, this issue is not that important, and may not occur very frequently, and can be trivially addressed by incorporating the user and item indices into IGMC. It will be more convincing if the authors can give more supporting materials in the paper.
2. For Theorem 1, the authors hold one argument that matrix factorization gives maximized capacity for learning personalized user preferences from historical rating patterns, however, it does not make sense to the reviewer. Can you provide any references or explain a bit more? Besides, what are the implications of Theorem 1 in helping us understanding the proposed IRCF ?
3. For Theorem 2, the authors only discuss the influences of the size of $\mathcal{U}_1$. How about other variables, e.g., $B, H, M_2$, etc. 
4. In the performance comparisons, the authors use RMSE in Table 1 and 2, while MAE in Table 3. This seems weird to the reviewers. Why do not you adopt the same metric, say either RMSE or MAE, since the experiments are actually the same type.

Generally speaking, the paper is of high quality. The idea is clear, and the technical solution is interesting and solid with a theoretical guarantee. Most experimental results are convincing. Besides, the writing of the paper is clear and easy to understand.

------
### Post rebuttal
Great thanks to the authors for the detailed replies. After reading them, I decided to keep my rating.
review_title: An interesting paper with limited novelty and solid technical solution

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposed an inductive collaborative filtering method, called IRCF. The goal is to possess expressiveness (against feature-driven methods) as well as generalization (against one-hot encoding based methods). In IRCF, there are a matrix factorization model for support users and a relation model for query users. The former is trained with transductive learning to obtain support users embeddings and item embeddings. The relation model then generates query user embeddings as weighted sum of support user embeddings by examining relational graph between support and query users.

Pros:

1. The paper is well-written and easy to follow. 
2. The experimental results are satisfying. 
3. The Theorem 1 and Theorem 2 reflect the tradeoff between capacity and generalization, which can guide the way of selecting support users. 
4. The idea of using a set of pretrained embeddings as bases may be generalized to other inductive tasks.

Cons/Questions:

1. A detailed related work section is expected. There have been many works studying inductive recommendation problems w/ or w/o user features. As far as I known, lots of methods like FISM generate user embeddings by aggregating embeddings of historical items, which naturally support inductive learning. In this paper, the proposed method IRCF views query user embeddings as weighed sum of support user embeddings, but the weights are still based on aggregating embeddings of historical items (i.e., d_u' in Eq. 4). Moreover, the support users are analogous to a set of bases, and each user can be represented by a combination of the bases. Thus, it is hard to assess the novelty without a related work section.
2. It is unclear on how to handle user bias terms b_u in Eq. 19 and Eq. 22 for query users or new users.
3. It seems that you assume C is a conical combination coefficients in Eq.4. Why not to use the unnormalized scores in Eq. 4, which matches the Theorem 1 better?
review_title: This is a borderline paper and slightly above the threshold

Meta-review: The paper is somewhat borderline, though reviews mostly lean positive. Unfortunately after calibrating compared to other submissions, the work remains somewhat below the bar compared to higher-scoring papers.

The reviewers praise the topic, the method, and the experiments (although some of this praise is a little mixed or lukewarm). The most negative review raises several specific concerns about the evaluation methodology, as well as some concerns about data leaks etc. While serious, the authors rebuttal to these claims seems reasonably convincing. While the remaining issues appear not to be dealbreakers, there are nevertheless some lingering concerns which ultimately put the paper slightly below the bar.

The AC notes that their initial inclination was to accept this paper, though it was suggested that the score be lowered after calibration compared to other submissions, mainly due to doubt regarding these lingering issues.
Decision: Reject