Inductive Collaborative Filtering via Relation Graph Learning
Authors: Qitian Wu
Hengrui Zhang
Xiaofeng Gao
Hongyuan Zha
Venue:  (2021)
Keywords: collaborative filtering
matrix completion
inductive learning
relation learning
recommender systems

Abstract: Collaborative filtering has shown great power in predicting potential user-item ratings by factorizing an observed user-item rating matrix into products of two sets of latent factors. However, the user-specific latent factors can only be learned in transductive setting and a model trained on existing users cannot adapt to new users without retraining the model. In this paper, we propose an inductive collaborative filtering framework that learns a hidden relational graph among users from the rating matrix. We first consider a base matrix factorization model trained on one group of users' ratings and devise a relation inference model that estimates their underlying relations (as dense weighted graphs) to other users with respect to historical rating patterns. The relational graphs enable attentive message passing from users to users in the latent space and are updated in end-to-end manner. The key advantage of our model is the capability for inductively computing user-specific representations using no feature, with good scalability and superior expressiveness compared to other feature-driven inductive models. Extensive experiments demonstrate that our model achieves state-of-the-art performance for inductive learning on several matrix completion benchmarks, provides very close performance to transductive models when given many training ratings and exceeds them significantly on cold-start users.

Under review as a conference paper at ICLR 2021
INDUCTIVE COLLABORATIVE FILTERING VIA RELA- TION GRAPH LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Collaborative filtering has shown great power in predicting potential user-item rat- ings by factorizing an observed user-item rating matrix into products of two sets of latent factors. However, the user-specific latent factors can only be learned in transductive setting and a model trained on existing users cannot adapt to new users without retraining the model. In this paper, we propose an inductive collabo- rative filtering framework that learns a hidden relational graph among users from the rating matrix. We first consider base matrix factorization model trained on one group of users' ratings and devise a relation inference model that estimates their underlying relations (as dense weighted graphs) other users with respect to historical rating patterns. The relational graphs enable attentive message pass- ing from users ner. The key advantage of our model the capability for inductively computing user-specific representations using no feature, with good scalability and superior expressiveness compared to other fcature-driven inductive models. Extensive ex- periments demonstrate that our model achieves state-of-the-arr performance for inductive learning on several matrix completion benchmarks, provides very close performance to transductive models when given many training ratings and exceeds them significantly cold-start users.
in the latent space and updated in end-to-end man-
INTRODUCTION
As information explosion has become one major factor affecting human life in the decade, recom- mender systems, which can filter useful information and contents of user's potential interests, play an increasingly indispensable part in day-to-day activities. Recommendation problems can be gen- crally formalized as matrix completion (MC) where one has a user-item rating matrix whose entries, which stand for interactions of users with items (ratings click behaviors), are partially observed. The goal of MC is to predict missing entries (unobserved or future potential interactions) in the matrix based the observed ones.
Modern recommender systems need to meet two important requirements in order for desirable effec- tiveness and practical utility. First of all, recommendation models should have enough expressive- ness to capture diverse user interests and preferences so that the systems can accomplish personal- ized recommendation. Existing methods based on collaborative filtering (CF) or, interchangeably, matrix factorization (MF) have shown great power in this problem by factorizing the rating matrix into classes of latent factors (i.e., embeddings) for users and items respectively, and further lever- age dot-product of two factors to predict potential ratings (Koren et all 2009;; Rendle et all 2009 Srebro et all, 2004: Zheng all, 20166). Equivalently, for each user. the methods consider a one-hot user index as input, assume a user-specific embedding function (which maps user index to a latent factor), and use the learnable latent factor to represent user's preferences in low-dimensional space. One can select proper dimension size control balance between capacity and generalization. Re- cent works extend MF with complex architectures, like multi-layer perceptrons (Dziugate & Roy, 2015), recurrent units (Monti et all 2017), autoregressive models (Zheng all, 20168), graph neural networks (van den Berg et all, 201 etc., and achieve state-of- tfate-o-thhe-ee -tthh-rrt results on most benchmarks.
The second requirement stems from a key observation from real-world scenarios: recommender sys- tems often interact with a dynamic open world where new users. who are not exposed to models during training, may appear in test stage. This requires that models trained one group of users

---

Under review as a conference paper at ICLR 2021
user-item rating matrix
item embedding
Lssers
items
quary support
1
0
0
transductive matrix completion
0
2
matrix factorization
1
2 5
3
0
5
3
1 3
user embedding
4
5
inductive matrix completion
relation inference
hidden relational graph
1
3
4
5
message
passing
embedding
5
4
2
Figure I: Model framework of inductive relational matrix factorization.
manage adapt to unseen users. However, the srove-mentioned CF models would fail in this situ- ation since the user-specific embeddings are parametrized for specific users and need to be learned collaboratively with all other users in transductive setting. One brute-force way is retrain the whole model with an augmented rating matrix, but extra time cost would be unacceptable for online systems. There are quite a few studies that propose inductive matrix completion models using user features (Jain & Dhillon, 2013 all 20L3: Cheng et all. 2016; Ying et all, 2018; Zhong et all 2018). Their different thinking paradigm is to target user-sharing mapping from user features to user representations, instead of from one-hot user indices used by CF models. Since the feature space is shared among users, such methods are able to adapt a model trained on existing users to unseen users. Nevertheless, feature-driven models often suffer from limited expressiveness with low-quality features that have weak correlation with target labels. For example, users with the same age and occupation (commonly used features) may have distinct ratings on movies and music. Un- fortunately, high-quality features that can unveil user interests for personalized recommendation are often hard to collect due to increasingly concerned privacy issues.
A following question arises: Can we have a recommendation model that guarantees enough ex- pressiveness for personalized recommendation and enables inductive learning? In fact, simul- laneously meet the two requirements is a non-trivial challenge when high-quality user features are unavailable. First, to achieve either of them, one needs to compromise on the other. In fact, the one-hot user indices (together with learnable user-specific embeddings) give a maximized capacity for learning distinct user preferences from historical rating patterns. To make inductive learning possible, one needs to construct shared input feature space among users out of the rating matrix, as an alternative to one-hot user indices. However, the new constructed features have relatively in- sufficient expressive power. Second, the computation based on new feature space often bring extra costs for time and space, which limits model's scalability to large-scale datasets.
In this paper, we propose an inductive collaborative filtering model (IRCF) @ as general CF frame- work that achieves inductive learning for matrix completion and meanwhile guarantees enough ex- pressiveness and scalability. As shown in Fig. M. consider a base transductive matrix factorization model trained on one group of users (called support users) and a relation inference model that aims to estimate their relations to another group of users (called query users) w.r.t. historical rating pat- terns. The (multiple) estimated relational graphs enable attentively message passing from users to users in the latent space and compute user-specific representations in an inductive way. The output user representations can be used to compute product with item representations to predict ratings in the matrix, as is done by CF models. Compared with other methods, one key advantage of IRCF is the capability for inductively computing user-specific representations without using features.
Besides, our method possesses the following merits. 1) Expressiveness: general version of our model can minimize reconstruction loss to the same level as matrix factorization under a mild con- dition. Also, we qualitatively show its superior expressiveness than feature-driven and local local-graph- based inductive models that may fail in some typical cases. Empirically, IRCF provides very close performance to transductive CF models when given sufficient training ratings. 2) Generalization: IRCF manages to achieve state-of-the-art results on new (unseen) users compared with inductive models. Also, IRCF gives much better accuracy than transductive models when training data be- comes sparse and outperforms other competitors in extreme cold-start recommendation. 3) Scal-
'The codes will be released.
2

---

Under review as a conference paper at ICLR 2021
ability: Our model enables mini-batch training and efficient inference. In experiments. IRCF is averagely ten times faster than local-grapy graph-basee inductive model. 4) Flexibility: As a general CF framework, IRCFis flexible to incorporate with various architectures (e.g. MLP- MLP-based, GNN- basedd autoregressive, etc.) as a base transductive model as well as deal with implicit user feedbacks.
2 BACKGROUND AND RELATED WORKS
In this section, we present some background of the problem and discuss relationships to related works. We consider general matrix completion (MC) problem which deals with user-item rating matrix f {r..) ar N where M and N are the numbers of users and items, respectively. For implicit feedback, rui is a binary entry which denotes whether user u rated (or clicked on, reviewed, liked, purchased, etc.) item item i. The entries of R are partially observed and the goal is to estimate the missing values in the matrix In the following, we introduce related works and highlight their differences to our paper. In Fig. EX we provide an illustration for comparison with these methods.
not. For explicit feedback, Tui records rating value of user on
General Collaborative Filtering. Existing methods for MC are generally based on collaborative filtering (CF) or, interchangeably, matrix factorization (MF) where user u (resp. item i) corresponds to d-dimensional latent factor (i.e., one-hot embedding) Pa (resp. q:).Then one has prediction model fus M(P,, q.) where f can be basically specified as simple dot product or some complex architectures, like neural networks, graph neural networks, etc. One advantage of CF models is that the user-specific embedding p., (as learnable parameters) can provide enough expressive power for learning diverse personal preferences from user historical behaviors and decent generalization ability through collaborative learning with all the users and items. Furthermore, the user embed- dings possess rich profile information as representation of user preferences and can benefit various downstream tasks, like target advertisement, user-controllable recommendation (Ma et all 2019 all 2020), influence maximization (Khalil all 2010 Manchanda et all 2011), friend rec- ommendation, etc. However, such user-speeific embedding limits the model in transductive learning and when it comes to new users during test, one has to retrain the model (often associated with embeddings for both new and existing users as well items) with new augmented rating matrix. Admittedly, one can consider local updates for embeddings of new users with fixed item embed- dings learned from existing users. Nevertheless, such operation makes model learning for users independent from each other, which is prone for over-fitting compared with collaborative learning in CF models. Besides, it requires 'incremental" learning for each new user, while IRCF can deliver on-the-fl fly inference in online systems.
Feature-driven Recommendation. The CF models do not require any side information other than the rating matrix, but cannot be trained inductively due to the learnable user-specific embedding Pa- To address the issue, one can leverage side information such as attribute features to achieve inductive learning. Define user features (like age, occupation, etc.) as a,, and item features (like movie genre, director, etc.) as b,. The feature-driven model targets a prediction model f.r = g(a., b,). Since the space of a. is shared among users, model trained on one group of users can adapt to other users without retraining. However, feature-driven models often provide limited performance since the shared feature space is not expressive enough compared to one-hot embedding space. Another issue is that high-quality features are hard to collect in practice. We note that our model does not require user features for inductive collaborative filtering.
Inductive Matrix Completion. There are a few existing works that attempt to handle inductive matrix completion using only user-item rating matrix. (Harlford at 2018) (F-EAE) puts forward an exchangeable matrix layer that takes a whole rating matrix as input and inductively outputs pre- diction for missing ratings. However, the scalability of F-EAE is limited since it requires the whole rating matrix as input for training and inference for users, while IRCF enables mini-bate training and efficient inference. Besides, (Zhang & Chen, 2020) (IGMC) proposes to use local subgraphs of user-item pairs in a bipartite graph of rating information as input features and further adopt graph neural networks as representation tool to encode subgraph structures for rating prediction. The model achieves inductive learning via replacing users' one-hot index embeddings by shared input features (i.c., index-free local subgraph structures). Differently, IRCF maintains the ability to give
2 Also, in some situations, one targets ranking list of items for each user as top-N recommendation. In this paper, we focus on predicting missing values in the matrix and leave top-N recommendation for future works.
3

---

Under review as a conference paper at ICLR 2021
user-specific embeddings, which represent users' preferences and interests and can be used for down- stream tasks (like target advertisement, user-controllable recommendation, influence maximization, etc.) while F-EAE and IGMC merely output a prediction score without such user representation. Moreover, the expressiveness of IGMC is limited since the local subgraph structures can be indistin- guishable for users with distinct behaviors (see Section 3.2 for more discussions), while IRCF has equivalent expressiveness as original CF models.
Item-based CF Models. Some previous works use item embeddings as representation for users. (Cremonesi et all 20T0; KKbbbu et all 2013) proposes to use a combination of items rated by users to compute user embeddings and frees the model from learning parametric one-hot embeddings for users. Furthermore, there are quite few auto auto-encoder architectures for recommendation problem, leveraging user's rating vector (ratings on all the items) as observed input, estimate user embedding (as latent variables) based on that, and output prediction for missing values in the rating vector (Sede
ha 201%% Liang et al, 2018). R) With item embeddings and user's rating history, these methods achieve inductive learning for users and can adapt to users during test. On methodological level, IRCF has the following differences: 1) IRCF learns to use weighted combination of users' embed- dings compute embeddings for new users, and such combination weights possess interpretability for underlying social influence or user proximity; 2) IRCF considers both users' and items' one-hot embeddings in general CF models that maintain better capacity than item-based CF models that only considers learnable parameters in item embedding space.
3 METHODOLOGY
We propose new inductive collaborative filtering model without using features. We borrow the general idea from graph neural networks (GNNs) which define a computational graph over nodes and edges and further harness message passing through edges to obtain new node representations that aggregate neighbored information. Our high-level methodology stems from a key observation: there exist (or multiple) latent relational graph among users that represents preference proximity and behavioral interactions Based on the relational graphs, we can leverage message passing, propagating learned embeddings from one group of users to others, especially, in an inductive way.
We formulate model through two sets of users: support users (denoted by U4), for which we learn their embeddings in transductive setting, and query users (denoted by U2), for which we consider message passing inductively compute their embeddings. Assume [ M1 and [24 M2. Cor- respondingly, we have two rating matrices: R = frui ( M, xN (given by (41 and R2 = {rw'sh M (given by U2). Note that there is no strict requirement for Ui and U2 in our formulation. Two com- mon settings are Ui n U2 = 0, where query users are distinct from support users, and 2n = U2. where support users equal to query users in training. We consider both cases in Section 4.
We train a (transductive) matrix factorization model for Uj using R1, denoted as f.i = fe q.). where Pa. RR denotes user-specific embedding for user u in U1, q. E Rd denotes item-specific embedding for item i and fa can be simple dot-product or a neural network with parameter 0. In Appendix Q we present details for two specifications for fo using neural network and graph convo- lution network, which are used in our implementation. Denote P1 = {Pu] M, xd. Q = {q, Nxd and the objective function becomes
min Ds, (Ri, Ri), P,Q,8
(1)
where Ri (Fui) XN. Ds, (fi, R1) set with size Ti containing indices of observed entries Ri Here one can use cross-entropy or square loss for l(Tai, Pui).
E E(.i)sss 1(rui, Pui) and S E ([M) x [N]) is
The key question is how learn the relational graph among users and consider message passing for inductive representation learning. We next propose our inductive relation inference model.
Social networks and following networks in social media can he seen realizations of such relational graphs, but in most cases, the graphs unobserved and implicitly affect user's decisions and behaviors.
4

---

Under review as a conference paper at ICLR 2021
3.1 INDUCTIVE RELATION INFERENCE MODEL
Assume C = {Can' JM x Ma. where Cuu E R denotes weighted edge from user u E Ui to user u' e U2, and define Cu [Cl',, -2ul:".... CM,'' the u'-th column of C. Then we express embedding of user u' as Pu' = P1 the weighted sum of embeddings of support users. The rating can be predicted by Pw's fo q:) and the problem can be formulated as
min Ds (R2, R2), CQQ
(2)
where Rs = (fws)MaxN)
- E is a set with size T3 containing indices of observed entries in R2. The essence of above method is taking attentive pooling as message passing from support to query users. We first justify this idea by analyzing its capacity and then propose parametrized model that enables it for inductive learning.
- Jee, (Tari, fwr)) and S2 E ([M2) x
[N])
Theoretical Justification we use dot-product for fo in the MF model, then we have Pa's P We compare (Z) with using matrix factorization over R2:
min Ds. (R2, R2),
Q
where P2 = (Pw) } M2xd and have the following theorem.
Theorem 1. Assume (G) can achieve Ds. (R2, R2) < and the optimal P1 given by (0) satisfies fall then there exists at least one solution for C in (2) such that Ds, (R2, R2) <
The only condition that P1 is column-full-rank can be trivially guaranteed since d << N. The theorem shows that the proposed model can minimize the reconstruction loss of MC to at least the same level as matrix factorization which gives maximized capacity for learning personalized user preferences from historical rating patterns.
Parametrization We showed that using attentive pooling does not sacrifice model capacity than CF models under a mild condition. However, directly optimizing over C is intractable due to its O M2) parameter space and C, is user-specific which disallows inductive learning. Hence, we parametrize C with an attention network, significantly reducing parameters and enabling it for inductive learning, Concretely, we estimate the adjacency score between user u' and user u as
[ [ xPo]
Cu'u Endeth e [W dw W&PPo
(4)
where e 22dx1 Wg E Rdxd WR E Rdd are trainable parameters, || denotes concatenation and d. = EiEI, Here Iw = filmure > 0} includes the historically rated items of user u'. The attention network captures first-order user proximity on behavioral level and also maintains second- order proximity that users with similar historical ratings on items would have similar relations to other users. Besides, if Iw is empty (for extreme cold-start recommendation) we can randomly select a group of items or use (the embedding of) user's attribute features as d, if features are available. We provide details in Appendix D.
The normalization in requires computation for all the support users, which limits scalability to large dataset. Therefore, we use sampling strategy to control the size of support users in relation graph for each query user and further consider multi-head attentions that independently sample different subsets of support users. The attention score given by the I-th head is
(e(0) T [w! d d/ PP.
(5)
Em (n (ell)) [W()) |
( P
where 4 denotes a subset of support users sampled from U4, Each attention head independently aggregates embeddings of different subsets of support users and the final inductive representation for user u' can be given as
P W.
Σ (1) Sau
(
Pull aeth
Σ
Pal Pull
Σ weth
(L) Cuu (L)
Pu
(6)
Lueth
5

---

Under review as a conference paper at ICLR 2021
where W. RdxLd and w! Rdddd To keep the notation clean, we denote Pu' hww (dwe) and w = U[sfoll) w g w w (W.)-
There exist some common thinkings as for the rationals of model and inductive graph repre- sentation learning (Hamilton et all, 2017; Ying et al,, 2018: Zhang et al, 2018), which considers message passing from existing nodes new nodes over graphs. Differently, IRCF jointly estimate neighbored nodes for target node (given by attention scores) and learn node representations based on that, while the latter often assumes given observed graph and directly learn node representa- tions. Furthermore, IRCF can deal with nodes with new users with no historical edge, while the latter would fail for new nodes with no observed edge (if without node attribute features).
Optimization The training process consists of two contiguous stages. First, we pretrain MF model in transductive setting via (l)) and obtain embeddings Pl, Q and network fo. Second, we train our relation model he with fixed P1, Q via
min Ds. (R2, R2)- w.
We found that using fixed Q in the second stage contributes to much better performance on test ratings of query users than optimizing over it.
We further analyze the generalization ability of inductive model on query users. Also, consider fa as dot-product operation and we assume Cu'u E R+ to simplify the analysis. In the next theorem, we show that the generalization error D(R2, R2) Egu ( (rrss, Pars) on query users would be bounded by the numbers of support users and observed ratings of query users.
Assume that 1) 'D is L-Liscchitz 2) Vfa's E R2 we have lfwsl S B, and 3) the LI-norm of cw bounded by H. Then with probability at least 8 over the random choice of S2 E ([M2) [N]) holds that for any R2.
Theorem 2.
2M In M1
In(1/8)
D(Rg, R2) s Ds, RR R2) +0
2LHB
(8)
T2
The theorem shows that smaller size of Ui would make the gen- eralization еггоr bound tighter. Looking at both Theorem and 2, we will find that the configuration of U4 has an important effect on model capacity and generalization ability. On one hand, we need to make support users in 212 'representative" of diverse user behavior patterns on item consumption in order to guarantee enough model capacity. Also, we need to control the size U2 in order to main- tain generalization ability. Based on these insights, how to properly select support users can be an interesting direction for future in- vestigation. Our experiments in Section 4 provide more interesting discussions on this point.
(a)
(b)
Figure 2: Feature-driven and local-graph-based models fail in (a) and (b), respectively. IRCF works in both cases with superior expressiveness.
3.2 DISCUSSIONS
To shed more lights on model expressiveness, we provide comparison with feature-driven and local-graph-based inductive models through cases in Fig. A Here we assume ratings are within {- (positive, denoted by red line, and negative, denoted by black line). The solid lines are observed ratings for training and dash lines are test ratings. In Fig. D (a). local-graph-baat based models can give right prediction relying on different 1-hop local subgraphs of (wi, i2) and (u2, while feature-driven models will fail once #1 and u2 have the same features. In Fig. Ø (b), local-graph- based models will fail since (w,, iz) and (u2,4a) possess the same 1-hop local subgraphs though with distinct ground-truth ratings on ig. By contrast, CF models and IRCF can recognize that us has similar rating patterns with ii and different from u2 thus pushing the embedding of #j (resp. uz) close to (resp. distant from) ug, which guides the model to right prediction. Note that the first case becomes a common issue when the feature space is small while the second case becomes general when the rating patterns of users not distinct enough throughout dataset, which induces similar local subgraph structures. In short, IRCF achieves inductive learning and maintains as good expressiveness transductive models.

---

Under review as a conference paper at ICLR 2021
Table RMSES for all the users (AlI), query users (Query) and new users (New) in different datasets. We highlight the best scores among all the (resp. inductive) models with bold (resp. underline).
Douban
ML-100K
ML-IM Query
Method
Inductive
Feature
All
Query
New
Query
New
All
New
PMF
No
No
0.737
0.718
0.932
1.003
0.851
0.946
NNMF
No
No
0.729
0.712
0,925
0,987
0.848
0,940
GCMC
No
No
0.731
0.710
0.911
0.989
0.838
0.941
NIMC
Yes
Yes
0.772
0.745
0.766
1.015
1.065
1.089
0.873
0.995
1.059
PinSAGE
Yes
Yes
0.769
0.743
0.763
1.008
1.055
1.083
0.859
0,961
1.055
BOMIC
Yes
Yes
0.735
0.713
0.764
0.931
1.001
1.088
0.847
0.953
1.057
F-EAE
Yes
No
0.738
0.920
0.860
IGMC
Yes
No
0.721
0.718
0.743
0.905
0,997
1.031
0.857
0.956
0.997
IRCF-NN
Yes
No
0.731
0.712
0.749
0.931
0.999
1.057
0.844
0,952
0.991
IRCF-GC
Yes
No
0.732
0.712
0.719
0.905
0.981
0.999
0.839
0.944
0.956
Table 2:
RMSES (resp. AUCs)
Table 3: RMSEs on
10a
on query users in ML-10M (resp. cold-start users in ML-
Training time per eporc (4)
IRC (wans)
IM using features.
1GMC
-100
ML-10M better)
Amazon (t better)
ML-1M
Method
Method
better)
PMF
0.928
0.917
Wide&Deep
1.0260
20f
NNMF
0.922
0.921
GCMC
0.9688
GCMC
0.919
0,922
AGNN
1.0087
10a Dataset size
IRCF-NN
0.924
0,942
MeLU
0,9625
Figure 3: Scalability test in ML-1M.
IRCF-GC
0.919
0.948
IRCF-HY
0,9367
4 EXPERIMENTS
In this section, we conduct experiments to verify the proposed model. We consider five benchmark datasets Douban, Movielens- /00K (ML-100K), Movielens- /M (ML-IM), Movieleny-IOM (ML-10M) and Amazon-
The statistics datasets and more details are in Appendix E. For Douban and ML-100K, (van den Berg et all 2017 Hartood ef all 2018: Zhang & Chen, 2020). For ML- ML-IM and 1OM, we also follow previous works and use 9:1 training/test spliting. For Amazon Boooks with implicit user feedbacks, we use the last ten ratings of each user for test and the remaining for training.
the training/test split provided by (Moni all 2017),, which is used by
We consider two specifications for fe in model: IRCF-NN, which adopts multi-layer perceptron for f (following the design of NNMF (Dzuugate & Roy, 2013), and IRCF-GC, which uses graph convolution network for (following the design of GCMC (van den Berg et all 2017).. The details are in Appendix C We also extend IRCF with user attribute features as IRCF-HY, a hybrid model that takes both user onc-hot indices and features input (see details in Appendix D.I.), for cold- start recommendation. IRCF-HY considers mee-ho//multi-oot embeddings for various features and further a neural network for prediction, following the design of Wide&Deep network (Cheng et all 2016). In Appendix ERA we present detailed information for hyper-parameter settings. We leave out 5% training data as validation set for parameter tuning and use early stoping strategy in training. For each dataset we run experiments with five different random seeds and report average results (the improvements in the paper are all significant so we omit confidence intervals for results).
4.1 COMPARATIVE EXPERIMENTS
There can be different configurations for support users and query users. To simulate real-world scenarios where new users are often with fewer historical ratings, we divide users into two sets: users with more than training ratings, denoted as Z7, and users with less than training ratings, denoted as Uz. We basically set 8 = 30 for Douban and three Movielens datasets, and ō 20 for Amazon- -Bokk.. Then we consider two situations: 1) we set Ud = U1 and Uz Uz i.c. we treat Ui support users and U/ query users; 2) we set in U22 U1. i.e., we treat U7 as both
7

---

Under review as a conference paper at ICLR 2021
1.16
TRE NANE OCMC
IKCT-N NOOLE
0.90
8 1130
0:30
1.11
Sparsity
(a)
(b)
(C)
(d)
Figure 4: (a) Overall RMSE w.r.t # sparsity ratio. (b) User specific RMSE w.r.t # user's training ratings.(c) Attention weights of query users (y-axis) on support users (x-axis). (d) Support uses'
accumulated attention weights w.r.t. # historical ratings.
support and query users". In the first case, we can use test performance on U2 to investigate model's expressiveness in inductive learning; in the second case, we can use test performance on U2 to study model's 's generalization on new unseen users during test. Other split ways like random split or using different 0's and their impact on model performance thoroughly discussed in Appendix EL
Comparison with transductive & inductive models We compare with three state-of ta-t---rrt trans- ductive methods PMF (Salakhutdmov & Madi 2007),, NNMF (Dziugaite & Roy, 2015). GCMC
(van den Berg et all 2012), three inductive feature-driven methods NIMC (Zhong et all 2018).
PinSAGE (Ying et ale 2018), BOMIC (Ledent et all 2020), and two recently proposed inductive methods without using features F-EAE (Hariorr ct all 2015) and IGMC (Zhang & Chen 2020).
For these competitors, we use all the training data for training. In Table we report test RMSEs for all the users, inductive learning on query users and generalization on new users in Douban, ML-1000k and ML-IM. As we can see, IRCF-NN gives very close RMSEs on all the users and query users to NNMF in three datasets while IRCF-GC performs nearly as well as GCMC. The empirical results prove that our inductive model possesses the same expressive power as corresponding transductive model. Compared with inductive methods, IRCF-GC achieves the best RMSEs for query users in three datasets. The results demonstrate the superior expressiveness of proposed inductive model comapred to other fcature-driven and local-graph- graph-bbshddddd based inductive models. Besides, for new unseen users, our model IRCF-GC gives state-of-the-ar performance, achieving significant improvement over the best competitor, quantitatively 3.2% Douban, 3.1% % on MI-100K and 4.1% on ML.
In Table D. we compare IRCF-NN and IRCF-GC with PMF, NNMF and GCMC in ML-10M and Amazon-Books. Since the two datasets have no feature, we cannot train feature-driven methods. Also, F-EAE and IGMC are both hard to scale to such large datasets. In Amazon-Booke we replace square loss in our objective as cross-entropy loss and harness AUC as metric to align with implicit feedbacks. We can see that our inductive models achieve very similar RMSEs in ML-10M as trans- ductive models, and even significantly outperform transductive models in Amazon- Books. In fact, Amazon dataset is a very sparse one with rating density 0.0012. One implication here is that our inductive model can provide better performance than transductive models for users with few ratings.
Comparison with cold-start recommendation models We also wonder if our inductive model can handle extreme cold-start users who have no historical rating. Note that cold-start users are different and more challenging compared to new (unseen) users. For new users, the model can still use observed historical ratings as input during test, though it cannot be trained on these ratings. In order to enable cold-start recommendation, we leverage attribute features in Moviclens- IM. We use the dataset provided by (Lee et all 2019), which contain attribute features and split warm-start and cold-start users, and follow its evaluation protocol: using the warm-start users' training ratings for training and cold-start users' test ratings for test. We compare with Wide&Deep network (Cheng al, 201G) GCMC (using feature vectors) and two -tate-of-the-art cold-start recommendation models AGNN (Qian et all 2019) and MeLU (Lee et all 2019). In Table B. we present test RMSES for all the models, and we can see that our model IRCF-HY gives the best results, achieving 2.6% improvement over the best competitor MeLU even on the difficult zero-shot recommendation task. The result shows that our inductive model is promising approach to handle new users with no historical behavior in real-world dynamic systems.
*In such case, we train the base MF model and inductive model over the same users in order.
8

---

Under review as a conference paper at ICLR 2021
4.2 FURTHER DISCUSSIONS
Sparse Data and Few-shot users A successful recommender system is supposed to handle data sparsity and few-shot users with few historical ratings. Here we construct several sparse datasets by using 50%, 20%, 10%, 5%, 1% and 0.1% training ratings in Moviclens-IN. and then compare the test RMSEs of query users in Fig. Hay Also, in Fig. Hb we compare the test RMSEs for users with different numbers of historical ratings when under 50% sparsity. As shown in Fig. Hag as the dataset becomes sparser,the RMSES of all the models suffer from drop, but the drop rate of our inductive models IGCF-NN and IGCF-GC is much smaller compared with transductive models NNMF and GCMC. In Fig. Hb we find that users with more historical ratings usually have better RMSE scores compared with few-shot users. By contrast, our inductive models IRCF-NN and IRCF-GC exhibits a more smooth decrease and even outperform other transductive methods NNMF and GCMC for users with very few ratings. In the extrame cases with less than five historical ratings, notably, IRCF-GC achieves 2.5% improvement on RMSE compared with the best transductive method GCMC.
Attention Weight Distribution Fig. Hc we visualize attention weights of IRCF-NN from query users to support users in Movielens- 10M. As we can see, there is an interesting phenomenon that some of support users are very 'important" and most query users give high attention weights on them, which indicates that the representations of these support users are informative and combination of them can provide powerful expressiveness for query users' preferences. In Fig. Hdj we further plot support users' accumuultedd attention weights w.r.. # historical ratings. We can that larger attention weights concentrate on support users with more historical ratings. Such observation gives an important hint for selecting optimal support users: important support users more likely exist in users with more observed ratings. In Appendix E we compare different split ways for support and query users and provide more discussions and results on this point.
Scalability Test We further investigate the scalability of IRCF-GC compared with two GNN-based counterparts IGMC and GCMC. We statistic the training time per epoch on Movielens- 1M using a GTX 1080Ti with 11G memory. Here we truncate the dataset and use different numbers of ratings for training. The results are shown in Fig. 3 (with log-scale axis). As we can see, when dataset size becomes large, the training times per epoch of three models all exhibit linear increase. IRCF spends approximately one more time than GCMC, while IGMC approximately ten times slower than IRCF. Nevertheless, while IRCF costs one more training time than GCMC, the latter cannot tackle new unseen users without retraining a model test stage.
5 CONCLUSIONS
In this paper, we propose a new inductive collaborative filtering framework that learns hidden rela- tional graphs among users to allow sufficient message passing in the latent space. The new model accomplishes inductively computation for user-specific representations without compromising on expressiveness and scalablity. Through extensive experiments, we show that the model can achieve state-of-the-ane performance on inductive matrix completion and outperforms transductive models for users with few training ratings. Interesting future directions include: 1) consider the selection for support users as a decision problem; 2) use meta-learning or transfer learning techniques over inductive framework; 3) exploit our framework in cross dooss- recommendation tasks.
The core idea of IRCF opens a new way for next generation of representation learning, i.e., one can consider a pretrained representation model for one of existing entities and generalize their rep- resentations (through some simple transformations) to efficiently compute inductive representations for others, enabling the model flexibly handle new coming entities an open world. We believe that this novel and effective framework can inspire more researches in broad areas of AI.
REFERENCES
Yukuo Cen, Jianwei Zhang, Xu Zou, Chang Zhou, Hongxia Yang. and Jie Tang, Controllable multi- interest framework for recommendation. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash (eds.), KDD, pp. 2942-2951. 2020.
Heng-Tze Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan

---

Under review as a conference paper at ICLR 2021
Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. Wide & deep learning for recommender systems. In DLRS. pp. 7-10, 2016.
Paolo Cremonesi, Yehuda Koren, and Roberto Turrin. Performance of recommender algorithms on top-n recommendation tasks. RecSys, pp. 39-46. ACM, 2010.
Gintare Karolina Dziugaite and Daniel M. Roy. Neural network matrix factorization. CoRR, abs/1511.064433 2015.
William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NeurlPS, pp. 1024-1034, 2017.
Jason S. Hartford, Devon R. Graham, Kevin Leyton-Browe and Siamak Ravanbakhsh. Deep models of interactions across sets. In ICML, pp. 1914-1923, 2018.
Prateek Jain and Inderjit S. Dhillon. Provable inductive matrix completion. CoRR, 2013.
Santosh Kabbur, Xia Ning, and George Karypis. FISM: factored item similarity models for top-n recommender systems. In KDD, pp. 659-667. ACM, 2013.
Elias Khalil, Hanjun Dai, Yuyu Zhang, Bistra Dilkina, and Le Song. Learning combinatorial optimization algorithms over graphs. In NeurlPS, pp. 6348-6358, 2017.
Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, 42(8):30-37, 2009.
Antoine Ledent, Rodrigo Alves, and Marius Kloft. Orthogonal inductive matrix completion. CoRR. abs/2004.01653, 2020.
Hoyeop Lee, Jinbae Im, Seongwon Jang, Hyunsouk Cho, and Sehee Chung. Melu: Meta-learned user preference estimator for cold-start recommendation. In KDD, pp. 1073-1082, 2019.
Dawen Liang, Rahul G. Krishnan, Matthew D. Hoffman, and Tony Jebara. Variational autoencoders for collaborative filtering. In Pierre-Antoine Champin, Fabien L.. Gandon, Mounia Lalmas, and Panagiotis G. Ipeirotis (eds.), WWW, pp. 689-698. ACM, 2018.
Jianxin Ma, Chang Zhou, Peng Cui, Hongxia Yang. and Wenwu Zhu. Learning disentangled repre- sentations for recommendation. In NeurlPS, pp. 5712-5723,, 2019.
Sahil Manchanda, Akash Mittal, Anuj Dhawan, Sourav Medya, Sayan Ranu, and Ambuj K. Singh. Learning heuristics over large graphs via deep reinforcement learning. CoRR, abs/1903.03333. 2019.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. Adaptive computation and machine learning. MIT Press, 2012.
Federico Monti, Michael M. Bronstein, and Xavier Bresson. Geometric matrix completion with recurrent multi-graph neural networks. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), NIPS, pp. 3697-3707, 2017.
Tieyun Qian, Yile Liang, and Qing Li. Solving cold start problem in recommendation with attribute graph neural networks. CoRR, abs/1912. 123998 2019.
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt- Thieidd,--eee...... BPR: bayesian personalized ranking from implicit feedback. In UAI, pp. 452-461, 2009.
Ruslan Salakhutdinov and Andriy Mnih. Probabilistic matrix factorization. John C. Platt, Daphne Koller, Yoram Singer, and Sam Roweis Nedr.I.. pp. 1257-1264, 2007.
Suvash Sedhain, Aditya Krishna Menon, Scott Sanner, and Lexing Xie. Autoree: Autoencoders meet collaborative filtering. In WWW (Companion Volume), pp. 111-112, ACM, 2015.
10

---

Under review as a conference paper at ICLR 2021
Ajit Paul Singh and Geoffrey J. Gordon. Relational learning via collective matrix factorization. In KDD, pp. 650-658, 2008.
Nathan Srebro, Noga Alon, and Tommi S. Jaakkola. Generalization error bounds for collaborative prediction with low-rank matrices. In NeurIPS, pp. 1321-1328, 2004.
Rianne van den Berg, Thomas N. Kipf, and Max Welling. Graph convolutional matrix completion. CoRR, abs/ 720.02233,,, 2017.
Miao Xu, Rong Jin, and Zhi-Hua Zhou. Speedup matrix completion with side information: Appli- cation to multi-label learning. NeurlPS, pp. 2301-2309, 2013.
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systemm In Yike Guo and Faisal Farooq (eds.), SIGKDD, pp. 974-983. ACM, 2018.
Muhan Zhang and Yixin Chen. Inductive matrix completion based on graph neural networks. In ICLR, 2020.
Ziwei Zhang, Peng Cui, and Wenwu Zhu. abs/1181.04222, 2018.
Deep learning on graphs: A survey,
CoRR,
Yin Zheng, Bangsheng Tang, Wenkui Ding, and Hanning Zhou. A neural autoregressive approach to collaborative filtering. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), ICML, vol- ume 48, pp. 764-773, 2016a.
Yin Zheng, Bangsheng Tang, Wenkui Ding, and Hanning Zhou. A neural autoregressive approach to collaborative filtering. In ICML., pp. 764-773, 2016b.
Kai Zhong, Zhao Song, Prateek Jain, and Inderjit S. Dhillon. Nonlinear inductive matrix completion based on one-layer neural networks. CoRR, abbs/1005.10 2018
Kai Zhong, Zhao Song, Prateek Jain, and Inderjit S. Dhillon. Provable non-linear inductive matrix completion. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Gameet (eds.), NIPS, pp. 1 14455-1 1445, 2019.

---

Under review as a conference paper at ICLR 2021
user ene-hot
user embeddings
user-item pair (u, rating vector
5
Fué
item miing
0
4
0
(a) general CF model
(b) local-graph-based inductive model
support user embeddlng
query user embedding
user rating vector
user rating vector
5
0
3
0
0
2
item embeddings
user
ON
embedding
(c) item-based model
model IRCF
Figure 5: Comparison with related works methodological level. (a) General collaborative fil- tering assumes user-specific one-hot embeddings for users and learn them collaboratively among all the users in one dataset. It disables inductive learning due such learnable one-hot embeddings. (b) docal-graph--aset inductive model (Zhang & Chen, 2020) extracts local subgraph structures within I-hop neighbors of each user-item pair (i.e., rated items of the user and users who rated the item) from bipartite graph of all the observed user-item ratings and use GNNs to encode such graph structures for rating prediction. Note that the model requires that the local subgraphs do not contain user and item indices, it assumes no user-specific embeddings.
(c) Item-based model leverages embeddings of user's historically rated items to compute user's embeddings via some pooling methods. The learnable parameters only lie in the item space. (d) Our model IRCF adopts item-based embedding to compute attention scores on different support users and aggregate one-hot embeddings of support users compute user-specific embeddings for query users, which maintains ability to produce user representations with enough expressiveness and achieves inductive learning.
A LINKS TO RELATED WORKS
We provide thorough discussion on the relationships and differences to related works. In Fig. B we present a illustration for comparison with different methods.
B PROOFS IN SECTION 3
B. PROOF OF THEOREM 1
Proof. The proof is trivial by construction. Assume the optimal P2 for (G) as P Since P1 given by (II) volumn-full-rant for any column vector p in P (u' E U2), there exists c such that
P = / Hence, C* -===(e [ is solution for (2) and gives Ds, (Rz, R2) < <c.
LI
B.2 PROOF OF THEOREM 2
With fixed a true rating matrix R2 be learned and probability distribution over (M2] x [N]. which is unknown to the learner, we consider the problem under the framework of standard PAC learning. We can treat the matrix R2 as funetion (u/, - Ta4. Let R, set of matrices in N. denotes the hypothesis class of this problem. Then the input to the learner is a sample of R2 denoted as
Proof.
T ((1 de, ruga ruga)) (14, Sz),
where S2 = {(L). &t}} E ([M,) x [N]) a set with size T2 containing indices of the ob- served entries in Ra and each (u in Sa is independently chosen according to the distribution
12

---

Under review as a conference paper at ICLR 2021
P. When using T training examples for the leamer, it minimizes the enror Ds, (Rz, R2) To E(w,ness (Tars, We are interested in the generalization error of the learner, which is defined as
D22,, R2) B(w () (rris Puri)]
The (empirical) Rademacher complexity of R. w.r.t. the sample T is defined
Rady (R)
T2 E
sup REER Am1
where of 1-1, is random variable with probability Pr(a, = 1) Pr(o, -1) 3. We assume is L-Lipschitz w.r.t the first argument and [4] is bounded by a constant. Then a general result for generalization bound of R is
Lemma 1. (Generalization bound (Mohri etll al (0T2)): : For a sample T with random choice of S2 = [[M2] x [ it holds that for any R2 E R and confidence parameter 0 < 8 1 Pr((((R2, R2) I Dz (R2, R2) + G) 1 8.
(9)
where,
In(1/5) T2
G = 2T. Rad(x) + +0
Based the lemma, need further estimate the Rademacher complexity in our model plete the proof. In our model, R2 C 1 Q and the entry fu' is given by fu's = p.qq Piq: (where Cy' is the u' -'h colunm vector of C). Define A as a set of matrices.
Mi
A = (A E [0, 12 M2 aw/l
Σ
aann = 13.
Then we have
T2
T2 Rady (R) = E,
sup CEC
T. o,C Piqi.
(10)
t=1
Σ Ri )
E
sup
(:: (1,ii = Piq.)
(11)
CEC
a/=l
Fick muy
M
< H Eg
sup AEA 12
Σ
or R1
(12)
till mu
M2
H Eg
max
Σ
Oefats
(13)
The last equation is due the fact that a. a probability distribution for choosing entries in R1
the 14 column of matrix R. In fact, we can treat the max inside the sum over all u' E U2 as a mapping k from u' [M2] to u E [M1]. Let = {x: [Ma] - [M,]] be the set of all mappings from [Mz] to [M and then the above formula can be written as
E2
Σ oetuie
(14)
uE[M
M2 sup
EE
(15)
KEK
w'=1 to =u!
T2
Eg
sup
Te OeTn(uad
(16)
KEK:
< B T2
2M, log M1.
(17)
13

---

Under review as a conference paper at ICLR 2021
The last inequality is according to the Massart Lemma. Hence, we have
2M log M,
Rady(R) & HB
(18)
T2
Incorporating (I) into H)), we will arrive at the result in this theorem.
C SPECIFICATIONS OF IRCF
In Section 3, we present general framework for inductive relational collaborative filtering (IRCF) without using features. In the following, we provide two different specifications for fo in IRCF with neural networks (IRCF-NN) and graph convolution networks (IRCF-GC) which are used in our experiments.
C.I NEURAL NETWORK AS BASED MODEL (IRCF-NN)
follow architecture NNMF (Dzuuaate Roy M K and use neural network for fg. Here we combine three-layer neural network and a shallow dot- dot-product operation. Concretely,
fe (Pu. q:) (P. q. nn nn([pu/qq Pa O q:l)) + ba + bi,
(19)
where nn is a three-layer neural network using tanh activation, @ denotes element-wise product and bns by are bias terms for user and item i, respectively.
C.2 GRAPH CONVOLUTION NETWORK AS BASED MODEL (IRCF-GC)
follow the architecture used GCMC (van den Berg et al, 2017) and adopt graph convolution network for fe (). Besides user-specific embedding for user and item-specific embedding for item i, we consider embeddings for user u's rated items and users who rated on item i.e., the one-hop
neighbors of user and item in user-item bipartite graph. Denote M₁ {ilrus * 0} user u's
rated items and M2 = {u|ra * 0} as users who rated on item i. We consider graph convolution to aggregate information from neighbors,
m. ReLU WWal ieN.
Σ Wyg:).
(20)
n. ReLU
w WyPa).
(21)
Then we define the output function
S (P,, qı (PuhweN; (q.)) = nn ((p.. O q lp. @ m.llnn @ q:llnn ② m..) + b, + b;, (22)
where nn' is three-layer neural network using ReLU activation.
D EXTENSIONS OF IRCF
IRCF can be extended to feature-based setting and flexibly deal with zero-shot learning, as shown in Section 4. Here, we provide details of feature-based IRCF (IRCF-HY) which indeed is hybrid model that considers both user features and one-hot user indices. Furthermore, we discuss in the views of transfer-learning and meta-lcarning that be incorporated with our framework future study.
D.1 HYBRID MODEL WITH FEATURES (IRCF-HY)
Assume a, denotes user u's feature vector, i.e., concatenation of all the features (often in- cluding binary, categorical and continuous variables) where categorical features can be denoted by one-hot multi-hot vectors. one has m user features in total, then a, can be
a [aw]
|a,2 |a.3
[amm]
14

---

Under review as a conference paper at ICLR 2021
Then consider user-sharing embedding function y:( which can embed each feature vector into a d-dimensional embedding vector:
Yu = [y (a Y2 a.2) (a.s) I| ... Il (aum)) )}
Similarly, for item feature b, = b b,2 b . [bo]. we have its embedding representation:
2; = |1 (b,1) )| Z2 (b,2 Zz (bss)]l [l,,, (b,n)l
Also, we assume user-specifie index embedding Pa and item- sem-sppccficc index embedding qi for user and item i, respectively, as is in Section 3. The prediction for user u's rating item can be
Aus go (Pu: Yu, 2;;
(23)
where go can be shallow neural network with parameters denoted by 0. To keep notation clean, and 2 = [z,, 22, ... znn Then for support users in Un with rating
denote Y = Y2, ,mm y
matrix Rj, we consider the optimization problem.
min Ds. (Rg By), P 2,0
(24)
based on which we get learned feature embedding functions Y, Z as well as transductive embedding matrices Pi, Q which we further use compute inductive embeddings for query users.
For query users, feature embeddings can be obtained by the learned Y and Z in (244), i.e., Yu' [Ya'l (aw//)lll-. ... Iyy wmm (aw'm)) where aw raw feature vector of user u'. Then have a relation inference model hw that consists of a multi-head attention function and use user feature as input dy Yu'. The inductive user-specific representation can given by P = (dw) (i.e., (G)
and (f), similar as the CF setting in Section 3. The rating of user u' on item i can be predicted by Pw's go (Pw' (ww.yws q: zz). Also, the optimization of the second stage is
min Ds. (R2, R2). w.00
(25)
D.2 ZERO-SHOT LEARNING
For zero-shot recommendation where test users have no historical rating, we have no information about users if without any side information. In such case, most CF models would fail for person- alized recommendation and degrade to a trivial one which outputs the same result (or the same distribution) all the users using the popularity of items. For IRCF, the set A., would be empty for users with no historical rating, in which situation we can randomly select a group of support users to construct A, used for computing attentive scores with support users. Another method to directly use average embeddings of all the support users as estimated embeddings for query users. In such case, the model degrades to ItemPop (using the numbers of users who rated the item for prediction).
On the other hand, if side information is available, our hybrid model IRCF-HY can leverage user features for computing inductive preference embeddings, which enables zero-shot recommendation. We apply this method to zero-shot recommendation on Movielens- 1M using features in Section and achieve superior RMSE.
D.3 TRANSFER LEARNING META-LEARNIH
Another extension of IRCF is to consider transfer learning cross-domain recommendation tasks (Singh & Gordon, 200X or when treating recommendation for different users as different tasks like all 201U). Transfer learning and meta learning have shown power learning generalizable models that can adapt to new tasks. In our framework, we can also take advantage of transfer learning (few-shot learning or zero-shot learning) or mete-learning algorithms to train our relation inference model hg. For example, if using model-agnosiie meta- meta-learning algorithm for the second- stage optimization, we can first compute one-step (or multi-step) gradient update independently for each user in batch and then average them as one global update for the model. The meta-learning can be applied over different groups of users or cross-domain datasets.
E DETAILS IN IMPLEMENTATIONS
We provide implementation details that presented in Section 4 in order for reproducibility.
15

---

Under review as a conference paper at ICLR 2021
Table Statistics of five datasets used in our experiments. Amazon-Booke dataset has implicit user feedbacks while other four datasets have explicit feedbacks. Explicit feedbacks mean user's rating values range within [1. 2, 3, 4, 5) and implicit feedbacks mean we only know whether user has rated on an item or not. f A dataset has both user/item features, only user or item features, no feature.
Dataset
# Users/Items
# Ratings
Density
Featuref
# Supp/Query Users
Douban
3,000/3,000
136,891
0.0152
User User/ltem
2,131/869
Movielens-100k
943/1,682
100,000
0.0630
671/272
Movielens-IN 1M
6,040/3,706
1,000,209
0.0447
User/Item
5, 14/926
Movielens- (M
69,878/10,677
10,000,054
0.0134
No
Mmazon-Booke
52,643/91,599
2,931,466
0.0012
No
49,058/3,585
E. DATASETS
The statistics of datasets used in experiments summarized in Table ! We the prepro- cessed versions of Douban and Movielens-1006 provided by (Mont et al, 2017).. For Movielens- IM and Moviclens-I0M B, we use the same 9:1 training/test split as previous works. The original ML-IM dataset has few user features and recent work (Lee all 2019) collects new dataset with more features for both users and items We use this version for our feature-based setting on ML- in experiments. The raw dataset for Beaazon-Booke is very large and sparse one and we filter out infrequent items and users with less than five ratings.
E.2 HYPER-PARAMETEE SETTINGS
We present details for hyper-parameter settings in different datasets. We use L 4 attention heads for our inductive relation inference model among all the datasets. For Douban and ML-100K, each attention head randomly samples 200 support users for computing attention weights. For ML-IM and ML-10M, we set sample size 500; for Amazon-Boode
we set it as 2000. We use Adam optimizer and learning rates are searched within [0.1, 1, 0.01, 0.001, 0.0001]. For the first-stage trans- ductive training, we consdier 1.2 regularization for user and item embeddings. The regularization weights are searched within [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0. 0.2]. The mini-batch sizes are searched within [64, 256, 512, 1024, 2048] to keep a proper balance between training efficiency and for architectures are used in three implementations.
performance. Also, different
IRCF-NN. For Douban and ML-100K, use embedding dimension d 16 and neural size 48 32 - 32 for fe. For ML- 1M, ML-10M and Amazon-Books we use = 32 and neural size 96 64 64 1 for fo-
IRCF-GC. For Douban and ML.-10K,, we use embedding dimension = 32 and neural size 128
32 32 for fo. For ML-1M, ML-10M and Amazon-Books, we use d 64 and neural size 256 64 64 for fa-
IRCF-HY. We use embedding size d = 32 for each feature in ML-1M well sper-specific and item-specific index embeddings. The neural size of go is set as 320 64 64 - 1.
E.3 DETAILS FOR COMPARATIVE METHODS
For each comparative model our experiments, we mainly rely on the designs and setups in their papers and, if necessary, tune its byper-parameten to obtain optimal results in each dataset. Here we provide some implementations details for MeLU (Lee et all 2019), AGNN (Qian et al), 2019), IMC (Zhong et al. 2019) and BOMIC (Ledent al. 2020).
"https:/grouplensscc http:/uuwwww.oorrrdddas////ooiee https///ggghibcoo//ooopp/p/////a http://mmeennggnosc...cda//aaaat
edu//dattaamazon/
16

---

Under review as a conference paper at ICLR 2021
MeLU. We use the souree codes provided by the paper The author provides two versions of the model, MeLU-1 and McLU-5, different from using one-step or steestep local updates for each global update. We report the best results of the two in our experiments.
AGNN. We build user-user (resp. item-item) graph according to cosine similarities of fea- tures and rating records of users (resp. items). In specific, we add an edge from to j when is among j's top 10 nearest nodes. In order for the consistency of feature dimension, we use a linear model transform the original one-hot or multi-hot representation of each fea- ture into vector with fixed dimension. When training the eVEE model, use Euclidean distance instead of cross-entropy between the original features and reconstructed features for calculating reconstruction loss.
IMC. We first use a two-layer neural network to transform the multi-hot user/item features into di-eeniiniionll vector representations. Then we calculate the dot-product of the the features of user-item pairs to make predictions.
BOMIC. We use the version BOMIC+ in our experiments. To be specific, the model incorporates user features, item features, item bias, user bias well as user/item index embeddings to predict the possible rating between users and items.
Evaluation Metries We provide details for our adopted evaluation metrics, In our experiments, we follow evaluation protocols commonly used in previous works in different settings. Three metrics used in our paper are follows.
AUC: This is measurement for consistency of recommendation list ranked by predicted scores and ground-truth clicking list with 1s before Os. It counts the average area under the curve of true-positive v.s. false-positive curve for one user's ranking list:
AUC. ΣiEro ΣjEr. 8(Buns > 0)
(26)
[I
where It (ilrus > 0} and I_ = Lilrag =0 denote the sets of clicked items and not clicked items by user u respectively. The indicator 8(fmi > Puj) returns when fua >> faj and 0 otherwise. AUC is used For implicit feedbacks. Since only have ground-truth positive examples (clicked items) for each user, we negatively sample five items as negative examples (non-clicked items) for each user-item rating in dataset.
RMSE: Root Mean Square Error measures the averaged L2 distance between predicted ratings and ground-truth ratings:
Σ (Pui ru))2
RMSE,
ie T.
(27)
It
MAE:Mean Absolute Error measures the averaged LI distance between predicted ratings and ground-truth ratings:
Σ Pi
rual
$EZ!!
MAE,
(28)
I
F DISCUSSIONS ON DIFFERENT CONFIGURATIONS FOR SUPPORT AND QUERY USERS
In our experiments in Section 4, we basically consider users with more than å training ratings as U1 and the remaining as Ug based on which construct support users and query users to study model's expressiveness in inductive learning and generalization on new unseen users. Here we provide further discussions two spliting ways and study the impact on model performance.
Threshold: we select users with more than training ratings as 01 and users with less than ò training ratings as U2.
17

---

Comparing threshold split and random split, we can find that when using the same ratio of support users and query users (i.e., the same column in Table 5) RMSEs on new users with threshold split are always better than those with random split. Such observation shows that support users with more Under review as a conference paper at ICLR 2021
Table RMSEs on all the users (ALL), query users (Query) and new users (New) of IRCF-NN in Movielens- IM using different configurations for support and query users. The results calibrate with IRCF-NN in Table II where we consider threshold split with 8 30 in ML-IM.
20
30
40
50
60
70
(RMSE)
0.8440
0.8437
0.8439
0.8440
0.8444
0.8451
Threshold
Query (RMSE)
0,9785
0.9525
0.9213
0.9166
0.9202
0.9160
New (RMSE)
0,9945
0.9912
0.9902
0.9883
0.9911
0.9929
0.97
0.85
0.75
0.68
0.62
0.57
(RMSE)
0.8446
0.8536
0.8587
0.8637
0.8669
0.8689
Random
Query (RMSE)
0,8863
0.8848
0.8760
0.8805
0.8824
0,8855
New (RMSE)
0.9901
0.9923
0.9956
0.1001
1.0198
1.0262
Random: set ratio y (0, 1) and randomly sample y x 100% of users in the dataset U1 The remaining users are grouped as U2.
We consider 5 = [20, 30, 40, 50, 60, 70] and [0.97. 0.85, 0.75, 0.68, 0.62, 0.57] (which exactly gives the same ratio of [ and [ as corresponding in threshold split in Movielens- IM dataset. For each spliting, we also consider two situations for support users and query users: 1) 24 = U and U2 U2; U1 1dz UT The results of IRCF-NN are presented in Table DI where report test RMSEs on all the users, query users and new users.
As we can see from Table 5L in threshold split, increases (we have fewer support users and more query users and they both have more training interactions average), test RMSEs for query users exhibit a decrease. The reason two-folds: 1) since support usres have more training ratings, the transductive model can learn better representations: 2) since query users have more training ratings, the inductive model would have better generalization ability. On the other hand, with different spliting thresholds, test RMSEs for new users remain in fixed level. The results demonstrate that our model performance on new unseen users not sensitive to different split thresholds. However, in random split, when y decreases (also we have fewer support users and more query users but their average training ratings stay unchanged). RMSEs for new users suffer from an obvious decrease. One possible reason is that when support users in the dataset are more likely to be ignored. (As is shown Fig. [[C) in Section 3, there exist some important support users that give high attention weights for query users.) such support users are missing, the performance would be affected due to insufficient expressive power of inductive model.
smaller ratio of support users in random split, important
training ratings would more effective for inductive learning and again accord with the results in Fig. Hdi which demonstrates important support users who give large attention weights for query users tend to exist users with many training ratings.
For example, using y 0.97 in random split will result in the same sizes of 27, and Und as using 5 20 in threshold split.
18


Reviews:
[{"review_id": "q_3yh7OE3zs", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed an inductive collaborative filtering method, called IRCF. The goal is to possess expressiveness (against feature-driven methods) as well as generalization (against one-hot encoding based methods). In IRCF, there are a matrix factorization model for support users and a relation model for query users. The former is trained with transductive learning to obtain support users embeddings and item embeddings. The relation model then generates query user embeddings as weighted sum of support user embeddings by examining relational graph between support and query users.\n\nPros:\n\n1. The paper is well-written and easy to follow. \n2. The experimental results are satisfying. \n3. The Theorem 1 and Theorem 2 reflect the tradeoff between capacity and generalization, which can guide the way of selecting support users. \n4. The idea of using a set of pretrained embeddings as bases may be generalized to other inductive tasks.\n\nCons/Questions:\n\n1. A detailed related work section is expected. There have been many works studying inductive recommendation problems w/ or w/o user features. As far as I known, lots of methods like FISM generate user embeddings by aggregating embeddings of historical items, which naturally support inductive learning. In this paper, the proposed method IRCF views query user embeddings as weighed sum of support user embeddings, but the weights are still based on aggregating embeddings of historical items (i.e., d_u' in Eq. 4). Moreover, the support users are analogous to a set of bases, and each user can be represented by a combination of the bases. Thus, it is hard to assess the novelty without a related work section.\n2. It is unclear on how to handle user bias terms b_u in Eq. 19 and Eq. 22 for query users or new users.\n3. It seems that you assume C is a conical combination coefficients in Eq.4. Why not to use the unnormalized scores in Eq. 4, which matches the Theorem 1 better?", "extra_scores": {}, "extra_text": {}}, {"review_id": "i3MMD7Jk74X", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:  \nThe work proposes a relational learning scheme that extends standard collaborative filtering approach and aims at improving recommendations quality for new users. The proposed method extracts relations from historical rating data within a preselected subset of support users and utilizes this knowledge to better represent newly introduced (query) users.\n\nReasons for score:  \nThe paper misses comparison with a large set of competitive techniques based on incremental learning approach and provides no justification for this omission. Evaluation methodology seem to have test data leak, which may be the major source of performance gains instead of the model itself.\n\nPros:  \nThe authors present an interesting view on relational learning problem within the collaborative filtering setting. Generating recommendations online in an instant fashion for both known and new users is indeed a very relevant problem of a great practical importance. The proposed relational learning-based modification of standard collaborative filtering schemes is described clearly and incorporates novel ideas. The authors also prove two theorems, which further support feasibility of their approach and provide some hints on the expected behavior. \n\nCons:  \nBoth in the abstract and in the text, the authors state that standard CF techniques are unable to deal with new users without the need to retrain the entire model. This statement is wrong. For example, in the case of warm start scenario with matrix factorization techniques one can easily update the model specifically for a newly introduced user by performing a few update steps using only ratings of this user. In the case of SGD this would be just a few \"half-steps\" of gradient descent with fixed matrix of item embeddings. This can be generalized to neural-networks based approaches. Similarly, for the ALS algorithm (e.g., [1]) it would require \"half-step\" of solving a linear system w.r.t. new user embedding (see eq. 4 in [1]), which can be performed efficiently and in fact is one of the standard approaches in many production systems.\nFurthermore, this can be even reduced to an analytical solution in the case of PureSVD approach [2]: it only requires learning item embeddings, and the user embeddings are simply represented as a weighted sum of the embeddings of items they interacted with (see eq. 6 in [2]). It's similar to the way the d_u variable is defined in this paper in eq. 4. Finally, a natural generalization of the latter representation would be an autoencoder, e.g. MultVAE [3] or RecVAE [4], which naturally resolves the warm-start scenario without the need for any modifications as there's no need for a separate user representation.\nTherefore, in order to make comparison complete I would suggest to include some incremental learning techniques as well as autoencoder solutions and clearly demonstrate how the proposed method compares to them in terms of recommendation quality, flexibility, and computational efficiency.\n\nThe second major point here is the evaluation methodology. In eq. 4, matrices W_q, W_k represent trainable parameters. So how are they trained? Equation 7 explicitly states that training is done on historical data from query users (matrix R_2) and no other data splits are present.  If that's the case, than there's a test data leak: historical rating data of query users is used to train parameters of the model and then the same users are used to evaluate the performance of the model. It doesn't correspond to the warm start scenario. The weights W_q, W_k must be fixed first (after they were trained) and then used to generate representations of users that were never shown to the model before with eq. 6. Hence, there should be at least two disjoint subsets of query users: one for validation and another one for actual testing of the model. Unfortunately, I couldn't find any hints on such a splitting neither in the main text, nor in the Appendix, which makes me believe there's no such splitting. Avoiding test data leaks is absolutely crucial for a fair comparison.\n\nMinor comments:  \nThe problem of generating recommendations is not the same as the problem of rating prediction/matrix completion. It's important to keep this distinction in mind. The standard task for recommender systems is generating an ordered list of relevant items. The quality of this cannot be measured with RMSE. In fact, there's a strong evidence that models that perform well in terms of RMSE metric may not be good at all in terms of more appropriate metrics like precision, recall, nDCG, MAP, MRR, etc. Please, consider adding more appropriate metrics into the work.\nAlso note that AUC is not the best choice for that matter as it makes no distinction between proper ranking of irrelevant items and proper ranking of relevant items. Considering that the majority of items in recommendations are typically irrelevant, high AUC scores may not reliably represent actual performance of algorithms in their ability to generate lists of relevant items.\n\nReferences:  \n[1] Hu, Yifan, Yehuda Koren, and Chris Volinsky. \"Collaborative filtering for implicit feedback datasets.\" In 2008 Eighth IEEE International Conference on Data Mining, pp. 263-272. Ieee, 2008.  \n[2] Cremonesi, Paolo, Yehuda Koren, and Roberto Turrin. \"Performance of recommender algorithms on top-n recommendation tasks.\" In Proceedings of the fourth ACM conference on Recommender systems, pp. 39-46. ACM, 2010.  \n[3] Liang, Dawen, Rahul G. Krishnan, Matthew D. Hoffman, and Tony Jebara. \"Variational autoencoders for collaborative filtering.\" In Proceedings of the 2018 World Wide Web Conference, pp. 689-698. 2018.  \n[4] Shenbin, Ilya, Anton Alekseev, Elena Tutubalina, Valentin Malykh, and Sergey I. Nikolenko. \"RecVAE: A New Variational Autoencoder for Top-N Recommendations with Implicit Feedback.\" In Proceedings of the 13th International Conference on Web Search and Data Mining, pp. 528-536. 2020.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "PH6GdhXhJ0P", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThis work proposed an inductive recommendation framework on user-item relation graphs. Such a framework relies on the user-item relations without the requirement of side-information and perceives certain flexibility in terms of the parametrization for user/item representations. The authors also provided theoretical analysis to highlight some mathematical insights out of this framework. The proposed method is evaluated on three real-world datasets and compared with several baselines.\n\nOverall I find the work was well-reasoned and executed in a relatively good shape, thus recommending acceptance.\n\n\n##########################################################################\n\nStrength:\n- Relevant topic to the ICRL community and could have potential impact in real-world applications\n- The proposed method is well reasoned and technically sound\n- Experiments are executed in a decent shape\n\n##########################################################################\n\nWeakness:\n- Motivations behind its technical contributions can be further sharpened; comparisons to previous related studies on the inductive graph learning domain can be further improved\n- Some gaps between the current experiment setup and real-world recommendation senarios\n\n##########################################################################\n\nDetailed Comments:\n\nI'll address the above potential weakness in details here.\n\nI personally find a bit difficult to digest the motivations of this work and how it differentiated from previous inductive graph learning work until diving into its detailed parametrizations. Fig. 1 and its descriptions are helpful in terms of illustrating the inductive setting, but not quite informative in terms of concrete contributions of this work conceptually.\nMy takeaway from the proposed framework is, the attentive pooling method falls into the aggregator family of inductive graph learning, despite that the aggregation and sampling scheme are performed on user side globally instead of on the user-item local neighborhoods. In this regard, it may also be helpful to highlight the (mathematical) difference between this work and existing inductive graph learning (e.g. pinSage) after eq.5/6.\n\nAlthough the experimentations are executed in a good shape, there are still some gaps between the current setup and real-world recommendation requirements. \n- The proposed method is largely evaluated on the rating prediction setting, AUC is reported on the amazon dataset but no Top-K ranking metrics are performed during the experiments. It is acceptable given these metrics are consistent with the optimization objective, however, the notable gap between pointwise prediction setting and the real-world online top-K ranking setting needs to be called out.\n- Another concern about the current evaluation protocol is, it enforces the temporal dynamics on the user side and assumes item representations remains the same - again it is consistent with the proposed method (i.e., Q remains the same) thus expected to favor it. The question is, whether these assumptions are consistent with real-world senarios. As far as I know, both movieLens and Amazon datasets have associated timestamps, what the real temporal dynamics here and what would be the warm/cold item/user distribution look like if splitting data chronologically?\n\t\n\nMinor Concerns:\n\t- Annotations in  Figure 4 can be further enlarged for visibility\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "IJedtHABwCD", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Quick summary\nThis work explores a popular problem, i.e., collaborative filtering, in an inductive setting, which is very important for real-world recommender systems. To address the challenges in the inductive settings, i.e., learning accurate representations for users who do not occur in the training data, the authors propose to construct a relational graph between users in the training data and new users based on a standard matrix factorization model and then use an attentive message passing framework to inductively compute user-specific representations. Besides, the authors prove the expressive and generalization capabilities of the proposed framework. Extensive experiments are conducted to demonstrate the effectiveness of the proposed framework both in transductive and inductive settings, as well as the scalability.\n\n### Clarity\nThe presentation of the paper is good.\n\n### Originality\nGenerally speaking, the inductive collaborative filtering is of limited novelty, while the technical solution is novel and solid, especially the part in constructing a relational graph between support users and query users.\n\n### Pros\n1. The technical solution is interesting and solid, with a clear presentation in the paper.\n2. The proofs of Theorem 1&2 are interesting, which theoretically shows the expressiveness and generalization abilities of the proposed model.\n3. The experiments are extensive, most of which are convincing.\n4. The presentation of the paper is good.\n\n### Cons\n1. The major concern is that the novelty of inductive collaborative filtering with GNN is limited since Zhang & Chen 2020 [1] proposed the IGMC framework, which has done a comprehensive study on the inductive CF problem. Though the authors point out the difference between the proposed IRCF and Zhang's work, they do not give adequate materials to support their arguments. For example, the mentioned disadvantage of IGMC is that *the subgraphs in IGMC are ignorant of user and items indices*, however, from the perspective of the author, this issue is not that important, and may not occur very frequently, and can be trivially addressed by incorporating the user and item indices into IGMC. It will be more convincing if the authors can give more supporting materials in the paper.\n2. For Theorem 1, the authors hold one argument that matrix factorization gives maximized capacity for learning personalized user preferences from historical rating patterns, however, it does not make sense to the reviewer. Can you provide any references or explain a bit more? Besides, what are the implications of Theorem 1 in helping us understanding the proposed IRCF ?\n3. For Theorem 2, the authors only discuss the influences of the size of $\\mathcal{U}_1$. How about other variables, e.g., $B, H, M_2$, etc. \n4. In the performance comparisons, the authors use RMSE in Table 1 and 2, while MAE in Table 3. This seems weird to the reviewers. Why do not you adopt the same metric, say either RMSE or MAE, since the experiments are actually the same type.\n\nGenerally speaking, the paper is of high quality. The idea is clear, and the technical solution is interesting and solid with a theoretical guarantee. Most experimental results are convincing. Besides, the writing of the paper is clear and easy to understand.\n\n------\n### Post rebuttal\nGreat thanks to the authors for the detailed replies. After reading them, I decided to keep my rating.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is somewhat borderline, though reviews mostly lean positive. Unfortunately after calibrating compared to other submissions, the work remains somewhat below the bar compared to higher-scoring papers.

The reviewers praise the topic, the method, and the experiments (although some of this praise is a little mixed or lukewarm). The most negative review raises several specific concerns about the evaluation methodology, as well as some concerns about data leaks etc. While serious, the authors rebuttal to these claims seems reasonably convincing. While the remaining issues appear not to be dealbreakers, there are nevertheless some lingering concerns which ultimately put the paper slightly below the bar.

The AC notes that their initial inclination was to accept this paper, though it was suggested that the score be lowered after calibration compared to other submissions, mainly due to doubt regarding these lingering issues.
Decision: Reject