Inductive Collaborative Filtering via Relation Graph Learning

1 INTRODUCTION . As information explosion has become one major factor affecting human life in the decade , recommender systems , which can filter useful information and contents of user ’ s potential interests , play an increasingly indispensable part in day-to-day activities . Recommendation problems can be generally formalized as matrix completion ( MC ) where one has a user-item rating matrix whose entries , which stand for interactions of users with items ( ratings or click behaviors ) , are partially observed . The goal of MC is to predict missing entries ( unobserved or future potential interactions ) in the matrix based on the observed ones . Modern recommender systems need to meet two important requirements in order for desirable effectiveness and practical utility . First of all , recommendation models should have enough expressiveness to capture diverse user interests and preferences so that the systems can accomplish personalized recommendation . Existing methods based on collaborative filtering ( CF ) or , interchangeably , matrix factorization ( MF ) have shown great power in this problem by factorizing the rating matrix into two classes of latent factors ( i.e. , embeddings ) for users and items respectively , and further leverage dot-product of two factors to predict potential ratings ( Koren et al. , 2009 ; Rendle et al. , 2009 ; Srebro et al. , 2004 ; Zheng et al. , 2016b ) . Equivalently , for each user , the methods consider a one-hot user index as input , assume a user-specific embedding function ( which maps a user index to a latent factor ) , and use the learnable latent factor to represent user ’ s preferences in a low-dimensional space . One can select proper dimension size to control balance between capacity and generalization . Recent works extend MF with complex architectures , like multi-layer perceptrons ( Dziugaite & Roy , 2015 ) , recurrent units ( Monti et al. , 2017 ) , autoregressive models ( Zheng et al. , 2016a ) , graph neural networks ( van den Berg et al. , 2017 ) , etc. , and achieve state-of-the-art results on most benchmarks . The second requirement stems from a key observation from real-world scenarios : recommender systems often interact with a dynamic open world where new users , who are not exposed to models during training , may appear in test stage . This requires that models trained on one group of users manage to adapt to unseen users . However , the above-mentioned CF models would fail in this situation since the user-specific embeddings are parametrized for specific users and need to be learned collaboratively with all other users in transductive setting . One brute-force way is to retrain the whole model with an augmented rating matrix , but extra time cost would be unacceptable for online systems . There are quite a few studies that propose inductive matrix completion models using user features ( Jain & Dhillon , 2013 ; Xu et al. , 2013 ; Cheng et al. , 2016 ; Ying et al. , 2018 ; Zhong et al. , 2018 ) . Their different thinking paradigm is to target a user-sharing mapping from user features to user representations , instead of from one-hot user indices used by CF models . Since the feature space is shared among users , such methods are able to adapt a model trained on existing users to unseen users . Nevertheless , feature-driven models often suffer from limited expressiveness with low-quality features that have weak correlation with target labels . For example , users with the same age and occupation ( commonly used features ) may have distinct ratings on movies and music . Unfortunately , high-quality features that can unveil user interests for personalized recommendation are often hard to collect due to increasingly concerned privacy issues . A following question arises : Can we have a recommendation model that guarantees enough expressiveness for personalized recommendation and enables inductive learning ? In fact , to simultaneously meet the two requirements is a non-trivial challenge when high-quality user features are unavailable . First , to achieve either of them , one needs to compromise on the other . In fact , the one-hot user indices ( together with learnable user-specific embeddings ) give a maximized capacity for learning distinct user preferences from historical rating patterns . To make inductive learning possible , one needs to construct a shared input feature space among users out of the rating matrix , as an alternative to one-hot user indices . However , the new constructed features have relatively insufficient expressive power . Second , the computation based on new feature space often bring extra costs for time and space , which limits model ’ s scalability to large-scale datasets . In this paper , we propose an inductive collaborative filtering model ( IRCF ) 1 as a general CF framework that achieves inductive learning for matrix completion and meanwhile guarantees enough expressiveness and scalability . As shown in Fig . 1 , we consider a base transductive matrix factorization model trained on one group of users ( called support users ) and a relation inference model that aims to estimate their relations to another group of users ( called query users ) w.r.t . historical rating patterns . The ( multiple ) estimated relational graphs enable attentively message passing from users to users in the latent space and compute user-specific representations in an inductive way . The output user representations can be used to compute product with item representations to predict ratings in the matrix , as is done by CF models . Compared with other methods , one key advantage of IRCF is the capability for inductively computing user-specific representations without using features . Besides , our method possesses the following merits . 1 ) Expressiveness : A general version of our model can minimize reconstruction loss to the same level as matrix factorization under a mild condition . Also , we qualitatively show its superior expressiveness than feature-driven and local-graphbased inductive models that may fail in some typical cases . Empirically , IRCF provides very close performance to transductive CF models when given sufficient training ratings . 2 ) Generalization : IRCF manages to achieve state-of-the-art results on new ( unseen ) users compared with inductive models . Also , IRCF gives much better accuracy than transductive models when training data becomes sparse and outperforms other competitors in extreme cold-start recommendation . 3 ) Scal- 1The codes will be released . ability : Our model enables mini-batch training and efficient inference . In experiments , IRCF is averagely ten times faster than local-graph-based inductive model . 4 ) Flexibility : As a general CF framework , IRCF is flexible to incorporate with various architectures ( e.g . MLP-based , GNN-based , autoregressive , etc . ) as a base transductive model as well as deal with implicit user feedbacks . 2 BACKGROUND AND RELATED WORKS . In this section , we present some background of the problem and discuss relationships to related works . We consider a general matrix completion ( MC ) problem which deals with a user-item rating matrix R = { rui } M×N where M and N are the numbers of users and items , respectively . For implicit feedback , rui is a binary entry which denotes whether user u rated ( or clicked on , reviewed , liked , purchased , etc . ) item i or not . For explicit feedback , rui records rating value of user u on item i . The entries of R are partially observed and the goal is to estimate the missing values in the matrix2 . In the following , we introduce related works and highlight their differences to our paper . In Fig . 5 , we provide an illustration for comparison with these methods . General Collaborative Filtering . Existing methods for MC are generally based on collaborative filtering ( CF ) or , interchangeably , matrix factorization ( MF ) where user u ( resp . item i ) corresponds to a d-dimensional latent factor ( i.e. , one-hot embedding ) pu ( resp . qi ) .Then one has a prediction model r̂ui = f ( pu , qi ) where f can be basically specified as simple dot product or some complex architectures , like neural networks , graph neural networks , etc . One advantage of CF models is that the user-specific embedding pu ( as learnable parameters ) can provide enough expressive power for learning diverse personal preferences from user historical behaviors and decent generalization ability through collaborative learning with all the users and items . Furthermore , the user embeddings possess rich profile information as representation of user preferences and can benefit various downstream tasks , like target advertisement , user-controllable recommendation ( Ma et al. , 2019 ; Cen et al. , 2020 ) , influence maximization ( Khalil et al. , 2017 ; Manchanda et al. , 2019 ) , friend recommendation , etc . However , such user-specific embedding limits the model in transductive learning and when it comes to new users during test , one has to retrain the model ( often associated with embeddings for both new and existing users as well as items ) with new augmented rating matrix . Admittedly , one can consider local updates for embeddings of new users with fixed item embeddings learned from existing users . Nevertheless , such operation makes model learning for users independent from each other , which is prone for over-fitting compared with collaborative learning in CF models . Besides , it requires ‘ incremental ’ learning for each new user , while IRCF can deliver on-the-fly inference in online systems . Feature-driven Recommendation . The CF models do not require any side information other than the rating matrix , but can not be trained inductively due to the learnable user-specific embedding pu . To address the issue , one can leverage side information such as attribute features to achieve inductive learning . Define user features ( like age , occupation , etc . ) as au and item features ( like movie genre , director , etc . ) as bi . The feature-driven model targets a prediction model r̂ui = g ( au , bi ) . Since the space of au is shared among users , a model trained on one group of users can adapt to other users without retraining . However , feature-driven models often provide limited performance since the shared feature space is not expressive enough compared to one-hot embedding space . Another issue is that high-quality features are hard to collect in practice . We note that our model does not require user features for inductive collaborative filtering . Inductive Matrix Completion . There are a few existing works that attempt to handle inductive matrix completion using only user-item rating matrix . ( Hartford et al. , 2018 ) ( F-EAE ) puts forward an exchangeable matrix layer that takes a whole rating matrix as input and inductively outputs prediction for missing ratings . However , the scalability of F-EAE is limited since it requires the whole rating matrix as input for training and inference for users , while IRCF enables mini-batch training and efficient inference . Besides , ( Zhang & Chen , 2020 ) ( IGMC ) proposes to use local subgraphs of user-item pairs in a bipartite graph of rating information as input features and further adopt graph neural networks as representation tool to encode subgraph structures for rating prediction . The model achieves inductive learning via replacing users ’ one-hot index embeddings by shared input features ( i.e. , index-free local subgraph structures ) . Differently , IRCF maintains the ability to give 2Also , in some situations , one targets a ranking list of items for each user as top-N recommendation . In this paper , we focus on predicting missing values in the matrix and leave top-N recommendation for future works . user-specific embeddings , which represent users ’ preferences and interests and can be used for downstream tasks ( like target advertisement , user-controllable recommendation , influence maximization , etc . ) while F-EAE and IGMC can merely output a prediction score without such user representation . Moreover , the expressiveness of IGMC is limited since the local subgraph structures can be indistinguishable for users with distinct behaviors ( see Section 3.2 for more discussions ) , while IRCF has equivalent expressiveness as original CF models . Item-based CF Models . Some previous works use item embeddings as representation for users . ( Cremonesi et al. , 2010 ; Kabbur et al. , 2013 ) proposes to use a combination of items rated by users to compute user embeddings and frees the model from learning parametric one-hot embeddings for users . Furthermore , there are quite a few auto-encoder architectures for recommendation problem , leveraging user ’ s rating vector ( ratings on all the items ) as observed input , estimate user embedding ( as latent variables ) based on that , and output prediction for missing values in the rating vector ( Sedhain et al. , 2015 ; Liang et al. , 2018 ) . With item embeddings and user ’ s rating history , these methods achieve inductive learning for users and can adapt to new users during test . On methodological level , IRCF has the following differences : 1 ) IRCF learns to use weighted combination of users ’ embeddings to compute embeddings for new users , and such combination weights possess interpretability for underlying social influence or user proximity ; 2 ) IRCF considers both users ’ and items ’ one-hot embeddings in general CF models that maintain better capacity than item-based CF models that only considers learnable parameters in item embedding space .

Summary: This work explores a popular problem, i.e., collaborative filtering, in an inductive setting, which is very important for real-world recommender systems. To address the challenges in the inductive settings, i.e., learning accurate representations for users who do not occur in the training data, the authors propose to construct a relational graph between users in the training data and new users based on a standard matrix factorization model and then use an attentive message passing framework to inductively compute user-specific representations. Besides, the authors prove the expressive and generalization capabilities of the proposed framework. Extensive experiments are conducted to demonstrate the effectiveness of the proposed framework both in transductive and inductive settings, as well as the scalability.