Collaborative Filtering With A Synthetic Feedback Loop

1 INTRODUCTION . Recommendation systems are important modules for abundant online applications , helping users explore items of potential interest . As one of the most effective approaches , collaborative filtering Sarwar et al . ( 2001 ) ; Koren & Bell ( 2015 ) ; He et al . ( 2017 ) and its deep neural networks based variants He et al . ( 2017 ) ; Wu et al . ( 2016 ) ; Liang et al . ( 2018 ) ; Li & She ( 2017 ) ; Yang et al . ( 2017 ) ; Wang et al . ( 2018 ) have been widely studied . These methods leverage patterns across similar users and items , predicting user preferences and demonstrating encouraging results in recommendation tasks Bennett & Lanning ( 2007 ) ; Hu et al . ( 2008 ) ; Schedl ( 2016 ) . Among these work , beside “ useritem ” pairs , side information , e.g. , user reviews and scores on items , are involved and have achieved remarkable success Menon et al . ( 2011 ) ; Fang & Si ( 2011 ) . Such side information is a kind of user feedback to the recommended items , which is promising to improve the recommendation systems . Unfortunately , both the user-item pairs and user feedback are extremely sparse compared with the search space of items . What is worse , when the recommendation systems are trained on static observations , the feedback is unavailable until it is deployed in real-world applications — in both training and validation phases , the target systems have no access to any feedback because no one has observed the recommended items . Therefore , the recommendation systems may suffer overfitting , and their performance may degrade accordingly , especially in the initial phase of deployment . Although real-world recommendation systems are usually updated in an online manner with the assist of increasing observed user behavior Rendle & Schmidt-Thieme ( 2008 ) ; Agarwal et al . ( 2010 ) ; He et al . ( 2016 ) , introducing a feedback mechanism during their training phases can potentially improve the efficiency of the initial systems . However , this is neglected by existing learning frameworks . Motivated by the above observations , we propose a novel framework that achieves collaborative filtering with a synthetic feedback loop ( CF-SFL ) . As shown in Figure 1 , the proposed framework consists of a “ recommender ” and a “ virtual user. ” The recommender is a collaborative filtering ( CF ) model , that predicts items from observed user behavior . The observed user behavior reflects intrinsic preferences of users , while the recommended items represent the potential user preferences estimated by the model . Regarding the fusion of the observed user behavior and the recommended items as inputs , the virtual user , which is the key of our model , imitates real-world scenarios and synthesizes user feedback . In particular , the virtual user contains a reward estimator and a feedback generator : the reward estimator estimates rewards based on the fused inputs ( the compatible representation of the user observation and its recommended items ) , learned with a generative adversarial regularizer . The feedback generator provides feedback embeddings to augment the original user embeddings , conditioned on the estimated rewards as well as the fused inputs . Such a framework constructs a closed loop between the target CF model and the virtual user , synthesizing user feedback as side information to improve recommendation results . The proposed CF-SFL framework can be interpreted as inverse reinforcement learning ( IRL ) approach , in which the recommender learns to recommend user items ( policy ) with the estimated guidance ( feedback ) from the proposed virtual user . The proposed feedback loops can be understood as an effective rollout procedure for recommendation , jointly updating the recommender ( policy ) and the virtual user ( the reward estimator and the feedback generator ) . Eventually , even if side information ( i.e. , real-world user feedback ) is unobservable , our algorithm is still applicable to synthesize feedback in both the training and inference phases . The proposed framework is general and compatible with most CF methods . Experimental results show that the performance of existing approaches can be remarkably improved within the proposed framework . 2 PROPOSED FRAMEWORK . In this section , we first describe the problem we are interested in and give a detail description of each module that is included in the framework . 2.1 PROBLEM STATEMENT . Suppose we have N users with M items in total , we denote the observed user-item matrix as X = [ xi ] ∈ { 0 , 1 } N×M , where each vector xi = [ xij ] ∈ RM , i = 1 , ... , N , represents observed user behaviors . xij = 1 indicates the the j-th item is bought or reviewed via the i-th user and otherwise the j-th item can either be irrelevant to the i-th user or we have not enough knowledge about their relationship . The desired recommendation system aims to predict each user ’ s preference , denoted as ai = [ aij ] ∈ RM , whose element aij indicates the preference of the i-th user to the j-th item . Accordingly , the system recommends each user with the items having large aij ’ s . Ideally , for each user , xi just contains partial ( actually , very sparse ) information about user preference and a practical recommendation system works dynamically with a closed loop — users often generate feedback to the recommended items while the recommendation system considers these feedback to revise recommended items in the future . Therefore , we can formulate the whole recommendation process as ati = π ( xi , v t i ) , v t+1 i = f ( xi , a t i ) , for i = 1 , ... , N , ( 1 ) where π ( · ) represents the target recommender while f ( · ) represents the coupled feedback mechanism of the system . vi ∈ Rd is the embedding of user feedback to historical recommended items . At each time t , the recommender predicts preferred items according to observed user behaviors and previous feedback , and the user generates feedback to the recommender . Note that ( 1 ) is different from existing sequential recommendation models Mishra et al . ( 2015 ) ; Wang et al . ( 2016 ) because those methods ignore the feedback loop as well , which just updates recommender π according to observed sequential observations , i.e. , xti for different time t ’ s . 1 Unfortunately , the feedback information is often unavailable in the training and inference phases . Accordingly , most of existing collaborative filtering-based recommendation methods ignore the feedback loop in the system , learning the target system purely from static observation user-item matrix X Liang et al . ( 2018 ) ; Li & She ( 2017 ) . Although in some scenarios side information like user reviewers is associated with the observation matrix , the methods using such information often treat it as a kind of static knowledge rather than dynamic feedback . They mainly focus on fitting the 1When the static observation xi in ( 1 ) is replaced with sequential observation xti , ( 1 ) is naturally extended to a sequential recommendation system with a feedback loop . In this work , we focus on the case with static observations and train a recommender system accordingly . groundtruth recommended items with the recommender π ( · ) given fixed xi ’ s and vi ’ s , while ignore the imitation of the whole recommendation-feedback loop in ( 1 ) . Without the feedback mechanism f ( · ) , π ( · ) tends to over-fit the observed user behavior and static side information , which may degrade in practical dynamical scene . To overcome the problems mentioned above , we propose a collaborative filtering framework with a synthetic feedback loop ( CF-SFL ) , which explains the whole recommendation process from a viewpoint of reinforcement learning . As shown in Figure 1 , besides traditional recommendation module the proposed framework further introduces a virtual user , which imitates the recommendationfeedback loop even if the real-world feedback are unobservable . 2.2 RECOMMENDER . In our framework , the recommender implements the function π ( · ) in ( 1 ) , which takes the observed user behavior xi and his/her previous feedback embedding vti as input and recommends items accordingly . In principle , the recommender π ( · ) can be defined with high flexibility , which can be arbitrary collaborative filtering methods that predicting items from user representations , such as WMF Hu et al . ( 2008 ) , CDAE Wu et al . ( 2016 ) , VAE Liang et al . ( 2018 ) etc . In this work , we formulate the recommender from the viewpoint of reinforcement learning . In particular , the recommendation-feedback loop generates a sequence of interactions between each user and the recommender , i.e. , ( sti , a t i ) T t=1 for i = 1 , ... , N . Here , s t i = [ xi ; v t i ] is the representation of user i at time t , which is a sample in the state space S describing user preference . ati indicates the recommended items provided by the recommender , which is a sample in the action space A of the recommender . Accordingly , we can model the recommendation-feedback loop as a Markov Decision Process ( MDP ) M = 〈S , A , P , R〉 , where P : S×A×S 7→ R is the transition probability of user preference and R : S ×A 7→ R is the reward function used to evaluate recommended items . The recommender π ( · ) works as a policy parametrized by θ , i.e. , πθ ( a|s ) , which corresponds to the distribution of items conditioned on user preference . The target recommender should be an optimal policy that maximizes the expected reward : J ( πθ ) = ∑T t=1 Eπθ [ R ( st , at ) ] , where R ( st , at ) means the reward over the state-action pair ( st , at ) . For the i-th user , given sti , the recommender selects potentially-preferred items via ati = arg maxa∈A πθ ( a|sti ) . ( 2 ) Note that different from traditional reinforcement learning tasks , in which both S andA are available while P and R are with limited accessibility , our recommender just receives partial information of state — it does not observe users ’ feedback embedding vi . In other words , to optimize the recommender , we need to build a reward model and a feedback generator jointly , which motivates us to introduce a virtual user into the framework . 2.3 VIRTUAL USER . The virtual user module aims to implement the feedback function f ( · ) in ( 1 ) , which not only models the reward of the items provided by the recommender but also generates feedback to complete the representations of state . Accordingly , the virtual user contains the following two modules : Reward Estimator The reward estimator parametrizes the function of reward , which takes the current prediction ati and user preference s t i as input and evaluate their compatibility . In this work , we implement the estimator with parameter φ , which is defined as Rφ ( s t i , a t i ) = sigmoid ( g ( h ( xi , a t i ) ) ) . ( 3 ) In this work , we use the static part of the state sti , i.e. , the observed user behaviors xi as input . h ( · , · ) is the fusion function which merges xi and ati into a real value vector ( the fused input is shown in Figure 5 and described in Appendix ) . g ( · ) is the single value regression function that translates the fused input into a single reward value . The sigmoid function is used to restrict the regression value between 0 and 1 . Feedback Generator The feedback generator connects the reward estimator with the recommender via generating a feedback embedding , i.e. , vit+1 = Fψ ( h ( xi , a t i ) , R ( s t i , a t i ) ) , ( 4 ) where ψ represents the parameters of the generator . Specifically , the a parametric function Fψ ( · , · ) considers the fused input and the estimated reward and returns a feedback embedding vit ∈ Rd to the recommender . R ( sti , a i t ) indicates the compatibility between the recommended items and user preferences , and h ( xi , ait ) , which is a vector rather than a scalar like reward , further enriches the information of the reward to generate feedback embeddings . Consequently , the recommender receives the informative feedback as a complementary component of the static observation xi to make an improved recommendation via ( 2 ) .

Summary: The paper proposes to learn a "virtual user" while learning a "recommender" model, to improve the performance of the recommender system. The "virtual user" is used to produce the "reward" signal for training the recommendation system (which is trained using RL). Jointly learning the recommender and the "virtual user" provides a synthetic feedback loop that helps to improve the performance of the recommendation system. The paper formulates the training of the "virtual user" as an inverse RL problem and uses adversarial regularizer.