Title: COLLABORATIVE FILTERING WITH A SYNTHETIC FEEDBACK LOOP Anonymous

Abstract: We propose a novel learning framework for recommendation systems, assisting collaborative filtering with a synthetic feedback loop. The proposed framework consists of a "recommender" and a "virtual user." The recommender is formulizd as a collaborative-filtering method, recommending items according to observed user behavior. The virtual user estimates rewards from the recommended items and generates the influence of the rewards on observed user behavior. The recommender connected with the virtual user constructs a closed loop, that recommends users with items and imitates the unobserved feedback of the users to the recommended items. The synthetic feedback is used to augment observed user behavior and improve recommendation results. Such a model can be interpreted as the inverse reinforcement learning, which can be learned effectively via rollout (simulation). Experimental results show that the proposed framework is able to boost the performance of existing collaborative filtering methods on multiple datasets.

1. INTRODUCTION:

Recommendation systems are important modules for abundant online applications, helping users explore items of potential interest. As one of the most effective approaches, collaborative filtering Sarwar et al. (2001) ; Koren & Bell (2015) ; He et al. (2017) and its deep neural networks based variants He et al. (2017) ; Wu et al. (2016) ; Liang et al. (2018) ; Li & She (2017) ; Yang et al. (2017) ; Wang et al. (2018) have been widely studied. These methods leverage patterns across similar users and items, predicting user preferences and demonstrating encouraging results in recommendation tasks Bennett & Lanning (2007) ; Hu et al. (2008) ; Schedl (2016) . Among these work, beside "useritem" pairs, side information, e.g., user reviews and scores on items, are involved and have achieved remarkable success Menon et al. (2011) ; Fang & Si (2011) . Such side information is a kind of user feedback to the recommended items, which is promising to improve the recommendation systems.

Unfortunately, both the user-item pairs and user feedback are extremely sparse compared with the search space of items. What is worse, when the recommendation systems are trained on static observations, the feedback is unavailable until it is deployed in real-world applications -in both training and validation phases, the target systems have no access to any feedback because no one has observed the recommended items. Therefore, the recommendation systems may suffer overfitting, and their performance may degrade accordingly, especially in the initial phase of deployment. Although real-world recommendation systems are usually updated in an online manner with the assist of increasing observed user behavior Rendle & Schmidt-Thieme (2008) ; Agarwal et al. (2010) ; He et al. (2016) , introducing a feedback mechanism during their training phases can potentially improve the efficiency of the initial systems. However, this is neglected by existing learning frameworks. Motivated by the above observations, we propose a novel framework that achieves collaborative filtering with a synthetic feedback loop (CF-SFL). As shown in Figure 1 , the proposed framework consists of a "recommender" and a "virtual user." The recommender is a collaborative filtering (CF) model, that predicts items from observed user behavior. The observed user behavior reflects intrinsic preferences of users, while the recommended items represent the potential user preferences estimated by the model. Regarding the fusion of the observed user behavior and the recommended items as inputs, the virtual user, which is the key of our model, imitates real-world scenarios and synthesizes user feedback. In particular, the virtual user contains a reward estimator and a feedback generator: the reward estimator estimates rewards based on the fused inputs (the compatible representation of the user observation and its recommended items), learned with a generative adversarial regularizer. The feedback generator provides feedback embeddings to augment the original user embeddings, conditioned on the estimated rewards as well as the fused inputs. Such a framework constructs a closed loop between the target CF model and the virtual user, synthesizing user feedback as side information to improve recommendation results. The proposed CF-SFL framework can be interpreted as inverse reinforcement learning (IRL) approach, in which the recommender learns to recommend user items (policy) with the estimated guidance (feedback) from the proposed virtual user. The proposed feedback loops can be understood as an effective rollout procedure for recommendation, jointly updating the recommender (policy) and the virtual user (the reward estimator and the feedback generator).

Eventually, even if side information (i.e., real-world user feedback) is unobservable, our algorithm is still applicable to synthesize feedback in both the training and inference phases. The proposed framework is general and compatible with most CF methods. Experimental results show that the performance of existing approaches can be remarkably improved within the proposed framework.

2. PROPOSED FRAMEWORK:

In this section, we first describe the problem we are interested in and give a detail description of each module that is included in the framework.

2.1 PROBLEM STATEMENT Suppose we have N users with M items in total, we denote the observed user-item matrix as X = [x i ] ∈ {0, 1} N ×M , where each vector

x i = [x ij ] ∈ R M , i = 1, .

.., N , represents observed user behaviors. x ij = 1 indicates the the j-th item is bought or reviewed via the i-th user and otherwise the j-th item can either be irrelevant to the i-th user or we have not enough knowledge about their relationship. The desired recommendation system aims to predict each user's preference, denoted as a i = [a ij ] ∈ R M , whose element a ij indicates the preference of the i-th user to the j-th item. Accordingly, the system recommends each user with the items having large a ij 's.

Ideally, for each user, x i just contains partial (actually, very sparse) information about user preference and a practical recommendation system works dynamically with a closed loop -users often generate feedback to the recommended items while the recommendation system considers these feedback to revise recommended items in the future. Therefore, we can formulate the whole recommendation process as

EQUATION

where π(•) represents the target recommender while f (•) represents the coupled feedback mechanism of the system. v i ∈ R d is the embedding of user feedback to historical recommended items. At each time t, the recommender predicts preferred items according to observed user behaviors and previous feedback, and the user generates feedback to the recommender. Note that (1) is different from existing sequential recommendation models Mishra et al. (2015) ; Wang et al. (2016) because those methods ignore the feedback loop as well, which just updates recommender π according to observed sequential observations, i.e., x t i for different time t's.1 Unfortunately, the feedback information is often unavailable in the training and inference phases. Accordingly, most of existing collaborative filtering-based recommendation methods ignore the feedback loop in the system, learning the target system purely from static observation user-item matrix X Liang et al. (2018) ; Li & She (2017) . Although in some scenarios side information like user reviewers is associated with the observation matrix, the methods using such information often treat it as a kind of static knowledge rather than dynamic feedback. They mainly focus on fitting the groundtruth recommended items with the recommender π(•) given fixed x i 's and v i 's, while ignore the imitation of the whole recommendation-feedback loop in (1). Without the feedback mechanism f (•), π(•) tends to over-fit the observed user behavior and static side information, which may degrade in practical dynamical scene.

To overcome the problems mentioned above, we propose a collaborative filtering framework with a synthetic feedback loop (CF-SFL), which explains the whole recommendation process from a viewpoint of reinforcement learning. As shown in Figure 1 , besides traditional recommendation module the proposed framework further introduces a virtual user, which imitates the recommendationfeedback loop even if the real-world feedback are unobservable.

2.2. RECOMMENDER:

In our framework, the recommender implements the function π(•) in (1), which takes the observed user behavior x i and his/her previous feedback embedding v t i as input and recommends items accordingly. In principle, the recommender π(•) can be defined with high flexibility, which can be arbitrary collaborative filtering methods that predicting items from user representations, such as WMF Hu et al. (2008) , CDAE Wu et al. (2016) , VAE Liang et al. (2018) etc. In this work, we formulate the recommender from the viewpoint of reinforcement learning.

In particular, the recommendation-feedback loop generates a sequence of interactions between each user and the recommender, i.e., (s t i , a t i ) T t=1 for i = 1, ..., N . Here,

s t i = [x i ; v t i ]

is the representation of user i at time t, which is a sample in the state space S describing user preference. a t i indicates the recommended items provided by the recommender, which is a sample in the action space A of the recommender. Accordingly, we can model the recommendation-feedback loop as a Markov Decision Process (MDP) M = S, A, P, R , where P : S ×A×S → R is the transition probability of user preference and R : S × A → R is the reward function used to evaluate recommended items. The recommender π(•) works as a policy parametrized by θ, i.e., π θ (a|s), which corresponds to the distribution of items conditioned on user preference. The target recommender should be an optimal policy that maximizes the expected reward:

J(π θ ) = T t=1 E π θ [R(s t , a t )]

, where R(s t , a t ) means the reward over the state-action pair (s t , a t ). For the i-th user, given s t i , the recommender selects potentially-preferred items via a t i = arg max a∈A π θ (a|s t i ).

(2)

Note that different from traditional reinforcement learning tasks, in which both S and A are available while P and R are with limited accessibility, our recommender just receives partial information of state -it does not observe users' feedback embedding v i . In other words, to optimize the recommender, we need to build a reward model and a feedback generator jointly, which motivates us to introduce a virtual user into the framework.

2.3. VIRTUAL USER:

The virtual user module aims to implement the feedback function f (•) in (1), which not only models the reward of the items provided by the recommender but also generates feedback to complete the representations of state. Accordingly, the virtual user contains the following two modules:

Reward Estimator The reward estimator parametrizes the function of reward, which takes the current prediction a t i and user preference s t i as input and evaluate their compatibility. In this work, we implement the estimator with parameter φ, which is defined as

EQUATION

In this work, we use the static part of the state s t i , i.e., the observed user behaviors x i as input. h(•, •) is the fusion function which merges x i and a t i into a real value vector (the fused input is shown in Figure 5 and described in Appendix). g(•) is the single value regression function that translates the fused input into a single reward value. The sigmoid function is used to restrict the regression value between 0 and 1.

Feedback Generator:

The feedback generator connects the reward estimator with the recommender via generating a feedback embedding, i.e.,

EQUATION

h(x i , a 0 i )

< l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > t=0

Reward Estimator

Recommender:

Feedback Generator t=1

Reward Estimator

Feedback Generator a 1 i < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > h(x i , a 1 i )

< l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > R(s 1 i , a 1 i )

< l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t >

t=T-1

Reward Estimator:



Feedback Generator t=T Reward Estimator Recommender v T 1 i < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > v T i < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > v 0 i < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t >

R(s T 1 i , a T 1 i )

< l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > R(s 0 i , a 0 i )

< l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t >

x i < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t >

x i < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t >

x i < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t >

x i < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > v 1 i < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > a T 1 i < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > a T i < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > a 0 i < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > h(x i , a T 1 i )

< l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > h(x i , a T i )

< l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > L sup < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > L R < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > where ψ represents the parameters of the generator. Specifically, the a parametric function F ψ (•, •) considers the fused input and the estimated reward and returns a feedback embedding v i t ∈ R d to the recommender. R(s t i , a i t ) indicates the compatibility between the recommended items and user preferences, and h(x i , a i t ), which is a vector rather than a scalar like reward, further enriches the information of the reward to generate feedback embeddings. Consequently, the recommender receives the informative feedback as a complementary component of the static observation x i to make an improved recommendation via (2).

3. LEARNING ALGORITHM:



3.1. LEARNING TASK:

Based on the proposed framework, we need to jointly learn the policy corresponding to the recommender π θ , the reward estimator R φ and the feedback generator F ψ . Suppose that we have a set of labeled samples D = {x i , ãi }, where x i ∈ R M is the historical behaviors of user i derived from the user-item matrix X and ãi is the ground truth of the recommended item for the user based on his/her behavior x i . We formulate the learning task as the following min-max optimization problem.

EQUATION

where

L(π θ , R φ , F ψ ) = i Lsup(ai, ãi; π θ , F ψ ) supervised loss -Ea∼π θ [log(R φ (s, a))] -Eã ∼D [1 -log(R φ (s, ã))]

Collaboration with adversarial regularizer .

(6)

In particular, the first term L sup in (6) can be any supervised loss based on labeled data D, e.g., the evidence lower bound (ELBO) proposed in VAEs Liang et al. (2018) (and used in our work). This term ensures the recommender to fit the groundtruth labeled data. The second term considers the following two types of interactions among these three modules:

• The collaboration between the recommender policy π θ and the feedback generator F ψ towards a better predictive recommender. • The adversarial game between the recommender policy π θ and the reward estimator R φ .

Accordingly, given current reward model, we update the recommender policy π θ and the feedback generator F ψ to maximize the expected reward derived from the generated user preference s and the recommended item a. On the contrary, given the recommended policy and the feedback generator, we improve the reward estimator R φ by sharpening its criterion -the updated reward estimator maximizes the expected reward derived from the generated user preference and the ground truth of item while minimize the expected reward based on the recommended item. Therefore, we solve (5) via alternating optimization. The updating of π θ and F ψ is achieved by minimizing

EQUATION

And the updating of R φ is achieved by maximizing

EQUATION

Both these two updating steps can be solved effectively via stochastic gradient descent.

3.2. UNROLLING FOR LEARNING AND INFERENCE:

Because the proposed framework contains a closed loop among learnable modules, during training we unroll the loop and let the recommender interact with the virtual user in T steps. Specifically, at the initial stage, the recommender takes the observed user behaviour x i and an all-zero initial feedback embedding v 0 i , to make recommendations. At each step t, the recommender predicts the items a t i given x i and v t i to the virtual user, and receives the feedback embedding v t+1 i . The loss is defined according to the output of the last step, i.e., a T and v T , and the modules are updated accordingly. After the model is learned, in the testing phase we need to infer the recommended item in the same manner, unrolling the feedback loop and deriving a T as the final recommended item. The details of unrolling process are illustrated in Figure 2 , and the detailed scheme of our learning algorithm is shown in Algorithm 1 in appendix.

4. CF-SFL AS INVERSE REINFORCEMENT LEARNING:

Our CF-SFL framework automatically discovers informative user feedback as side information and gradually improve the training for the recommender. Theoretically, it closely connects with Inverse Reinforcement Learning (IRL). Specifically, we jointly learn the reward function R(•, •) and the policy (recommender) π(•, •) from the expert trajectories D E (the observed labeled data). D E typically consists of state-action pairs generated from an expert policy π E with the corresponding environment dynamics ρ E . And the goal of the IRL is to recover the optimal reward function R * (•, •) as well as the corresponding recommender π * . Formally, the IRL is defined as:

EQUATION

Intuitively, the objective enforces the expert policy π E to induce higher rewards (the max part), than all other policies. This objective is sub-optimal if the expert trajectories are noisy, i.e., the expert is not perfect and its trajectories are not optimal. This will render the learned policy always performs worse than the expert one. Besides, the illed-defined IRL objective often induces multiple solutions due to flexible solution space, i.e., one can assign an arbitrary reward to trajectories not from expert, as long as these trajectories yields lower rewards than the expert trajectories. To alleviate these issues, some constraints are placed into the objective functions, e.g., a convex reward functional, ψ : R S×A → R, which usually works as a regularizer.

{R * , π * } = arg max

EQUATION

To imitate the expert policy and provide better generalization, we adopt the adversarial regularizer Ho & Ermon (2016) , which defines ψ with the following form:

ψ(R) E π E [q(R(s, a))] if R(s, a) ≥ 0 +∞ otherwise ,

where q(x) = xlog(1e -x ). This regularizer places low penalty on reward functions R that assign an amount of positive value to expert state-action pairs; however, if R assigns low value (close to zero, which is the lower bound) to the expert, then the regularizer will heavily penalize R. With induced adversarial regularizer, we obtain a new imitation learning algorithm for recommender:

min θ ψ * (ρ π -ρ π E ) -λH(π θ ) (12)

Intuitively, we want to find a saddle point (R φ , π θ ) of the expression:

EQUATION

where R(s, a) ∈ (0, 1). Note equation 11 is derived from the objective of traditional IRL. However, distinct from the traditional approach, we propose a feedback generator to provide feedbacks to the recommender. In terms of the reward estimator, it tends to assign lower rewards to the predicted results by the recommender π θ and higher rewards for the expert policy π E , which aims to discriminate π θ from π E :

EQUATION

Similar to standard IRL, we update the generator to maximize the expected reward with respect to log R(s, a), moving towards expert-like regions of user-item space. In practice, we incorporate feedback embedding to update the user preferences, and the objective of the recommender is:

EQUATION

where 2018) uses variational inference to scale up the algorithm for large-scale dataset and has shown significant success in recommender systems with a usage of multinormial likelihood. Our CF-SFL is a general framework that can adapt to these models seamlessly.

v t i = F ψ (h(x i , a t i ), R(s t i , a t i )).

5. RELATED WORK:

RL in Collaborative Filtering.

For RL-based methods, contextual multi-armed bandits are firstly utilized to model the interactive nature of recommender systems. Thompson Sampling (TS) Chapelle & Li (2011); Kveton et al. (2015) ; Zhang et al. (2017) and Upper Confident Bound (UCB) Li et al. (2010) are used to balance the trade-off between exploration and exploitation. Zhao et al. (2013) combine matrix factorization with bandits to include latent vectors of items and users for better exploration. The MDP-Based CF model can be viewed as a partial observable MDP (POMDP) with partial observation of user preferences Sunehag et al. (2015) . Value function approximation and policy based optimization can be employed to solve the MDP. Zheng et al. (2018) and Taghipour & Kardan (2008) modeled web page recommendation as a Q-Learning problem and learned to make recommendations from web usage data. Sunehag et al. (2015) introduced agents that successfully address sequential decision problems. Zhao et al. (2018) propose a novel pagewise recommendation framework based on deep reinforcement learning. In this paper, we consider the recommending procedure as sequential interactions between virtual users and recommender; and leverage feedbacks from virtual users to improve the recommendation. Recently, Chen et al. (2019) proposed an off-policy corrections technique, and successfully applied it in real-world applications. 2011). To directly compare with existing work, we employed the same pre-processing procedure as Liang et al. (2018) . A summary statistics of these datasets are provided in Table 1 . 

6. EXPERIMENTS:



Methods:



ML-20M:

Netflix MSD R@20 R@50 NDCG@100 R@20 R@50 NDCG@100 R@20 R@50 NDCG@100 SLIM 0. Evaluation Metrics We employ Recall@r2 together with NDCG@r3 as the evaluation metric for recommendation, which measures the similarity between the recommended items and the ground truth. Recall@r considers top-r recommended items equally, while NDCG@r ranks the top-r items and emphasizes the importance of the items that are with high ranks. Set-up For our CF-SFL framework, the architectures of its recommender, reward estimator and feedback generator are shown in Table 2 .

To represent the user preference, we normalize x i and v t i (t > 0) independently and concatenate the two into a single vector. To learn the model, we pre-train the recommender (150 epochs for ML-20M and 75 epochs for Netflix and MSD) and optimize the entire framework (50 epochs for ML-20M and 25 epochs for the other two). 2 regularization with a penalty term 0.01 is applied to the recommender, and Adam optimizer Kingma & Ba (2014) with batch in size of 500 is employed. The framework is built with Tensorflow and will be publicly available upon publication. Baselines To demonstrate the superiority of our framework, we consider multiple state-of-theart approaches as baselines, which can be categorized into two types: (i) Linear based models: SLIM Ning & Karypis (2011) and WMF Hu et al. (2008) . (ii) Deep neural network based models: CDAE Wu et al. (2016) , VAE Liang et al. (2018) and aWAE Zhong & Zhang (2018) . It should be noted that our CF-SFL is a generalized framework, which is compatible with all these approaches. In particular, as shown in Table 2 , we implement our recommender as the VAE-based model Liang et al. (2018) for a fair comparison. In the following experiments, we will show that besides such a setting the recommender can be implemented by other existing models as well. Performance Analysis All the evaluation metrics are averaged across all the test sets. (i) Quantitative results: we test various methods and report their results in Table 3 . With the proposed CF-SFL framework, we significantly improve the performance of the baselines on all the evaluation metrics. These experimental results demonstrate the power of the proposed CF-SFL framework, which provides informative feedback as the side information. Particularly, we observed that the performance between the base model (VAE * ) is similar to that of its variation with the reward estimator (VAE † ). It implies that simply learning a feedback from the reward estimator via back-propagation 3 , the trajectory of VAE † in Figure 3 , is similar to that of the base VAEs (VAE * ). In contrast, the trajectories of our CF-SFL methods are more smooth and able to converge to a better local minimum. This phenomenon further verifies that our CF-SFL learns informative user feedback with better stability. (d) With an increasing of time steps T in a particular range (T ≤ 8 for ML-20M), CF-SFL achieves faster and better performance. One possible explanation is the learning with our unrolled structure -parameters are shared across different time-steps, and a more accurate gradient is found towards the local minimum. (e) We find ML-20M and MSD are more sensitive to the choice of T when compared with Netflix. Therefore, the choice of T should adjust to different datasets.

(iii) CF-SFL with dynamic time steps: As shown in Figure 2 , the learning of CF-SFL involves a recurrent structure with T times steps. We investigate the choice of T and report its influence on the performance of our method. Specifically, the NDCG@100 with different T 's is shown in Figure 4 . Within 6 time steps, CF-SFL consistently boots the performance on all the three datasets. Even with a larger time steps, the results remain stable. Additionally, the inference time of CF-SFL is linear on time steps T . To achieve a trade-off between performance and efficiency, in our experiments we set T to 8 for ML-20M and Netflix and 6 for MSD. et al. (2018) . We find that our CF-SFL is capable of generalizing to most of the exisiting collaborative filtering approaches and boosts their performance accordingly. The gains achieved by our CF-SFL may vary depending on the choice of recommender.

7. CONCLUSION:

We propose a CF-SLF framework to simulate user feedback. It constructs a virtual user to provide informative side information as user feedback. Mathematically we formulate the framework as an IRL problem and learn the optimal policy by feeding back the action and reward. Specially, a recurrent architecture was built to unrolled the framework for efficient learning. Empirically we improve the performance of state-of-the-art collaborative filtering method with a non-trivial margin. Our framework serves as a practical solution making IRL feasible over large-scale collaborative filtering. And it will be interesting to investigate the framework in other applications, such as sequential recomender systems etc.

:

When the static observation xi in (1) is replaced with sequential observation x t i , (1) is naturally extended to a sequential recommendation system with a feedback loop. In this work, we focus on the case with static observations and train a recommender system accordingly.

https://en.wikipedia.org/wiki/Precision_and_recall

https://en.wikipedia.org/wiki/Discounted_cumulative_gain

Figures and Tables:

Figure 2: Unrolling the recurrent CF-SFL framework into an iterative learning process with multiple time steps T.

Figure 3: Performance (NDCG@100) boosting on the validation sets.

Figure 4: The blue curve summarizes NDCG@100 and red curves report the computational cost for model inference in each epoch. In each sub-figure, we vary the time steps from 0 to 12 (T=0 is the base recommender). is inefficient. Compared with such a naive strategy, the proposed CF-SFL provides more informative feedback to the recommender, and thus, is able to improve recommendation results more effectively. (ii) Learning Comparison: In Figure 3, we show the training trajectory of the baselines (VAE, VAE+reward estimator) and the CF-SFL with multiple time steps. There are several interesting findings. (a) The performance of the base VAE doesn't improve after the pre-training steps, e.g., 75 epochs for Netflix. In comparison, the proposed CF-SFL framework can further improve the performance once the whole model is triggered. (b) The CF-SFL yields fast convergence once the whole framework is activated. (c) Coincide with results in Table3, the trajectory of VAE † in Figure3, is similar to that of the base VAEs (VAE * ). In contrast, the trajectories of our CF-SFL methods are more smooth and able to converge to a better local minimum. This phenomenon further verifies that our CF-SFL learns informative user feedback with better stability. (d) With an increasing of time steps T in a particular range (T ≤ 8 for ML-20M), CF-SFL achieves faster and better performance. One possible explanation is the learning with our unrolled structure -parameters are shared across different time-steps, and a more accurate gradient is found towards the local minimum. (e) We find ML-20M and MSD are more sensitive to the choice of T when compared with Netflix. Therefore, the choice of T should adjust to different datasets. (iii) CF-SFL with dynamic time steps: As shown in Figure2, the learning of CF-SFL involves a recurrent structure with T times steps. We investigate the choice of T and report its influence on the performance of our method. Specifically, the NDCG@100 with different T 's is shown in Figure4. Within 6 time steps, CF-SFL consistently boots the performance on all the three datasets. Even with a larger time steps, the results remain stable. Additionally, the inference time of CF-SFL is linear on time steps T . To achieve a trade-off between performance and efficiency, in our experiments we set T to 8 for ML-20M and Netflix and 6 for MSD.

Collaborative Filtering. Collaborative filtering (CF) can be roughly categorized into two groups: CF with implicit feedbackBayer et al. (2017);Hu et al. (2008) and those with explicit feedbackKoren (2008);Liu et al. (2010). In implicit CF, user-item interactions are binary in natural (i.e., 1 if clicked and 0 otherwise) as oppose to explicit CF where item ratings (e.g., 1-5 stars) are typically the subject of interests. Implicit CF has been widely studied, examples including factorization of useritem interactionsHe et al. (2016);Koren (2008);Liu et al. (2016);Rendle (2010);Rennie & Srebro (2005) and ranking based approachRendle et al. (2009). And our CF-SFL is a new framework for implicit CF.Currently, neural network based models have achieved state-of-the-art performance for various recommender systemsCheng et al. (2016);He et al. (2018; 2017);Zhang et al. (2018);Liang et al. (2018). Among these methods, NCFHe et al. (2017) casts the well-established matrix factorization algorithm into an entire neural framework, combing the shallow inner-product based learner with a series of stacked nonlinear transformations. This method outperforms various of traditional baselines and has motivated many following works such as NFMHe et al. (2017), Deep FMGuo et al. (2017) and Wide and DeepCheng et al. (2016). Recently, deep generative has achieved remarkable success. VAEsLiang et al. (

Table content: <table/>

Basic information of the considered datasets.

Table content: <table><tr><td/><td colspan="2">ML-20M Netflix</td><td>MSD</td></tr><tr><td># of users</td><td colspan="3">136,677 463,435 571,355</td></tr><tr><td># of items</td><td>20,108</td><td colspan="2">17,769 41,140</td></tr><tr><td># of interactions</td><td>10.0M</td><td colspan="2">56.9M 33.6M</td></tr><tr><td># of held-out-users</td><td>10.0K</td><td>40.0K</td><td>50.0K</td></tr><tr><td>% of sparsity</td><td>0.36%</td><td colspan="2">0.69% 0.14%</td></tr></table>

Performance comparison between our CF-SFL framework and various of baselines. VAE * is the results based on our own runs and VAE † is the VAE model with our reward estimator.

Table content: <table/>

Architecture of our CF-SFL framework.

Table content: <table><tr><td colspan="3">Recommender Reward Estimator Feedback Generator</td></tr><tr><td>Input R M</td><td>Input R 64</td><td>Input R 65</td></tr><tr><td>M × 600, tanh</td><td>64 × 128, ReLU</td><td>64 × 128, ReLU</td></tr><tr><td>600 × 200 (x2)</td><td>128 × 128, ReLU</td><td/></tr><tr><td>Sample R 200</td><td/><td>128 × 128, ReLU</td></tr><tr><td colspan="2">200 × 600, tanh 128 × 128, ReLU</td><td/></tr><tr><td colspan="2">600 × M softmax 128 × 1, sigmoid</td><td>128 × 128</td></tr></table>

Performance of our CF-SFL with various of recommenders are reported. We study the usefulness of our CF-SFL on different recommenders and present the results in Table4. Specifically, two types of recommenders are being considered: the linear approaches like WARPWeston et al. (2011) andMF Hu et al.  (2008), and deep learning methods, e.g.,DAE Liang et al. (2018)  and the variation of VAE in Liang

Table content: <table><tr><td>Generalization Study As afore-</td><td/><td/><td/></tr><tr><td>mentioned, our CF-SFL is a general-</td><td/><td/><td/></tr><tr><td>ized framework which is compatible with many existing collaborative fil-tering approaches.</td><td colspan="4">Recommender w/o CF-SFL w CF-SFL Gain (10 -3 ) WARP 0.31228 0.33987 +27.59 MF 0.41587 0.41902 +3.15</td></tr><tr><td/><td>DAE</td><td>0.42056</td><td>0.42307</td><td>+2.51</td></tr><tr><td/><td>VAE</td><td>0.42546</td><td>0.43472</td><td>+9.26</td></tr><tr><td/><td colspan="2">VAE-(Gaussian) 0.42019</td><td>0.42751</td><td>+7.32</td></tr><tr><td/><td>VAE-(β = 0)</td><td>0.42027</td><td>0.42539</td><td>+5.02</td></tr><tr><td/><td>VAE-Linear</td><td>0.41563</td><td>0.41597</td><td>+0.34</td></tr></table>

Peer review (iclr2020):
The paper proposes to learn a *virtual user* while learning a *recommender* model, to improve the performance of the recommender system. The *virtual user* is used to produce the *reward* signal for training the recommendation system (which is trained using RL). Jointly learning the recommender and the *virtual user* provides a synthetic feedback loop that helps to improve the performance of the recommendation system. The paper formulates the training of the *virtual user* as an inverse RL problem and uses adversarial regularizer. The paper proposes an interesting idea but more experiments (and explanation) is needed to bring out the usefulness of the work. In general, the writing needs to be polished as well. =============== Following are the items that need to be improved: ## Significance of the results * The results in Tables 3 and 4 provide only marginal improvements over the baseline. These improvements do not appear to be statistically significant. It would help if the authors comment on why these results appear significant and also provide variance values/curves for the results. ## Role of the feedback general F * Is there any separate loss for training F or is it always trained along with pi? * Is the feedback capturing some sort of *memory* or *past preferences/behaviors* of the user? If yes, wouldn*t using a recurrent recommender also capture these effects without needing the feedback model F? Note that I am not criticizing the choice of F. I am trying to understand the role played by F (in addition to the recommender pi). * If there is no separate loss for F, I wonder how would the performance change if the F network was to be removed and the reward value was to be fed into a recurrent recommended. (The paper seems to have considered a special case where a non-recurrent recommender is used with the reward value). The reason I am stressing on this is that one of the key distinctions of the authors* work is the use of feedback generator and it would be useful to quantify the benefits on this modification. ## Others * How is the static representation, x, computed? From eq 16 (appendix), it appears that x is a binary vector that captures what items have been purchased/reviewed. Is that correct? If yes, wouldn*t x have an enormous dimensionality? * The recommendation system is operating in a sequential decision-making setup. The formulation of R and F do no consider any sequential information. For example, let us say that the recommender recommends the items a1, a2 and a3 in 3 timesteps. The reward at time 3 is a function of x and a3 and the information of a1 and a2 is not used. * The loss function has many components and I want to make sure I understand what gradients flow where. So please correct me if I missed something; * supervised learning loss (from the real data) trains pi and F (equation 6). * pi and F collaborate to get a high reward from R (since we do not have the ground truth corresponding to R). No gradient flows to R. * Adversarial game between pi and R - which is used to update R. * I understand that the loss is defined according to the output of the last step but do the gradients flow through all the intermediate steps? * The training/inference procedure seems to be doing something strange. Let us assume that T = 5. So the 5 items are recommended to the *virtual user* and only the 5th item is recommended to the real user. Now the recommendation of the 5th item depends on the first 4 items that the real user has not seen. * In equation 15, what does the subscript F stand for? * What algorithm is used to train the policy pi? ================= The following are the items that should be corrected in the future iterations of the paper. Please note that these things did not impact the score. * It seems that the irrelevance of an item (for a user) is treated the same way as missing information about the relevance of the item. Could the authors discuss this more in the paper? * Is there any reason why this approach is only used with CF methods? * Some writing choices seem to make the problem statement more complex than it is. For example, the authors discuss how their work is different than traditional RL instead of simply stating that their work is set up as a POMDP. * Articles are missing (or mixed) up at many places.