Collaborative Filtering With A Synthetic Feedback Loop
Authors: Wenlin Wang
Hongteng Xu
Ruiyi Zhang
Wenqi Wang
Lawrence Carin
Venue:  (2020)

Abstract: We propose a novel learning framework for recommendation systems, assisting collaborative filtering with a synthetic feedback loop. The proposed framework consists of a ``recommender'' and a ``virtual user.'' The recommender is formulizd as a collaborative-filtering method, recommending items according to observed user behavior.  The virtual user estimates rewards from the recommended items and generates the influence of the rewards on observed user behavior.  The recommender connected with the virtual user constructs a closed loop, that recommends users with items and imitates the unobserved feedback of the users to the recommended items. The synthetic feedback is used to augment observed user behavior and improve recommendation results.  Such a model can be interpreted as the inverse reinforcement learning, which can be learned effectively via rollout (simulation). Experimental results show that the proposed framework is able to boost the performance of existing collaborative filtering methods on multiple datasets. 

Under review as a conference paper at ICLR 2020
COLLABORATIVE FILTERING WITH A SYNTHETIC
FEEDBACK LOOP
Anonymous authors Paper under double-blind review
ABSTRACT
We propose novel learning framework for recommendation systems, assisting collaborative filtering with synthetic feedback loop. The proposed framework consists of a "recommender" and "virtual user." 'hhe recommender is formulizd as collaborative-flttrinng method, recommending items according to observed user behavior. The virtual user estimates rewards from the recommended items and generates the influence of the rewards on observed user behavior. The recom- mender connected with the virtual user constructs a closed loop, that recommends users with items and imitates the unobserved feedback of the users to the recom- mended items. The synthetic feedback is used to augment observed user behavior and improve recommendation results. Such a model can be interpreted as the inverse reinforcement learning, which can be learned effectively via rollout (sim- ulation). Experimental results show that the proposed framework is able to boost the performance of existing collaborative filtering methods on multiple datasets.
INTRODUCTION
Recommendation systems important modules for abundant online applications, helping users explore items of potential interest. As one of the most effective approaches, collaborative filter- 2001): Koren & Bell (2015); He al. (2017) and its deep neural networks based 2017); Wu et al. Liang et al. (2018); Li & She|(2017) Yang et al. (2017); Wang et al. (2018) have been widely studied. These methods leverage patterns across similar users and items, predicting user preferences and demonstrating encouraging results in recommendation tasks|Benned & Lanning (2007): Hu al. (2008); Schedl (2016). Among these work, beside "user- item" pairs, side information, e.g., user reviews and scores on items, are involved and have achieved remarkable success Menon et al. (2011); I): Fang & Si (20IT). Such side information is a kind of user feedback to the recommended items, which is promising to improve the recommendation systems.
ing Sarwar al.
variants He et al.
Unfortunately, both the user-item pairs and user feedback are extremely sparse compared with the search space of items. What is worse, when the recommendation systems trained static obser- vations, the feedback is unavailable until it is deployed in real-world applications - in both training to any feedback because
and validation phases, the target systems have served the recommended items. Therefore, the recommendation systems may suffer overfitting, and their performance may degrade accordingly, especially in the initial phase of deployment. Although real-world recommendation systems are usually updated in an online manner with the assist of in- creasing observed user behavior Rendle & Schmidt- Thiemee (2008) ; Agarwal al. (2010); He al. (2016), introducing a feedback mechanism during their training phases can potentially improve the efficiency of the initial systems. However, this is neglected by existing learning frameworks.
has ob-
Motivated by the above observations, we propose novel framework that achieves collaborative filtering with a synthetic feedback loop (CF-SFI.). As shown in Figure [ the proposed framework consists of "recommender" and "virtual user." The recommender is collaborative filtering (CF) model, that predicts items from observed user behavior. The observed user behavior reflects in- trinsic preferences of users, while the recommended items represent the potential user preferences estimated by the model. Regarding the fusion of the observed user behavior and the recommended items inputs, the virtual user, which is the key of our model, imitates real-world scenarios and synthesizes user feedback. particular, the virtual user contains a reward estimator and a feed- back generator: the reward estimator estimates rewards based on the fused inputs (the compatible

---

Under review as a conference paper at ICLR 2020
representation of the user observation and its recommended items), learned with generative adver- sarial regularizer. The feedback generator provides feedback embeddings to augment the original user embeddings, conditioned on the estimated rewards as well as the fused inputs. Such a frame- work constructs a closed loop between the target CF model and the virtual user, synthesizing user feedback as side information improve recommendation results.
The proposed CF-SFL framework can be interpreted as inverse reinforcement learning (IRL) approach, in which the recommender learns to recommend user items (policy) with the estimated guidance (feed- back) from the proposed virtual user. The proposed feedback loops can be understood as an effective rollout procedure for recommendation, jointly up- dating the recommender (policy) and the virtual user (the reward estimator and the feedback generator). Eventually, even side information (i.e., real-world user feedback) is unobservable, our algorithm is still applicable to synthesize feedback both the train- ing and inference phases. The proposed framework is general and compatible with most CF methods. of existing approaches can be remarkably improved within the proposed framework.
Alccommended
Virtual User
Fosed
Reward Estimator
Ocseved Uver Butaeiirr
Roward
Racommender
Foodbaok Emshardin
Feedback Generator
Figure 1: Illustration of our proposed CF-SFL framework for collaborative filtering.
Experimental results show that the performance
2 PROPOSED FRAMEWORK
In this section, we first describe the problem we are interested in and give a detail description of each module that is included in the framework.
2.1 PROBLEM STATEMENT
Suppose we have N users with M items in total, we denote the observed user-item matrix as X = [x.] E {0, 1] N where each vector æ; [may]] RM, i = I,....N N. represents observed user behaviors. zij = indicates the the j-th item is bought or reviewed via the i-th user and otherwise the j-th item can either be irrelevant to the i-th user relationship. The desired recommendation system aims to predict each user's preference, denoted as a, = [ay]] E RMM, whose element diy indicates the preference of the i-th user the j-th item. Accordingly, the system recommends each user with the items having large 0:y's 's.s.
have not enough knowledge about their
Ideally, for each user, at; just contains partial (actually, very sparse) information about user prefer- ence and practical recommendation system works dynamically with closed loop users often generate feedback the recommended items while the recommendation system considers these feedback to revise recommended items in the future. Therefore, we can formulate the whole recom- mendation process as
a = π ( vf), of = s(x,, at), for i 1,..., N,
where (-) represents the target recommender while f(-) represents the coupled feedback mecha- nism of the system. 0, E Rd is the embedding of user feedback to historical recommended items. At each time f, the recommender predicts preferred items according to observed user behaviors and previous feedback, and the user generates feedback to the recommender. Note that (T) different from existing sequential recommendation models Mishra et al. (2015): Wang et al. (2016 because those methods ignore the feedback loop as well, which just updates recommender π according to observed sequential observations, i.e., æς for different time t's
Unfortunately, the feedback information is often unavailable in the training and inference phases. Accordingly, most of existing collaborative filtering-based based recommendation methods ignore the feedback loop in the system, learning the target system purely from static observation user-item matrix X Liang et al. (2:18): Li & She (2017). Although in some scenarios side information like user reviewers is associated with the observation matrix, the methods using such information often treat it as kind of static knowledge rather than dynamic feedback. They mainly focus fitting the
When the static observation æ; in 11 is replaced with sequential observation æ is naturally extended sequential recommendation system with feedback loop. In this work, we focus the case with static observations and train recommender system accordingly.
2

---

Under review as a conference paper at ICLR 2020
groundtruth recommended items with the recommender iT-) given fixed 's and v, ss while ignore the imitation of the whole recommendation- feeebbacn-e loop in (T). Without the feedback mechanism f()), x()) tends to over-fit fit the observed user behavior and static side information, which may degrade in practical dynamical scene.
To overcome the problems mentioned above, we propose collaborative filtering framework with synthetic feedback loop (CF-SFL), which explains the whole recommendation process from a view- point of reinforcement learning. As shown in Figure I besides traditional recommendation mod- ule the proposed framework further introduces a virtual user, which imitates the recommendation- feedback loop even if the real-world feedback are unobservable.
2.2 RECOMMENDER
In our framework, the recommender implements the function sT-) in (I. which takes the observed user behavior 2; and his/her previous feedback embedding of as input and recommends items ac- cordingly. In principle, the recommender x--) can be defined with high flexibility, which can be arbitrary collaborative filtering methods that predicting items from user representations, such as WMF Hu al. (2008), CDAE Wu al. (2016), VAE Liang al. (2018) etc. In this work, we formulate the recommender from the viewpoint of reinforcement learning.
In partieular, the recommendation-ffeedhaee loop generates sequence of interactions between each user and the recommender, i.e., (s (tt,a))== for = I.... N. Here, sf [ay; v/] is the representation of user i time which is a sample in the state space S describing user preference. at indicates the recommended items provided by the recommender, which is sample in the action space A of the recommender. Accordingly, we can model the recommendation--eeddeee Decision Process (MDP) M = A. P, R), where P: S x S IR is the transition probability of user preference and R: S x A + IR is the reward function used to evaluate recommended items. The recommender x-) works distribution items conditioned on user preference. The target recommender should be an optimal
loop as Markov
policy parametrized by 0, i.e., Ta s), which corresponds to the
policy that maximizes the expected reward: ((TO) = Σn_
t=1 Erg [R(s', a')], where R(s', a²) means the reward over the state-action pair (s', a²). For the i-th user, given sf the recommender selects potentially sereefially-prefferred items via
a = arg max No (als().
(2)
Note that different from traditional reinforcement learning tasks, in which both S and A are available while P and R are with limited accessibility, our recommender just receives partial information of state it does not observe users' feedback embedding t; In other words, optimize the recommender, we need to build a reward model and a feedback generator jointly, which motivates us introduce a virtual user into the framework.
2.3 VIRTUAL USER
The virtual user module aims to implement the feedback function f(-) in I). which not only models the reward the items provided by the recommender but also generates Teedback to complete the representations of state. Accordingly, the virtual user contains the following two modules:
Reward Estimator The reward estimator parametrizes the function of reward, which takes the cur- rent prediction at and user preference sf as input and evaluate their compatibility. In this work, implement the estimator with parameter Φb, which is defined as
( (s) a!) = sigmoid(g(h(x,, a))))
(3)
In this work, we use the static part of the state s i.e., the observed user behaviors æ; input. A(:,)) the fusion function which merges æ; and at into a real value vector (the fused input is shown in Figure and described in Appendix). 9(-) is the single value regression function that translates the fused input into a single reward value. The sigmoid function is used to restrict the regression value between 0 and 1.
Feedback Generator The feedback generator connects the reward estimator with the recommender via generating feedback embedding, i.c.,
e Fx (h(x,, at), R(s), a()).
3

---

Under review as a conference paper at ICLR 2020
t-0
t=1
t=T-1
R(st.a)
R(s). a))
R(e
CR
Reward
Reward Esiimator
Rewerd Estirator
b h2 o.)
h( ag
Ma2 4
I (args aq
Feeebackk Generator
Feedbast Gasendior
Feedback Ganerator
Comp
Recommeende
Recommender
Recommender
Recommender
2
Figure Unrolling the recurrent CF-SFL framework into an iterative learning process with multiple time steps T.
where vp represents the parameters of the generator. Specifically, the parametric function (ts considers the fused input and the estimated reward and returns a feedback embedding of E Rd to the recommender. R(s a;) indicates the compatibility between the recommended items and user preferences, and h(,,, a,), which is vector rather than scalar like reward, further enriches the information of the reward to generate feedback embeddings. Consequently, the recommender receives the informative feedback complementary component of the static observation æ; to make an improved recommendation via (2
LEARNING ALGORITHM
3.1 LEARNING TASK
Based on the proposed framework, we need to jointly learn the policy corresponding to the recom- mender Te. the reward estimator Rs and the feedback generator Fut. Suppose that have set of labeled samples D {æ,, a.). where æ; RM is the historical behaviors of user i derived from the user-item matrix X and a; is the ground truth of the recommended item for the user based on his/her behavior æ;. We formulate the learning task as the following min-max max optimization problem.
min F& maX L(To. Rø F&p)
(5)
where
[( Rø, Fw)
E, Crp Cup (a., a.; no, F+) Ba [log( (s, a))] Ba-D/1 log(Ro((s a))]
(6)
supervised
regularizer
In particular, the first term Suup in 66 can any supervised loss based on labeled data D, e.g.. the evidence lower bound (ELBO) proposed in VAÉs Liang et al. (2018) (and used in our work). This term ensures the recommender to fit the groundtruth labeled data. The second term considers the following two types of interactions among these three modules:
The collaboration between the recommender policy no and the feedback generator F towards better predictive recommender.
The adversarial game between the recommender policy me and the reward estimator Rg-
Accordingly, given current reward model, update the recommender policy no and the feedback generator Fe to maximize the expected reward derived from the generated user preference and the recommended item On the contrary, given the recommended policy and the feedback generator, we improve the reward estimator Rs by sharpening its criterion - the updated reward estimator maximizes the expected reward derived from the generated user preference and the ground truth of item while minimize the expected reward based on the recommended item. Therefore, we solve via alternating optimization. The updating of тo and F& is achieved by minimizing
Co F+)) E, Lapla, ā no. Fw) Ea-n, (og(R,((s. a))}
And the updating of Rs is achieved by maximizing
CA(R)) Ear,, log( s, a))l Ea~p|1 (o, -
(8)
Both these two updating steps can be solved effectively via stochastic gradient descent.

---

Because the proposed framework contains closed loop among learnable modules, during training we unroll the loop and let the recommender interact with the virtual user in T steps. Specifically, the initial stage, the recommender takes the observed user behaviour æ; and all-zero initial feedback embedding vi, to make recommendations. At each step I, the recommender predicts the items a given z; and of to the virtual user, and receives the feedback embedding uf+1 The loss defined according to the output of the last step, i.c., a" and v², and the modules are updated accordingly. After the model is learned, in the testing phase need to infer the recommended item Under review as a conference paper at ICLR 2020
3.2 UNROLLING FOR LEARNING AND INFERENCE
in the same manner, unrolling the feedback loop and deriving a as the final recommended item. The details of unrolling process illustrated in Figure Z. and the detailed scheme of our learning algorithm is shown in Algorithm in appendix.
CF-SFL AS INVERSE REINFORCEMENT LEARNING
Our CF-SFL framework automatically discovers informative user feedback as side information and gradually improve the training for the recommender. Theoretically, it closely connects with Inverse Reinforcement Learning (IRL). Specifically, we jointly learn the reward function R(- and the pol- icy (recommender) π ((; +) from the expert trajectories DE (the observed labeled data). Dg typically consists of state-action pairs generated from an expert policy TH with the corresponding environ- ment dynamics PE- And the goal of the IRL. to recover the optimal reward function Rt (..),.) as well as the corresponding recommender π Formally, the IRL is defined as:
(R', n é (RI(TE) = arg max RERSXA
Σ pe(s, a) R(5, [max H(T) + Eplss R(s,,a) (9) xell
max min - (T) + PERSXA TEll
Σ (e(ss.a)((ss.0) a) P ((s.a))((s..)
(10)
L(x.R)
Intuitively, the objective enforces the expert policy TE to induce higher rewards (the max part), than all other policies. This objective is sub-optimal if the expert trajectories are noisy, i.e. the expert is not perfect and its trajectories are not optimal. This will render the learned policy always performs worse than the expert one. Besides, the illed-define IRL. objective often induces multiple solutions due to flexible solution space, i.e., one assign arbitrary reward to trajectories not from expert, as long as these trajectories yields lower rewards than the expert trajectories. To alleviate these issues, some constraints are placed into the objective functions, e.g., convex reward functional, rb RSA + R, which usually works as regularizer,
RR", m = arg max min C( R) (2)))((R)) (R)
(11)
RERSXA xell
To imitate the expert policy and provide better generalization, we adopt the adversarial regular- izerHo Ho & Ermon (2016), which defines &b with the following form:
(R) & {
Eng [a([(ss a))l if R(s, 2 +00
otherwise
where g(z) log(l This regularizer places low penalty on reward functions R that assign an amount of positive value to expert state-action pairs; however, if R assigns low value (close zero, which is the lower bound) to the expert, then the regularizer will heavily penalize R. With induced adversarial regularizer, we obtain a new imitation learning algorithm for recommender:
min 1b" (p= pae) (HT)
(12)
Intuitively, we want to find saddle point (Ro, Ta) of the expression:
Er, (osg([(ssu a))] + E. log(R[(s a))] (H(T)))
(13)
where a) E (0, 1). Note equation 1I is derived from the objective of traditional IRL. How- ever, distinct from the traditional approach, we propose a feedback generator provide feedbacks to the recommender. In terms of the reward estimator, it tends to assign lower rewards to the pre- dicted results by the recommender To and higher rewards for the expert policy Tg, which aims to discriminate n. from Tg:
Ln Er, Dog( R(s,
(og(R(()) a))] E.g log (os(R(ss))))
(14)
5

---

Under review as a conference paper at ICLR 2020
Similar to standard IRL, we update the generator to maximize the expected reward with respect to log R(s, a). moving towards expert-like regions of user-item space. In practice, we incorporate feedback embedding to update the user preferences, and the objective of the recommender is:
LF = l of], a)]l (Ta)
(15)
where of = (h a.) a()).
RELATED WORK
Collaborative Filtering. Collaborative filtering (CF) can be roughly categorized into two groups: CF with implicit feedback Bayer et al. (2017): Hu et (2008) and those with explicit feedback Ko- ren (2008); Liu et al. (010)) In implicit cit CF, user-item interactions are binary in natural (i.e., I ii clicked and otherwise) as oppose to explicit CF where item ratings (c.g., 1-5 stars) are typically the subject of interests. Implicit CF has been widely studied, examples including factorization of user- item interactions He et al. (0166; Koren 2008); Liu el. (2016); |Rendle (2010); Rennie & Srebro
(2055) and ranking based approach Rendle al. (2009). And our CF-SHL is new framework for implicit CF.
Currently, neural network based models have achieved state-of-the-art performance for various rec- ommender systems Cheng et al.] (2016); He et al.] (2018) 2017); ZZhag et al. (2018); Liang et al. (2018 Among these methods, NCF He et al. (2017) casts the well-established matrix factorization algorithm into an entire neural framework, combing the shallow inner- unner-product based learner with a series of stacked nonlinear transformations. This method outperforms various of traditional base- lines and has motivated many following works such as NFM He et al. (2017), Deep FM Guo et al. (2077) and Wide and Deep Cheng al, (2016). Recently, deep generative has achieved remarkable success. VAEs VAEs|Liang al. (2018) uses variational inference to scale up the algorithm for large-scale dataset and has shown significant success recommender systems with a usage of multinormial likelihood. Our CF-SFL is general framework that adapt to these models seamlessly.
RL in Collaborative Filtering. For RL-based methods, contextual multi-armed bandits are firstly utilized to model the interactive nature of recommender systems. Thompson Sampling (TS) Chapelle & Li (201T); Kveton et al. (2015): Zhang et al. (2017) and Upper Confident Bound (UCB) et al. (2010) are used to balance the trade-off between exploration and exploitation. /Zao (et al.] (2013) combine matrix factorization with bandits to include latent vectors of items and users for better exploration. The MDP-Based CF model can be viewed a partial observable MDP (POMDP) with partial observation of user preferences [Sunehag et al. (2015). Value function ap- proximation and policy based optimization can be employed to solve the MDP. Zheng et al. (2018) and Taghipour& & Kardan (2008) modeled web page recommendation as a Q-Learning problem and learned to make recommendations from web usage data. (Sunehag et al. (2015) introduced agents that successfully address sequential decision problems. Zhhao et al. (2018) propose novel page- wise recommendation framework based on deep reinforcement learning. the recommending procedure as sequential interactions between virtual users and recommender; and Recently./Chen leverage feedbacks from virtual users to improve the recommendation. Recently, [Chen et al. (2019) proposed off- off-policy corrections technique, and successfully applied it in real-world applications.
this paper, we consider
EXPERIMENTS
Datasets We investigate the effective- ness of the proposed CF-SFL framework on three benchmark datasets of recom- mendation systems. (i) MovieLens-20M (ML-20MM, a movie recommendation ser- vice contains tens of millions user-movie ratings. (ii) Netflix-Prize (Netflix), an- other user-movie ratings dataset collected by the Netflix Prize Bennett & Lanning (2007). (ii)) Million Song Dataset (MSD), a user-song rating dataset, which is released as part of the Million Song Dataset Dataset[Bertin rectly compare with existing work, we employed the same pre-processing procedure as Liang et al. 2018). A summary statistics of these datasets provided in Table 1
Table 1: Basic information of the considered datasets.
ML-200
Netflix
MSD
of users
136,677
463,435
571,355
# of items
20, 80,108
17,769
41,140
# of interactions
10.0M
56.9M
33.6M
# of held-out-users
10.0K
40.0K
50.0K
% of sparsity
0.36%
0.69%
0.14%
et al. (201T). To di-

---

Under review as a conference paper at ICLR 2020
Performance comparison between our CF-SHL framework and various of baselines. VAE* is the results hased on our own rums and VAR is the VAF model with our reward estimator.
Table 3:
ML-20M
Netflix
MSD
Methods
R@20
R@50
NDCG@ 100
R@20
R@50
NDCG@ 100
R@20 RØ50 NDCG@ 100
SLIM
0.370
0.495
0.401
0.347
0.428
0.379
WMF
0,360
0.498
0.386
0.316
0.404
0.351
0.211
0.312
0.257
CDAE
0.391
0.523
0.418
0.343
0.428
0.376
0.188
0.283
0.237
aWAE
0.391
0.532
0.424
0.354
0.441
0.381
VAE
0.395
0.537
0.426
0,351 0.444
0,386
0.266
0.364
0.316
VAE*
0.395
0.535
0,425
0.350
0.444
0.386
0.260 0.356
0.311
VAE!
0.396
0.536
0.426
0.352
0.445
0.387
0.263
0.360
0.314
CF-SFL
0.404
0.542
0.435
0.355
0.449
0.394
0.273
0.369
0.323
Notlkx
MBD
8.445
VAg
VAD Dannertrrr oBL EoAn (-3)
0.308
$,4400
CF-O7L (AAE Tea)
(Th0)
ch-on, La ubs ronk (-:)
0.427S
0.008
0.315
0.384
0.300
2,000
M/
3.38
0.4m7
0.500
2/250
2.500
140 200
60
Spocits
Figure 3: Performance (NDCG@ 100) boosting on the validation sets.
Evaluation Metries We employ Recall together with NDCG@ the evaluation metric for
recommendation, which measures the similarity between the recommended items and the ground truth. Recall@r considers top-r recommended items equally, while NDCG@r ranks the top-r items and emphasizes the importance of the items that are with high ranks.
Set-up For our CF-SFL frame- work, the architectures of its recom- mender, reward estimator and feed- back generator
Table 2: Architecture our CF-SFL framework.
Recommender
Reward Estimator Feedback Generator
Input Rd
Input R
Input R
shown in Table To represent the user preference, we normalize æ; and of > 0) indepen- dently and concatenate the two into a single vector. To learn the model, we pre-train the recommender (150 epochs for MI.-20M and 75 epochs for Netflix and MSD) and optimize the entire framework (50 epochs for ML-20M and 25 epochs for the other two). ls regularization with a penalty term 0.01 is
M x 600, tanh
64 x 128, ReLU
x4x x 128, ReLU
600 x 200 (x2)
128 x 128, ReLU
Sample R200 200 x 600, tanh
128 128, ReLU
128 x 128, ReLU
600 x M softmax
128 x 1, sigmoid
128 x 128
applied to the recommender, and Adam optimizer Kingma & Ba (2014) with batch in size of 500 is employed. The framework is built with Tensorflow and will be publicly available upon publication.
Baselines To demonstrate the superiority of our framework, we consider multiple state-of-
art approaches as baselines, which can be categorized into two types: (i) Linear based models:
SLIM Ning & Karypis (201T) and WMF Hu et al.) (2008). (ii) Deep neural network based models: CDAE et al. (2016). VAE Liang et al. (2018) and AWAE/Zong & Zhang (2018). It should be noted that our CF-SFL is generalized framework, which is compatible with all these approaches. In particular, as shown in Table we implement our recommender as the VAE-based model Liang
et al. (2018) for fair comparison. In the following experiments, we will show that besides such setting the recommender can be implemented by other existing models as well.
Performance Analysis All the evaluation metrics are averaged across all the test sets. (i) Quantitative results: we test various methods and report their results in Table With the proposed CF-SFL framework, we significantly improve the performance of the baselines on all the evaluation metrics. These experimental results demonstrate the power of the proposed CF-SFL framework, which provides informative feedback as the side information. Particularly, we observed that the per- formance between the base model (VAE*) is similar to that of its variation with the reward estimator (VAEt). It implies that simply learning a feedback from the reward estimator via pack-propagation
7

---

Under review as a conference paper at ICLR 2020
11, 2001
VILD
(ol) eepech
NOCO9100
0.033
0330
0.22
0319
0.0.2
5.500
100
0.315
0.022
0425
Recurient steps
Figure 4:
The blue curve summarizes NDCG@ 100 and red curves report the computational cost for model inference in each epoch. In each sub-figure, we vary the time steps from to 12 (T=0 is the hase recommender).
inefficient. Compared with such a naive strategy, the proposed CF-SFL provides more informative feedback to the recommender, and thus, is able to improve recommendation results more effectively.
(ii) Learning Comparison: In Figure 3 we show the training trajectory of the baselines (VAE, VAE+reward estimator) and the CF-SFL with multiple time steps. There are several interesting findings. (a) The performance of the base VAE doesn't t improve after the pre-training steps, e.g., 75 epochs for Netflix. In comparison, the proposed CF-SFL. framework can further improve the perfor- mance once the whole model is triggered. (b) The CF-SFL vields fast convergence once the whole framework is activated. (c) Coincide with results in Table 3 the trajectory of VAE in Figure 3 is similar to that of the base VAEs (VAE*). In contrast, the trajectories of our CF-SFL methods are more smooth and able to converge to a better local minimum. This phenomenon further verifies that CF-SFL learns informative user feedback with better stability. (d) With an increasing of time steps T in a particular range (T < 8 for ML-20M), CF-SFL achieves faster and better performance. One possible explanation is the learning with our unrolled structure - parameters different time-steps, and a more accurate gradient is found towards the local minimum. (e) We find ML-20M and MSD are more sensitive the choice of I when compared with Netflxx Therefore, the choice of I" should adjust to different datasets.
shared across
(iii) CF-SFL with dynamic time steps: As shown in Figure 2 the learning of CF-SFL involves a recurrent structure with 7 times steps. We investigate the choice of T and report its influence on the performance of our method. Specifically, the NDCG@100 with different 7 is shown in Figure 4 Within 6 time steps, CF-SFL consistently boots the performance on all the three datasets. Even with a larger time steps, the results remain stable. Additionally, the inference time of CF-SFL is linear on time steps T. To achieve a trade-off between performance and efficiency, our experiments we set T to 8 for ML-20M and Netflix and 6 for MSD.
Generalization Study As afore- mentioned, our CF-SFL is general- ized framework which is compatible with many existing collaborative fil- tering approaches. We study the use- fulness of our CF-SFL different recommenders and present the results in Table 4 Specifically, two types of recommenders are being considered: the linear approaches like WARPWe ston et al. (2011) and MF Hu et al. (2008), and deep learning methods, e.g., DAE Liang et al. 2018 and the variation of VAE in Liang et al. (2018). We find that our CF-SFL is capable of generalizing to most of the exisiting collab- orative filtering approaches and boosts their performance accordingly. The gains achieved by our CF-SFL may vary depending on the choice of recommender.
Table 4: Performance of our CF-SFL with various of rec- ommenders are reported.
Recommender w/o CF-SFL w CF-SFL Gain (10 ")
WARP
0.31228
0.33987
+27.59
MF
0.41587
0.41902
+3.15
DAE
0,42056
0,42307
+2.51
VAB
0.42546
0.43472
+9.26
VAE-(Gaussia
0.42019
0.42751
+7.32
VAE-(8 = 0)
0.42027
0.42539
+5.02
VAE-Lincar 0.41563
0.41597
+0.34
7 CONCLUSION
We propose a CF-SLF framework to simulate user feedback. It constructs a virtual user to provide informative side information as user feedback. Mathematically we formulate the framework as an IRL problem and learn the optimal policy by feeding back the action and reward. Specially, a recur- rent architecture was built to unrolled the framework for efficient learning. Empirically we improve the performance of state-of-the-art collaborative filtering method with a non-trivial margin. Our framework serves as a practical solution making IRL feasible over large-scale collaborative filter- ing. And it will be interesting to investigate the framework in other applications, such as sequential recomender systems etc.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Deepak Agarwal, Bee-Chung Chen, and Pradheep Elango. Fast online learning through offline initialization for sen-iiesitive recommendation. In KDD, 2010.
Immanuel Bayer, Xiangnan He, Bhargav Kanagal, and Steffen Rendle. A generic coordinate descent framework for learning from implicit feedback. In wwW, 2017.
James Bennett and Stan Lanning. The netflix prize. In KDD cup and workshop, 2007.
Thierry Bertin-Mahieux, Daniel PW Ellis, Brian Whitman, and Paul Lamere. The million song dataset. In /smir, 2011.
Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling. In NIPS, 2011.
Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed H Chi. Top-k off-policy correction for reinforce recommender system. In WSDM, 2019.
Xu Chen, Hongteng Xu, Yongfeng Zhang, Jiaxi Tang, Yixin Cao, Zheng Qin, and Hongyuan Zha. Sequential recommendation with user memory networks. In WSDM, 2018.
Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, al. Wide & deep learning for rec- ommender systems, In Proceedings of the 1st Workshop on Deep Learming for Recommender Systems, 2016.
Yi Fang and Luo Si. Matrix co-factorization for recommendation with rich side information and im- plicit feedback. In Proceedings of the 2nd International Workshop Information Heterogeneity and Fusion in Recommender Systems, 2011.
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: factorization- machine based neural network for ctr prediction. arXiv preprint 1703.004700447 2017.
Xiangnan He, Hanwang Zhang, Min-Yen Kan, and Tat-Seng Chua. Fast matrix factorization for online recommendation with implicit feedback. In SIGIR, 2016.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nic, Xia Hu, and Tat-Seng Chua. Neural collab-
orative filtering. In www, 2017.
Xiangnan He, Xiaoyu Du, Xiang Wang, Feng Tian, Jinhui Tang, and Tat-Seng Chua. Outer product- based neural collaborative filtering. arXiv preprint arXiv: (eo::8888.0312 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In NIPS, 2016.
Yifan Hu, Yehuda Koren, and Chris Volinsky. Collaborative filtering for implicit feedback datasets. In ICDM, 2008.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, arXiv preprint arXiv:1412.6988. 2014.
Yehuda Koren. Factorization meets the neighborhood: multifaceted collaborative filtering model. In KDD, 2008
Yehuda Koren and Robert Bell. Advances in collaborative filtering. In Recommender systems hand- book. 2015.
Branislav Kveton, Csaba Szepesvari, Zheng Wen, and Azin Ashkan. Cascading bandits: Learning to rank in the cascade model. ICML, 2015.
Lihong Li. Wei Chu, John Langford, and Robert E Schapire. A cantextual-bandie approach to personalized news article recommendation. In WWW. 2010.
Xiaopeng Li and James She. Collaborative variational autoencoder for recommender systems. In KDD, 2017.

---

Under review as a conference paper at ICLR 2020
Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. Variational autoencoders for collaborative filtering. WWW, 2018.
Nathan N Liu, Evan W Xiang, Min Zhao, and Qiang Yang. Unifying explicit and implicit feedback for collaborative filtering. In CIKM, 2010.
Yong Liu, Peilin Zhao, Xin Liu, Min Wu, and Xiao-Li Li. Learning optimal social dependency for recommendation. arXiv preprint arXiv: 1603. 04522, 2016.
Aditya Krishna Menon, Krishna-Pree Prasad Chitrapura, Sachin Garg, Deepak Agarwal, and Nagaraj Kota. Response prediction using collaborative filtering with hierarchies and side-information. In KDD, 2011.
Rajhans Mishra, Pradeep Kumar, and Bharat Bhasker. A web recommendation system considering sequential information. Decision Support Systems, 2015.
Xia Ning and George Karypis. Slim: Sparse linear methods for top-n recommender systems. In ICDM, 2011.
Steffen Rendle. Factorization machines. ICDM, 2010.
Steffen Rendle and Lars Schmidt-Thiems Online-updating regularized kernel matrix factorization models for large-scale recommender systems. In Recsys, 2008.
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. Bpr: Bayesian personalized ranking from implicit feedback. In UAI, 2009.
Jasson DM Rennie and Nathan Srebro. Fast maximum margin matrix factorization for collaborative prediction. In ICML, 2005.
Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. Item-based collaborative filtering recommendation algorithms. In WWW, 2001.
Markus Schedl. The Ifm- lb dataset for music retrieval and recommendation. In ICMR, 2016.
Peter Sunehag, Richard Evans, Gabriel Dulac-Arnold, Yori Zwols, Daniel Visentin, and Ben Cop- pin. Deep reinforcement learning with attention for slate markov decision processes with high- dimensional states and actions. arXiv preprint arXiv1151.001124, 2015.
Nima Taghipour and Ahmad Kardan. hybrid web recommender system based on q-learning. In SAC, 2008
Jiaxi Tang and Ke Wang. Personalized top-n sequential recommendation via convolutional sequence embedding. In WSDM, 2018.
Qinyong Wang, Hongzhi Yin, Zhiting Hu, Defu Lian, Hao Wang, and Zi Huang. Neural memory streaming recommender networks with adversarial training. In KDD, 2018.
Weiqing Wang, Hongzhi Yin, Shazia Sadiq, Ling Chen, Min Xie, and Xiaofang Zhou. Spore: sequential personalized spatial item recommender system. In ICDE, 2016.
Jason Weston, Samy Bengio, and Nicolas Usunier. Wsabie: Scaling up to large vocabulary image annotation. IJCAI, 2011.
Chao-Yuan Wu, Amr Ahmed, Alex Beutel, Alexander J Smola, and How Jing. Recurrent recom- mender networks. In WSDM, 2017.
Yao Wu, Christopher DuBois, Alice X Zheng, and Martin Ester. Collaborative denoising auto- encoders for top-n recommender systems. In ICDM, 2016.
Carl Yang. Lanxiao Bai, Chao Zhang, Quan Yuan, and Jiawei Han. Bridging collaborative filtering and semi-supervised learning: a neural approach for poi recommendation. In KDD, 2017.
Ruiyi Zhang, Chunyuan Li, Changyou Chen, and Lawrence Carin. Learning structural weight un- certainty for sequential decisi. arXiv preprint arXiv: 18888000088 2017.
10

---

Under review as a conference paper at ICLR 2020
Shuai Zhang, Lina Yao, Aixin Sun, Sen Wang, Guodong Long, and Manqing Dong. Neurec: On nonlinear transformation for personalized ranking. arXiv preprint arXiv:1885.03002, 2018.
Xiangyu Zhao, Liang Zhang, Zhuoye Ding, Long Xia, Jiliang Tang, and Dawei Yin. Recom- mendations with negative feedback via pairwise deep reinforcement learning. arXiv preprint arXiv:1802.06501, 2018.
Xiaoxue Zhao, Weinan Zhang, and Jun Wang. Interactive collaborative filtering. In CIKM, 2013.
Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. Dr:: A deep reinforcement learning framework for news recommendation. WWW, 2018.
Jingbin Zhong and Xiaofeng Zhang. Wasserstein autoencoders for collaborative filtering. arXiv preprint arXiv:1809.05660 2018

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
Algorithm 1: CF-SFL training with stochastic optimization
1: Input: A user-item matrix X and labeled pairs D, the unrolling step T, the size of batch b. 2: Output: Recommender ne, reward estimator Rs. and feedback generator Fe
3: Initialization: randomly initialize 0, & and 1b;
/* stage 1: pretrsin the rezcmmender +/
4: while not converge do
5: Sample batch of (æ;, a;)²=1 from D;
6: Update f via minimizing Lsup-
7: end while
/* st.ae 2:
pretrain the reward astimator x/
8: while not converge do
9: Sample a batch of (æ,, ā.};= from and calculate a.));
Sample another batch of user {mx) and set v; w-=0 =0
Infer the recommended items {a, and calculate (R(s,, a Mai
12: Update e via maximizing 8
end while
/* stage 3: alternative train all the modules +/
14:
while not converge do Sample batch of {æ,, āi 1 from D, initialize feedback embedding vd O;
15:
* Update zecommender and reedback generato: */
16: Feed (x:))=_ and (0) into the recommender and infer {a. ya= through a 7-step recurrent structure.
17:
Collect the corresponding reward [R(s " Update O and sp via minimizing (
18:
Reward est estimatoz update step
Sample batch of {a,; ā; ā=) from D, and calculate { (R(s,, ā:)),=
Sample batch of {x.)/- infer the recommended items {a } and calculate [R(si, a }
21:
Update & via maximizing
8
end while
A.1 FUSION FUNCTION
Here we give a detail description of the fused function we have proposed. A straight-forward way to build the fusion function h(æ,, a;) is to concatenate æ; and a,, and feed it into linear layer to learn lower dimensional representation. However, in practice this method is infeasible since the dimension of items, M, is extremely high and the usage of the concatenation will make the problem even worse. To this end, we introduce a sparse layer. This layer includes a lookup table R Mxd Once we have inferred the the recommended items a, based on the observation Li, we build the the fused input as
h(æ;, a.))
Σ
B(zi))BB
Σn
k= agsy Br
(16)
27g
where d is the Dirac Delta function and takes value 1 if Dij 1. [x, is number of in æ; The parameters of the lookup table will be automatically learned during the training phrase. We show an example illustrate the working scheme for the proposed fusion function in Figure The benefits for the proposed approach can be summarized as two folds: 1) it reduce the computational cost of the standard linear transformation under the general sparse set up and saves number of parameters in our proposed adversarial learning framework; 2) This lookup table shared across the observation and the recommended items, building unified space for the users existing preference and missing preference. Empirically such shared knowledge boosts the performance of our CF-SFL framework.
12

---

Under review as a conference paper at ICLR 2020
B
D
-1.2
0.5
x.5
h (xi, ai
0.6
x.5
-0.8
1.2
-0.44
0.08
ai
0.6
-0.7
-1.2
0.5
X .1
.5
0
-0.8
1.2
x.4
0.3
1.1
0.6
-0.7
X.5
Figure 5: An example of the our fused function working scheme. The user behavior æ; and the recommended items a. share the same lookup table B. [-0.04, 0.08) is the fused input for the given example. This method works efficient if and a, sparse.
13


Reviews:
[{"review_id": "rJx5gsqRKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Although I assume somebody well-versed in the recent collaborative filtering literature would not have trouble, I had too much difficulty understanding the setup and the model to be able to recommend the paper for acceptance.  Perhaps if you clarify the following questions in your rebuttal and the final version of the paper, I can view the paper more favorably:\n\n- In the Problem Statement section, it says that we assume that there is an unknown user preference vector a_i for each user with elements in reals.  How should we think about what a_i is?  Is it a binary vector? A probability distribution over items?  An arbitrary vector of reals? Nonnegative reals?  \n- In the problem statement, we have that \"recommender predicts preferred items . . . and the user generates feedback to the recommender.\"  Can you make this concrete?  Would feedback be like a binary label for whether each of the recommended items was of interest?  Or whether any were of interest? \n- In general, I had a very difficult time figuring out the difference between the information in x_i and \"feedback\".  Is it that x_i only has information about what user i selected, but no information about what the user was presented and did not select?  Is the feedback then the combination of what was presented and whether or not it was selected?\n- In Section 2.2, we discuss a_i^t as the recommended items provided by the recommender, sampled from the action space A -- this seems like a different a_i from the one in the Problem Statement section?  If so, this is an unfortunate notation clash.\n- If we're producing sets of items at a time, as your wording frequently indicates, should we think of the action space A as a set of sets of items?  \n- When describing the \"recommendation-feedback loop as a Markov Decision Process\", you introduce a transition probability over user preferences given actions.  Are you understanding a user's preferences to be changing over time?  Or is it just our estimate of the user's preferences that are changing over time?  \n- In Figure 2, does the Recommender have state that can accumulate all the virtual feedbacks v_i in each step?\n- After equation (1), it says that v_i is \"the embedding of user feedback to historical recommended items\", but in Figure 2 it seems like v_i can only have information about the preceding recommendation and estimated reward, rather than cumulative.  Can you clarify?\n- Is the reward estimator estimating something that could actually be observed?  Suppose we had a user in the loop -- can you give an example of what a reward would be for a particular set of actions?  Would it be 0/1 for whether the user clicked on one of the actions?   \n- In the setup for the learning task, Section 3.1 says that x_i is the \"historical behaviors of user i derived from the user-item matrix X and \\tilde{a}_i is the ground truth of the recommended item for the user based on his/her behavior x_i.\" I don't understand this.  x_i has all the items the user has clicked.  What new information is in \\tilde{a}_i?  Is it referring to the a_i in the problem statement, some vector representing preferences?  Is such a vector ever known during training?  Please clarify.\n- In the paragraph before equation 7, you say \"given the recommended policy and the feedback generator\" -- should that be \"given the recommender policy\"?\n- In Figure 2, we see the recommender producing what seems to be a vector a_i.  Is this a set of recommendations?  Or a distribution over items from which we could sample recommendations? Or a single recommendation, in which case the graphic should change?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SkeAqcyO_H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is essentially an attempt to incorporate a form of reinforcement learning into recommender systems, via the use of a synthetic feedback loop and a \"virtual user\".\n\nOverall this seems like a nice attempt to combine Inverse Reinforcement Learning frameworks with collaborative filtering algorithms.\n\nWithout a background in reinforcement learning, it was difficult for me to assess exactly how this compares to similar reinforcement learning work, or exactly how \"obvious\" the technical contribution is.\n\nNevertheless combining reinforcement learning with recommender systems is a topic of growing interest, and it is nice to see a paper in this space that makes a contribution with strong quantitative experiments, i.e., it is able to compete with (reasonably) strong baselines.\n\nThe experiments mostly look good, with a few representative, and quite large, datasets. The baselines are perhaps not state-of-the-art but represent reasonable points of comparison and are enough for a proof-of-concept. The paper is quite thorough in terms of experimental comparisons.\n\nOverall this seems like an interesting approach and a reasonably timely paper, making what seems like a compelling contribution to a current hot topic, and passes the bar in terms of experiments. Regarding the merits of the technical contribution, I'll perhaps have to defer to other reviewers, but overall the contribution seems above the bar.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "Hkg4-Ixptr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to learn a \"virtual user\" while learning a \"recommender\" model, to improve the performance of the recommender system. The \"virtual user\" is used to produce the \"reward\" signal for training the recommendation system (which is trained using RL). Jointly learning the recommender and the \"virtual user\" provides a synthetic feedback loop that helps to improve the performance of the recommendation system. The paper formulates the training of the \"virtual user\" as an inverse RL problem and uses adversarial regularizer.\n\nThe paper proposes an interesting idea but more experiments (and explanation) is needed to bring out the usefulness of the work. In general, the writing needs to be polished as well.\n\n===============\n\nFollowing are the items that need to be improved:\n\n## Significance of the results\n\n* The results in Tables 3 and 4 provide only marginal improvements over the baseline. These improvements do not appear to be statistically significant. It would help if the authors comment on why these results appear significant and also provide variance values/curves for the results.\n\n## Role of the feedback general F\n\n* Is there any separate loss for training F or is it always trained along with \\pi?\n\n* Is the feedback capturing some sort of \"memory\" or \"past preferences/behaviors\" of the user? If yes, wouldn't using a recurrent recommender also capture these effects without needing the feedback model F? Note that I am not criticizing the choice of F. I am trying to understand the role played by F (in addition to the recommender pi).\n\n* If there is no separate loss for F, I wonder how would the performance change if the F network was to be removed and the reward value was to be fed into a recurrent recommended. (The paper seems to have considered a special case where a non-recurrent recommender is used with the reward value). The reason I am stressing on this is that one of the key distinctions of the authors' work is the use of feedback generator and it would be useful to quantify the benefits on this modification.\n\n\n## Others\n\n* How is the static representation, x, computed? From eq 16 (appendix), it appears that x is a binary vector that captures what items have been purchased/reviewed. Is that correct? If yes, wouldn't x have an enormous dimensionality?\n\n* The recommendation system is operating in a sequential decision-making setup. The formulation of R and F do no consider any sequential information. For example, let us say that the recommender recommends the items a1, a2 and a3 in 3 timesteps. The reward at time 3 is a function of x and a3 and the information of a1 and a2 is not used.\n\n* The loss function has many components and I want to make sure I understand what gradients flow where. So please correct me if I missed something;\n\n    * supervised learning loss (from the real data) trains pi and F (equation 6).\n    * pi and F collaborate to get a high reward from R (since we do not have the ground truth corresponding to R). No gradient flows to R.\n    * Adversarial game between pi and R - which is used to update R.\n\n* I understand that the loss is defined according to the output of the last step but do the gradients flow through all the intermediate steps?\n\n* The training/inference procedure seems to be doing something strange. Let us assume that T = 5. So the 5 items are recommended to the \"virtual user\" and only the 5th item is recommended to the real user. Now the recommendation of the 5th item depends on the first 4 items that the real user has not seen.\n\n* In equation 15, what does the subscript F stand for?\n\n* What algorithm is used to train the policy pi?\n\n=================\n\n\nThe following are the items that should be corrected in the future iterations of the paper. Please note that these things did not impact the score.\n\n* It seems that the irrelevance of an item (for a user) is treated the same way as missing information about the relevance of the item. Could the authors discuss this more in the paper?\n\n* Is there any reason why this approach is only used with CF methods?\n\n* Some writing choices seem to make the problem statement more complex than it is. For example, the authors discuss how their work is different than traditional RL instead of simply stating that their work is set up as a POMDP.\n\n* Articles are missing (or mixed) up at many places.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to learn a "virtual user" while learning a "recommender" model, to improve the performance of the recommender system. A reinforcement learning algorithm is used for address the problem the authors defined. Multiple reviewers raised several concerns regarding its technical details including the feedback signal F, but the authors have not responded to any of the concerns raised by the reviewers. The lack of authors involvement in the discussion suggest that this paper is not at the stage to be published.
Decision: Reject