RaCT: Toward Amortized Ranking-Critical Training For Collaborative Filtering 
Authors: Sam Lobel*
Chunyuan Li*
Jianfeng Gao
Lawrence Carin
Venue:  (2020)
Keywords: Collaborative Filtering
Recommender Systems
Actor-Critic
Learned Metrics

Abstract: We investigate new methods for training collaborative filtering models based on actor-critic reinforcement learning, to more directly maximize ranking-based objective functions. Specifically, we train a critic network to approximate ranking-based metrics, and then update the actor network to directly optimize against the learned metrics. In contrast to traditional learning-to-rank methods that require re-running the optimization procedure for new lists, our critic-based method amortizes the scoring process with a neural network, and can directly provide the (approximate) ranking scores for new lists.

We demonstrate the actor-critic's ability to significantly improve the performance of a variety of prediction models, and achieve better or comparable performance to a variety of strong baselines on three large-scale datasets.


Published as a conference paper at ICLR 2020
RACT: TOWARDS AMORTIZED RANKING-CRITICAL TRAINING FOR COLLABORATIVE FILTERING
Sam Lobel"i, Chunyuan Li Jianfeng Gao?, Lawrence Carin
Brownn University Microsoft Research, Redmond Duke University samuel wwwelloeeee..... chunyl@microsoft com
ABSTRACT
We investigate new methods for training collaborative filtering models based on actor-critic reinforcement learning, to more directly maximize ranking-bat basedd--addddd objec- tive functions. Specifically, we train a critic network to approximate ranking-bass metrics, and then update the actor network directly optimize against the learned metrics. In contrast to traditional learning-to-rank methods that require re-running the optimization procedure for new lists, our critic-based method amortizes the scoring process with a neural network, and can directly provide the (approximate) ranking scores for new lists. We demonstrate the actor-critic's ability to signifi- cantly improve the performance of variety of prediction models, and achieve better or comparable performance to a variety of strong baselines on three large-scale datasets.
INTRODUCTION
Recommender systems are an important means of improving user's web experience. Collaborative filtering is a aipele- technique in recommender systems (Ricci et al., (2015), in which patterns across similar users and items are leveraged to predict user preferences (Su & Khoshgoftaar 2009).
This naturally fits within the learning paradigm of latent variable models (LVMs) (Bishop 2006),
where latent representations capture the shared patterns. Due their simplicity and effectiveness. LVMs are still a dominant approach. Traditional LVMs employ linear mappings of limited modeling capacity (Paterek 2007 Mnih & Salakhutdinov 2008), and growing body of literature involves applying deep neural networks (DNNs) to collaborative filtering to create more expressive mod- els (He et al. 2017: Wu Wu
al.) 2016 Liang et al. 2018). Among them, variational autoencoders (VAEs) (Kingma& & Welling 2013 Rezende et al. (014)) have been proposed as non-linear extensions LVMs (Liang et al. 2018). Empirically, VAES significantly outperform many competing LVM- based methods. One essential contribution to the improved performance is the use of the multinomial likelihood, which is argued by Liang et al. (2018) to be a close proxy to ranking loss.
This property is desirable, because in recommender systems generally care more about the ranking of predictions than an individual item's 's score. Hence, prediction results are often evaluated using top-N ranking- based metrics, such as Normalized Discounted Cumulative Gain (NDCG) (Jävellin & Kekäläinen 2002). The VAË is trained to maximize the likelihood of observations; as shown below, this does not necessarily result in higher ranking-based scores. A natural question concerns whether one may directly optimize against ranking-basse metrics, which are by nature non-diffirentiable and piecewise-constann the coimaation-retreeval community, where
Previous work on learning-to-ran to-rank has been explored this question in of ranking loss are consid- Weston et al. 2013).
ered (Weimer et al. 2008 Liu et al. 2009
2014
In this paper, we borrow the actor-critic idea from reinforcement learning (RL) Sutton et al. 1998) to propose an efficient and scalable learning-ta to-kkkk algorithm. The critic is trained to approximate the ranking metric, while the actor is trained to optimize against this learned metric. Specifically, with the goal of making the actor-critic approach practical for recommender systems, we introduce novel feature- based critic architecture. Instead of treating raw predictions the critic input, and hoping the neural network will discover the metric's 's structure from massive data, we consider
"Corresoonding author Equal Contribution

---

Published as a conference paper at ICLR 2020
engineering sufficient statistics for efficient critic learning. Experimental results on three large-scale datasets demonstrate the actor-crite crioic' ability to significantly improve the performance of a variety of latent-variable models, and achieve better comparable performance strong baseline methods.
2 BACKGROUND: VAES FOR COLLABORATIVE FILTERING
Vectors are denoted as bold lower-case letters æ, matrices as bold uppercase letters X, and scalars as lower-case non-bold letters z. We use o for function composition, @ for the clement-wise multiplication, and for cardinality of set. ò(-) is the indicator function.
We use n E [I) to index users, and m E {I,.... 3 to index items. The user-item interaction matrix X E {0, NxM collected from the users' implicit feedback is defined as:
f1, if interaction of user with item m is observed;
Tum (0,
otherwise.
(I)
Note that Trm does not necessarily mean user n dislikes item m; they may simply be unaware of the item. Further, num = 1 is not equivalent to saying user n likes item m, but that there is at least interest.
VAE model VAEs have been investigated for collaborative filtering (Liang et al., 2018), where this principled Bayesian approach is shown to achieve strong performance on large-scal scale datasets. Given the user's 's interaction history æ [21,..., Far]" E {0, 10, our goal is predict the full interaction behavior with all remaining items. To simulate this process during training, a random binary mask b E (0, 1] is introduced, with the entry
un-masked, and masked. Thus, 2h = x @ b is the user's partial interaction history. The goal becomes recovering the masked interactions: Fp
(1 xy),, which equivalent recovering the full æ 2 is known.
In LVMs, each user's binary interaction behavior is assumed to be controlled by a k-dimensional user-dependent latent representation
RRK When applying VAEs collaborative filtering (Liang et al. 2018 8), the user's ' latent feature is represented as distribution (( obtained from some partial history 2 of æ. With the assumption that (=x(<x) follows a Gaussian form, the inference of for the corresponding x: is performed as:
( (N, diag(o)))) with μe, 12
fa()))) 2y @ b, Ber(a),
where a is the hyper- parameter of a Bernoulli distribution, fo is Φ p--aametterized neural network, which outputs the mean μ and variance の² of the Gaussian distribution.
the generative process make predictions. In Liang et al. (2018) a multinomial distribution is used to model the likelihood of items. Specifically, construct pe [c 2), z is transformed produce a probability distribution over M items, from which the interaction vector x is assumed to have been drawn:
After obtaining user's latent representation 2,
x Mult(T). with π Softmax(go(z))
(3)
where is 0- parameterized neural network. The output is normalized via a softmax function to produce probability vector π E A M- (an (M 1)-simplex) over the entire item set.
Training Objective
Learning VAE parameters ( 0) yields the following generalized objective:
Ca(x; 0, () = Le with Lg - (=[m) (=(m) log po (z|z) [z) and LR <L=<L((a<>)))]lpl
(4)
where LE is the negative log likelihood (NLL) term, LR the KL regularization term with standard normal prior p(z), and B is a weighting hyper-parameter. When 3 = 1, we can lower-bound
the log marginal likelihood of the data using equation 4 as -La=1 (z=; 0 Φ)) log p(z). This is
commonly known as the evidence lower bound (ELBO) in variational inference (Blei et al. (017). Thus equation 4 is the negative 3-regularized ELBO. To improve the optimization efficiency, the reparametrization trick (Kingma & Welling 2013; Rezende et al. 2014) is used to draw samples to obtain an unbiased estimate of the ELBO, which is further optimized via stochastic optimization. We call this procedure maximum likelihood estimate (MLE)-barl based training, as it effectively maximizes the (regularized) ELBO. The testing stage of VAEs for collaborative filtering is detailed in Section of the Supplement.
z ~ 20 (2(x))
2

---

Published as a conference paper at ICLR 2020
Advantages of VAEs The VAE framework successfully scales to relatively large datasets by making use of amortized inference (Gershman & Goodman 2014): the prediction for all users share the same procedure, which effectively requires evaluating two functions the encoder (-) and the decoder go(+). Crucially, as all users share the same encoder/decoder, the number of parameters required for an autoencoder is independent of the number of users. This is in contrast to some traditional latent factor collaborative filtering models (Paterek] [0077] Hu et al., 20088 Mnihh & Salakhutdinov 2008). where a unique latent vector is learned for each user. The reuse of encoder/decoder for all users is well-aligned with collaborative filtering, where user preferences are analyzed by exploiting the similar patterns inferred from past experiences (Liang al.] 2018). VAEs has the two advantages simultaneously: expressive representation power non-linear model, and the number of parameters being independent of the number of users.
Pitfalls of VAEs Among various likelihood forms, it was argued in Liang et al. (2018) that multinomial likelihoods are closer proxy the ranking loss than the traditional Gaussian or logistic likelihoods. Though simple and ef- fective, the MLE procedure may still diverge with the ultimate goal in recommendation of correctly suggesting the top-ranked items. To illustrate the divergence between MI.E-based training and ranking-basse based evaluation, con- sider the example in Figure [ For the target {1, 1, 0, 0), two different predictions A and B are provided. In MLE, the training loss is the multinomial NLL: predicted probability. From the NLL point of view, B is a better prediction than A, because B shows a lower loss than A. However, B ranks an incorrect item highest, and therefore would return a worse recommendation than A. Fortunately, NDCG is calculated directly from the ranking, and captures this dependence. This inspired us to directly use ranking- evaluation metrics to guide training. For details on calculating NDCG, refer to Section E.2 of the Supplement.
Target
NDCG
Prediction
0.8
0.1
0.05
0.05
0.08
Prodiction
0.3
0.3
0.35
0.05
-log 0.09
0.693
Figure I: Difference between MLEebased training loss and ranking-based evaluation. For A, -1 log 0.8 1 log 0.1 = log 0.08; For B, -l 0.3 xlog 0.3 log 0.09. NLL assigns a better value to the misranked example than to the properly-ranked one. NDCG always assigns maximum value properly-ranked scorings.
x log π,, where π is the
3 RANKING-CRITICAL TRAINING
We introduce novel algorithm for recommender system training, which we call Ranking- Crittccl Training (RaCT). RaCT learns differentiable approximation to the ranking metric, which the prediction network then leverages as target for optimization through gradient ascent. This is in contrast to existing methods in collaborative filtering, which define an objective relaxation ahead of time. This methodology of learning approximations to functions which cannot be optimized directly stems from the actor-critic paradigm of RL, which we adapt for collaborative filtering.
Any ranking- based evaluation metric can be considered as a "black box" function w { 2x, b} + y [0, 1]. which takes in the prediction to compare with the ground -ruth (conditioned on the mask b), and outputs a scalar y to rate the prediction quality. As in equation 2 b partitions a user's interactions into those that are "observed" and "unobserved" during inference. As are only interested in recovering the unobserved items in recommendation, we compute the ranking score of predicted items T π 3 (1 2x) based on the ground- truth items Tp.
One salient component of a ranking- based Oracle metric w is to sort π.. The sorting operation is non- differentiable, rendering it impossible directly use w" the critic. While REINFORCE WWiliimss 1992) may appear to be suited to tackle the non-differentiable problem, it suffers from large estimate variance (Silver al. 2014), especially in the collaborative filtering problem, which has a very large prediction space. This motivates consideration of a differentiable neural network to approximate the mapping executed by the Oracle. In the actor-critic framework, the prediction network is called the actor, and the network which approximates the oracle is called the critic. The actor begins by making a prediction (action) given the user's interaction history as the state. The critic learns to estimate the value of each action, which we define the task-specific reward, i.e., the Oracle's 's output. The value predicted by the critic is then used to train the actor. Under the assumption that the critic produces the exact values, the actor is trained based on an unbiased estimate of the gradient of the prediction value in terms of relevant ranking quality metrics. In Figure 2 we illustrate the actor-critic paradigm in (b), and the traditional auto-encoder shown in (a) can be used as the actor in paradigm.
3

---

Published as a conference paper at ICLR 2020
Actor
Critic
Encoder
Decoder
Encoder
Decoder
Ranking Scores
X
Z
π
Z
(x)
Go (z(
fø
9o (z)
X
Wish (T)
(a) Traditional auto-encoder paradigm
(b) Proposed actor-critic paradigm
Figure 2: Illustration of learning parameters {Φ, }} in the two different paradigms. (a) Learning with MLE, as in VAEs; (b) Learning with learned ranking-critic. The actor be viewed as the function composition of encoder (o) and (e()) in VAEs. The critic mimics the ranking-based evaluation scores, so that can provide ranking-
ranking-sensiitve sensitive feedback in the actor learning.
Naive critic Conventionally one may concatenate vectors [Tp Tpl as input to a neural network, and train network output the measured ranking scores y. However, this naive critic is impractical, and failed in our experiments. Our hypothesis is that since this network architecture has a huge number of parameters to train (as the input data layer is of length 2M, where M > 10k), it would require rich data for training. Unfortunately, this is impractical: { x) } E M are very high-dimense dimensninniinll, and the implicit feedback used in collaborative filtering is naturally sparse.
Feature-based critic The naive critic hopes deep network can discover structure from massive data by itself, leaving much valuable domain knowledge unused. We propose more efficient critic, that takes into account the structure underlined by the assumed likelihood in MLE (Miyato & Koyama 2018). We describe our intuition and method below, and provide the justification from the perspective of adversarial learning in Section D of the Supplement.
Consider the computation procedure of the evaluation metric as function decomposition w WyoW including two steps:
Wp - h, feature engineering of prediction into the sufficient statisties h wi h + J. neural approximation of the mapping from the statistics h to the estimated ranking score ý. using a w--parameterized neural network.
The success of this two-step critic largely depends on the effectiveness of the feature h. We hope
feature h is (i) compact that fewer parameters in the critic we can simplify training: (ii) easy-to- compute so that training and testing is efficient; and (iii) informative so that the necessary information is preserved. We suggest to use 3-dimensional vector as the feature, and leave more complicated feature engineering as future work. In summary, our feature is
[Cg. (Hol. [l.
(5)
where (i) LE is the negative liglliieldood in equation 4 detined in the MLE training loss. (ii) |Hl]
is the number unobserved items that a user will interact, with Hgg = {m (rm = and 6n 0). (ii) [7/1] is the number of observed items that user has interacted, with H/1 = {m|rm = 1 and b,. 1}.
The NLL characterizes the prediction quality of the actor's output π against the ground-truth æ in an item-to-in flem-to-item comparison manner, e.g., the inner product between two vectors - log as in the multinomial NI.L (Liang et al., 2018). Ranking is made easier when there are many acceptable items to rank highly (e.g. when iHol is large), and made difficult when predicting from very few interactions (e.g. when [H1] is small), motivating these two features. Including these three features allows the critic guide training by weighting the NLL's relation ranking given this context about the user. Interestingly, this idea to consider the importance of user behavior statistics coincides with the scaling trick in SVD (Nikolakopoulos et al.) 2019b).
Note that Hoo and [741 are user-specific, indicating the user's frequency to interact with the system, which can be viewed as side-information about the user. They are only used as features in training the critic to better approximate the ranking scores, and not in training the actor. Hence, we do not use additional information in the testing stage.
Actor Pre-training In order to be a helpful feature for the critic, the NLL must hold some rela- tionship to the ranking-basse based objective function. But for the high-dimensional datasets common to collaborative filtering, the ranking score is near-uniformly zero for randomly-initialzzed actor. In
4

---

Published as a conference paper at ICLR 2020
this situation, trained critic will not propagate derivatives the actor, and therefore the actor will not improve. We mitigate this problem by using a pre-trained actor, such as VAEs that have been trained via MLE.
Critic Pre-training Training generic critic to approximate the ranking scores for all possible predictions is difficult and cumbersome. Furthermore, it is unnecessary. In practice, a critic only needs to estimate the ranking scores on the restricted domain of the current actor's 's outputs. Therefore, we train the critic offline on top of the pre-trained MLE-based actor. To train the critie, we minimize the Mean Square Error (MSE) between the critic output and true ranking score y from the Oracle:
(h, y; ub) = 1 (h) yll?
(6)
where the target y is generated using its non con-differential definition, which plays the role of ground truth simulator in training.
Actor-critic Training Once the critic is well trained, we fix its parameters vp and update the actor parameters () to maximize the estimated ranking score
L Φ 0) Wab (h),
where h is defined in equation 5 including NLL feature extracted from the prediction made in equa- tion together with count features. During back-propagation, the gradient of LA wrt the prediction π aLa It further updates the actor parameters, with the encoder gradient dCs. aLs On 86 8 86
8e 8h yx
and the decoder gradient dC aLa 0x Updating the actor changes its predictions, so must update the critic to produce the correct ranking scores for its new input domain.
00 8m
The full RaCT training procedure is summarized in Algorithm 1 in the Supplement. Stochastic optimization is used, where a batch of users 21 = {æa|i E B} is drawn at each iteration, with B as a random subset of user index in {1,... .N. The pre-training of the actor in Stage 1 and the critic in Stage 2 are important; they provide good initialization to the actor-critic training in Stage 3 for fast convergence. Further, we provide an alternative interpretation view our actor-critic approach in equation 6 and equation ( the perspective of adversarial learning (Goodfellow et al. 2014) in the Supplement. This can partially justify our choice of feature engineering.
4 RELATED WORK
Deep Learning for Collaborative Filtering There are many recent efforts focused on developing
deep learning models for collaborative filtering (Sedhain et al., 2015) Xue et al.] [017]] He et al. 2018a b) Zhang et al., 2017| Chen et al.. 2017). Early work on DNNs focused on explicit feedback settings (Georgiev& & Nakov 2013 Salakhutdinov et al., 2007 Zheng et al. 2016), such rating predictions. Recent research gradually recognized the importance of implicit feedback (Wu et al. 2016 He et 2017 2017 Liang et al. 2018), where the user's preference is not explicitly presented (Hu ct al. 2008). This setting is more practical but challenging, and is the focus of our work. The proposed actor-critic method belongs the general two-level architectures for recommendation systems, where a coarse to fine prediction procedure is used. For systematic method comparison for top-N recommendation tasks, we suggest referring to Dacrema al. (2019). Our method is closely related to three papers, on VAES (Liang et al. 2018), collaborative denoising autoencoder (CDAE) (Wu et al. 2016) and neural collaborative filtering (NCF) (He et al. 2017). CDAE and NCF may suffer from scalability issues: the model size grows linearly with both the number of users as well as items. The VAE (Liang et al., 2018) alleviates this problem via amortized inference. Our work builds on top of the VAE, and improves it by optimizing to the ranking- based metric.
Learned Metrics in Vision & Languages Recent research in computer vision and natural language processing has generated excellent results, using learned instead of hand-crafted metrics. Among the rich literature of generating realistic images via generative adversarial networks (GANs) (Goodfellow al.]]201 2014 Radford et al. 2015 Karras et al. (2018), our work is most similar to Larsen et al. (2016), where the VAE objective (Kingma & Welling 2013) is augmented with the learned representations in
the GAN discriminator (Goodfellow et al. 2014) to better measure image similarities. For language
generation, the discrepancy between werd-level MLE training and sequence-level semantic evaluation
has been alleviated with GANs RL. techniques (Bahdanau et al. 2016 Ren et al. 2017 Lin et al.
5

---

Published as a conference paper at ICLR 2020
ML-20M
Nettix
MSD
0.403
0.282
0.000
Valcatin NDCG 700
RaC1
Valdeiion NWGleiionNDCCGg) NDCGB1100
RaC T
Valdweloon NDCO@-100
RaCT
0.43
VAE
0.3004
0.315
VAE
0.432
M Mwmm
0.810
0.593
0:03
0.508
0.88
50
100
125
180
175
202
40
100
90
Epach of actor
# Epach of
# Epach of actor
(a) ML-20M dataset
(b) Netflix dataset
(c) MSD dataset
Figure 3: Performance improvement (NDCG@ 100) with RaCT over the VAE baseline.
2017). The RL approach directly optimizes the metric used at test time, and has shown improvement on various applications, including dialogue (Li et al. (016), image captioning (Rennie et al.) 2017) and translations (Ranzato et al., 2013). Despite the significant successes in other domains, there has been little if any research reported for directly learning the metrics with deep neural networks for collaborative filtering. Our work fills the gap, and we hope it inspires more research in this direction.
Learning to Rank (I.2R) The idea of L2R has existed for two decades in the information-retrien community. The goal is maximize a given ranking- anking-based evaluation metric (Liu et al. 2009 Li] 2014), generally through optimizing objective relaxations (Weimer et al.] 2008). Many L2k methods used in recommendation, such as the popular pairwise L2R methods BPR (Rendle et al., 2009)
and WARP (Weston et al. (01T), are trained by optimizing a pairwise classification function that penalizes mis-ranked pairs of items. Through negative sampling (Hu et al.) 2008). these methods can scale to extremely sigh-dimensional output spaces. However, it is computationally expensive to compute low-variance updates to model when the number of items is large.
An alternative to the pairwise approach is lisnwise loss functions, which minimize loss calculated from a user's entire interaction history. By considering the entire interaction history these methods can more closely model ranking, and generally perform better than their pairwise counterparts (Xia) let al. 2008). Furthermore, compared to methods which calculate relative ranking for each pair (Weston et al. 20IT), the per-user amortization of rank-calculation can be computed more efficiently. NLL is an example of a listwise loss function, as it is calculated over user's 's entire interaction history. Interestingly. NLL is also used as the loss function for ListNet (Cao al., (007), a classic listwise 1,2R method designed to probabilistically maximize Top-1 Recall. The VAE framework under NLL can be seen as a principled extension of this method Top-N collaborative filtering. Our ranking- critical training further extends this methodology by explicitly calculating the relationship between a differentiable listwise loss function and the desired ranking- based evaluation function.
EXPERIMENTS
Experimental Settings We implemented our algorithm in TensorFlow. The source code to re- produce the experimental results and plots is included as Supplementary Material. We conduct experiments three publicly available large-scale datasets, which represent different item recom- mendation scenarios, including user-movie ratings and user-song play counts. This is the same set of user-item consumption datasets used in Liang et al. (2018), and we keep the same pre-processing steps for fair comparison. The statistics of the datasets, evaluation protocols and hyper-parameters are summarized in the Supplement. VAE (Liang al. 2018 is used as the baseline, which plays the role of our actor pre- vre-trannnng. The NCDG@ 100 ranking metric is used as the critic's 's target in training.
Baseline Methods We use manking-criticat training to improve the three MLE-based methods described in Section 2.1: VAE, DAE, and MF. We also adapt traditional L2R methods as the actors in our framework, where the L.2R loss is used to replace LE in equation 5 to construct the feature. We consider WARP and LambdaRank, two pairwise loss functions designed for optimizing NDCG, for these experiments. We also compare our approaches with four representative baseline methods in collaborative filtering. CDAE (Wu et al. 2016) is a grrongl--perfoming neural-network based method, weighted MF (Hu et al. 2008) ) is a linear latent-factor model, and SLIM (Ning & Karypis similarity models. We additionally compare with
2011) and EASE (teck 2019) are item-to-itee Bayesian Pairwise Ranking (Rendle et al. 2009), but as this method did not yield competitive performance on these datasets, we omit the results.

---

Published as a conference paper at ICLR 2020
Table Comparison on three large datasets. The best testing set performance is reported. The results below the line are from Liang et al. 2018), and VAE shows the VAE results based on runs. Blue indicates improvement over the VAE baseline, and bold indicates overall best.
Datasel
ML-20M
Netflix
MSD
Metric
Rw
Rw50
NDCG@ 100
KGP20
R@:3
NDCG 100
RC
Rw50
NDCG 1O0
RaCT
0.403
0.543
0.434
0.357
0.450
0.392
0.268
0.364
0.319
VAET
0.396
0.536
0.426
0.350
0.443
0.385
0.260
0.356
0.310
WARP
0.310
0.448
0.348
0.273
0.360
0.312
0.162
0.253
0.210
LambdaRank
0.395
0.534
0.427
0.352
0.441
0.386
0.259
0.355
0.308
EASE
0.391
0.521
0.420
0.362
0,445
0,393
0,333
0.428
0,389
VAE
0.395
0.537
0.426
0.351
0.444
0,386
0.266
0,364
0.316
CDAE
0.391
0.523
0,418
0.343
0.428
0.376
0.188
0.283
0.237
WMF
0.360
0.498
0,386
0.316
0,404
0.351
0.211
0.312
0.257
SLIM
0.370
0.495
0.401
0.347
0.428
0.379
5.1 OVERALL PERFORMANCE OF RACT
Improvement over VAE In Figure 3 we show the learning curves of RaCT and VAE on the validation set. The VAE converges training stage, e.g., 150 epochs on ML-20 dataset, after which the VAE's performance is not improving. By contrast, when the RaCT is plugged in, the performance shows a significant immediate boost. For the amount of improvement gain, RaCT takes only half the number of epochs that VAE takes in the end of actor pre- pre-traini For example, RaCT takes 50 epochs (from 150 to 200) to achieve an improvement of 0.44-0.43 0.01, while VAE takes 100 epochs (from 50 to 150) to achieve improvement of 0.43-0.424 0.006.
plateau by the time that the RaCT finishes its actor pre-
Training/Evaluatte Correlation We visualize scatter plots between learning objectives and evaluation metric for all users on ML-20M dataset in Figure More details and enlarged visualization shown in Figure 6 of the Supplement.
r- 0,1558
0=0.80 0.8500
True NDCG@100
Trun NDCG@100
The Pearson's 's correlation is computed.
&
NLL exhibits low correlation with the target NDCG (r is close to zero), while
the learned metric in RaCT shows much strongly
Training NLL (Mulinommial)
Training Oritic Outpul
higher positive correlation. indicates RaCT optimizes a more direct objective than an MLE approach. Further, NLL should in theory have negative correlation with the target NDCG, as we wish that minimizing NLL can maximize NDCG. However, in practice it yields positive correlation. We hypothesize that this is because the number of interactions for each user may dominate the NLL values. That partially motivates us to consider the number of user interactions features.
(a) MLE
(b) RaCT
Figure 4: Correlation between the learning objectives
(MLE or RaCT) and evaluation metrics on training.
Comparison with traditional L2R methods As examples of traditional L2R methods, we compare to our method using WARP (Weston al.] 2011) and LambdaRank (Burges et al. (007) as the ranking- critical objectives. We use implementations of both methods designed specifically to maximize roughly 2 and 10 times more computationally
NDCG. We observe that WARP and LambdaRank expensive than RaCT per epoch, respectively. Table [ shows the results of RaCT, WARP and LambdaRank, using the same amount of wall-clock training time. We observe the trends that WARP degrades performance, and LambdaRank provides performance roughly equal to VAE. WARP's s poor performance perhaps due to poor approximation of the ranking when the number of items is large.
Comparison with existing methods In Table competing methods in terms of three evaluation metries: NDCG@ 100, Recall @0, and Recall 50. We use the published code of Liang et al. (2018), and reproduce the VAE as our actor pre-training. We further use their reported values for the classic collaborative filtering methods CDAE, WMF, and SLIM. Our reproduced VAE results are very close to Liang et al. (2018) on the ML-20M and
we report our RaCT performance, and compare with
https://gi
/

---

Published as a conference paper at ICLR 2020
0440
Actor
Before
After
Gain
04435
[Ce. 17fl.. (P6.
VAE
0.4258
0.4339
8.09
Ice. [N61l
VAE (Gaussian)
0.4202
0.4224
2.21
[Ge]
VAE (3 = 0)
0.4203
0.4255
5.17
VAE (Linear)
0.4156
0.4162
0.53
0,43
M M w mww.
DAE (Liang et al. 2018)
0.4205
0.4214
0.87
0433
MF (Liang et al. 2018
0.4159
0.4172
1.37
0421
WARP
0.3123
0.3439
31.63
120
200
Epoch oficcoo
Table 2: Performance gain (x (x10-9) for various actors.
Figure 5: Ablation study on features.
Netflix datasets, but slightly lower on the MSD dataset. The RaCT is built on top of our VAE runs, and consistently improves its baseline actor for all the evaluation metrics and datasets, as seen by comparing the rows RaCT and VAE The proposed RaCT also significantly outperforms competing LVMs, including VAE, CDAE, and WMF.
When comparing to EASE (Steck) 2019), our method performs substantially better for ML.-20M, comparably for Netflix, and is substantially outperformed for MSD. We observe similar trend when comparing SLIM (an item-to-item similarity method) and CDAE (a latent variable method). As SLIM and EASE rely recreating the Gram-matrix G x their performance should improve with the the number of users (Steck 2019). However this performance may come at a computational cost, as inference requires multiplication with unfactored M M matrix. EASE requires computing dense item-to-item similarity matrix, making its inference on MSD roughly 30 times more expensive than for VAE RaCT. A practitioner's choice between these methods should be informed by the specifics of the dataset as well as demands of the system.
In the Supplement, we study the generalization of RaCT trained with different ranking- metiinss in Section F.1 and break down the performance improvement with different cut-off values of NDCG in Section and with different number of interactions of X in Section E.4
5.2 WHAT ACTOR CAN BE IMPROVED BY RACT?
In RL, the choice of policy plays crucial role in the agent's 's performance. Similarly, we would like to study how different actor designs impact RaCT performance. Table Zhole/ssswwss the performance of various policies before and after applying RaCT. The results NDCG@ @ 100 are reported. The VAE, DAE and MF models follow the setup in Liang et al. (2018).
We modify one component of the VAE time, and check the change of performance improvement (1) VAE (Gaussian): we change likelihood form from multinomial to Gaussian, and observe smaller performance improvement. This shows the importance of having closer proxy of ranking-based loss. (2) VAE (3 O): we remove the KL regularization by setting 0, and replace the posterior sampling with delta distribution. We see a marginally smaller performance improvement. This compares stochastic and deterministic policy. The stochastic policy (i.e., posterior sampling) provides higher exploration ability for the actor, allowing more diverse samples generated for the critic's 's training. This is essential for better critic learning. (3) VAE (Linear): we limit the expressive ability of the actor by using a linear encoder and decoder. This significantly degrades performance, and the RaCT cannot help much in this case. RaCT shows improvements for all MLE-based methods, including DAE and MF from Liang et al. (2018). also shows significant improvement over WARP. Please see detailed discussion in Section F.5 of the Supplement.
that RaCT can provide.
5.3 ABLATION STUDY ON SEATURE-BASER CRITIC
In Figure 5] we investigate the importance of the features we designed in equation 5 using results from the ML-20M dataset. The full feature vector consists of three elements: h = [E |Hol, [ l]. LE is mandatory, because it links the actor to the critic; removing it would break the back bropppronnn to train the actor. We carefully remove /7/o or [741 from at each time, and observe that it leads to performance degradation. In particular, removing (Hul results in severe over-fitting issue. When both counts are removed, we observe an immediate performance drop, as depicted by the orange curve. Overall, the results indicate that all three features necessary our performance improvement.
8

---

Published as a conference paper at ICLR 2020
6 CONCLUSION & DISCUSSION
We have proposed an actor-critic framework for collaborative filtering on implicit data. The critic learns to approximate the ranking scores, which in turn improves the traditional MLE-based nonlinear LVMs with the learned ranking-critical objectives. To make it practical and efficient, we introduce few techniques: a reature-based critic reduce the number of learnable parameters, posterior sampling as exploration for better critic estimates, and pre-training of actor and critic for fast convergence. The experimental results on three large-scale datasets demonstrate the actor-critic's 's ability to significantly improve the results of a variety of latent-variable models, and achieve better or comparable performance to strong baseline methods.
Though RaCT improves VAEs, it does not start from the best performing actor model. The very recent work by Dacrema et al. (2019) conducts a systematic analysis of algorithmic proposals for top-N recommendation tasks. There are other simple and efficient methods that perform better than VAEs, such as pure RecWalk (Nikolakopoulos & Karypis, 2019) and Personalized Diffusions (Nikolakopoulos et al., 2019a). One interesting future research direction is to explore Garning-to-ranf. techniques for them.
models (Cremonesi et al.] 2010 Nikolakopoulos et al. (019b))
REFERENCES
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pincau, Aaron Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. arXiv preprint urXiv:1607.07066 2016.
Thierry Bertin-Mahieux, Daniel PW Ellis, Brian Whitman, and Paul Lamere. The million song dataset. In ISMIR, 2011.
Christopher Bishop. Pattern recognition and machine learning. Pattern Recognition and Machine Learning. 2006.
David Blei, Alp Kucukelbir, and Jon McAuliffe. Variational inference: A review for statisticians, Journal of the American Statistical Association, 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1009, 2018.
Christopher J Burges, Robert Ragno, and Quoe V Le. Learning to rank with nonsmooth cost functions. In NIPS, 2007.
Zhe Cao, Tao Qin, Tie- Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: From pairwise approach to listwise approach. In International conference machine learning. 2007.
Jingyuan Chen, Hanwang Zhang, Xiangnan He, Liqiang Nic, Wei Liu, and Tat-Seng Chua. Attentive collaborative filtering: Multimedia recommendation with and component-lewe level attention. In Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval, pp. 335-53 344. ACM, 2017.
Paolo Cremonesi, Yehuda Koren, and Roberto Turrin. Performance recommender algorithms on top-N recommendation tasks. In ACM Conference on Recommender systems, 2010.
Maurizio Ferrari Dacrema, Paolo Cremonesi, Dietmar Jannach. Are we really making much progress?
a worrying analysis of recent neural recommendation approaches. In ACM Conference on Recommender Systems, 2019.
Chelsea Finn, Paul Christiano, Picter Abbeol, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint 161::6611.0882 2016.
Kostadin Georgiev and Preslav Nakov. A non-iid framework for collaborative filtering with restricted boltzmann machines. In International conference machine learning, pp. 1148 1156, 2013.
Samuel Gershman and Noah Goodman. Amortized inference in probabilistic reasoning. In Proceedings of the Annual Meeting of the Cognitive Science Society, 2014.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, pp. 2672-2680, 2014.

---

Published as a conference paper at ICLR 2020
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT press Cambridge, 2016,
Xiangnan He, Lizi Liao, Hanwang Zhang. Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Proceedings of the 26th International Conference on World Wide Web, pp. 173-182. International World Wide Web Conferences Steering Committee, 2017.
Xiangnan He, Xiaoyu Du, Xiang Wang, Feng Tian, Jinhui Tang, and Tat-Seng Chua. Outer product-based neural collaborative filtering. IJCAI, 2018a.
Xiangnan He, Zhankui He, Xiaoyu Du, and Tat-Seng Chua. Adversarial personalized ranking for recommenda- tion. In The 41st International ACM SIGIR Conference on Research & Development Information Retrieval, pp. 355-364. ACM, 2018b.
Mining, 2008. ICDM '8. Eighth IEEE International Conference on, pp. 263-272. IEEE, 2008. Yifan Hu, Yehuda Koren, and Chris Volinsky. Collaborative filtering for implicit feedback datasets. Data
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ICML, 2015.
Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 2002.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. ICIR, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2014.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. ICLR, 2013.
Maciej Kula. Metadata embeddings for user and item cold-start recommendations. arXiv preprint arXiv: 2015.
Anders Boesen Lindbo Larsen, Stren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using a learned similarity metric. In International Conference on Muchine Learning, pp. 1558-1566. 2016.
Chunyuan Li, Hao Liu, Changyou Chen, Yuchen Pu, Liqun Chen, Ricardo Henao, and Lawrence Carin. Alice: Towards understanding adversarial learning for joint distribution matching. In NIPS, pp. 5495-5503, 2017.
Hang Li. Learning to rank for information retrieval and natural language processing. Synthesis Lectures on Human Language Technologies, ((3):1-121) 2014.
Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky, Deep reinforcement lcarning for dialogue generation. arXiv preprint or 1606.1600..0554 2016.
Dawen Liang, Minshu Zhan, and Daniel PW Ellis. Content-aware collaborative music recommendation using pre-trained neural networks. In ISMIR, 2015.
Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. Variational autoencoders for collabora- tive filtering. WWW, 2018.
Kevin Lin. Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-Ting Sun. Adversarial ranking for language generation. In NIPS, pp. 3155-3165, 2017.
Tie-
Yan Tie-Yan (13:225-3311 2009.
Liu et al. Learning to rank for information retrieval. Foundations and Trends(e in Information Retrieval,
Mchdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint ArXiv:1411.11 1784, 2014.
Takeru Miyato and Masanori Koyama. egans with projection discriminator. ICLR, 2018.
Andriy Mnih and Ruslan R Salakhutdinov. Probabilistic matrix factorization. In NIPS, 2008
Athanasios N Nikolakopoulos and George Karypis. ReeWalk:
Nearly uncoupled random walks for top-N recommendation. In ACM International Conference on Web Search and Data Mining, 2019.
Athanasios N Nikolakopoulos, Dimitris Berberidis, George Karypis, and Georgios B Giannakis. Personalized diffusions for top-N recommendation. In ACM Conference on Recommender Systems, 2019a.
10

---

Published as a conference paper at ICLR 2020
Athanasios N Nikolakopoulos, Vassilis Kalantzis, Efstratios Gallopoulos, and John D Carofalakis. EigenRec: generalizing pureSVD for effective and efficient top-N recommendations. Knowledge and Information Systems, 2019b.
Xia Ning and George Karypis. Slim: Sparse linear methods for top-n recommender systems. In 201 11th IEEE International Conference on Data Mining, pp. 497-506. IEEE, 2011.
Arkadiusz Paterek. Improving regularized singular value decomposition for collaborative filtering. In Proceedings of KDD cup and workshop, volume 2007, pp. 5-8, 2007.
David Pfau and Oriol Vinyals. Connecting generative adversarial networks and actor-critic methods, arXiv preprint 1rXi::6610.0145, 2016.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: (0155111.0444 1.04344, 2015.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeicch Zaremba. Sequence level training with recurrent neural networks. ICLR, 2015.
Zhou Ren, Xiaoyu Wang. Ning Zhang, Xutao Lv, and Li-Jia Li. Deep reinforcement learning- based image captioning with embedding reward. CVPR, 2017.
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme Bpr: Bayesian personalized ranking from implicit feedback. In Proceedings of the twenty-fifth conference uncertainty in artificial intelligence, pp. 452-461. AUAI Press, 2009.
Steven Rennie, Etienne Marcheret, Youssef Mrouch, Jarret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In CVPR, volume 1. pp. 3, 2017.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. ICMI., 2014.
Francesco Ricci, Lior Rokach, and Bracha Shapira. Recommender systems: introduction and challenges. In Recommender systems handbook, pp. 1-34. Springer, 2015.
Ruslan Salakhutdinov, Andriy Mnih, and Geoffrey Hinton. Restricted boltzmann machines for collaborative filtering. In Proceedings of the 24th international conference on Machine learning, pp. 791-798. ACM, 2007.
Suvash Sedhain, Aditya Krishna Menon, Scott Sanner, and Lexing Xic. Autoree: Autoencoders meet collabora- tive filtering. In Proceedings of the 24th International Conference World Wide Web, pp. 111-112, ACM, 2015.
Suvash Sedhain, Aditya Krishna Menon. Scott Sanner, and Darius Braziunas. the effectiveness linear models for one-class collaborative filtering. In AAAI, 2016.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. 2014.
Harald Steck. Embarrassingl shallow autoencoders for sparse data. 2019.
Xiaoyuan Su and Taghi M Khoshgoftaar. A survey of collahorative filtering techniques. Advances artificial intelligence, 2009, 2009.
Richard S Sutton, Andrew G Barto, Francis Bach, et al. Reinforcement learning: An introduction. MIT press, 1998.
Markus Weimer, Alexandros Karatzoglou, Quoc V Le, and Alex J Smola. Cofi rank-maxing margin matrix factorization for collaborative ranking. In NIPS, pp. 1593-1600, 2008.
Jason Weston, Samy Bengio, and Nicolas Usunier. Wsabie: Scaling up large vocabulary image annotation. In IJCAI, volume 11, pp. 2764-2770, 2011.
Jason Weston, Hector Yee, and Ron J Weiss. Learning to rank recommendations with the k-order statistic loss. In Proceedings of the 7th ACM conference on Recommender systems, 345-248. ACM, 2013.
Ronald I Williams. Simple statistical frllient- algorithms for connectionist reinforcement learning. Machine learning, 2(3-4):29-256, 1992.
11

---

Published as a conference paper at ICLR 2020
Yao Wu, Christopher DuBois, Alice X Zheng, and Martin Ester. Collaborative denoising auto-encoders for top-n recommender systems. In Proceedings of the Ninth ACM International Conference on Web Search and Data Mining, 2016.
Xia, Tie- Yan Liu. Jue Wang, Wensheng Zhang, and Hang Listwise approach to learning to rank theory and algorithm. 2008.
Hong-Jian ManngJiannue Xue, Xinyu Dai, Jianbing Zhang, Shujian Huang, and Jiajun Chen. Deep matrix factorization models for recommender systems. In IJCAI, pp. 1203--2099 2017.
Shuai Zhang, Lina Yao. and Aixin Sun. Deep learning based recommender system: A survey new perspectives. arXiv preprint wwt.iv::07..00000
2017.
Zheng, Bangsheng Tang. Wenkui Ding, and Hanning Zhou, neural autoregressive approach to collaborative filtering, arXiv preprint 1rX.v.160..09477.0 2016.
12

---

Published as a conference paper at ICLR 2020
Summary of contributions: Sam and Chunyuan conceptualized Beaning-to-rank for VAEs. Sam created and implemented the current algorithm, made the model work, and ran all experiments. Chunyuan set up the experiments, led and completed the manuscript writing. Lawrence edited every version of the manuscript. Jianfeng proofread an early version of the manuscript.
A TESTING STAGE OF VAES FOR COLLABORATIVE FILTERING
We focus on studying the performance various models under strong generalization (Ling ct al. 2013) as in (Liang al. 2018). All users split into vaaiiinn/vliiaiiio/eesttttt sets. The models learned using the entire interaction history the users the training set. To evaluate, we use a part of the interaction history from held-out (validation and test) users infer the user-level representations from the model, and compute quality metrics by quantifying how well the model ranks the rest of the unseen interaction history from the held-out users. Specifically, for held-out user with the full history æ, we take 2 = @ b offline using the randomly generated mask b. 2% is then frozen as the testing input, and is fed into various trained models during the evaluation stage to get the prediction it. The recovered interaction zt न @ .r) for the masked seen part is then evaluated by ranking- -ased metries.
B BACKGROUND ON TRADITIONAL LEARNING-TO-RANT METHODS
Formally, the Bayesian Personalized Ranking (BPR) (Rendle et 2009 loss for the n-th user
CBPR Σ Σ
(TTy - Tni),
(8)
SEK+ JEK+JEK-
where (-) is the sigmoid function, K.+ denotes the of items that the user has interacted with before, and K_ denotes the complement item set.
The Weighted Approximate-
Pairwise (WARP) model (Weston al. 2011) has been shown to perform better than BPR for implicit feedback (Kula 2015):
CWARP
Σ nIcK JEK-
Σ
w(r,) + Tay kni)),
where w(-) is a weighting function for different ranks, and r; is the rank for the i-th item for n-th user. common choice weighting function ur()) for optimizing NDCG is w(rr E 0s with a; = 1/1. WARP improves BPR by the weights w-() and the margin between positive and negative items.
C PSEUDO-CODE FOR RACT
We summarize the full training procedure of RaCT in Algorithm 1.
D INTERPRETATION WITH GANS
We can view our actor-critic approach in equation and equation 7 from the perspective of Gen- erative Adversarial Networks (GANs). GANs constitute framework construct generator G that can mimic a target distribution, and have achieved significant success in generating realistic images (Goodfellow et al. 2014; Radford al., (2055) Karras
2018 Brock et al., 2018). The
most distinctive feature of GANs is the discriminator D that evaluates the divergence between the current generator distribution and the target distribution (Goodfellow et al., 2014 Li al. (017). The GAN Icarning procedure performs iterative training between the discriminator and generator, with the discriminator acting as an increasingly meticulous critic to refine the generator. In our work, the actor be interpreted the generator, while the critic be viewed as the discriminator.
Note that GANs and actor-critic critic models learn the metric functions (Finn et al., 2016), and it has been shown in Pfau & Vinyals (2016) that GANs can be viewed as actor-critic in an environment where the actor cannot affect the reward. This is exactly our setup. One key difference that we know the Oracle metric, and the critic is trained mimic the Oracle's 's behaviour.
13

---

Published as a conference paper at ICLR 2020
Algorithm 1: Our full ranking-critical training with stochastic optimization.
Input:
Interaction matrix X: Actor parameters (encoder & and decoder 0), Critic parameters vb. Initialize: Randomly initialize weights Φ, A and ab /+ stage 1: Pretrain the actor via MLE
while not converged do do Sample batch of users Lt;
0Cs in equation Update {0, )) with gradient aCe and uΦ 00
&end
Pretrain the critic via MSE while not converged do do Sample batch of users UI; Construct features h in equation and target from the Oracle;
/+ Stage 2:
10
Update tb with gradient &Ce in equation
12 end
13
/+ Stage 3: Alternative training of actor and critic
14 for 1. 2. 1......, T do
15
Sample batch of users U1;
16
/+ Actor step
Update {6, Φ) with gradient
ac and aca in equation 7 00
17
18
Critic step
*/
19
Construct features h in equation and target y from the Oracle;
Update zp with gradient aCc in equation
20
8xp
end
Conditioned on interaction history corrupted from æ. the actor predicts the distribution parameter
over items, which further constructs the likelihood p(æ rT). ). We use q designate the data empirical distribution, the target conditional is q( T). It can be formulated as the standard adversarial loss for the conditional GAN (Mirza & Osindero 2014). It has been shown that the optimal critic (Goodfellow
al. 2014 Li al. 2017 for conditional GAN can be represented the log likelihood ratio
( T p(a: TT)
R" (TT, x) log
(10)
In the collaborative filtering setup, we often make the assumptions that p(z| are simple distributions, such multinomial in VAEs (Liang et al. 2018) and Gaussian in ME. This simplification allows the parameterization of critic following the following form (Miyato & Koyama 2018):
Rd R" x x Vv(TT) + C
(11)
where æ the target, v(T) a layer of the critic with input æ,, and V and are the parameters to learn. Most notably, this formulation introduces the prediction information via inner product, opposed to concatenation. The form equation i indeed the form we proposed for NLL feature æ log n with V = I and v(-) log(.). C includes the normalizer for the prediction probability (Miyato & Koyama 2018), which is related to the count features in equation
E EXPERIMENTAL SETUP
E. DATASETS
We conduct experiments on three publicly available datasets. Table summarizes the statistics of the data. These three ten-mil mom-miillonszze datasets represent different item recommendation scenarios, including user-movie ratings and user-song play counts. This is the same set of medium- to large scale user-item consumption datasets used in Liang et al. (2018), and we keep the same pre-processing steps for fair comparison.
14

---

Published as a conference paper at ICLR 2020
Table 3: Summary Statistics of datasets after all pre-processing steps. Interactions# is the number of non-zero entries. Sparsity% refers to the percentage of zero entries in the user-item interaction matrix X. Items is the number of total items. #O# is the number of validation/test users held out of the total number of users in the 5th column #s
Dataset
Interaction#
Sparsity%
Item#
User#
HO#
ML-20M
10.0M
99.64%
20,108
136,677
10K
Netflix
56.9M
99.31% %
17,769
463,435
40K
MSD
33.6M
99.86%
41,140
571,355
50K
1. MovieL.ens220M (MI-20M)::
This is the user-movie rating data collected from movie recommendation service The data is binarized by keeping ratings of four or higher and setting other entries as unobserved. Only users who have watched at least five movies considered.
2. Netflix Prize (Netflix): This the user-movie rating data from the Netflix Prize Similarly to ML-20M, the data is binarized by keeping ratings of four higher, and only users who have watched at least five movies are kept.
3. Million Song Dataset (MSD): This is the user-song play count data from the Million Song Dataset (Bertin-Mahieur Mahieux et al. 2011). We binarize play counts, and keep users who have listened to least 20 songs as well as songs that are listened to by at least 200 users.
E.2 EVALUATION PROTOCOL
In the testing stage, we get the predicted ranking by sorting the multinomial probability Tp. For each user, we compare the predicted ranking of the held-out items with their true ranking. Two ranking- based metrics considered, Recall @ R and the truncated NDCG (NDCG@R) R), where R the cut-off ayper-parameter. While Recall R considers all items ranked within the first R to be equally important, NDCG@R uses monotonically increasing discount emphasize the importance of higher ranks versus lower ones.
Formally, define m(r) as the item rank r, and Ho the held-out unobserved items that user will interact.
R 28lm()) -
DCG@R
(12)
log(r +
By dividing DCG@R by its best possible value, obtain NDCG@R in [0,
8(m(r) E Hol
Recall@R
(13)
min(I. (Hol)
The denominator normalizes Recall R in [0, 1], with maximum value corresponding to the case that all relevant items are ranked in the top R positions.
E.3 EXPERIMENT HYPER-PARAMETER
We set typer-parameteer by following Liang al. (2018) for comparisons. For VAE, the dimension of the latent representation is 200. When KL regularization removed (B = 0), i.e., for DAE and MI, we instead apply & regularization (0.01) on weights to prevent overfitting. Adam optimizer (Kingma & Ba 2014) is used, with batch size of [ 500 users. For ML-20M, the actor is pre-trained for 150 epochs, and alternative training for 50 epochs. On the other two datasets, the actor pre-trained for 75 epochs, and alternative training for 25 epochs. The critic is pre-trained for 50 epochs for all three datasets. The alternative training has equal update frequency for actor and critic. This schedule ensures that the have the same total number of actor training epochs as Liang et al. (2018): 200 epochs for ML-20M, 100 epochs for the other two datasets.
movi 1ens/20mm
https://uw..etffii
15

---

Published as a conference paper at ICLR 2020
Table 4: Network architectures. The arrow indicates the flow between two layers. For each layer, we show the number of units on top of its following activation function. BN indicates Batch Normalization.
Networks
Architectures
M
600
200
Encoder
Actor
Linear
Tanh Linear& Exp
200
600
M
Decoder
Linear
Tanh
Sof tmax
3
100
100
10
Critic
BN
ReLU
ReLU
ReLU Sigmoid
Table 5: Summary of training schedule aaper-parameters. Smax indicates the maximum value of B. In the actor pre-training stage, the number of epochs used for increasing and fixing 3 are shown in row 3 and 4, respectively.
Dataset
ML-20M
Netflix
MSD
fmax
0.2
0.2
0.1
# epochs for annealing
100
75
75
# epochs for fixing
50
0
0
# epochs for actor pre-training
150
75
75
# epochs for critic pre-training
50
50
50
# epochs for alternative training
50
25
25
Table Performance trained with different metrics. (ML-20M)
Training
Testing
Recall @@2
@ecall
NDCG@ 100
RaCT (Recall@ 100)
0.40316
0.54317
0.43392
RaCT (NDCG@ 100)
0.40269
0.54304
0.43395
VAE
0.39623
0.53632
0.42586
fully-connected (FC) architecture is used for all networks, as detailed in Table Please refer to Goodfellow al. (2016) for the activation functions. Batch Normalization (loffe & Szegedy 2015) is used to normalize the input features, because the magnitude of the inputs (NLL) change as training progresses. The encoder outputs the mean and variance of the varational distribution; the variance implemented via an exponential function.
F ADDITIONAL EXPERIMENTAL RESULTS
F.1
GENERALIZATION ACROSS RANKING METRICS
To study the generalization ability of RaCT, we consider training the critic against Recall @ 100, in addition NDCG@ 100. The only difference that Recall treats each item equally important, while NDCG treats the higher ranking items as more important. The results are shown in Table 6 Indeed, the RaCT gets slightly better testing Recall values when trained against the Recall metric, and the reverse holds for NDCG. More importantly, RaCT allows generalization across different ranking metrics: all testing metric values are significantly improved when trained against either Recall or NDCG.
Following Liang et al. (2018), we compare with NCF on two small datasets, ML- (6,040 users, 3,704 items) and Pinterest (55,18 187, users, 9,916 items). This is because the prediction stage of NCF is slow, due to a lack of amortized inference as in VAE. We use their publicly available datasets and metrics for fair comparison. The results are evaluated with small cut-off value R, only study the highly ranked items: NDCG@ 10 and Recall 10. The performance are compared in Table 7 Our observation that DAE performs better than VAE on these two datasets is consistent with Liang
16

---

Published as a conference paper at ICLR 2020
r=
0.1556
r= 0.8506
1.0
1.0
0.8
True NDCG@100 100
0.6
True NDCG@10
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
2000
6000
0000
8000
10000
0.0
02
0.4
1.0
Training NLL (Multinormial)
Training Critic Output
(a) Training NLL
(b) Training NDCG
0.0499
0.4881
True NDCG@10)
True NDCG@100
2000
4000
6000
8000
0.0
0.2
0.4
0.6
0.8
1.0
Testing NLL (Mullinormial)
Testing Critic Output
(c) Testing NLL
(d) Testing NDCG
Figure 6: Correlation between the learning objectives (NLL or RaCT) and evaluation metrics NDCG.
Table 7: Comparison between our RaCT with NCF on two small datasets. NCF results are from Liang
al. (2018).
Dataset
Metric
NCF
DAE
RaCT
VAE
RaCT
Recall 10
0.705
0.722
0.722
0.704
0.706
ML-IM
NDCG @ 10
0.426
0.446
0.446
0.433
0.434
Recall @ 10
0.872
0.886
0.887
0.873
0.878
Pinterest
NDCG@ @ 10
0.551
0.580
0.581
0.564
0.568
et al. (2018). In general, RaCT shows higher improvement when a larger dataset (Pinterest), or a stochastic actor (VAE) is considered. This is because the sizes of the two datasets relatively small, the critic can be better trained when more samples are observed. On the larger Pinterest dataset, the auto-encoder variants perform better than NCF by big margin, and RaCT further boosts the performance.
F.2 CORRELATION BETWEEN TRAINING METRICS
Figure 6 explores the relationship between NLL, the learned RaCT metric, and NDCG, We ensure that the best model for each method is used: the model after actor pre- trainini (Stage 1) is used for NLL plots, and the model after the actor-crite alternative training (Stage 3) is used for RaCT plots. The bottom row of plots displays the output of these models on the testing data. This demonstrates that RaCT's connection to the ranking metric generalizes to unseen data.
17

---

Published as a conference paper at ICLR 2020
0.45
200
NDCG@R
100
0.40
50
20
0.35
5
20
50
100
200
R: Cut-off value in NDCG@R
Figure 7: The improvement various cut-off value R in evaluation. Given specific R, the dashed line shows the VAE, and square dot shows the RaCT.
1.0
0.45
RaCT VAE
NDCC@10 100
NDCG@100
0.6
0.4
0.42
0.41
200
400
600
500
1000
200
400
600
800
1002
(a) Scatter plot
(b) NDCG
Figure Improvement breakdown over different user interactions. NDCG@ 100 and activity levels. Note only # interactions < 1000 is visualized, there is long tail (> (>1000) the distribution. (b) Comparison of the mean NDCG@ 100 values for four user groups.
(a) Scatter plot between
F.3 BREAKDOWN ANALYSIS FOR DIFFERENT CUT-OFF VALUES
NDCG@ 100 only reflects the ranking quality at the cut-off value R = 100. i.e, the top-100 ranking items. To study the ranking quality different range the predicted list, we consider large range of R, and report the corresponding NDCG values. We consider R 5, 20, 50, 100, 200, and report the results Figure The NDCG@R values are improved for various R, though the critic is trained against NDCG @ 100. This is because the NDCG metrics of different R are highly correlated, the RaCT can generalize across them.
F.4 BREAKDOWN ANALYSIS DIFFERENT NUMBER OF INTERACTIONS
In Figure 8 show performance improvement actoss increasing interactions. We use ML-20M dataset for this case study. The # interactions is the number items each user interacts with (ground- truth), indicating the user's activity level. Figure & shows the seatter plots between NDCG@ @ 100 values and various number of interactions on the testing dataset, for both VAE and our RaCT methods. RaCT generally improves VAE for large range of user interactions. We further categorize the users in four groups according to their number of interactions: <250, 250 250-500 500, 501 - 501-750, >750, and plot the mean of N 100 values for two methods Figure ( RaCT improves VAE except for users with high activity level (>775).. This is probably because the number of the most active users is small, as observed Figure 8 [a). yields lack of training data for critic learning, which potentially hurts the performance.
F.5 ON THE PERFORMANCE IMPROVEMENT OF ACTORS VIA RACT.
We also consider the two other auto-encoder variants used in Liang et al. (2018) as the actor. (I)
The DAE in Liang et al. (0188 8) chooses smaller architecture M - 600 M. which achieves better performance than the larger architecture as in our VAE (8 = 0) by prevent over-fitting. While observe the same result, it interesting to note that the VAE (8 0) shows much larger improvement gain than DAE (Liang et al., 2018) when trained with our RaCT technique, and eventually significantly outperforms the latter. This shows that the additional modeling capacity is
18

---

Published as a conference paper at ICLR 2020
necessary capture the more complex relationship in prediction, when the goal is ranking rather than MLE. (2) The MF in Liang et al. (2018) employs a Gaussian likelihood, which also gets slight improvement with the RaCT. Overall, we conclude that the RaCT method improves all the MI.E-based variants.
We also ranking-loss-based WARP the actor. For the large datasets considered in this paper, calculating the full WARP-loss for each user is impractically slow. We derive a simple approximation WARP which runs in quasilinear time the number of items. Even so, it takes around 30 minutes per epoch on ML-20M dataset, roughly 30 times slower than the VAE. WARP yields the score 0.3 which is lower than other baseline methods. This consistent with the studies in Liang al. (2018); Sedhain et al. (2016). However, when RaCT is applied, WARP gets significant improvement; in fact, the largest improvement gain of all the actors. This indicates the RaCT is a more direct and effective approach for learning to rank on large datasets.
19


Reviews:
[{"review_id": "rJe1xRyEKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a novel reinforcement learning (RL) based recommendation algorithm (i.e., Ranking-Critical Training, RaCT) for modeling users' implicit feedback, which is illustrated in Figure 2(b). Specifically, the authors apply the actor-critic RL paradigm to approximately optimize the ranking-oriented loss/objective function in collaborative filtering with implicit feedback, which is a difficult and important problem. In particular, the critic network is used to approximate the ranking-oriented metric, and the actor network is used for optimizing this metric. Moreover, for efficiency, the authors also propose a feature-based critic to replace the original one.\n\nEmpirical studies on three large datasets, i.e., ML20M, Netflix and MSD, show very promising results in comparison with the state-of-the-art methods on the studies problem. The results are convincing.\n\nThe paper is well presented, including clear background and good illustration, etc.\n\nThe proposed model is believed to be interested by the community of both recommender systems and reinforcement learning. I thus recommend acceptance.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "r1edoAD6tS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work presents a two-level architecture for building an efficient ranking solution for standard collaborative filtering task in recommender systems. The core of this solution is the actor-critic approach, where the critic (motivated by ideas from RL) tries to directly approximate a ranking-based metric using as an input prediction from the actor (VAE framework) and the ground truth knowledge from an Oracle. Both actor and critic require a pre-training step, after which the critic is set to propagate adjustments to actor parameters. Experiments demonstrate the advantage of such approach over competing methods.\n\nThe work presents an interesting idea of feature-based critic for learning a better ranking. The choice of features is surprisingly effective considering how simple they are – just basic statistics. On the other hand, the motivating example with NLL vs NDCG is not convincing. Comparing to much simpler linear SOTA methods, the improvement over multinomial VAE is not impressive. This is probably due to VAE being not a real SOTA in the first place, however other possible candidates for actor were not demonstrated. Anyway, I believe that this is an important direction for research and can be considered for acceptance if the authors adjust the text according to the comments below. \n\nMain arguments\nThe text is well structured, easy to read. Key concepts are explained carefully with the appropriate level of details. Open-source repository with the source code is definitely a plus. However, there’s a number of inaccuracies, which should be addressed.\n\nIn the example demonstrating VAE's pitfalls, it is stated that prediction A is intuitively better than prediction B. I’d argue with that intuition. For me, B looks more consistent as there’s a smaller gap between predictions for positive targets. In A one of the predictions is much lower, whcih seems more like an outlier. Moreover, shouldn’t NDCG be 1 in both cases? The ordering of positive examples is the same in both cases. Please, consider addiing more explanations to clarify your intuition or replace the example with something less contradicting. \n\nThe  statement “making RaCT scalable to large-scale datasets” is arguable from the technical viewpoint. As far as I know, there’s still no good support for sparse calculations in NN frameworks. When you take user mini batch you have to convert it to a dense matrix with explicitly stored zeros. It’s not a problem for the datasets tested in the paper (with <100K items). However, consider 100M items. What will be the size of user batch that will fit GPU memory? Will it be still large enough and effective from the learning viewpoint (smaller batches are likely to reduce generalization)? As it will mostly contain zeroes (few know interactions vs ~100M unknown), efficiency of calculations with this batch will also be low comparing to operations in sparse format. In that sense the datasets used in the paper are not really large scale.\nAnother false statement: “This is much more efﬁcient than most traditional latent factor collaborative ﬁltering models …, where a time-consuming optimization procedure is typically performed to obtain the latent factor for a user who is not present in the training data”. All factorization models admit folding-in procedure, which doesn’t depend on the size of the problem (i.e., the number of users or items) and depends only on the rank of decomposition (the number of latent variables). This is among the reasons why MF is so popular in industrial applications.\n\nFurthermore, SVD-based models (I’m talking about the real SVD, not MF models like SVD++) are absolute winners here as they do not even require running any additional optimization steps. Due to the orthogonality property, the solution for folding-in has an analytical form in the case of SVD, e.g., given a low-rank item embeddings matrix V with orthonormal columns, the folding-in solution (vector of predicted item scores for a user) is simply given by an orthogonal projector\nr = VV^T p,\nfor any vector of user preferences p, be it a known user or a new one (i.e., not present in the training data). It is basically a simple encoder-decoder architecture with a single linear layer. Training SVD-based models can also very efficient and can be scaled to almost billion-size problems, e.g. https://github.com/criteo/Spark-RSVD. Moreover, one can significantly improve standard PureSVD model [Cremonesi, Koren, Turrin 2010] by a simple scaling trick as it is shown in the EigenRec model [Nikolakopoulos 2019], you can also see more details on this in the blog https://www.eigentheories.com/blog/to-svd-or-not-to-svd. I also find it interesting that the scaling trick in SVD partially intersects with the idea of importance of user behavior statistics in the critic model,  as it also depends on how many clicks a user has made.\n\nI ran my own experiments with VAE and SVD in google colab (I can provide a link to the notebook if needed). With a proper tuning the SVD-based solution gives NDCG=0.415, which is within 3% difference of the Mult-VAE’s best result on the ML-20M dataset. It takes around 200s to train the SVD model of rank 600 on the CPU in the google colab. For comparison, one epoch of VAE training takes 26s in the same google colab environment (which I believe is using Tesla K80 GPU). The same NDCG level is achieved by Mult-VAE after approximately 20 epochs, which amounts to more than 500s of training time, twice longer than SVD. Speaking about practicality, even a student will have a decent CPU nowadays, but not everyone will have Tesla K80. SVD merely requires doing `from scipy.sparse.linalg import svds` in most cases. Moreover, there are even better solutions than SVD (in terms of accuracy), still linear, not even based on MF, e.g., RecWalk [Nikolakopoulos and Karypis 2019] and even better one - Personalized Diffusions model [Nikolakopoulos, Berberidis, Karypis, Giannakis 2019] - that are shown to significantly outperform VAE models.\n\nThe general conclusion is that even though you improve VAE, you are actually starting from not the best model and there’s much more room for improvement (demonstrated by much simpler models) than you show in your work. Please consider this fact and adjust wording in your text accordingly. Especially in the statements like “VAEs signiﬁcantly outperform many state-of-the-art methods” or “the proposed RaCT also signiﬁcantly outperforms other state-of-the-art methods, including VAE, CDAE, WMF and SLIM” as it is really about outperforming strong baselines not real SOTA.\nIn addition to that, recent studies show that many modern neural networks-based approaches underperform even simple KNN-based models, see work by [Dacrema, Cremonesi, Jannach 2019] on “A Worrying Analysis of Recent Neural Recommendation Approaches”. It includes VAE, DAE and NeuCF that you reference in your paper. \n\nIn spite of that, the statement “[LVM]… may yield suboptimal performance, especially for large datasets (He et al., 2017)” is at best questionable and requires more investigation as the referenced here paper is about NeuCF. This also makes me doubt in the claims, where you talk about approximation of the real ranking measure by the critic. Examples with RecWalk and PersDiff show that even without optimizing any ranking metric you can get a decent quality in terms of NDCG. Even though I find the general explanation in the text intuitive and appealing, I believe, more rigorous research should be done before we can have something reliable to say about ranking metric approximation. I’d suggest being more careful with such statement in the text. Even in your own experiments a simple EASE model is sometimes better, even though it has nothing to do with approximating any elaborate ranking metric (it's closer to the nearest neighbours search).\n\n\nOther comments\n1)\t“performance comes at a computational cost, as inference requires multiplication with an unfactored M ×M matrix” is arguable to a certain extent. In the case of SLIM there’s a sparse matrix with zero diagonal, so once it’s computed and stored in a sparse format, it’s not really a problem to do calculations with it (the training is expensive, though).\n2)\t “which is argued to be a close proxy to ranking loss” a reference required here.\n3)\tThe two-level architectures are very popular in general. At the first level, a compact list of candidates is generated by a quick to compute algorithm like MF,  then the second-level algorithm based on decision-trees or multiarmed bandits (MAB) is trained on a much smaller output space. Comparing with such approaches (especially with MAB as they are widely used in industry, probably the second popular approach after MF) would make the work more complete.\n4)\t“Previous work on learning-to-rank has been explored this question…” something is not right here.\n5)\tUnfortunately, a lot of prior studies do not report confidence intervals, and this paper is not an exception. It makes it hard to reason about statistical significance of the results and about generalization capabilities of the approach.\n\nReferences:\nCremonesi, Paolo, Yehuda Koren, and Roberto Turrin. \"Performance of recommender algorithms on top-n recommendation tasks.\" In Proceedings of the fourth ACM conference on Recommender systems, pp. 39-46. ACM, 2010.\nNikolakopoulos, Athanasios N., Vassilis Kalantzis, Efstratios Gallopoulos, and John D. Garofalakis. \"EigenRec: generalizing PureSVD for effective and efficient top-N recommendations.\" Knowledge and Information Systems 58, no. 1 (2019): 59-81.\nNikolakopoulos, Athanasios N., and George Karypis. \"Recwalk: Nearly uncoupled random walks for top-n recommendation.\" In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pp. 150-158. ACM, 2019.\nNikolakopoulos, Athanasios N., Dimitris Berberidis, George Karypis, and Georgios B. Giannakis. \"Personalized diffusions for top-n recommendation.\" In Proceedings of the 13th ACM Conference on Recommender Systems, pp. 260-268. ACM, 2019.\nDacrema, Maurizio Ferrari, Paolo Cremonesi, and Dietmar Jannach. \"Are we really making much progress? A worrying analysis of recent neural recommendation approaches.\" In Proceedings of the 13th ACM Conference on Recommender Systems, pp. 101-109. ACM, 2019.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SyxRaUE0Yr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper builds on the line of work of developing latent variable models (LVMs) for collaborative filtering with implicit feedback (e.g. which items a user has previously clicked on).  While variational autoencoders allow convenient construction of nonlinear LVMs, they are trained by maximizing the multinomial likelihood for the items selected.  Thus the training objective is not perfectly matched to the evaluation objective, which is typically something like NDCG @N or RECALL @N, neither of which is differentiable.  This paper proposes an actor-critic RL approach to train the nonlinear LVM directly for the NDCG loss.  The idea is to create a critic model that learns to approximate NDCG, and to train the actor to optimize this approximate objective.  However, they find that they are unable to build an effective critic when taking the predictions and ground truth directly.  However, they find that with a simple set of 3-features, they can build an effective critic that gives improved or competitive performance on several metrics on 3 large-scale datasets.  When their method (RaCT) is not the best, it is beaten by EASE, which is computationally much more expensive.\n\nThis work seems to fit naturally into the line of work on nonlinear LVMs.  While porting MLE training to RL training is a standard operation these days, in this case it wasn't entirely trivial -- one couldn't just use REINFORCE and be done.  Introducing features and an actor/critic setup isn't an amazing innovation, but the method works well, has computational benefits, and has compared to many other models on 3 datasets.  The paper is very polished and nice to read so that somebody can really learn from it. I vote for accepting this paper.\n\nSome specific questions and comments:\n- In the 'Pitfals of VAEs' section, it says \"A preserves the ranking order of the target, while B does not.\"  I'm seeing that A and B give the same rankings, so I don't understand the comment.\n- I don't quite understand the explanation for why REINFORCE wouldn't be effective.  I'm more convinced by the fact that the critic failed to learn when given the full prediction set and ground truth.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers generally agreed that the application and method are interesting and relevant, and the paper should be accepted.

I would encourage the authors to carefully go through the reviewers' suggestions and address them in the final.
Decision: Accept (Poster)