RaCT: Toward Amortized Ranking-Critical Training For Collaborative Filtering 

1 INTRODUCTION . Recommender systems are an important means of improving a user ’ s web experience . Collaborative filtering is a widely-applied technique in recommender systems ( Ricci et al. , 2015 ) , in which patterns across similar users and items are leveraged to predict user preferences ( Su & Khoshgoftaar , 2009 ) . This naturally fits within the learning paradigm of latent variable models ( LVMs ) ( Bishop , 2006 ) , where latent representations capture the shared patterns . Due to their simplicity and effectiveness , LVMs are still a dominant approach . Traditional LVMs employ linear mappings of limited modeling capacity ( Paterek , 2007 ; Mnih & Salakhutdinov , 2008 ) , and a growing body of literature involves applying deep neural networks ( DNNs ) to collaborative filtering to create more expressive models ( He et al. , 2017 ; Wu et al. , 2016 ; Liang et al. , 2018 ) . Among them , variational autoencoders ( VAEs ) ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) have been proposed as non-linear extensions of LVMs ( Liang et al. , 2018 ) . Empirically , VAEs significantly outperform many competing LVMbased methods . One essential contribution to the improved performance is the use of the multinomial likelihood , which is argued by Liang et al . ( 2018 ) to be a close proxy to ranking loss . This property is desirable , because in recommender systems we generally care more about the ranking of predictions than an individual item ’ s score . Hence , prediction results are often evaluated using top-N ranking-based metrics , such as Normalized Discounted Cumulative Gain ( NDCG ) ( Järvelin & Kekäläinen , 2002 ) . The VAE is trained to maximize the likelihood of observations ; as shown below , this does not necessarily result in higher ranking-based scores . A natural question concerns whether one may directly optimize against ranking-based metrics , which are by nature non-differentiable and piecewise-constant . Previous work on learning-to-rank has been explored this question in the information-retrieval community , where relaxations/approximations of ranking loss are considered ( Weimer et al. , 2008 ; Liu et al. , 2009 ; Li , 2014 ; Weston et al. , 2013 ) . In this paper , we borrow the actor-critic idea from reinforcement learning ( RL ) ( Sutton et al. , 1998 ) to propose an efficient and scalable learning-to-rank algorithm . The critic is trained to approximate the ranking metric , while the actor is trained to optimize against this learned metric . Specifically , with the goal of making the actor-critic approach practical for recommender systems , we introduce a novel feature-based critic architecture . Instead of treating raw predictions as the critic input , and hoping the neural network will discover the metric ’ s structure from massive data , we consider ∗Corresponding author †Equal Contribution engineering sufficient statistics for efficient critic learning . Experimental results on three large-scale datasets demonstrate the actor-critic ’ s ability to significantly improve the performance of a variety of latent-variable models , and achieve better or comparable performance to strong baseline methods . 2 BACKGROUND : VAES FOR COLLABORATIVE FILTERING . Vectors are denoted as bold lower-case letters x , matrices as bold uppercase letters X , and scalars as lower-case non-bold letters x . We use ◦ for function composition , for the element-wise multiplication , and | · | for cardinality of a set . δ ( · ) is the indicator function . We use n ∈ { 1 , . . . , N } to index users , and m ∈ { 1 , . . . , M } to index items . The user-item interaction matrix X ∈ { 0 , 1 } N×M collected from the users ’ implicit feedback is defined as : xnm { 1 , if interaction of user n with item m is observed ; 0 , otherwise . ( 1 ) Note that xnm = 0 does not necessarily mean user n dislikes item m ; they may simply be unaware of the item . Further , xnm = 1 is not equivalent to saying user n likes item m , but that there is at least interest . VAE model VAEs have been investigated for collaborative filtering ( Liang et al. , 2018 ) , where this principled Bayesian approach is shown to achieve strong performance on large-scale datasets . Given the user ’ s interaction history x = [ x1 , ... , xM ] > ∈ { 0 , 1 } M , our goal is to predict the full interaction behavior with all remaining items . To simulate this process during training , a random binary mask b ∈ { 0 , 1 } M is introduced , with the entry 1 as un-masked , and 0 as masked . Thus , xh = x b is the user ’ s partial interaction history . The goal becomes recovering the masked interactions : xp = x ( 1− xh ) , which is equivalent to recovering the full x as xh is known . In LVMs , each user ’ s binary interaction behavior is assumed to be controlled by a k-dimensional user-dependent latent representation z ∈ RK . When applying VAEs to collaborative filtering ( Liang et al. , 2018 ) , the user ’ s latent feature z is represented as a distribution q ( z|x ) , obtained from some partial history xh of x . With the assumption that q ( z|x ) follows a Gaussian form , the inference of z for the corresponding x is performed as : qφ ( z|x ) = N ( µ , diag ( σ2 ) ) , with µ , σ2 = fφ ( xh ) , xh = x b , b ∼ Ber ( α ) , ( 2 ) where α is the hyper-parameter of a Bernoulli distribution , fφ is a φ-parameterized neural network , which outputs the mean µ and variance σ2 of the Gaussian distribution . After obtaining a user ’ s latent representation z , we use the generative process to make predictions . In Liang et al . ( 2018 ) a multinomial distribution is used to model the likelihood of items . Specifically , to construct pθ ( x|z ) , z is transformed to produce a probability distribution π over M items , from which the interaction vector x is assumed to have been drawn : x ∼ Mult ( π ) , with π = Softmax ( gθ ( z ) ) ( 3 ) where gθ is a θ-parameterized neural network . The output π is normalized via a softmax function to produce a probability vector π ∈ ∆M−1 ( an ( M − 1 ) -simplex ) over the entire item set . Training Objective Learning VAE parameters { φ , θ } yields the following generalized objective : Lβ ( x ; θ , φ ) =LE+βLR , with LE=−Eqφ ( z|x ) [ log pθ ( x|z ) ] and LR=KL ( qφ ( z|x ) ||p ( z ) ) ( 4 ) where LE is the negative log likelihood ( NLL ) term , LR is the KL regularization term with standard normal prior p ( z ) , and β is a weighting hyper-parameter . When β = 1 , we can lower-bound the log marginal likelihood of the data using equation 4 as −Lβ=1 ( x ; θ , φ ) ≤ log p ( x ) . This is commonly known as the evidence lower bound ( ELBO ) in variational inference ( Blei et al. , 2017 ) . Thus equation 4 is the negative β-regularized ELBO . To improve the optimization efficiency , the reparametrization trick ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) is used to draw samples z ∼ qφ ( z|x ) to obtain an unbiased estimate of the ELBO , which is further optimized via stochastic optimization . We call this procedure maximum likelihood estimate ( MLE ) -based training , as it effectively maximizes the ( regularized ) ELBO . The testing stage of VAEs for collaborative filtering is detailed in Section A of the Supplement . Advantages of VAEs The VAE framework successfully scales to relatively large datasets by making use of amortized inference ( Gershman & Goodman , 2014 ) : the prediction for all users share the same procedure , which effectively requires evaluating two functions – the encoder fφ ( · ) and the decoder gθ ( · ) . Crucially , as all users share the same encoder/decoder , the number of parameters required for an autoencoder is independent of the number of users . This is in contrast to some traditional latent factor collaborative filtering models ( Paterek , 2007 ; Hu et al. , 2008 ; Mnih & Salakhutdinov , 2008 ) , where a unique latent vector is learned for each user . The reuse of encoder/decoder for all users is well-aligned with collaborative filtering , where user preferences are analyzed by exploiting the similar patterns inferred from past experiences ( Liang et al. , 2018 ) . VAEs has the two advantages simultaneously : expressive representation power as a non-linear model , and the number of parameters being independent of the number of users . The same NLL , but different DCG Pitfalls of VAEs Among various likelihood forms , it was argued in Liang et al . ( 2018 ) that multinomial likelihoods are a closer proxy to the ranking loss than the traditional Gaussian or logistic likelihoods . Though simple and effective , the MLE procedure may still diverge with the ultimate goal in recommendation of correctly suggesting the top-ranked items . To illustrate the divergence between MLE-based training and ranking-based evaluation , consider the example in Figure 1 . For the target x = { 1 , 1 , 0 , 0 } , two different predictions A and B are provided . In MLE , the training loss is the multinomial NLL : −x logπ , where π is the predicted probability . From the NLL point of view , B is a better prediction than A , because B shows a lower loss than A . However , B ranks an incorrect item highest , and therefore would return a worse recommendation than A. Fortunately , NDCG is calculated directly from the ranking , and so captures this dependence . This inspired us to directly use ranking-based evaluation metrics to guide training . For details on calculating NDCG , refer to Section E.2 of the Supplement . 3 RANKING-CRITICAL TRAINING . We introduce a novel algorithm for recommender system training , which we call Ranking-Critical Training ( RaCT ) . RaCT learns a differentiable approximation to the ranking metric , which the prediction network then leverages as a target for optimization through gradient ascent . This is in contrast to existing methods in collaborative filtering , which define an objective relaxation ahead of time . This methodology of learning approximations to functions which can not be optimized directly stems from the actor-critic paradigm of RL , which we adapt for collaborative filtering . Any ranking-based evaluation metric can be considered as a “ black box ” function ω : { π ; x , b } 7→ y ∈ [ 0 , 1 ] , which takes in the prediction π to compare with the ground-truth x ( conditioned on the mask b ) , and outputs a scalar y to rate the prediction quality . As in equation 2 , b partitions a user ’ s interactions into those that are “ observed ” and “ unobserved ” during inference . As we are only interested in recovering the unobserved items in recommendation , we compute the ranking score of predicted items πp = π ( 1− xh ) based on the ground-truth items xp . One salient component of a ranking-based Oracle metric ω∗ is to sort πp . The sorting operation is nondifferentiable , rendering it impossible to directly use ω∗ as the critic . While REINFORCE ( Williams , 1992 ) may appear to be suited to tackle the non-differentiable problem , it suffers from large estimate variance ( Silver et al. , 2014 ) , especially in the collaborative filtering problem , which has a very large prediction space . This motivates consideration of a differentiable neural network to approximate the mapping executed by the Oracle . In the actor-critic framework , the prediction network is called the actor , and the network which approximates the oracle is called the critic . The actor begins by making a prediction ( action ) given the user ’ s interaction history as the state . The critic learns to estimate the value of each action , which we define as the task-specific reward , i.e. , the Oracle ’ s output . The value predicted by the critic is then used to train the actor . Under the assumption that the critic produces the exact values , the actor is trained based on an unbiased estimate of the gradient of the prediction value in terms of relevant ranking quality metrics . In Figure 2 , we illustrate the actor-critic paradigm in ( b ) , and the traditional auto-encoder shown in ( a ) can be used as the actor in our paradigm . Naive critic Conventionally one may concatenate vectors [ πp , xp ] as input to a neural network , and train a network to output the measured ranking scores y . However , this naive critic is impractical , and failed in our experiments . Our hypothesis is that since this network architecture has a huge number of parameters to train ( as the input data layer is of length 2M , where M > 10k ) , it would require rich data for training . Unfortunately , this is impractical : { π , x } ∈ RM are very high-dimensional , and the implicit feedback used in collaborative filtering is naturally sparse . Feature-based critic The naive critic hopes a deep network can discover structure from massive data by itself , leaving much valuable domain knowledge unused . We propose a more efficient critic , that takes into account the structure underlined by the assumed likelihood in MLE ( Miyato & Koyama , 2018 ) . We describe our intuition and method below , and provide the justification from the perspective of adversarial learning in Section D of the Supplement . Consider the computation procedure of the evaluation metric as a function decomposition ω = ω0◦ωψ , including two steps : • ω0 : π 7→ h , feature engineering of prediction π into the sufficient statistics h ; • ωψ : h 7→ ŷ , neural approximation of the mapping from the statistics h to the estimated ranking score ŷ , using a ψ-parameterized neural network . The success of this two-step critic largely depends on the effectiveness of the feature h. We hope feature h is ( i ) compact so that fewer parameters in the critic ωψ can simplify training ; ( ii ) easy-tocompute so that training and testing is efficient ; and ( iii ) informative so that the necessary information is preserved . We suggest to use a 3-dimensional vector as the feature , and leave more complicated feature engineering as future work . In summary , our feature is h = [ LE , |H0| , |H1| ] , ( 5 ) where ( i ) LE is the negative log-likelihood in equation 4 , defined in the MLE training loss . ( ii ) |H0| is the number of unobserved items that a user will interact , withH0 = { m|xm = 1 and bm = 0 } . ( iii ) |H1| is the number of observed items that a user has interacted , withH1 = { m|xm = 1 and bm = 1 } . The NLL characterizes the prediction quality of the actor ’ s output π against the ground-truth x in an item-to-item comparison manner , e.g. , the inner product between two vectors −x logπ as in the multinomial NLL ( Liang et al. , 2018 ) . Ranking is made easier when there are many acceptable items to rank highly ( e.g . when |H0| is large ) , and made difficult when predicting from very few interactions ( e.g . when |H1| is small ) , motivating these two features . Including these three features allows the critic to guide training by weighting the NLL ’ s relation to ranking given this context about the user . Interestingly , this idea to consider the importance of user behavior statistics coincides with the scaling trick in SVD ( Nikolakopoulos et al. , 2019b ) . Note that |H0| and |H1| are user-specific , indicating the user ’ s frequency to interact with the system , which can be viewed as side-information about the user . They are only used as features in training the critic to better approximate the ranking scores , and not in training the actor . Hence , we do not use additional information in the testing stage . Actor Pre-training In order to be a helpful feature for the critic , the NLL must hold some relationship to the ranking-based objective function . But for the high-dimensional datasets common to collaborative filtering , the ranking score is near-uniformly zero for a randomly-initialized actor . In this situation , a trained critic will not propagate derivatives to the actor , and therefore the actor will not improve . We mitigate this problem by using a pre-trained actor , such as VAEs that have been trained via MLE . Critic Pre-training Training a generic critic to approximate the ranking scores for all possible predictions is difficult and cumbersome . Furthermore , it is unnecessary . In practice , a critic only needs to estimate the ranking scores on the restricted domain of the current actor ’ s outputs . Therefore , we train the critic offline on top of the pre-trained MLE-based actor . To train the critic , we minimize the Mean Square Error ( MSE ) between the critic output and true ranking score y from the Oracle : LC ( h , y ; ψ ) = ‖ωψ ( h ) − y‖2 , ( 6 ) where the target y is generated using its non-differential definition , which plays the role of ground truth simulator in training . Actor-critic Training Once the critic is well trained , we fix its parameters ψ and update the actor parameters { φ , θ } to maximize the estimated ranking score LA ( h ; φ , θ ) = ωψ ( h ) , ( 7 ) where h is defined in equation 5 , including NLL feature extracted from the prediction made in equation 4 , together with count features . During back-propagation , the gradient of LA wrt the prediction π is ∂LA∂π = ∂LA ∂h ∂h ∂π . It further updates the actor parameters , with the encoder gradient ∂LA ∂φ = ∂LA ∂π ∂π ∂φ and the decoder gradient ∂LA∂θ = ∂LA ∂π ∂π ∂θ . Updating the actor changes its predictions , so we must update the critic to produce the correct ranking scores for its new input domain . The full RaCT training procedure is summarized in Algorithm 1 in the Supplement . Stochastic optimization is used , where a batch of users U = { xi|i ∈ B } is drawn at each iteration , with B as a random subset of user index in { 1 , · · · , N } . The pre-training of the actor in Stage 1 and the critic in Stage 2 are important ; they provide good initialization to the actor-critic training in Stage 3 for fast convergence . Further , we provide an alternative interpretation to view our actor-critic approach in equation 6 and equation 7 from the perspective of adversarial learning ( Goodfellow et al. , 2014 ) in the Supplement . This can partially justify our choice of feature engineering .

Summary: This paper builds on the line of work of developing latent variable models (LVMs) for collaborative filtering with implicit feedback (e.g. which items a user has previously clicked on).  While variational autoencoders allow convenient construction of nonlinear LVMs, they are trained by maximizing the multinomial likelihood for the items selected.  Thus the training objective is not perfectly matched to the evaluation objective, which is typically something like NDCG @N or RECALL @N, neither of which is differentiable.  This paper proposes an actor-critic RL approach to train the nonlinear LVM directly for the NDCG loss.  The idea is to create a critic model that learns to approximate NDCG, and to train the actor to optimize this approximate objective.  However, they find that they are unable to build an effective critic when taking the predictions and ground truth directly.  However, they find that with a simple set of 3-features, they can build an effective critic that gives improved or competitive performance on several metrics on 3 large-scale datasets.  When their method (RaCT) is not the best, it is beaten by EASE, which is computationally much more expensive.