RaCT: Toward Amortized Ranking-Critical Training For Collaborative Filtering 

1 INTRODUCTION . Recommender systems are an important means of improving a user ’ s web experience . Collaborative filtering is a widely-applied technique in recommender systems ( Ricci et al. , 2015 ) , in which patterns across similar users and items are leveraged to predict user preferences ( Su & Khoshgoftaar , 2009 ) . This naturally fits within the learning paradigm of latent variable models ( LVMs ) ( Bishop , 2006 ) , where latent representations capture the shared patterns . Due to their simplicity and effectiveness , LVMs are still a dominant approach . Traditional LVMs employ linear mappings of limited modeling capacity ( Paterek , 2007 ; Mnih & Salakhutdinov , 2008 ) , and a growing body of literature involves applying deep neural networks ( DNNs ) to collaborative filtering to create more expressive models ( He et al. , 2017 ; Wu et al. , 2016 ; Liang et al. , 2018 ) . Among them , variational autoencoders ( VAEs ) ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) have been proposed as non-linear extensions of LVMs ( Liang et al. , 2018 ) . Empirically , VAEs significantly outperform many competing LVMbased methods . One essential contribution to the improved performance is the use of the multinomial likelihood , which is argued by Liang et al . ( 2018 ) to be a close proxy to ranking loss . This property is desirable , because in recommender systems we generally care more about the ranking of predictions than an individual item ’ s score . Hence , prediction results are often evaluated using top-N ranking-based metrics , such as Normalized Discounted Cumulative Gain ( NDCG ) ( Järvelin & Kekäläinen , 2002 ) . The VAE is trained to maximize the likelihood of observations ; as shown below , this does not necessarily result in higher ranking-based scores . A natural question concerns whether one may directly optimize against ranking-based metrics , which are by nature non-differentiable and piecewise-constant . Previous work on learning-to-rank has been explored this question in the information-retrieval community , where relaxations/approximations of ranking loss are considered ( Weimer et al. , 2008 ; Liu et al. , 2009 ; Li , 2014 ; Weston et al. , 2013 ) . In this paper , we borrow the actor-critic idea from reinforcement learning ( RL ) ( Sutton et al. , 1998 ) to propose an efficient and scalable learning-to-rank algorithm . The critic is trained to approximate the ranking metric , while the actor is trained to optimize against this learned metric . Specifically , with the goal of making the actor-critic approach practical for recommender systems , we introduce a novel feature-based critic architecture . Instead of treating raw predictions as the critic input , and hoping the neural network will discover the metric ’ s structure from massive data , we consider ∗Corresponding author †Equal Contribution engineering sufficient statistics for efficient critic learning . Experimental results on three large-scale datasets demonstrate the actor-critic ’ s ability to significantly improve the performance of a variety of latent-variable models , and achieve better or comparable performance to strong baseline methods . 2 BACKGROUND : VAES FOR COLLABORATIVE FILTERING . Vectors are denoted as bold lower-case letters x , matrices as bold uppercase letters X , and scalars as lower-case non-bold letters x . We use ◦ for function composition , for the element-wise multiplication , and | · | for cardinality of a set . δ ( · ) is the indicator function . We use n ∈ { 1 , . . . , N } to index users , and m ∈ { 1 , . . . , M } to index items . The user-item interaction matrix X ∈ { 0 , 1 } N×M collected from the users ’ implicit feedback is defined as : xnm { 1 , if interaction of user n with item m is observed ; 0 , otherwise . ( 1 ) Note that xnm = 0 does not necessarily mean user n dislikes item m ; they may simply be unaware of the item . Further , xnm = 1 is not equivalent to saying user n likes item m , but that there is at least interest . VAE model VAEs have been investigated for collaborative filtering ( Liang et al. , 2018 ) , where this principled Bayesian approach is shown to achieve strong performance on large-scale datasets . Given the user ’ s interaction history x = [ x1 , ... , xM ] > ∈ { 0 , 1 } M , our goal is to predict the full interaction behavior with all remaining items . To simulate this process during training , a random binary mask b ∈ { 0 , 1 } M is introduced , with the entry 1 as un-masked , and 0 as masked . Thus , xh = x b is the user ’ s partial interaction history . The goal becomes recovering the masked interactions : xp = x ( 1− xh ) , which is equivalent to recovering the full x as xh is known . In LVMs , each user ’ s binary interaction behavior is assumed to be controlled by a k-dimensional user-dependent latent representation z ∈ RK . When applying VAEs to collaborative filtering ( Liang et al. , 2018 ) , the user ’ s latent feature z is represented as a distribution q ( z|x ) , obtained from some partial history xh of x . With the assumption that q ( z|x ) follows a Gaussian form , the inference of z for the corresponding x is performed as : qφ ( z|x ) = N ( µ , diag ( σ2 ) ) , with µ , σ2 = fφ ( xh ) , xh = x b , b ∼ Ber ( α ) , ( 2 ) where α is the hyper-parameter of a Bernoulli distribution , fφ is a φ-parameterized neural network , which outputs the mean µ and variance σ2 of the Gaussian distribution . After obtaining a user ’ s latent representation z , we use the generative process to make predictions . In Liang et al . ( 2018 ) a multinomial distribution is used to model the likelihood of items . Specifically , to construct pθ ( x|z ) , z is transformed to produce a probability distribution π over M items , from which the interaction vector x is assumed to have been drawn : x ∼ Mult ( π ) , with π = Softmax ( gθ ( z ) ) ( 3 ) where gθ is a θ-parameterized neural network . The output π is normalized via a softmax function to produce a probability vector π ∈ ∆M−1 ( an ( M − 1 ) -simplex ) over the entire item set . Training Objective Learning VAE parameters { φ , θ } yields the following generalized objective : Lβ ( x ; θ , φ ) =LE+βLR , with LE=−Eqφ ( z|x ) [ log pθ ( x|z ) ] and LR=KL ( qφ ( z|x ) ||p ( z ) ) ( 4 ) where LE is the negative log likelihood ( NLL ) term , LR is the KL regularization term with standard normal prior p ( z ) , and β is a weighting hyper-parameter . When β = 1 , we can lower-bound the log marginal likelihood of the data using equation 4 as −Lβ=1 ( x ; θ , φ ) ≤ log p ( x ) . This is commonly known as the evidence lower bound ( ELBO ) in variational inference ( Blei et al. , 2017 ) . Thus equation 4 is the negative β-regularized ELBO . To improve the optimization efficiency , the reparametrization trick ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) is used to draw samples z ∼ qφ ( z|x ) to obtain an unbiased estimate of the ELBO , which is further optimized via stochastic optimization . We call this procedure maximum likelihood estimate ( MLE ) -based training , as it effectively maximizes the ( regularized ) ELBO . The testing stage of VAEs for collaborative filtering is detailed in Section A of the Supplement . Advantages of VAEs The VAE framework successfully scales to relatively large datasets by making use of amortized inference ( Gershman & Goodman , 2014 ) : the prediction for all users share the same procedure , which effectively requires evaluating two functions – the encoder fφ ( · ) and the decoder gθ ( · ) . Crucially , as all users share the same encoder/decoder , the number of parameters required for an autoencoder is independent of the number of users . This is in contrast to some traditional latent factor collaborative filtering models ( Paterek , 2007 ; Hu et al. , 2008 ; Mnih & Salakhutdinov , 2008 ) , where a unique latent vector is learned for each user . The reuse of encoder/decoder for all users is well-aligned with collaborative filtering , where user preferences are analyzed by exploiting the similar patterns inferred from past experiences ( Liang et al. , 2018 ) . VAEs has the two advantages simultaneously : expressive representation power as a non-linear model , and the number of parameters being independent of the number of users . The same NLL , but different DCG Pitfalls of VAEs Among various likelihood forms , it was argued in Liang et al . ( 2018 ) that multinomial likelihoods are a closer proxy to the ranking loss than the traditional Gaussian or logistic likelihoods . Though simple and effective , the MLE procedure may still diverge with the ultimate goal in recommendation of correctly suggesting the top-ranked items . To illustrate the divergence between MLE-based training and ranking-based evaluation , consider the example in Figure 1 . For the target x = { 1 , 1 , 0 , 0 } , two different predictions A and B are provided . In MLE , the training loss is the multinomial NLL : −x logπ , where π is the predicted probability . From the NLL point of view , B is a better prediction than A , because B shows a lower loss than A . However , B ranks an incorrect item highest , and therefore would return a worse recommendation than A. Fortunately , NDCG is calculated directly from the ranking , and so captures this dependence . This inspired us to directly use ranking-based evaluation metrics to guide training . For details on calculating NDCG , refer to Section E.2 of the Supplement . 3 RANKING-CRITICAL TRAINING . We introduce a novel algorithm for recommender system training , which we call Ranking-Critical Training ( RaCT ) . RaCT learns a differentiable approximation to the ranking metric , which the prediction network then leverages as a target for optimization through gradient ascent . This is in contrast to existing methods in collaborative filtering , which define an objective relaxation ahead of time . This methodology of learning approximations to functions which can not be optimized directly stems from the actor-critic paradigm of RL , which we adapt for collaborative filtering . Any ranking-based evaluation metric can be considered as a “ black box ” function ω : { π ; x , b } 7→ y ∈ [ 0 , 1 ] , which takes in the prediction π to compare with the ground-truth x ( conditioned on the mask b ) , and outputs a scalar y to rate the prediction quality . As in equation 2 , b partitions a user ’ s interactions into those that are “ observed ” and “ unobserved ” during inference . As we are only interested in recovering the unobserved items in recommendation , we compute the ranking score of predicted items πp = π ( 1− xh ) based on the ground-truth items xp . One salient component of a ranking-based Oracle metric ω∗ is to sort πp . The sorting operation is nondifferentiable , rendering it impossible to directly use ω∗ as the critic . While REINFORCE ( Williams , 1992 ) may appear to be suited to tackle the non-differentiable problem , it suffers from large estimate variance ( Silver et al. , 2014 ) , especially in the collaborative filtering problem , which has a very large prediction space . This motivates consideration of a differentiable neural network to approximate the mapping executed by the Oracle . In the actor-critic framework , the prediction network is called the actor , and the network which approximates the oracle is called the critic . The actor begins by making a prediction ( action ) given the user ’ s interaction history as the state . The critic learns to estimate the value of each action , which we define as the task-specific reward , i.e. , the Oracle ’ s output . The value predicted by the critic is then used to train the actor . Under the assumption that the critic produces the exact values , the actor is trained based on an unbiased estimate of the gradient of the prediction value in terms of relevant ranking quality metrics . In Figure 2 , we illustrate the actor-critic paradigm in ( b ) , and the traditional auto-encoder shown in ( a ) can be used as the actor in our paradigm . Naive critic Conventionally one may concatenate vectors [ πp , xp ] as input to a neural network , and train a network to output the measured ranking scores y . However , this naive critic is impractical , and failed in our experiments . Our hypothesis is that since this network architecture has a huge number of parameters to train ( as the input data layer is of length 2M , where M > 10k ) , it would require rich data for training . Unfortunately , this is impractical : { π , x } ∈ RM are very high-dimensional , and the implicit feedback used in collaborative filtering is naturally sparse . Feature-based critic The naive critic hopes a deep network can discover structure from massive data by itself , leaving much valuable domain knowledge unused . We propose a more efficient critic , that takes into account the structure underlined by the assumed likelihood in MLE ( Miyato & Koyama , 2018 ) . We describe our intuition and method below , and provide the justification from the perspective of adversarial learning in Section D of the Supplement . Consider the computation procedure of the evaluation metric as a function decomposition ω = ω0◦ωψ , including two steps : • ω0 : π 7→ h , feature engineering of prediction π into the sufficient statistics h ; • ωψ : h 7→ ŷ , neural approximation of the mapping from the statistics h to the estimated ranking score ŷ , using a ψ-parameterized neural network . The success of this two-step critic largely depends on the effectiveness of the feature h. We hope feature h is ( i ) compact so that fewer parameters in the critic ωψ can simplify training ; ( ii ) easy-tocompute so that training and testing is efficient ; and ( iii ) informative so that the necessary information is preserved . We suggest to use a 3-dimensional vector as the feature , and leave more complicated feature engineering as future work . In summary , our feature is h = [ LE , |H0| , |H1| ] , ( 5 ) where ( i ) LE is the negative log-likelihood in equation 4 , defined in the MLE training loss . ( ii ) |H0| is the number of unobserved items that a user will interact , withH0 = { m|xm = 1 and bm = 0 } . ( iii ) |H1| is the number of observed items that a user has interacted , withH1 = { m|xm = 1 and bm = 1 } . The NLL characterizes the prediction quality of the actor ’ s output π against the ground-truth x in an item-to-item comparison manner , e.g. , the inner product between two vectors −x logπ as in the multinomial NLL ( Liang et al. , 2018 ) . Ranking is made easier when there are many acceptable items to rank highly ( e.g . when |H0| is large ) , and made difficult when predicting from very few interactions ( e.g . when |H1| is small ) , motivating these two features . Including these three features allows the critic to guide training by weighting the NLL ’ s relation to ranking given this context about the user . Interestingly , this idea to consider the importance of user behavior statistics coincides with the scaling trick in SVD ( Nikolakopoulos et al. , 2019b ) . Note that |H0| and |H1| are user-specific , indicating the user ’ s frequency to interact with the system , which can be viewed as side-information about the user . They are only used as features in training the critic to better approximate the ranking scores , and not in training the actor . Hence , we do not use additional information in the testing stage . Actor Pre-training In order to be a helpful feature for the critic , the NLL must hold some relationship to the ranking-based objective function . But for the high-dimensional datasets common to collaborative filtering , the ranking score is near-uniformly zero for a randomly-initialized actor . In this situation , a trained critic will not propagate derivatives to the actor , and therefore the actor will not improve . We mitigate this problem by using a pre-trained actor , such as VAEs that have been trained via MLE . Critic Pre-training Training a generic critic to approximate the ranking scores for all possible predictions is difficult and cumbersome . Furthermore , it is unnecessary . In practice , a critic only needs to estimate the ranking scores on the restricted domain of the current actor ’ s outputs . Therefore , we train the critic offline on top of the pre-trained MLE-based actor . To train the critic , we minimize the Mean Square Error ( MSE ) between the critic output and true ranking score y from the Oracle : LC ( h , y ; ψ ) = ‖ωψ ( h ) − y‖2 , ( 6 ) where the target y is generated using its non-differential definition , which plays the role of ground truth simulator in training . Actor-critic Training Once the critic is well trained , we fix its parameters ψ and update the actor parameters { φ , θ } to maximize the estimated ranking score LA ( h ; φ , θ ) = ωψ ( h ) , ( 7 ) where h is defined in equation 5 , including NLL feature extracted from the prediction made in equation 4 , together with count features . During back-propagation , the gradient of LA wrt the prediction π is ∂LA∂π = ∂LA ∂h ∂h ∂π . It further updates the actor parameters , with the encoder gradient ∂LA ∂φ = ∂LA ∂π ∂π ∂φ and the decoder gradient ∂LA∂θ = ∂LA ∂π ∂π ∂θ . Updating the actor changes its predictions , so we must update the critic to produce the correct ranking scores for its new input domain . The full RaCT training procedure is summarized in Algorithm 1 in the Supplement . Stochastic optimization is used , where a batch of users U = { xi|i ∈ B } is drawn at each iteration , with B as a random subset of user index in { 1 , · · · , N } . The pre-training of the actor in Stage 1 and the critic in Stage 2 are important ; they provide good initialization to the actor-critic training in Stage 3 for fast convergence . Further , we provide an alternative interpretation to view our actor-critic approach in equation 6 and equation 7 from the perspective of adversarial learning ( Goodfellow et al. , 2014 ) in the Supplement . This can partially justify our choice of feature engineering .

Summary: The authors propose a novel reinforcement learning (RL) based recommendation algorithm (i.e., Ranking-Critical Training, RaCT) for modeling users' implicit feedback, which is illustrated in Figure 2(b). Specifically, the authors apply the actor-critic RL paradigm to approximately optimize the ranking-oriented loss/objective function in collaborative filtering with implicit feedback, which is a difficult and important problem. In particular, the critic network is used to approximate the ranking-oriented metric, and the actor network is used for optimizing this metric. Moreover, for efficiency, the authors also propose a feature-based critic to replace the original one.