Beyond Greedy Ranking: Slate Optimization via List-CVAE

1 INTRODUCTION . Recommender systems modeling is an important machine learning area in the IT industry , powering online advertisement , social networks and various content recommendation services ( Schafer et al. , 2001 ; Lu et al. , 2015 ) . In the context of document recommendation , its aim is to generate and display an ordered list of “ documents ” to users ( called a “ slate ” in Swaminathan et al . ( 2017 ) ; Sunehag et al . ( 2015 ) ) , based on both user preferences and documents content . For large scale recommender systems , a common scalable approach at inference time is to first select a small subset of candidate documents S out of the entire document pool D. This step is called “ candidate generation ” . Then a function approximator such as a neural network ( e.g. , a Multi-Layer Perceptron ( MLP ) ) called the “ ranking model ” is used to predict probabilities of user engagements for each document in the small subset S and greedily generates a slate by sorting the top documents from S based on estimated prediction scores ( Covington et al. , 2016 ) . This two-step process is widely popular to solve large scale recommendation problems due to its scalability and fast inference at serving time . The candidate generation step can decrease the number of candidates from millions to hundreds or less , effectively dealing with scalability when faced with a large corpus of documents D. Since |S| is much smaller than |D| , the ranking model can be reasonably complicated without increasing latency . However , there are two main problems with this approach . First the candidate generation and the ranking models are not trained jointly , which can lead to having candidates in S that are not the highest scoring documents of the ranking model . Second and most importantly , the greedy ranking method suffers from numerous biases that come with the visual presentation of the slate and context in which documents are presented , both at training and serving time . For example , there exists positional biases caused by users paying more attention to prominent slate positions ( Joachims et al. , 2005 ) , and contextual biases , due to interactions between documents presented together in the same slate , such as competition and complementarity , relative attractiveness , etc . ( Yue et al. , 2010 ) . In this paper , we propose a paradigm shift from the traditional viewpoint of solving a ranking problem to a direct slate generation framework . We consider a slate “ optimal ” when it maximizes some type of user engagement feedback , a typical desired scenario in recommender systems . For example , given ∗Google DeepMind , London , UK . Correspondence to : Ray Jiang < rayjiang @ google.com > . †The University of Hong Kong a database of song tracks , the optimal slate can be an ordered list ( in time or space ) of k songs such that the user ideally likes every song in that list . Another example considers news articles , the optimal slate has k ordered articles such that every article is read by the user . In general , optimality can be defined as a desired user response vector on the slate and the proposed model should be agnostic to these problem-specific definitions . Solving the slate recommendation problem by direct slate generation differs from ranking in that first , the entire slate is used as a training example instead of single documents , preserving numerous biases encoded into the slate that might influence user responses . Secondly , it does not assume that more relevant documents should necessarily be put in earlier positions in the slate at serving time . Our model directly generates slates , taking into account all the relevant biases learned through training . In this paper , we apply Conditional Variational Auto-Encoders ( CVAEs ) ( Kingma et al. , 2014 ; Kingma and Welling , 2013 ; Jimenez Rezende et al. , 2014 ) to model the distributions of all documents in the same slate conditioned on the user response . All documents in a slate along with their positional , contextual biases are jointly encoded into the latent space , which is then sampled and combined with desired conditioning for direct slate generation , i.e . sampling from the learned conditional joint distribution . Therefore , the model first learns which slates give which type of responses and then directly generates similar slates given a desired response vector as the conditioning at inference time . We call our proposed model List-CVAE . The key contributions of our work are : 1 . To the best of our knowledge , this is the first model that provides a conditional generative modeling framework for slate recommendation by direct generation . It does not necessarily require a candidate generator at inference time and is flexible enough to work with any visual presentation of the slate as long as the ordering of display positions is fixed throughout training and inference times . 2 . To deal with the problem at scale , we introduce an architecture that uses pretrained document embeddings combined with a negatively downsampled k-head softmax layer within the List-CVAE model , where k is the slate size . The structure of this paper is the following . First we introduce related work using various CVAE-type models as well as other approaches to solve the slate generation problem . Next we introduce our List-CVAE modeling approach . The last part of the paper is devoted to experiments on both simulated and the real-world datasets . 2 RELATED WORK . Traditional matrix factorization techniques have been applied to recommender systems with success in modeling competitions such as the Netflix Prize ( Koren et al. , 2009 ) . Later research emerged on using autoencoders to improve on the results of matrix factorization ( Wu et al. , 2016 ; Wang et al. , 2015 ) ( CDAE , CDL ) . More recently several works use Boltzmann Machines ( Abdollahi and Nasraoui , 2016 ) and variants of VAE models in the Collaborative Filtering ( CF ) paradigm to model recommender systems ( Li and She , 2017 ; Lee et al. , 2017 ; Liang et al. , 2018 ) ( Collaborative VAE , JMVAE , CVAE-CF , JVAE-CF ) . See Figure 1 for model structure comparisons . In this paper , unless specified otherwise , the user features and any context are routinely considered part of the conditioning variables ( in Appendix A Personalization Test , we test List-CVAE generating personalized slates for different users ) . These models have primarily focused on modeling individual document or pairs of documents in the slate and applying greedy ordering at inference time . Our model is also using a VAE type structure and in particular , is closely related to the Joint Multimodel Variational Auto-Encoder ( JMVAE ) architecture ( Figure 1d ) . However , we use whole slates as input instead of single documents , and directly generate slates instead of using greedy ranking by prediction scores . Other relevant work from the Information Retrieval ( IR ) literature are listwise ranking methods ( Cao et al. , 2007 ; Xia et al. , 2008 ; Shi et al. , 2010 ; Huang et al. , 2015 ; Ai et al. , 2018 ) . These methods use listwise loss functions that take the contexts and positions of training examples into account . However , they eventually assign a prediction score for each document and greedily rank them at inference time . In the Reinforcement Learning ( RL ) literature , Sunehag et al . ( 2015 ) view the whole slates as actions and use a deterministic policy gradient update to learn a policy that generates these actions , given concatenated document features as input . Finally , the framework proposed by ( Wang et al. , 2016 ) predicts user engagement for document and position pairs . It optimizes whole page layouts at inference time but may suffer from poor scalability due to the combinatorial explosion of all possible document position pairs . 3 METHOD . 3.1 PROBLEM SETUP . We formally define the slate recommendation problem as follows . LetD denote a corpus of documents and let k be the slate size . Then let r = ( r1 , . . . , rk ) be the user response vector , where ri ∈ R is the user ’ s response on document di . For example , if the problem is to maximize the number of clicks on a slate , then let ri ∈ { 0 , 1 } denote whether the document di is clicked , and thus an optimal slate s = ( d1 , d2 , . . . , dk ) where di ∈ D is such that s maximizes E [ ∑k i=1 ri ] . 3.2 VARIATIONAL AUTO-ENCODERS . Variational Auto-Encoders ( VAEs ) are latent-variable models that define a joint density Pθ ( x , z ) between observed variables x and latent variables z parametrized by a vector θ . Training such models requires marginalizing the latent variables in order to maximize the data likelihood Pθ ( x ) = ∫ Pθ ( x , z ) dz . Since we can not solve this marginalization explicitly , we resort to a variational approximation . For this , a variational posterior density Qφ ( z|x ) parametrized by a vector φ is introduced and we optimize the variational Evidence Lower-Bound ( ELBO ) on the data loglikelihood : logPθ ( x ) = KL [ Qφ ( z|x ) ‖Pθ ( z|x ) ] + EQφ ( z|x ) [ − logQφ ( z|x ) + logPθ ( x , z ) ] , ( 1 ) ≥ −KL [ Qφ ( z|x ) ‖Pθ ( z ) ] + EQφ ( z|x ) [ logPθ ( x|z ) ] , ( 2 ) where KL is the Kullback–Leibler divergence and where Pθ ( z ) is a prior distribution over latent variables . In a Conditional VAE ( CVAE ) we extend the distributions Pθ ( x , z ) and Qφ ( z|x ) to also depend on an external condition c. The corresponding distributions are indicated by Pθ ( x , z|c ) and Qφ ( z|x , c ) . Taking the conditioning c into account , we can write the variational loss to minimize as LCVAE = KL [ Qφ ( z|x , c ) ‖Pθ ( z|c ) ] − EQφ ( z|x , c ) [ logPθ ( x|z , c ) ] . ( 3 ) 3.3 OUR MODEL . We assume that the slates s = ( d1 , d2 , . . . dk ) and the user response vectors r are jointly drawn from a distribution PDk×Rk . In this paper , we use a CVAE to model the joint distribution of all documents in the slate conditioned on the user responses r , i.e . P ( d1 , d2 , . . . dk|r ) . At inference time , the List-CVAE model attempts to generate an optimal slate by conditioning on the ideal user response r ? . As we explained in Section 1 , “ optimality ” of a slate depends on the task . With that in mind , we define the mapping Φ : Rk 7→ C. It transforms a user response vector r into a vector in the conditioning space C that encodes the user engagement metric we wish to optimize for . For instance , if we want to maximize clicks on the slate , we can use the binary click response vectors and set the conditioning to c = Φ ( r ) : = ∑k i=0 ri . Then at inference time , the corresponding ideal user response r ? would be ( 1 , 1 , . . . , 1 ) , and correspondingly the ideal conditioning would be c ? = Φ ( r ? ) = ∑k i=0 1 = k. As usual with CVAEs , the decoder models a distribution Pθ ( s|z , c ) that , conditioned on z , is easy to represent . In our case , Pθ ( s|z , c ) models an independent probability for each document on the slate , represented by a softmax distribution . Note that the documents are only independent to each other conditional on z . In fact , the marginalized posterior Pθ ( s|c ) = ∫ z Pθ ( s|z , c ) Pθ ( z|c ) dz can be arbitrarily complex . When the encoder encodes the input slate s into the latent space , it learns the joint distribution of the k documents in a fixed order , and thus also encodes any contextual , positional biases between documents and their respective positions into the latent variable z . The decoder learns these biases through reconstruction of the input slates from latent variables z with conditions . At inference time , the decoder reproduces the input slate distribution from the latent variable z with the ideal conditioning , taking into account all the biases learned during training time . To shed light onto what is encoded in the latent space , we simplify the prior distribution of z to be a fixed Gaussian distribution N ( 0 , I ) in R2 . We train List-CVAE and plot the predictive prior z . As training evolves , generated output slates with low total responses are pushed towards the edge of the latent space while high response slates cluster towards a growing center area ( Figure 3 ) . Therefore after training , if we sample z from its prior distribution N ( 0 , I ) and generate the corresponding output slates , they are more likely to have high total responses . Since the number of documents in D can be large , we first embed the documents into a low dimensional space . Let Ψ : D 7→ Sq−1 be that normalized embedding where Sq−1 denotes the unit sphere in Rq . Ψ can easily be pretrained using a standard supervised model that predicts user responses from documents or through a standard auto-encoder technique . For the i-th document in the slate , our model produces a vector xi in Rq that is then matched to each document from D via a dot-product . This operation produces k vectors of logits for k softmaxes , i.e . the k-head softmax . At training time , for large document corpora D , we uniformly randomly downsample negative documents and compute only a small subset of the logits for every training example , therefore efficiently scaling the nearest neighbor search to millions of documents with minimal model quality loss . We train this model as a CVAE by minimizing the sum of the reconstruction loss and the KLdivergence term : L = βKL [ Qφ ( z | s , c ) ‖Pθ ( z |c ) ] − EQφ ( z|s , c ) [ logPθ ( s | z , c ) ] , ( 4 ) where β is a function of the training step ( Higgins et al. , 2017 ) . During inference , output slates are generated by first sampling z from the conditionally learned prior distribution N ( µ ? , σ ? ) , concatenating with the ideal condition c ? = Φ ( r ? ) , and passed into the decoder , generating ( x1 , . . . , xk ) from the learned Pθ ( s|z , c ? ) , and finally taking arg max over the dot-products with the full embedding matrix independently for each position i = 1 , . . . , k .

Summary: This paper pr poses a conditional generative model for slate-based recommendations. The idea of slate recommendation is to model an ordered-list of items instead of modelling each item independently as is usually done (e.g., for computational reasons). This provides a more natural framework for recommending lists of items (vs. recommending the items with the top scores).