Seq2Slate: Re-ranking and Slate Optimization with RNNs

1 INTRODUCTION . Ranking a set of candidate items is a central task in machine learning and information retrieval . Many existing ranking systems are based on pointwise estimators , where the model assigns a score to each item in a candidate set and the resulting slate is obtained by sorting the list according to item scores ( Liu et al. , 2009 ) . Such models are usually trained from click-through data to optimize an appropriate loss function ( Joachims , 2002 ) . This simple approach is computationally attractive as it only requires a sort operation over the candidate set at test ( or serving ) time , and can therefore scale to large problems . On the other hand , in terms of modeling , pointwise rankers can not easily express dependencies between ranked items . In particular , the score of an item ( e.g. , its probability of being clicked ) often depends on the other items in the slate and their joint placement . Such interactions between items can be especially dominant in the common case where display area is limited or when strong position bias is present , so that only a few highly ranked items get the user ’ s attention . In this case it may be preferable , for example , to present a diverse set of items at the top positions of the slate in order to cover a wider range of user interests . A significant amount of work on learning-to-rank does consider interactions between ranked items when training the model . In pairwise approaches a classifier is trained to determine which item should be ranked first within a pair of items ( e.g. , Herbrich et al. , 1999 ; Joachims , 2002 ; Burges et al. , 2005 ) . Similarly , in listwise approaches the loss depends on the full permutation of items ( e.g. , Cao et al. , 2007 ; Yue et al. , 2007 ) . Although these losses consider inter-item dependencies , the ranking function itself is pointwise , so at inference time the model still assigns a score to each item which does not depend on scores of other items . There has been some work on trying to capture interactions between items in the ranking scores themselves ( e.g. , Qin et al. , 2008 ; 2009 ; Zhu et al. , 2014 ; Rosenfeld et al. , 2014 ; Dokania et al. , 2014 ) . Such approaches can , for example , encourage a pair of items to appear next to ( or far from ) each other in the resulting ranking . Approaches of this type often assume that the relationship between items takes a simple form ( e.g. , submodular ) in order to obtain tractable inference and learning algorithms . Unfortunately , this comes at the expense of the model ’ s expressive power . In this paper , we present a general , scalable approach to ranking , which naturally accounts for high-order interactions . In particular , we apply a sequence-to-sequence ( seq2seq ) model ( Sutskever et al. , 2014 ) to the ranking task , where the input is the list of candidate items and the output is the resulting ordering . Since the output sequence corresponds to ranked items on the slate , we call this model sequence-to-slate ( seq2slate ) . The order in which the input is processed can significantly affect the performance of such models ( Vinyals et al. , 2016 ) . For this reason , we often assume the availability of a base ( or “ production ” ) ranker with which the input sequence is ordered ( e.g. , a simple pointwise method that ignores the interactions we seek to model ) , and view the output of our model as a re-ranking of the items . To address the seq2seq problem , we build on the recent success of recurrent neural networks ( RNNs ) in a wide range of applications ( e.g. , Sutskever et al. , 2014 ) . This allows us to use a deep model to capture rich dependencies between ranked items , while keeping the computational cost of inference manageable . More specifically , we use pointer networks , which are seq2seq models with an attention mechanism for pointing at positions in the input ( Vinyals et al. , 2015b ) . We show how to train the network end-to-end to directly optimize several commonly used ranking measures . To this end , we adapt RNN training to use weak supervision in the form of click-through data obtained from logs , instead of relying on ground-truth rankings , which are much more expensive to obtain . Finally , we demonstrate the usefulness of the proposed approach in a number of learning-to-rank benchmarks and in a large-scale , real-world recommendeation system . 2 RANKING AND SLATE OPTIMIZATION AS SEQUENCE PREDICTION . The ranking problem is that of computing a ranking of a set of items ( or ordered list or slate ) given some query or context . We formalize the problem as follows . Assume a set of n items , each represented by a feature vector xi ∈ Rm ( which may depend on a query or context ) . Let π ∈ Π denote a permutation of the items , where each πj ∈ { 1 , . . . , n } denotes the index of the item in position j . Our goal is to predict the output ranking π given the input items x . For instance , given a specific user query , we might want to return an ordered set of music recommendations from a set of candidates that maximizes some measure of user engagement ( e.g. , number of tracks played ) . In the seq2seq framework , the probability of an output permutation , or slate , given the inputs is expressed as a product of conditional probabilities according to the chain rule : p ( π|x ) = n∏ j=1 p ( πj |π1 , . . . , πj−1 , x ) , ( 1 ) This expression is completely general and does not make any conditional independence assumptions . In our case , the conditional p ( πj |π < j , x ) ∈ ∆n ( a point in the n-dimensional simplex ) models the probability of any item being placed at the j ’ th position in the ranking given the items already placed at previous positions . Therefore , this conditional exactly captures all high-order dependencies between items in the ranked list , including those due to diversity , similarity or other interactions . Our setting is somewhat different than a standard seq2seq setting in that the output vocabulary is not fixed . In particular , the same index ( position ) is populated by different items in different instances ( queries ) . Indeed , the vocabulary size n itself may vary per instance in the common case where the number of items to rank can change . This is precisely the problem addressed by pointer networks , which we review next . POINTER-NETWORK ARCHITECTURE FOR RANKING We employ the pointer-network architecture of Vinyals et al . ( 2015b ) to model the conditional p ( πj |π < j , x ) . A pointer network uses non-parametric softmax modules , akin to the attention mechanism of Bahdanau et al . ( 2015 ) , and learns to point to items in its input sequence rather than predicting an index from a fixed-sized vocabulary . Our seq2slate model , illustrated in Fig . 1 , consists of two recurrent neural networks ( RNNs ) : an encoder and a decoder , both of which use Long Short-term Memory ( LSTM ) cells ( Hochreiter and Schmidhuber , 1997 ) . At each encoding step i ≤ n , the encoder RNN reads the input vector xi and outputs a d-dimensional vector ei , thus transforming the input sequence { xi } ni=1 into a sequence of latent memory states { ei } ni=1 . At each decoding step j , the decoder RNN outputs a d-dimensional vector dj which is used as a query in our attention function . The attention function takes as input the query dj ∈ Rd and the set of latent memory states computed by the encoder { ei } ni=1 and produces a probability distribution over the next item to include in the output sequence as follows : sji = v > tanh ( Wenc · ei +Wdec · dj ) ( 2 ) pθ ( πj = i|π < j , x ) ≡ pji = { es j i / ∑ k/∈π < j e s j k if i /∈ π < j 0 if i ∈ π < j , where Wenc , Wdec ∈ Rd×d and v ∈ Rd are learned parameters in our network , denoted collectively by parameter vector θ . The probability pji = pθ ( πj = i|π < j , x ) , is obtained via a softmax over the remaining items and represents the degree to which the model points to input i at decoding step j . To output a permutation , the pji are set to 0 for items i that already appear in the slate . Once the next item πj is selected , typically greedily or by sampling ( see below ) , its embedding xπj is fed as input to the next decoder step . The input of the first decoder step is a learned d-dimensional vector , denoted as go in Fig . 1 . Importantly , pθ ( π|x ) is differentiable for ant fixed permutation π which allows gradient-based learning ( see Section 3 ) . We note the following . ( i ) The model makes no explicit assumptions about the type of interactions between items . If the learned conditional in Eq . ( 2 ) is close to the true conditional in Eq . ( 1 ) , then the model can capture rich interactions—including diversity , similarity or others . We demonstrate this flexibility in our experiments ( Section 4 ) . ( ii ) x can represent either raw inputs or embeddings thereof , which can be learned together with the sequence model . ( iii ) The computational cost of inference , dominated by the sequential decoding procedure , is O ( n2 ) , which is standard in seq2seq models with attention . We also consider a computationally cheaper single-step decoder with linear cost O ( n ) , which outputs a single vector p1 , from which we obtain π by sorting the values ( similarly to pointwise ranking ) . 3 TRAINING WITH CLICK-THROUGH DATA . We now turn to the task of training the seq2slate model from data . A typical approach to learning in ranking systems is to run an existing ranker “ in the wild ” and log click-through data , which are then used to train an improved ranking model . This type of training data is relatively inexpensive to obtain , in contrast to human-curated labels such as relevance scores , ratings , or rankings ( Joachims , 2002 ) . Formally , each training example consists of a sequence of items { x1 , . . . , xn } and binary labels ( y1 , . . . , yn ) , with yi ∈ { 0 , 1 } , representing user feedback ( e.g. , click/no-click ) . Our approach easily extends to more informative feedback , such as the level of user engagement with the chosen item ( e.g. , time spent ) , but to simplify the presentation we focus on the binary case . Our goal is to learn the parameters θ of pθ ( πj |π < j , x ) ( Eq . ( 2 ) ) such that permutations π corresponding to “ good ” rankings are assigned high probabilities . Various performance measuresR ( π , y ) can be used to evaluate the quality of a permutation π given the labels y , for example , mean average precision ( MAP ) , precision at k , or normalized discounted cumulative gain at k ( NDCG @ k ) . Generally speaking , permutations where the positive labels rank higher are considered better . In the standard seq2seq setting , models are trained to maximize the likelihood of a target sequence of tokens given the input , which can be done by maximizing the likelihood of each target token given the previous target tokens using Eq . ( 1 ) . During training , the model is typically fed the ground-truth tokens as inputs to the next prediction step , an approach known as teacher forcing ( Williams and Zipser , 1989 ) . Unfortunately , this approach can not be applied in our setting since we only have access to weak supervision in the form of labels y ( e.g clicks ) , rather than ground-truth permutations . Instead , we show how the seq2slate model can be trained directly from the labels y .

Summary: The authors consider the problem of re-ranking an initial ranker that doesn’t consider interactions between items (e.g., a point-wise ranker) with a pointer-network approach that considers these interactions when re-ordering the input ranking. Notably, this is performed during decoding as opposed to {pairwise, list-wise} learning to rank approaches that consider interactions during training, but emit an item-wise score during inference. Operationally in practice, this has to be trained from click-through data for which the authors consider both a RL approach (Reinforce) and supervised training (a sequence-level hinge loss function) and decoded either with a single-step greedy policy or a sampling procedure. Experiments are conducted on learning-to-rank benchmarks where interactions are introduced to test the validity of the method and on a real-world, large-scale recommendation engine — showing solid improvements in both cases.