AN ATTENTION-BASED DEEP NET FOR LEARNING TO RANK

1 INTRODUCTION . Learning to rank applies supervised or semi-supervised machine learning to construct ranking models for information retrieval problems . In learning to rank , a query is given and a number of search results are to be ranked by their relevant importance given the query . Many problems in information retrieval can be formulated or partially solved by learning to rank . In learning to rank , there are typically three approaches : the pointwise , pairwise , and listwise approaches Liu ( 2011 ) . The pointwise approach assigns an importance score to each pair of query and search result . The pairwise approach discerns which search result is more relevant for a certain query and a pair of search results . The listwise approach outputs the ranks for all search results given a specific query , therefore being the most general . For learning to rank , neural networks are known to enjoy a success . Generally in such models , neural networks are applied to model the ranking probabilities with the features of queries and search results as the input . For instance , RankNet Burges et al . ( 2005 ) applies a neural network to calculate a probability for any search result being more relevant compared to another . Each pair of query and search result is combined into a feature vector , which is the input of the neural network , and a ranking priority score is the output . Another approach learns the matching mechanism between the query and the search result , which is particularly suitable for image retrieval . Usually the mechanism is represented by a similarity matrix which outputs a bilinear form as the ranking priority score ; for instance , such a structure is applied in Severyn & Moschitti ( 2015 ) . We postulate that it could be beneficial to apply multiple embeddings of the queries and search results to a learning to rank model . It has already been observed that for training images , applying a committee of convolutional neural nets improves digit and character recognition Ciresan et al . ( 2011 ) ; Meier et al . ( 2011 ) . From such an approach , the randomness of the architecture of a single neural network can be effectively reduced . For training text data , combining different techniques such as tf-idf , latent Dirichlet allocation ( LDA ) Blei et al . ( 2003 ) , or word2vec Mikolov et al . ( 2013 ) , has also been explored by Das et al . ( 2015 ) . This is due to the fact that it is relatively hard to judge different models a priori . However , we have seen no literature on designing a mechanism to incorporate different embeddings for ranking . We hypothesize that applying multiple embeddings to a ranking neural network can improve the accuracy not only in terms of “ averaging out ” the error , but it can also provide a more robust solution compared to applying a single embedding . For learning to rank , we propose the application of the attention mechanism Bahdanau et al . ( 2015 ) ; Cho et al . ( 2015 ) , which is demonstrated to be successful in focusing on different aspects of the input so that it can incorporate distinct features . It incorporates different embeddings with weights changing over time , derived from a recurrent neural network ( RNN ) structure . Thus , it can help us better summarize information from the query and search results . We also apply a decoder mechanism to rank all the search results , which provides a flexible list-wise ranking approach that can be applied to both image retrieval and text querying . Our model has the following contributions : ( 1 ) it applies the attention mechanism to listwise learning to rank problems , which we think is novel in the learning to rank literature ; ( 2 ) it takes different embeddings of queries and search results into account , incorporating them with the attention mechanism ; ( 3 ) double attention mechanisms are applied to both queries and search results . Section 2 reviews RankNet , similarity matching , and the attention mechanism in details . Section 3 constructs the attention-based deep net for ranking , and discusses how to calibrate the model . Section 4 demonstrates the performance of our model on image retrieval and text querying data sets . Section 5 discusses about potential future research and concludes the paper . 2 LITERATURE REVIEW . To begin with , for RankNet , each pair of query and search result is turned into a feature vector . For two feature vectors x0 ∈ Rd0 and x′0 ∈ Rd0 sharing the same query , we define x0 ≺ x′0 if the search result associated with x0 is ranked before that with x′0 , and vice versa . For x0 , { x1 = f ( W0x0 + b0 ) ∈ Rd1 , x2 = f ( W1x1 + b1 ) ∈ Rd2 = R , and similarly for x′0 . Here Wl is a dl+1 × dl weight matrix , and bl ∈ Rdl+1 is a bias for l = 0 , 1 . Function f is an element-wise nonlinear activation function ; for instance , it can be the sigmoid function σ ( u ) = eu/ ( 1 + eu ) . Then for RankNet , the ranking probability is defined as P ( x0 ≺ x′0 ) = e x2−x′2/ ( 1 + ex2−x ′ 2 ) . Therefore the ranking priority of two search results can be determined with a two-layer neural network structure , offering a pairwise approach . A deeper application of RankNet can be found in Song et al . ( 2014 ) , where a five-layer RankNet is proposed , and each data example is weighed differently for each user in order to adapt to personalized search . A global model is first trained with the training data , and then a different regularized model is adapted for each user with a validation data set . A number of models similar to RankNet has been proposed . For instance , LambdaRank Burges et al . ( 2006 ) speeds up RankNet by altering the cost function according to the change in NDCG caused by swapping search results . LambdaMART Burges ( 2010 ) applies the boosted tree algorithm to LambdaRank . Ranking SVM Joachims ( 2002 ) applies the support vector machine to pairs of search results . Additional models such as ListNet Cao et al . ( 2007 ) and FRank Tsai et al . ( 2007 ) can be found in the summary of Liu ( 2011 ) . However , we are different from the above models not only because we integrate different embeddings with the attention mechanism , but also because we learn the matching mechanism between a query and search results with a similarity matrix . There are a number of papers applying this structure . For instance , Severyn & Moschitti ( 2015 ) applied a text convolutional neural net together with such a structure for text querying . For image querying , Wan et al . ( 2014 ) applied deep convolutional neural nets together with the OASIS algorithm Chechik et al . ( 2009 ) for similarity learning . Still , our approach is different from them in that we apply the attention mechanism , and develop an approach allowing both image and text queries . We explain the idea of similarity matching as follows . We take a triplet ( q , r , r′ ) into account , where q denotes an embedding , i.e . vectorized feature representation of a query , and ( r , r′ ) denotes the embeddings of two search results . A similarity function is defined as SW ( q , r ) = qTWr , and apparently r ≺ r′ if and only if SW ( q , r ) > SW ( q , r′ ) . Note that we may create multiple deep convolutional nets so that we obtain multiple embeddings for the queries and search results . Therefore , it is a question how to incorporate them together . The attention mechanism weighs the embeddings with different sets of weights for each state t , which are derived with a recurrent neural network ( RNN ) from t = 1 to t = T . Therefore , for each state t , the different embeddings can be “ attended ” differently by the attention mechanism , thus making the model more flexible . This model has been successfully applied to various problems . For instance , Bahdanau et al . ( 2015 ) applied it to neural machine translation with a bidirectional recurrent neural network . Cho et al . ( 2015 ) further applied it to image caption and video description generation with convolutional neural nets . Vinyals et al . ( 2015 ) applied it for solving combinatorial problems with the sequence-to-sequence paradigm . Note that in our scenario , the ranking process , i.e . sorting the search results from the most related one to the least related one for a query , can be modeled by different “ states. ” Thus , the attention mechanism helps incorporating different embeddings along with the ranking process , therefore providing a listwise approach . Below we explain our model in more details . 3 MODEL AND ALGORITHM . 3.1 INTRODUCTION TO THE MODEL . Both queries and search results can be embedded with neural networks . Given an input vector x0 representing a query or a search result , we denote the l-th layer in a neural net as xl ∈ Rdl , l = 0 , 1 , . . . , L. We have xl+1 = f ( Wlxl + bl ) , l = 0 , 1 , . . . , L− 1 where Wl is a dl+1 × dl weight matrix , bl ∈ Rdl+1 is the bias , and f is a nonlinear activation function . If the goal is classification with C categories , then ( P ( y = 1 ) , . . . , P ( y = C ) ) = softmax ( WLxL + bL ) , where y is a class indicator , and softmax ( u ) = ( eu1/ ∑d i=1 e ui , . . . , eud/ ∑d i=1 e ui ) for u ∈ Rd . From training this model , we may take the softmax probabilities as the embedding , and create different embeddings with different neural network structures . For images , convolutional neural nets ( CNNs ) LeCun et al . ( 1998 ) are more suitable , in which each node only takes information from neighborhoods of the previous layer . Pooling over each neighborhood is also performed for each layer of a convolutional neural net . With different networks , we can obtain different embeddings c1 , . . . , cM . In the attention mechanism below , we generate the weights αt with an RNN structure , and summarize ct in a decoder series zt , etm = fATT ( zt−1 , c m , αt−1 ) , m = 1 , . . . , M , αt = softmax ( et ) , ct = ∑M m=1 αtmc m , zt = φθ ( zt−1 , ct ) . Here fATT and φθ are chosen as tanh layers in our experiments . Note that the attention weight αt at state t depends on the previous attention weight αt−1 , the embeddings , and the previous decoder state zt−1 , and the decoder series zt sums up information of ct up to state t. As aforementioned , given multiple embeddings , the ranking process can be viewed as applying different attention weights to the embeddings and generating the decoder series zt , offering a listwise approach . However , since there are features for both queries and search results , we consider them as separately , and apply double attention mechanisms to each of them . Our full model is described below .

Summary: The paper proposed an attention-based deep neural network for implementing 'learning to rank' algorithm. Particularly, the proposed method implements a listwise approach which outputs the ranks for all search results given a query. The search results are claimed to be sorted by their degree of relevance or importance to the query. However, it is not clear to me how the ranking was decided in equation 6 by the softmax function. For example, as per section 4, the documents of the same topic are considered related, then how the proposed model was trained with one document having higher relevance than others in the same topic category.