AN ATTENTION-BASED DEEP NET FOR LEARNING TO RANK
Authors: Diego Klabjan
Baiyang Wang
Venue:  (2020)
Keywords: learning to rank
deep learning

Abstract: In information retrieval, learning to rank constructs a machine-based ranking model which given a query, sorts the search results by their degree of relevance or importance to the query. Neural networks have been successfully applied to this problem, and in this paper, we propose an attention-based deep neural network which better incorporates different embeddings of the queries and search results with an attention-based mechanism. This model also applies a decoder mechanism to learn the ranks of the search results in a listwise fashion. The embeddings are trained with convolutional neural networks or the word2vec model. We demonstrate the performance of this model with image retrieval and text querying data sets.

Under review as a conference paper at ICLR 2020
AN ATTENTION-BASED DEEP NET FOR LEARNING TO RANK
Anonymous authors Paper under double-bling review
ABSTRACT
In information retrieval, learning to rank constructs muchine-basede ranking model which given a query, sorts the search results by their degree of relevance or importance to the query. Neural networks have been successfully applied to this problem, and in this paper, we propose an attention- based deep neural network which better incorporates different embeddings of the queries and search results with an attention- learn the ranks of the search results listwise fashion. The embeddings are trained with convolutional neural networks or the word2vec model. We demonstrate the performance of this model with image retrieval and text querying data sets.
mechanism. This model also applies a decoder mechanism
INTRODUCTION
Learning to rank applies supervised sumi-super machine learning construct ranking models for information retrieval problems. In learning to rank, a query is given and a number of search results be ranked by their relevant importance given the query. Many problems in information retrieval can be formulated or partially solved by learning to rank. In learning to rank, there are typically three approaches: the pointwise, pairwise, and listwise approaches Liu (201T). The pointwise approach assigns an importance score to each pair of query and search result. The pairwise approach discerns which search result is more relevant for certain query and a pair of search results. The listwise approach outputs the ranks for all search results given a specific query, therefore being the most general.
For learning to rank, neural networks are known to enjoy a success. Generally in such models, neural networks are applied to model the ranking probabilities with the features of queries and search results as the input. For instance, RankNer Burges et al. (2005) applies a neural network to calculate probability for any search result being more relevant compared to another. Each pair of query and search result is combined into feature vector, which is the input of the neural network, and ranking priority score is the output. Another approach learns the matching mechanism between the query and the search result, which is particularly suitable for image retrieval. Usually the mechanism is represented by a similarity matrix which outputs a bilinear form as the ranking priority score; for instance, such a structure is applied in [Severyn & Moschitti (2015).
We postulate that could be beneficial to apply multiple embeddings of the queries and search results to a learning to rank model. It has already been observed that for training images, applying committee of convolutional neural nets improves digit and character recognition Ciresan et al. (201I): Meier al. (20TT) From such an approach, the randomness of the architecture of a single neural network can be effectively reduced. For training text data, combining different techniques such as tf-idf, latent Dirichlet allocation (LDA) et al. 2003), or word2vec Mikolov et al. (2013), has also been explored by Das et al.) (2015). This is due to the fact that it is relatively hard to judge different models a priori. However, we have seen no literature on designing a mechanism to incorporate different embeddings for ranking. We hypothesize that applying multiple embeddings to a ranking neural network can improve the accuracy not only in terms of "averaging out" the error, but it can also provide more robust solution compared to applying single embedding.
For learning to rank, we propose the application of the attention mechanism Bahdanau et al. (015): ; Cho et al. (2015), which is demonstrated to be successful in focusing different aspects of the input so that it can incorporate distinct features. incorporates different embeddings with weights changing over time, derived from recurrent neural network (RNN) structure. Thus, it can help us

---

Under review as a conference paper at ICLR 2020
better summarize information from the query and search results. We also apply decoder mechanism to rank all the search results, which provides a flexible list ranking approach that can be applied to both image retrieval and text querying. Our model has the following contributions: (1) it applies the attention mechanism to listwise learning to rank problems, which we think is novel in the learning to rank literature; (2) it takes different embeddings of queries and search results into account, incorporating them with the attention mechanism; (3) double attention mechanisms are applied to both queries and search results.
Section 2 reviews RankNet, similarity matching, and the attention mechanism in details. Section constructs the attention- based deep net for ranking, and discusses how to calibrate the model. Section 4 demonstrates the performance of our model on image retrieval and text querying data sets. Section 5 discusses about potential future research and concludes the paper.
LITERATURE REVIEW
To begin with, for RankNet, each pair of query and search result is turned into a feature vector. For two feature vectors To E TRd and v/ E IR sharing the same query, we define xo < zó if the search result associated with zo is ranked before that with 2/y, and vice versa. For zo.
11 f(Wo20 + bu) R x22 = f(W] + b1) E R2
and similarly for s/g Here Wi is a de+1 x de weight matrix, and by E Rd+++ is a bias for l = 0, 1. Function f is an element-wiss wise nonlinear activation function; for instance, it can be the sigmoid
function (u) = =e"/( ee"(11 + e²).. Then for RankNet, the ranking probability is defined as P(zo < an)) - c 100===/(1++ =22-2))) Therefore the ranking priority of two search results can be determined with a two-layer neural network structure, offering a pairwise approach. A deeper application of RankNet can be found in Song et al. (2014), where five-layer RankNet is proposed, and each data example is weighed differently for each user in order to adapt to personalized search. A global model is first trained with the training data, and then different regularized model is adapted for each user with a validation data set.
A number of models similar to RankNet has been proposed. For instance, LambdaRank Burges et al. (2006) speeds up RankNet by altering the cost function according to the change in NDCG caused by swapping search results. LambdaMART Burges (2010) applies the boosted tree algorithm to LambdaRank. Ranking SVM [Jachims) (2002) applies the support vector machine to pairs of search results. Additional models such as ListNet et al. (2007) and FRank Tsai et al. (2007) can be found in the summary of Liu (2011).
However, we are different from the above models not only because we integrate different embeddings with the attention mechanism, but also because we learn the matching mechanism between a query and search results with similarity matrix. There are number of papers applying this structure. For instance, Sevvrrn & Moschitti (2015) applied a text convolutional neural net together with such structure for text querying. For image querying, Wan et al. (2014) applied deep convolutional neural nets together with the OASIS algorithm Chechik et al. (2009) for similarity learning. Still, our approach is different from them in that we apply the attention mechanism, and develop an approach allowing both image and text queries.
We explain the idea of similarity matching as follows. We take a triplet (q, r") into account, where q denotes an embedding, i.e. vectorized feature representation of query, and (r, r') denotes the embeddings of two search results. similarity function is defined as Sw (g. r) WWr, and apparently < r' if and only if (w, > Sw (w.rr').
Note that we may create multiple deep convolutional nets so that we obtain multiple embeddings for the queries and search results. Therefore, it is question how to incorporate them together. The attention mechanism weighs the embeddings with different sets of weights for each state t, which are derived with a recurrent neural network (RNN) from t = 1 to t = T. Therefore, for each state t, the different embeddings can be "attended" differently by the attention mechanism, thus making the model more flexible. This model has been successfully applied to various problems. For instance, Bahdanau et al. (2015) applied it to neural machine translation with a bidirectional recurrent neural network. Chho et al. (2015) further applied it to image caption and video description generation with
2

---

Under review as a conference paper at ICLR 2020
convolutional neural nets. Vinyals et al. (2015) applied it for solving combinatorial problems with the aequence-to-sequeee paradigm.
Note that in our scenario, the ranking process, i.c. sorting the search results from the most related one to the least related one for a query, can be modeled by different "states." " Thus, the attention mechanism helps incorporating different embeddings along with the ranking process, therefore providing listwise approach. Below we explain our model in more details.
MODEL AND ALGORITHM
3.1 INTRODUCTION TO THE MODEL
Both queries and search results can be embedded with neural networks. Given an input vector To representing a query or search result, we denote the 1-th layer in neural net as x E R4, 1= 0, ...... .. L. We have 21+1 f(W + bt), 1 1..... L where Wi is d++1 di weight matrix, b, E Rd44 is the bias, and f is nonlinear activation function. If the goal is classification with categories, then (P(y = 1). .................... P(y = C)) = so softmmmm tmma (Wr, + by.), where y is class indicator, and softmar( = (e"
e / eth) for u E R
From training this model, we may take the softmax probabilities the embedding, and create different embeddings with different neural network structures. For images, convolutional neural nets (CNNs) LeCun et al. (1998) are more suitable, in which each node only takes information from neighborhoods of the previous layer. Pooling over each neighborhood is also performed for each layer of a convolutional neural net.
With different networks, we can obtain different embeddings c!...., C,M In the attention mechanism below, we generate the weights o₁ with an RNN structure, and summarize et in a decoder series 2,,
c0",,,,,,,0111 00-1), -..... M, 04 so softm (lmaaa(e)).
Ctm
C: Em=1 atmem
24 (0(2)-1,
Here SATT and the are chosen as tanh layers in our experiments. Note that the attention weight oz state & depends on the previous attention weight a;-1, the embeddings, and the previous decoder state 2t-1. and the decoder series 2, sums information of ct up to state t.
As aforementioned, given multiple embeddings, the ranking process can be viewed applying different attention weights to the embeddings and generating the decoder series 2t, offering listwise approach. However, since there are features for both queries and search results, we consider them as separately. and apply double attention mechanisms to each of them. Our full model is described below.
3.2 MODEL CONSTRUCTION
For each query Q, we represent M different embeddings of the query as q == (91..... where each gm is multi-dimmesionnl. For the search results RI...., ; Rr associated with Q, which are to be ranked, we can map each result into N different embeddings with same structure, and obtain r. = (Fri,...., TTN) as the embeddings of Rr / = I.... ... T. Each Ttn is also multi-dimensional. All these embeddings can either correspond to raw data or they can be the output of a parametric model, ie. the output of a CNN in the case of images. In the latter case, they are trained jointly with the rest of the model.
Below we represent queries or search results in their embeddings. We assume that given the query q. the results F1,... .r retrieved in the order fI...., Pr in equation (12) below. Note that "r....,,rf = (7 (f.....,fy! Fr).
We observe that both the query and the results Pi...., PT can be "attended" with different weights To implement the attention- mechanism, we assign different weights for different components of for each different / 1,...; Specifically, assign the attention weights 01, -.. Or for the
3

---

Under review as a conference paper at ICLR 2020
query, and $1...., Br for the search results. To achieve this, need to assign the pre-softmax weights e, and fe, t l....." T. Therefore we first let
Ctm = CATT (2 9m g(r......, ar.: ftn ( (r (T,,.... FT), 11 ,.......... M, 1, ,,..., N, t= I.....T.
(1)
Note that there are two different attention functions. While they both consider the previous weights 0t-1 1 and 31- the pre-softmax weight 6tm considers the m-th component of the query (m and all information from the search results, and Stn considers the opposite. In our experiments, y takes an average while averages over the n-th embedding of each result. Here 2t-1 is the value of the decoder at state 1. Letting e (e:1, eLM), ft (fai...., fiN), we impose the softmax functions for = I,.
a softmar(es), 3r so ffmmaz(f).
The attention weights a and 3i can then be assigned to create the context vectors e for the query and de for the search results, which defined as follows,
M
Otem Im de = Σ
.r).
(3)
m==
n=l
With the context vectors and de depending on the state we can finally output the decoder 2t,
2t = Φ0 00 de), 1
Equations (8-11) are carried out iteratively from = to / T. The hidden states ,1,...,277 together with the context vectors ci..... cr, are used to rank results ri, ri,...,fy. For each rt. we define context vector d which can be directly compared against the context vector e for queries.
de,e Σ Benreay (= 1=,,...,
This context vector dg,,, unlike d, is not only specific for each state L, but also specific for each result t' = 1..... .......T T. Now suppose in terms of ranking that f1.... fz-1 have already been selected. For choosing it, we apply the softmax function for the similarity scores se,e between the query q and each result rzr.
0.8
softmax soore recurrent
of p.
(fnnctinn
NDCGS
NDCG3
attention mochanism
error (%6)
0.6
embediings
0.5
convolutional
layers
0.4
query
search
embeddings
Figure 1: The attention-based model for im- age retrieval.
Figure 2: The rates of AuRN-HL given different numbers of embeddings with 95% upper and lower confidence bounds.
Thus we have
(P(ft = relii,...,,f fr-1) x softmar(8tt.) St.r We, df. V Va, re E fri,..., .r....,TTII... .TM Fr
4

---

Under review as a conference paper at ICLR 2020
Therefore Pi...., Fr can be retrieved in a listwise fashion. Equations (1-6) complete model, which is shown in Figure in the case of image retrieval.
3.3 MODEL CALIBRATION
To calibrate the parameters of this model, apply the stochastic gradient descent algorithm which trains a minibatch, a subsample of the training data at each iteration. To rank search results for the testing data, apply the beam search algorithm Reddy (1977) with beam width of 3, which keeps a number of paths, i.e. sequences of fi,..., ft at state of the highest log-likelihoo.. Other paths trimmed at each state and finally the path with the highest log-likelihood is chosen.
Alternatively, we may consider the hinge loss function to replace the softmax function. Com- paring potentially chosen result iz against any other unranked result ry', with Yy' PT) ff.. P;-11, we apply the hinge loss function, similar to Chechik et al. (2009) r(r,, = ma St,f. + +t,,r, When training the model, for single query q and all related search results, we apply the stochastic gradient descent algorithm to minimize Σ Er, [(Fr, For this method, we maximize St,r,, at each step, i.e. apply beam search with beam width of 1.
DATA STUDIES
We perform studies on three datasets: MNIST presented in Section 4.1 CIFAR-10 listed in Appendix,
and 20 Newsgroups in Section 4.2
4.1 THE MNIST DATA SET
The MNIST data set contains handwritings 0-9 digits. There are 50,000 training, 10,000 validation, and 10,000 testing images. We take each image among them query. For each query image, construct the set of retrieved images as follows. We first uniformly at random sample a random Next we select & related random images (an image is related if it is of the same digit as the query image). Finally, we randomly select 30-k unrelated images (an image is unrelated if it corresponds to a different digit than the query image). The order is imposed so that images of the same digit are considered related and images of different digits not related.
integer & from 2.. 9}.
For this data set, we pretrain convolutional neural networks based on different I,2 regularization rates for all layers and Dropout [Srivastava et al. (2014) regularization for fully connected (FC) layers with respect to the values shown in Table 1. We vary the regularization rate because it can be expensive to determine most suitable regularization rate in practice, and therefore applying the attention mechanism to different regularization rates can reduce the impact of improper regularization rates and find better model.
Table 2: Errors of "mean" " and "max" pooling for MNIST.
Table 1: Different regularization rates applied MNIST.
error
MAP
NDCG3
NDCGs
Dropout's for Cony.
AuRN-
0.44%
0.65%
0.52%
Dropout's 's p for FC
sd.
(0.01%)
(0.02%)
(0.02%)
L s X for All
0 le-5 1o-5
1c-4
1c-4
AIRN--L-max
0.44%
0.66%
0.53%
sd.
(0.01%)
(0.02%)
(0.02%)
The pretrained models are standard digit classification. The softmax layers of the five models as embeddings are plugged into our full model (where the CNNs are trained together with the rest). We initialize the similarity matrix W with the identity matrix and the remaining parameters with zeros. Our model is trained given such embeddings with a batch size of 100, a learning rate of 0.001, and 20 epochs. We choose the best epoch using the validation data
We compare our algorithm against OASIS Chechik et J. (2009). Ranking SVM, and LambdaMART. in the setting of image retrieval problems Wan et al. (2014); Wang et al. (2014); Wu et al. (2013). For OASIS, we set the batch size and maximum learning rate to be the same as our model. For Ranking SVM and LambdaMART, combine the embeddings of query and search result to
5

---

Under review as a conference paper at ICLR 2020
create features, apply sklearn and pyltr in Python, and use NDCG, and 100 trees for LARbdaMART. To compare the results, we apply MAP and p Liu 2011 as the criteria, and calculate the standard deviations averaged over five runs for each model. The error rates, which are minus MAPs and NDCGs, are shown in Table 3. The lowest error rates are in bold. The top row shows the average value while the bottom one (sd.) exhibits the standard deviation for 5 randomized runs.
We name our model as an "Attention- based Ranking Network" (AuRN). In Table 3, "AttNNSM' or "AuRNN-L" denote our model with softmax or hinge loss ranking functions. "OASIS-I" "OASIS-5" denote, respectively. OASIS trained with the first set of embeddings, or the average of all five sets of embeddings. For Ranking SVM and LambdaMART, we use all embeddings. "RSVM-100% means that 10th pairs are sampled for the SVM, "AMART-2%%" means that 2% of all feature vectors are sampled for each tree in LanbdaRank, and so forth.
Table Errors of different image retrieval methods for MNIST.
error
OASIS-1
OASIS-5
AuRN-SM
Aut-N-HI
MAP
0.55%
0.47%
0.46%
0.44%
sd.
(0.02%)
(0.01%)
(0.01%)
(0.01%)
NDCG3
0.81%
0.70%
0.68%
0.65 %
sd.
(0.02%)
(0.01%)
(0.01%)
(0.02%)
NDCGs
0.65%
0.57%
0.55%
0.52%
sd.
(0.02%)
(0.01%)
(0.02%)
(0.02%)
error
RSVM-100k
RSVM-500k
AMART-2%
AMART-5%
MAP
0.52%
0.54%
0.87%
0.88%
sd.
(0.02%)
(0.01 %%)
(0.02%)
(0.03%)
NDCG3
0,74%
0.75%
0.93%
0.96%
sd.
(0.03%)
(0.01%)
(0.01%)
(0.02%)
NDCGs
0.61%
0.62%
0.89%
0,91%
sd.
(0.02%)
(0.01%)
(0.02%)
(0.03%)
From Table 3. we observe that AutN--H outperforms other methods, and AttRN-M achieves the second best. Moreover, OASIS-5 clearly outperforms OASIS-1, which demonstrates the benefit of applying multiple embeddings. Ranking SVM and LambdaMART do not seem very suitable for the data, and increasing the sample size does not seem to improve their performance.
needs to be investigated how sensitive AutRN-L is against the number of embeddings. In Figure 2, we show the eгror rates of AuRN-HL with different numbers of embeddings. Here we apply regularization parameters (1, 0.8, le-4) and (1, 0.8, 2e-4) to the new embeddings. From Figure 2, the error rates steadily decrease the number of embeddings increases, which is consistent with the intuition that adding more information to the model should yield better results. We also observe that the error rates tend to stabilize the number of embeddings increases.
However, while applying 7 embeddings slightly lowers the error rates as shown in Figure 2, the training time is roughly 33% longer than for embeddings. Therefore we consider applying 7 embeddings to be unnecessary, and consider the current practice to be representative of AttRN.
Table 2 shows the differences in error rates by changing the pooling functions g and h2 in (7) from "mean" to "max." We observe that the changes are very small, which means that AuRN is quite robust to such changes.
Table 4: Errors of "mean" and "max" pooling as for MNIST.
CIrOr 0.44%
MAP
NDCGs
NDCG;
AtRN--LLmman
0.65%
0.52%
sd.
(0.01% %)
(0.02%)
(0.02%)
AURN-IHL-maN
0.44%
0.66%
0.53%
sd.
(0.01%)
(0.02%)
(0.02%)
Table 4 shows the changes in weights of the convolutional neural networks. The unregularized CNN is used as an example. The L norms sum over the weights of each layer and are averaged over 5 randomized runs. They tend to increase after AuRN is trained, possibly because it requires the weights to be larger to adjust for the ranking mechanism.

---

Under review as a conference paper at ICLR 2020
Table 5: L norms of each layer in the convolutional neural net before and after training AttRN as for MNIST.
L.
1st Conv.
2nd Conv.
PC
Softmax
Before
4.99
9.66
25.74
7.15
sd.
(0.05)
(0.03)
(0.02)
(0.04)
After
5.26
9.91
25.90
7.81
(0.05)
(0.03)
(0.02)
(0.04)
query/
considered related
unrelated
query/images considered consdddeed related
unrelated
5
5
5
5
$
J
9
9
7
good
5
5
S
5
5
3
8
6
6
good
2
2
2
2
2
2
4
4
0
)
9
3
7
7
7
7 2
7
7
7
7 7 7
2
2
poor
pour
6
5
5
5
5
$
5
5
6
Figure 3: Actual query images and the 1 to 8th retrieved images in the MNIST data set.
Figure 4: Actual query images and the 1st to 7th retrieved images in the CIFAR-10 data set.
Next we display three good cases of retrieval together with three poorer ones from AuRN--L in terms of the actual images, which are shown in Figure We observe that the poorer cases of retrieval in the MNIST data set are basically due to the distortion of the query images.
4.2
THE 20 NEWSGROUPS DATA SET
The 20 Newsgroups data set contains online news documents with 20 different topics. There 11,293 training documents, from which 1,293 are held out for validation, and 7,528 testing documents. The topies can be categorized into 7 superclasses: religion, computer, "for sale," cars, sports, science, and politics. We consider each document as a query, and randomly choose 30 retrieved documents follows: the number of documents of the same topic is uniformly distributed from number of documents of the same superclass but different topics is also uniformly distributed from 3 7. and the remaining documents are of different superclasses. We impose the order on different documents so that: (1) documents with the same topic are considered to be related, and otherwise not; (2) documents with the same superclass are considered be related, and otherwise not. We train our model with the first type of order, and calculate eтrоr rates for both orders.
7, the
We pretrain the documents with three different word2vec- type Mikolov et al. (2013) models: CBOW, skip-gram, and GLOVE Penningtnnn et al. (2014). We note that all these techniques apply neighboring words to predict word occurrences with neural network based on different assumptions. There- fore, applying the attention mechanism to incorporate the three models weaken the underlying distributional assumptions and result in more flexible word embedding structure.
We apply text8 as the training corpus for CBOW and skip-gram, and download the pretrained weights for GLOVE from Wikipedia 2014 and Gigaword 5. Each vector representing a word of dimension represent it tf-idf vector Aggarwal & Zhai (2012) and apply the word2vee weights to transform it into a 300-dimensional vector, which is the same as VecAvg in
300. For each document,
Socher et al. (2013). For this instance, the embeddings fixed and not modified during the training
7

---

Under review as a conference paper at ICLR 2020
process of our model. We also impose softmax layer corresponding to the 20 classes top of word2vec and use the resulting final weights as pretrained weights to AttNN. We compare our model against OASIS well Ranking SVM and LambdaMART. The results of RankNet are not shown here because we have found to be unsuitable for this data set.
For AttRN and OASIS, we apply batch size of 100 and learning rate of 0.0001, and 50 epochs. Other computational details are the same as in the previous data sets. Table considers any two documents of the same topic be related, while Table 9 considers any documents of the same superclass to be related.
Table Errors of different text retrieval methods for 20 Newsgroups with regard different topics.
emror
OASIS-1
OASIS-3
AuRN-SM
AuRN-HL
MAP
15.56%
15.07%
14.78%
14.79%
sd.
(0.01%)
(0.01%)
(0.01%)
(0.004%)
NDCG,
17.83%
17.25%
16,87% %
16,93%
sd.
(0.02%)
(0.02%)
(0.02%)
(0.01%)
NDCG,
17.16%
16.59%
16.28%
16.29%
sd.
(0.02%)
(0.02%)
(0.01%)
(0.02%)
eтror
RSVM-20K
RSVM-100k
AMART-2%
AMART-5%
MAP
23.24%
19.98%
32.90%
31.58%
sd.
(0.60%)
(0.10%)
(1.20%)
(0.87%)
NDCG;
24.35%
21.20%
34.62%
33.55%
sd.
(0.20%)
(0.05%)
(1.26%)
(0.88%)
NDCG,
24.73%
21.37%
35.33%
34.04%
sd.
(0.48%)
(0.08%)
(1.27%)
(0.95%)
Table 7: Errors of different text retrieval methods for 20 Newsgroups with regard different superclasses.
error
OASIS-1
OASIS-3
AuRN-SM
AuRN-HI
MAP
30.58%
30.15%
30.14%
30.195
sd.
(0.02%)
(0.01%)
(0.01%)
(0.04%)
NDCG,
13.59%
13.14%
12.93%
12.95%
sd.
(0.02%)
(0.02%)
(0.02%)
(0.01%)
NDCG,
18.36%
17.92%
17.82%
17.79%
sd.
(0.03%)
(0.01%)
(0.01%)
(0.01 %))
eтror
RSVM-20K
RSVM-100k
AMART-2%
AMART-5%
MAP
35.87%
34.03%
41.52%
40,27%
sd.
(0.30%)
(0.06%)
(0.91%)
(0.55%)
NDCGs
19.61%
16.83%
29.35%
28.34%
sd.
(0.25%)
(0.04%)
(1.24%)
(0.63%)
NDCG,
24.92%
22.10%
34.05%
32.65%
sd. (0.38%)
(0.08%)
(L.14%)
(0.70%)
Tables 8 and show that AuRN-SM is the most suitable for 20 Newsgroups while AutRN-H achieves the second best, and relatively close. We also observe that MAP for superclasses too strict for this data set, because all documents of the same superclass are taken into account. NDCG scores are more realistic because a threshold is given and less related search results are weightei...ed.
To conclude, AuRN--L has the most robust performance across all three data sets, and is thus the recommended choice. Moreover, for the problems considered in this work, OASIS is more suitable than Ranking SVM and LambdaMART, and thus competitive baseline.
5 CONCLUSION
In this paper, we proposed a new neural network for learning-to-rante problems which applies the attention mechanism to incorporate different embeddings of queries and search results, and ranks the search results with a listwise approach. Data experiments show that our model yields improvements over state-of of- fotte-of-theaart the-art techniques. For the future, it would be of interest to consider improving the RNN structure in the attention mechanism, and tailoring the embedding part of the neural network to this problem.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
C. Aggarwal and C. Zhai (eds.). Mining text data. Springer, New York, USA, 2012.
D. Bahdanau, K. Cho, and Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations, San Diego, USA, 2015.
D. Blei, A. Ng, and M. Jordan. Latent dirichlet allocation. Journal of Machine Learning Research, 3: 993-1022, 2003.
C. Burges. From ranknet to lambdarank to lambdamart: An overview. Technical report, June 2010.
C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to rank using gradient descent. In Proceedings of the 22th International Conference on Machine Learning (ICML 2005), Bonn, Germany, 2005.
C. Burges, Regno, and Q Le. Learning to rank with nonsmooth cost functions. In Advances in Neural Information Processing Systems 19 (NIPS 2006), Vancouver, Canada, 2006.
Z. Cao, T. Qin, T. Liu, M. Tsai, and H. Li. Learning to rank: from pairwise approach to listwise approach. In Proceedings of the 24th International Conference on Machine Learning (ICML 2007). Corvalis, Oregon, USA, 2007.
G. Chechik, V. Sharma, U. Shalit, and S. Bengio. An online algorithm for large scale image similarity learning. In Advances Neural Information Processing Systems 22 (NIPS 2009), Vancouver, Canada, 2009.
K. Cho, A. Courville, and Y. Bengio. Describing multimedia content using iitention-baseed encoder- decoder networks. IEEE Transactions on Multimedia, 17(11), 2015.
D. Ciresan, Meier, L. Gambardella, and J. Schmidhuber. Convolutional neural network committees for handwritten character classification. In ICDAR, pp. 1250-1254, 2011.
R. Das, M. Zaheer, and C. Dyer. Gaussian Ida for topic models with word embeddings. In The 2015 Conference the Association for Computational Linguistics (ACL 2015), 2015.
T. Joachims. Optimizing search engines using clickthrough data. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '02, pp. 133- New York, NY, USA, 2002. ACM.
Y. LeCun, I.. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11))2278-233) 1998.
Liu. Learning to var information retrieval. Springer Berlin Heidelberg, Germany, 2011.
U. Meier, D. Ciresan, Gambardella, and J. Schmidhuber. Better digit recognition with committee of simple neural nets. In ICDAR, pp. 1135-1139, 2011.
T. Mikolov, Sutskever, K. Chen, G. Corrado, and Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26 (NIPS 2013), Lake Tahoe, Nevada, USA, 2013.
J. Pennington, R. Socher, and C. Manning. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar, 2014.
R. Reddy. Speech understanding systems: summary of results of the five-year research effort carnegie-mellon university. Technical Report, 1977.
A. Severyn and A. Moschitti. Learning to rank short text pairs with convolutional deep neural networks. In Proceedings of the 38th International ACM SIGIR Conference Research and Development in Information Retrieval, Santiago, Chile, 2015.
R. Socher, A. Perelygin, J. Wu, J. Chuang, C. Manning, A. Ng. and C. Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Stroudsburg, PA, October 2013. Association for Computational Linguistics.

---

Under review as a conference paper at ICLR 2020
Y. Song, H. Wang, and X. He. Adapting deep ranknet for personalized search. In The 7th ACM WSDM Conference, New York City, USA, 2014.
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 14: 1929-1958, 2014.
M. Tsai, T. Liu, T. Qin, H. Chen, and W. Ma. Frank: a ranking method with fidelity loss. In The 30th Annual International ACM SIGIR Conference, Amsterdam, the Netherlands, 2007.
O. Vinyals, M. Fortunato, and N. Jaitly, Pointer networks. In Advances in Neural Information Processing Systems 28 (NIPS 2015), Montreal, Canada, 2015.
J. Wan, D. Wang. S. Hoi, P. Wu, J. Zhu, Y. Zhang, and J. Li. Deep learning for content- image retrieval: a comprehensive study. Proceedings of the 22nd ACM international conference on Multimedia, Orlando, Florida, USA, 2014.
Wang, Song, Leung, C. Rosenberg, Wang, J. Philbin, B. Chen, and Y. Wu. Learning fine-grained image similarity with deep ranking. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, Ohio, USA, 2014.
P. Wu, S. Hoi, H. Xia, P. Zhao, D. Wang. and C. Miao. Online multimodal deep similarity learning with application to image retrieval. Proceedings of the 21st ACM international conference on Multimedia, Barcelona, Spain, 2013.
A APPENDIX
A.1 THE CIFAR-10 10 DATA SET
The CIFAR-IO 10 data set contains images of 10 different classes. There are 50,000 training images, from which we take 5,000 images for validation, and 10,000 testing images. Again we took each image as a query, and randomly created 30 search results with the number of related images uniformly distributed from to 9. The order is imposed in the same way as MNIST except that it is based on different classes. We use convolutional neural networks in the same way as MNIST based on regularization rates shown in Table 5.
Table 8: Different regularization rates applied to CIFAR-10.
Dropout's 's p for Cony.
0.25
0.25
0.25
0.1
0.1
Dropout's s for FC
0.5
0.3
0.1
0.3
0.1
L.'ss
5c-4
5c-4
5c-4
5e-4
5c-4
We apply a batch size of 50, a learning rate of 0.0005 and 20 epochs to train AtRN, Other compu- tational details the same in the MNIST data set. The error rates and the standard deviations are shown Table 6. AttRN-HL. again achieves the best performance followed by AttRN- and OASIS-5 in terms of MAP.
10

---

Under review as a conference paper at ICLR 2020
Table 9: Errors of different image retrieval methods for CIFAR-10.
error
OASIS-1
OASIS-5
AIRN-SM
AuRRN-HI
MAP
16.46%
13.72%
13.65%
13.41
sd. (0.08%)
(0.08%)
(0.06%)
(0.07%)
NDCGs
19.19%
15.95%
15.93%
15.61%
sd.
(0.11%)
(0.08%)
(0.06%)
(0.08%)
NDCGs
17.94%
14.86%
14.80%
14.55%
sd.
(0.06%)
(0.08%)
(0.08%)
(0.07%)
error
RSVM-TOOk
RSVM-500K
AMART-2%
AMART-5%
MAP
13.74%
13.77%
18.49%
19.27%
sd.
(0.20%)
(0.13%)
(0.06%)
(0.14%%
NDCGg
15.91%
15.87%
19.34%
20.05%
sd.
(0.20%)
(0.15%)
(0.22%)
(0.46%%
NDCGs
14.78%
14.81%
18.71%
19.28%
sd.
(0.20%)
(0.14%)
(0.11%)
(0.26%%
Table 7 also shows that replacing the pooling functions from "mean" to "max" again yields very small changes eπror rates, which means that AttR is quite robust.
Table 10: Errors of "mean" and "max" pooling as for CIFAR-10.
emror
MAP
NDCG;
NDCG; 14.55%
AL-Na-H--mean
13.41%
15.61%
sd.
(0.07%)
(0.08%)
(0.07%)
AuRR HL-
13.40%
15.61%
14.54%
sd.
(0.07%)
(0.08%)
(0.07%)
Again display three good cases of retrieval and three poorer from AttRN-HL in terms of the actual images, shown in Figure 4 below. We observe that the images considered as related tend to be in the same superclass the query, but may be in different class due to the variation within each class of images and blurred images.


Reviews:
[{"review_id": "rJedKG9y5B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use attention mechanism for combining different embeddings of the queries and search results. Besides, a decoder mechanism is used to do listwise ranking for the results. The experiments show that the proposed approach outperforms some classic learning-to-rank baselines.\n\nThis paper is below the bar of acceptance for the following reasons:\n\n1.\tLimited technical contribution: some previous papers have explored the idea of learning attention weights for combining different embeddings, and simply applying this idea to learning-to-rank application does not seem to be a big contribution.\n\n2.\tChoice of datasets: the datasets used in this paper are typically used for tesing classification models rather than ranking models. In these datasets, for each query image/doc, there are many images/docs of the same class that could be considered relevant, which makes the ranking task less challenging. Since the paper focuses on learning-to-rank problem, probably the authors should consider include more datasets dedicated to learning-to-rank problems.\n\n3.\tInsufficient baselines: the baseline methods used in the paper are not very recent (e.g., OASIS, RankSVM and LambdaMart have been proposed for more than 10 years). There have been many neural-network based retrieval/ranking methods proposed in the past 5 years. Hence, the experimental results could be more convincing if the paper include more \n\n4.\tLack of justification for the model architecture: some design choices of the model are not well-motivated/justified. For example, how does the decoder mechanism using multiple states in the model (listwise) help improve the ranking results compared to pairwise ranking? Ablation study could help whether such decoder mechanism help show the usefulness of this module.\n\n5.\tParameter sensitivity study: study on how hyper-parameter values affects the model performance could also help.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BygEsdrNqr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed an attention-based deep neural network for implementing 'learning to rank' algorithm. Particularly, the proposed method implements a listwise approach which outputs the ranks for all search results given a query. The search results are claimed to be sorted by their degree of relevance or importance to the query. However, it is not clear to me how the ranking was decided in equation 6 by the softmax function. For example, as per section 4, the documents of the same topic are considered related, then how the proposed model was trained with one document having higher relevance than others in the same topic category.  \n\nThere are other confusions that need to be addressed for better understanding. For example, how softmax probabilities can be used as an embedding as described in the line: “From training this model, we may take the softmax probabilities as the embedding, and create different embeddings with different neural network structures. ” Also, what does the line means: “the number of documents of the same topic is uniformly distributed from 3 to 7, the number of documents of the same superclass but different topics is also uniformly distributed from 3 to 7, and the remaining documents are of different super classes.”  \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxrToa2YB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose to use attention to combine multiple input representations for both query and search results in the learning to rank task. When these representations are embeddings from differentiable functions, they can be jointly learned with the neural network which predicts rankings. A limited set of experiments suggest the proposed approach very mildly outperforms benchmark approaches.\n\nMajor comments\n\nTo the best of my knowledge, this is the first paper to apply attention to the learning to rank problem. However, the main methodological innovation seems to be the use of attention to create and train an ensemble of models; this has been previously explored in the literature (e.g., [Kim et al., ECCV 2018]).\n\nThe paper is also missing important context in that it omits developments in using deep learning for the learning to rank problem (e.g., [Pang et al., CIKM 2017; Ai et al., WWW 2018]). The experimental evaluation does not include any other deep methods; thus, it is not clear if the (very minor) improvement in performance are due to the deep models or the proposed attention approach.\n\nThe datasets used in the experiments are not appropriate for evaluating learning to rank algorithms. A variety of learning to rank datasets are available, and these should be used rather than (or in addition to) the toy datasets considered here. Examples: http://arogozhnikov.github.io/2015/06/26/learning-to-rank-software-datasets.html, http://quickrank.isti.cnr.it/istella-dataset/, https://www.cl.uni-heidelberg.de/statnlpgroup/nfcorpus/, \n\nMinor comments\n\nConcerning Section 3.3, in what sense is SGD used to “calibrate” the model? It seems as though the authors just mean it is used to “train” the model. However, is there some other meaning of calibration (e.g., in the sense of a Brier score) here?\n\nIn Table 1, what is the meaning of a dropout p value of 1? In most deep learning frameworks (e.g., Keras and PyTorch), this would mean all nodes are dropped out.\n\nIn what sense are the “5 randomized runs” for the experiments randomized? Are different train, test splits used? or just different random seeds? or something else?\n\nHow is it that the error rates are higher when using superclasses for evaluation?\n\nTypos, etc.\n\nThe paper has several significant problems with the “\\cite”s and “\\ref”s in the paper. First, the “\\cite”s should presumably be “\\citep”s or something since the references are not set off from the rest of the text. Second, the paper includes references to equation numbers which are not present in the paper, such as “equation (12)”. It seems as the equations are in the paper, but are included in some unnumbered environment (“\\begin{align*}” or some such). This makes it very difficult to track down to which equations the authors intend to refer. Third, the reference numbers to figures and tables in the text is wrong. For example, the text refers to “Tables 8 and 9” for 20 newsgroups (at the end of Section 4). Clearly, this is supposed to be Tables 6 and 7. It seems like the authors moved the CIFAR-10 discussion to the appendix but did not update the references in the text.\n\nTables 2 and 4 are exactly the same.\n\nFigure 4 is not referenced in the text.\n\nIt would be helpful to put Figure 2 a bit closer to where it is discussed in the text.\n\nThe references are not consistently formatted.\n\n“components of for each” -> “components for each”\n\nPlease define acronyms like MAP at least once.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers felt the paper should be rejected and no rebuttal was offered. So the paper is rejected.
Decision: Reject