In defense of dual-encoders for neural ranking
Authors: Aditya Krishna Menon
Sadeep Jayasumana
Seungyeon Kim
Ankit Singh Rawat
Sashank J. Reddi
Sanjiv Kumar
Venue: ICLR 2022 Submitted (2022)
Keywords: cross-attention
dual encoder
neural ranking
distillation

Abstract: Transformer-based models such as BERT have proven successful in information retrieval problem,  which seek to identify relevant documents for a given query. There are two broad flavours of such models:  cross-attention (CA) models, which learn a joint embedding for the query and document, and dual-encoder (DE) models, which learn separate embeddings for the query and document. Empirically, CA models are often found to be more accurate, which has motivated a series of works seeking to bridge this gap. However, a more fundamental question remains less explored:  does this performance gap reflect an inherent limitation in the capacity of DE models, or a limitation in the training of such models? And does such an understanding suggest a principled means of improving DE models? In this paper, we study these questions, with three contributions. First, we establish theoretically that with a sufficiently large embedding dimension, DE models have the capacity to model a broad class of score distributions. Second, we show empirically that on real-world problems,  DE models may overfit to spurious correlations in the training set, and thus under-perform on test samples. To mitigate this behaviour, we propose a suitable distillation strategy, and confirm its practical efficacy on the MSMARCO-Passage and Natural Questions benchmarks.

Under review as a conference paper at ICLR 2022
IN DEFENSE OF DUAL-ENCODER, FOR NEURAL RANKING
Anonymous authors Paper under double-blind review
ABSTRACT
Transformer-basee models such as BERT have proven successful in information retrieval problem, which seek to identify relevant documents for a given query. There are two broad flavours of such models: cross-attention (CA) models, which learn a joint embedding for the query and document, and dual-encoder (DE) models, which learn separate embeddings for the query and document. Empirically, CA models are often more accurate, which has motivated a series of works seeking to bridge this gap. However. a fundamental question remains less explored: does this performance gap reflect a limitation in the capacity of DE models, or in the training of such models? First, we establish theoretically that with a sufficiently large encoder size, DE models can capture a broad class of scores without cross-attention. Second, we show empirically that on real-world problems, the gap between CA and DE models may be due to the latter overfitting to the training set. To mitigate this behaviour, we propose distillation strategy that focuses preserving the ordering amongst documents, and confirm its efficacy on benchmark neural re-ranking datasets.
this paper, we study this question, with three contributions.
INTRODUCTION: TRANSFORMER-BANE BASED NEURAL RANKING
Information retrieval (Mitra& & Craswell, 2018 8) is the classic problem of identifying relevant documents for given query. Typically, such retrieval is performed in two-step manner (Matveeva et al.. 2006): one uses an initial model to efficiently retrieve a candidate set of documents for query, and then uses a second model to re-rank these candidates. The retrieval stage is generally implemented by a model with low inference cost, for which candidate document generation is tractable; if this candidate set is reasonably small, it is feasible to use more complex models in the re-ranking stage.
Transformer- based models such as BERT (Devlin et al., 2019) have proven successful both the retrieval and re-ranking stages. Such nearal ranking models have two flavours: dual-encoder (DE) models (Lee et al., 2019; Chang et al., 2020; Karpukhin et al., 2020; Xiong et al., 2021; Luan et al., 2021), which learn separate (factorised) embeddings for the query and document; and cross-attention (CA) models (Nogueira & Cho, 2019; Dai & Callan, 2019; Yilmaz et al., 2019; MacAvancy et al., 2019; Gao et al., 2020), which learn a joint embedding for the query and document. Only DE models are applicable for retrieval, as they admit efficient nearest neighbour search (Guo et al., 2020; Johnson al., 2021) identify candidate documents. Both CA and DE models are applicable for re-ranking; however, empirically, CA models have proven more accurate (Hofstätter et al., 2020a).
Does the re-ranking performance gap between CA and DE models reflect limitation in the inherent capacity of the latter's 's factorised representation, or in the training of DE models? These questions are of conceptual interest, but are also practically relevant: DE models allow for more efficient inference owing to the ability to pre-compute document embeddings, which enables fast similarity search when presented with a candidate query (Khattab & Zaharia, 2020; Hofstätter et al., 2020b). Several works have explored means of improving DE models, including by changing' the scoring layer (Khattab & Zaharia, 2020; MacAvaney et al., 2020; Hofstätter et al., 2020b), and by distilling predictions from a CA model (Lu et al., 2020; Izacard & Grave, 2020; Yang & Seo, 2020; Hofstätter al., 2020a; Miech et al., 2021), I). However, the root cause of the gap between CA and DE models remains elusive.
In this paper, we study these questions, with the following contributions:
Unless otherwise stated, we "DE" mean dual-encoder with dot-product scoring per Equation 2.

---

Under review as a conference paper at ICLR 2022
madel
0.95
bis
0.45
mote:
OST
Embedding dts
128
Emoedding sice
Figure I: Comparison of cross-attention (CA) and dot-product hased dual-encoder (DE) models on the MSMARCO-Passage re-ranking task. Using 6-layer BERT models with varying embedding size (Ture et al., 2019), we report the train and dev set MRR @ 10 averaged over three independent trials. For sufficiently large embedding dimension, the DE model closely matches the performance of the CA on the training set; however, there is a sizable gap in the test set performance. This points to the poorer DE model performance being largely an issue of generalisation, rather than model capacity. Suitable distillation (DIST) from the CA model manages to prevent such overfitting potentially by worsening the training performance and largely bridges the gap between the two models. See $3.2 for details on the experimental setup, and $4 for details our distillation strategy.
we establish theoretically that with a sufficiently large embedding dimension (and under mild assumptions), any continuous ground-trun scores can be modelled by generic DE models (Proposition 1), and in particular by sufficiently deep transformer- Thus, in principle, there is no fundamental restriction in using DE versus CA models for re-ranking.
DE models (Proposition 2).
(ii) we show empirically (Figure 1) that on real-worke problems, CA and DE models can achieve similar training performance, but DE models may do worse on test data due to overfitting ($3.2). Thus, the DE models may suffer due to poorer generalisation ability rather than capacity.
(iii) to mitigate the above, we propose distillation strategy focussing on mimicking the teacher's s ordering amongst documents ($4). This generalises the recent margin MSE loss (Hofstätter et al., 2020a) ()4.1), and justifies the utility of softmax cross-entropy based distillation for re-ranking (Proposition 3, $4.2). We confirm the efficacy of this strategy on the re-reranking benchmarks SAOMARRCO-Passaee (Nguyen et al., 2016) and Natural Questions (Kwiatkowski et al., 2019).
Overall, our results aim to shed light on the fundamental differences between CA and DE models, and give simple yet effective distillation strategy to boost the performance of the latter.
2 BACKGROUND AND NOTATION
Retrieval problems. Suppose we have query space Q and document space D. For a given query q E Q, the goal of information retrieval is to identify set of relevant documents (g) & Craswell, 2018). This is typically achieved by learning a scorer s: x D + R that predicts the relevance of a query and document, and simply selecting the top-k highest scoring documents for a given query. Typically, this scorer is itself implemented in two phases (Matveeva et al., 2006; Nogueira& & Cho, 2019; Chang et al., 2020; Miech et al., 2021; Hofstätter et al., 2021). In the retrieval phase, one identifies an initial candidate set of documents via an initial scorer Syet- In the re-ranking phase, one re-scores only these candidate documents via a distinct scorer 8,ik to obtain Dra( We may thus regard s: 0 x D + R as a suitable composition of both Sret and Strk-
(g) C D (Mitra
To learn a scorer, suppose we have supervision {(g., dn, yn) vn) bi
where for each query 9n we have a list of K associated documents d, = (d.t....,dzy) E D with ground-trute truth relevance labels Yn = (Ix1...., )rK) E {0, 1]K Typically, dn comprises a small subset of D (i.e., K & [D], since it is rarely feasible to collect relevance labels for all documents. Consider parameterised class of scorers 8 {s(,, (e(-..;)) :;
D + R 0 E }). For any given e θ let "))....,((.. s(gn, dnk: 0)) I denote the vector of model scores on the provided documents. One may then learn a scorer 8 via minimising R(0) = = Eng(N) E(yn, Sn (0)) for loss
(0) = (s(gn, d,1; ()................. ........
e: R x K R such as the mean square error lmsc n, (0)) = ΣxETKy (Ynk (Yuk Sak (0))
2

---

Under review as a conference paper at ICLR 2022
and the softmax eross-entropy (assuming for simplicity 1 y 1): for temperature >> 0,
exp(r-1 8nk(()) (p
leol s,, - Σ kE[K]
Vnk log Prkk (0) where Pn) = Σ Eve[K] exp(r Snk (0))
(1)
To instantiate either objective, one missing ingredient is the precise parameterisation of 8. We now describe one possible parameterisation, given by transformer model.
Cross-attention and dual-encoders. Transformers ( (Vaswani et al., 2017) are sequence-to toruuenceet--eqqeeeee -equuenece models with good empirical performance on language tasks. Suppose we have a sequence (.1,,.... ,T,) of I. tokens (e.g., words in language), with X = (x1, (x,, sequence of token embeddings (e.g., word embeddings). transformer encoder maps this sequence to Z = (Z1, (Z1,22..... 22..... ZL) E R D via a composition of attention and feedforward layers. The former is a mapping attn: : DxL + R DDLL which in its simplest ("single head") form is attn =
X2.....Xi.) E DxL being corresponding
X + Wv.A, where A a X WWWWXX/VD for softmax a(-) and weights Wo. WK. Wv.
Intuitively, this embeds a token via weighted sum of all tokens' embeddings. The weighting is given by the attention matrix A E LxL,, which is a learned similarity measure between pairs of tokens. BERT (Devlin et al., 2019) is a canonical instantiation of the transform encoder, coupled with effective pre-training tasks shown to improve downstream performance in small-sample regimes.
Transformers can readily score (query, document) relevance: given a pair (g. d) E Q x 'D of tokenised queries and documents, one may pass their concatenation through a transformer encoder T. The resulting embedding may be pooled (e.g., using the [CLS] token (Devlin et al., 2019)) to yield
s(g. d) w' pool (o (q.d))
for weights E TR.. Such cross-attention (CA) models apply attention layers on the queries and documents jointly, which intuitively allow for rich interactions.
A distinct strategy is to separately embed queries and documents (e.g., via BERT encoders), and score them with their dot-product (Lee et al., 2019; Chang et al., 2020; Karpukhin et al., 2020; Zhan et al., 2020; Ma et al., 2021; Xiong et al., 2021; Luan et al., 2021; Qu et al., 2021; Zhan al., 2021a):
s(g. d) (T)) T (To))
Such dual-encoder (DE) models have their roots in Siamese networks for image processing (Bromley et al., 1993), and have received recent interest in a range of language problems (Reimers & Gurevych, 2019; Gillick et al., 2019; Karpukhin al., 2020). A salient feature of Equation 2 that one can index the query and document embeddings. This implies that DE models can have significantly lower the inference cost compared to CA models, which makes them appealing to use for both the retrieval and re-ranking stages. Indeed, DE models are widely used for retrieval, owing to their amenability to fast approximate nearest neighbour search (Guo et al., 2020; Johnson et al., 2021). Unfortunately, for re-ranking, one typically observes large gap between DE and CA performance (see $3.2).
Cross-attention to dual-encoder distillation. One strategy to bridge the performance gap between CA and DE models is distillation (Ba & Caruana, 2014; Hinton et al., 2015). Suppose have supervision {Gn, dz, yx)) where tn E R2K are the scores from a "teacher" (c.g.. CA) model. Then, we may minimise Rdist (0) = N EnE(N] f(t,,, S,, (0)). noting that use the teacher supervision t, rather than "one-hot" labels yn;
may also combine the teacher and one-hot labels. Two canonical instantiations are logit matching (Ba & Caruana, 2014), which uses the the mean square error l ms
(to, (tm, S, (0)) between the teacher and student scores, and probability matching (Hinton et al., 2015), which uses the softmax cross-entropy between the teacher and student softmax probabilities, i.e., E(t,, s, lee (ptach, sn) where Pic +
Exrejk] exp(r=-11,,') [_,,,') for temperature > 0. Empirically, distillation has been shown to at least partially bridge the chasm between CA and DE performance (Lu et al., 2020; Izacard & Grave, 2020; Yang & Seo, 2020; Hofstätter et al., 2020a; Miech et al., 2021).
exp(r 1.f44)
3 THE CAPACITY OF DUAL-ENCODER MODELS
We show that while DE models have sufficient capacity to represent complex score functions, in practice they may overfit and thus perform poorly test data.
3

---

Under review as a conference paper at ICLR 2022
3.1 How EXPRESSIVE ARE DUAL-ENCODERS THEORY?
To study the expressivity of DE models, we ask: how well can such models approximate a given score function s*: Q D IR? This concerns the theoretical capacity of representing s", and side-steps the issue of estimating it from samples; we discuss the latter subsequently. For our purposes, the DE family comprises any model of the form sq, d) z(g) w(d). where z: Q - RR and w: D + R² are (arbitrarily powerful) query- and document embedding functions. Under mild assumptions on D, we may show that DE models with sufficiently large embeddings can model a broad class of s*.
Proposition 1. Suppose D is a compact metric space, and s" =q,-): D + R is continuous for each q E Q. Then, for any e > 0, 3 vectors by; wa of at most countably infinite dimension such that
(vy E Q, d E D) is (q, 2/4 Wa s.
The proof relies on the classic Mercer's theorem for kernel methods, and is provided in the Appendix. We make a few remarks on the above. First, note that by symmetry. we could equally swap the roles of Q and 2 above. Second, the assumption of DD being a compact metric space is mild: we may do this, e.g., by identifying each document with a normalised Euclidean embedding. Third, while the above employs kernel-based argument, we do not require that s" is symmetrie; rather, s s"(q, .) is simply a function to be approximated by an element in a reproducing kernel Hilbert space.
Fourth, one can gain intuition by assuming Q and D are finite spaces. Here, we may associate with any s*: Q x ② + R a matrix S E ||] where S = s* st(g,, d). By the singular value decomposition. S ZW for matrices Z e Rl W || where r = rank(S*). Thus, we may always write s* (g, d) = Sy.d = 2(g) w (() by simply defining z(1) = Zg: and w(d) = W.a-
Finally, the above does not restrict the function class used to represent either z or w. Restricting these to transformers, we may appeal to their universal approximation power (Yun et al., 2020): sufficiently deep transformer encoder can approximate any continuous mapping T: R Dx + Px arbitrary precision. Now suppose we represent queries Q C DxL as embeddings of sequences of L tokenised elements, with a fixed [CLS] token at the first position. Then, any continuous z: Q RDD can be approximated by reading the first token embedding of a suitable transformer T.
Proposition 2. Fix some compact Q C RDL For any continuous 2: Q - RD and é' > 0, 3 transformer encoder T: RDL DXL such that [|x(Q)) T(Q)a a 11 é' for almost every Q E Q.
Remark. Luan et al. (2021) similarly analysed the expressive power of CA versus DE models, but through a slightly different lens: they formalised the expressiveness of DE score functions based on random projections, showing that the resulting embedding size may need to scale with the document length. This elegant analysis sheds light on one possible failure mode of DE models, i.e., scoring the relevance of queries and overly long documents. By contrast, we do not specifically restrict the form of the score function, which can be more reflective of transformer-basse DE models; further, we shall now identify distinct, fundamental reason for the gap between DE and CA model performance.
3.2 How EXPRESSIVE ARE DUAL-ENCODERS PRACTICE?
The previous section suggests that, at least in theory, DE models ought to be sufficiently expressive as model broad class of scores. However, this is at odds with the wealth of empirical results suggesting a non-trivial performance gap between DE and CA models (Lu et al., 2020; Izacard & Grave, 2020; Hofstääter et al., 2020a; Miech et al., 2021). What is the cause of this discrepancy? Note that the previous section considers only the capacity of DE models, i.e., the theoretical existence of a DE model capable of representing a given score function. A distinct question is whether is feasible to learn such a model from a finite number of samples, and with a capacity restriction on the function class (e.g., a fixed embedding dimension and depth of transformer encoder).
To study this, we perform a simple experiment on the benchmark MSMARCO- Passage dataset (Nguyen et al., 2016), which concerns query to passage retrieval; we defer a detailed discussion of this dataset and our training protocol to $5. We focus on the re-ranking setting, wherein both the CA and DE models operate on the outputs of a B.255 retrieval model (Robertson & Zaragoza, 2009). We train a series of BERT-based CA and DE models on the standard ("small") triplets training set, employing 6-layer BERT models from Ture et al. (2019) with varying embedding size. For each such model, we compute the mean reciprocal rank (MRR)@ @ 10 (Radev et al., 2002) the provided train and dev set.
4

---

Under review as a conference paper at ICLR 2022
CA
DIST
Densty
Demalty
2 Density
0.10
0.10
0.05
0.05
0.00
-40
0.00
-40
-20
20
40
2..2
0.2
0.4
0.6 0.8
CA score
DE score
Norma lised margin
Figure 2: Comparison of CA (left) and DE (middle) model predictions on the MSMARCO passage test set. The CA model more confidently distinguishes positives from negatives, with a pronounced peak in scores for the latter. For visual clarity, the scores are translated to have mean zero; see Figure for an uncentered plot. We further see that the margins of the DE model are distinctly smaller than CA (right); this may be mitigated with suitable distillation (DIST).
Figure compares the train and test MRR @ 10 for cross-attention (CA) and dual-encoder (DE) models. See Table 2 (Appendix) for a numeric summary. With increased embedding dimension, the DE model closely matches the performance of the CA on the training set. This is consistent with the previous section, which suggests suitably deep transformer-bass
DE models ought to model any ground truth scores; however, there is a sizable gap on the test set. This points to the DE models suffering from generalisation, rather than capacity. See Appendix C.1 for similar results with 2-layer BERT model.
3.3
WHAT CAUSES THE QUAL-ENCODER GENERALISATION GAP?
We now attempt to further understand the cause for the DE model's generalisation gap. We perform a more fine-grained inspection of the CA versus DE model predictions by studying the predicted scores for positive and negative pairs. More precisely, for each query q in the test set, consider the distribution of scores for positive and negative (query, document) pairs (q, d')) and (g, ))
Figure 2 shows that both CA and DE models possess clear modes for the positive and negative pairs. However, the CA model can be seen to not "waste" its capacity by unnecessarily modelling fine-grained distinctions amongst negative pairs; rather, it collapses most negatives to a small range of scores, and focusses intead on clearly separating the positives and negatives. By contrast, the DE model has far more diffuse negative scores. This results in a greater overlap in the positive and negative scores, implying the model has more difficulty making distinctions between these pairs. From the right plot, we also see that the DE model has smaller normalised margins (s(g, d')) s(g, d ) between the positive and negative pairs, where p is the maximal score range.
The DE score distribution for negatives is plausible: when updating DE model for a given (g, d) pair, the factorised nature of the model implies a non-trivial influence on the scores for all other pairs (q, d') and (q', d) sharing either the query or document. Consequently, it is intuitively harder for the model to arrive at solution that makes fine-grained distinctions. This can be verified qualitatively; the DE model is often seen to make errors on pairs which have only superficial relevance (Appendix C.4).
We make few remarks here. First, similar trends to the above holds on the training set; see Figure 5 (Appendix). Second, for visual clarity, in Figure 2 we apply a constant offset to each model's 's scores to ensure they are mean zero. Figure 6 (Appendix) provides uncentered plot, showing the DE model scores to have a strong baseline shift over the CA model.
3.4 How CAN MITIGATE DUCL-ENCODER GENERALISATION GAP?
Having identified the overfitting problem plaguing DE models, one may naturally ask how to mitigate this. One natural option is to apply standard regularisation strategies, such as dropout; however, our experiments reveal these to be largely ineffective in mitigating DE overfitting (see Appendix C.10). Alternately, the recent literature has identified two promising (complementary) strategies. The first is to modify the scoring function used to compute the final DE model score based on the query and document embeddings; i.e., we replace Equation 2 with s(g. d) = score score(T(q). TT))))((d))) for some scoring function score (see, e.g., MacAvaney et al., 2020; Khattab & Zaharia, 2020; Luan et al., 2021). Such strategies have proven effective; however, as these involve some overhead over dot-product scoring, it is natural to ask whether can extract more gains out of the latter. The second strategy is
5

---

Under review as a conference paper at ICLR 2022
to distill the predictions of the CA model into the DE model. By itself, this does not require changing the dot-product scoring in Equation 2. While distillation has been widely studied for classification settings, its use for re-ranking settings is less well-developed. We now study this issue more closely.
4 IMPROVING DUNL-ENSODERS VIA DISTILLATION
We now propose family of distillation schemes to improve the performance of DE models.
4.1 FROM LOGIT MARGIN MATCHING
Recall that in distillation settings, each sample is the form (7, y t). where for query and documents d E DK, we have ground-truth relevance labels y E {0, 1]. and "teacher" model scores t IR (e.g., the outputs of CA model). To fit scores s R2 for a "student" model (e.g.. a DE model), one can employ a suitable loss s). Per $2, the simplest choice is the logit matching or mean square error loss. Let us momentarily assume that for each query we have single associated positive and negative document, with respective teacher scores t = (f1, 122) and student scores s ($1. 82). Then,
the loss simply minimises msse (t, ||t $ = (41 51) )2 (12 s2)².
=1)²
While intuitive, this loss makes the implicit assumption that the range of scores for teacher and student are commensurate. When this is violated - as we empirially observe with the baseline shift between CA and DE model scores, per discussion in $3.2 and Figure (Appendix) - the loss may be ineffective. To get around this, the margin MSE loss (Hofstätter et al., 2020a) matches margins, via
fumse (t.s) ((t) - 12) - (s1 - -2))
Clearly, this loss is translation invariant, and thus is immune baseline shifts in the scores. However, two issues remain. First, the loss is tied to triplet data with single positive and negative document. Intuitively, it ought to be beneficial to leverage the teacher scores for multiple documents, with t. RK for generic K. One natural means of achieving this is E(,,s) = ExEP ExEPENEN ((t,tt))(sssss))))) where P, N denote the set of positive and negative documents, with PUN [K].
Second, the above loss demands matching all documents' margins exactly. However, this may be an ineffective use of model capacity: for re-ranking tasks, it is primarily important to ensure clear separation between the positive and negative documents, and overcome the overlapping scores observed in Figure 2. To this end, suppose then that the negative documents are sorted in descending order of the teacher scores t = (f1,.... .t....,,.. We propose to extend Equation 3 via
Emisse (t. (t, s) >((t= (tt. (g-) (5a 83.)) + > [s; 1 JEN
(4)
sEP
where j* E N denotes the index of the negative document with highest teacher score. This multi- margin MSE SE) loss only attempts to match the margins for the highest scoring negative j"; for all other negatives j }J" we simply encourage them to have lower score than j While one may design other intuitively plausible losses, an appealing property of Equation 4 is its relation to the recently proposed RankDistil framework (Reddi et al., 2021), as shall be discussed in $4.3.
4.2
FROM MARGIN TO PROBABILITY MATCHING
The above extended the logit matching distillation loss for re-ranking settings, with care account for differing score ranges for teacher and student. Per $2, the other classic distillation loss is the softmax cross-entropy (Equation 1), which has seen prior use for dual-encoder distillation (Lu et al., 2020; Izacard & Grave, 2020; Yang & Seo, 2020; Hofstätter et al., 2020a; Miech et al., 2021). In fact, we now show that the softmax cross-entropy generalises the margin MSE loss in a precise sense.
Proposition 3. Fix any teacher and student scores t, E 1².. Let ( s; S; denote the softmax cross- cross-entropy loss (Equation 1) with temperature 7 > 0, and fmmse the margin MSE loss (Equation 3). Then, for temperature scaled sigmoid function o = (1 + exp(-7-11 =))-
-1, we have
St TT) = KT KT - (2;)) || [(( (2:(111877)) 823
1 θ
lim +000
,2
loco (toott, (t)
=
l mmee for (1,2).
Os,
8
Os;

---

Under review as a conference paper at ICLR 2022
We make few remarks. First, lace swaps the square loss for binary KL divergence KL (a [6] = -a log _ + (1 a) log t in Equation 3. Unlike the square loss, the softmax cross-entropy is Lipschitz, which is typically favourable for generalisation (Luxburg & Bousquet, 2004). Second, the softmax and margin MSE loss derivatives with respect the student logits converge in the high temperature limit. This gives a precise sense in which the softmax cross-entropy is smooth approximation to the margin MSE. Intuitively, this is plausible: the former similarly invariant to constant translations to the scores, thus allowing for differing ranges in the teacher and student.
Third, our argument subtly different from Hinton et al. (2015), which establishes convergence of the softmax and logit matching derivatives: crucially, the latter only holds under the assumption that the teacher and student logits are centered. We reiterate that this assumption typically does nor hold for CA and DE models, per $3.2 and Figure 6 (Appendix). (While it is benign to translate all teacher logits by a fixed constant, this may not guarantee the assumption as the average score K Lie(k] [R] 4 is typically not constant across samples.) By contrast, Proposition 3 makes no such assumption.
Finally, Proposition assumes the supervision comprises single positive and negative document, to facilitate comparison with Equation 3. The case of K > 2 documents with teacher and student scores t, E RK poses no conceptual issue: we may seamlessly minimise the softmax cross-entropy between the two vectors. Since typically K < (D), this does not correspond to matching the full teacher distribution over all possible "classes". Despite this distinction to the standard use of distillation for classification problems, one may justify the loss as follows. For positive (negative) documents P (Q),
St, x
Σ KE(K]
log
Σ iEP
(s aNd
Σ JEN
(8 (E)=
Eiep e Lits EjEN et 1.z
This approaches max [max,epls; sk-]+ max.enls; sx+1+} a +00, where k- is the highest scoring document under the teacher. The loss thus encourages the student to preserve the teacher's ' highest scoring document, which is useful for re-ranking. Compared to Equation 4, this does not explicitly delineate the highest scoring negative from other negatives; however, empirically, we shall see that the departure from square loss endows the loss with comparable efficacy.
4.3 DISCUSSION EXTENSIONS
Connection to RankDistil. Equation 4 can be related to the recently proposed RankDistil framework by Reddi et al. (2021). Here, the goal is design distillation losses to match the teacher and student ranking over labels, particularly for the top-k elements. A canonical instantiation is the binary RankDistil formulation (RankDistil-B), (tankB(t, s) -iep(t s,)² +
-ijnls, - - Intuitively, this seeks to match the teacher and student score on positive documents, while ensuring all negative documents fall below some threshold 7n- The relation between this loss and Equation 4 is analogous that between logit matching and margin MSE: Equation matches margins rather than raw scores, which are thus suitable for settings of differing ranges of the teacher and student scores.
Distillation for retrieval phase. Our focus thus far has been on bridging the observed gap between the CA and DE models for re-ranking. However, the losses presented above are equally applicable during the retrieval phase, where they can similarly be expected improve DE model performance. These losses are complementary to proposals from several recent works improving the retrieval performance of dual-encoder models, including using suitable hard negative mining for training (Xiong et al., 2021; Zhan et al., 2021 and quantisation (Zhan et al., 202
5 EXPERIMENTAL RESULTS
We now present experiments that demonstrate the viability of the proposed distillation scheme in $4. We show that this scheme yields better performance compared to several baselines, helping reduce the gap between CA and DE models by mitigating the overfitting issue identified in $3.2.
Per $4.3, we reiterate that our primary goal is to explore the feasibility of bridging the gap between CA and DE models for re-ranking. This is distinct from the more common of DE models in the literature for retrieval (Chang et al., 2020; Xiong et al., 2021; Zhan et al., 20)la). While we do not make special effort to incorporate tricks from that literature that beneficial for retrieval
7

---

Under review as a conference paper at ICLR 2022
MSMARCO re-rank
MSMARCO retrieval
TREC DL19 re-rank
NQ re-rank
Model
MRR
nDCG
MRR
mDCG
MRR
mDCG
MRR
mDCG
Baselines: one-hot
BM25 (Robertson & Zaragoza, 2009)
0.194
0.241
0.194
0.241
0.6891
0.501
ANCE (Xiong et al., 2021)
0.330
0.677!
Cross-attention BERT (12 (12-layer)
0.370
0.430
N/A
N/A
0.829
0.749
0,746
0.673
Dual-encoder BERT (6-layer)
0.310
0,360
0.281
0.331
0.834
0.677
0.676
0.601
Distilled dual-encoder: prior work
MSE (Hofstätter et al., 2020a)
0.289
0.343
0.000
0.000
0.000
0.000
0.659
0.591
Margin MSE (Hofitätter et al,, 2020s)
0.334
0.392
0.319
0.375
0.8679
0.718
0.673
0.594
RankDistil-B -B (Reddi al,, 2021)
0.249
0,301
0.000
0.000
0.000
0.000
0.649
0,561
Distilled dual-eneoder: this work
MESE (Equation 4)
0.349
0.337
0.394
0.852
0.714
0.699
0.625
Softmax CE (Eqquation 1)
0.346
0.405
0.334
0.392
0.846
0.726
0.682
0.607
Table 1: Summary of MRR@ and nDCG@ 10 for all methods on MSMARCO Passage and Natural Questions (NQ). We compare cross-attention, dual-encoder, and distilled dual-encoder BERT models. We highlight the best performing DE based model. Distilling the dual qualdencoder with our proposed techniques significantly improves performance over one-hot training and existing distillation techniques. Results marked are quoted from the corresponding reference, "N/A" are not applicable (e.g., the cross-attention model is not feasible to apply for retrieval), and are not available from the reference. O See Appendix C.11 for analysis of potential label noise influencing the results. See text for discussion.
(e.g., using hard negatives), we do demonstrate that our proposed distillation techniques can improve performance for this stage over simply using one-hot training labels.
Datasets. We present results on MSMARCO- Passage (Nguyen et al., 2016) and Natural Questions (NQ) (Kwiatkowski et al., 2019). For MSMARCO, we report results on the standard dev set, and the TREC DI.19 test set (Craswell et al., 2020). MSMARCO comprises (query, passage) records and their relevance labels from human rater; for each query, the associated passages are retrieved from a BM25 model. The training set is canonically represented in triplet form, comprising a query, and a single positive and negative passage. NQ comprises user questions and Wikipedia passages potentially containing their answers; we use processed version from Karpukhin et al. (2020) (cf. Appendix B).
Models. We use transformer encoders initialised with the standard pre-trained BERT model checkpoints. Following Hofstätter et al. (2020a), we use BERT-Base Base for the CA model, and a 6-layer BERT model (Turc et al., 2019) with embedding size 768 for all DE models. For the DE models, tie the query and document encoder parameters. For distillation, we use the CA model as the "teacher" and the DE model the "student". On MSMARCO, for ease of comparison, we use the "TI" CA model annotations from Hofstätter et al. (2020a), which achieves similar re-re-ranking performance as our independently trained CA model. See Appendix B for further training details.
Metrics. We report the MRR @ 10 and nDCG@ 10 for all methods. Note that we are primarily interested in re-ranking tasks, wherein the test set comprises documents retrieved by BM25 baseline. and our models are used to re-rank these candidates. For MSMARCO, we additionally consider full retrieval performance obtained by scoring all (8.8M) passages, but reiterate that maximising performance on this metric requires additional negative mining that we do not explore.
Baselines. We consider methods that leverage either the observed ("one-hot") training labels on the triplet data, or the predictions from the cross-attention ("teacher") model on the triplet data. For the former, we employ softmax cross-entropy against the using a cross-attention and dual-encoder model. For the latter, we employ the logit MSE loss (Ba & Caruana, 2014), margin MSE loss (Hofstätter et al., 2020a), and the binary version of RankDistil (RankDistil-B B) (Reddi et al., 2021).
These are compared against the proposed ME (Equation 4), and the softmax cross-entropy (Equation 1). The latter methods handle an arbitrary number of documents K in the supervision (i.e., they are not restricted to triplet data). We thus aggregate the triplet data by query, and retain the top-20 passages with highest teacher model score. We study the sensitivity to K in Appendix C.7.
5.1 RESULTS AND DISCUSSION
Table 1 summarises the results of all methods. We discuss a few salient findings.
8

---

Under review as a conference paper at ICLR 2022
Baseline performance. Comparing the performance of CA and DE models in Table I, make two initial observations. First, there is a sizable performance gap between the models when using one-hot labels. Second, in line with Hofstätter et al. (2020a), naive logit matching via the MSE underperforms, due to the vastly different scales of the two models; this is however assuaged by the margin MSE loss.
Efficacy of distillation. Both the SE and the softmax cross-entropy losses significantly outperform existing methods on the MSMARCO and NQ re-reranking tasks, and shrinks the gap between the DE and CA models. On the TREC DL19 test set, our proposed methods appear fall short of the margin MSE MRR @ 10; a closer inspection of the apparent loss cases (Appendix C.11) reveals these are unanimously the results of false negatives in the provided labels.
Figure/ 1 shows that SE distillation nearly closes the gap between CA and DE models with sufficiently large embedding dimension. Intriguingly, the test gains comes with some sacrifice of training MRR for the highest embedding size. This is consistent with findings in the classical distillation literature, where distillation can harm training accuracy but improve test accuracy (Cho & Hariharan, 2019). To further study this, Figure 4 (Appendix) shows the learning curve on the MSMARCO train and test set, which demonstrates a consistent gap in test set performance as training progresses. Distillation yields a training MRR that closely matches that of the DE model for the first 100K training steps; subsequently, however, the training MRR under distillation begins to saturate, while that of the DE model with one-hot labels keeps increasing. Intuitively, this suggests that the DE model starts exploring a poorly generalising part of the parameter space beyond this point.
As final illustration, Figure 2 (right) shows that softmax cross-entropy distillation makes the DE scores better behaved: it shifts the margins of the DE model to be closer to the the CA model.
Beyond the dot-producd generic scoring functions. We have thus far focussed on results using the standard dot-product based scoring for the DE models, per Equation 2. It is of interest as to what impact more complex scoring functions, per $3.4, have final model performance. To study this, we consider the ColBERT model (Khattab & Zaharia, 2020), which computes the average of the maximum query-document token similarities. Consistent with Khattab & Zaharia (2020), using this model with one-hot labels by itself improves performance significantly over the dot-product.
When further combined with distillation, the performance exceeds that of the CA teacher, reaching an MRR @ 10 of 0.376 for the MSMARCO re-ranking task under the softmax cross-entropy loss. (Full results in Appendix, Table 7.) This further highlights that factorised representations need not imply a loss in performance, provided they are suitably trained. We defer to future work similar analysis of the conceptually similar multi-vector scoring layer of Luan et al. (2021).
Full-retrieval performance. Thus far, we have focussed on the re-ranking performance of all models, wherein the predictions from a BM25 retriever are provided as input. This allows for the comparison of DE and CA models on an equal footing. One may however naturally wonder how the proposed methods fare in the retrieval phase, wherein the models score all possible passages. Table also summarises the results in this setting MSMARCO. We again see that the proposed distillation techniques offer strong gains over standard one-hot training of the DE model, as well as existing distillation techniques. We remark also that the poor retrieval performance of the MSE loss is result of the model strongly overfitting to the teacher scores on the provided documents.
Additional results. In the Appendix, present several additional results and ablations, including the sensitivity of performance to the number of documents K in the distilled supervision (Table 6); the effectiveness of label smoothing as opposed to full distillation (Appendix C.9); and the impact of other regularisation strategies aimed at minimising overfitting (Appendix C.10).
CONCLUSION AND FUTURE WORK
Several avenues for future work remain open. For example, in the literature on distilling between cross-attention models, a common strategy is to match more than simply output logits (Sun et al., 2019; Turc et al., 2019; Sanh et al., 2019; Xu et al., 2020; Sun et al., 2020; Jiao et al., 2020; Wang et al., 2020). Are such strategies useful when distilling to a dual encoder? Further, our distillation strategy assumes access scores from a cross-attention model. Can one do away with this altogether, e.g., using deff-disiillation and related ideas (Furlanello et al., 2018)? Finally, the efficacy of the proposed losses when combined with recent advances for retrieval (Xiong al., 2021) 1) is of interest.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
This work concerns the mathematical and empirical analysis of techniques for neural re-ranking. particular problem encountered in information retrieval. We do not foresee the former having undue societal effects. The latter does not explicitly consider issues of fairness in retrieval, which is an important yet under-studied dimension. We do not foresee our distillation techniques as unduly amplifying biases in existing algorithms; nonetheless, we consider more careful study of the topic of fairness in retrieval an important avenue for future work.
REFERENCES
Jimmy Ba and Rich Caruana. Do deep nets really need be deep? In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using "Siamese" time delay neural network. In Proceedings of the 6th International Conference on Neural Information Processing Systems, NIPS 93, pp. 737-744, San Francisco, CA.
USA, 1993. Morgan Kaufmann Publishers Inc.
Wei-Cheng Chang, Felix X. Yu, Vin-Wen Chang, Yiming Yang, and Sanjiv Kumar. Pre-training tasks for embedding-base baaed large-scale retrieval. In 8th International Conference on Learning Representations, ICLR 2020, Addis Abuba, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
J. H. Cho and B. Hariharan. On the efficacy of knowledge distillation. In IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4793-4801, 2019.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. Overview of the TREC 2020 deep learning track. In Ellen M. Voorhees and Angela Ellis (eds.), Proceedings of the Twenty-Ninth Text REtrieval Conference, TREC 2020, Virtual Event [ /Gaithersburg, Maryland, USAJ. November 16-20, 2020, volume 1266 of NIST Special Publication. National Institute of Standards and Technology (NIST), 2020.
Zhuyun Dai and Jamie Callan. Deeper text understanding for IR with contextual neural language modeling. In Benjamin Piwowarski, Max Chevalier, Éric Gaussier, Yoelle Maarek, Jian- Yun Nie, and Falk Scholer (eds.), Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, pp. 985-988. ACM, 2019.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN. USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171-415 Association for Computational Linguistics, 2019.
Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born-again neural networks. In Proceedings of the 35th International Conference on Machine Learning, JCML 2018, pp. 1602-1611, 2018
Luyu Gao, Zhuyun Dai, and Jamie Callan. Understanding BERT rankers under distillation. In Proceedings of the 2020 ACM SIGIR International Conference Theory of Information Retrieval, ICTIR '20, pp. 149-152, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450380676.
Daniel Gillick. Sayali Kulkarni. Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ic, and Diego Garcia-Olano Learning dense representations for entity retrieval, 2019.
Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiy Kumar. Accelerating large-scale inference with anisotropic vector quantization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020. 13-18 July 2020, Virtual Event, volume 1 119 of Proceedings of Machine Learning Research, pp. 3887-3896. PMLR, 2020.
10

---

Under review as a conference paper at ICLR 2022
Geoffre Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015.
Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. Improving efficient neural ranking models with cross-acchitecture knowledge distillation. CoRR, abs/2010.02666, 2020a. URL https: /arxiviivvv.// wttpxivv..rg/bb//111e10 02666.
Sebastian Hofstätter, Markus Zlabinger, and Allan Hanbury. Interpretable & time-budget-constraile contextualization for re-ranking. In Giuseppe De Giacomo, Alejandro Catalá, Bistra Dilkina, Michela Milano, Senén Barro, Alberto Bugarín, and Jérôme Lang (eds.), ECAI 2020 24th European Conference on Artificial Intelligence, 29, volume 325 of Frontiers Artificial Intelligence and Applications, pp. 513-520. IOS Press, 2020b.
Sebastian Hofstätter, Bhaskar Mitra, Hamed Zamani, Nick Craswell, and Allan Hanbury. Intra- Document Cascading: Learning to Select Passages for Neural Document Ranking. Proc. of SIGIR, 2021.
Gautier Izacard and Edouard Grave. Distilling knowledge from reader to retriever for question answering, 2020.
Xiaoqi Jiao, Yichun Yin. Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguisties: EMNLP 2020, pp. 4163 4174, Online, November 2020. Association for Computational Linguistics.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535-5477 047355557,440 2021.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.
6769-6781, Online, November 2020. Association for Computational Linguistics.
Omar Khattab and Matei Zaharia. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT, pp. 39-48. Association for Computing Machinery, New York, NY, USA, 2020. ISBN 9781450380164
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Anna Korhonen, David R. Traum, and Llufs Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 6086-6096. Association for Computational Linguistics, 2019.
Tsung- Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In IEEF International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 2999-3007. IEEE Computer Society, 2017.
Wenhao Lu, Jian Jiao, and Ruofei Zhang. Twinbert: Distilling knowledge to twin-structured compressed bert models for large-scale retrieval. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 2645-2652, New York, NY, USA, 2020.
Yi Luan, Jacob Eisenstein. Kristina Toutanova, and Michael Collins. Sparse, Dense, and Attentional Representations for Text Retrieval. Transactions of the Association for Computational Linguistics, 9:329-345, 04 2021. ISSN 2307-387X.
11

---

Under review as a conference paper at ICLR 2022
Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1448-6458 PMLR, 2020.
Ulrike von Luxburg and Olivier Bousquet. Distance- based classification with lipschitz functions. J. Mach. Learn. Res., 5:669-695, December 2004. ISSN 1532-4435.
Xueguang Ma, Kai Sun, Ronak Pradeep, and Jimmy Lin. replication study of dense passage https//200.04.......
retriever. CoRR, abs/2104.05740, 2021. URL hattps://arxxt
Sean MacAvaney, Andrew Yates, Arman Cohan, and Nazli Goharian. Cedr: Contextualized embeddings for document ranking. In SIGIR, 2019.
Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, Nazli Goharian, and Ophir Frieder. Efficient Document Re-Ranking for Transformers by Precomputing Term Representations, pp. 49-58. Association for Computing Machinery, New York, NY, USA, 2020.
ISBN 145088001
Irina Matveeva, Chris Burges, Timo Burkard, Andy Laucius, and Leon Wong. High accuracy retrieval with multiple nested ranker. Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '06, pp. 437-444, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933697.
Charles A. Micchelli, Yuesheng Xu, and Haizhang Zhang. Universal kernels. Journal of Machine Learning Research, 7(95):2551-261) 2006.
Antoine Miech, Jean-Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Andrew Zisserman. Thinking fast and slow: Efficient text-to-visual retrieval with transformers. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 9826 99369983.. Computer Vision Foundation / IEEE, 2021.
Bhaskar Mitra and Nick Craswell. An introduction to neural information retrieval. Foundations and Trends@ in Information Retrieval, 13(1):1-126, 2018. ISSN 1554-0669. doi: 10.1561/1500000061
Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems 32: Anmual Conference Neural Information Processing Systems 2019, NeurlPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 4696-4705, 2019.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: human generated machine reading comprehension dataset. In Tarek Richard Besold, Antoine Bordes, Artur S. d'Avila Garcez, and Greg Wayne (eds.), Proceedings of the Workshop on Cognitive Computation: Inlegrating neurul and symbolic approaches 2016, volume 1773 of CEUR Workshop Proceedingy. CEUR-WS.org. 2016.
Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with BERT. CoRR, abts/1901.0400 2019. URL http:: /ttp:v/////i/ee//a orgg abs/1901. 04085.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao Daxiang Dong, Hua Wu, and Haifeng Wang. RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLI 2021, Online, June 6-11, 2021, pp. 5835-5847, Association for Computational Linguistics, 2021.
Dragomir R. Radev, Hong Qi, Harris Wu, and Weiguo Fan. Evaluating web-based question answering systems. In Proceedings of the Third International Conference on Language Resources and Evaluation (LREC'02), Las Palmas, Canary Islands Spain, May 2002. European Language Resources Association (ELRA).
12

---

Under review as a conference paper at ICLR 2022
Sashank Reddi, Rama Kumar Pasumarthi, Aditya Menon, Ankit Singh Rawat, Felix Yu, Seungyeon Kim, Andreas Veit, and Sanjiv Kumar. RankDistil: Knowledge distillation for ranking. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 2368 2376. PMLR, 13-15 Apr 2021.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. 11 2019. URL https: //arii///////////// org/ Abo8/ 10084.
Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr. 3(4):333-389 April 2009. ISSN 1554-0669.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, distilled version of BERT: smaller, faster, cheaper and lighter. In 5th Workshop Energy Efficient Machine Learning and Cognitive Computing NeurlPS 2019, 2019. URL
Bharath K. Sriperumbudur, Kenji Fukumizu, and Gert R.G. Lanckriet. Universality, characteristic kernels and RKHS embedding of measures. Journal of Machine Learning Research, 12(70): 2389-2410, 2011.
Ingo Steinwart and Andreas Christmann. Support Vector Machines. Information Science and Statistics. Springer New York, New York, NY, 2008. ISBN 978-0-387-77241-7.
Siqi Sun, Yu Cheng, The Gan, and Jingjing Liu. Patient knowledge distillation for BERT model compression. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 4322-4331. Association for Computational Linguistics, 2019.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. MobileBERT: compact task-agnostic BERT for resource-limited devices. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2158-2170 Online, July 2020.
Association for Computational Linguistics.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 20/6, Las Vegas, NV, USA, June 27-30, 2016, pp. 1826-2826. 2016.
Tulia Ture, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: The impact of student initialization on knowledge distillation. CoRR, abs/1900.0899000 089662 2019. URL http:// http://axxvvvv// org/ org/abs/ abs//9/s/999 1901..988000 08962.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference Neural Information Processing Systems, NIPS'17, pp. 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Wenhui Wang, Fura Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trdined transformers. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), in Neural Information Processing Systems, volume 33, pp. 5776-5788. Curran Associates, Inc., 2020.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations, 2021.
Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. BERT-of- Compressing BERT by progressive module replacing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 7859-7869, Online, November 2020. Association for Computational Linguistics.
13

---

Under review as conference paper at ICLR 2022
Sohee Yang and Minjoon Seo. Is retriever merely approximator of reader? CoRR, abs/2010.10999, 2020. URL https: / arxiv. org/ abs/2010...... 10999.
Zeynep Akkalyoncu Yilmaz, Wei Yang, Haotian Zhang, and Jimmy Lin. Cross-domain modeling of sentence-level evidence for document retrieval. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3490-3496, Hong Kong, China, November 2019. Association for Computational Linguistics.
Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Are transformers universal approximators sequence-to-sequence functions? International Conference Learning Representations, 2020.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. RepBERT: Contextualized text embeddings for first-stage retrieval. CoRR, 154998 2020. URL https///://axxii.orrr abs/2006. 15498.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. Optimizing dense retrieval model training with hard negatives. CoRR, abs/2104.08051, 2021a. URL https: //arxiv.org org. abs abw/aa/s...0.///...//... 08051.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. Jointly optimizing query encoder and product quantization to improve retrieval performance. In CIKM, 2021b.
14

---

Under review as a conference paper at ICLR 2022
A
PROoFS
Proof of Proposition 1. For any q E 9, let us write 8q = s -): D R. Pick reproducing kernel Hilbert space H over DD with associated unee-pproduct -.. Suppose further H has associated kernel k: D x D - IR that is continuous, and universal (Micchelli et al., 2006). Then, H is dense the space of continuous functions (Sriperumbudur al., 2011). Recall that by assumption sy is continuous. Consequently, each 8q can be approximated to arbitrary precision (with respect to the foc. metric) by some 9 E H. Since 4 E Jt, by the reproducing property an RKHS,
(vd D) s' (q (sg. ka)sc,
where in an abuse of notation hi :), E H. Observe that we may interpret oy as an "embedding" for the query q, and ki a "weight vector" for the document d.
may appear this stage that we are done; however, each of sy and ki may uncountably infinite dimensional objects, and the inner product in H( may not be the standard dot-product. Nonetheless, under the stated assumptions (in particular, compactness of D), we may appeal to Mercer representation of the kernel, following Steinwart & Christmann (2008, Theorem 4.51). In particular, we may construct Mercer feature map c: D - &g for H, such that
20
(4h E 30) (3(m E (()
Σ an & (d).
Further, the inner product between any h1, h2 E H with "Mercer coefficients" (an (an))=11 (bx) is expressible as
(h1 h2)9c
Dn.
7=1
iee. familiar dot-product. our context, write the Mercer coefficients for sy as (zgn) and for ka as (w Thus, we may write
(s kaya = Zg Wa
where the vectors Zn, wa have at most countably infinite dimension.
Proof of Proposition 2. Any continuous z: R PxL. R can be trivially associated with
2: RDxL RR L where each column ž,І Such mapping preserves continuity. By Yun et al. (2020, Theorem 2), we may thus find transformer encoder T: DxL - RDxL such that fo [(Q))= I S é' Since the integrad is nonnegative, we must also have [īz(Q) T(Q) 2 dQ < for any given Q. Consequently, we must also have (z()). T(Q)a <
almost surely, which by definition implies [[z(Q)) T(Q). l12 <
Proof of Proposition 3. For temperature T > 0, simple algebra reveals Equation I to be
exp(r- ty)
exp(r-1 .8y)
lsce(t. (t s)==)= -
Σ
exp(T t1) + exp( (xp(r-1.15 1.) .z) llog
exp(r-1 1 s1) + exp(T- 11)))
vE(1.2)
Σ 1 11(b--t1) log(1 + a ( (x2-
=E(=1)
Σ alz (t)-))): log(o(2 (81 s2):T)).
=
#E|=11
This is equivalent to KLpin (a(t1 t2: [I ( S2; T)), i.e., swapping out the square loss for the KL divergence in Equation 3. In fact, one can make stronger connection: observe that
2 Os
Cmmse (t, s) 2 ((s1 82) (t1 tz))
15

---

Under review as a conference paper at ICLR 2022
① Os
laco (t,)=
Σ =E(#1)
olz (41 ((1-)));))
log(o(z (51 82);7)
8s1
.1
+ e 81
1
1 11 + e
1
1
1
(82 (82)
1 12
4
as T - +00, where the last line follows from the Taylor series approximation
2 + 0(2))
+ exp(-2)
28 - O.
B EXPERIMENT HYPERPARAMETENS
For all models, at the output layer apply dropout rate 0.1 and layer normalisation. We use sequence length of 30 for queries, and 200 for passages. We optimise all methods for maximum of 3 x 10y steps using Adam with weight decay, with batch size of 128 and a learning rate of 2.8 x 10-5 (i.e., a 4x scaling of the choices in Hofstätter et al. (2020a)). Following Hofstätter et al. (2020a), we perform early stopping based the nDCG@ 10 metric.
For the Natural Questions dataset, we use the processed version used in Karpukhin et al. (2020), that contains questions, positive passages with a correct answer to each question, and corpus of all Wikipedia passages. To train DE, CA, and multi-negative DIST models, we use 19 BM-25 hard-negative passages for each question along with a positive passage. For single-negative DIST models we use a single BM-25 negative from the same collection. To calculate MRR @ 10 and nDCG@ 10 metries, we use the queries in the dev set with 200 passages containing positives, 100 BM-25 hard-negatives and up to 100 random negatives.
C ADDITIONAL EXPERIMENTS
C.I CA VERSUS DE MODELS: EFFECT ARCHITECTURE
Figure 3 presents an analogue of Figure 1, but using instead 2-layer BERT model. As before, we initialise using the pre-trained models developed in Ture et (2019). The same general trends hold: for sufficiently large embedding dimension, the DE model closely matches the performance of the CA on the training set; however, there a sizable gap in the test set performance. Table presents numeric summary of the train and test performance for CA and DE models, under different choices of base BERT-model architecture (in terms of number of layers, as well as the final embedding dimension).
C.2 CA VERSUS DE MODELS: EVOLUTION OF TRAIN AND TEST PERFORMANCE
The above results summarise the results at the completion of training. For finer- understanding of how performance evolves during training, Figure presents the learning curves for CA, DE, and distilled DE (DIST) models the train and test set. Here, we use the small-bert-6-766 architecture for all models. We observe that the CA and DE models initially have a non-trivial gap in performance, but this shrinks over time; this further points to the initialisation of the CA models being more favourable for generalisation. Note also that beyond certain point, both models continually improve their training performance, but are completely neutral in terms of test performance. Interestingly, distillation largely tracks the DE model performance, but then worsens on the training set while improving the test set. This suggests that distillation helps explore better part of the search space.
16

---

Under review as a conference paper at ICLR 2022
MRR@ 10 train
MRR@ 10 test
Model
CA
DE
CA
DE
small-hert-2-128
0.356
0.292
0.309
0.221
smaal-bbrt-2-256
0.358
0.302
0.322
0.253
122
0.398
0.374
0.335
0.268
small-hert-2-768
0.429
0.406
0.334
0.281
1mall-bert---1222
0,385
0.317
0.334
0.249
small-hert-6-2556
0.427
0.407
0,350
0.282
small-bert-e bert-6-512
0.493
0.491
0.360
0.293
small small-bert-6-775 bert-6-768
0.522
0,518
0.364
0.310
bert-base
0.658
0.635
0.368
0.309
Table 2: Comparison of cross-attention (CA) and dot-product based dual-encoder (DE) models on the MSMARCO-Passar
re-ranking task. Notably, for sufficiently large embedding dimension, the DE model closely matches the performance of the CA on training set; however, there is sizable gap on the test set. This points to the poorer DE model performance being largely an issue of generalisation, rather than capacity.
0.425
modee
model
GA
0.400
WBR train
0.315
MRR810 test.
0.350
0.28
0.525
0.26
0.300
0.24
0.275
0.22
128
256
512
768
128
256
512
768
Embeddling siz
Embedding size
Figure 3: Comparison of BERT- based cross coos-attentiinn (CA) and dual-encoder (DE) models on the MSMARCO- Passage re-ranking task. For varying embedding dimension of an underlying 2-layer BERT model, we report the train and dev set MRR @ 10. We the same trend in Figure 1: for sufficiently large embedding dimension, the DE model closely matches the performance of the CA on the training set; however, there a sizable gap in test set performance.
C.3
CA VERSUS DE MODELS: COMPARISON OF SCORE DISTRIBUTIONS
Figure compares the score distributions for the CA and DE models on the training set. We observe the same general trends as the test set (Figure 2). Figure 6 further presents the test score distributions without zero-centering. We see that the DE model exhibits strong baseline shift, wherein the scores are of the order of ~ 700.
Figure studies the score distributions for the CA and DE models after varying numbers of training steps. At initialisation, as expected, both models have essentially overlapping distributions for the positives and negatives. However, here too, we observe that the DE model scores operate on a much wider range than the DE model. The middle row continues this study by looking at behaviour after 10, 000 steps of training, which a small fraction of the 300, 000 steps used for training of all models. Here, we start to observe a clearer separation between the positive and negative scores for both models. Interestingly, the CA model already sees a sharp peak on the negatives, unlike the DE model. The bottom shows the test set score distributions of the final trained models, to complement Figure from the body. We see the same general trend as the training set: the CA model is seen to more confidently distinguish positives from negatives, and operate narrower range of scores.
17

---

Under review as conference paper at ICLR 2022
0.5
0.3
method
0.2
DIST
CRV
0.0
train
0.0
0.2
0.4
0.6
0.8
1.0
Training step
Figure 4: Learning curve for CA, DE, and DIST models on the MSMARCO- Passage train and test set. Here, we use the small-bert-6-766 architecture. Distillation is seen to saturate training performance beyond certain point, while still resulting in solution with better generalisation the test set.
0.175
0.175
0.125
0.125
0.015
0.015
0.025
0.025
20
Figure 5: Comparison of CA and DE model predictions on MSMARCO- Passage train set. observe same general trends as the test set (Figure 2). For visual clarity, the scores are translated to have mean zero.
0,200
0.200
0.150
0.150
0.100
0.100
0.500
0.050
8.025
3.000
0.000
CA
Figure 6: Comparison of CA and DE model predictions the WAMAMMMM-MSSssse test Here, do center scores to have mean zero. We see that DE model scores possess a strong baseline shift over the model scores.
C.4 CA VERSUS DE MODELS: QUALITATIVE ANALYSIS
Table 3 provides a sample of (query, passage) pairs from MSMARCO-Pasaag dev set, where there is large discrepancy between the CA and DE model scores. Specifically, consider pairs that the CA model scores low, but the DE model scores high. Interestingly, these pairs typically involve strong token overlap between the query and passage indicating certain degree of topicality - but are fact genuine negative pairs. This reflects that the DE model may be unable to capture certain fine-grained distinctions.
18

---

Under review as a conference paper at ICLR 2022
0.003
50330
Fodele
Naguive
10
1.005
3,025
3.8
3,0020
-400
-230
250
DE
DIST
0.35
0.35
0.95
Peativ
Megative
0.25
0.25
6:25
Denale
2.35
2.35
049
5.00
0:75
CA srere
DC soore
DIST scone
0.5
0.23
Megatiee
0.20
5.23
0.20
3.15
3.15
Derell
8.05
0.30
0.33
6:00
3.55
1.05
6:35
0.00
CA score
DC acore
31ST scom
Figure 7: Comparison of CA, DE and distilled DE (DIST) model scores on the MSMARCO-Passage dev set at initialisation (top row), after 10, 000 steps of training (middle row), and at the completion of 300, 000 steps of training (bottom row). At initialisation, all models have overlapping score distributions, but the CA model operates at much tighter range. After a steps of training, sees a peaky score distribution on the negative. while the DE model has a diffuse distribution. The distilled model however manages to overcome this, and produce a sharper distribution than even CA.
CA model already
Query
Passage
effects of yeast on body
The Side Effects Chemotherapy on the Body. Chemotherapy drugs are powerful cnough to kill rapidly growing cancer cells, but they also can harm perfectly healthy cells, causing side effects throughout the body.
age not to take off shoes airline security
Airline Identification Requirements. Airlines do not typically require identification from passengers under the age of 18, but there are exceptions. Children under the age of 2 may ride on parent's lap without purchasing a ticket, but the airline will require identification, such as a birth certificate, to prove the child's date of birth
actress who plays alice on the magicians
Then portrayed as the animated Alicc's real life counterpart actress Mia Wasikowska more mature, grown Alice in Disney's 's 2010 semi-sequel, live action/CGI film Alice and Wonderland Directed by Tim Burton. In the Broadway musical version, she will be played by Taylor Louderman...
can you absorb metals from plants
Answer 1: Photosynthesis the ability of plants to absorb energy of light, and convert into energy for the plant. To do this, plants have pigment molecules which absorb the energy of light very well. The pigment responsible for most light-harvestin hy plants is chlorophyll, a green pigment. The green color indicates that it is absorbing all the non-green light- the blues ( (425-450 nm), the reds and yellows (600-700 nm).he pigment responsible most light-harvesting by plants is chlor
Table 3: Sample of (query, passage) pairs from SAMMARCO-Passaeee dev set with largest discrepancy between the CA and DE model scores. In most of these cases, the passage is not relevant to the query; however, there is high degree of token overlap between the two, indicating superficial similarity.
19

---

Under review as conference paper at ICLR 2022
CA
6
DE
DIST
Density
4
3
2
-0.2 0.0
0.8 1.0 Normalised margin
0.2
Figure 8: SE distillation model margins the MSMARCO Pasco-e test Compared to the DE model trained with one-hot labels, the distilled model seen to more confidently distinguish positives from negatives, evidenced by the margin distribution having more mass on larger values.
Model
Supervision
Loss
MRR @ 10
nDCG@ 10
Cross-attention
One-hot
Softmax CE
0.370
0.430
Dual-encoder
One-hot
Softmax CE
0.249
0.299
Dual-encoder
Teacher
MSE
0.180
0,232
Teacher
Margin MSE
0.251
0.299
Teacher
KL
0,273
0.321
Teacher
M
0.261
0.308
Table Re-ranking results on MSMARCO-Passee dev set, using a small- sellll-bert---111 128 model.
C.5 IMPACT OF DISTILLATION: EFFECT ON SCORE DISTRIBUTIONS
We have previously seen in Figure 7 that our proposed M distillation strategy can improve the distribution of scores for positives and negatives. Figure shows the impact that such distillation has on the margins between positive and negatives, compared to standard training on one-hot labels. The distilled model seen to more confidently distinguish positives from negatives, evidenced by the margin distribution having more mass larger values.
C.6 IMPACT OF DISTILLATION: EFFECT OF STUDENT MODEL ARCHITECTRRE
The preceding distillation results have all employed a amall-bert-6-i 6-8 model. It is of interest how sensitive the results are to this choice. To study this, we report in Table 4 the re-ranking results on model. We observe largely consistent trends as before: distillation using our proposed losses can close the gap between CA and DE model performance. Table 5 repeats this for small-bert-2-77 model.
MSMARCO-Pasaan when using small-bert-6- 128
C.7 IMPACT OF DISTILLATION: EFFECT OP NUMBER OF DOCUMENTS IN SUPERVISION
The results reported in the body employed total of 20 documents per query for the softmax CE and MESE methods. As this involves significantly more information than the triplet data used, e.g., in training the margin MSE loss, it is of interest tease apart how much of the gains from the method come from this increased supervision, versus the loss itself. Table studies the effect of varying the number of documents. We see that there diminishing returns beyond certain point: using 100 documents yields qualitatively similar performance to using 20 documents. At the same time, there marked difference in performance when using documents versus 10.
20

---

Under review as conference paper at ICLR 2022
Model
Supervision
Loss
MRR@10
nDCG@ 10
Cross-attention
One-hot
Softmax CE
0.370
0.430
Dual-encoder
One-hot
Softmax CE
0.281
0.331
Dual-encoder
Teacher
MSE
0.259
0.313
Teacher
Margin MSE
0.313
0.368
Teacher
KL
0.311
0.370
Teacher
M
0.286
0.337
Table 5: Re-ranking results on MSMARCO-Pasage dev using small-bert-2-768 model.
# of documents
Loss
MRR@ 10
nDCG @ 10
2
Softmax CE
0.327
0.386
M
0,323
0.379
10
Softmax CE
0.333
0.391
M
0.326
0.382
20
Softmax CE
0.338
0.396
M
0.349
0.406
100
Softmax CE
0.334
0.393
M
0.325
0.381
Table 6: Sensitivity analysis of distillation losses to number of documents per sample.
C.8 RESULTS WITH COIBERT SCORER
Table 7 presents results using the ColBERT scorer. We find that with distillation using the softmax CE loss (Equation 1), exceed the performance of the cross-attention teacher model for the re-ranking task. This further indicates the viability of dual-encoder models for neural ranking.
MSMARCO rerank
TREC DL19
MRR@ 10
nDCG@10
MRR@10 nDCG@10 10
Baselines: one-hot
Cross-attention BERT
0.370
0.430
0.829
0.749
Dual-encoder ColBERT
0.356
0.416
0.839
0.703
Distilled ColBERT: prior work
MSE (Hofstätter al., 2020a)
0.365t
0.428t
Margin-MSE (Hofstätter et al., 2020a)
0.370
0.431
0.862t
0.738t
Distilled ColBERT: this work
M
0.371
0.430
0.875
0.726
Softmax CE
0.376
0.437
0.835
0.737
Table 7: Results on MSMARCO Passage dev with ColBERT model.
C.9
BEYOND DISTILLATION? EFFECT OF LABEL SMOOTHING
special case of distillation is label smoothing (Szegedy et al., 2016), which can be understood using teacher that predicts the uniform distribution over all labels, and combining the result
21

---

Under review as conference paper at ICLR 2022
Smoothing
MRR @ 10 train
MRR @ 10 test
0.0
0.394
0.308
0.1
0.460
0.297
0.2
0,455
0,296
0.3
0.453
0,292
0.9
0.382
0.292
0.99
0.305
0.278
Table Effect label smoothing on DE models, on the re-ranking task MSMARCO-Pasaage dataset. For all models, we use the softmax CE loss on triplet data.
with the one-hot label. Specifically, this involves mixing the observed one-hot labels with uniform distribution over all labels, with the mixing controlled by weight E [0, 1); 0 corresponds to standard one-hot training. Label smoothing has also proven effective as a means of preventing harmful overfitting, and thus improving generalisation (Müller al., 2019; Lukasik et al., 2020) in classification settings.
For the MSMARCO PasAac dataset, studied the effect of varying degrees of label smoothing for DE models, using the softmax CE loss on triplet data. We use a small-bert-6-766 model as before; for computational ease, we report results after 300, 000 steps of training, which point all models stable test error. Table 8 reveals that, surprisingly, smoothing has a minimal effect on test per performance, even high smoothing levels. This despite smoothing reducing the training performance higher values, which is expected given its equivalence to injecting symmetric label noise to each sample. Further study of the viability of smoothing for retrieval problems would be of interest.
C.10 BEYOND DISTILLATION? EFFECT OF ALTERNATE REGULARISATION STRATEGIES
Distillation provides one means of preventing the DE model from overfitting to the training set. One may of course conceive of other plausible strategies, such as:
increasing the dropout rate in the final (embedding) layer of the transformer. The experiments in the body employ 10% dropout (following prior work, e.g., Hofstääter al. (2020a)), which improves performance considerably over having no dropout; it is thus of interest whether there are benefits to further increasing this.
performing token dropout at the input layer. Intuitively, such corruption prevents the model from overly relying on individual tokens, thus preventing spurious token matching (as present in some cases in Table 3).
adding masked language model (Devlin et al., 2019) loss for the student. This loss is core component of BERT, and has thus proven be effective in learning generically useful sequence embeddings. Intuitively, adding such self-supervised objective to training can plausibly improve the model's 's robustness.
modifying the base loss from square log-loss to the focal loss (Lin et al., 2017). This loss has proven effective in vision tasks involving high class- inballnnaaene... In our setting, as there typically only handful of relevant documents, but several millions of irrelevant documents, is interest whether such a loss can positively impact results.
Table 9 summarises results for each of these strategies on the MSMARCO Passage re-ranking task. Here, we focus on their effect on the standard (one-hot) training a dot-produet based DE model on triplet data. Unfortunately, performance of the baseline model. Thus, mitigating the overfitting observed in Figure requires more effort than appealing standard strategies effective in classification tasks.
find that none of these strategies significantly improve the generalization
C.11 ANALYSIS OF TREC DL.9 PREDICTIONS
Table suggests a notable gap in performance for our proposed RankDistil variant and the margin MSE loss (Hofstätter et al., 2020a). Here, we take closer look at the loss cases for our method. The
22

---

Under review as conference paper at ICLR 2022
Strategy
Train MRR @ 10
Test MRR @ 10
Baseline DE
0.619
0.310
Increased embedding dropout
0.588
0.299
Token dropout
0.572
0.291
Masked language loss
0.548
0.299
Focal loss
0.546
0.307
Table 9: Results of various regularisation strategies on MSMARCO Passage dev set. For all rows, we use a DE model trained on the triplet data.
TREC DL.19 data comprises total of 43 queries, each with between 100 to 500 rated passages. Of these queries, the MRR @ 10 of the DE models trained with margin MSE and softmax CE agree on 10. On the queries with disagreement, 6 cases favour margin MSE, and 4 cases favour RankDistil-S
For the 6 queries where RankDistil-SM ostensibly underperforms, we inspect the top-5 scoring passages for both margin MSE and sofumax CE in Table and 11. The cells shaded blue correspond to passages rated positive. Several other passages however to be equally valid answers the source query. Indeed, we submit that in all cases, the predictions from RankDistil are at least the same quality as the margin MSE
C.12 IMPACT NEGATIVE MINING ON RE-RANKING PERFORMANCE
We consider the value of adding additional negatives during DE model training. Following Karpukhin al. (2020); Qu et al. (2021), we consider the use within-batch (also referred to as in-batch) and uniform negatives. For the latter, in each training minibatch, we draw sample of Buni documents drawn uniformly at random from the entire set of documents; each such document treated as negative document in the loss. For the former, from each training minibatch {(g, d; ) compute the set of all observed documents Dobs U _ {dix }. For each sample (4t di ys), we then use each element in Dobs - _ [d } as a negative document in the loss.
Table 12 summarises the results for the MSMARCO re-ranking task. We find that adding these negatives has a small gain for the one-hot model performance. However, for the distilled objectives, there limited gain (and sometimes even a degradation) performance. More study of this issue warranted, but one hypothesis is that the re-ranking task is inherently concerned with ranking the outputs BM25 model. These outputs are precisely used to construct the training data used for distillation. It is thus possible that adding additional negatives which are unlikely to appear as candidates for re-ranking - do not bring significantly useful information.
We emphasise here also that the results reported are for the re-ranking, as opposed to retrieval task. For the latter, adding wuthin-batch and uniform negatives is intuitively and empirically valuable, as noted in Karpukhin et al. (2020); Qu et al. (2021). For example, in training the one-hot DE model on the triplet data, the retrieval MRR @ 10 is 0.281. When adding within-bated and uniform negatives, this increases 0.314. Given that the re-ranking MRR @ 10 remains relatively unchanged (0.312 versus 0.310), this is further indication of the re-ranking and retrieval objectives not being perfectly aligned.
23

---

Under review as a conference paper at ICLR 2022
Query
Top scoring passages
Margin MSE
RankDistil-SM
who robert gray
Who is Henry Gray? Henry Gray an African-Americnn blues piano player and singer. He has been play...
Robert Grey (born 21 April 1951 Marefield, Leicestershire) is an En- glish musician best known as L...
Robert Gray was the Democratic can- didate for governor of Mississippi in the 2015 elections. Gray won...
Who is Henry Gray? Henry Gray is an African-America blues piano player and singer. He has been play...
Kenneth Gray Kenneth Gray is an actor, known Love, Lies and Murder (1991), Retribution (1987...
Robert Gray was Democratic can- didate for governor of Mississippi the 2015 elections. Gray won...
Kenneth Gray (I) Actor Kenneth Gray is an actor, known for Love, Lies and Murder (1991), Retributio...
Robert Gray. A surprise came on the Democratic side in the race for Mississippi Governor, Robert Gra.
Robert Grey (born 21 April 1951 in Marefield, Leicestershire) glish musician best known
William Thomas Gray. Billy Gray was born January 13, 1938 in Los Angeles, California, USA as Will...
define visceral?
Definition of visceral. 1 felt as if in the internal organs of the body: deep a visceral co...
Definition of Visceral. Visceral: Re- ferring to the viscera, the internal organs of the body, specifi...
Definition of Visceral. Visceral: Re- ferring to the viscera, the internal organs of the body, specifi.
Definition of Visceral. Visceral: Re- ferring to the viscera, the internal organs of the body, specifi...
felt in in the internal organs of the hody: deep visceral...
Definition of visceral.
Medical Definition of Visceral. Vis- ceral: Referring the viscera. the internal organs of the body.... ..
Definition of Visceral. Visceral: Re- ferring to the viscera, the internal organs the body, specifi.
Definition of visceral. 1: felt in or as if the internal organs the body: deep visceral co...
Detinition of visceral. 1 felt in in the internal organs of the body: deep visceral...
Define visceral: felt in if the internal organs of the body deep; not intellectual inst.
what is the daily life of thai people
T he following concepts are part of Thai everyday life: or JAI YEN is more way
The Daily Life of Thai Monk The Sangha World in Thailand consists of about 200,000 monks and 85,00...
The following concepts are part of Thai everyday life: or JAI YEN is more way o...
An important thing in everyday life is SANUK. Thai people love to have fun together. SANUK can repre .. T he following concepts are of Thai everyday life: or JAI YEN is more a way
The population of Thailand is approx- imately 67.5 million people, with an annual growth rate of about...
For the rapcore band, see Every Day Life. Everyday life or Daily life or Routine life phrase us...
The following concepts are part Thai everyday life: or YEN more a way o...
Everyday life. Everyday life, daily life or routine life comprises the ways in which people typicall...
The population of Thailand is approx- imately 67.5 million people, with annual growth rate of about...
Table 10: Comparison of top-5 scoring passages for margin MSE and softmax CE on TREC DL.19 test set. The cells shaded blue correspond to passages rated positive. Several other passages however seen to be equally valid answers to the source query.
24

---

Under review as a conference paper at ICLR 2022
Query
Top scoring passages
Margin MSE
RankDistil-SM
what are the three percenters?
The Three Percenters, formed in late 2008, are a lousely organized move- ment centered around an obscu...
Try understand that being Three Percenter (Threeper. 3%, 3 Percenter, etc) is more of an idea I...
111% Club Merchandise & II1% Logo Gear. The Three Percenters Club official merchandise is designed a... Rhodes has written supportively of the Three Percenters, while at least two participants carried the...
But this still doesnt answer the ques- tion. So how do you know you are Three Percenter? Try t...
So let me see if can help. In fact, let me provide you with well over 50 ways to tell whether or n...
Try to understand that being a Three Percenter (Threeper, 3%, Percenter, etc) is more of an idea t...
The Three Percenters, formed in late 2008, are loosely organized move- ment centered around an obscu...
But this still doesnt answer the ques- tion. So how do you know if you Three Percenter? Try L...
111% Club Merchandise & I11% Logo Gear. The Three Percenters Club official merchandise is designed a...
how are
some warm
Most sharks are cold-blooded. Some, like the Mako and the Great white shark, are partially warmblood...
Most sharks are cold-blooded. Some, like the Mako and the Great white shark, are partially warmblood...
sharks blooded
Most sharks cold-blooded. Some, like the Mako and the Great white shark, are partially warmblood...
Most sharks cold-hlooded. Some, like the Mako and the Great white shark, partially warmblood...
Are White Sharks warm- boon--dddddd or cold-blooded? White sharks are part of the fish family, so they mus...
Are White Sharks warm- moro-bloded or cold-blooded? White sharks are part of the fish family, so they mus...
Great white sharks are some of the only warm blooded sharks. This al- lows them to swim in colder wate... These sharks can raise their temper- ature ubout the temperature of the water; they need to have oc...
Great white sharks are some the only warm blooded sharks. This al- lows them swim colder wate...
The Salmon Shark is one of the warmest of the warm-bodied sharks. Measurements its epax- ial (upper...
what are the so- cial determinants health
The social determinants health are the circumstances in which people are born, grow up, live, wor...
Social determinants of health reflect the social factors and physical condi- tions of environment
Social determinants of health reflect the social factors and physical condi- tions of the environment..
Social determinants of health nomic and social conditions that in- fluence the health of people...
The social determinants of health are linked the economic and social conditions and their distrib...
Social determinants of health are con- ditions in the environments which people are born, live, lea...
Social determinants of hcalth are con- ditions in the environments in which people are born, live, lea...
Back to Top. Social determinants of health are conditions in the environ- ments in which people are bo...
Determinants health are factors that contribute to person's 's current state health. These fact....
The social determinants of health are the circumstances in which people are born, grow up, live, wor...
Table 11: Comparison of top-5 scoring passages for margin MSE and softmax CF on TREC DL.19 test set. The cells shaded blue correspond to passages rated positive. Several other passages are however seen to be equally valid answers to the souree query.
25

---

Under review as conference paper at ICLR 2022
Supervision
Loss
MRR @ 10
One-hot
Softmax CE
0.310
negative mining
0.312
Teacher
Margin MSE
0.334
+ negative mining
0.335
Teacher
SE
0.349
+ negative mining
0.324
Teacher
KL
0.346
+ negative mining
0.342
Table 12: Results of negative mining for DE models, on MSMARCO passage re-ranking task.
26


Reviews:
[{"review_id": "klPNCxlXBlo", "reviewer": "Reviewer_FgFj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Closing the gap between cross-attention models and dual-encoder models is an essential research problem for industrial applications. The main contribution of this work lies in revealing the underlying reasons of gap between CA and DE models, and proposing a new KD method for re-ranking tasks. The whole paper is clear and well-written, \n\nBelow are some questions or confusions I had: \n1)\tI am a bit confused by the relationships between capacity, overfitting and generalization ability (or better OOD performance) as discussed in this work. It is obvious that a model with relatively large capacity tends to easily overfit the training data, while to get better OOD performances we need to establish new learning methods or design better model architectures. The BERT model has a large number of parameters, so it is not surprising that a dual-encoder model, twice size of a CA model, can overfit the training dataset, which is widely accepted in practice. To narrow the gap, the proposed is a modified KD loss, and it would be more interesting to give some suggestions on the network architectures, as the well-known ColBERT.   \n\n2)    The main contribution of this work is the Dual-encoders(DE) model based distillation, and this topic has been studied a lot. The proposed M3SE loss substantially outperforms previous distillation techniques on both re-ranking and full retrieval settings. However, M3SE seems to be a natural extension of previous work MarginMSE by adapting to multiple positive and negative documents. Considering that there is almost one positive doc for each query in MS MARCO and NQ, Equation 4 actually reduces to $(s_{+}-s_{j}^{*})^2+\\sum_{j \\in N} \\[s_j-s_{j}^{\\*} \\]^2_{+}$  in the experiment. This means that in M3SE, the teacher provides an benchmark for student model that what should the score of positive document be close to and what should its negative ones be lower to. It makes sense but seems to be a natural extension of marginal MSE.\n\n3)\tIn this work, 6-layer BERTs are used as student model. The authors are suggested to further study the question of how to find the optimal student network architectures, e.g. a relatively deep and thin network, under the constraints of inference time.\n", "summary_of_the_paper": "This paper aims to narrow the performance gap between cross-attention BERT and dual-encoder BERT for re-ranking task. The authors empirically and theoretically analyze the underlying reasons of the performance gap. The gap could be mitigated by the proposed knowledge distillation method, where the original cross-attention BERT model acts as the “teacher” and the more efficient dual-encoder BERT model is used as student. Comprehensive experiments confirmed the effectiveness of the proposed KD method for various re-ranking tasks.   ", "summary_of_the_review": "This paper tries to reveal the underlying reasons of performance gap between CA and DE models, and a KD method is further proposed to improve the performance of dual-encoder model, while the analysis on the gap cannot guide the design of new student matching architecture or new distillation method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iQ9ZNQF0zGa", "reviewer": "Reviewer_yXc2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n\nPros:\n\n1. Good theoretical analysis on the power of dual encoders. I like this analysis as the starting point of the paper. \n2. The generalization gap of dual encoders makes sense to me. The empirical results in Figure 2 are interesting. However, I would like to see more deep insight. For example, analysis on special query-document cases might be useful.\n3. The two modification methods on knowledge distillation are interesting. The empirical results show the benefits of the proposed methods. However, I have some questions about this part (refer to the Cons part).\n.\n\n\nCons:\n\nMy main concern of the proposed approach is the experimental design parts. For information retrieval models, only utilizing the human-labeled dataset for knowledge distillation seems to be a weak baseline.  There has been some work on constructing datasets to improve the dual encoders (also called data distillation). For example, we can easily construct hard negative samples (cited in the paper) to improve the dual encoders. My feeling is that adding these results might make the paper even stronger.\n", "summary_of_the_paper": "In information retrieval problems, there are two kinds of models: dual-encoder models and cross-attention models. Cross-attention models generally outperform dual encoder models by a large margin. Due to the latency requirement of real search systems, cross-attention models might not meet the requirements. So one ideal solution is to transfer the knowledge from cross-attention models to dual encoder models. This paper follows this line of research ideas by reducing the gap between the two models. \n\nFirstly, the paper theoretically analyzes the effects with sufficiently large embedding dimensions. Secondly, the paper empirically verifies the theoretical results. Thirdly, the paper proposes a new method that further bridges the gap between the two methods. The writing is clear and well-structured. I like the idea and logic of the paper. I have some questions about the experiment part.\n", "summary_of_the_review": "Overall, I like this paper. I would like more empirical results on data distillations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "B-9JtLUxSqr", "reviewer": "Reviewer_oYj4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* They did a theoretical analysis which shows that DE models are sufficiently expressive to model a broad class of scores and there should not be a gap between CA and DE models.\n* They identified that the gap is mainly caused by the generalization ability of the DE models. They have similar performance as CA models on the training data while performing much worse on the testing data.\n* They proposed distillation algorithms that aim to match the margins between the teacher (CA) and student (DE) models. The experimental results verify that the proposed approach can further reduce the gap.\n\nWeaknesses:\n* It remains unclear to me why the DE models are less generalizable. Do the authors have some any explanations about this?\n* The theoretical analysis can not guide the algorithm design to reduce the gap.\n* They did not compare with other distillation algorithms.", "summary_of_the_paper": "This paper investigate the gap problem between cross-attention (CA) and dual-encoder (DE) models for document reranking. In general, cross-attention models perform much better than dual-encoder models while consuming more computational cost. The authors first leverage Mercer's theorem to prove that dual-encoders models are sufficiently expressive to model a broad class of scores using countably infinite dimension, which mainly the gap between CA and DE models should not exist from theoretical analysis. They further identified the gap is caused by the generalization ability of the DE models. To reduce the overfitting on the training data, they proposed to use distillation algorithms. Specially, they design the loss function which matches the margins between the teacher (CA) and student (DE) models. They also extend the margin to probability matching by using softmax cross-entropy loss. Through the experiments, they demonstrated that their proposed distillation approaches can further reduce the gap between DE and CA models.\n\nThe main contributions of this paper is that they show there should not be a gap between DE and CA models from theoretical analysis. They found that the gap is caused by the generalization ability of CA model and proposed distillation approach to reduce the gap.", "summary_of_the_review": "They have theoretically analyzed that the DE models are expressive enough and the gap should not exist between DE and CA models. They found the gap is because DE models are less generalizable and proposed an effective distillations algorithm to further reduce the gap. Taking these advantages into consideration, I think it is good paper to be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7t2Hu5RM8Ep", "reviewer": "Reviewer_4c6M", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe paper is well written and easy to follow.  Good analysis throughout, comparing the score distributions of the two architectures.  The experiments are sound and proofs are correct.\n\nWeak points:\nThe theory presented is correct, but not really informative in practice.  Prop. 1 has no bounds on the dimension of the embeddings.  Without such bound, it's easy to see how bi-encoders might represent any function, however this is not really useful.  I don't think this theory deserves the central treatment it currently has in the presentation, I would remove or relegate to the appendix.  On the other hand, it's interesting to see empirically that bi-encoders have similar training accuracy with cross-encoders.\n\nNovelty - there are multiple recent works which have demonstrated the usefulness of distilling a cross-encoder into a bi-encoder for retrieval.  Authors have cited some of them, but here are two more:\nhttps://arxiv.org/pdf/2010.10999.pdf\nhttps://arxiv.org/pdf/2010.08191.pdf\n\nAll of these works use the standard cross-entropy / KL-divergence loss for distillation, which the authors have shown to work as well or better than their proposed M3SE loss.  From a practical standpoint, the novel contribution of this work is pretty weak.  \n\nAs mentioned before, some of the empirical analysis is certainly interesting, but I don't think it is surprising enough to warrant publication by itself.", "summary_of_the_paper": "The paper contrasts cross-encoder vs. bi-encoder architectures for re-ranking in information retrieval.  The authors argue that bi-encoders under-perform cross-encoders, not for lack of capacity, but due to poor generalization, and back this claim with some theory and empirical results.  The authors then propose distillation of the cross-encoder model into a bi-encoder, which is shown to improve bi-encoder results.  While this has been proposed before, the distillation loss function the authors propose is slightly different than the standard cross-entropy (which the authors show is closely related).", "summary_of_the_review": "This is a well written paper, however the contribution is not significant, due to multiple recent works already having covered this idea pretty well.  The theory presented is also neither informative nor surprising.  Some of the empirical analysis is certainly interesting, but I don't think it is surprising enough to warrant publication by itself.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Strengths:
* Well-written paper
*Theoretical analysis demonstrates that dual encoder models have similar capacity as CA models
*New distillation algorithm for learning DE students from CA teachers

Weaknesses:
* No reviewer seems particularly excited about this work 
* Theoretical analysis doesn’t provide actionable insight -- it does not directly motivate the suggested distillation methods
* Empirical results are lacking -- reviewers asked for qualitative examples of improvements from their distillation method
Decision: Reject