EmbedDistill: A geometric knowledge distillation for information retrieval
Authors: Seungyeon Kim
Ankit Singh Rawat
Manzil Zaheer
Sadeep Jayasumana
Veeranjaneyulu Sadhanala
Wittawat Jitkrittum
Aditya Krishna Menon
Rob Fergus
Sanjiv Kumar
Venue: Submitted to ICLR 2023 (2023)
Keywords: Knowledge distillation
dual encoder
cross encoder
information retrieval
query generation
embedding matching
retrieval
re-ranking

Abstract: Large neural models (such as Transformers) achieve state-of-the-art performance for information retrieval. In this paper, we aim to improve distillation methods that pave the way for the deployment of such models in practice. The proposed distillation approach supports both retrieval and re-ranking stages and crucially leverages the relative geometry among queries and documents learned by the large teacher model. It goes beyond existing distillation methods in the information retrieval literature, which simply rely on the teacher's scalar scores over the training data, on two fronts: providing stronger signals about local geometry via embedding matching and attaining better coverage of data manifold globally via query generation. Embedding matching provides a stronger signal to align the representations of the teacher and student models. At the same time, query generation explores the data manifold to reduce the discrepancies between the student and teacher where the training data is sparse. Our distillation approach is theoretically justified and applies to both dual encoder (DE) and cross-encoder (CE) models. Furthermore, for distilling a CE model to a DE model via embedding matching, we propose a novel dual pooling-based scorer for the CE model that facilitates a more distillation-friendly embedding geometry, especially for DE student models.

Under review as a conference paper at ICLR 2023
EMBEDDISTILL: A GEOMETRIC KNOWLEDGE DISTILLATION FOR INFORMATION RETRIEVAL
Anonymous authors Paper under double-blind review
ABSTRACT
Large neural models (such as Transformers) achieve state-of-the-arr performance for information retrieval. In this paper, we aim to improve distillation methods that pave the way for the deployment of such models in practice. The proposed distillation approach supports both retrieval and re-ranking stages and crucially leverages the relative geometry among queries and documents learned by the large teacher model. It goes beyond existing distillation methods in the information re- trieval literature, which simply rely the teacher's scalar scores the training data, on two fronts: providing stronger signals about local geometry via embed- ding matching and attaining better coverage of data manifold globally via query generation. Embedding matching provides a stronger signal to align the repre- sentations of the teacher and student models. At the same time, query generation explores the data manifold to reduce the discrepancies between the student and teacher where the training data is sparse. Our distillation approach is theoretically (CH) models. Furthermore, for distilling a CE model to a DE model via embedding matching, we propose novel dual pooling-based scorer for the CE model that facilitates a more distillation-frienndly embedding geometry, especially for DE student models.
justified and applies to both dual encoder (DE) and cross-encoder
INTRODUCTION
Neural models for information retrieval (IR) are increasingly used to capture the true ranking in var- ious applications, including web search (Mitra & Craswell, 2018), recommendation (Zhang et al., 2019), and question-answentin
(QA; Chen et 2017). Notably, the recent success of Transform- ers (Vaswani et al., 201 based pre-trained language models (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020) on a wide range of natural language understanding tasks has prompted their utilization in IR to capture query-document relevance (see, e.g., Dai & Callan, 2019b; MacAvaney et al., 2019a; 9a; Nogueira & Cho, 2019; Lee et al., 2019; Karpukhin et al., 2020, and references therein).
A typical IR system comprises two stages: (1) A retriever first selects small subset of potentially relevant candidate documents (out of a large collection) for a given query; and (2) A re-ranker then identifies a precise ranking among the candidates provided by the retriever. Dual-encoder (DE) models are the de-facto architecture for retrievers (Lee et al., 2019; Karpukhin et al., 2020) Such models independently embed queries and documents into a common space, and capture their relevance by simple operations on these embeddings such as the inner product. This enables offline creation of document index and supports fast retrieval during inference via efficient maximum inner product search (MIPS) implementations (Johnson et al., 2021; Guo et al., 2020), with query embedding generation primarily dictating the inference latency. Cross-encoded (CE) models, on the other hand, are preferred re-rankers, owing to their excellent performance (Nogueira & Cho, 2019; Dai & Callan, 2019a; Yilmaz al., 2019). CE model jointly encodes query-document pair while enabling early interaction among query and document text. Employing CE model for retrieval is often infeasible, as it would require processing a given query with every document the collection at inference time. In fact, even in the re-ranking stage, the inference cost of CE models high enough (Khattab & Zaharia, 2020) to warrant exploration of efficient alternatives (Hofstätter et al., 2020; Khattab & Zaharia, 2020; Menon et al., 2022).
Knowledge distillation (Bucilà et al., 2006; Hinton et al., 2015) provides a general strategy to address the prohibitive inference cost associated with high-quality large neural models. In the IR literature,

---

Under review as a conference paper at ICLR 2023
most existing distillation methods only rely on the teacher's query-document relevance scores (see, e.g., Chen et al., 2021; Lu et al., 2020; Hofstätter et al., 2020; Ren et al., 2021; Santhanam et al,, 2021) or their proxies (Izacard & Grave, 2021). However, given that neural IR models are inherently emhedding- based, it is natural to ask: is it useful to go beyond matching of the teacher and student models' scores, and directly aim align their embedding spaces?
With this in mind, we propose a novel distillation method for IR models that utilizes an embedding matching task train the student. The proposed method supports cross-architecture distillation and improves upon existing distillation methods for both retriever and re-ranker models. When distilling large DE model into a smaller DE model, for given query (document), one can minimize the dis- tance between the query (document) embeddings of the teacher and student after compatible projec- tion layers to account for any dimensionality mismatch. In contrast, defining an embedding matching task for distilling CE model into a DE model is not as straightforward. For Transformers-based CE models,
common to use the final embedding of special token, e.g., [CLS] in BERT (De- relevance (Nogueira & Cho, 2019). However,
vlin et al., 2019), to compute query-documen note in Sec. 4.2, this token embedding does not capture semantic similarity between the query and document. To make CE models more amenable to distillation via embedding matching, propose a modified CE scoring approach by utilizing a novel dual- dual-pooling strategy: this separately pools the final query and document token embeddings, and then computes the inner product between the pooled embeddings as the relevance score.
Our key contributions toward improving IR models via distillation are:
We propose a novel distillation approach for neural models, namely EmbedDistill, that goes beyond score matching and aligns the embedding spaces of the teacher and student models.
We consider novel DE to DE distillation setup showcase the effectiveness of our embedding matching approach (Sec. 4. 1). Specifically, we consider a student DE model with an asymmet- ric configuration, consisting of a small query encoder and frozen encoder inherited from the teacher. Such a configuration significantly reduces query embedding generation latency during inference, while leveraging the teachers' high-quality document index.
We show that our proposed distillation approach can leverage synthetic data improve student by further aligning the embedding spaces of the teacher and student (Sec. 4.3).
We theoretically justify both embedding matching and query generation components of our pro- posed method (Sec. 5). Further, we provide a comprehensive empirical evaluation of the method (Sec. 6) on two standard IR benchmarks Natural Questions (Kwiatkowski et al., 2019a) and MSMARCO (Nguyen et al., 2016).
Finally. we demonstrate the utility of embedding matching for CE to DE distillation on MS- MARCO by utilizing novel pooling strategy for CE models, namely dual pooling (Sec. 4.2), which might be of independent interest.
RELATED WORK
Here, we review some existing Transformers-based IR models, and discuss prior work distillation and data augmentation for such models. We also cover prior efforts on aligning representations dur- ing distillation for non-IR settings. Unlike our problem setting where the DE student is factorized, these works mainly consider distilling a single large Transformer into smaller one.
Transformers-basee based architectures for IR. Besides DE and CE models described in Section 1, intermediate configurations (MacAvaney et al., 2020; Khattab & Zaharia, 2020; Nie et al., 2020; Luan et al., 2021) have been proposed. Such models first independently encode the query and doc- ument, and then apply more complex late interaction between the two. Interestingly, Nogueira et al. (2020) explore generative encoder-decoder style model for re-ranking, where a T5 (Raffel et al., 2020) model takes query-ddcumentt pair as input and its score for certain target tokens (c.g.. True/False) defines the relevance score for the pair. In this paper, we focus on basic DE and CE models to showcase the benefits of our proposed geometric distillation approach. Exploring embed- ding matching for the aforementioned architectures is an interesting avenue for future exploration.
Distillation for IR. Traditional distillation techniques have been widely applied in the IR literature, often to distill a teacher CE model to a student DE model (Chen et al., 2021; Li et al., 2020). Recently, distillation from DE model (with complex late interaction) another DE model (with
2

---

Under review as a conference paper at ICLR 2023
Score beaed (ld Audonn
Soore
Fookin Student Cuuney Froodde
Posingg Student foc Frccder
Paoling Studket Query Ercoder
Poolin Encodar
Teacher
Excecer
Erecier
Qurey inkene
Doo Vilns
Quary lakara
Quary tooees
Dei sione
Query
Dax tikens
(b) Score-basee DE to DE distillation
(a) Score-hased CE to DE distillation
Figure 1: Illustration of score-based distillation for IR (cf. Section 3.2). Fig. la describes distil- lation from a teacher [CLS]-pooled CE model to a student DE model. Fig. 1b depicts distillation from teacher DE model to a student DE model. Both distillation setup employ symmetric DE configurations where query and document encoders of the student model are of the same size.
inner-product scoring) has also been considered (Lin et al., 2021; Hofstätter et al., 2021). As for distilling across different model architectures, Lu et al. (2020); Izacard & Grave (2021) consider distillation from teacher CE model to student DE model. Hofstätter et al. (2020) conduct an extensive study of knowledge distillation across wide-range of model architectures. Most existing distillation schemes for rely on only teacher scores; by contrast, we propose a geometric approach that also utilizes the teacher embeddingg. Many recent efforts (Qu et al., 2021; Ren et al., 2021; Santhanam et al., 2021) show that iterative multi-stage
improves upon single-stage distillation (Qu et al., 2021; : Ren et al., 2021; Santhanam et al., 2021). These approaches use model from the previous stage to obtain labels (Santhanam et al., 2021) as well as mine harder- negatives (Xiong al., 2021). We only focus the single-stage stage training in this paper. Multi-stage procedures are complementary to our work, as one can employ our proposed ambedding-matehle matching approach in various stages of such a procedure.
Distillation with representation alignments. Outside of the IR context, a few prior works proposed utilize alignment between hidden layers during distillation (Romero et al., 2014; Sanh al., 2019; Jiao et al., 2020; Aguilar et al., 2020; Zhang & Ma, 2020). Chen et al. (2022) utilize the represen- tation alignment to re-use teacher's classification layer. Our work differs from these it needs to address multiple challenges presented by IR setting: 1) cross-architecture distillation such as CE to DE distillation; 2) partial representation alignment of query or document representations as opposed to aligning representations for the entire input, i.e., a query-documents pair; 3) catering representation alignment approach to novel IR setups such as asymmetric DE configuration; and 4) dealing with negative sampling due to large number of classes (documents). To the best of our knowledge, our work is the first the IR literature that goes beyond simply matching scores (or its proxies).
Semi-supervised learning for IR. Data augmentation or semi-supervised learning has been previ- ously used ensure data efficiency in IR (see, e.g., Zhao al., 2021; MacAvaney al., 2019b). More interestingly, data augmentation via large pre-trained models have enabled performance im- provements well. Doc2query (Nogueira al., 2019b;a) performs document expansion by gener- ating queries that are relevant to the document and appending those queries to the document. Query expansion has also been considered, c.g., for document re-ranking (Zheng et al., 2020). Notably, generating synthetic (query, passage, answer) triples from a text corpus to augment existing training data for QA systems also leads to significant gains (Alberti et al., 2019; Oguz al., 2021). Fur- thermore, even zero-shot approaches, where no labeled query-document pairs are used, can also perform competitively to supervised methods. Such methods train DE model by relying on inverse cloze task (Lee et al., 2019; Izacard et al., 2021), synthetic query-document pairs given a target text corpus (Ma et al., 2021), or relevance scores from large pretrained models (Sachan et al., 2022). Unlike these works, we utilize querr-ggen capability ensure tighter alignment between the embedding spaces of the teacher and student.
BACKGROUND
Let Q and D denote the query and document spaces, respectively. An IR model is equivalent to scorer s: x 'D > R, i.e., assigns a (relevance) score s(q, d) for a query-document pair (g, d) e Q D. Ideally, we want to learn an IR model or scorer that is faithful to the true query-document relevance, i.c., s(q, d) > s(g. d') iff the document d is more relevant to the query than document d' We assume access n labeled training examples of the form S,, {(g, di, ys) kejo)- Here,
3

---

Under review as a conference paper at ICLR 2023
d (dd......,d....) Dí, E [nd], denotes a list of L documents and y; (Va,1 (V6.1y.....ii.
{0, 1)2h denotes the corresponding labels such that Vig = 1 iff the document d,, is relevant to the query g. Given Sn, we learn an IR model by minimizing
R(s; 8,)
n E iigjn) (i).
(1)
where ( (s = (s(ge: d1.t).... ..... s(q, d1, denotes the loss incurs (%, d, y.). Due to space constraint, we present a few common choices for the loss function & in Appendix A.
While this learning framework is general enough to work with any IR models as the scorers, next, we formally describe two families IR models that are prevalent in the recent literature.
3.1 TRANSFORMER-BASHI IR MODELS: CROSS-ENCODERS AND DUAL-ENCODERS
Let query q= (g".............. .q'..... . and document d = (d' ..... d= consist m, and m2 tokens, respec- tively. We now discuss how Transformers-bart -assd CE and DE models process a given (q, d) pair.
Cross-encoder model. Let [g: denote the mi --2 length sequence obtained by concatenating and d. Furthermore, let ō be the sequence obtained by adding special tokens such [CLS] and [SEP] o at appropriate locations. Now, given an encoder-only Transformer model Enc, the relevance score for the query- docymmee pair (q. d) is defined as
s(q, d) = (w, pool [Enc()))) = (w, embg.a).
where w is a d-dimensional classification vector, and pool(-) denotes pooling operation that trans- form Enc(o) contextualized token embeddings produced by Enc to a joint embedding vector embt q,d [CLS] |pooling a common strategy that simply outputs the embedding of the [CLS] token.
Dual-encoder model. Let ÿ and d be the sequences obtained by adding appropriate special tokens to and d, respectively. A DE model comprises two (encoder-only) Transformers Encq and Encp, which we call query and document encoders, respectively.' Let emb, = pool (ooc(((((()) and emb, pool (d) denote the query and document embeddings, respectively. Now, one can define s(q. d) (pool emby, emb,) to be the relevance score assigned to (g, d) by the DE model.
3.2 SCORE-BASED DISTILLATION FOR IR MODELS
Most distillation schemes for IR (e.g., Chen et al., 2021; Lu et al., 2020; Hofstääter et al., 2020; Ren al., 2021; Santhanam al., 2021) rely on teacher relevance scores (ef. Fig. 1). particular, given a training set 8, and teacher with scorer s", one learns a student with scorer s by minimizing
R(s S':) 1 Enetal La (a sy,a),;
where la captures the discrepancy between 8 and s². See Appendix A for common choices for la-
EMBEDDING-MATCHING BASED DISTILLATION
Since modern neural IR models are inherently embedding-based we propose to explicitly align the embedding spaces the teacher and student via novel distillation method, namely EmbedDistill. Note that our proposal goes beyond existing distillation methods in the IR literature that only utilize the teacher scores. Next, we describe EmbedDistill for two prevalent settings: (1) distilling large DE model to smaller DE model;² and (2) distilling CE model to DE model.
4.1 DE TO DE DISTILLATION
Given (q. d) pair, let emb; and emb;, be the query and document embeddings produced by the query encoder Ency and document encoder Eact, of the teacher DE model, respectively. Similarly, let
is common to employ dual-encnder models where query and document encoders are shared. We focus DE to DE distillation setup the CE to CE distillation is special case of the former with the classification vector (cf. Eq. 2) being the trivial second encoder.
4

---

Under review as a conference paper at ICLR 2023
emb; and emb; denote the query and document embeddings produced by student DE model with (Enci, Enc")) as its query and document encoders. Now, EmbedDistill optimizes the following embedding alignment loss in addition to the score-matching loss from Sec. 3.2:
Remb (t, S;,) - E (q.))E.. (Iembl, -rrop (emb;)]
+ |embb, proj (emb:))))
(4)
where proj is an optional trainable layer that is required if the teacher and student produce different sized embeddings. Alternatively, one can employ other variants of EmbedDistill, e.g., focusing on only aligning the query embeddings takes the following form (cf. Fig. 2).
REmb, (t, S.) = - ges..
yes, |/emb; - proj (ero))) ||-
(5)
Asymmetric DE. We also propose novel student DE configuration where the student employs the teacher's s document encoder (i.e., E = Ench)) ) and only train its query encoder, which is much smaller compared the teacher's query encoder. For such setting, it is natural only employ the embedding matching loss in Eq. 5 as the document embeddings are aligned by design (cf. Fig. 2).
Note that this asymmetric student DE does not incur an increase in latency despite use of large teacher document encoder. This is because the large document encoder only needed to create good quality document in- dex offline, and only the query en- coder is evaluated at inference time. Thus, for DE to DE distillation, we prescribe the asymmetric DE con- figuration universally. Our theoreti- cal analysis and experimental results suggest that the ability to inherit the document tower from the teacher DE model can drastically improve the final performance, especially when combined with EmbedDistill.
dec
Soo based dieillationn
Score
matsiing
Pooling
Ing
PPoliing
Pooking Student Query Enoodar
Teacher Query Enooder
Teacher Doc Encoder
Teacher Encoder
Quenyy
Quary bolions
Figure 2: Proposed DE to DE distillation with query embed- ding matching. The figure describes setting where student employs an asymmetric DE configuration with a small query encoder and large (non-trainable) document encoder in- herited from the teacher. The smaller query encoder ensures small latency for encoding query during inference, and large document encoder leads to good quality document index.
4.2 CE TO DE DISTILLATION
Let Enc' denote the single teacher CE model encoder, and (Ency, Ency)) denote the student DE model's query and document encoders. When distilling from a CE to DE model, defining an effective embedding matching task is not as straightforward as in Sec. 4.1: since CE models jointly encode query-document pairs, it is not obvious how to extract individual query and document embeddings.
As a na've solution, for a given (q, pair, one can simply match a joint transformation of the student's query embedding emb; and document embedding embद, to the teacher's joint embeddings emb q.d- However, we observed that including such an embedding matching task often leads to severe over-fitting, and results in a poorly generalizable student. Since s' d) = (w, emb during CE model training, the joint embeddings emb', g.d for relevant and irrelevant (4, d) pairs are encourage to be aligned with w and ww, respectively. This produces degenerate embeddings that do not capture semantic query-to-qurry or secumenn-to--ocumeent relationships. We notice that even the final query and document token embeddings lose such semantic structure. Thus, a teacher CE model with (g. d) = (we, emb', a does not add value for distillation beyond score-matching: in fact, it hurts to include naïve embedding matching. Next, we propose a modified CE model training strategy that facilitates EmbedDistill.
CE models with dual pooling, We propose dual pooling to produce two embeddings emb' 7 (q.d)
and emb))((q.d)) from a CE model that serve as the proxy query and document embeddings, respec- tively. Accordingly, we define the relevance score as s'. d) (emb) (q. emb' (g.))) We explore two variants of dual pooling: (1) special token-based pooling that pools from [CLS] and [SEP]; and (2) segment-ber based weighted mean pooling that separately performs weighted averaging on the query and document segments the final token embeddings. See Appendix B for details.
5

---

Under review as a conference paper at ICLR 2023
In addition employing the dual pooling, we also utilize a reconstruction loss dur- ing the CE training, which measures the likelihood of predicting each token of the original input from the final token embed- dings. This loss encourages reconstruc- tion of query and document tokens based on the final token embeddings and pre- vents the degeneration of the token embed- dings during training the IR task. Given proxy embeddings from the teacher CE we can perform EmbedDistill with the loss defined in Eq. 4 or Eq. 5 (cf. Fig. 3).
Queyy Maa
Scone-tereed did lation
Embedding maithing
Pooling Quarry
Pooling)
Teacher
Cross Encoder
Encoder
Figure 3: Illustration of CE to DE distillation using EmbedDistill, with CE model employing dual pooling.
4.3 SPEKIIPECIFIC ONLINE DATA GENERATION
Data augmentation as general technique has been previously considered in the IR literature (see, e.g., Nogueira al., 2019b; Oguz et al., 2021; Izacard al., 2021; Ma et al., 2021), especially in data-limited, out-of-domain, or zero-shot settings. Since EmbedDistill aims to align the embed- dings spaces the teacher and student, the ability to generate similar queries or documents can naturally help enforce such aligmment globally on the task-specific manifold. Given set of unla- beled task specific query and document pairs Um, we can further add the embedding embedding-alignmeen loss Remb (t, 8; Um) to training objective (ef. Eq.4). Interestingly, for DE to DE distillation setting, our approach can even benefit from a large collection of task-specific queries O' or documents D' Here, as we can independently add the additional embedding matching losses REmb, (t, (t,ss s; Q') and REmb,D (t,, 8; D') that focus on queries and documents, respectively.
IMPROVEMENTS IN THE GENERALIZATION OF STUDENT
Note that we motivate EmbedDistill well as asymmetric DE configuration where the student DE model inherits the teacher's document encoder from their potential ability to ensure a better align- ment between the teacher and student embedding spaces. In this section, we provide a theoretical justification for both of these proposals by showing that they indeed result in better generalization performance for the student and reduce the gap between the teacher and the student. We break our analysis into two steps: I) First, we show that, using EmbedDistil and inheriting the document en- coder from the teacher, the student's empirical risk (as measured by the distillation objective) gets closer the teacher's population risk; and 2) Second, argue that both of these techniques have favorable implications on the distillation loss of the student via uniform deviation bounds.
The following result studies the gap between student's expected empirical (distillation) risk and teacher's ' population risk (see Appendix C.I for a formal statement and proof). For simplicity, we focus on (cf. Sec. 3). The result can be extended L > 1 with more complex notation.
Proposition (Expected risk bound (informal)). Let label y E (0, 1} indicate the relevance of query-document pair (q, d). Suppose that the score-based distillation loss la in Eq. 3 is based on binary cross entropy loss (Eq. 11 in Appendix A). Let one-hot (abee-dependent) loss in Eq. be the binary cross entropy loss (Eq. 9 in Appendix A). Assume that all encoders have the same output dimension. Under regularity conditions, we have
E [R(s s²: S..) EE(ss v)] 0 Eull [emllemmb; [mb] F.y [llem;; - emb; [2]
(qg.)))) |iigmoid ((mbb) embū)) vl)
Proposition can be viewed as the bias of the student wrt. the teacher, as realized by the distillation. It shows that the bias can be upper bounded by three terms: 1) the expected difference between the doc embeddings of the student and the teacher, 2) the expected difference between the query embeddings, and 3) the teacher's error in modeling the true label probability. Observe that the student's ' bias is smaller when the embeddings of the student match those of the teacher. In particular, when the student inherits the document encoder from the teacher (as in Fig. 2), the error in the first

---

Under review as a conference paper at ICLR 2023
Recall
Recall @
Recall & 100
Method
82M
16M
82M
16M
82M
16M
Train student directly
39.5
64.5
59.9
82.0
76.3
Distill from teacher
48.3
44.9
67.2
61.1
80.9
74.8
Inherit document embeddings
56.9
47.2
74.3
64.0
85.4
77.0
Query embedding matching
61.8
56.7
78.7
74.6
89.0
85.9
Query generation
61.7
57.1
79.4
75.2
89.6
86.7
Train student only using embedding
matching and inherit doe embeddings
63.7
57.9
80.3
74.6
90.3
85.7
Query generation
80.5
76.0
90.4
86.6
Table I: Full recall performance of various student DE models on NQ dev set, including symmet- ric DE student model (82M 16M transformer for both encoders), and asymmetric DE student model (82M or 16M transformer as query encoder and document embeddings inherited from the teacher). All distilled students used the same teacher (110M parameter RoBERTa-base models as both encoders), with the full Recall = 64.6, Recall @ 20 = 81.7, and Recall @ 100 91.5.
term vanishes. These observations also justify EmbedDistill which either employ Eq. 4 or Eq. 5 (when student inherits teacher's document encoder).
Now, we analyze the deviation of a student model from its training empirical risk at the test time, which typically captured by the uniform deviation bounds based on quantities like Rademacher complexity. Again, restrict ourselves to binary cross-entropy loss with L 1 for simplicity. Duc to space constraints, we present an informal statement of the result (see Appendix C.2 for a more precise statement and proof).
Let la be a distillation loss which is Lea Lipschitz in its first argument. Let IJ and 9 denote the function classes for the query and document encoders, respectively. Further assume that, for each query and document encoder in our function class, the query and document embeddings have their 22-norm bounded by K. Then, we have the following uniform deviation bounds
Proposition 2.
sup E 71 sE(m]
1 Σ
Ca(sa de.de) - Ela(s s4,0) S Es.
48K Vn
VVog [ (NNu,)) du; (6)
Σ Ca (S di
i s.y,a)) - Blaass sq S Es..
48K
Vlog N (u,J) du. (7)
sup "EF nt
iE(m)
Here, 9" is a fixed document encoder and (u,,)) denotes the u-covering number of a function class.
Note that Eq. 6 and Eq. 7 corresponds to uniform deviation when train without and with frozen document encoder, respectively. is clear that the bound in Eq. 7 is lower than Eq. 6 (due to absence of Nuu, 9) term which is always larger than 1), which alludes to desirable impact of employing frozen document encoder (in terms of deviation in train and test performance). When we further employ EmbedDistill (e.g., with the loss in Eq. 5), it regularizes the function class of query encoders; effectively, reducing it to F" with | uniform deviation bounds as N (u, F') (u,
< |3). This has further desirable implication for reducing the
6 EXPERIMENTS
We now conduct comprehensive evaluation the proposed distillation approach. Specifically, we highlight the utility of the proposed approach for both DE to DE and CE DE distillation. We also showcase the benefits of combining our distillation approach with query generation methods.
6.1 EXPERIMENTAL SETUP
Benchmarks and evaluation metrics. We focus two popular IR benchmarks - Natural Ques- tions (NQ; Kwiatkowski et al. 2019b) and MSMARCO (Nguyen et al., 2016), which focus on finding the relevant passage/document given question and search query, respectively. NQ provides both standard test and dev sets, whereas MSMARCO has only standard dev set publicly available. In what follows, we use the terms query (document) and question (passages) interchangeably. For NQ, we use the standard full recall (strict) as well as the relaxed recall metric (Karpukhin et al., 2020) to evaluate the retrieval performance. For MSMARCO, we focus on the standard metrics Mean Reciprocal Rank (MRR)@ @ 10, and normalized Discounted Cumulative Gain (nDCG) 10. See Appendix for a detailed discussion on these evaluation metries.

---

Under review as a conference paper at ICLR 2023
Model architectures. We follow the standard Transformers-base IR model architectures similar
@RR@10
nDCG @ 10
Method
82M
16M
82M
16M
Train student directly
29.7
26.3
35.2
31.4
to Karpukhin et al. (2020); Qu et al.
Distill from teacher
31.6
28.4
37.2
33.5
(2021); Oguz al. (2021). Our CE model is based on a RoBERTa-base model (Liu et al. (2019); 12-layer, 768 dim, 124M parameters). We utilized various sizes of DE models Dis-
Inherit doc embeddings
32.4
30.2
38.0
35.8
Query embedding matching
32.8
31.9
38.6
37.6
Query generation
33.0
32.0
38.8
37.7
Train student only using embedding
matching and inherit doe embeddings
32.7
31.8
38.
37.5
Query generation
33.0
31.8
38.9
37.5
based on RoBERTa-base, tilRoBERTa (Sanh et al. (2019);
Table Performance of various DE models on MSMARCO dev set. A teacher model (1 (11MM parameter RoBERTa- base models as both encoders) achieving MRR @ 10 of 33.1 and nDCG@ @ 10 of 38.8 is used. The table shows performance of the symmetric DE student model (82M or 16M transformer as both encoders), and asymmetric DE student model (82M or 16M transformer as query encoder and document embed- dings inherited from the teacher).
6-layer, 768 dim, 82M parameters 2/3 of base), or RoBERTa-mini (Ture et al. (2019); 4-layer, 256 dim, 16M parameters - 1/8 of base). For query generation (cf. Sec. 4.3), we employ BART-base (Lewis et al., 2020), an encoder-decoder model, to generate similar questions from each training example's s input question (query). We randomly mask 10% of tokens and inject zero mean Gaussian noise with {0.1, 0.2] between the encoder and decoder. See Appendix E for details.
6.2 DE TO DE DISTILLATION
For both NQ and MSMARCO, teacher DE models are based on RoBERTa-base model (see Ap- pendix F for the training details). For DE to DE distillation, we consider two kinds of configurations for the student DE model: (1) Symmetric. We use identical question and document encoders. We evaluate DistilRoBERTa and RoBERTa-miní for both datasets. (2) Asymmetric. The student DE model inherits its document embeddings from the teacher DE model, which are not trained during the distillation. For query encoder, we use BERtil or RoBERTa-mini which are smaller than the teacher document encoder.
Student DE model training. We train student DE models using combination of (i) one-hot loss (ef. Eq. 8 in Appendix A) on training data; (ii) distillation loss in (cf. Eq. 10 in Appendix A); and (ii) embedding matching loss in Eq. 5. We used [CLS]-pooling for all student encoders. Unlike DPR (Karpukhin et al., 2020), we do not use hard negatives from BM25 or other models, which greatly simplifies our distillation procedure.
Results and discussion. To understand the impact of various proposed configurations and losses, we train models by sequentially adding components and evaluate on NQ and MSMARCO dev set as shown in Table and 2, respectively. (See Table 5 in Appendix G.1 for performance on NQ in terms of the relaxed recall.) We begin by training a symmetric DE without distillation. As expected, moving to distillation brings in considerable gains.
Next, we swap the student document encoder with document embeddings from the teacher (non- trainable), which leads good jump in the performance. Now can introduce EmbedDistill with Eq. 5 for aligning query representations between student and teacher. The two losses are combined with weight of 1.0 and 5.0 for NQ and MSMARCO, respectively. This improves performance significantly, e.g., it provides ~ 5 and ~ points increase in recall @ on NQ with students based on BERTa and RoBERTa-mini, respectively. We further explore the utility of EmbedDistill in aligning the teacher and student embedding spaces in Appendix H.I.
On top of the two losses (standard distillation and embedding matching), we also employ REmb. s; 2') from Sec. 4.3 on additional questions (per input question) generated from BART for further gain. We also try variant where we eliminate the standard distillation loss and only employ the embedding matching loss in Eq. 5 along with inheriting document embedding from the teacher. This configuration without the standard distillation loss leads to excellent performance (with query generation again providing additional gains).
worth highlighting that DE models trained with the proposed methods (e.g. asymmetric DE with embedding matching and generation) achieve 99% of the performance in both NO/MSMARCO tasks with a query encoder that is half the size of that of the teacher. Furthermore, even with 1/8th
8

---

Under review as a conference paper at ICLR 2023
size of the query encoder, our proposal can achieve 91-97% of the performance. This is particularly useful for latency critical applications with minimal im- pact on the final performance.
Method
Flayyers
R@200
R@ 100
DPR (Karpukhin et 2020)
12
78.4
85.4
DPR PAQ (Oguz et al., 2021)
12
84.0
89.2
DPR PAQ (Oguz et al,, 2021)
24
84.7
89.2
ACNE (Xiong et al., 2021)
12
81.9
87.5
RocketQA (Qu et al., 202
12
82.7
88.5
Finally, we take our best student mod- els for NQ based on the dev set, i.e., one trained using only embedding match- ing loss with inherited document embed- ding and using data augmentation from query generation, and evaluate on the NQ test set (cf. Table 3). We compare with various prior work and note they worked with considerably bigger models in terms of depth (12 or 24 layers) and width (upto 1024 dims), while our ap- proach obtains competitive performance with fewer than 50% of the parameters. Note that, even with 6 layers, our student model is very close (99%) to its teacher.
MSS-DPR (Sachan et al., 2021) MSS-DPR (Sachan al., 2021)
12
84.0
89.2
24
84.8
89.8
Our teacher
12 (2x124M)
80.7
87.3
Student proposed method Student w/ proposed method
(82M)
80.4
86.8
4 (16M)
77.4
85.3
Table 3: Performance of EmbedDistill for DE to DE dis- tillation on NQ test set. Note that the prior work men- tioned in the table rely on techniques such as negative mining and multi-stage training. In contrast, we explore the orthogonal direction of smbedding-mattchee that im- proves single-stage distillation, which be combined with the aforementioned techniques.
6.3 CE TO DE DISTILLATION
Method
MRR 10
[CLS]-pooled teacher
37.1
We consider two CE teachers for MSMARCO:
Dual-pooled teacher
37.0
standard [CLS]-pooled CE teacher, and the Dual-pooled CE teacher (ef. Sec. 4.2). Both teachers are based on RoBERTa-base and trained on triples in the training set for 300K steps with cross-entropy loss.
Standard distillation from [CLS]-pooled teacher
33.0
+Joint matching
32.4
Stundard distillation from Dual-pooled teacher +Query matching
33.3
33.7
Table 4: Performance of DE models distilled from [CLS]-pooled and Dual-pooled CE models on MSMARCO. While both teacher models per- form similarly, embedding matching-based distil- lation only works with the Dual-pooled teacher. See Appendix G for nDCG@ 10 metric.
Student DE model training. We consid- ered the following distillation variants: stan- dard ccsre-based distillation from the [CLS]- pooled teacher, and our novel Dual-pooled CE teacher (with and without embedding matching loss). For each variant, we initialize encoders of the student DE model with two RoBERTa-base models and train for 500K steps on the training triples. We performed the naive joint embedding matching for the [CLS]-pooled teacher (cf. Sec. 4.2) and employed the query embedding matching (cf. 5) for the Dual-pooled CE teacher. In either case, embedding matbbcinnngmmthinnggg loss is added on top of the standard cross entropy loss with the weight of 1.0 (when used).
Results and discussion. Table 4 evaluates the effectiveness of the dual pooling and the embedding matching for CE to DE distillation. As described in Sec. 4.2, the traditional [CLS]-pooled teacher did not provide any useful embedding for the embedding matching (see Appendix H.2 for the further analysis of the resulting embedding space). However, with the Dual-pooled teacher, embedding matching does boost student's 's performance.
7 CONCLUSION
We propose EmbedDistill - a novel distillation method for IR that goes beyond simple score match- ing. We specialize it to distill a DE model into another DE model by (a) reusing the teacher's document encoder in the student and (b) aligning query embeddings of the teacher and student. This simple approach delivers immediate quality and computational gains in practical deployments and we demonstrate them on MSMARCO and NQ benchmarks. We show that query generation technique further improves the performance of the distilled student. We generalize the proposed approach distill CE model to DE model and show the benefits MSMARCO. Finally, our theoretical analysis alludes to the favorable implications of both embedding matching and inheriting document encoder in DE to DE distillation setting. A more comprehensive and systematic analysis of embedding matching-bass based distillation for IR is an exciting avenue for future research.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Gustavo Aguilar, Yuan Ling, Yu Zhang, Benjamin Yao, Xing Fan, and Chenlei Guo. Knowledge distillation from internal representations. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7350-7357, 2020.
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. Synthetic QA cor- pora generation with roundtrip consistency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6168-6173, Florence, Italy, July 2019. Associa- tion for Computational Linguisties. doi: 10. 10.185//11//19-16222 URL https://aclanthology. org/P19-1620.
Olivier Bousquet, Stéphane Boucheron, and Gábor Lugosi. Introduction Statisticul Learning The- ory, pp. 169-207. Springer Berlin Heidelberg, Berlin, Heidelberg, 2004. ISBN 978-3-540-288650 9. doi: 0001//997788-4--288888 9_8. URL http:://boi..rr//...e////////// 8.
Cristian Bucilă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. Proceedings of the 12th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '06, pp. 535-541, New York, NY, USA, 2006. ACM.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pp. 1870-1879, Vancouver, Canada, July 2017. Association for Computational Linguisties. doi: 10. 0011863//1/P17--1771 URL https: //aclantte Maclanttolooy.org/eee 1171.
Defang Chen, Jian-Ping Mei, Hailin Zhang, Can Wang, Yan Feng, and Chun Chen. Knowledge distillation with the reused teacher classifier. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11933-11943 2022.
Xuanang Chen, Ben He, Kai Hui, Le Sun, and Yingfei Sun. Simplified tinybert: Knowledge distilla- tion for document retrieval. In Djoerd Hiemstra, Marie-Francine Moens, Josiane Mothe, Raffaele Perego, Martin Potthast, and Fabrizio Sebastiani (eds.). Advances in Information Retrieval, pp- 241-248, Cham, 2021. Springer International Publishing. ISBN 978-3-030-72240-11.
Zhuyun Dai and Jamie Callan. Deeper text understanding for IR with contextual neural language modeling. In Benjamin Piwowarski, Max Chevalier, Éric Gaussier, Yoelle Maarek, Jian- Yun Nic, and Falk Scholer (eds.), Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, pp. 985-988. ACM, 2019a.
Zhuyun Dai and Jamie Callan. Context-aware sentence/passage term importance estimation for first stage retrieval. arXiv preprint arXiv:1910.1663 2019b.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171-4186. Association for Computational Linguisties, 2019.
Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Ku- mar. Accelerating large-scale inference with anisotropic vector quantization. In International Conference on Machine Learning, 2020. URL http::://aaxiv.org//aaa arggaabs/998..0393..
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015.
Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. Im- proving efficient neural ranking models with cross crostarcch knowledge distillation. CoRR, abs/2010.02666, 2020. URL hwtttttss///aiii htw.voorg/aaaao argg/bss/o0/ e2666.
10

---

Under review as a conference paper at ICLR 2023
Sebastian Hofstätter, Sheng- Cheng-Chich Lin, theng-Hong Yang, Jimmy Lin, and Allan Hanbury. Ef- ficiently teaching an effective dense retriever with balanced topic aware sampling. In Pro- ceedings of the 44th International ACM SIGIR Conference Information Retrieval, SIGIR '21, pp. 113-122, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450380379.
Research and Development in
doi: 10.1145/3404835
URL https:
//doi. 10. 1t////.1.111///411 114511444/343551111 344153333448835.5888111
Gautier Izacard and Edouard Grave. Distilling knowledge from reader to retriever for question
answering. In International Conference on Learning Representutions, 2021. URL https://
openreview. www.eneeoooundddeeeeee 6wysdb.
Gautier Izacard, Mathild Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave, Unsupervised dense information retrieval with contrastive learning. arXiv preprint 11Xiv:2112.09118, 2021.
Xiaoqi Jiao, Yichun Yin. Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for natural language understanding. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2020, pp. 4163-4174, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/w 1/20300////222..// 12116655/1120000.mnnegemm...... https://aclanthool httpllloyyyorgg///// Findings-eenlp.23
URL
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 73)::33--577, 2021. doi: 10.100//BDDXAA.001999955222
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6788 6781, Online, November 2020. Association for Computational Linguistics.
Omar Khattab and Matei Zaharia. ColBERT: Efficient and Effective Passage Search via Contextual- ized Late Interaction over BERT, pp. 39-48. Association for Computing Machinery, New York, NY, USA, 2020. ISBN 9781450380164.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le. and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the URL
Association for Computational Linguistics, 7:452-466 466, 2019a. doi:
https://aclanthollog org//19- 1026.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019b.
Michel Ledoux and Michel Talagrand. Probability in Banach spaces. Springer- Verlag, 1991.
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pp. 6086-6096. Association for Computational Linguistics, 2019.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, On- line. July 2020. Association for Computational Linguistics. doi: 1/2286633/12022accl 703. URL https: http:///alcaatttoooooooooo org/ 2020. ac a- main.
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. Paq: 65 million probably-aske -sseed questions and what you can do with them. Transactions of the Association for Computational Linguistics, 9:1098-1115,
2021.
11

---

Sean MacAvaney, Andrew Yates, Kai Hui, and Ophir Frieder. Content-based weak supervision for ad-hoc re-ranking. SIGIR' 19, pp. 993-996 New York, NY. USA, 2019b. Association for doi: 10.1145/3331184433333 Under review as a conference paper at ICLR 2023
Canjia Li, Andrew Yates, Sean MacAvaney, Ben He, and Yingfei Sun. Parade: Passage representá- tion aggregation for document reranking, arXiv preprint arXiv:2008.0900 09093, 2020.
Sheng- Sheng-Chieh Lin, Jheng- Hong Yang, and Jimmy Lin. In-batch negatives for knowledge dis- tillation with tightly-coupled teachers for dense retrieval. Proceedings of the 6th Work- shop on Represenlation Learning for NLP (RepLANL.-2222) pp. 163-173, Online, August 2021. Association for Computational Linguistics. doi: 16.33/121112021 rrpp44nlp-1.17. 1epp..e URL
https://acllantholggr htt.://ooog............o.tooai-.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1wt..1997.11... 2019.
Wenhao Lu, Jian Jiao, and Ruofei Zhang. Twinbert: Distilling knowledge to twin-structured compressed bert models for large-scale retrieval. In Proceedingy of the 29th ACM Interna- fional Conference on Information & Knowledge Management, CIKM '20, pp. 2645-2652, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450368599. doi:
10.1145//340033.34424444 URL https: ort.:/...rg///.. 3145/3340531.
Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. Sparse, dense, and attentional representations for text retrieval. Transactions of the Association for Computational Linguistics, 9:329-345, 2021. doi: 10.1162/tacl 00369. URL https:://alaanthhloggg/.// org/ tec..taccl.....................a 20.
Ji Ma, Ivan Korotkov, Yinfei Yang, Keith Hall, and Ryan McDonald. Zero-shot neural pas- sage retrieval via domain-targeted synthetic question generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lingnistics: Main Volume, pp. 1075-1088, Online, April 2021. Association for Computational Linguistics. doi: URL https::/aclanttholloee org/ 2021.
10.18653/v1/2021.eaa
92.
Sean MacAvaney, Andrew Yates, Arman Cohan, and Nazli Goharian. CEDR: Contextualized em- beddings for document ranking. In Proceedings of the 42nd International ACM SIGIR Confer- ence on Research and Development Information Retrieval, SIGIR'19, 19, pp. 1101-1104, New York, NY, USA, 2019a. Association for Computing Machinery. ISBN 9781 9781450361729. 00.1145/33311144333333 URL /ttp:::::/d..oooo org/10. 1ttp:///....111/311188.333333 3331317.
doi:
Computing Machinery. ISBN 9781450361729.
URL https:
//doi. orgoi.org/// 1145/3331184. 33115/333118.3333316.. 3331316.
Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, Nazli Goharian, and Ophir Frieder. Efficient Document Re-Ranking for Transformers by Precomputing Term Repre- sentations, pp. 49-58. Association for Computing Machinery, New York, NY, USA, 2020. ISBN 9781450380164.
Aditya Krishna Menon, Sadeep Jayasumana, Seungyeon Kim, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. In defense of dual-encoders for neural ranking, 2022. URL https: //openreview net httt/forrumidd=bguee PqqQQ
Bhaskar Mitra and Nick Craswell. An introduction to neural information retrieval. Foundations and Trends@ in Information Retrieval, 13(1):1-126, 2018. ISSN 1554-0669. doi: 10.1561/ 1500000061. URL http:://xx...o///////////// http//xxxdooo...//o www.doi..oooooo.... org/10. 11556/556000006.111
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A human generated machine reading comprehension dataset. In Tarek Richard Besold, Antoine Bordes, Artur S. d' Avila Garcez, and Greg Wayne (eds.), Proceed- ings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016, volume 1773 of CEUR Workshop Proceedings. CEUR-WS.org, WSS.or, 2016.
Ping Nie, Yuyu Zhang, Xiubo Geng, Arun Ramamurthy, Le Song, and Daxin Jiang. DC-BERT: decoupling question and document for efficient contextual encoding. In Jimmy Huang, Yi Chang,
12

---

Under review as a conference paper at ICLR 2023
Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu (eds.). Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pp. 1829-1832. ACM, 2020. doi: URL
10.1145/339727113444 1344012777..
org/10. 1011111133333 34012777...
Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with BERT. CoRR, abs/1901.0440 .00085, 2019. URL http:://axxxv..rgabbbssss 0408..
Rodrigo Nogueira, Jimmy Lin, and Al Epistemic. From doc2query to docttittquery. Online preprint, 6, 2019a.
Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. Document expansion by query prediction. arXiv preprint arXiv: (004.99444477 2019b.
Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. Document ranking with pre- trained sequence-to- -euuunneee model. In Findings of the Association for Computational Linguis- tics: EMNLP 2020, pp. 708-718 Online, November 2020. Association for Computational Lin- URL https://aclanthollonn https////accannoooooooooo/////// org/2020.
guisties. doi: 10. 18653/v 1/ 1/2020. grnpDofinning--mmlp..6.3
findings- emn 1ammmp.66... 63.
Barlas Oguz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Pik- tus, Xilun Chen, Sebastian Riedel, Wen-tau Yih, Sonal Gupta, al. Nomain-matchen pre-training tasks for dense retrieval. arXiv preprint arXiv:2107 1177..13602 36002, 2021.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. RocketQA: An optimized training approach dense passage retrieval for open-domain question answering. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tite Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.). Proceedings of the 2021 Conference of the North American Chupter of the Associa- tion for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pp. 5835-5847. Association for Computational Linguistics, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text- to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http: //jmtt.org//aatee v2tp//v2//////// 2022200-77.....
Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. Rocketqav2: A joint training method for dense passage retrieval and passage re- ranking. In Proceedings of EMNLP, 2021.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint 14Xiv::412..65 2014.
Devendra Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L. Hamil- ton, and Bryan Catanzaro. End-to-end training of neural retrievers for open- open-domain question answering. In Proceedings of the 59th Anmual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pp. 6648 66662, Online, August 2021. Association for Computational Linguistics. doi: N0.18653/v1/20211
1a222221.allloonn long.511. URL htt http:///aaaaattoo....oooooo wwwp//rr..//ec..e.e Long.
Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. Improving passage retrieval with zero-shot question generation. arXiv preprini arXiv:2204.00) 007499 2022.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.0/108. 2019.
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. Col- bertv2: Effective and efficient retrieval via lightweight late interaction. CoRR, abs/21 12.01488, 2021.
Julia Ture, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv: / /900.0000000 00960, 2019.
13

---

Under review as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Proceedings of the 31st Inter- national Conference Neural Information Processing Systems, NIPS' 17, pp. 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 578151086096..
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Aroold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations, 2021. URL https: //apenreview.ee net/ for
Zeynep Akkalyoncu Yilmaz, Wei Yang, Haotian Zhang, and Jimmy Lin. Cross-domain modeling of sentence-level evidence for document retrieval. In Proceedings of the 2019 Conference Em- pirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (FMNL.-IJCNIIP), pp. 1490-3396, Hong Kong, China, November 2019. Association for Computational Linguistics.
Linfeng Zhang and Kaisheng Ma. Improve object detection with feature-based knowledge distilla- tion: Towards accurate and efficient detectors. In International Conference on Learning Repre- sentations, 2020.
Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. Deep learning based recommender system: A survey and new perspectives. ACM Comput. Surv., 52(1), feb 2019. ISSN 0360-0300. doi: 10.1145/3285029, URL https://doi.onn org.ao.rr//o.oo 1145/118502222
Chen Zhao, Chenyan Xiong, Jordan Boyd-Graber, and Hal Daumé III. Distantly-supervised dense retrieval enables open-domain question answering without evidence annotation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 9612-9622, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2000 htt0://....m...mmmaaiil emn lp- main.
wemnlp-mainn7555 7566. URL https::/actantthooogg... org.ooy....rg......
Zhi Zheng, Kai Hui, Ben He, Xianpei Han, Le Sun, and Andrew Yates. BERT-QE: Contextualized Query Expansion for Document Re-ranking. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp.
Online, November 2020. Association for Computa-
tional Linguistics. doi: 10.18653/v 100/3//112020..0 1/2020..nninng-eemllpp.... 424. URL org/202e. eeweiinngseeell.....
14

---

Under review as a conference paper at ICLR 2023
A LOSS FUNCTIONS
Here, we state various (per-cxample) loss functions that most commonly define training objectives for IR models. Typically, one hot training with original label is performed using softmax-bass cress-entropy loss functions:
exp(s(qi, d(,5))
Σ
yiu log
Σ
exp(s(gi, di.
SE(L]
J'ELL
Alternatively, it's also common employ one-vs-all loss function based binary cross-entropy loss as follows:
(si.d....)) =
Σ JE(L]
yoit log
1 (1
dri)))
+
(1 (1- Vi.)) log (1 exp(s(qs, )
(9)
As for distillation, one can define distillation objective based on the softmax based cross-entropy loss as
exp(st.g)
exp(s
la (a :g. = E
log
(10)
E()) exp(s
Eye( exp(si
jE[N]
where si == sq d) and sh,, == (qi, di denote the teacher and student scores, respectively. On the other hand, the distillation objective with the binary cross-entropy takes the form:
la la(%), (.. = Σ
1
log
+
expl
exp(- (
jE[L]
log
(1I)
+
exp(st,)
exp(si,)
Finally, distillation based on the meas square error (MSE) loss (aka. logit matching) employs the following loss function:
La La sqyde = Σ
(s - (g:
(12)
jELL]
B DUAL POOLING DETAILS
In this work, we focus on two kinds of dual pooling strategies:
Special tokens-based dual pooling, Let poolcLs and pOolSEP denote the pooling opera- tions that return the embeddings of the [CLS] and [SEP] tokens, respectively. We define emb't (-..dd pool (Enc²(0)),
emb (q = pool (Fnce
(13)
where ő denotes the input token sequence to the Transformers-based encoder, which con- sists of { query, document, special } tokens.
Segment-based weighted-mean dual pooling, Let Enc'(o)) and Enc' Enc'(o) denote the final query token embeddings and document token embeddings produced by the encoder, respectively. We define the proxy query and document embeddings
emb) ( mean (Enc' (o)l)))
emb 2 (q.4) = mean (Enc² (ō)[p).
(14)
where mean (-) denotes the weighted mean operation. We employ the specific weighting scheme where each token receives weight equal to the inverse of the square root of the token-sequence length.
It is common to employ temperature scaling with softmax operation. We do not explicitly show the tem- perature parameter for ease of exposition.
15

---

Under review as a conference paper at ICLR 2023
C DEFERRED DETAILS AND POOFS FROM SECTION 5
In this section we present more precise statements and proofs of Proposition and 2 (stated infor- mally in Section 5 of the main text) along with the necessary background. First, for the ease of exposition, we define new notation which will facilitate theoretical analysis in this section.
Notation Denote the query and document encoders as f: Q + R$ and D + R for the student, and F: Q + R. G: - E for the teacher (in the dual-encoder setting). With denoting a query and d denoting document, [()) and g then denote query and document embeddings by the student. We define F()) and (dd) similarly for embeddings by the teacher."
C.I BOUND ON THE EXPECTED RISK
Proposition 3 (Formal statement of Proposition 1). Given example (g,d,y) E D x {0,1}, let sf. d) f(a) g(d) be the scores assigned to the (q, d) pair by a dual- model with and g as query and document encoders, respectively. Let & and la be the binary cross-entropy loss (cf. Eq. 9 with L = 1) and the distillation-specife loss based on it (cf. Eq. 11 with L = 1), respectively. In particular,
f(s (q. == yylog (F(4)
((d)) G(d) (1 - log [1 - (F())
(q, = - (F()) G(d)) log a (f(g) g(d))
[1 -1-a (F(2) G(d)) log [1 (f(g) 9(d))
(f())T((d))
where a is the sigmoid function. Assume that
1. All encoders J,g, F, and G have the same output dimension k 2
2. There exist KQ, KD E oo) such that supge max (11f(1)122 (()()) s Kę and supdep max (llg(d) 22, [[G(d]]l2} I KD.
Given sample [lgs, di v1}} 3nd P(q, d, y). we have
la(s! (9 (q,, di), P.G d,)) El(s (E(sFF((g.d)))) 2) < 22KgB(ll(44) G(d)laa
E
2K (1I5 - F())ll)) + Kg la (F(1) G(d) - yl
Proof of Proposition 3. We first note that the distillation loss can be rewritten fa(sf (q, d), (q, where >(v) = log[1 ee"l is the softplus functtion Similarly, the one-hot (label-dependent) loss can be rewritten as
F.G (q, d)) (1 (F G(d) f(4) g(d) + x(-((g)) g(d)),
( (g, d). = (1 y)
As shorthand, define
R(S.)) = 1 Σ Ea( (g (4), 8 P.G (ge, d4),,
F.G (g, R(F,G) := (E)dd ( d).d).))
as the empirical risk based on the distillation loss, and the population risk based on the label- dependent loss, respectively. With this notation, the quantity to upper bound can be rewritten as
=0
=00
E - =E R(f.g) -(f.)R((++++++)) R(f.G) R(F,G) R(F.)) R(F.G)
R(F.))
(15)
= 'Note that, as per the notations in the main text, we have (f.g) = (Encă, Ench) and (F,G) (Encyy Ench)). Similarly, have (emb), emb')) (f( (dd) and (emb),, emb;) ((q))G(d))
16

---

Under review as a conference paper at ICLR 2023
We start by bounding E [ as
B La (q: ;
_ Σ Ea Ca(ssiCaan (gn, F (gi
"
E (, (q
[ ((00--10F0)((0)))111000000000110
E [3()-
(
od f() G(4]
- 14000000--0-00----000-00-------0
K& [ll(d] -
G(d)))
2 (lg - (2)
(16)
where (a) we use the fact that a Lipschitz continuous function with Lipschitz constant 1, and at (b) we use Cauchy-Schwar inequality.
Similarly for [[]. we proceed
2 =
Ea La ( (qi,d),, d (Fr,) -
Σ Ea F.G. 8 di))
-=1
E ((1=(((())((()) ((0
= (
s 1 [IG(d)]I1(() - =(0)
S B (1) F(2)12)
(17)
3 can be bounded
B gg] E[R(F, G) R(FF,G
R(F,G))
B(g,du))
B(g,dwv) - ( +(
- ((gd)))) (()
B(g.dw) (1-o(F(g)) G(d)) -)} (()))(())
I KgKDBeadt
G)))) -//-
(18)
Combining (15), (16), (17), and (18) gives the result.
C.2 UNIFORM DEVIATION BOUND
Let I denote the class of functions that map queries in Q their embeddings in R* via the query encoder. Define analogously for the doc encoder, which consists of functions that map documents D to their embeddings in & To simplify exposition, we assume that each training example consists of single relevant or irrelevant document for each query, i.e., L in Section 3. Let
F9 = d(. + f(g) T g(d) | f e 9}
Given 8, = ((i de, v.) : [m]]. let (e, denote the e-covering number of function class H. with respect to (Px)) norm, where [|bll2,,(..)) = |bb] = 1. Σ [l((q, d.)11. Depending on the context, the functions in H. may map IR or Rdd
17

---

Under review as a conference paper at ICLR 2023
Proposition 4. Let s' be scorer of teacher model and la be distillation loss function which is -iipschcit in its first argument. Let the embedding functions in 3 and 9 output vectors with & norms most K. Define the uniform deviation
sup - E
(,
(a((()))) gdd)) (F) g(d), sq.))
fEJ
For any 9" EG. have
4
Es, (5, EEs..
Vn Vlog.Nu, N Nu,NN, (u, + log N (u, du,
48K
Es, En (F, (F. Es,
Vlog Nu, F) du.
vn
Proof of Proposition 4. We first symmetrize excess risk to get Rademacher complexity, then bound the Rademacher complexity with Dudley's ' entropy integral.
For training set S,, the empirical Rademacher complexity of class of functions H that maps x D to defined by
Rad, (FC) = E, sup Σ hEDr ==1
1
Ech(gs d:).
where {e,} denote i.i.d. Rademacher random variables taking the value in {+1, with equal probability. By symmetrization (Bousquet et al., 2004) and the fact that la is Lea Lipschitz in its first argument, we get
Es,, & (3, - 2L, Es, Rad,
Then, Dudley's entropy integral (see, e.g., Ledoux & Talagrand, 1991) gives
12
( <
Vlog (u, du.
Vn J
From Lemma with K2 = KD K, for any u > 0,
Nuu,99) S N
2K F) N
2K 9)
Putting these together,
24Lta Es, (5,
Vlog N (u/2K,,)) + log N (u/2K, 9) du.
(19)
Vn
Following the same steps with 9 replaced by {g* }, we get
24Tx,
Es, E (F, (9
Nlu/2K du
(20)
vn
By changing variable in Eq. 19 and Eq. 20, we get the stated bounds.
For f + R.a g: R define fg x R by folq, d) = f(1) Lemma 1. Let fi,..., fN be 6-cover o and g1, 91.-..,9h 99s be 6-cover of 9 in L2(P,) norm. Let sup, sup suppss |
S Ko and supges supaED (lg(d)]l2 S Kp. Then,
(fa [ [M]
is a (KQ + K-cver JG.
Proof Lemma 1. For arbitrary E F, E S, there exist j E (f1y...,N).b (91...-9MM gM} such that || fllxx I 1lg - Øll. < c. It is sufficient show that llfg - fūln I (Kş + KD)c. Decomposing using triangle inequality.
llfg föll. = [[g fg + fg fğll.
< [[g fōll. n. + |[g fg l.
(21)
18

---

Under review as a conference paper at ICLR 2023
To bound the first term, using Cauchy-Schwartz Schwartz inequality, we can write
1
(fl)) g(d,) - f(()) I(di)) - sup f
Σ 11 - (d)))
T2 2m1
qED
Therefore
flfg fōln, s Kellg 11 T11n Kge.
Similarly
llfg fölln - Kollf - flln - D
Plugging these in Eq. 21, we get
llfg fölnn S (Ko KD)E.
This completes the proof.
D EVALUATION METRIC DETAILS
For NQ, we evaluate models with full strict recall metric, meaning that the model is required to find a golden passage from the whole set of candidates (21M). Specifically, for k 1, recall @ & or R@ k denotes the percentage of questions for which the associated golden passage is among the k passages that receive the highest relevance scores by the model. In addition, we also present results for relaxed recall metric considered by Karpukhin et al. (2020), where @@k denotes the percentage of questions where the corresponding answer string is present in at least one of the & passages with the highest model (relevance) scores.
For MSMARCO, we follow the standard evaluation metries Mean Reciprocal Rank(MRRR @ 10 and normalized Discounted Cumulative Gain (nDCG)@ 10, which are computed with respect to BM25 generated 1000 candidate passages for each query. We report 100 x MRR@ 10 and 100 xnDCG@ @ 10, per the convention followed in the prior works.
E QUERY GENERATION DETAILS
We introduced query generation to encourage geometric matching in local regions, which can aid in transferring more knowledge confusing neighborhoods. As expected, this further improves the distillation effectiveness top of the embedding matching. To focus the local regions, we generate queries from the observed examples by adding local perturbation in the data manifold (embedding space). Specifically, we employ off-the-shelf encoder-decoder model (BART). First, we embed an observed query in the corresponding dataset. Second, we add a small perturbation the query embedding. Finally, we decode the perturbed embedding generate new query in the input space. Formally, the generated query x' given an original query æ takes the form x' = Dec(Ene(2) + c), where Enc() and Dec() correspond to the encoder and the decoder from the off- the-shelf model, respectively, and is an isotropic Gaussian noise. Furthermore, we also randomly mask the original query tokens with small probability. We generate two new queries from an observed query and use them as additional data points during our distillation procedure.
As a comparison, we tried adding the same size of random sampled queries instead of the ones generated via the method described above. That did not show any benefit, which justifies the use of our query/question generation method.
F ADDITIONAL TRAINING DETAILS
Training for teacher models. For the teacher DE model on NQ, we initialize its question and document encoders by two pre-trained RoBERTa-base models (12 layers). Following (Oğuz al., 2021), the model is further pre- pre-trained on PAQ (Lewis et al., 2021) for 800K steps, and then fine- tuned on NQ train set with the help of in in-batch negatives (Karpukhin et al., 2020) for 40K step.
As for the teacher DE model on MSMARCO, it's known that directly training a DE model on MSMARCO training set leads to poor performance (Menon al., 2022). Thus, first train
19

---

Under review as a conference paper at ICLR 2023
[CLS]-pooled CE model triples in MSMARCO training set by using cross-entropy loss. We subsequently use the same triples to distill the resulting CE model to a DE model that has two pre- trained RoBERTa-base models its two encoders. We utilize cross-entropy based distillation loss in Eq. 10.
Optimization. For all of our experiments, up period and a linear decay schedule. We use the initial learning rate of 10-5 and 2.8 x 10 for experiments NQ and MSMARCO, respectively. We chose batch sizes to be 128.
ADAM weight decay optimizer with short warm
G ADDITIONAL EXPERIMENT RESULTS
G.1 ADDITIONAL EXPERIMENT RESULTS ON NQ
See Table 5 for the performance of various DE models on NQ, as measured by the relaxed recall metric.
Recall
Recall @
Recall @ 100
Method
82M
16M
82M
16M
82M
16M
Train student directly
82.0
61.7
84.4
79.9
93.8
90.6
Distill from teacher
69.8
64.6
84.5
79.7
92.4
89.2
Inherit document embeddings
76.3
67.5
88.7
81.0
94.3
89.6
Query embedding matching
80.6
75.9
91.4
88.1
96.3
94.5
Quety generation
80.7
77.1
91.6
89.2
96.5
94.9
Train student only using embedding
matching and inherit doe embeddings
81.0
75.5
91.7
87.3
96.4
93.8
Query generation
91.8
88.5
96.5
94.5
Table 5: Relaxed recall performance of various student DE models on NQ dev set, including sym- metric DE student model (82M or 16M transformer for both encoders), and asymmetric DE student model (82M or 16M transformer as query encoder and document embeddings inherited from the teacher). All distilled students used the same teacher (1 (110M parameter RoBERTa-base models as both encoders), with the performance (in terms of relaxed recall) of Recall @5 82.5, Recall @ 20 = 92.6, Recall 100 97.1.
G.2 ADDITIONAL EXPERIMENT RESULTS ON MSMARCO
See Table for CE to DE distillation results on MSMARCO, as measured by the nDCG@ 10 metric.
Method
nDCG@10
[CLS]-pooled teacher
43.0
Dual-pooled teacher
42.8
Standard distillation from [CLS]-tcacher
38.8
+Joint matching
38.0
Standard distillation from Dual-pooling teacher
39.2
+Query matching
39.4
Table 6: Performance of CE to DE distillation on MSMARCO, as measured by the nDCG@ 10 metric. As for the teacher CE models, consider kinds of CE models based on two different pooling mechanism.
H EMBEDDING ANALYSIS
H.1 DE TO DE DISTILLATION
Traditional score matching-base distillation might not result in transfer of relative geometry from teacher to student. To assess this, look at the discrepancy between the teacher and student query embeddings for all q, q' pairs: /mmb; emb;, emb emb ||- Note that the analysis based on NQ, and focus the teacher and student DE models based RoBERTa--base
20

---

Under review as a conference paper at ICLR 2023
15.
Standard distillation
Embedding
matching
Density
10
5
0
-1.0
-0.5
0.0
0.5
1.0
Discrepancy in distance from teacher
Figure 4: Histogram of teacher-student distance discrepancy in queries.
All positive
All negative
(q, d pairs
(g. pairs
000
[CLSle CE model
Dual pooled Pairwise distance matrix
[CLSS-pooled
9, macy credit card phone number
(q.dpp) (qidii)
92' phone number to experian credit bureau
** (gi,d)) d()
(gt, díy) (e:dy (g, dgp))
Dual pooled model
9, colloids chemistry definition
4, is phosphorus diatomic
(gs, dit)
(gs, dhi))
(gs, 4,) (as deg)
(gsddi)
(gs, dzy)
(g).d))
(gs, dia)
9g what cancer doctor called
(gts
(q.. d
(gr, djj
(gr. dii)
9. physiological disease
Figure 5: Illustration of geometry expressed by [CLS]-pooled Ceadee our Dual-pooled CE model on queries from MSMARCO and 12 passages based pairwise distance matrix across these 72 pairs. [CLS]-pooled CE embeddings degenerates as all positive and negative query-document pairs almost collapse two points and fail capture semantic information. In contrast, our Dual-pooled CE model leads to much richer representation that can express semantic information.
and DistilRoBERTa, respectively. As evident from Fig. 4, embedding matching loss significantly reduces this discrepancy.
H.2 CE TO DE DISTILLATION
We qualitatively look at embeddings from CE model in Fig. 5. The embedding emb;, from [CLS]- pooled CE model does not capture semantic similarity between query and document as it is solely trained classify whether the query-document pair is relevant or not. In contrast, the (proxy) query embeddings emb', emb;)(yydd) from our Dual-pooled CE model with reconstruction loss do not degener- ate and its embeddings groups same query whether conditioned together. Furthermore, other related queries are closer than unrelated queries. Such informative embedding space would aid distillation to DE model via embedding matching.
positive negative document
21


Reviews:
[{"review_id": "q2NFrp27k7A", "reviewer": "Reviewer_VV1L", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new distillation approach that supports both retrieval and re-ranking stages and crucially leverages the relative geometry among queries and documents learned by the large teacher model. The paper provides stronger signals about local geometry via embedding matching and attains better coverage of data manifold globally via query generation.", "summary_of_the_review": "Overall, the paper is well-written and easy to follow. But it can be greatly improved after the concern about the technical details on question generation and the experimental settings. ", "strength_and_weaknesses": "Strength:\n1. The paper is generally well-written and easy to follow.\n2. The paper proposes a new idea of distillation approach that goes beyond score matching and aligns the embedding spaces of the teacher and student models.\n3. The paper has solid theoretical analysis on the proposed method\n\nWeakness:\n1. The question generation method is not well described. As a main contribution emphasized by the authors, more details are required.\n2. Both the CE and DE models are based on the RoBERTa-base or RoBERTa-mini, where the model size is no larger than 124M. The authors can try some larger models to show the distillation performance on these larger models.\n3. The idea that aligns the embedding space between teacher and student seems not novel.\n4. No distillation baselines are compared in the paper. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "pdOLVEifwG", "reviewer": "Reviewer_CgJL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel embedding-based knowledge distillation method for Transformer in QA, including DE2DE and CE2DE. Moreover, they give a theory that proves the effectiveness of the distillation method on the generalization of student networks. ", "summary_of_the_review": "It is a valuable topic to perform feature-based distillation for transformer in QA tasks. But more comparisons are needed to prove the effectiveness, and the writing can be improved. Moreover, embedDistill brings an extra aligned projection matrix, which may take more training resources.", "strength_and_weaknesses": "Strength:\n\n     -- The theory proof makes the method more convincing.\n     -- Try to distill the knowledge based on the embeddings(or, say, outputs) from Transformer.\n\n\nWeakness:\n\n    -- Poor writing and too many long sentences in the Introduction Section.\n    -- Lack of comparison. The experiments seem poor, and they only compare the proposed embedDistill with naïve logit-based distillation, and they do not compare with other methods mentioned in related work. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cj0GtaPWjKj", "reviewer": "Reviewer_zx15", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new distillation technique for learning Information Retrieval (IR) models whereby not only the scores of the documents are aligned with the teacher, but also the representations. The paper exhibits two propositions showing (1) a relationship involving the empirical distillation loss and the representations (2) the bias between the empirical and expected distillation loss. Experiments show that the regularization is based on embeddings. The authors show that this improves the result with two types of distillation (dense to dense, and cross-encoder to dense).\n", "summary_of_the_review": "This paper proposes to regularize the distillation loss by aligning the query and document representations (or a substitute when they are not readily available, e.g. for cross-encoders), and clearly shows its benefit on two standard IR datasets. The theoretical section fails to show anything (IMHO) that could really be leveraged or provide insight on what this distillation is doing, and this lessen the interest of the paper (a qualitative or more focused theoretical analysis would have been more valuable).\n", "strength_and_weaknesses": "This paper shows that aligning embeddings (with an ad-hoc procedure in the case of cross-encoders where queries and documents do not have a dense representation) helps improve the performance of student models when distilling. \n\nThe various experiments conducted clearly show that the proposed methodology do help in a variety of settings (dense to dense, cross-encoder to dense, dense to lighter dense) with two representative datasets (MS Marco and Natural Questions), showing the robustness of the approach. \n\nThe paper provides two propositions justifying the approach (I did not check the proofs in the appendix, but they seem to be sensible), but I failed to understand how they help showing in theory how aligning the embeddings does help improving the student:\n\n1) Proposition 1 shows that the difference between the distillation loss and the teacher loss is bounded by the expectation of embeddings distance and a dense teacher loss: how this shows that minimizing the proposed loss (distillation loss + embeddings loss) will help improving generalization?\n\n2) Proposition 2 states that the difference between the empirical and theoretical loss is bounded by some quantity that is lower when one part of the network is frozen (i.e. the document embeddings part in this paper). If this is the case, first, it seems that just proving the bound would be enough (which should be doable since the Rademacher complexity of the sum of two classes of functions is the sum of their complexities, by decomposing the query representation as $q + \\delta_q$ and the same for the document); after that, I wonder what is the practical or theoretical value of such bounds in this paper's context.\n\nI might add that from a theoretical point of view, it would have been interesting to understand (1) what is the last term of the equation of proposition 1 and how it relate to the performance of the model (2) to study the difference between the \"[CLS]\" classification and the one based on the dense embeddings obtained through pooling.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VFUdmjTjquO", "reviewer": "Reviewer_mS5Z", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a distillation approach that supports both retrieval and re-ranking stages and crucially leverages the relative geometry among queries and documents learned by the large teacher model. ", "summary_of_the_review": "This paper explores two ways of distilling into dual-encoder text retrieval models. However, it lacks novelty and comparison with other approaches.", "strength_and_weaknesses": "## Strength\n1. This paper provides a theoretical justification of the proposed method.\n1. This paper includes a good analysis of the embeddings, which empirically justifies their approach.\n1. The ablation study (Table 2) is informative.\n\n## Weaknesses\n1. The mathematical expressions seem to be overused. There are a lot of unnecessary formulas in the paper, e.g., Eq. 1,2,3\n1. The proposed method is not new as there have been many works on distillation for retrieval, both CE to DE and DE to DE.\n1. There are mentions of prior studies on text retrieval + knowledge distillation. However, they are not compared in the experiments. \n1. I hope to see evaluation on more datasets, e.g., BEIR benchmark. \n1. The performance of the student model is somewhat disappointing. It's maybe because of the weak teacher. Why not use a SOTA model as the teacher?\n\n#### Minor issues:\n1. Figure 1 is too small.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject