LexMAE: Lexicon-Bottlenecked Pretraining for Large-Scale Retrieval
Authors: Tao Shen
Xiubo Geng
Chongyang Tao
Can Xu
Xiaolong Huang
Binxing Jiao
Linjun Yang
Daxin Jiang
Venue: ICLR 2023 poster (2023)
Keywords: Self-Supervised Learning
Lexicon Representation
Large-Scale Retrieval

Abstract: In large-scale retrieval, the lexicon-weighting paradigm, learning weighted sparse representations in vocabulary space, has shown promising results with high quality and low latency. Despite it deeply exploiting the lexicon-representing capability of pre-trained language models, a crucial gap remains between language modeling and lexicon-weighting retrieval -- the former preferring certain or low-entropy words whereas the latter favoring pivot or high-entropy words -- becoming the main barrier to lexicon-weighting performance for large-scale retrieval. To bridge this gap, we propose a brand-new pre-training framework, lexicon-bottlenecked masked autoencoder (LexMAE), to learn importance-aware lexicon representations. Essentially, we present a lexicon-bottlenecked module between a normal language modeling encoder and a weakened decoder, where a continuous bag-of-words bottleneck is constructed to learn a lexicon-importance distribution in an unsupervised fashion. The pre-trained LexMAE is readily transferred to the lexicon-weighting retrieval via fine-tuning. On the ad-hoc retrieval benchmark, MS-Marco, it achieves 42.6% MRR@10 with 45.8 QPS for the passage dataset and 44.4% MRR@100 with 134.8 QPS for the document dataset, by a CPU machine. And LexMAE shows state-of-the-art zero-shot transfer capability on BEIR benchmark with 12 datasets. 

Published as a conference paper at ICLR 2023
LEXMAE: LEXICON-BOTTLENECKEN PRETRAINING FOR LARGE-SCALE RETRIEVAL
Tao Shen, Xiubo Geng, Chongyang Tao, Can Xu, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang"
Microsoft. "shentaa.xigeng.choen caxu, binxjaaliniyaa.jjanng) @iicrosoft.com
ABSTRACT
In large-scala serge-scaleaaee retrieval, the eexicon-weightie paradigm, learning weighted sparse representations in vocabulary space. has shown promising results with high quality and low latency. Despite it deeply exploiting the lexicon-rereseenting capability of pre-trained language models, a crucial gap remains between language modeling retrieval the former preferring certain or low-entropy words whereas the latter favoring pivot or high-entropy words becoming the main barrier to lexicon-weighting performance for large-scale retrieval. To bridge this gap, we propose brand-new pre- pre-training framework, masked autoencoder (LexMAE), to learn importance-awant lexicon representations. Essentially, we present exsieon-bttteenecked module between a normal language modeling encoder and a weakened decoder, where a continuous bag-of-word bot- tleneck is constructed learn a rexicon-importaen fashion. The pre- pre-trained LexMAE is readily transferred to the lexicon- weiihting retrieval via the-tuunng. On the ad-hoc retrieval benchmark, MS MS-Marco, achieves 42.6% MRR @ 10 with 45.8 QPS for the passage dataset and 44.4% MRR @ 100 with 134.8 QPS for the document dataset, by CPU machine. And LexMAE shows state-of-the-ard zero-shot transfer capability on BEIR benchmark with 12 datasets.
and lexicon-weight
distribution in an unsupervised
INTRODUCTION
Large-scale retrieval, also known as first stage retrieval (Cai et al. 2021), aims to fetch top query- relevant documents from huge collection. In addition to its indispensable roles in dialogue systems (/hao et al. 2020), question answering (Karpukhin et al. 2020), search engines, etc., it also has been surging recent cutting-edge topies, e.g., retrieval-augeeeeeeee augmented generation (Lewis al. 2020) -augmented language modeling (Guu et al., 2020). As there are millions to billions of documents in collection, efficiency is the most fundamental prerequisite for large-scale retrieval. To this end, query-agnoste document representations (i.e., indexing the collection independently) and lightweight relevance metrics (e.g., cosine similarity, dot-product) have become the common practices to meet the prerequisite usually achieved by two-tower structure (Reimers & Gurevych 2019) a.k.a., bi-encoder and dual-encoder, representation learning literature.
and
Besides the prevalent 'dense-vectes retrieval' paradigm that encodes both queries and documents in the same low-dimension, real-valued latent semantic space (Karpukhin al.) 2020), another retrieval paradigm, 'lexicon-weightin retrieval', aims to leverage weighted sparse representation in vocabulary space (Formal et al. 202 Shen et al., 2022). learns to use a few lexicons in the vocabulary and assign them with weights represent queries and documents sharing a high-level inspiration with BM25 but differing in dynamic (with compression and expansion) lexicons and their importance weights learned in end-to-end manner. Although learning the representations in such high-dimensional vocabulary space seems intractable with limited human-annotated query-document pairs, recent surging pre- pre-trained language modeling (PLM), especially the masked language modeling (MLM), facilitates transferring context-aware retrieval by fine-tuning the PLM on the annotated pairs (Formal et al.) 2021b Shen al. 2022). Here, coordinate terms (full of synonyms and concepts) are highly related to relevance-centn tasks and mitigate the lexicon mismatch problem (Cai al. 202T), leading to superior retrieval quality.
knowledge into exxicon-weighting
"Corresponiing author.

---

Published as a conference paper at ICLR 2023
Due to the pretraining- weerraniingfinetuninng consistency with the same output vocabulary space, eixicon-baseld retrieval methods can fully leverage a PLM, including its masked language modeling (MLM) head, leading to better search quality (e.g., ~ 1.0% MRR @ 10 improvement over dense-vector ones by fine-tuning the same PI.M initialization (Formal et al. 2021a Hofstätter et al.
2020). Meantime,
attributed the high-dimensional sparse-control lable representations (Yang et al. 2021 Lassance &
Clinchant 2022), these methods usually enjoy higher retrieval efficiency than vector ones (c.g.. 10x faster with the identical performance in our experiments).
Nonetheless, there still exists a subtle yet crucial gap between the pre-training language modeling and the downstream saaicon-weightnng objectives. That is, MLM (Devlin al. 2019) aims to recover a word back given its contexts, so it inclines to assign high scores to certain (i.e., low-entropy) words, but these words are most likely to articles, prepositions, etc., or belong to collocations or common phrases. Therefore, language modeling is in conflict with the lexicon- weeiicnnweightingg representation for relevance purposes, where the latter focuses more on the high-entropy words (e.g., subject, predicate, object, modifiers) that are essential to the semantics of query or document. These can explain, in our experiments when being fine-tuned under the paradigm of lexicon- lexicon-weighting retrieval (Formal et al. 2021a), why moderate PLM (i.e., DistilBERT) can even outperform relatively large one (i.e., base) and why a well-trained PLM (e.g., RoBERTa) cannot even achieve a convergence.
To mitigate the gap. in this work we propose brand-new pre- pre-training framework, dubbed lexicon- bottlenecked masked autoencoder (LexMAE), to learn importance-aware lexicon representations for the transferable knowledge towards large-scale texicon-weighting retrieval. Basically, LexMAE pre-trains a language modeling encoder for document-specific lexicon-importanee distributions over the whole vocabulary reflect each lexicon's 's contribution to the document reconstruction. Motivated by recent dense bottleneck-enhacee pre-training (Gao & Callan 2022 Liu & Shao [022] Wang ct al. 2022). we present to learn the lexicon- -mpoortacee distributions in an unsupervised fashion by constructing continuous bag-of-wo sere-of-orrds (CBoW) bottlenecks upon the distributions.
LexMAE LeXMAE pre-training architecture consists of three components: i) a language modeling encoder (as most other PLMs, e.g., BERT, RoBERTa), ii) dexicon-bottlenee a weakened masking- decoder. Specifically, a mask-corrupted document from the collection is passed into the language modeling encoder to produce token-level LM logits in the vocabulary space. Besides an MLM objective for generie representation learning, a max max-pooling followed by normalization function is applied to the LM logits to derive lexicon- rexicon-importance impoorance distribution. To unsupervisedly learn such a distribution, the weights to produce a CBoW dense bottleneck, while the weakened masking-styl decoder is asked to reconstruct the aggressively masked document from the bottleneck. Considering the shallow decoder and its aggressive masking, the decoder in LexMAE is prone recover the masked tokens on the basis of the CBoW bottleneck, and thus the LexMAE encoder assigns higher importance scores to essential vocabulary lexicons the masked document but lower to trivial ones. This closely aligns with the target of the lexicon- weiighine retrieval paradigm and boosts its performance.
module, and iii)
module leverages as the
After pre-training LexMAE on large-scale collections, we fine-tune its language modeling encoder to get a lexicon- weighting retriever, improving previous state-of-the-ee the performance by 1.5% MRR @ 10 with ~13 speed-up on the ad-hoc passage retrieval benchmark. Meantime, LexMAE also delivers new state-of-the-arr results (44.4% MRR @ 100) on the ad-hoc document retrieval benchmark. Lastly, LcxMAE shows great zero-shot transfer capability and achieves state-of-the-ar performance on BEIR benchmark with 12 datasets, e.g., Natural Questions, HotpotQA, and FEVER.
lexicon-weighting
2 RELATED WORK
PLMed Dense-vector Retrieval. Recently, pre-trained language models (PLM), c.g., BERT (Devlin et al., 2019), RoBERTa (Liu et af. 2019), DeBERTa (He al. (021b). have been proven generic and effective when transferred to a broad spectrum of downstream tasks via fine-tuning. When transferring PLMs to large-scale retrieval, a ubiquitous paradigm is known as 'dense-vector retrieval' (Xiong al. 2021) encoding both queries and documents in the same low-dimension semantic space and then calculating query-document relevance scores on the basis of spatial distance. However, dense-vector retrieval methods suffer from the objective gap between cexicon-recovering
We released codes and models http:://ggiihbb...m/ttt///////// LexMAN
2

---

Published as a conference paper at ICLR 2023
language model pre-training and document cocumeent-omprresiing dense-vector tinentuning. Although natural remedy has been dedicated to the gap by constructing pseudo query- (uery-document pairs (Lee et al.
2019
Chang et al.; 2020; Gao & Callan 2022] Zhou et al., (202aa) or/and enhancing bottleneck dense
representation (Lu et al. 2021; Gao & Callan 2021] 2022] Wang et al. 2022 Liu & Shao 2022).
the methods are still limited by their intrinsic representing manners dense vector leading to large
index size and high retrieval latency applying speed-up algorithms, product-quantizttin (Zhan
al. 2022), however resulting in dramatic drops (e.g.. 3% 4% by Xiao al. (2022)
Lexicon-weighing Retrieval. In contrast to the almost unlearnable BM25, lexicon-weighing retrieval methods, operating lexicon-weights by neural model, are proposed to exploit language models for term-based retrieval (Nogucira et al. 2019b a. Formal ct al. 202 a 2022). According to different types of language models, there are two lines work: based on causal language models (CLM) (Radford et al. Raffel et al. 2020), (Nogueira et al. 2019a) use the concurrence between a document and query for lexicon- based sparse representation expansion. Meantime, based on masked language models (MLM) (Devlin et al., [2019] Liu et al. 2019), (Formal et al., 2021b) couple the original word with top coordinate terms (full synonyms and concepts) from the pre-trained MLM head. However, these works directly fine-tune the pre-trained language models, regardless of the objective mismatch between general language modeling and relevance-oriente lexicon weighting.
LEXMAE:
MASKED AUTOENCODER
Overview of LexMAE Pre- training. As illustrated in Figure [ our lexicon- bottlenecked masked autoen- coder (LexMAE) contains one encoder and one decoder with masked inputs in line with the masked autoencoder (MAE) family (He et al.] 2021a Liu & Shaol 2022), while is equipped with a lexicon-bottlenecked mod- ule for socemint-specific lexicon-importance learning. Given a piece of free-form text, x, from a large-scale collection, D, we aim to pre-train a language modeling encoder, glene), that represents x with weighted lexicons in the vocabulary space, i.e., [0, 1] 1 V denotes the whole vocabulary. Here, each a, P(w wy ax; glsnet) with w; E V denotes the importance degree of the lexicon w, to the whole text æ. To Iearn the distribution a for x in an unsupervised fashion, an additional decoder gidec) is asked to reconstruct based on a.
Masked Language Modeling Objective x
Language Model Sub-module
[elm))
C(d0m)
Seq Len
Vocab Size
Brad-disallowwd
Pool
norm
LM Logits
Lextoan Imporisnce
Weakened Masking-style
Decoder ((dee)
(Language Model Head g(lm))
50% masked i [[SEP]
LM Encoder glone)
CBoW
dot
Bottleneck
(Wrdd Emb Param Mat W(
[CLS] 30% masked #[SEP]
further masking
Figure 1: An illustration lexicon-bottleneeee bottlenocked masked autocncoder (Lex- MAE) pre-training architecture.
3.1 LANGUAGE MODELING ENCODER
Identical to most previous language modeling encoders, e.g., BERT (Devlin et al. 2019). the language modeling encoder, g(enc) in LexMAE is composed of three parts, i.e., word embedding module mapping the discrete tokens of x to dense vectors, a multi-layer Transformer (Vaswani et al.) (017) for deep contextualization, and a language modeling head mapping back to vocabulary space RIV
First, following the common practice of pre-training the encoder unsupervisedly, a masked language modeling (MLM) objective is employed to pre-train glene), Formally, given a piece text a E D. certain percentage (a%) of the tokens in masked to obtain F. in which 80% replaced with special token [MASK]. 10% replaced with a random token in V, and the remaining kept unchanged (Devlin et al. 2019 Then, the masked is fed into the language modeling encoder, (ienc) i.e.,
glene) Transformer- -M
(M;;;; glenet) |M
where Stene) denotes LM logits. Lastly, the MLM objective is to minimize the following loss, I, fellm)
Σ
Σ log P(w" æz1 gleno), where P(w') = softmax (S(
(2)
D jEM
3

---

Published as a conference paper at ICLR 2023
where Mi(eo) denotes the set of masked indices of the tokens in ह, w denotes the discrete variable over V at the j-th position of z, and 2, is its original token (i.e., golden label of the MLM objective).
3.2 SAXICON-BOTTEENEER
MODULE
Given token-level logits from Eq. ! defined in V, we calculate lexicon-importance distribution by
== P(w i;; ((e)c) (ormaalzz(Max-PPol((S922
E 1)
(3)
where
is pooling along with its sequence axis, which is proven more effective than
meca--ooling in lexicon representation (Formal et al., 2021a), Normalize(.) is a normalization
function (let Σ a; 1), which we simply take softmax our main experiments. P(w] [ ((ene)
is lexicon-importance distribution over V to indicate which lexicons in V is relatively important to
The main obstacle to learning the lexicon-importance distribution (( ž;; (lene) that we do not have any general-purpose supervised signals. Inspired by recent bottleneck-enharee
dense representation learning (Gao & Callan 2022. Liu Shao 2022; Wang et al., 2022), we propose to leverage the texicon-importanee distribution as clue for reconstructing back. As such, our language modeling encoder will be prone to focus more on the pivot or essential tokens/words in x. However, it is intractable to directly regard the high-dimensional distribution vector a E [0, 1]1y as a bottleneck since i) the distribution over the whole V has enough capacity to hold most semantics of (Yang) jet al., 2018), making the bottleneck less effective, and ii) the high-dimensional vector is hardly fed into a decoder for representation learning and text reconstruction.
Therefore, we further propose to construct a continuous bag-of-worde. (CBoW) bottleneck following the eexicon-importance distribution P(w [2; (lenc) derived from Eq. 3 That is
b = P
(4)
Here, W( [e(w)), e(wx) ....] E dx denotes the learnable word embedding matrix in the parameters (Kene) of language modeling encoder, where d denotes embedding size, e IRd is word embedding of the lexicon wt- Thereby, E Rd stands for dense-vector CBoW bottleneck, upon which a decoder (will be detailed in the next sub-section) is asked to reconstruct the original x back.
As aforementioned in our Introduction, there exists a conflict between MLM and lexicon- importance objectives, but we still apply an MLM objective in our encoder. This is because the MLM objective can serve as a regularization term to ensure the original token in x receive relatively high scores in contrast its coordinate terms and ii) the token-level noise introduced by the MLM task has been proven effective in robust learning.
Remark.
3.3 WEAKENED MASKING-STYLE DECODER
Lastly, instruct the bottleneck representation b and consequently learn the lexicon-importance distribution P(w].ī; glene),, we leverage a decoder to reconstruct x upon b. In line with recent bottleneck-enhane neural structures (Gao & Callan 2022] Wang et al.] [002),, we employ a
weakened masking-style decoder parameterized by ( which pushes the decoder to rely heavily on the bottleneck representation. It is noteworthy that the 'weakened' is reflected by two-fold: i) aggressively masking strategy and ii) shallow Transformer layers (says two layers).
In particular, given the masked input at the encoder side, T. we first apply an extra 8% masking operation, resulting in ã. That is, the decoder is required to recover all the masked tokens that are also absent in the encoder, which prompts the encoder to compress rich contextual information prefix ă with the bottleneck representation b, i.e., replacing the special token [CLS] with the bottleneck. Therefore, our weakened masking waskinggstyle decoding with a Transformer-basee language modeling decoder can be formulated as
into the bottleneck. Then,
ddde)) Transformer LM ā;; gldee)) R 1
(5)
where (idee) parameterizes this weakened masking-styl decoder. Lastly, similar to the MLM at the encoder side, the loss function defined as
I (dlm))
Σ Σ
log P(w = 1z; ((dzo)), where P(wi) = softmax (S0))))),
(6)
JEMCAN
where (idce) denotes the set of masked indices of the tokens in the decoder's input, J.
4

---

Published as a conference paper at ICLR 2023
3.4 PRE-TRAINE OBJECTIVE & FINE-TUNING FOR LEXICON- TEXICON-WEIGHTING RETRIEVER
The final loss of pre-training LexMAE is an addition of the losses defined in Eq. 2 and Eq. (9. i.e.,
t = L (elm)
+ L (dlm)
(7)
Meanwhile, we tie all word embedding metrics in our LexMAE pre-training architecture, including the word embedding modules and language model heads of both the encoder & decoder, as well as W((e) in Eq. (4. is noteworthy that we cut-off the gradient pack-propagation for W((w) in Eq. to make the training focus only on the lexicon- catioon-impottncee distribution P(w) [w; glenc) rather than W/we).
Task Definition of Downstream Large-scale Retrieval. Given a collection containing a number
of documents, i.c., D = {d. ) and query g, retriever aims to fetch list of text pieces Dy to contain all relevant ones. Generally, this is based on relevance score between g and every document d; in a Siamese manner, i.e., < Eno(g). Enc(d,) >, where Enc is an arbitrary representation model (e.g., neural encoders) and < , > denotes lightweight relevance metric (e.g., dot-product).
To transfer LexMAE into large-scale retrieval, we get rid of its decoder but only fine-tune the language modeling encoder for the lexicon- weighttng retriever. Basically, to leverage a language modeling encoder for lexicon-weighting representations, adopt (Formal et al. 2021a) and represent piece of text, æ, in bigh-dimensional vocabulary space by
v (x) log(L 1 + Max-Pool(mmax Transformer- (M00oorrmme-MMMM;;600000 ((r 0))) E R
(8)
where max(., 0) ensures values greater than or equal to zero for upcoming sparse requirements, and the saturation function log(1 [ Max- )) prevents some terms from dominating.
In contrast to a classification task, the retrieval tasks are formulated as contrastive learning problems. That is, only a limited number of positive documents, df is provided for a query q so we need sample a set of negative documents, N(a) (d }. from ID for the q. And we will dive into various sampling strategies to get N(()) in & Note that if no confusion arises, we omit the superscript (q) that indicates 'a query-specific' for clear demonstrations. By following /Shen et al. (2022), we can first derive likelihood distribution over the positive {d+} and negative N documents, i.e.,
cxp(x
p P(dla, {d4) N; (lene))
Ed'edd, JUN exp(u uld't Vel (d_)UN
where (-) E I. derived from Eq.18) denotes exxicon-weightee document d. Then, the loss function of the contrastive learning towards this retrieval task is defined as
representation for a query q or a
Σ, log P(d = d4 14, {d+} JUN; ((e)c) ) + FLOPS( d)=) Σ
log Pla=d, FAFLOPS (10)
where FLOPS(-) denotes a regularization term for representation sparsity (Paria et al. 2020) as first introduced by Formal al. (2021b) and À denotes a hyperparameter of it loss weight. Note that, to train a competitive retriever, we adapt the fine-tuning pipeline in (Wang et al. (2022). which consists of three stages (please refer to $A $B for our training pipeline and inference details).
Top-K Sparsifying. Attributed to inherent flexibility, we can adjust the sparsity of the lexicon- weighting representations for the documents to achieve a targeted dfficacy-efficinne trade-off. Here, the 'sparsity' denotes how many lexicons in the vocabulary we used to represent each document. Previous methods either tune sparse regularization strength (Formal et al. 2021a 2022) (e.g.. À in (O) propose other sparse hyperparameters (Yang et al.] [02 Lassance & Clinchant 2022)
(c.g., the number of activated lexicons), however causing heavy fine-tuning overheads. Hence, we present simple but effective sparsifying method, which only presents during embedding documents the inference phase so requires almost zero extra overheads. It only keeps top-K weighted lexicons the representations ((d) Iv by Eq.(8), while removing the others by assigning zero weights (see &Dr for details). We will dive into empirical efficacy-efficiceee analyses later in $42
4 EXPERIMENT
Benchmark Datasets. Colloing/Forman et al. (202la), we first employ the widely-used passage
retrieval datasets, MS-Marco (Nguyen et al. 2016). We only leverage its official queries (no
5

---

Published as a conference paper at ICLR 2023
Table I: Passage retrieval results on MS-Mareo Dev, TREC Deep Learning 2019 (DL'19), and TREC Deep Learning 2020 (DL'20). M@ 10 and nDCG denotes MRR @ 10 and nDCG@ 10, respectively. The 'coCon' denotes the coCondenser that continually pre-trained BERT in an unsupervised manner, and the subscript of pre-trained model denotes its scale (e.g., 'base' equal to 110M parameters). PPlease refer to Tahle
Pre-trained Reranker Mult MS-Marco Dev 21 model distilled negs Vec
Method
M@10R@100 R@1 R@1kkDCGDDD
Dense-vector Retriever
ANCE (Xiong et (2021)
RoBERTa RoBERTa, DistiBBERT
33.8
86.2
96.0
65.4
64.6
ADORE (Zhan el al.
2021
34.7
87.6
68.3
TAS-B (Holstätter cf al. 2021
34.7
97.8
71.2
69.3
TCT-ColBERT (I al. 2021
BERT hase
35.9
97.0
71.9
coCondenser (Gao & Callan 2022
coConhase
38.2
98.4
71.7
68.4
ColBERTv2 (Santianam a.. 2021
BERT hase
39.7
98.4
RocketQAv2 (Ren el al. 20216)
ERNIE
38.8
-f
AR2 (Zhang ct al. 2002
coCoNbase
39.5
SimLM Wang et al.
2022
SimL
41.1
98.7
71.2
69.7
Lexicon-base or Sparse Retriever
BM25 (Dai & Callan 2019)
18.5
58.5
85.7
51.2
47.7
DeepC'T (Dai & Callan 2019
BERT buse
24.3
91.3
55.1
RepCONC (Zhan et al. 2022
RoBERTas
34.0
86.4
66.8
SPLADE-max (Formal et 2021a
DistiBBERT
34.0
96.5
68.4
DistilPPLADE-ma (Formal al.
2021ap
DistilBERT
36.8
97.9
72.9
SelfDistil (Formal et al. 2022)
DistilBERT
36.8
98.0
72.3
Co-SelfDisnl (Formal ct 2022)
coCoN
37.5
98.4
73.0
LexMAE
LexMABase
42.6
93.1
98.8
73.7
72.8
augmentations (Ren et al. 2021b)), and report for MS-Marco Dev set, TREC Deep Learning 2019 (Craswell al. 2020). and TREC Deep Learning 2020 (Craswell al. 2021). Besides, we evaluate the zeoo-shot transferability of our model on BEIR benchmark (Thakur et al. 2021).
We employ twelve datasets covering semantic relatedness and relevance-basee retrieval tasks (i.c.. TREC-COVID, NFCorpus, Natural Questions, HotpotQA, FiQA, ArguAna, Tóuche-2020, DBPedia, Scidoes, Fever, Climate-FEVV FEVER, and SciFact) in the BEIR benchmark as they are widely-used across most previous retrieval works. Lastly, to check if our LexMAE is also compatible with long-context retrieval, we conduct document retrieval evaluations on MS-Marco Doc Dev. Note that if not specified in our analyzing sections of the remainder, the numbers are reported on MS-Marco passage dev.
Evaluation Metrics. We report MRR 10 (M@ 10) and Recall 1/50/100/1K for MS-Marco Dev (passage), and report NDCG@ @ 10 for both TREC Deep Learning 2019 (passage) and TREC Deep Learning 2020 (passage). Moreover, NDCG@ is reported on BEIR benchmark, while MRR @ 100 and Recall @ 100 are reported for MS-Marco Doc. Regarding R @ metric, found there two kinds of calculating ways, and we strictly follow the official evaluation (please refer to C
Setups. We pre-train the MS-Marco collection (Nguyen et al. 2016). where hyperparame- ters are identical to (Wang et al. (022): the encoder is initialized by BERT base (Devlin et al. 2019) whereas the others are randomly initialized, the batch size is 2048, the max length 144, the learning rate is 3 x 10-4 the number of training steps is 80k, the masking percentage (a%) of encoder is 30%, and that (a ,%) of decoder 50%. Meantime, the random seed is always 42, and the pre-training is completed on A100 GPUs within 14h. Please refer to $A. for our fine-tuning setups.
4.1 MAIN EVALUATION
MS-Marco Dev (Passage Retrieval). First, we compare our fine-tuned LexMAE with a wide range of baselines and competitors to perform large-scale retrieval in Table I It is shown that our method substantially outperforms the previous best retriever, SimLM, by very large margin (+1.5% MRR @ 10) and achieves a new state-of-the-art performance. Standing with different retrieval paradigms and thus different bottleneck constructions, such large performance margin verifies the superiority of lexicon- retrieval when a proper initialization is given. Meantime, the LexMAE dramatically superior (+5.1% MRR @ 10) to its baseline (Formal et al. (022), Co-Self- Disil, with the same neural model scale but different model initialization (coCondenser (Gao &
Callan 2022) v.s. LexMAE). This verifies that our lexicon- bottlenecked pre- pre-training is more effective than the hotte-hottlenecked one in lexicon- retrieval.

---

Published as a conference paper at ICLR 2023
Table Zero-shot transfer performance (nDCG@ 10) on BEIR benchmark. 'BEST ON" and 'AVERAGE' do not take the in-domain result into account. 'ColBERT" is its v2 version (Santhanam et al. 2021).
Method
BM25
D5 SPLADE ColBERT DPR ANCE GenQ TAS-B Contriever UnifieR LexMAE
In-Domain
22.5
33.8
43.3
42.5
38.8
40.8
40.8
47.1
48.0
TREC-COVID
65.6
71.3
71.0
73.8
33.2
65.4
61.9
48.1
59.6
71.5
76.3
NFCorpus
32.5
32.8
33.4
33.8
18.9
23.7
31.9
31.9
32.8
32.9
34.7
NQ
32.9
52.1
56.2
47.4
44.6
35.8
46.3
49.8
51.4
56.2
HotpotQA
60.3
58.0
68.4
66.7
39.1
45.6
53.4
58.4
63.8
66.1
71.6
FiQA
23.6
29.1
33.6
35.6
11.2
29.5
30.8
30.0
32.9
31.1
35.2
ArguAna
31.5
34.9
47.9
46.3
17.5
41.5
49.3
42.9
44.6
39.0
50.0
Tóuche-2020
36.7
34.7
27.2
26.3
13.1
24.0
18.2
16.2
23.0
30.2
29.0
DBPedia
31.3
43.5
44.6
26.3
28.1
32.8
38.4
41.3
40.6
42.4
Scidoes
15.8
16.2
15.8
15.4
7.7
12.2
14.3
14.9
16.5
15.0
15.9
Fever
75.3
71.4
78.6
78.5
56.2
66.9
66.9
70.0
75.8
69.6
80.0
Climate-FEVER FEVER
21.3
20.1
23.5
17.6
14.8
19.8
17.5
22.8
23.7
17.5
21.9
SciFact
66.5
67.5
69.3
69.3
31.8
50.7
64.4
64.3
67.7
68.6
71.7
BEST ON AVERAGE
1
1
3
0
0
0
0
41.1
42.4
47.0
47.0
26.4
37.7
39.8
40.4
44.3
44.5
48.7
TREC Deep Learning 2019 & 2020. As shown in Table 1 we also evaluate our LexMAF on both the TREC Deep Learning 2019 (DL'19) and the TREC Deep Learning 2020 (DL'20)) It is observed that LexMAE consistently achieves new state-of-the-an performance on both datascts.
Mcantime, we evaluate the LexMAE on BEIR benchmark, which contains twelve datasets, where ArguAna, Scidocs, Fever, Climate-FEVER, and SeiFact are semantic relat- edness tasks while TREC-COVID, NFCorpus, NQ, HotpotQA, FiQA, Tóuche-2020, and DBPedia are relevance-based retrieval tasks. To apply LexMAE pre-training to this benchmark, we pre-train LexMAB on BEIR collections and then fine-tune the pre-trained encoder on the in-domain supervised data of the BER benchmark. Lastly, we evaluate the fine-tuned LexMAE on both the in-domain evaluation set and the twelve out-of-domain datasets, whose results are listed in Table 2 It is observed that our LexMAE achieves the best in-domain performance. When performing the zero-shot transfer on the twelve out-of-domain datasets, LexMAE achieves the best 7 out of 12 datasets, and delivers the best overall metrics, i.e., 'BEST ON' and 'AVERAGE', verifying LexMAE's generalization.
BEIR benchmark.
Table 3: Document Retrieval on Marco Doc Dev.
MS-Marco Doc. Lastly, we evaluate document pre-traii LexMAE on the document collection and follow the fine-tuning pipeline of (Ma et al., 2022) (w/o distillation), where our setting is firstP of 384 tokens.
retrieval MS-Marco doc in Table
Method
M@ 100 R 100
BERT
38.9
87.7
ICT (Lee al
2019
39.6
88.2
B-PROP (M et al
2021
39.5
88.3
SEED (L al 2021
39.6
90.2
COSTA (Ma 2022
42.2
91.9
4.2 EFFICIENCY ANALYSIS AND COMPARISON
LexMAE
44.4
92.5
Here, we show efficacy-efficinncy correlations after applying our top-K sparsifying (please see see (3.4). key metrics of retrieval systems is the efficiency in terms of retrieval latency qquery-per-secome QPS), index size for inverted indexing, as well as representation size per document (for non-inverted indexing). On the one hand, as shown in Figure 2 our LexMAE achieves the best efficacy-efficinncy trade-off among all dense-vector, quantized-denss, and sisicon-based methods. Compared to the previous state-of-the-ee art retriever. SimLM. we improve its retrieval effectiveness by 1.5% MRR @ 10 with 14.1 x acceleration. With top-L. sparsifying, LexMAE can achieve competitive performance with M with 100+ QPS. addition, LexMAE shows a much better trade-off than the recent best PQ-IVF dense-vector retriever, RepCONC. Surprisingly, when only 4 tokens were kept for each passage, the performance of LexMAE (24.0% MRR @ 10) is still better than BM25 retrieval (18.5%).
On the other hand, as listed in Table 4 also compare different retrieval paradigms in the aspect of their storage requirements. Note that each activated (non-zero) zero) term in lexicon-weighed sparse vector needs 3 bytes (2 bytes for indexing and byte for its quantized weight). Compared to dense- vector methods, texicon-based methods, including our LexMAE, inherently show fewer storage requirements in terms of both index size of the collection and representation Byte per document.
7

---

Published as a conference paper at ICLR 2023
Method
IdxSize ReprByte M@10
SimLM
28
bp-32
Prefocu So
ColBERTv2
150G
17,203
39.7
MRR@00 MS- Mi--Marco Dev
AR2
BT-SPLADE
LexMAE
AR2
27G
3,072
39.5
SPLADEV2
RepCONC
top-16
SimLM
27G
3,072
41.1
BM25
4.3G
Avg 210
18.5
ADORE-IVE
top-5
SPLADE-max
2.0G
Avg 290
34.0
APLADE-mask
5,4G
Avg 915
37.3
DocTSquery
LexMAE
3.7G
42.6
25
top-
top-256 sparsify
3.5G
768
42.6
+3.25 QPS
top-128 sparsify
2.4G
384
42.3
20-
top-64 sparsify
1.4G
192
41.8
BM25 *
1op-32 sparsity
0.9G
96
40.0
15
top
top-16 sparsify
0.5G
48
36.0
10
10 QPS CPU Machine (query-emeedded)
10t
top-8 sparsify
0.4G
24
30.6
top-4 sparsify
0.3G
12
24.0
Figure 2: Retrievers their MS-Marco dev MRR@ and QPS,
top-2 sparsify
0.2G
6
15.2
top-1 sparsify
0.2G
1.9
including dense-vector methods (i.c., SimLM, AR2), quantized- dense methods (i.c., RepCONC (Zhan et al. 2022), ADORE-IVF
Table 4: Index sizes (IdxSize) of models with retrieval performance (MRR@ 10) on
(Zhan et | () ), and lexicon-based methods (i.e. SPLADEv2
Formal 202 BT-SPLADE (Lassance Clinchant 2022). Dqcuryy (Nogueira al. 2019a). BM25, and ours).
MS-Marco Dev. 'ReprByte' denotes the stor- age requirement an embedded passage.
Table 5: Ensemble & hybrid retrievers.
Table 6: Performance on different stages (see A for their details) of the fine-tuning pipeline on MS-Marco Dev.
An ensemble of SPLADE models.
Method
M@10
R@
LexMAE-pipeline
43.1
28.8
BM25 Negatives Hard Negatives Reranker- -iittile-
Method
LexMAE-cnsenil
43.1
28.8
M@
R@ lk
M@
R@ lk
M@ 10
R@
Rni (Shen
2022
40.7
26.9
Ensemble of SPLADE
40.0
coCondenser
35.7
97.8
38.2
98.4
40.2
98.3
COIL-full (Gao et al
2021a
35.5
SimLM
38.0
98.3
41.1
98.7
CLEAR Gao al.
2021b)
33.8
LexMAE
39.3
98.3
40.8
98.5
42.6
98.8
Meantime, compared to BM25 building its index the word level, the learnable lexicon-weighting methods, based on the smaller vocabulary of sub-words, are more memory- friendl..
4.3 FURTIER ANALYSIS
Analysis of Dense-Lexicon Complement. As verified by Shen et al. (022)). the lexicon- weirghiigg retrieval is complementary to dense-vector retrieval and simple linear combination of them can achieve excellent performance. As shown in Table 5 we conduct an experiment to complement our LexMAE with our re-implemented state-of-the-arr dense-vector retrieval method, i.e., SimL.M (Wang et al. 2022). Specifically, propose to leverage two strategies: i) ensemble; combination is applied to the retrieval scores of both paradigms, resulting in significant overheads due to twice large-scale retrieval; and ii) pipeline: retrieval pipeline is leveraged to avoid twice retrieval: our lexicon- lexicon-weighting weighting retrieval is to retrieve top-K documents from the collection and then our dense- vector retrieval is merely applied to the constrained candidates for dense scores. It is shown that improve the previous state-of-the-an hybrid retrieval method by 2.4% MRR @ 10 on MS-Marco Dev.
Zero-shot Retrieval. To figure if our LexMAE pre-training can learn exxcon-iimortaan distribution, we conduct a zero- shot retrieval on MS-Marco. Specifically, instead of softmax normalization function in Eq. (3). we present saturation function- based LI norm (i.c., LI-Norm(log(1 + ReLU(-))) and keep the other parts unchanged. W/o fine-tuning, we apply the pre- trained LexMAE to MS-Me MS-Marco retrieval task, with the sparse representation of log(1 (1 + ReLU(-))). As in Figure 3 our lexicon- importance embedding by LexMAE (110M parameters) beats BM25 in terms of large-scale retrieval and is competitive with a very large model, SGPT-CE (Muennighofil 2022) with 6.1B parameters.
8
28
$
5
24
22
BM25
1B LoWMAE
Figure 3: Zero-shot retrieval results (nDCG@ 10) MS-Marco Dev.
8

---

Published as a conference paper at ICLR 2023
2.0
4.0
softmax nomm
wat-torm
3.5
0.35
der derse-di
3.0
bottleneck
Enc
2,5
Ming10
2.0
0.25
sahnorm
1.5
1,5
10000
20000 30000 40000 50000 Fire-luting
Steps
Figure 4: MLM pre-training losses (99% moving average has been applied) of LexMAE's encoder and decoder for various bottlenecks.
Figure 5: Fine-tuning MRR @ 10 curve MS-Marco Dev.
Multi-stage Retrieval Performance. As shown in Table we exhibit more details about the retrieval performance of different pre-training methods on multiple fine-tuning stages (see SA). It can be seen that our LexMAE consistently achieves the best or competitive results on all the stages.
Table 7: First-stage (w/ BM25 negatives) fine-
4.4 MODEL CHOICES & ABLATION STUDY
tuning results on MS-Marco Dev with different choices & ablations of LexMAE pre-training.
We conduct extensive experiments to check our model choices and their ablations from multiple aspects in Table 7 Note that, our Lex.ME uses softmax-nom norm CBoW bottleneck, shares LM logits of encoder and bottleneck, and employs the inclusive masking strategy with 30% for the encoder and 50% for the decoder.
Method LexMAE
M@ 10
39.3
Bottleneck Choices
Saturated Norm in CBoW (Eq.3)
39.2
Dense CCLS]
(Eq.
38.6
First, we try three other bottlenecks in Eq.(5), i.e., saturated norm for CBoW (as detailed in 'Zero-shot Retrieval'), dense bottleneck by using [CLS], and no
Disable Bottleneck (Eq
38.5
Architecture Choices
Enable Grad to W in Bottleneck
38.9
bottleneck by cutting the bottleneck off. As in Figure their training loss curves show our CBoW bottlenecks
-haree Encoder-Decoder LM Heads
39.1
Add extra LM for Bottleneck
39.2
-DittlBERT- DistilBERTTmitttliiee Initialized
38.4
do help the decoding compared are inferior to [CLS] contextual dense vector. But, attributed pretraining-fintunii
no--bottleneck'. but
Masking Strategy (T in 5g.
consistency, CBoW bottlenecks are better in lexicon-weightinn retrieval.
-Exclusive
39.1
Fuully Random
39,2
As for the two different lexicon CBoW bottlenecks,
Masking Proportion (3. & $3.3)
we show their fine-tuning dev curves in Figure 5
EEc (15%) v.s. Dec (15%)
38.4
Enc (15%) Dee (30%)
38.8
'sat-norm' shows its great performance in early fine-
Enc (30%) v.s. Dee (30%)
39.0
tuning stages due to the same whereas max-oom" show better later fine-tuning
way
Enc (40%) v.s. Dec (60%)
39.0
Enc (30%) v.s. Dec (100%)
39.0
results due its generalization.
Then, we make some subtle architecture changes to LexMAE: i) enabling gradient back- bock-propagation the word embedding matrix leads to learning short-cut thus worse fine-tuning results; ii) both sharing the LM heads of our encoder and decoder (Eq.(I and Eq, (5) and adding an extra LM head specially for bottleneck LM logits (Eq.3) result in a minor drop, and iii) replacing BERT with DistilBERT (Sanh et al., 2019) for our iniialization still outperforms bunch of competitors.
Lastly, masking strategies other than our used 'inclusive' strategy in $3.3 do have minor effects on downstream fine-tuning. And, masking proportion of the encoder and decoder can affect the LexMAE's capability, and the negative 'affect' becomes unnoticeable when their proportion is large. In summary, pre-training LexMAE is very stable against various changes and consistently delivers great results. And please refer to $ for more experimental comparisons.
CONCLUSION
In this work, we propose to improve the eexicon-weighin retrieval by pre-training a lexicon- bottlenecked masked autoencoder (LexMAE) which alleviates the objective mismatch between the masked language modeling encoders and relevance-oriented lexicon importance. After pre- training LexMAE on large-scale collections, we first observe great zero-shot performance. Then after fine-tuning the LexMAE on the large-scale retrieval benchmark, we obtain state-of of-the-an retrieval quality with very high efficiency and also deliver ffate-of-the-as zero-shot transfer performance BEIR benchmark. Further detailed analyses on the ffficcc--fffcience trade-off in terms of retrieval latency and storage memory also verify the superiority of our fine-tuned LexMAE.

---

Published as a conference paper at ICLR 2023
REFERENCES
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.). Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL httpp:::/proceeeeteeett 1457c0d cb4bbcbb4777-4 2tw.d6/..0009911110011444e444422 www.eeaccc42ff4420
neurips. .1...c//pp hcc/paper/2220/hahh .html
Yinqiong Cai, Yixing Fan, Jiafeng Guo, Fei Sun, Ruqing Zhang, and Xueqi Cheng. Semantic models for the first-stage retrieval: A comprehensive review. CoRR, 00002110..481,,220 URL
https://arxiis
Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. Pre-training based large-scale retrieval. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
tasks for embedding basee
URL
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees, Overview of the TREC 2019 deep learning track. CoRR, abs/2003.07820, 2020. httthttss///r/
org/ aat/abb/200000 07820
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. Overview of the TREC 2020 deep learning track. CORR, abs/2102.07662,, 2021. URL
/210
07662
Zhuyun Dai and Jamie Callan. Context-aware sentence/passage term importance estimation for first stage retrieval. CoRR, abs/ abs/1910.10087, 2019. URL
/1900
10687
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: 10.186533/11/11--14222 URL
11653/v1/n1ll 1423
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. SPLADE v2:
Sparse lexical and expansion model for information retrieval. CoRR, abs/2109.10086, 2021a. URL
kttps://Ae wttp///aavvvvv/. www.orr.abb.... abs /2e///2133 2109.1100. 10086
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. SPLADE: sparse lexical and expansion model for first stage ranking. In Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai (eds.), SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pp. 2288 2292. ACM, 2021b. doi: 10. 10.1145/3004335.4663999 URL https //doi aeroi.org//0 111.1...11151111111113 3463098
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. From distillation to hard negative sampling: Making sparse neural IR models more effective. CoRR, 22w/2225504433. 2022. doi: 10.485550/rxxw.2225.00m 004733. 2205.00 04733
URL https:///oi. wrw.:io.orgg11.. wtt.///.......050///3......
Luyu Gao and Jamie Callan. Condenser: pre-training architecture for dense retrieval. In Marie- Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 I Conference on Empirical Methods in Natural Language Processing, EMNLP 2021,
10

---

Published as a conference paper at ICLR 2023
Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 981-993. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021 10.18653w1/202.emmmlp wwwp:////...eemmmp-pmme main.75. URL https
Luyu Gao and Jamie Callan. Unsupervised corpus aware language model pre-training for dense passage retrieval. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Pro- ceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, Pp. sociation for Computational Linguistics, 2022. doi: 10. 1/0522//112022.lllloooo https://doo wrw.oorgo.0. / vl www.............. wong-llnooo....oo...oooooo 203
As-
URL
Luyu Gao, Zhuyun Dai, and Jamie Callan. COIL: revisit exact lexical match in information retrieval with contextualized inverted list. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tüc Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-HLT 2021, Online, June 6- I, 2021, pp. 3030-3042. Association for Computational Linguistics, 2021a, doi: 10.18653/v1/2022
maacl-main.241 URL LRL
wtt../.../11..1////..0.. www.11166111//00 naacl main. 241
Luyu Gao, Zhuyun Dai, Tongfei Chen, Zhen Fan, Benjamin Van Durme, and Jamie Callan. Com- plement lexical retrieval model with semantic residual embeddings. In Djoerd Hiemstra, Marie- Francine Moens, Josiane Mothe, Raffaele Perego, Martin Potthast, and Fabrizio Sebastiani (eds.), Advances in Information Retrieval 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28- April 1, 2021, Proceedings, Part I, volume 12656 of Lecture Notes in Computer Science, pp. 146 160. Springer, 2021b. doi: 03000- 72113-8 10
URL
httpps://dii org/ wwt.://...11001110000
1007
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. REALM: retrieval- augmented language model pre-training. CoRR, abs/2002.08909, 2020. URL httth:://axxxiitt
bzq/abs 2002. 08909
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked 2021a. URL https://
autoencoders are scalable vision learners. CoRR,
22677763777
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: decoding-cnhanced bert with disentangled attention. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021b. | //openreview. net/ fese/forree htt//oormmmmm/mm X IaotutsD
Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. Improving efficient neural ranking models with cross-architecturee knowledge distillation. CoRR, abs/2010.02666 2020. URL
02666
Sebastian Hofstätter, sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. Efficiently teaching an effective dense retriever with balanced topic aware sampling. In Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai (eds.), SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pp. 113 122. ACM, 2021. doi: 10.1145/ 2404835.3462891.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. Poly-encoders: Architec- tures and pre-training strategies for fast and accurate multi-sentence scoring. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URU/httpa://ppeeeeiice
wtt/forumt Sts.rrgnn
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 7769-6781. Association for Computational Linguistics, 2020. doi: 10. 1t..8865/v1/222 emnlp-main.550 URL httph:tts: http://///6..orr0...118553333//3
emn p- ma i 550

---

Published as a conference paper at ICLR 2023
Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over BERT. In Jimmy Huang. Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock. Ji-Rong Wen, and Yiqun Liu (eds.). Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pp. 39 48. ACM, 2020. doi: 10.1145/3397271.4400 //doi.org/10 1115.....139391111111 3401075
| https
Carlos Lassance and Stéphane Clinchant. An efficiency study for SPLADE models. In Enrique Amigó, Pablo Castells, Julio Gonzalo, Ben Carterette, J. Shane Culpepper, and Gabriella Kazai (eds.), SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 15, 2022, pp. 2220 2226. ACM, 2022. doi: 3531833
10.1 14/4444749953531118
URL https::///// http:/oi.orrr////oo/oo /000 11000.....4 15/
Kenton Lee, Ming-Wai Weig Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Anna Korhonen, David R. Traum, and Lluís Mårquez (cds.). Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2. 2019, Volume 1: Long Papers, pp. 6086-6096. Association for Computational Linguistics, 2019. doi: 10.1 10.18653/v1/p19-100 1612. URL https? 7/doi. aodoi.or///0 10. 11w53/v1/p19-11011
Patrick S. H. Lewis. Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttlee, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmenntd generation for knowledge-intensive NLP tasks. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL httl:/ttss::pprcceeeeet/t ngs. neurips wwcpppppe///a/// heterr/220/haase Absttaattttttaaaaa html
Sheng-Chich Lin, Jheng-Hong Yang. and Jimmy Lin. In-batch negatives for knowledge dis- tillation with tightly-coupled teachers for dense retrieval. In Proceedings of the 6th Work- shop on Representation Learning for NLP (RepLANLP-202) pp. 163-173, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/w1/20211 repl4nlp- 1.17. URL rrgp
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roherta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019.
http///aaivv../////////////// 11692
Zheng Liu and Yingxia Shao. Retromae: Pre-training retrieval-orienteed transformers via masked auto-encoder. CoRR, 22003 2022. doi: 10.48550/arXi..2255. URL https
//doi. .tt.looog/1...55/aaaxi.....
2205.12033
Shuqi Lu, Chenyan Xiong, Di He, Guolin Ke, Waleed Malik, Zhicheng Dou, Paul Bennett, Tie- Yan Liu, and Arnold Overwijk. Less is more: Pre-training a strong siamese encoder using a weak decoder. CoRR, abs/2102.00206, 2021. httppstss://axxii wwwpiv..rrgabb///10100 09206
Xinyu Ma. Jiafeng Guo, Ruqing Zhang, Yixing Fan, Yingyan Li, and Xueqi Cheng. B-PROP: bootstrapped pre-training with representative words prediction for ad- hoc retrieval. In Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai (eds.), SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pp. 13218-1127..00 ACM, 2021. doi: 10.1145; /
Xinyu Ma, Jiafeng Guo, Ruqing Zhang, Yixing Fan, and Xueqi Cheng. Pre-train a discriminative text encoder for dense retrieval via contrastive span prediction. In Enrique Amigó, Pablo Castells, Julio Gonzalo, Ben Carterette, J. Shane Culpepper, and Gabriella Kazai (eds.), SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11- 15, 2022, pp. 848 858, ACM, 2022. doi: 10. 00.145/54774495.5537772 353177
URL l http:::tp//......... / wtw.ddi...r..oooo
12

---

Published as a conference paper at ICLR 2023
Niklas Muennighoff. SGPT: GPT sentence embeddings for semantic search. CoRR, abs/2202.08904,
2022. URL htllh:tts:/aaaa /ttttps://aaxii...mm wttpv//...r//bb///220000000
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A human generated machine reading comprehension dataset. In Tarek Richard Besold, Antoine Bordes, Artur S. d' Avila Garcez, and Greg Wayne (eds.), Proceed- ings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, December 9, 2016, volume 1773 of CEUR Workshop Proceed- ings. CEUR-WS.org 2016. URL http://tp///c.... ceur-ws /Vol 1 W7T3///////SS/////1 paper9. pdf
Rodrigo Nogueira, Jimmy Lin, and Al Epistemic. From doc2query to doctttitquery. Online preprint, 6, 2019a.
Rodrigo Nogueira, Wei Yang. Jimmy Lin, and Kyunghyun Cho. Document expansion by query prediction. CoRR, abs/1904 2019b. URL hwt.:htp///m http::://axxxv...r//bbbbs.. wttp://///////...00....
Biswajit Paria, Chih-Kuan Yeh, Ian Hs- Yen, Ning Xu, Pradeep Ravikumar, and Barnabás Póczos. Minimizing flops to learn efficient sparse representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Abuba, Ethiopia, April 26-30, 2020, OpenReview.net, 2020. URL URRppttts///ppeeeeee ew net/ forum? iaborrumiiiSyyee SygpC 6Ntvr
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. Rocketqa: An optimized training approach dense passage retrieval for open- domain question answering. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 202/ Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pp. 5835-5847, Association for Computational Linguistics, 2021. doi: 10. 10.18613/11/2021. www.nnaalllaaaaa 466
maacl-main..ee URL https: /ttp:://ddoi.coo http///......oo../oo////. 1tte//10..6553/11///1
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. J. Mach. Learn. Res., 21:140: 1-140:67, 2020. URL http: /m htt.m/mlrrror.. org
papers/v21/20-070 html
Nils Reimers and Iryna Gurevych. Sentence-bert, Sentence embeddings using siamese bert-networke In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNIP--JCNIIP 2019, Hong Kong, China, November 3-7, 2019, pp. 3980-3990. Association for Computational Linguistics, 2019. doi: 1410
10.18653/ URL
Ruiyang Ren, Shangwen Lv, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. PAIR: leveraging passage-centric similarity relation for improving dense passage retrieval. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACI/IJCNPP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pp. 2173-2183, Association for Computational Linguistics, 2021a. doi: 10.18653/ 1/2021 findings-acl. 1wt../..153331/11 / www.iiiiiiiingggg.
URL https
191
Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaogiao She, Hua Wu, Haifeng Wang. and Ji-Rong Wen. Rocketqav2: A joint training method for dense passage retrieval and passage re-ranking. In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings the 202/ Conference on Empirical Methods in Natural Language Processing. EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic. 7-11 November, 2021, pp. 2825- 2835. Association for Computational Linguistics, 2021b. doi: 10.18653/v1/2022 http//222111mmm2p. 1op-mmain222 224. .mm p-ma nn2241
URL httpt:::////.. htops:://oo.. http:://ooooooog1..111..0 wwt.ooo.......//11111.11
13

---

Published as a conference paper at ICLR 2023
Stephen E. Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Rem 3(4):333-38) 2009. doi: 10.156//1500000009. URL https
7/ ///oi.oorg/..1111 0tte://0.1561//500000000
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.011088 108, 2019. URL http:://axx:
org/abs/191n 01108
Keshav Santhanam, Omar Khattab, Jon
Christopher Potts, and Matei Zaharia. Colbertv2: Effective and efficient retrieval via lightweight late interaction. CoRR, abs/2 12..0118,,,,88, 2021. URL
httpp::/aat //arivv.aiivv.v/aa org/ abs 211..111000 11888
Tao Shen, Xiubo Geng, Chongyang Tao, Can Xu, Kai Zhang. and Daxin Jiang. Unifier: A unified retriever for large-scale retrieval. CoRR, ahs/2205. 1194, 2022. doi: (0.48550/arXiv22205.. 1194.
URL httl:ttts::// //toi://ooioo/ /ttttps://doi.rr/// 0ttp://11...55////// art.5///axxv.. 111941144
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogenous benchmark for zero-shot evaluation of information retrieval models. CoRR, abs/2104.08663, 2021. htRl:ttts:://at
abs 2004..10000000 08663
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neurul Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. neur neuri ps cc/ccc//ppe. html
1188-6008,000 2017. URL
Liang Wang, Nan Yang. Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Simlm: Pre-training with representation bottleneck for dense passage retrieval. CoRR, abs/2207.02578 2022. doi: 10.48550/aXx..2207.02778 URL
48550/arXiv 2207202777222220
Shitao Xiao, Zheng Liu, Weihao Han, Jianjin Zhang, Defu Lian, Yeyun Gong, Qi Chen, Fan Yang, Hao Sun, Yingxia Shao, and Xing Xie. Distill-vq: Learning retrieval oriented vector quantization by distilling knowledge from dense embeddings. In Enrique Amigó, Pablo Castells, Julio Gonzalo, Ben Carterette, J. Shane Culpepper, and Gabriella Kazai (eds.), SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain. July 11 15, 2022 pp. 1513-152231 ACM, 2022, doi: 10.1145334774955.531799 URL hhttp:://do... htt.:/oo..rgr/o//o 0. 11 00444544477994 3531799
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang. Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL Id=zeFr
net forum
Jheng-Hong Yang, Xueguang Ma, and Jimmy Lin. Sparsifying sparse representations for passage retrieval by top-k masking. CoRR, abs/2112.09628,, 2021. URL https:t/aax.....////////////////
2112.09628
Peilin Yang, Hui Fang, and Jimmy Lin. Anserini: Enabling the use of lucene for information retrieval research. In Noriko Kando, Tetsuya Sakai, Hideo Joho, Hang Li, Arjen P. de Vries, and Ryen W. White (eds.), Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017, pp. 1253 1256. ACM, 2017. doi: 10.1145/3077133633007777 URL 1145/3077 00 3080721
Zhilin Yang. Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. Breaking the softmax bottleneck: A high-rank RNN language model. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL htw.::ttts//ott https:://operrevie. net forum?id
Hkw7SG-C22
14

---

Published as a conference paper at ICLR 2023
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. Optimizing dense retrieval model training with hard negatives. In Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai (eds.), SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, doi: 10.1145//4404355344666
Canada, July 11-15, 2021, pp.
ACM, 2021.
URL
/tt www.:/o....g./......o Ttt...111//33444...0 3462880
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. Learning discrete representations via constrained clustering for effective and efficient dense retrieval. In K. Selcuk Candan, Huan Liu, Leman Akoglu, Xin Luna Dong, and Jiliang Tang (eds.), WSDM '22: The Fifteenth ACM International Conference on Web Search and Data Mining, Virtual Event / Tempe, AZ, USA, February 21 25, 2022, pp. 1328- 1336. ACM, 2022. doi: 10.1145/38885003449888 URL https: /oi org 10. 1145/ 3488560 3498443
Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen. Adversarial retriever-ranker for dense text retrieval. In International Conference on Learning Representations, 2022. URL/https:
htttts::/opennevieew..eee wwwtene/fooummmdd---mm 7XUKKUF
Xueliang Zhao, Wei Wu, Can Xu, Chongyang Tao, Dongyan Zhao, and Rui Yan. Knowledge- grounded dialogue generation with pre-trained language models. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 3377-3390. Association for Computational Linguistics, 2020. doi: 10. 10.18653/ www.ppp...mmmmmmmmmmmmmmm main. 272.. URL www.emm.......... ma..pppmain.....
https://dois org/10 1ee.c.1863333/
Jiawei Zhou, Xiaoguang Li, Lifeng Shang, Lan Luo, Ke Zhan, Enrui Hu, Xinyu Zhang, Hao Jiang, Zhao Cao, Fan Yu, Xin Jiang, Qun Liu, and Lei Chen. Hyperlink-induced induced pre-training for passage retrieval in open-domain question answering. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 7135-7146. Association for Computational Linguistics, 2022a. doi: 10. 111.///////22..aallllaaaa 493. URL .rg. http:///100.13//1/2/2232222 acl ong 493
Yucheng Zhou, Tao Shen, Xiubo Geng, Chongyang Tao, Can Xu, Guodong Long, Binxing Jiao, and Daxin Jiang. Towards robust ranker for text retrieval. CoRR, abs/2206.08063,, 2022b. doi: 10.44555//rXiv.2206.00 080663 URL (tRt:ttss: http:///ooo.o......3..... /1tp:/1...445000 48.500.88555////0 2206. 08063
A MULTI-STAGE RETRIEVER FINE-TUNING
A.1 FINE-TUNING PIPELINE
To train a state-of offte-of-thh-art exxicon-weighting retriever, adapt the fine-tuning pipeline in recent dense-vector retrieval method (Wang et al. 2022) as illustrated in Figure The major difference is the source of the involved reranker for knowledge distillation into retriever: In contrast to (Wang et al., 2022) that trains a reranker on the fly for retriever-specific reranker but suffers from high computation overheads, we propose leverage an off-the-shelf reranker by (Zhou al. 2022b)
Stage 1: BM25 Negatives. In the first stage, sample negatives for each query within top- Kj document candidates by BM25 retrieval system, which is denoted as Nibm25), Therefore, the contrastive learning loss of stage I of our retriever fine-tuning is written as
(d)) = Σ log P(d = d+ lg, {d+} 12bm255) glenet) + +A1 FLOPS.
(11)
Stage 2: Hard Negatives. Then, we sample the hard negatives N(ml for each query q within Kop-Kz candidates based on the relevance scores by the retriever we obtain in stage 1, and the training loss of our stage 2 is defined as
I (r2) = Σ, log P(d= d41g. {d4 Numl), glene)) + X₂ FLOPS.
(12)
15

---

Published as a conference paper at ICLR 2023
Pre-trained LM Encoder ((enc)
Fine-tuned Reranker ((rk)
Model Inibalization
Ki-Distill
Contrast
Contrast
Contrast
BM25 Negalives'
Retriever Stage
Stage1 Hard Negalives
Retriever Stage 2
Stage2 Hard Negalives
Retriever
Reranker-distilldd
Figure 6: An illustration of the fine-tuning pipeline of our retrievers. Here, the fine-tuned reranker is directly
adopted from (Zhou al. 2022b) avoid expensive reranker training.
Stage Reranker-Distileed Lastly, we further sample hard negatives N(hn2) for each query g within top- K3 candidates by the 2nd-stage retriever. Besides the contrastive learning objective, we also distill
reranker into our stage of the retriever, which written as
I ( Σ, KT-Div(P(dlg, {d+} } N(m2), glene)) || (d+} U N g(1)))
(13)
y log P(d d+14: {d+} U N(mm2) (lmn)) + A FLOPS.
Here, ((rh) parameterizes an expensive but effective cross-encoder as the reranker for knowledge distillation, and KL divergence, KL-Div(-II-), is used distillation loss with (likk frozen.
A.2 FINE-TUNING SETUPS
We share some hyperparameters across all three stages: learning rate is set to 2 x 10-5 by following Shen et al.) (2022), the number of training epochs set 3, model initialization is always our LexMAE, max document length is set to 144, and max query length is set to 32. Following (Wang) ct al. 2022). the y in Eq, (3) is set to 0.2. contrast to (Wang et al., 2022) using 4 GPUs for fine-tuning, we limited all the fine-tuning experiments on one A000 GPU. The batch size (w.r.t the number of queries) is set to 24 with 1 positive and 15 negative documents in the fine-tuning stage 1 and 2. whereas that is set to 16 with positive and 23 negative documents (to increase the number of negatives but fit one GPU memory by reducing the batch size). Another important hyperparameter is the depth (keeping how many top candidates negatives) of negative sampling. i.e., K in Eq. [1]-- 13 By following Wang et al. (2022) and Gao & Callan (022). we keep 1000 for BM25 negatives and 200 for other negatives, i.c K 1 1000, K₂ 200, K3 200. The only hyperparameter we
have tuned is the loss weight À in Eq. 11 13 i.e., A1 E (0.001, 0.002, 0.004) (corresponding to BM25 negatives) and X2/a E f0.004, 0.008, 0.016) (corresponding to hard negatives). Empirically, we found A = 0.002, X2 0.008, A = 0.008 achieving great arrffommanceefffccinn trade-off. Again, we fix the random seed always to 42 all our experiments without any tuning.
B LEXICON-WEIGHTING INFERENCE FOR LARGE- RETRIEVAL
In the inference phase of large-scale retrieval, there some differences between dense-vector and lexicon- lexicon-weighting retrieval methods.
As in IO), we use the dot-product between the real-valved sparse lexieon-weighting representa- tions as a relevance metric, where 'real-valved' is a prerequisite of gradient back-propagation and end-to-end learning. However, it is inefficient and infeasible leverage the real-valved sparse repre- sentations, especially for the open-source term- based retrieval systems, e.g., LUCENE and Anserini (Yang et al. 2017). Dollowing/Formall et al. (2021a), we adopt "quantization' and 'term-based system' to complete our retrieval procedure. That is, to transfer the high-dimensional sparse vectors back to the corresponding lexicons and their virtual frequencies, the lexicons are first obtained by keeping the non-zero elements in high-dim sparse vector, and each virtual frequency then is derived from straightforward quantization (i.e., [100 x v)).
In summary, the overall procedure of our large-scale retrieval based on fine-tuned LexMAE is i) generating the high-dim sparse vector for each document and transferring it to lexicons and frequencies. ii) building term-based inverted index via Anserini (Yang et al. (017)) for all documents in collection, iii) given a test query, generating the lexicons and frequencies, in the same way, and iv) querying the built index get top document candidates.
16

---

Published as a conference paper at ICLR 2023
C EXPLANATION OF DIFFERENT RECALL METRICS
Regarding @@N metric, we found there are two kinds of calculating ways, and we strictly follow the official evaluation one at htttps:/ggtttub.....bbb www/ummmmccssoc... wwwesnissttttttteeeeeeeee and https
//github. ctcpluu.coom/ooe wowm/cattorini/eeee..eo anserini. which is defined as
1
Ed,eD, La min(N, [B+])
Marco-RecalI@N
Σ OEQ
(14)
[Q
where there may be multiple positive documents 1B+ E D. Q denotes the test queries and D denotes top-K document candidates by a retrieval system. We also call this metric all-positive-maat Recall @N. On the other hand, another recall calculation method following DPR (Karpukhin et al.) 2020 is defined as
DPR-Recall@N
1 DdeDAdED
(15)
q
which we call one one-positive-enouee Recall@N. Therefore, The official (all-posittin) collposittve-macro) Recall@N usually less than DPR wete--poiiivveeeeeeee Recall@N, N. and the smaller N, the more obvious.
Therefore, we make the unofficial uueepsiitiveeenuugh R standalone in Table & for more precise comparisons. It is observed that LexMAE is still the best retriever among its competitors.
D SPARSIFYING LEXICON REPRESENTATIONS
Compared dense-vector retrieval methods (Zhan al. 2022 Xiao al. 2022 that rely on product-quantization (PQ) and inverted file (IVF) to compromise their performance (-3% 3% ~ 4%) for memory & time efficiency, the lexicon-weighting method with high-dimension sparse representation is intrinsically efficient for large-scale retrieval as demonstrated in $B fully compatible with traditional term-based retrieval system, e.g., BM25 only manipulating the term frequency and document frequency by the neural language modeling encoder.
To dive into LexMAE's efficacy-efficiency trade-off, we need adjust the sparsity of lexicon representations for the documents. In general, the 'sparsity' here denotes how many lexicons in the vocabulary we used to represent each document. Since the hyperparameter in Eq. denotes the strength of sparse regularization during fine-tuning and controls the efficacy-efficinncy trade-off for the retriever, it is straightforward tune the for the purpose of sparsifying. However, this requires fine-tuning the retriever multiple times with different for adequate data points, leading to huge computation overheads. What's worse, there is no certain correlation between the and the sparsity, leading uncontrollable sparsifying and increasing the number of trials. To make the sparsifying procedure more controllable, Yang et al.) (2021) and Lassance & Clinchant (2022) propose to sparsify the eexicon-weighing representations by controlling fine-tuning hyperparameters, e.g., constraining the number of activated lexicons, however still leading to extra fine-tuning efforts.
Therefore, in this work present simple but effective and controllable sparsifying method, which only presents during embedding documents in the inference phase and requires almost zero extra overheads. Specifically, it only keeps top-K weighted lexicons in the sparse lexicon representations ((()) E IVi by (). while removing the others by assigning zero weights, which can be formally written as
0 = (d)) C 1,
(16)
where denotes element-wise product, denotes the K-th large value in ((d) and {0, 1 is mask where an entry equals only if the corresponding value in top-K of (dd) If K is smaller than the number of activated (i.c., the weight (!()) > 0) lexicons in (dd), applying this sparsifying method would not make any change. All the sparsified lexicon representations 0 with different K values are derived from the same original representation ed()), so both the fine-tuning and embedding procedures are invoked only once, saving a lot of computing resources. Lastly, the sparsified lexicon (d) representations, 0k used to build the inverted index for large-scale retrieval.
17

---

Published as a conference paper at ICLR 2023
Table 8: Retrieval results on MS-Marce on one- positive-enough recall. Note that the official @5call@50 of our mune-tned LexMAE is 88.9%.
Table 9: Comparisons of retriever with re- trieval&rerank pipelines.
Method
M@
R@ IK
Retriever
Reranker
M@ 10
RocketQA (Qu ot al
2021
37.0
85.5
97.9
RepBERT
RepBERT
37.7
PAIR (Ren et al. 202 a
37.9
86.4
98.2
ME-HYBRID ME- HYBRID
39.4
RockeiQAv2 Ren al.
2021b
38.8
86.2
98.1
RocketQA
RocketQA
40.9
AR2 (Zhang al. 2022
39.5
87.8
98.6
RocketQAv2
RocketQAv2
41.9
UnitieKiesieon Shen et al
2022
39.7
87.6
98.2
LexMAE (retriever-only)
42.6
UnifieRaense Shen et al.
2022)
38,8
86.3
97.8
LexMAE
42.6
89.6
99.0
Table 10: Different pre-training objectives with its first-stage fine-tuning MRR @ 10 performance on MS-Marco.
LexMAE
SimLM
Enc- Dne MLM Condenser
MLM Enc Dec RTD
AutoEncoder
BERT
39.3
38.0
37.7
36,9
36.7
36.2
32.8
33.7
E
MORE EXPERIMENTS
Comparison Retrieval & Rerank Pipeline. Furthermore, our LexMAE retriever can out- perform many state-of-the-ar retrieval & rerank pipeline methods as in Table It is noteworthy that these rerankers (a.k.a., cross-attention model cross-encoder) are very costly as they must be applied to every query-document text concatenation, instead of query guery-agnostic representations from bi-encoder.
Comparisons with Different Pre-training Objectives. As listed in Table I0 we compare our pre- training objective with batch of other objectives by fine-tuning the pre-trained model on MS-Marco BM25 negatives. can be seen that our LexMAE improves the previous best by 1.3% MRR 10 in MS-Marco Dev.
F BACKGROUND: DENSE-VECTOR AND LEXICON-WEIGHTIE RETRIEVAL
With recent surging pre-trained language models (PLMs) by self-supervised learning, e.g.. GPT (Brown et al. (020), BERT (Devlin et al. 2019), RoBERTa (Liu et al. (019), and T5 (Raftel era to offer more expressively powerful
al. 2020), deep representation learning has entered text representations. As a practical task that relies heavily on text representation learning, large-scall retrieval directly benefits from these PLMs by leveraging the PLMS neural encoders and fine-tuning them on the downstream datasets. Thereby, recent works upon the PL.Ms propose to learn encoders for large-scale retrieval, which are coarsely grouped into two paradigms according to their encoding spaces, i.c., dense-vector and sekicon-weighting retrieval.
Dense-vector Encoding Methods. The most straightforward way to leverage the Transformer- based PLMs is directly representing document/query as a fixed-length real-valued low-dimension dense vector u E IR_ By following the previous common practice, the dense vector is derived from either contextual embedding of the special token "CCSS] word-level contextual embeddings. It is noteworthy that is embedding size and usually small (c.g.. 768 for base-size PLMs), and the 'fieed-lengt' is not limited to one vector for each collection entry. but maybe multi-vector (Humeau et al., 2020 Khattab & Zaharia 2020). Lastly, the relevance score between document and a query is calculated by a very lightweight metric, e.g., dot-product or cosine similarity (Khattab & Zaharial 2020 Xiong al. 2021 Zhan et al.) 2021: Gao & Callan) 2022). Although PL.M-based dense-vectt vector retrieval methods enjoy their off-the the shelf dense embeddings and easy-to-can to-calculate relevance metrics, the methods limited by their intrinsic representing manners i) real-valued vectors leading to large index size and high retrieval latency and ii) high-level vector representations losing the key relevance feature about lexicon overlap.
mean pooling over the sequence of
18

---

Published as a conference paper at ICLR 2023
Lexicon-weighing Encoding Methods. To make the best of word-level contextualizzation by considering either high concurrence (Nogueira et al. 2019a) or coordinate terms (Formal et al. 2021b) in PLMs, lexicon-weighing encoding methods, aligning more closely with the retrieval tasks, encode a query/document as a weighted sparse representation in vocabulary space (Formal et al. 22021b [Shen al. 2022]. It first weights all vocabulary lexicons for each word of document/query based on the contexts, leading and usually large, e.g., 30k). The text is then denoted by aggregating over all the lexicons in a sparse manner. More specifically, built upon causal language models (CLM) (Brown et al.) 2020 Raffel et al. query for
high- high-dimension sparse vector R (V) is the vocabulary size
2020), (Nogueira et al. 2019a) proposes to leverage the concurrence between a document and
sparse representation expansion. Built upon masked language models (MLM) (Dellin et al. 2019 Liu et al. (2019), Formal et al. (0021b) propose to couple the original word with top coordinate terms (full of synonyms and concepts) from the pre-trained MLM head. Lastly, the relevance is calculated by texical-bases matching metrics between the sparse lexicon representations (e.g., sparse dot-product and BM25 (Robertson & Zaragoza 2009)).
19


Reviews:
[{"review_id": "nkMdyMxrOm", "reviewer": "Reviewer_KtuL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a pre-training method specifically for large-scale retrieval in the text domain. Specifically, it uses an auto-encoder architecture, with a bottleneck on the pooled lexicon-based representations in between the encoder and the decoder. After pre-training, the model is adapted to the task of large-scale text retrieval (with a technique called lexicon-weighting retriever). In terms of empirical studies, the paper benchmarked on several text retrieval tasks, either with full fine-tuning, or with zero-shot transfer. Also a lot of analyses are done, including a very extensive one for efficiency. According to the tables and figures, the proposed approach (LexMAE) achieves impressive improvement over methods compared.", "summary_of_the_review": "Overall, I think the strengths outweighs the weaknesses for the paper. The paper does some task-specific pre-training, and shows some good results. The writing is probably okay for researchers within the domain but a bit effort-taking for ones outside of the domain. The results are impressive and the empirical validations are extensive, but there are concerns about the reproducibility of the paper.", "strength_and_weaknesses": "Strengths:\n- While I am less familiar with the detailed metrics used for text retrieval, the results presented in the paper is the most impressive part to me. dIt seems LexMAE can significantly improve prior-arts, and also maintain a very healthy speed-accuracy trade-off. The evaluation is also not just done on one dataset, but on multiple datasets. \n- I also think the technical contributions of the paper is of good value. For example in how to construct a reasonable bottleneck to make the task meaningful enough to pre-train representations, while also get aligned with the downstream task (text retrieval with lexicon-weighting).\n\nWeaknesses:\n- One high-level weakness is that pre-training is really supposed to work with not just one task, but with multiple tasks. Right now the pre-training is adapted in way that works particularly well for one task. I am not saying this is bad, but doing pre-training for just one task sounds a bit less scalable -- for 1000 tasks it has to design 1000 pre-training models and actually run them.\n- The writing of the paper is also a bit unfriendly to folks who are directly working on text retrieval. It mentioned about different ways of doing retrieval (dense or lexicon-weighting) at the beginning of the paper, and only explain such terms in more detail in later parts. It could be made better to just consolidate all the background knowledge into one section, for easier reference and make the paper more self-contained.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NNlX1pDfaN", "reviewer": "Reviewer_12su", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new training framework for a language model that represents text in a lexicon space. The main motivation is that the large-scale retrieval (first stage retrieval) requires a high retrieval performance and a low latency. The dense representation approach could not work smoothly with the indexing technique developed in the search engine (inverted index etc.) Therefore, a method that maps the text in a sparse latent space is required. To learn such model, the paper proposed an encoder-decoder style self-supervised training framework. The encoder first encodes the text as a sparse embedding. However as the sparse embedding spans over the whole vocabulary. It is trivial to decode the full text from the sparse embedding. To add the difficulties, the author proposes the encode the sparse embedding into a continuous bag-of-word vector. The bag-of-word vector is concatenated with the masked text to be the input for the decoder. The proposed approach achieves a better performance than other state-of-the-art methods. It also achieves faster inference speed (in terms of the query per second metrics.)", "summary_of_the_review": "I think the paper is novel and interesting to read. The observation of efficiency-efficacy is pretty interesting and important for real-world product. I think this paper is a good one and should be accepted.", "strength_and_weaknesses": "The paper is well written and the motivation is very clear. I love the idea of mapping the text into lexicon space and uses the inverted index as retrieval method to achieve better performance and better efficiency. \nIt is also great to see the improved performance for passage retrieval on MSMarco and better efficiency comparing with the dense embedding.\n\nI have some quick questions:\n1. Conceptually speaking, why would the sparse embedding (lexicon representation) approach achieves better performance than dense representation model?\n2. For Figure 2, how to calculate the query per second for the dense embedding approach? I just curious whether the sparse embedding approach achieves better efficiency than the dense representation, especially given the fact that there also exists retrieval system for dense representation (e.g. faiss[1] etc.) Are you arguing that the sparse representation is more efficient than the dense one on the CPU?\n3. For appendix D, is it sparsify instead of specify?\n4. For appendix D, I wonder what is the top-K referring to? Given a text, we can encode it into a sparse embedding. Does the top-K mean the top-K activations on the sparse embedding?\n5. I just curious what is the performance if not using the continuous bag-of-word technique?\n\n\n[1]Johnson, Jeff, Matthijs Douze, and Hervé Jégou. \"Billion-scale similarity search with gpus.\" IEEE Transactions on Big Data 7.3 (2019): 535-547.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "H2BXqU0i67", "reviewer": "Reviewer_Bg1k", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aimed to learn sparse representation for large-scale retrieval, which often enjoys low latency and smaller index size. The author proposed lexicon bottlenecked masked autoencoder (LexMAE). The encoder outputs an importance-aware lexicon representation and the decoder aims to reconstruct the missing tokens via the help of the lexicon representation. After three-staged fine-tuning, LexMAE not only achieved new SOTA results on MS-MARCO and TREC-DL benchmarks, but also enjoyed smaller index size and lower latency.  ", "summary_of_the_review": "This is a novel work with strong empirical performance, which may have a large impact on the large-scale retrieval community. I would like to learn more details of LexMAE and its ablation.\n\nSpecifically,\n- How scalable is LexMAE to a larger vocab space? In some applications (e.g., Product Search),  to index a wide range of model serial numbers or code-name of products, we may want the vocab space of size millions or more. Also, real-world text data can also be multilingual, hence the #sub-words can also be more than hundred of thousands.\n- Does the pre-trained only (un-supervised) LexMAE always perform better than BM-25? For Table 1&2, can you include the Recall@k numbers of pre-trained only LexMAE, without fine-tuning on any supervised data (in-domain as well for BEIR)?\n- How important is FLOP() term of Eq(10) in fine-tuning and how much it affect the performance?  \n", "strength_and_weaknesses": "*Strength*\n- Strong empirical results\n- Controllable index size and low latency\n- Predictions are explainable (advantage of sparse retrieval in general)\n\n*Weakness*\n- The sparse lexical presentation from encoder has a different form in pre-training (Eq. 3) and fine-tuning (Eq. 8). Why such disparity? Does performance drop if using the same?\n- Many hyper-parameters to tune. For example, three-stage fine-tuning, alpha, and lambda, etc.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "-OVtzVNBy0r", "reviewer": "Reviewer_v7Be", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The lexicon-weighting paradigm in large-scale retrieval has achieved certain results. However, the language modeling prefers certain or low-entropy words whereas the lexicon-weighting retrieval favors pivot or high-entropy words. In view of the above problems, a lexicon-bottlenecked masked autoencoder (LexMAE) is proposed to learn the vocabulary representation of importance perception. The lexicon-importance distribution is learned unsupervised by constructing a continuous word package bottleneck. The experimental results show the effectiveness of the proposed method.", "summary_of_the_review": "The writing structure of this paper is clear and the innovation is relatively new. The experiment has fully verified the performance of the method from many aspects, which is a good article on the whole.", "strength_and_weaknesses": "Strength:\n（1）The innovation of this paper is novel. This work proposes a lexicon-bottlenecked masked autoencoder. LexMAE encoder assigns higher importance scores to essential vocabulary lexicons of the masked document but lowers to trivial ones. It is used to solve the gap between language modeling and lexicon-weighting retrieval in the vocabulary-weighted paradigm.\n（2）The method proposed in this paper has a good effect and has achieved improvement on multiple datasets.\n（3）The experimental analysis of this paper is sufficient. In addition to verifying the effectiveness of the proposed method on multiple datasets, this paper also conducts efficiency analysis, Zero-shot Retrieval, Multi-stage Retrieval Performance, and other experiments to comprehensively verify the effectiveness of the method in this paper.\n\nWeaknesses:\n(1) The sparsity defined in this paper is lack description, so it is hard to understand the experiments in Figure 2 and Table 4.  The efficiency of the proposed model is represented by increasing the sparsity, so it is necessary to make it clear in section 3.4.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster