NAIL: Lexical Retrieval Indices with Efficient Non-Autoregressive Decoders
Authors: Livio Baldini Soares
Daniel Gillick
Jeremy R. Cole
Tom Kwiatkowski
Venue: EMNLP 2023 Main (2023)
Keywords: information retrieval
text retrieval
large language models
non-autoregressive decoders
TL;DR: We describe a method to reuse large-language models to construct document representations for retrieval that performs well without query-time neural processing.

Abstract: Neural document rerankers are extremely effective in terms of accuracy. However, the best models require dedicated hardware for serving, which is costly and often not feasible. To avoid this servingtime requirement, we present a method of capturing up to 86% of the gains of a Transformer cross-attention model with a lexicalized scoring function that only requires 10−6% of the Transformer’s FLOPs per document and can be served using commodity CPUs. When combined with a BM25 retriever, this approach matches the quality of a state-of-the art dual encoder retriever, that still requires an accelerator for query encoding. We introduce nail (Non-Autoregressive Indexing with Language models) as a model architecture that is compatible with recent encoder-decoder and decoder-only large language models, such as T5, GPT-3 and PaLM. This model architecture can leverage existing pre-trained checkpoints and can be fine-tuned for efficiently constructing document representations that do not require neural processing of queries.

NAIL: Lexical Retrieval Indices with Efficient Non-Autoregressive Decoders
Livio Baldini Soares
Daniel Gillick
Jeremy R. Cole
Tom Kwiatkowski
Google Deepmind
fliviobs, dgilliik,, tomkwiat j
Abstract
quickly at serving time. In many application set- tings, this may be impractical or costly, and as we will show, potentially unnecessary.
Neural document rerankers are extremely ef- fective in terms of accuracy. However, the best models require dedicated hardware for serving, which is costly and often not feasible. To avoid this serving-ting time requirement, we present method of capturing up to 86% of the gains a Transformer cross-attention model with lexicalized scoring function that only requires 10-5% of the Transformer's FLOPs per doc- ument and be served using commodity CPUs. When combined with a BM25 retriever, this approach matches the quality of state-of- the art dual encoder retriever, that still requires an accelerator for query encoding. We intro- duce NAIL Aoooruuoreeeessive Indexing with Language models) as a model architecture that is compatible with recent mecder- and decoder-only large language models, such as T5, GPT-3 and PaLM. This model architecture can leverage existing pre-trained checkpoints can be fine-tuned for efficiently construct- ing document representations that do not re- quire neural processing queries.
In particular, we explore a retrieval paradigm where documents are indexed by predicted query token scores. As result, scoring query-document pair (g. d) simply involves looking up the scores for the tokens in associated with d in the index. While the scores are predicted by neural network, the lookup itself involves no neural network infer- ence so can be faster than other approaches. How- ever, this also means that there can be cross- attention between specific query and document or globally learned semantic vector space. Given these shortcomings, it is unclear that such model, which offloads all neural network computa- tion to indexing time, can be practical alternative to its more expensive neural counterparts.
addition, while large pre-trained language models have been shown to generalize well over a number of language and retrieval tasks (Chowdh- ery et al., 2022; Raffel et al., 2020; Brown et al., 2020; Nogueira et al., 2019b; Ni et al., 2021), a key challenge that they have universally adopted a sequence-to-ssqueeee architecture which not obviously compatible with precomputing query scores. Naive approaches are either computation- ally infeasible (scoring all possible queries), or rely on sampling small, incomplete set of samples (such as in Lewis et al. 2021).
1 Introduction
We attempt to answer the following question: to what extent can the computationalll-imienn infer- ence in modern neural retrieval systems be pushed entirely to indexing time?
Neural networks have revolutionized informa- tion retrieval, both with powerful reranking models that cross-attend to query and document, and with dual-encoder models that map queries and docu- ments to a shared vector space, leveraging approx- imate nearest neighbor search for top-k retrieval. The strongest systems typically use a dual-encoder for retrieval followed by a cross-attention reranker to improve the ordering. However, both these com- ponents tend to be built on increasingly large Trans- formers (Ni et al., 2021; Nogueira dos Santos et al., 2020; Izacard et al., 2021; Hui et al., 2022) and thus rely on dedicated accelerators to process queries
To overcome this challenge, we introduce a novel use of non-autoregressive decoder architec- ture that is compatible with existing Transfomer- based language models (whether Encoder-Decee or Decoder-only, Chowdhery al. 2022). It allows the model, in a single decode step, to score vo- cabulary items parallel. This makes document indexing with our model approximately as expen- sive as indexing with document encoders used recent dual-encoder retrieval systems (Ni et al., 2021; Izacard et al., 2021; Formal et al., 2021a). We call the retrieval system based on this proposed

---

System
Cross-Atiention Enc.
Query/Dual Enc.
Lexical
M-33
MiniLM-L6
TinyBERT-L6
GTR-XXL
Contriever
Splade-v2
BERT-tny
Splade-doc
NAIL
10
1040
1040
10th
10
10a
10
10
10a
Table I: Estimated FLOPS required score (query, document) pair, using estimators by Clark (2020). For dual- encoder and lexical systems, document representations are precomputed. query is assumed of length 16 tokens, and document assumed length of 128 tokens. The standard versions of Splade-v2 and Contriever are hased BERT-hasc.
model NAIL Auon-uutorerressivee Indexing with Language models). We summarize our contribu- tions as follows:
both (1) unconditional with respect to the query, and (2) indexed using lexicalized features (specifi- cally, we use a vector of token scores). As result, this type of document representation can be pre- computed (at indexing time) and does not require expensive computation at query-time. Prior work on leveraging language models to produce such lexicalized term weighting can be roughly divided into two groups: those with just document-side encoders, and those with query-side and document- side encoders.
1. We advance prior work on learned sparse re- trieval by leveraging pretrained LMs with novel non-autoregressive decoder.
2. We describe a range of experiments using the BEIR benchmark (Thakur et al., 2021) that explore the performance and efficiency of our model as a reranker and as re- triever. As reranker, NAIL can recover 86% of the performance of large cross-attention reranker (Nogueira et al., 2020), while requir- ing 10-%% of the inference-time FLOPS, As retriever, NAIL has extremely high upper the performance
Examples of the first group include DeepCT (Dai and Callan, 2020), DeepTR (Zheng and Callan, 2015), and DeepImpact (Mallia al., 2021), Tilde v2 (Zhuang and Zuccon, 2021), and Splade- doc (Formal et al., 2021a). These systems amples of the model paradigm in which all neural network computation happens at indexing time. Our work can be seen as an at- tempt to update these systems (which use word2vec embeddings or encoder-only language models) modern encoder-decoder architectures. Splade-doc the most recent (and performant) of these, so in many cases the most useful point of comparison for our work. We include results for the best version of Splade-doc (Lassance and Clinchant, 2022).
bound for of all other retrievers in the zero-shot setting. Finally, by using BM25 retriever and NAIL reranker, we match state-of-the-ar dual- encoders (Ni et al., 2021; Izacard al., 2021) with 10-4% of the inference-time FLOPS.
exploring,
3. We propose our model as preferred solution when significant compute is available at index- ing time, but not on-demand at serving time, and we provide a cost analysis that illustrates when our approach could be preferred to previ- ous work that harnesses LLMs.
Examples of the second group include SPARTA (Zhao et al., 2021), CoIBERT (Khattab and Zaharia, 2020), ColBERT v2 (Santhanam et al., 2022), COIL. (Gao et al., 2021), Splade (Formal et al., 2021b), and Splade v2 (Formal et 2021 These sparse dual-encoders have proven them- selves competitive with dense dual-encoders, and have some advantages like improved interpretabil- ity. We demonstrate comparable performance without the need for any query-side encoder.
2 Related work
There has been much work in information retrieval leveraging neural networks, which cannot ade- quately cover in this paper. For comprehensive overview, we refer the reader the survey by Ham- barde and Proenca 2023. Here, we focus meth- ods that minimize the use of expensive neural meth- ods at query inference time (typically methods of sparse retrieval) and on those that leverage LLMs.
LM-based Document Expansion Another way to improve retrieval indices using language mod- els document expansion. This consists of aug- menting the terms in a document that do not oc- cur in its original text, but are likely to be useful for retrieval. When used combination with a lexicalized retrieval index, document expansion can be implemented without additional query-
LM-based Term Weighting Bag-of-words mod- cls, such as TF-IDF and BM25 (Robertson and Zaragoza, 2009), use term weighting based on cor- pus statistics to determine relevance of document terms to query terms. Our work can be seen as a way to construct document term weights that are

---

time computational requirements. Recent exam- ples of LM-based document expansion systems include Doc2Query (Nogueira et al., 2019c) and Doc2Query-T3 (Nogueira al., 2019a).
This form is familiar from both traditional lexical- ized retrieval and from more recent work on dense retrieval. In lexicalized retrieval, (e.g., TF-IDF and BM25) (Robertson and Zaragoza, 2009; Robert- son and Walker, 1994), by and @4 assign non-zero scores to sub-strings q and d. On the other hand, in dense retrieval (Karpukhin et al., 2020; Ni et al., 2021; Izacard et al., 2021), Φg and Qd are neural networks that map q and d to dense vectors. Note that this formulation does not allow for deeper in- teractions between d and q. such as cross-encoder scorers, as these cannot be computed efficiently and without an accelerator at query time.
Other forms of document expansion include the Probably asked questions database (Lewis al., 2021) which, via an expensive offline system, uses a generative language model to produce lists of questions for every document the corpus.
We agree with Lin and Ma (2021) that doc- ument expansion typically improves the quality of retrieval systems, irrespective of representation used. Our approach, however, makes no assump- tions about which terms should be used to index a document, allowing the model to score all tokens in the vocabulary.
We investigate an alternative formulation of Equation than either traditional lexicalized re- trieval or dense retrieval. In this formulation, Od can be an arbitrarily complex neural network, but Oq must be a sparse featurization that can be quickly computed on commodity CPUs. This way, it is possible to push all costly neural network in- ference to indexing time, and avoid the need for accelerators at serving-time. For this paper, we choose 0g to be simple tokenizer, but we believe that our results could also extend to more complex sparse featurizations.
Non-autoregressive decoders Non-autoregres- sive sequence-to-sequeen models have been pre- viously proposed and studied, particularly in the context of machine translation (Gu et al., 2018; van den Oord et al., 2018; Lee et al., 2018), moti- vated by the computational complexity of standard auto-regressive decoding, which requires decode step per generated token. Non-autoregressive de- coding breaks the inter-step dependency and thus provides two computational benefits: (1) single step through the decoder can produce outputs for more than one position, and (2) computation can be easily parallelized since are is no time- wise de- pendencies between computations.
3.1 Independent prediction of query tokens Given the choice of og described above, we need to learn function du that can assign high scores to tokens that are are likely to occur in query as- sociated with the input document and low scores to tokens that are unlikely to appear in such query. This goal differs from related work on query pre- diction for document expansion (Nogueira et al., 2019b; Lewis et al., 2021) where only few likely query terms are added to the set of document terms.
While these systems use non-autoregressive de- coding perform iterative generation of text, we know of no existing work that uses non- autoregressive decoding to produce document rep- resentations or for retrieval purposes.
3
NAIL Model
Instead of aiming to predict small number of queries that related to d, we aim predict featurization of d that be used to score any query. Given that important motivation of this work is to make use of large pretrained language models, we must also investigate how best to adapt the segueeccett--qqueeee generative architecture that most such models have adopted. In particular, the Transformer-base based language models adopt an autoregressive decoding strategy, where the model predicts a single token position at a time, condi- tioned on the output of previous predictions. A naive decoding strategy, of decoding every possible target query ahead of time, is not computationally feasible, requiring 324 = 10 decode steps (or more generally, D where V is the vocabulary and I is the length of the query).
A major goal of this work is to investigate retrieval methods that forego neural computation and the need for specialized accelerator hardware at query time. As such, we focus on a method that uses a large neural model to precompute the required representations of the retrieval items (documents) ahead of time. Then, at retrieval time, the method performs only basic featurization (e.g., tokeniza- tion) of the queries.
Specifically, we investigate query-document scoring functions that score the compatibility of a query-document pair with the inner-product of separate featurizations of the query (g(g) and doc- ument (d).
score(g, d) (0,(())
(1)

---

melck
Max
(c)
(b)
Encoder
Parallel Decoder
Multh hot vector
Cen
1961 Muni Gegarin first huree being travel
Soviet
Input: Prompt Possag9
Target: Query
Figure 1:
model adapts
encoder-decoder architecture to predict query token scores given an input passage The
architecture is modified eneoder (a) reads an input passage. The decoder (b) initialized from pretrained T5 checkpoint, decoding parallelized few ways to be non-autoregressive: the only inputs are the standard position embeddings, efficiency, and output each position the full distribution over the vocabulary. Finally. take max over the position axis (c) to produce vector token scores corresoonding to multi-hot vector of tokens appearing in target query.
How do we generate document representations, using sequence-to-e to-sequunce architecture, in a computationally efficient way?
based on the previous predicted tokens. Each output token additionally conditions on a rela- tive position embedding based on the current decode position. Here, instead there are a fixed number of decode positions which all proceed simultaneously, conditioning only on the input and fixed position embedding.
To do this, while also making use of pre-trained Transformer language models, we modify the de- coder stack to support independent predictions of the output tokens (also known in the literature as non-autoregreessvve decoding, Lee al. 2018; Gu al. 2018). In addition, we modify the output of the model that instead of generating token se- quence, it generates sequence of scores over the vocabulary. We use this predicted sequence of vec- tor of scores over the vocabulary as a representation of the document in our system.
2. both the standard T5 model and our adapta- tion of it, each token position outputs distri- bution over the entire output vocabulary. Nor- mally, this produces single sequence tokens by sampling or taking the maximum probability token at each position. Here, we instead pool over all positions, taking the maximum token score produced at any position.
Our model architecture is illustrated in Figure 1. In this model, each output token is predicted inde- pendently from other output tokens, and is condi- tioned only input sequence and positional infor- mation. This allows the model to produce output for all positions in parallel. In addition, because the output representation is no longer single token, obtain
A simpler alternative would be to have the model decode for only single position and then the produced distribution the scores for each token. However, we found that the model was able represent a more diverse and setter-performin dis- tribution of query tokens when it could distribute their predictions over multiple output positions.
but scores over the entire vocabulary, representation for scoring any possible query q in single step of the decoder.
3.2 Contrastive training
The NAII. model is based on the T5 architec- ture (Raffel et al., 2020) and, for the experiments in Section 5, we start with pre-trained T5 checkpoints. There are several ways to use such model to pre- dict feature scores. NAIL uses the T5 vocabulary as its fcaturization, consisting of 32,000 tokens. In order to quickly score all 32,000 tokens, we modify the baseline model in two ways:
Similar to previous work that has trained dual en- coders for retrieval, we utilize negative training examples in order to do contrastive learning. In particular, we assume training data of the form D {(90, of d5)...., (gns d d,)) made up of triples that associate a query gi with posi- tive passage d; and set k negative passages d = (d/.o..... ). J. The negative passages are typically related to the query but are worse re- trievals than the positive passages.
1. The standard encoder-decoder model proceeds regorsegresy,v predicting the next token

---

tuning stage that relies on question-answent via explicit hard negatives. We present the details of each of the training steps in Sections 4.1 and 4.2. We train NAIL by assembling D into batches of m examples and calculating an in-batch softmax that includes both positive and negative passages from the batch (Ni al., 2021). Let a single batch of m examples be
pseudo-passage. In independent cropping, two con- tiguous spans of text are sampled from the passage. As the spans are selected independently, overlaps between them are possible. For the inverse cloze task, a contiguous span is initially selected from the passage, forming pseudo-query. The second span encompasses the remainder of the passage with the sub-sequence selected in the first span omitted. In both tasks, we use the C4 corpus (Raffel et al., 2020), a cleaned version of Common Crawl's web crawl corpus. In each training batch, half of the examples are from the independent cropping task and half are from the inverse cloze task. In addition, each target has a single correct corresponding input, and all other inputs serve as negatives.
bi = ((tirm, diam= of feem)).....
(g (jimm 1-mm1: diemeimm-1 d tam (m-))
and let d, be all of the positive and negative candi- date passages in this batch. The per-example loss for a query and positive passage d' drawn from batch b; is
== (dg(a)) )o(d")) log Σ expp((,(()) d'Ed,
(3)
We found this pre-training to be very important to calibrate language model scores to lexical re- trieval scores. One possible reason is that while highly frequent words (stop words) typically have high score in LMs, they are known to be insignif- icant or harmful in ranking retrievals independent of the context or inputs in which they occur. Addi- tional discussion of the need for pre-training can be found in Appendix B.2. We run pre-training for 500k steps on batches of 2048 items, the largest size are able to fit into accelerator memory.
and we train the model incrementally minimize the per-batch loss, summed over all m examples in the batch. Note that the number of explicit negative passages can vary under this setup, as the positive passages for other queries serve as implicit negative passages for every other query. More details about the training setup are given in the following section.
4 Model Training and Experiments
To train the NAIL. model, we have empirically found it beneficial to perform two stages of training (1) a pre-training stage the uses self-supervised tasks over large, unlabeled text corpus, and (2) a fine- data
4.2 Fine-tuning
We finetune our model on the MS-MARCO dataset (Nguyen et al., 2016). It consists of roughly 500,000 queries, each with corresponding set of gold passages (typically one per query) along with set of 1,000 negative passages produced by run- ning a BM25 system over the full corpus of 8.8M passages. We construct training examples using the gold passage positive, along with a sample of the BM25 candidate passages as hard negatives.
Our model is implemented within the T5X framework (Roberts et al., 2022) and we initial- ize model weights with published 15.1.1 check- points (Raffel et al., 2020). Unless otherwise noted, the NAIL model size used in the experiments is XL, with roughly 3 billion parameters. We saw no further gains from increasing parameters further.
We investigate a variable number of MS- MARCO hard negatives and find that more hard negatives improves MS-MARCO performance but worsens BEIR performance. More details can be found in Appendix B. Similar to pre-training, each batch consists of 2048 total passages.
To be compatible with T5 checkpoints, we also adopt the T5 vocabulary and attendant Senten- cePiece tokenizer (Kudo and Richardson, 2018). The vocabulary consists of 32,000 tokens extracted from a English-focused split of Common Crawl.
4.3 Evaluation Methodology
4.1 Pre-training
For evaluation, we focus on the public, readily- available, datasets available in the BEIR (Thakur et al., 2021) suite and which have baseline num- bers present in the Icaderboard, which totals 12 distinct datasets. We specifically target BEIR since contains a heterogeneous set of retrieval datasets, and equally importantly, evaluates these datasets in zero-shot setting. While neural models have made
For pretraining, we combine two related self- supervision tasks for retrieval: inverse cloze and in- dependent cropping (Lee et al., 2019; Izacard et al., 2021). Both of these tasks take in a passage from a document and generate a pair of spans text, forming a positive example. One of the generated spans serves as a pseudo-query and the other as a

---

huge gains over BM25 on in-domain data, BEIR shows that a varicty of neural retrievers underper- form relative to BM25 on out-of-domain data.
nDCG@
Cross Enc.
Lexical
MoTs
MiLM
BM25
NAIL
MS-Marco
0.398
0.401
0.228
0.377
BEIR results are typically presented as two sep- arate tasks, where most systems are only evalu- ated on either the rerunking variant or the full re- trieval variant. In the full retrieval variant, sys- tems must retrieve over the provided corpus of document passages, which range from a few thou- sand to a few million, and they are evaluated on their recall 100 and their nDCG@ 10 (Järvelin and Kekäläinen, 2002), providing a view into their ability to retrieve the gold passages into the top 100 and the ordering of the top ten passages, respec- tively. In the reranking variant, models do not have to do retrieval, and the recall @ 100 is fixed to the performance of an fff-th BM25 system, so only nDCG@ @ 10 is reported.
Arguana
0.288
0.415
0.472
0.522
Climate-Fever
0.28
0.24
0.186
0.206
DBPedia-cntity
0.478
0.542
0.320
0.376
Fever
0.85
0.802
0.650
0.692
FIQA-2018
0.514
0.334
0.254
0.411
HotPoiQA
0.756
0.712
0.602
0.644
NFCorpus
0,384
0.36
0.343
0.367
Natural Questions
0.633
0.53
0.326
0.487
SciDocs
0.197
0.164
0.165
0.160
SciFact
0.777
0,682
0.691
0,710
Treo-Covid
0.795
0.722
0.688
0.766
Touché 2020
0.3
0,349
0.347
0,240
BEIR Avg
0.511
0.481
0.405
0.458
BEIR- MS-Marco
0.521
0.488
0.420
0.465
Total FLOPS
10
10
10*
Table BEIR results on reranking task (top 100 results from BM25). Note that we use BM25 candidates from ElasticSearch system. Results all systems, Mo(no) (3B), Mi(ni) and BM25 are copied from the BEIR reranking leaderboard. Note MS- MARCO is in-domain the trained models.
5 Experimental Evaluation
We compare NAIL other systems that have pub- lished results on BEIR. To compare with some sparse systems that have not been evaluated on BEIR datasets, we also make of use the MS- MARCO passage ranking task. We focus on an- swering the following questions:
et al., 2020) and MiniL.--L6' 1 MiniLM-L6 is BERT-based models trained on MS-MARCO us- ing cross-encodes classifier. MonoT5-3B uses a T5-based model fine-tuned on MS-MARCO, using generative loss for reranking.
How does NAIL perfom reranker, particu- larly when compared to much more expensive neural reranker systems?
Results Table 2 shows the reranking results. The baseline comparison for NAIL's 's performance here is BM25 alone: using BM25 without reranker is the only other method that does not need to neural network for each query. We see that NAIL improves over BM25 fairly consistently. The im- provement on MS-MARCO, which has in-domain training data, especially striking. On BEIR, NAIL improves performance 10 out of the 12 datasets increasing the average score by over points.
How does NAIL compare to recent term weight- ing retrieval systems that use language models?
How does NAIL compare with similarly trained dual-encoder system that uses an expensive query-side encoder?
Further experimental work is also presented in appendices, including: qualitative analysis (Ap- pendix A), sensitivity to herd-nega in the batch loss (Appendix B. effects of ablating pre-training or fine-tuning (Appendix B.2), and analysis of sparsifying document representations to make them more efficient for indexing (Ap- pendix C).
While cross-ent cooseer models are more powerful, they are also more expensive. Cross-encoder mod- els must run inference on all 100 documents for each query. Thus, NAIL uses 8 to 9 orders of mag- nitude fewer FLOPS than cross encoder models, corresponding to almost 1 trillion fewer FLOPS for a single query. Moreover, NAIL significantly closes the gap between the BM25 bascline and the top performing cross-encoder rerankers, capturing 86% of the gains on MS MARCO and 45% of the gains the broader suite of BEIR tasks. Thus, it presents attractive alternative to expensive
5.1 Reranking
In the reranking BEIR task, each system must rerank the 100 passages returned by an off-the- shelve BM25 system.
Baselines In this section we divide approaches into two types of systems: lexical-based ap- proaches and cross-encoders. In the cross-encoder category, we compare to MonoT5-3B (Nogueira
huugihggiiccoooooscccooeeemmmmme M-LL- v2
M--L-6-

---

rerankers when compute is limited.
used (Lassance and Clinchant, 2022). As with SPLADE v2, SPLADE-doc is trained using dis- tillation of cross-eneoder reranker, with additional fine-tuning on MS-MARCO.
5.2 Full Corpus Retrieval
In the full corpus retrieval task. each system must retrieve and rank from each dataset's 's corpus. Because NATL is very cheap to run as reranker, it is reasonable to compare the BM25+NAIL results from Section 5.1 to direct retrieval systems that do not include reranking step, but typically consume many orders of magnitude more FLOPs at query time. Table 3 presents this comparison.
Results Table 3 shows the results for nDCG@ 10 and recall 100 BEIR full corpus retrieval for all systems that report it. We stratify the results into two sets, (1) MS-MARCO, which with the exception of BM25, is used as a training dataset, and (2) the average over all the other BEIR datasets, which are evaluated as zero-shot.
As NAIL could be used to populate an inverted index, we investigate how well NAIL works when scoring all candidates in the corpus, which is an upper-bound for a NAIL-only retrieval system. These results are presented NAIL-cxh in Table 3.
On the out-of-domain BER tasks, BM25+NAIL beats all but one of the neural retrieval systems, de- spite not encoding the query with a neural network and being limited recall to BM25. Addition- ally, we note that NAIL-exh outperforms all other retrieval systems according to the recall @ 100 met- ric, suggesting potential for a NAIL-based retriever that uses NAIL to populate an inverted index. How- ever, given the lower nDCG@ 10 than BM25+NAIL, this may only be worthwhile to implement if com- bined with a different reranker. Note that while re- call @ 100 is highest for NATL on the out-of-domain BEIR tasks, NAIL does worse than other models like GTR-XXL on the in-domain MSMARCO task. This is in part due to the training recipes used by other work to optimize for MS-MARCO perfor- mance, including model distillation and large non- public corpora of QA pairs.
We later present a brief investigation into the effect sparsification of the NAIL output, to further understand the potential for using NAIL to populate a sparse inverted index for retrieval.
Baselines For full retrieval, compare NAIL to lexical-based and dual-encoder systems.
GTR-XXL (Ni et al., 2021) is one of the largest and best performing dual-encoder systems publicly available. pre-trained on a large, non-public, corpus of 2 billion QA pairs scraped from the web, and fine-tuned on MS-MARCO. Contriever a dual-encoder system which employs novel self- supervised pretraining task (Izacard et al., 2021) and is fine-tuned on ASSMARCO,; we describe in more detail in Section 5.4.
SPLADE-doc also does not require a query-time encoder. We observe that NAIL lags on the in- domain evaluation but outperforms SPLADE-doc on both metrics of the zero-shot datasets in BEIR. As with many of the other retrievers, SPLADE- doc was distilled from a cross cross-attention reranker teacher, which may account for this in-domain gain in performance (Gao and Callan, 2022: Formal et al., 2022).
SPLADE v2 (Formal et al., 2021a) develops query and document encoders to produce sparse representations, differing from dense dual-encoded systems. The query and document representations in SPLADE v2 are used for slightly different objec- tives. The query encoder is used to perform query expansion, and the document encoder is used to produce sparse representations for indexing. This system is trained via distillation of cross-encoder reranker, and finally fine-tuned MS-MA.C.. Colbert v2 adopts late interaction model that produces multi-vector representations for both doc- uments and passages. In this model, per-token affinity between query and document tokens are scored using per-token representations. It trained via distillation of eross-encodes reranker.
5.3 Comparison to Term Weighting Models
In this work we are primarily interested in the zero- shot multi-domain retrieval task represented by BEIR. However Table also contains comparison to other recent systems that use LMs to compute term weights, using the in-domain MS-MARCO passage retrieval task that they focused on. For NAIL, we report both the version which uses BM25 retrievals (in that case, the recall metric is derived from the BM25 system) and the system described in the previous section which uses exhaustive scor- ing. The results demonstrate that both NAIL-cxh and BM25+NAIL. outperform the other term weight-
Besides BM25 and NAIL, SPLADE-doc++ is the only other retriever that does not require neural network inference at query time. This model is a variant of SPLADE v2 where the query encoder is dropped, and only the document encoder is

---

Metric
Dual encoder
Query encoder
Lexical (no inf. net.
GTR XXL
Contriever
SPLADE v2
Colbert v2
BM25
SPLADE doe"
NAIL-extt
BM25+NAIL
MS-MARCO
10
0.442
0.433
0.228
0.431
0.396
0.377
recall 100
91.6
89.1
66.0
88.4
89.5
66.0
1DCG@10
0.459
0.429
D.465
(arg, over 12 datasets)
recall 100
64.4
64.4
64.6
61.8
66.5
64.6
large QA corpus Pt. w/ distillation
No
No
No
Yes
Yes
No
Yes
No
No
self-supervision
No
Yes
Table 3: BEIR nDCG@ 10 and recall @ 100 results the fiall retrieval task. The SPLADE-doct results are pre- viously unpublished, corresponding to the model described in (Lassance and Clinchant, 2022), and obtained via correspondence with authors. Other numbers are obtained from their respective publications.
metric
DeepCT
Deeplmpact*
COIL-tok
uniCOIL
SPLADE-doc
BM25+NAIL
NAIL-exh
MRR@10
0.243
0,326
0.341
0.315
0.322
0,363
0.356
@ 1000
0.913
0.948
0,949
0.946
0.814
0.981
Table 4: Evaluation on the MS-MARCO dev set for passage ranking task. Numbers reported are taken from corresponding publications: DeepCT (Dai and Callan, 2020), DeepImpact (Mallia al., 2021), COIL-tok (Gao et al., 2021). uniCOIL (Lin and Ma, 2021), SPLADE-doc (Formal et al., 2021a). Results obtained without document expansion, except for DeepImpact which includes terms from doc2query-TSS (Nogueira et al., 2019a).
ing models presented on the MRR @ 10 metric for the MS-MARCO passage ranking task. With re- spect to recall, NAIL-exh clearly improves over the previous systems. Exhaustive scoring is much more expensive than the other systems shown; however, given the sparsification results shown in Figure 3, we believe sparse version of NAIL would be com- petitive with the models presented.
Table 8 (Appendix E). We observe that the BEIR reranking task, NAIL matches both the in-domain and zero-shot performance of the Contriever model, despite lacking a query time neural network. With- out using BM25 for initial retrievals, both methods perform slightly worse on nDCG@ 10 for the zero- shot BEIR tasks, but they remain comparable.
5.5 Performance versus query-time FLOPS
5.4 Comparison to Contriever
We have motivated this work by asking how much can we leverage large language models at indexing time while making query time computational costs small enough for a commodity CPU. As the results in this section show, there are tradeoffs between reranking improvements and computational costs. To illustrate this tradeoff, we present results of per- centage nDGC@ 10 improvement over BM25 ver- query-time FLOPS in Figure 4 (Appendix D). In general, think lexicalized approaches like NAIL provide an interesting point on this curve, where much higher performance than BM25 be achieved for only small amount more compute. Note that Lassance and Clinchant (2022) discuss smaller versions of Splade; see Table for the ap- proximate reduction.
There several confounding factors in comparing the systems presented in Tables 2 and 3. As men- tioned, each system uses different training recipes and training data while also having slightly differ- ent architectures. Training techniques presented in the baselines presented in this work include un- supervised pretraining, hard negative mining, and distillation from a cross-attention teacher. These factors can make it difficult to pinpoint the cause the variance in performance across models. However, NAIL and Contriever (Izacard et al., 2021) share training recipes to large extent, hav- ing both similar pretraining stage followed by fine-tuning on MS-MARCO. Contriever is re- cently introduced dual-encoder model that inspired the pretraining task in this work. However, archi- tecturally, NAIL and Contriever are quite differ- ent. NAIL'S query representation is not learned and is tied to the fixed set of vocabulary terms; this approach is potentially less powerful than fully learned dense representation.
Concluding Remarks
We introduce a new model for sparse, lexicalized retrieval, called NAIL that adapts expensive pre- trained sequence-to-sequence language models for document indexing. The main elements of NAIL are (1) the use non non- -on-autoregressive decoder, (2)
The summary of the comparison is available in

---

the use of vocabulary based representation for doc- uments and queries, (3) self- self-supervise training approach that is critical for good performance.
We believe the BEIR suite of datasets presents an improvement over prior text-based retrieval for QA, particularly focusing on wider range of datasets and in zero-shot setting. Nonetheless, BEIR does not cover some domains in which NAIL may be under-perform, Beyond multi-lingualig discussed above, we do not know how our model behaves when needing to reason about numbers or program- ming, or other domains of text which typically do not tokenize well.
With NAIL,, we foeus on offloading all neural computation to indexing time, allowing serving to operate cheaply and without the use of accelera- tors. Evaluating retricval on BEIR, we show that the NAIL approach is as effective as recent dual- encoder systems and captures up to 86% of the performance gains of a cross-attention model on MS-MARCO while being able to serve requests on commodity CPUs.
This paper demonstrates that NAIL is competi- tive with other model expensive and complex neu- ral retrieval systems. However, we do not present a highly optimized implementation of NAIL as a stan- dalone retriever. An efficient implementation based on an inverted index is needed before NAIL can be used for end-to-end retrieval in high-traffic applica- tions. Further work in sparsification of document representations (see Appendix C) is not explored in this work and is likely needed to achieve efficient indexing.
Limitations
The work presented in this paper contains several limitations. In this section we focus on limitations relating to (1) the choice of document representa- tion (vocabulary-basee vector of weights) and (2) empirical analysis using BEIR suite of datasets.
As described in Section 4, we inherit the vocabu- lary from the T5 models as basis for our document representation. This choice limits the applicability of NAIL in various ways:
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu. Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems. volume 33, pages 1877-1901. Curran Associates. Inc.
1. The vocabulary is derived from an English- focused portion of the web. As a consequence, there are very few non-English word pieces encoded in the vocabulary, such as Unicode and other scripts. We expect this will have a significant, but unknown, impact on applying our system to non-Linglish text.
2. In order to better support multi-lingual re- trieval, we expect that the vocabulary of the model will need to be extended. For example. the multi-lingual T5, (mT5, Xue et al. 2021) contains 250 thousand items, almost 8-fold increase compared to T5. This work does not study what the impact of vocabulary size increase can be the quality of document representations and subsequently, on retrieval performance.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao. Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghe- mawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fe- dus, Denny Zhou, Daphne Ippolito, David Luan, Hycontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi. David Dohan, Shivani Agrawal, Mark Omernick. Andrew Dai, Thanumalayan Sankara- narayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta,
3. Unlike learned dense representations, our vocabulary- representations may have more limited representational power. Re- cent work demonstrate that even in the case of learned dense representations, multiple representations can improve model perfor- mance (Lee et al., 2023; Zhou and Devlin, 2021). This work also does not evaluate the bound on such vocabulary-basee repre- sentations.

---

Jason Wei, Kathy Meier-Hellstem, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language modeling with pathways.
Kailash A. Hambarde and Hugo Proenca. 2023, Infor- mation retrieval: Recent advances and beyond.
Kai Hui, Honglei Zhuang, Tao Chen, Zhen Qin, Jing Lu, Dara Bahri, Ma, Jai Gupta, Cicero Nogueira dos Santos, Yi Tay, and Donald Metzler. 2022. ED2LM: Eincoderddecoder to language model for faster document re- re-ranking inference. In Find- ings of the Association for Computational Linguis- tics: ACL 2022.
Kevin Clark,
Luong, Quoe V. Le, and Christopher D. Manning. 2020. Electra: Pre- training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations. ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OvenReview.net.
Zhuyun Dai and Jamie Callan, 2020. Context-award term weighting for first stage passage retrieval. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, SIGIR '20, page 1533-1536, New York. NY, USA. Association for Computing Machin- ery.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Se- bastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Unsupervised dense in- formation retrieval with contrastive learning.
Kalervo Järvelin and Jaana Kekäläänen. 2002. Cumu- lated gain- evaluation of ir techniques. ACM Trans. Inf. Syst., 20(4):422 446.
Zhuyun Dai, Vincent Y. Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming-We Wei Chang. 2022. Promptagator: Few-shot dense retrieval from 8 examples.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference Empirical Methods in Nat- ural Language Processing (EMNLP), pages 6769- 6781, Online. Association for Computational Lin- guistics.
Thibault Formal, Carlos Lassance. Benjamin Pi- wowarski, and Stéphane Clinchant. 2022. From dis- tillation to hard negative sampling: Making sparse neural ir models more effective. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22, page 2353-2359 New York, NY, USA. Association for Computing Machinery.
Omar Khattab and Matei Zaharia. 2020. Colbert: Effi- cient and effective passage search via contextualived late interaction over bert. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, page 39-48, New York, NY, USA. Association for Computing Machinery.
Thibault Formal, Carlos Lassance, Benjamin Pi- wowarski, and Stéphane Clinchant. 2021a. Splade v2: Sparse lexical and expansion model for informa- tion retrieval. CoRR, abs/2109. 10086.
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 202 lb. Splade: Sparse lexical and expan- sion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2288-22 22982
Taku Kudo and John Richardson. 2018. SentencePicce: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonsirations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Luyu Gao and Jamie Callan. 2022. Unsupervised cor- pus aware language model pre-training for dense passage retrieval. Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 2843-2853, Dublin, Ireland. Association for Computational Lin- guistics.
Carlos Lassance and Stéphane Clinchant. 2022. An ef- ficiency study for splade models. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22, page 2220-2226, New York, NY, USA. Association for Computing Machinery.
Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021. COIL: Revisit exact lexical match in information retrieval with contextualized inverted list. In Pro- ceedings of the 2027 Conference of the North Amer- iean Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3030-3042, Online. Association for Computational Linguistics.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. Deterministic non-autoregressive neural se- quence modeling by iterative refinement. In Pro- ceedings of the 2018 Conference on Empirical Meth- ods Natural Language Processing, pages 1173- 1182, Brussels, Belgium. Association for Computa- tional Linguistics.
Jiatao Gu, James Bradbury, Caiming Xiong, Vic- tor O.K. Li, and Richard Socher. 2018. Non- autoregressive neural machine translation. In Inter- national Conference on Learning Representations.
Jinhyuk Lee, Zhuyun Dai, Sai Meher Karthik Duddu, Tao Lei, Iftekhar Naim, Ming-Wer Chang, and Vin- cent Zhao. 2023. Rethinking the role of token retrieval in multi-vector retrieval.

---

Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Compu- lational Linguistics, Florence, Italy. Association for Computational Linguistics.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. RockeiQA: An opti- mized training approach to dense passage retrieval for open-domain question answering. In Proceed- ings of the 2021 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5835 5555- Online. Association for Computational Linguisties.
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Kütteer Aleksandra Piktus, Pon- tus Stenetorp, and Sebastian Riedel. 2021. PAQ: 65 million probably-asked questions and what you can do with them. Transactions of the Association for Computational Linguisties, 9:1098-1115.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Re- search, 21(140):1-67.
Jimmy Lin and Xueguang Ma. 2021. A few brief notes on deepimpact, coil, and conceptual framework for information retrieval techniques.
Antonio Mallia, Omar Khattab, Torsten Suel, and Nicola Tonellotto. 2021. Learning passage impacts for inverted indexes. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development Information Retrieval, SIGIR '21, page 1723-1727, New York, NY, USA. Associ- ation for Computing Machinery.
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sha- ran Narang. Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowyez. Alex Salcianu, Mare van Zee, Jacob Austin, Sebas- tian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashehenko, Aakanksha Chowdhery, Jas- mijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee. Dan Garrette, James Lee- Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin- Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlam, and Andrea Gesmundo. 2022. Scaling up models and data with t5x and seqio. arXiv preprint Niv:2203.17788
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Proceedings of the Workshop Cognitive Computation: Inie- grating neural and symbolic approaches 2016 co- located with the 30th Annual Conference on Neural Information Processing Systems.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gus- tavo Hernández Ábrego, Ji Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming-Wes Wei Chang, and Yinfei Yang. 2021. Large dual encoders are generalizable retrievers. CoRR, abs/2112.07899.
Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Be- yond. Found. Trends Inf. Retr., 344):333-389
Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with pre- model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 708-718, Online. Association for Computational Linguistics.
Stephen Robertson and Steve Walker. 1994. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In An- nual International ACM SIGIR Conference on Re- search and Development in Information Retrieval.
trained sequence-to-sequeee
Keshav Santhanam, Omar Khattab, Jon Christopher Potts, and Matei Zaharia. 2022. Col- BERTv2: Effective and efficient retrieval via lightweight late interaction. In NAACL 2022.
Rodrigo Nogueira. Jimmy Lin, and AI Epistemic. 2019a. From doc2query to docttutqquery. Online preprimt, 6.
Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019b. Document expansion by query prediction. arXiv preprint arXiv: arXiv:1904.08:: 1904.08355.
Nandan Thakur, Nils Reimers, Andreas Rücklé Ab- hishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evalua- tion of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Sys- lems Datasets and Benchmarks Track (Round 2).
Rodrigo Frassetto Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019c. Document expansion by query prediction. CoRR, abs/1904.08375.
Cicero Nogueira dos Santos, Xiaofci Ma, Ramesh Nal- lapati, Zhiheng Huang, and Bing Xiang. 2020. Be- yond [CLS) through ranking by generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1722-1727, Online, Association for Computa- tional Linguistics.
Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals. Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, Norman Casagrande, Do- minik Grewe, Seb Noury, Sander Dieleman, Erich Elsen, Nal Kalchbrenner, Heiga Zen, Alex Graves, Helen King, Tom Walters, Dan Belov, and Demis

---

Hassabis. 2018. Parallel WaveNet: Fast high- fidelity speech synthesis. In Proceedings of the 35th International Conference on Machine Learning, vol- ume 80 of Proceedings of Machine Learning Re- search, pages 3918-3926. PMLR.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. Approximate nearest neigh- bor negative contrastive learning for dense text re- trieval. CoRR, abs/2007.008DD
Linting Xue, Noah Constant, Adam Roberts, Mi- hir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 483 498, Online. Association for Computa- tional Linguistics.
Tiancheng Zhao, Xiaopeng Lu, and Kyusong Lee. 2021. SPARTA: Efficient open-domain question swering via sparse transformer matching retrieval. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguisties: Human Language Technologies. pages 565-575, Online. Association for Computa- tional Linguisties.
Guoqing Zheng and Jamie Callan, 2015. Learning reweight terms with distributed representations. In Proceedings of the 38th International ACM SIGIR Conference Research and Development in Infor- mation Retrieval, SIGIR '15, page 575-584, New York. NY, USA. Association for Computing Machin- ery.
Giulio Zhou and Jacob Devlin. 2021. Multi- vector attention models for deep re-ranking. Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5452-5456, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Shengyao Zhuang and Guido Zuccon. 2021. Fast
passage re-ranking with contextualized exact term matching and efficient passage expansion.

---

A Qualitative Analysis
highly associated with the wh-word 'what'. On the other hand, passages about individuals or groups of individuals (Examples (3) and (4)) are more highly associated with 'who'.
In this section, we present a qualitative analysis of the tokens that score highest according to the NAIL model for a given input. We choose the Natu- ral Questions (NQ) subset of the BEIR benchmark for this analysis, as the queries tend to be com- plete questions that are easily interpretable. Table 5 shows the percentage of NAIL'S top predicted to- kens that appear in the passage input to the NATL model along with the gold query that is paired with this passage in the NQ development set. Figure 2 presents the top predicted terms for randomly sampled set of passages.
Finally, the predicted terms in Figure 2 contain a lot of small surface-form variations of the same root word, with different segmentations and capital- izations treated separately by the query tokenizer. For example, the tokens 'chic', 'chi', 'CHI', 'Ch', 'ch', 'CH' in Example (2) are all probably com- ing from different forms of the word 'Chicago' presented in different contexts. This redundancy illustrates drawback of our featurization: unlike neural models, it does not abstract over diverse surface forms. Future work could examine more efficient and discriminative featurizations than the tokenization used in this work.
top-100
top-1000
pretrained
tuned
pretrained
tuned
query passage
53
94
65
54
90
88
B Alternate training recipes
Table 5: Percent of NQ query and gold passage tokens contained in the top 100 and 1000 scores from NAIL.
Our primary goal has been to determine the extent to which the performance of an expensive neural network can be captured in fast, sparse, featur- ization for general purpose retrieval. Subsequently, we have prioritized a training recipe that aligned with previous work and well suited to the multi- domain BER task. However, the performance of learned retrievers as rerankers very sensitive to the exact nature of the training recipe, and in this section we present analyses of the choices we made, and the associated trade-offs on BEIR and MSMARCO performance.
Almost all of the tokens in both the input pas- sages and the unseen query are present in NAIL'S 's top 1000 predictions (Table 5). However, tuning towards MS-MARCO significantly increases the number of query tokens predicted in the top 100 and 1000 positions, while simultaneously reduc- ing the number of passage tokens predicted. This is unsurprising: the fine-tuning stage represents a domain shift from the pre-training task, which is predicting document tokens, toward predicting query tokens. One indication of this shift is the in- crease in the prevalence of 'wh' words (what, who, where) in the top terms from the finetuned model in Figure 2.
B.1 Hard- sard-negative selection for fine-tuning
One key choice in contrastive learning is the dis- tribution of negative examples used in Equation 3. This is commonly a combination of hard negatives, which are chosen to be challenging for a single ex- ample, and batch negatives, which are drawn from the distribution of all positive and hard-negative candidates across training examples (Karpukhin et al., 2020; Xiong et al., 2020; Qu et al., 2021).
Figure 2 also illustrates some other interesting shifts in NAIL'S output during fine-tuning. For ex- ample, in Example (3) the pre-trained model pre- diets many dates associated with the Eagles (e.g., album release years). These are likely to occur in adjacent passages in the same document as the in- put passage, so they are good predictions for the pre-training task (Section 4.1). However, they are
# of hard negatives
MS-MARCO nDCG@1 10
Avg. BEIR nDCG@
very unlikely to occur in queries associated with the input passage, and thus they are replaced in the fine- tuned predictions with terms that are more likely to occur in queries targeting the passage ("sang'. "sing", 'wrote', 'who', 'released').
0.377
0.465
0.378
0.461
0.391
0.460
0.394
0.457
63
0.397
0.457
Figure 2 also illustrates NAIL'S ability to pre- dict the type of query that is likely to be paired with a given passage. Passages containing defini- tions, such as the one presented in Example (1), are
Table 6: Efffect of varying the number of hard negatives on reranking evaluation for MS-MARCO and BEIR. The BEIR average is computed without MS-MAA MARCO.

---

query
NQ gold passage
Top terms from pre- trained model (predictions)
Top terms from finetuned model (predictions)
(not shown
(sole input model)
model)
(1) what is non con- trolling interest on balance sheet
accounting, minority interest (or non- controlling interest) is the portion of a subsidiary corporation's stock that not owned by the parent corporation. The mag- nitude of the minority interest the sub- sidiary company is generally less than 50% outstanding shares, or the corporation would generally cease to he subsidiary of the parent.[1]
minority, controlling, pas- sage, subsidiary. Minor, 50%, interest, accounting. control. I, Interest, ques- tion, Accounting, con- trol, non, Non, generally, owned, Answer, sub- sidiaries
minority, interest, Defi- nition, ities, controlling, Non, non, what, ity, inter- ests, control, own, Inter- est, Minor, shares, owned, ownership, Both, account- ing, interest, stock, does, mean, Control
how many episodes are chicago fire season
The fourth season of Chicago Fire, an American drama television series with executive producer Dick Wolf, and pro- ducers Derek Haas, Michael Brandr, and Matt Olmstead, was ordered on Febru- ary S. 2015, by NBC.[1] and premiered on October 13, 2015 and concluded May 17, 2016./21 The season contained 23 episodes.[3]
NBC, stead, Wolf, Fire, fourth, Chicago, scason, 2016, firefighters, con- cluded, episodes, Ha, 4, 5, contained, aire, 6. 23, 4., 4., premiere, char- acters, episode, Identify, fire
chic, fire, fourth, seasons, chi, scason, Fire, shows, ich, air, CHI, ch, Ch, four, episodes, when, se- ries, Season, CH. show, Chicago. Fi, 4, chie, fire- fighters, NBC, four, aire
(3) who sings love will keep us alive by the eagle
"ovv Will Keep Us Alive" song writ- ten by Jim Capaldi, Paul Carrack, and Pe- ter Vale, and produced by the Eagles, Elliot Scheiner, and Rob Jacobs. was first per- formed hy the Fagles in 1994, during their Hell Freezes Overicunion tour, with lead vocals by bassist Timothy B. Schmit.
Eagle, Schm, live, 1994, I Love, Keep, Us, alive, song. Free, performed, lyries, Glen, 1976, 1995, rack, IVE, 1977, during, 1975, 1993, keep. 1972. 1974, 1996, 1997, Don. album
Eagle, alive, sang, love, live, song, Cap, written, keep, sing, live, wrote, lov, Love, kept, Will, who, will, hell, keep, II, keeps, Live, tim, Us, gle, singer, songs, cap, IVE, Car, written
(4) nitty gritty dirt band fishin in the dark album
'Fishin' in the Dark" is song written by Wendy Waldman and Jim Photoglo and recorded by American country mu- sic group The Nitty Gritty Dirt Band, It was released in June 1987 as the second single from their album Hold On.[I] It reached number-one on the U.S. and Cana- dian country charts. It was the band's 's third number-one single on the U.S. country mu- sic charts and the second in Canada. Af- ter it became available for download, it has sold over a million digital copies by 2015.121 was certified Platinum by the RIAA on September 12, 2014.[3]
Wald, itt, glo, hin, Dir. Dark, Wendy, Fi, RIA. fishing, dark. 1987, song. 5. 4. million. 3, Gr, Fish, 5., single, became, Hold, Band, number, 1986, I, (4), 6, country, band, reached, Jim, 500,000, 1988
hin. fi, dark, iiy, fishing, song, Dir, Wald, sang, sing, hold, wald, fish, ?. ity. Fish, band, ge. who, shing, band, hit, dir, songs, held, ies, Wendy, singer, dirty. Hold, re- leased, Band, ISH, dirt, country, fish Dark, Song, ities, written, music, sin- gle, Country, ddy, when, wrote
Figure 2: Sample of top token predictions from pre-trained only and pre-trained-fine-tuue NAIL, models. The table shows a few evaluation examplars from the Natural Questions evaluation set included in BEIR. We display the corresponding question associated with the answer passage for the benefit of the reader, but this is not shown to the model. We have explicitly removed stop words and non-words (control sequences). Note that due to the use of SentencePiece tokenizer (Kudo and Richardson, 2018), tokens do not necessarily correspond to full words.

---

IDCG@10
Finetuned
Pretrained only
Pretrained Finetuned
1.00
only
only
MSMARCO
0.367
0.212
0.377
0.75
0.87
0.89
0.89
BEIR
0.422
0.416
0,465
0.50
0.52
0.64
0.66
Table 7: Effect of pretraining NAIL for the BEIR reranking task. The BEIR nDCG@ 10 metric corre- sponds to average score of datasets excluding MS- MARCO.
0.48
0.25
BEIR
0.00
100
500
1000
5000 10000
Terms per document (log scale)
Our pretraining task (described in Section 4.1) does not hard negatives; however, the MS- MARCO fine-tuning task includes hard negatives created by running BM25 retrieval over the set of candidate passages. Table 6 shows how BEIR and MS-MARCO results change as we change the ber of MS-MARCO hard-negatives that sam- ple during fine tuning. As this number increases, the MS-MARCO performance also increases until it matches the performance of the cross attentotn rerankers in Table 2 when 63 hard negatives are sampled for each training example. However, in- creasing the number MS-MARCO hard nega- tives also hurts BEIR performance.
Figure 3: Effect of sparsification of document represen- tation on recall @ 100, using a top-k strategy.
thousand tokens down to 100 tokens. For both MS- MARCO and other BEIR datasets, recall @ 100 falls considerably when using only the top 100 tokens. Nonetheless, with only two thousand tokens are able maintain the same level of performance for MS-MARCO and roughly 97% of the recall perfor- mance on BEIR. This observation, along with the results in Table 3, suggest that NAIL could be used to populate an efficient inverted index for retrieval, with little loss of recall. Such an index could serve as a more powerful alternative BM25. We leave this future work.
B.2 Effects of pretraining and fine-tuning
The training recipe, presented Section 4.1, has two stages beyond the language model training from Raffel et al. (2020). Table shows that both stages benefit both the BEIR and MSMARCO re- sults. However, NAIL still yields a nice improve- ment over BM25 across the BEIR tasks using only the pre-training task. This is encouraging because these data are heuristically generated rather than relying on human relevance labels, so they can be trivially applied to new domains. The MS-MARCO results are unsurprisingly more dependent on fine- tuning on MS-MARCO. Pre-trained NAIL does not outperform BM25 on MS- MS-MARCO without fine- tuning. More sophisticated methods of synthetic training data generation, such as Promptagator(Dai et al., 2022), could also help improve NAIL further, but we leave this to future work.
D Performance versus query-time FLOPS
C Sparsification
To further explore the potential for using NAIL. for full retrieval, we experiment with a naive ap- proach to sparsifying NAIL document representa- tions. Specifically, we simply order tokens by their scores and keep the top-k scoring tokens.
Figure 4 illustrates different systems with varying tradcoff between computational cost and retrieval performance. See Section 5.5 for the discussion on this figure.
Figure demonstrates the effect on the re- @ 100 metric of reducing the number of terms per document from the original vocabulary of 32

---

%6 Improvement of nDCG 10 on BEi
30
MS
20
SPLADE-v2
NAIL
10
BM25
5
10
15
Order of magnitude of FLOPS for one query
Figure 4: Improvement
BM25 and extra FLOPS score one query on the BEIR retrieval task. The NAIL and MonoT5 use BM25 retrievals: SPLADE-v2 uses its own retrievals over the full corpus. Note that the vast majority
computation SPLADE and dual encoders is in encoding the query; reranking BM25 re- trievals would not reduce computation.
E Comparison to Contriever
Table 8 compares the reranking performance of the Contriever system with NAIL. See Section 5.4 for the discussion on this comparison.
aDCGG 10
Contriever
RM25+Contr.
NAIL-exh
NAI5
MS-MAROO
0.407
0.371
0.396
0.377
Avg. BEIR
0.445
0.463
0.432
0.465
Table 8: Comparison of Contriever and NAIL on BEIR and MS-MARCO. We obtain Contriever reranking per- formance by using their released model and ranking the same set of BM25 candidates NAIL. The average BEIR nDCG@ & 10 does not include MS-MARCO.


Reviews:
[{"review_id": "Vuf4L06abO", "reviewer": "Reviewer_oAAC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer oAAC for a thorough review of our work. We are encouraged by the reviewer’s agreement that leveraging LLMs for retrieval, in an efficient manner, is an important problem. Nonetheless, the review provides low scores in excitement and soundness, listing a few reasons for rejection. Below, we touch upon each of the reasons mentioned, and hope that through these clarifications, the reviewer can reassess their evaluation of the work as we think most of the issues raised can be addressed with modest edits to the paper.\n\n\n**Flops computation**: We apologize for lack of clarity with respect to inference-time FLOPS computation we presented in the paper. There are two classes of models for which we present inference-time FLOPS: methods with neural inference and methods with lexical-based scores. For the neural-based models, as noted in the caption of Table 1, we adopted the methodology used in Clark et al 2020 (https://openreview.net/pdf?id=r1xMH1BtvB). Specifically, we used their open-source code at https://github.com/google-research/electra/blob/master/flops_computation.py. Their method covers all Transformers-based models we compare to in the paper.\n\nFor lexical-based models, we compute the cost of scoring query terms, which typically use a constant and small (typically one or two) number of floating point operations per query term (word or token).\n\nWe can make these computations more clear in the final version of the paper.\n\n\n**NAIL-Exhaust**: In Section 5.3 and Table 4 we present a proof of concept experiment to demonstrate the potential for using NAIL as an indexing system rather than using it as a reranker. As we describe in that section, we have not properly implemented an indexing scheme for retrieval, but instead use an *exhaustive* approach to retrieval where for each query, we score all documents in the corpus for that query. As we mention in the section, this exhaustive implementation is impractical. But we hope the results of the experiment are insightful and motivating for future designs of practical and sparse NAIL-based indexes.\n\n**32k^16 = 10^72, Section 3.1**: We can clarify this computation in the final version of the paper. But essentially, the first term comes from the size of the model’s English-based vocabulary, which has 32,768 items (32k). We must explicitly score each of these vocabulary items for each query term (which we limit query length to 16 tokens), so 32,768^16 (32k^16). Which is approximately 10^72 decodes.\n\n**Notation inconsistency**: Thank you for catching inconsistency in Section 3.2, specifically around equations 2 and 3. We will fix this inconsistency.\n\n**Two pre-training tasks**: We have not done an extensive analysis of the mix of pre-training tasks for NAIL. The main reason for this is that pre-training incurs a non-negligible cost which we seeked to minimize. Informally, we compared two pre-training setups: (a) the same setup as described in the Izacard et al 2022 (https://arxiv.org/abs/2112.09118) which applies only independent cropping task, and (b) created a mixture of the two tasks described in the paper (independent cropping and inverse cloze). We noticed that the combination of tasks performed slightly better in the query generation task. We stipulate that, unlike in the analysis done by Izacard et al., the query generation task is different from constructing dense document representations, and the inverse cloze task is more similar to query generation than the independent cropping task.\n\n**Distilled models**: We agree that distilled models are an alternative and interesting way to reduce inference-time costs of language models. Due to space constraints, we limited the number of comparisons we have presented in the paper, but we are open to expanding these comparisons further.\n\nIn Table 1, we do present FLOPS information on a few distilled models for retrieval from the literature, including MiniLM-L6 (derivative of Wang et al 2020 - https://arxiv.org/abs/2002.10957), TinyBERT-L6 (derivative of Chen et al 2020 - https://arxiv.org/abs/2009.07531), and BERT-tiny (derivate of Turc at al 2019 - https://openreview.net/pdf?id=BJg7x1HFvB ).\n\nOut of these, we selected MiniLM-L6 as representative of these models since it performs the best according to the BEIR leaderboard at https://docs.google.com/spreadsheets/d/1L8aACyPaXrL8iEelJLGqlMqXKPX2oSP_R10pZoy77Ns/edit#gid=867044147. Results from this model are presented alongside our main results in Table 2 (MiLM column). Fundamentally, even the smallest of the distilled cross-encoder models (for example, BERT-tiny) is still orders of magnitude more expensive for inference than lexicalized approaches, and as Table 2 shows, provide only a couple of points improvement over our proposed NAIL reranker.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper investigates the important problem of leveraging LLM efficiently in retrieval in the case in which computationally intensive approaches are not feasible at the time of inference. The authors conduct a comprehensive literature review. The authors compare their proposed models with different approaches in neural retrieval i.e. cross-encoder, dual encoder, and sparse retrieval.  The authors discussed the limitation of their proposed model from different aspects. The model's effectiveness is reasonable considering the reduction in FLOPs relative to other neural retrieval methods.", "reasons_to_reject": "The main contribution of the paper is reducing the computational cost at the inference which is measured by FLOPs. However, the authors did not provide any experimentations detail of how they measure FLOPs which makes the reproducibility and confirmation of results challenging. \n\nThe final step of similarity between query representation and document representation is not clear as well as how an index was created using the NAIL-Exhaust has not been explained. Also, the intuition behind why representing documents based on the non-autoregressive token prediction and comparing to the query token representation is not clearly explained. \n\nThe writing and structure of the paper need to be improved: there are instances of undefined parameters and confusing equations (32k^16 = 10^72, Section 3.1) in the paper as well as inconsistency in notations (passages once are referred to as d and later referred to as p in section 3.2)\n\nThe authors need to provide more rationale and discussion on why pre-training on two inverse cloze and in-dependent cropping task is needed and helpful for Nail. \n\nLastly, since the authors propose Nail as a mechanism for reducing computational cost at inference, comparing Nail with approaches using Distilled models can be good comparison to different approaches for reducing inference time computational cost complexity.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper provides a sparse retrieval framework to leverage LLM models to learn a representation for documents at the indexing time with the goal of omitting the high computation that LLM-based retrieval models require at the inference time. Authors train a seq-2-seq model with a modification to the decoder module to generate tokens in a non-auto-regressive approach to represent documents. The model outputs a probability distribution over all vocabulary at each position and the final distribution over the vocabulary is calculated by taking the maximum over all positions. The authors train the model in two steps: 1) Pre-training on inverse cloze and in-dependent cropping task and 2) fine-tuning on the MsMarco passage retrieval task. The authors conduct experiments on MSMarco as well as the BEIR benchmark which consists of several retrieval datasets in different domains. They compare their model with the cross-encoder and dual-encoder approaches that were previously investigated on these datasets. The experiments show NAIL achieves up to 86% of the performance of other transformer-based retrieval approaches (including cross-encoder, dual encoder, and spare retrieval) by reducing the FLOPs from 10^13 in a cross-encoder architecture to 10^4.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oJfiT7XYm7", "reviewer": "Reviewer_Esnj", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer Esnj for their positive review. We agree that this work has surprising results in that predicting vocabulary items (in lieu of generation) can yield such positive results, even when compared to more powerful techniques. The performance of the studied technique is shown to be robust across a range of retrieval tasks, strengthening the curiosity of this finding.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The evaluation results really demonstrate the good performance of the proposed method.\n2. The idea of predicting on vocabulary instead of generating tokens is interesting.", "reasons_to_reject": "N/A", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work introduces a new model called NAIL for sparse, lexicalized retrieval. It adapts a non-autoregressive decoder for document indexing, and predicting scores on vocabulary instead of generating token sequences, which largely reduces the computation cost. Experiments show that NAIL largely reduces the computation cost while capturing up to 86% performance gains of cross-attention model.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JD3vtjwsev", "reviewer": "Reviewer_w6rA", "summary": "", "questions": "A. As shown in Figure 1, the input to the Encoder is prompt+passage. Does the prompt design indeed affect the performance?   \nB. The Authors used multi-hot vectors for query representation, seemingly ignoring the word order information that contributes to semantics. Can you elaborate more on this concern?  \nC. Should the defined contrastive loss in Equation 3 be maximized instead of minimized?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer w6rA for their positive and encouraging comments. We are happy to hear that the reviewer enjoyed the experimental work presented and that the paper was easy to follow. We agree that the use of non-autoregressive decoder for indexing is novel and worthy of more research. We hope the community can benefit from this initial work which we focused primarily on query-time efficiency.\n\nBelow we address the three questions the reviewer has posed:\n\n\n**Question A (prompt design)**: Unfortunately, we have not experimented extensively with prompt design, so we cannot provide good experimental evidence on this matter. Nonetheless, it is our belief that the specific prompt used is much less influential on model behavior when the model is trained on a given task, versus on zero- and few-shot settings where it has been shown that the prompt design is quite important.\n\nSpecifically, in our case, we have added a prompt to the input so that the model can differentiate between tasks: the two pre-training tasks (inverse cloze and independent cropping) and the fine-tuning task (query term prediction). Typically for task differentiation, when the model is trained on many examples, the prompt can be as simple as a marker or simple string to allow the model to differentiate between the tasks.\n\n\n**Question B (word order)**: We briefly touch upon the issue of order agnostic representation in the “Limitations” section of the paper. Given our experience with the model, and specifically looking at many error cases in model ranking, we believe this is currently the main weakness in the proposed lexicalized (token-based) query representation. There are many cases where n-grams, whether denoting an entity name, idiomatic expression, complex predicate, etc., cannot be easily inferred from an unordered bag of tokens (for example: “house”, “door”, “white”, “blue”, “with”).  Nonetheless, we believe that studying an “extreme” case of single token representations without much processing of the query (beyond tokenization) is a compelling design point to study. We hope it inspires others to go beyond our simple query representation to explore other low-cost options.\n\nFor future work, we think it is likely the most interesting way to extend this line of work is to design an efficient representation that can go beyond single tokens. Precisely because it can enable more representational power than unigram representations. A few ideas include:\n\nIt would be interesting to study the gap between representational power of 1-gram vs. n-grams query representations for retrieval and/or ranking. This can be done with a simulated oracle or some other clever experimental design.\nWe also think it could be useful to study efficient ways to match precomputed n-gram representations. One option in this line of work could be to extend the base model vocabulary from 10s of thousands of sentence-pieces to millions of sentence-pieces (e.g., Liang et al 2023 - https://arxiv.org/abs/2301.10472).\nStudy the use of denoising non-autoregressive decoders, where a large number (e.g., 100-1000) of n-grams can be sampled in an efficient manner and used for indexing instead of along with unigram representations.\n\n**Question 3 (Equation 3 max/min)**: Thank you for catching this error, we agree with your comment.  Equation 3 defines a contrastive loss based on the score function described in Equation 1. As described, scores with higher values indicate better compatibility between query and document. As such, minimizing Equation 3 is incorrect, we agree with the reviewer. We will fix Equation 3 to invert the sign, making it closer to the code that trains the model, specifically:\n\n$$\n\\mathcal{L} = - \\langle \\phi_q(q_i), \\phi_d(p^+) \\rangle +  {\\rm log} \\sum_{p' \\in \\mathbf{p}_i} {\\rm exp}(\\langle \\phi_q(q_i), \\phi_d(p')\\rangle)\n$$\n\nSo that minimizing the sign inverted version of Equation 3 is the correct training procedure.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The research question is well-motivated and properly justified. Using a non-autoregressive decoder for document indexing is novel and non-trivial. Thorough experiments and analysis confirm both the effectiveness and efficiency of the proposed retrieval system. The paper is clear-written and easy to follow.  ", "reasons_to_reject": "N/A", "questions_to_authors": "A. As shown in Figure 1, the input to the Encoder is prompt+passage. Does the prompt design indeed affect the performance?   \nB. The Authors used multi-hot vectors for query representation, seemingly ignoring the word order information that contributes to semantics. Can you elaborate more on this concern?  \nC. Should the defined contrastive loss in Equation 3 be maximized instead of minimized?", "paper_topic_and_main_contributions": "The paper introduces a novel retrieval system named NAIL to accelerate the inference time by pushing workload to indexing time. The idea is to precompute the document indexing using a non-autoregressive decoder to score tokens in parallel and only perform tokenization of queries at the retrieval time. Experiments on the BEIR benchmark and MS-MARCO show that NAIL can significantly maintain performance compared to state-of-the-art cross-encoder and dual-encoder systems while being more computationally efficient. It pushes further the progress of sparse retrieval and is of great practical value in modern retrieval systems. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Equation 2:  unmatched parenthesis "}, "extra_text": {}}]

Author rebuttal:

Meta-review: All authors found merits in the submission, especially in the improvement of efficiency in reranking process. Although some reviewers pointed out the problem of failing to show FLOPs improvement explicitly, the authors described these information in the rebuttal. I strongly suggest the authors revise the manuscript per the comments of the reviewers.
Decision: Accept-Main