Semi-Parametric Retrieval via Binary Bag-of-Tokens Index
Authors: Jiawei Zhou
Li Dong
Furu Wei
Lei Chen
Venue: ICLR 2025 Poster (2025)
Keywords: information retrieval
efficient retrieval
retrieval-agumented applications
RAG
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A semi-parametric neural retrieval system that supports both  parametric and non-parametric index

Abstract: Information retrieval has transitioned from standalone systems into essential components across broader applications, with indexing efficiency, cost-effectiveness, and freshness becoming increasingly critical yet often overlooked. In this paper, we introduce SemI-parametric Disentangled Retrieval (SiDR), a bi-encoder retrieval framework that decouples retrieval index from neural parameters to enable efficient, low-cost, and parameter-agnostic indexing for emerging use cases. Specifically, in addition to using embeddings as indexes like existing neural retrieval methods, SiDR supports a non-parametric tokenization index for search, achieving BM25-like indexing complexity with significantly better effectiveness. Our comprehensive evaluation across 16 retrieval benchmarks demonstrates that SiDR outperforms both neural and term-based retrieval baselines under the same indexing workload: (i) When using an embedding-based index, SiDR exceeds the performance of conventional neural retrievers while maintaining similar training complexity; (ii) When using a tokenization-based index, SiDR drastically reduces indexing cost and time, matching the complexity of traditional term-based retrieval, while consistently outperforming BM25 on all in-domain datasets; (iii) Additionally, we introduce a late parametric mechanism that matches BM25 index preparation time while outperforming other neural retrieval baselines in effectiveness.

Published as a conference paper at ICLR 2025
SEMI-PARAMETRIC RETRIEVAL VIA BINARY OF TOKENS INDEX
Jiawei Zhou 1.3 Li Dong' Furu Wei?
Lei Chen
The Hong Kong University of Science and Technology* Microsoft Research² The Hong Kong University of Science and Technology (Guangzhou) ( zhoubu, Leichen @ust .kk (lidongl, fuwei) emicrosoft.com
ABSTRACT
Information retrieval has transitioned from standalone systems into essential com- ponents across broader applications, with indexing efficiency, cost-effectiveness, and freshness becoming increasingly critical yet often overlooked. this paper, we introduce Seml-paramenn parametric Disentangled Retrieval (SIDR), bi-encoder retrieval framework that decouples retrieval index from neural parameters to enable efficient, low-cost, and parameter-agnostie indexing for emerging use cases. Specifically, in addition to using embeddings as indexes like existing neural retrieval methods, SIDR supports non-parametric bag-of-tokens index for search, achieving BM25- like indexing complexity with significantly better effectiveness. Our comprehensive evaluation across 16 retrieval benchmarks demonstrates that SIDR outperforms both neural and term- based retrieval baselines under the same indexing work- load: (i) When using an parametric embedding- -based index. SIDR exceeds the performance of conventional neural retrievers while maintaining similar training complexity; (ii) When using matches the complexity of traditional term-based retrieval BM25, while consis- tently outperforming it on in-domain datasets; (iii) Additionally, we introduce a late parametric mechanism that matches BM25 index preparation time for search while outperforming both BM25 and other neural retrieval baselines in effectiveness. Code is available at attps://gtt github..... com/ cchouuuussiie
panaaaaretticc tokenization-bant based index, SIDR
INTRODUCTION
In recent years, information retrievers has evolved from end-to-end systems to essential components in various applications, including question answering (Kolomiyets & Moens, 2011; Zhu et al., 2021). classification (Long et al., 2022), recommendation (Dong et al., 2020; Manzoor & Jannach, 2022) and dialog systems (Liu al., 2024b). This evolution has notably accelerated with the advent of the retrieval-augmeened generation (RAG) paradigm (Bommasani et al., 2021; Guu et al., 2020; Yu al., 2022; Mialon al., 2023), in which the retrieval component enables large language models (LLMs) to access relevant data from external sources, effectively addressing challenges such as like hallucination (Ji et al., 2023; Zhang al., 2023), obsolescence (Wang al., 2023), and privacy concerns (Huang et al., 2022).
Traditional retrieval systems provide end-to-end search services and build indexes offline, without concern for cost or latency. In contrast, advanced retrieval components integrate with various downstream models, requiring greater flexibility to meet the diverse needs applications. We present several emerging retrieval scenarios where current neural retrievers face limitations, and introduce specific index properties in our proposed framework designed to overcome these challenges.
Scenario 1: Online indexing for RAG applications with real-time knowledge sources. In RAG applications that depend on real-time knowledge source, such as up-to-the-minute internet information (Liu et al., 2023) and user-uploaded content requiring immediate responses (Wang et al., 2024), efficient online indexing essential as it determines the time lag between the availability of data and its application. Effective neural retrieval with efficient online indexing facilitates the rapid assimilation and filtering of real-time datastreams, reducing computational burdens and mitigating hallucinations (Liu et al., 2024a; Shuster et al., 2021) when L.MMs process long contexts.

---

Published as a conference paper at ICLR 2025
Scenario 2: Low-cost index for exploration and deployment. Concerns about data privacy (Huang et al., 2022; Arora et al., 2023) and licensing is- sues (Min et al., 2023) are driving startups and indi- vidual developers towards building their own local retrieval pipelines for RAG applications. In this con- text, it is common to construct temporary retrieval indexes that are frequently modified and rebuilt to cater to different data sources and to optimize for varying chunk size selections, allowing for effective analysis of large datastores (Shao et al., 2024) and the fine-tuning of config infiguraaiions. Given their sensi- tivity to resource constraints, smaller entities often prioritize cost reduction over maximizing effective- ness, making Low-cost Index more important than achieving state-of-the-aa performance.
Index Construction
CPU hour
Tokenizer
Encoder P
Wikipedia (Unindexed)
GB
Bag-of- Token Index
Embedding Indec
Semi semi-parametric Retrieval
Parametric Retrieval
Query
Encoder Q
Figure I: Comparison of storage (2 GB vs. 31 GB) and resource costs (1 CPU hour vs. 30 GPU hours) between two indexes.
Scenario 3: Parameter-agnostic index for co-training retrievers with LLMs. A significant challenge in co-training neural retrievers with LLMs is the index update issue (Asai et al., 2023) caused by in-training retrieval (Guu et al., 2020; Izacard et al., 2022a; Shi et al., 2023). Specifically, during training, neural retriever parameterized by 0 is learned to search information from datastore DD to enhance the LLMs. The retrieval index, denoted as Ep (D), consists of the neural embeddings of the datastore D. As the parameters update
0', the datastore index needs to be rebuilt (D) + Eg (D) to prevent it from becoming stale. This process is computationally expensive and compromises the training objectives. Developing neural retrieval that supports Parameter-agnostic Index could address this issue and streamline the co- co-training pipelines.
To meet these emerging needs, our paper introduces the Sem -aam---mettric Disentangled Retrieval framework (SIDR), which decouples retrieval index from neural parameters to facilitate an efficient, low-cost, and non-pann -arrameettiic indexing setup. Specifically, our framework involves learning paramet- ric term weighting within language model vocabulary space, where aon-aaametric representations can be straightforwardly defined and constructed via tokenization. By aligning these two types of rep- resentations, encoder within the bi-encoder framework can optionally utilize tokenization-base based representations as a shortcut for indexing large data volumes. As a result, SIDR simultancously supports parametric index that utilizes neural embeddings and non-parametric index that employs bag-of-tokens representations. As illustrated in Figure 1, using the parammeetttic index for the Wikipedia corpus drastically reduces the indexing cost and time from 30 GPU hours to just CPU hour and reduces storage size from 3IGB to 2GB. This design offers flexibility in choosing indexes with varying complexity to meet diverse retrieval scenarios and co-training propose.
Our comprehensive evaluations across 16 retrieval benchmarks demonstrate that SIDR outperforms both neural and term-based retrieval basclines under comparable indexing workloads. Specifically, our framework with a non- non-paraeettic index achieves a 10.6% improvement in lop-1 accuracy in-domain compared to BM25, while maintaining indexing efficiency on par with BM25. Additionally, when utilizing parametric index, our framework surpasses neural retrieval methods by 2.7% with similar training complexity. Furthermore, our late parametric approach that retrieves from a non-paramen parmmaraettiic index and re-ranks the results on-the-fly, achieving indexing efficiency comparable to BM25 while maintaining the effectiveness of neural retrieval.
We summarize contributions from two main aspects:
From IR perspective: We introduce a versatile cami-parametric retrieval framework that supports both parametric and non- con-parametric indexes to accommodate diverse downstream scenarios. The parametric index uses neural embeddings for effectiveness, while the non non-parametric relies solely on tokenization for efficiency. We further propose late parametric mechanism maximize the trade-off between retrieval effectiveness and indexing efficiency.
From RAG perspective: Our approach introduces in-training retrieval fixed non-parametric index, which avoids index staleness and eliminates the need for costly index rebuilds within the retriever's training loop. This simplifies the co-training the retrieval system with other models.
2

---

Published as a conference paper at ICLR 2025
2 BACKGROUND
Information retrieval task. Given query q and datastore D, information retrieval (Manning. 2009) aims to identify the most relevant passage p E 'D based on q. This task is typically performed using bi-encoder framework, which employs passages into vector representations. The retrieval process can be formulated as:
independent encoders to embed queries and
argmax flg. D)
argmax (q..p)
VpED
In this equation, p is the retrieved passage, and f is a function measures the relevance between q and p, usually calculated as the inner product of their vector representations.
Notation. We use (-) to denote general neural embedding process, applicable to both dense and sparse retrieval. Specifically, for term-based and sparse lexical retrieval, a -dimenniional representation is used, where each dimension represents the weight of token word within vocabulary V. We denote this embedding function as ViMnde, (.) - RIV,, where the subscript indicates the model architecture, and reflects whether the embedding involves learnable parameters.
Traditional term-based retrieval. Traditional term-based retrieval, such as TF-IDF (Ramos et al., 2003) and BM25 (Robertson et al., 2009), assess relevance based on weighted term overlap, which can be described as:
fBM2S (q. p) (ViM2s((), VBM2s(p)) (WBM25 VBow (g), WBM25 VBow (p))
These methods do not involve learned neural parameters and are therefore categorized as non- parametric (Min al., 2022; Freeman al., 2002). They employ million-scale dimensional bag-of-words (BoW) representation VBow(+) with heuristic statistical metrics determining the term weighting UBM25 for each dimension. Due the efficiency and costttfffctsvesess in constructing the term-based index VBM2s(D), these methods are still widely used in industry applications.
Neural retrieval. Unlike term-based retrieval that is heuristic-driven, neural retrieval (Karpukhin et al., 2020; Zhu et al., 2023) is data-driven and parameterized, tailored to learn on specific datasets and tasks. The relevance assessment is defined as:
fo p) = (Eo(g). Eo(r))
While neural retrievers are effective with ample training data, the construction of the parametric index (D) requires embedding the entire datastore, which introduces significant computational costs and latency that hinder their widespread adoption.
Neural retrievers can further categorized into entangled retrievers (a.k.a. dense retrievers) and disentangled retrievers (a.k.a. sparse lexical retrievers), categories which we detail in Appendix A. Specifically, entangled retrievers utilize latent representations with dimensions such as 768, 1024, or 2048. contrast, disentangled retrievers employ disentangled representations, with dimensions equal to vocabulary sizes, such as 30522 for BERT's vocabulary, where each dimension reflects the importance of each token.
METHODOL.OGY
As an overview, Semi-parametii Disentangled Retrieval (SIDR) disentangled retrieval system that builds the VDR architecture (Zhou et al., 2024), incorporating modifications in the learning objective and utilizing in-training retrieval for negative mining to enhance its effectiveness. At downstream, SIDR supports both embedding- based index and bag-of-tokens index. The primary goal of our work to expand the functionality of the current retrieval system to better accommodate emerging scenarios.
In the following sections, we delve into details representation ($3.1), rational ($3.2), training objectives ($3.3), search pipelines ($3.4), and the application of in-training retrieval techniques ($3.5).
3

---

Published as a conference paper at ICLR 2025
Training
Inference
Align sage query and passage representations during training
Switch
sign
search inference
Disentangled Space [Parametric]
0.80 e-9
dig
0600
project
zlign
project
pooject
sign
prjeett
project
search
project
project
Entangled Space
CB
search
Disentangled Space (Non-parametric]
eriiel
cribed
vrobed
westeed
urrieel
Woberize
Wioriize
Text Data
Query
Passage
Pessage
Pastege
Passing
Pessige
Entangled Retriever
Disentangled Retriever
Semi- parametric
Full Parameeric
Semi-parametric
Disentangled Retriever
Search
Search (Option
(Ours: SIDR)
(Option
Figure 2: Left: Training frameworks of entangled retriever, disentangled retriever and our proposed semi-parametric disentangled retriever SIDR; Right: Different inference pipelines of S1DR.
3.1 PARAMETRIC AND NON- DON-PARAMETRIC REPRESENTATION
Disentangled retrievers typically represent data language model vocabulary space, which can be interpreted of tokens with weights. Our parametric representation, Vo(x),, uses token weights learned by neural encoder, whereas the non-parametric representation, (r), can be viewed as using unweighted tokens generated by tokenizer.
Parametric representation. We inherit the VDR encoder (Zhou et al., 2024), which extends the conventional MI.Ms architecture with three modification: (i) replacing the softmax activation with elulp to map dimensional values from (0, 1) to (0, +o);; (ii) applying max-pont max-pooling to aggregate token representations into global representation; and (iii) employing top-k sparsify (Stopk) to prune the less significant dimensional values to zero. These modifications can be expressed as follows:
f [ +1 if cx
elul elulp(x)
otherwise
Vo(x) Stopk o MaxPool ((Mn)))--eup((())), 44, E x}
Significantly, these modifications aggregate token representations into global one, while preserving the property of assigning larger values to more relevant dimensions.
Non-parametric representation. The non non-parametric Bag-of-tokens (BoT) representation for sequence of tokens x is defined as follows:
f1
if V E otherwise
(Bor(z) = MaxPool o (VBo Vt, æ); (B
(I) can be seen as the result of applying max pooling to the one-hot representations of all tokens in æ, assigning each i-th dimension value or O, depending on whether the 5th token V is present in Compared VBMes(-), (-), VBor(-) is tokenizer-speciffe,, with dimensionality the scale of tens of thousands, and uses binary values that require less storage space, making well-suited for tensorization and efficient GPU computation.
3.2 RATIONALE OF SEMI-PARAMETRIC ALIGNMENT.
In this section, we elaborate on the rationale behind the semi semi- -arammetattiic design. The semi coni-parametric alignment designed be consistent with mask language models (MLMs) (Devlin et al., 2018) pre-training objecitve.
Mask language model pre-training. During pre-training, MLMs are optimized to predict masked tokens by leveraging the context. Specifically, given input sequence of tokens [t, f2...., ., ( (t........ t,,] with t; masked, the MLM uses its prediction head (MLMH) with soft- max function to produce the probability VMLMMI, + softmax (Mask(I.)[o) of the masked token over
4

---

Published as a conference paper at ICLR 2025
vocabulary. The ground truth probability is the one-hot representation of the token & In this paper, we refer to this type of representation as the bag-of-tokene (BoT) representation, denote as VBOT The mask token prediction task can be viewed alignment between the vocabulary distribution of the masked token position with the one-hot representation Vnor(4;) in masked setup:
aliga
MMLMMo (Mask(t,) (t.)
(Bo
(1)
As result, the representation VMLMH, (4, [x) tends to assign large values to the dimension correspond- ing to 4, or that are semantically related to &ę based on the context a.
Semi- paramiaaaaetric alignment. can be expressed as follows:
The alignment between parametric and non non-parametric representation
Stopk MaxPool ( (VMILMI, Vt, E x) align Stopk o MaxPool (VBor(f+), Vt, E x)
The semi-parametric alignment is modeled after the MLM pre-training objective, as detailed in Equation I, and expands by aligning multi-token representations between the query and passage. The consistency between upstream pre-trate vre-training and downstream tuning supports the alignability of these two representations.
3.3 SEMI-PARAMETRUE CONTRASTIVE LEARNING
In a batch containing N instances, each instance consists of query q. positive passage p., and set of of negative passages. Our training objective is based on contrastive learning (Jaiswal et al., 2020), which aims maximize the similarity of positive pairs flg:, p.) for all instances i, while minimize the similarity of all negative pairs, denoted as f ((),P;) for all j o i. The loss function defined as follows:
N
(p(.ac))
L(g. EElog
ENpeR efla.p) log Evee cl()))))
iml
qpopp
plog
This results in a final loss that integrates both parametric and sarametrrm components: Lspura (q. p) L(Vo(g), (p))
Togmi (9. p) ((Vo()). Vn-r(p))/2 L(VBoT(g) Vo(p)/
Linal (4. p) Luua(q. p) + puamppaa p)
The parametric contrastive loss Lp aims to align the parametric representations of q and p. a common objective for retrieval training. The semi-parametricc contrastive loss Lsp ensures interaction between the parammeetticc and parametric representations, which forms the foundation of our model to support BoT index (D).
3.4 SEARCH PIPELINES AND INDEX TYPES
Our framework supports both a parametric embedding- based index and non-parametric tokenization- based index. Below, we discuss search functions and the corresponding index type.
Full parametric search (SiDRrat) utilizes a parametric index (D)), which relies embeddings derived from a neural encoder for the datastore. The relevance defined as:
fe D) = (Ve(g), (D())
This is the common indexing process for neural retrieval systems, which are effecttvee but involve higher costs and longer latency for embedding the entire D to obtain the index VD(D))
Semi-parametrie beta search (SIDR;) leverages non-paramen nor-maramt index VBOT based on BoT representations of the datastore, which are constructed solely by a tokenizer. The relevance is defined as:
fa(g. D) (Va(g), (B))(D))
Beta search applies BoT representations on the index side, eliminating the need for neural embeddings during index processing for large datastores, making it suitable for various applications.
5

---

Published as a conference paper at ICLR 2025
Late parametric with top-m re-rank (SIDRg (m)) is search pipeline that starts search with non-parametric index to retrieve top-m passages, denote as Dm, and then embeds them for re-ranking:
fa(a, D) (V) V
folg. mm) (Vo(g), (D()..))
Late parametric retrieval requires a first-stage retriever to support non-parametric index, followed by a second-stage bi-encoder retriever to re-rank and cache the passage embeddings. From an efficiency and cost perspective, the late parametrie with top-m re-rank only requires embedding at most N₂ x m passages, where N₂ is the number of queries. It starts the search with a non-parametric index, distributing the embedding workload throughout the online search process, achieving a fast retrieval setup. When dealing with exploratory scenarios that have limited queries or extremely large datastores, this approach becomes more efficient, as N₂ x m remains much smaller than (D].
3.5 IN-TRAINING RETRIEVAL FOR NEGATIVE SAMPLING
While semi-paramen panimmaareetticc retrieval offers advantages for in-training retrieval by eliminating the need for re-indexing, also has drawbacks, notably its limited effectiveness due to using non-parametric representations index side. To enhance their performance, we integrate beta search in the training loop to dynamically source hard negative passages, leveraging the strengths of semi comi-parametric design to counterbalance their limitations. Specifically, during the training, SIDR employs beta search to retrieve the top-m passages in real-time using Vo(g) to search on non- con-parametric index (Bor(D) and get the top-m results Dn. Subsequently, each passage in Dm is assessed whether it is negative based on exact matches with the answer strings. For each query, one negative is randomly selected from the identified negatives. This method is exclusively used for the Wikipedia benchmark, which provides answer strings to distinguish between negative and positive passages.
While in-training retrieval increasingly adopted to enhance retrieval training (Zhan et al., 2021; Xiong et al., 2020) and facilitate co-training retrievers with LLMs (Shi et al., 2023), previous approaches have necessitated periodic index refreshment the training loop. In contrast, our approach uniquely leverages a fixed index VBor(D), eliminating the need for re-indexing D. Our analysis shows that incorporating in-training retrieval with our non-pes non-parametric index does not add significant latency; however, slight latency increase occurs due to the string matching process to identify negatives, which discussed in Section 4.3.
EXPERIMENTAL SETUP
4.1 DATASETS
Wiki21m benchmark. Following established benchmark in retrieval literature (Chen et al., 2017; Karpukhin et al., 2020), we train our model on the training splits of Natural Questions (NQ; Kwiatkowski et al., 2019), TriviaQA (TQA; Joshi et al., 2017), and WebQuestions (WQ; Berant et al., 2013) datasets, and evaluated it on their respective test splits. The retrieval corpus used is Wikipedia, which contains over 21 million 100-word passages.
BEIR benchmark. We train our model on MS MARCO passage ranking dataset (Bajaj et al., 2016), which consists of approximately 8.8 million passages with around 500 thousand queries. The performance is assessed both in-domain on MS MARCO and in zero-shot setting across 12 diverse datasets within the BEIR benchmark (Thakur et al., 2021).
4.2 BASELINES
Primary baselines. We primarily compare our model with several established retrieval baselines, selected due to their similar model sizes and training complexities, ensuring a comparable training cost. These include dense retrieval DPR, term-based retrieval BM25, and sparse lexical retrieval models such as SPLADE (Formal et al., 202 uniCOIL (Lin & Ma, 2021), and VDR (Zhou et al., 2024). Specifically, VDR, refer to directly using the VDR models to perform beta search.
Advanced baselines. We also introduce advanced retrieval systems for baselines, including ANCE (Xiong et al., 2020), LexMAE (Shen et al., 2022), SPLADE-v2 (Formal al., 2021a),

---

Published as a conference paper at ICLR 2025
Contriever (Izacard et al., 2021), GTR (Ni al., 2021), E5 (Wang et al., 2022), and Dragon (Lin et al., 2023a). These systems leverage larger foundational models (Ma et al., 2023; Ni et al., 2021), 1), retrieval-oriented pre-training (Fan et al., 2022; Zhou et al., 2022), or knowledge distillation (Formal et al., 2022) to enhance performance. We have categorized them as advanced baselines due to their significantly higher training costs associated with the additional training techniques. Future work may explore their integration with our model to assess potential benefits.
4.3 IMPLEMENTATION DETAILS
Hyperparameters. For the NQ, TQA, and WQ datasets, our model is trained for 80 epochs, utilizing in-training retrieval for negative sampling. For the MS MARCO dataset, the training duration is set to 40 epochs. We utilize a batch size of 128 and an AdamW optimizer (Loshchilov & Hutter, 2018) with learning rate set x 10-5 Our model use a top-k sparsification with & = 768, matching the dimensionality of conventional dense retrieval embeddings. For computational devices, our systems are equipped with NVIDIA A A100 GPUs and Intel Xeon Platinum 8358 CPUs.
Training cost. The training durations for DPR, VDR, and SIDR on the NQ dataset are 5, 8, and hours respectively, with 80 epochs under similar conditions. For SIDR, the training time per epoch without in-training retrieval minutes, which is identical VDR. This duration increases minutes per epoch when incorporating in in-training retrieval. The additional time is primarily due to the string matching process required to identify negative samples from the retrieved top-k passages.
EXPERIMENTS
5.1
MAIN RESULTS
Table I:
Top- 1/5/20 retrieval accuracy on test sets (i.e., percentage of questions for which the answers
is found in the retrieved passages). Bold numbers indicate the best performance within each setting.
NQ
TQA
WQ
top1
top5
top20
topl
top5
top20
topl
top5
top20
Parametric Index
DPR
46.0
68.9
80.2
54.1
71.5
80.0
37.4
59.7
73.2
VDR
68.0
79.9
$2.9
71.3
79.3
37.1
58.7
72.5
SIDR
49.1
69.3
80.7
56.2
73.0
80.5
40.2
61.0
73,2
Non-parametric Index
BM25
22.7
43.6
62.9
48.2
66.4
76.4
19.5
42.6
62.8
VDR3
12.3
30.0
46.8
16.9
31.6
45.9
7.7
22.4
39,2
SIDR,
39.8
62.9
76.3
50.4
70.7
79.5
32.1
54.1
69.8
Late parametric with top-m re re-rank
BM25+DPR (m
32.2
43.6
62.9
54.8
66.4
76.4
28.0
42.6
62.8
BM25+DPR 20)
39.4
55.5
62.9
55.4
71.0
76.4
34.6
53.2
62.8
BM25+DPR (m 100)
44.4
63.6
73.5
56.6
72.3
80.5
39.9
59.2
70.2
SIDR, (m G)
44.9
60.6
76.4
54.9
70.7
79.6
38.0
54.1
69.8
SIDR, (m 20)
49.5
68.2
76.4
56.7
72.9
79.5
39.6
59.5
69.8
SIDR, (m 100)
50.3
70.7
80.6
56.8
73.3
81.3
41.5
62.0
73.5
Wiki2lm benchmark results. As shown in Table 1, when using parametric index, RiDR outperforms DPR and VDR in top-1 retrieval accuracy by 2.6% and 3.8%, respectively. These enable neural retrieval with support for
results demonstrate that although our primary objective non-parametric indexing, our modifications do not diminish effectiveness with an embedding-basse index; in fact, they may even improve it. This enhancement also suggests that current benchmarks may favor lexical relevance. likely due to their construction relying term-based retrieval methods.
When utilizing a non-parametric index, SIDR significantly surpasses VDR, and BM25 top-1 accuracy by 28.5% and 10.6%, respectively. Unlike BM25, which relies on empirically derived heuristic term weights for indexing, SIDR, employs binary values in a significantly smaller vo- cabulary space, facilitating tensorization GPUs. With ample in in-domain training data, SIDR, significantly outperforms BM25, demonstrating the exceptional learnability and generalizability of neural retrievers in this context.
7

---

Published as a conference paper at ICLR 2025
Additionally, late parametric methods improve 31DR, by enabling on-the-fly re-ranking of the top-m passages. Our results show that by re-ranking the top-20 passages, SIDR, (m = 20) matches the performance of SiDRfull, and by extending re-ranking to the top-100 passages, surpasses all primary baselines. Beyond effectiveness, the most significant advantage of late parametric methods their substantial reduction in computational costs. For example, in evaluations on the NQ test split with 3k queries, fuD requires embedding the entire D, which consists of 21 million passages. In contrast, S (m = 100) only needs to embed Ng X m passages, amounting to just 1% of the passages in D. yet achieves superior effectiveness. This underscores the exceptional suitability of late parametric methods for exploration or evaluation scenarios.
BEIR benchmark results. As shown in Table 2, S1DRjul surpasses VDR, DPR, and other primary baselines in the BEIR benchmark when using either a parametric index or non non-parametric index with late parametric techniques, consistent with the findings from the Wiki21 benchmark. However, when relying solely on non-parametric index, SIDR, outperforms VDR, and BM25 on in-domain datasets but falls behind BM25 on most out-of-domaan datasets. We attribute this performance decline to three factors. First, due the lack of answer strings accurately identify negative passages in MS MARCO, we do not implement in-training retrieval during training on MS MARCO, which likely contributes weaker performance. Second, non-pare con-paraettric indexes lack neural parameters, they are more sensitive to shifts in data distribution, which may lead to weaker effectiveness in out-of-domain scenarios. Lastly, many BEIR datasets exhibit a lexical bias due to their construction using BM25, as noted in the BEIR paper (Thakur et al., 2021), which inherently gives BM25 an advantage.
Table 2: Retrieval performance MS MARCO (MRR @ 10) and BEIR benchmark (NDCG @ 10).
Bold numbers indicate the best performance within each setting.
Climate- Climatt-FEVER FEVER
MS MARCO
TREC-COVID
Touchee 2020
Argazána
DBPedia
PEVER
HoqpoQA
NPCopuu
SCIDOCs
SeFFat
RQ4
NQ
Avg.
Advanced Retrieval Baxelnes
LexMAR
48.0
50.0
21.9
42.4
80.0
35.2
71.6
34.7
56.2
15.9
71.7
76.3
29.0
48.7
Splade
43.3
47.9
23.5
43.5
78.6
33.6
68.4
33.4
52.1
15.8
69.3
71.0
27.2
47.0
Contriever
44.6
23.7
41.3
75.8
32.9
63.8
32.8
49.8
16.5
67.7
59.6
23.0
44.3
GTR hase
42.0
51.1
24.1
34.7
66.0
34.9
53.5
30.8
49.5
14.9
60.0
53.9
20.5
41.2
ES-base
43.1
51.4
15.4
41.0
58.2
36.4
63.3
36.1
62.9
19.0
73.1
79.6
28.3
47.1
Dragon
39.3
48.9
22.2
41.7
78.1
35.6
64.8
32,9
53.1
15.4
67.5
74.0
24.9
46.6
ANCE
33.8
41.5
19.8
28.1
66.9
29.5
45.6
23.7
44.6
12.2
50.7
65.4
28.4
38.0
Purametric Index
DPR
30.2
40.8
16.2
30.4
63.8
23.7
45.2
26.1
43.2
10.9
47.4
60.1
22.1
35.8
UniCOIL
32.9
35.5
15.0
30.2
72.3
27.0
64.0
32.5
36.2
13.9
67.4
59.7
25.9
39.4
SPLADE
34.0
43.9
19.9
36.6
73.0
28.7
63.6
31.3
46.9
14.5
62.8
67.3
20.1
42.4
VDR
34.3
48.6
17.6
39.0
74.0
28.8
65.5
33,0
47.2
15.3
67.3
67.8
29.8
44.5
SIDRAa
34,2
53.0
17.9
39.3
71.5
29.8
65.4
33,0
47.7
15,1
66.2
68.0
29.7
44.7
Non-porametric Index
BM25
18.7
31.5
21.3
31.3
75.3
23,6
60.3
32.5
32.9
15.8
66.5
65.6
36.7
41.1
VDR
6.1
14.1
6.1
7.9
28.4
6.4
5.7
23.9
6.8
8.1
54.5
21,9
9.2
16.1
SIDR
19.0
38.6
10.8
20.8
46.5
19.8
49.4
27.9
25.3
11.1
64.2
53.5
23,7
32.5
Late parametric with top-10 re-rank
SIDR [m 10)
26.3
44.0
12.1
24.9
57.3
22.8
55.4
30.2
33.1
12.2
65.5
54.9
25.2
36.5
SIDR [m 20)
29.2
48.0
13.8
30.6
62.3
25.7
58.7
32.3
38.4
13.8
59.1
27.3
39.6
SIDR, 100)
32.9
51.5
16.6
37.8
69.2
29.3
63.3
33.2
44.7
14,8
65.7
65.9
29.0
43.4
5.2 RETRIEVAL LATENCY
We evaluated the latency of various retrieval systems across different stages using NQ test split and Wikipedia corpus, as shown in Table 3. The comparison assumes that both BM25 and SiDR indexes fit entirely into CPU/GPU memory. Further details can be found in Appendix B.
Indexing stage. The indexing stage converts the textual corpus into scarchable format. Both SIDR and BM25 use tokenzation-based index and can complete indexing within 1 hour CPU, much faster than the over 20 hours required on GPUs for embedding- based index. The indexing stage often accounts for large portion of the overall time and cost the retrieval pipeline. Our BoT index is efficient, more cost-effective and benefiting from parallelization, making it a flexible option for practical retrieval-Be based applications.
8

---

Published as a conference paper at ICLR 2025
Table 3: Latency at each stage of the retrieval pipeline. T()): computations of tokenization; Eo(:): computation of neural model forward. t: Computations performed a single GPU; times in parentheses indicate the latency if performed on a single CPU thread
Ep)p) in search stage refers to the passage embedding used for late parametric re-ranking.
Indexing
Search
Model
Total
T(D)
Fg(D)
Eatg)
T(²)
f(g.
Fo(p)
Total
BM25
0.6h
40s (2m)
40s
0.6h
DPR
20.3h
12s (2m)
41ms/ (2m)
12s
20.3h
VDR
23.7h
15s (2m)
130ms/ (30ml)(20m) (20m)
15s
23.7h
f1DR
23.7h
15s (5s)(2m)
300ss (20m)
15s
23.7h
SIDR3
0.5h
(5s)(2m)
30ms (3m)
15s
0.5h
SIDR3 (m = 20)
0.5h
15s (5s)(2m)
30mss
4mf
4m
0.6h
SIDRa (m 100)
0.5h
(5s)(2m
30mss (3m)
20m
20m
0.9h
Search stage. The search stage processes online incoming queries and retrieves relevant items from the indexed data. As shown in the table, , achieves significantly higher efficiency compared to BM25 and performs on par with dense retrieval methods when utilizing GPU resources. This advantage arises because the BoT index (D has fixed dimensionality, enabling tensorization for inner product calculations on the GPU. In contrast, BM25 term-based index VBM25(D) operates with millions of dimensions and relies on an inverted index for efficiency.
ANALYSIS
We assess the impact of proposed components and various influencing factors, as detailed in Table 4.
Impact of proposed components. Our ablation study confirms the significance of each component in our approach. Removing the remi-parametric loss (w/o SP objective) leads to a drop accuracy of 5.3% for SuDR and a substantial 31 31.5% for SIDR,, render- ing beta search non funfffcciinal.. Morcover, excluding in-training retrieved negatives (w/o retrieved neg) re- sults in a decrease in top-1 accuracy: 4.2% for SIDR and 7.5% for SIDR,, These results highlight the effec- tiveness of using beta search for in-training retrieval. Unlike static BM25 negatives, which quickly diminish in effectiveness as the model learns, beta search utilizes parametric queries that evolve with the model, contin- ually ensuring that the negatives are challenging and relevant throughout the training process.
Table 4: Ablation study of SıDRrull and SIDR, on NQ dataset.
top1
top5
top20
Parametric Index
SIDR:a
69.3
w/ retrieved neg (m==))
47.9
68.4
79.6
retrieved neg (m==100)
69.0
w/ retrieved neg (MARCO)
39.7
63.9
77.5
retrieved neg (WIKI Sm))
69.1
wlo retrieved neg
44,9
66.9
78.8
w/o nog
57.4
75.1
wlo SP objoctive
43.8
68.0
79.9
Non-parametric Index
SiDR;
39.8
62.9
76.3
retrieved neg (m=1)
62.3
w/ retrieved nog (mm=100)
37.3
62.4
76.5
retrieved neg (MARCO)
29.5
54.8
70.1
w/ retrieved neg (WIKI 8m)
37.5
62.4
76,3
w/o retrieved neg
32.3
56.0
72.1
w/o neg
24.4
49.1
68.2
Effect of negative sample hardness. We explore how the difficulty of retrieved negatives affects model effectiveness. The parameter m indicates the size of the passage pool from which negatives are identified and then randomly drawn, with lower m values yielding harder negatives. While these harder negatives can improve contrastive learning, they also increase the risk of misclassifying weak positives as negatives. Our results (w/ retrieved neg m=(1,100)) indicate that adjusting m to I or 100, compared to the baseline of 20, degrades the performance. Thus, optimal balance, effectively challenging the model while reducing the likelihood of misclassification.
w/o SP objective
12.3
30,0
46.8
w/ vary lentgh
37.5
61.2
76.1
value of 20 provides the
Effeet of negative sample source. Our results (w/ retrieved neg MARCO) indicate that switching the source of negative samples from the Wikipedia corpus to the MS MARCO with 8.8 million passages leads to notable drop in performance. In parallel experiment (w/ retrieved neg WIKI 8m) using a Wikipedia corpus of the same size, performance remained consistent with our bascline, indicating that corpus size is not the main factor behind the observed decline. Instead, the source of negatives plays a crucial role in performance. The disparity stems from differences in the writing styles and structures unique to each corpus (e.g.. Wikipedia passages typically include short title preceding the text), which cause the model to focus on superficial, corpus-specific features rather than developing deeper understanding of the relevance.

---

Published as a conference paper at ICLR 2025
Impact of text length non-parametric index. Unlike the BM25 term-based index, the BoT index lacks term weighting. meaning longer texts may activate more dimensions, resulting in higher inner product scores. To assess the impact of text length the effectiveness of SiDR segmented the Wikipedia corpus into passages ranging from 50 to 200 words, while maintaining the same overall number of passages. Our results (w/ vary lentgh) show that the top-1 accuracy of s decreased slightly from 39.8% to 37.5%, indicating minimal impact on performance. This slight drop can be explained by the sub-linear growth in unique tokens as text length increases and the high sparsity of the representations, where increasing activations has little impact on relevance.
re-
Comparison of in-training retrieval. We compared in-training retrieval across different systems, as shown in Table 5. Compared to BM25, S1DR, has up to 30x lower latency when using GPU resources for large corpora. In contrast to dense retrieval methods like DPR, 51DR, requires less GPU allocation and uses a fixed index, which eliminates the need for periodic re-indexing during the training loop and ensures that the training objective is not compromised by a stale index. Appendix C.
Table 5: In-training retrieval latency per
batch, with the storage size and GPU mem- ory allocation for corresponding index.
Method
Latency
Storage
GPU
BM25 DPR SIDR,
38
2.3GB
<Ims
31.5GB
31GB
<Ims
2.7GB
10GB
Additional details can be found in
7 RELATED WORK
Disentangled Retrieval Disentangled retrieval, also known sparse lexical retrieval, develops sparse representations for queries and documents within pre-defined vocabulary space, where each dimension reflecting the importance of specific token. These methods have proven effective text matching (Dai & Callan, 2020; Bai et al., 2020; Formal et al., b021b; 2022; Ram et al., 2022) and have been utilized to enhance search efficiency in subsequent studies (Gao et al., 2021a; Shen et al., 2022; Lin al., 2023b; Lin & Lin, 2023). Notably, several works like TILDE (Zhuang & Zuccon, 2021b;a), and SPARTA (Zhao et al., 2021) use bag-of-tokens query representations for efficient online query processing. These methods fall under the category of semi- cemi-parametric retrieval as they employ non-parametric representations on the query side. Complementing these efforts, our work focuses on addressing the challenges associated with the index side, which is inherently more complex due to the greater length and contextual depth of documents. We discuss the taxonomy of neural retrieval in more detail in Appendix A.
In-training Retrieval Retrieving data in the training loop of retrieval models an emerging yet challenging practice that serves several critical purposes. This includes acquiring negative samples for contrastive learning (Zhan et al., 2021; Robinson et al., 2021), sourcing relevant instances for data augmentation (Blattmann al., 2022; Shi al., 2023), and facilitating the training of retrieval- language models (Asai et al., 2023) in an end-to-end manner. However, this process is complicated due to the need for frequent re-indexing of the corpus as the training of the retriever progresses. Recent research has explored strategies like asynchronous index updates (Guu et al., 2020; Xiong et al., 2020; Izacard al., 2022b: Shi et al., 2023) or building temporary indexes on- on-the-fly from the current training batch (Zhong et al., 2022; Min et al., 2022). Our work proposes a semi cami-parmmtrie framework which supports a nor-maramttric index, thereby avoiding these complications and streamlining the in-training retrieval practice.
8 CONCLUSIONS
In this paper, we introduce SIDR, semi-parametric bi-encoder retrieval framework that supports both parametric and non-parametric indexes to address the emerging needs of retrieval-based ap- plications. Unlike traditional neural retrieval methods that rely solely on embeddings as indexes, SIDR additionally incorporates nor-parae bag-of-tokenn index. The flexibility of SIDR makes it particularly well-suited for applications requiring efficient or low-cost indexing and facilitates co-training with fixed index.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Lei Chen's work is partially supported by National Key Research and Development Program of China Grant No. 2023YFF0725100 National Science Foundation of China (NSFC) under Grant No. U22B2060, Guangdong- Hong Kong Technology Innovation Joint Funding Scheme Project No. 2024A0505040012, the Hong Kong RGC GRF Project 16213620, RIF Project R6020-19, AOE Project AoF/E-603/18. theme-based project TRS T41 -61--20// CRF Project C2004-21G, Guangdong Province Science and Technology Plan Project 2023A050503001 Guangzhou municipality big data intelligence key lab, 2023A03J0012, Hong Kong ITC ITF grants MHX/078/21 and PRP/004/22XX, Zhujiang scholar program 2021JC02X 170, Microsoft Research Asia Collaborative Research Grant, joint research lab and 2023 HKUST Shenzhen- Hong Kong Collaborative Innovation Institute Green Sustainability Special Fund, from Shui On Xintiandi and the InnoSpace GBA.
REFERENCES
Simran Arora, Patrick Lewis, Angela Fan, Jacob Kahn, and Christopher Ré. Reasoning over public and private data in eetiieval-based systems. Transactions of the Association for Computational Linguistics, 11:902-921, 2023.
Akari Asai, Sewon Min. Zexuan Zhong, and Danqi Chen. Retrieval-basee language models and applications. In Proceedings of the s Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts), pp. 41-46, 2023.
Yang Bai, Xiaoguang Li, Gang Wang, Chaoliang Zhang, Lifeng Shang, Jun Xu, Zhaowei Wang, Fangshan Wang, and Qun Liu. Sparterm: Learning term-based sparse representation for fast text retrieval. arXiv preprint arXiv:2010.00766, 2020.
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint 1611:0611 2016.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang, Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1533-1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https: http://aaaaaannccac.aooooooooolt ott////-1/1a-
Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Bjöm Ommer. Retrieval- augmented diffusion models. Advances Neural Information Processing Systems, 35:15309- 15324, 2022.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.07255 2021.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes, Reading wikipedia to answer open- domain questions. arXiv preprint arXiv:1704.0005) 2017.
Zhuyun Dai and Jamie Callan. Context-aware term weighting for first stage passage retrieval. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pp. 1533-1536. 2020.
Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800. 2018.
Manqing Dong, Feng Yuan, Lina Yao, Xiwei Xu, and Liming Zhu. Mamo: Memory-augmente
meta-optimization for cold-start recommendation. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 688-697, 2020.
Yixing Fan, Xiaohui Xie, Yinqiong Cai, Jia Chen, Xinyu Ma. Xiangsheng Li, Ruqing Zhang, Jiafeng Guo, et al. Pre-training methods information retrieval. Foundations and Trends@ in Information Retrieval, 16(3):178-311 78,
11

---

Published as a conference paper at ICLR 2025
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. Splade v2: Sparse lexical and expansion model for information retrieval. arXiv preprint arXiv:2109.10083: 2021a.
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. Splade: Sparse lexical and expansion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR 2021b.
Conference on Research and Development in Information Retrieval, pp.
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. From distillation to hard negative sampling: Making sparse neural ir models more effective. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2353-2359, 2022.
William T Freeman, Thouis R Jones, and Egon C Pasztor. Example-based super-resolution. IEEE Computer graphics and Applications, 22(2):56-65 65, 2002.
Luyu Gao, Zhuyun Dai, and Jamie Callan. Coil: Revisit exact lexical match in information retrieval with contextualized inverted list. arXiv preprint arXiv:2 100.022044.7866, 2021a.
Luyu Gao, Zhuyun Dai, Tongfei Chen, Zhen Fan, Benjamin Van Durme, and Jamie Callan. Com- plement lexical retrieval model with semantic residual embeddings. In Advances in Information Retrieval: 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28-April 1, 2021, Proceedings, Part I 43, pp. 146-160. Springer, 2021b.
Kelvin Guu, Kenton Lee, Zora Tung. Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929-3938. PMLR, 2020.
Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. Are large pre-trained language models leaking your personal information? arXiv preprint v.Xi:2205.122688 2022.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint (t.:::2112.091111 2021.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208. 032999, 2022a.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval augmented language models. arXiv preprint 1XXi:22088.03299 2022b.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Technologies, 91):2, 2020.
Ziwei Ji, Naycon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-33 2023.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 1601- 1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/ v1/P17-11477 URL https: //t.a/aaaaattoooooyooooooooo org/P17-1147.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber. Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, Online, November 2020. Association for Computational Linguistics. doi: (0.18653/v1/ 120000001200m.mmmmmmmmm m5.n..50. URL https: / a ant
/tlgg/220..m ammnl-maa i...in..... 550.
Oleksandr Kolomiyets and an information retrieval perspective. Information Sciences, 181424)5412-5444 2011.
Moens. A survey on question answering technology from
12

---

Published as a conference paper at ICLR 2025
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Jurek Leonhardt, Koustav Rudra, Megha Khosla, Abhijit Anand, and Avishek Anand. Efficient neural ranking using forward indexes. In Proceedings of the ACM Web Conference 2022, pp. 266-276, 2022.
Jimmy Lin and Xueguang Ma. few brief notes on deepimpact, coil, and a conceptual framework for information retrieval techniques. arXiv preprint arXiv:2106. 14807, 2021.
Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang. Ronak Pradeep, and Rodrigo Nogueira. Pyserini: python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 2356-2362, 2021.
Sheng-Chied Lin and Jimmy Lin. dense representation framework for lexical and semantic matching. ACM Transactions on Information Systems, 41(4):1-29, 2023.
Sheng-Chin Chigh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. How to train your dragon: Diverse augmentation towards generalizable dense retrieval. arXiv preprint arXiv:2302.07452. 2023a.
Bheng-Chiehl Lin, Minghan Li, and Jimmy Lin. Aggretriever: A simple approach to aggregate textual representations for robust dense passage retrieval. Transactions of the Association for Computational Linguistics, 11:436-455 452, 2023b.
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024a.
Xiao-Yame Yang Liu, Guoxuan Wang, and Daochen Zha. Fingpt: Democratizing internet-scale data for financial large language models. arXiv preprint arXiv:2307.10485 2023.
Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, and Bryan Catan- zaro. Chatqa: Surpassing gpt-4 on conversational qa and rag. arXiv preprint arXiv:2401.1022 2024b.
Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, and Anton van den Hengel. Retrieval augmented classification for long-tail visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6959-6969, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In International Confer- on Learning Representations, 2018.
Xueguang Ma, Liang Wang, Nan Yang. Furu Wei, and Jimmy Lin. Fine-tuning llama for multi-stage text retrieval. arXiv preprint arXiv:2310.08319, 2023.
Christopher Manning. introduction to information retrieval. Cambridge university press, 2009.
Ahtsham Manzoor and Dietmar Jannach. Towards retrieval eetrieval-based conversational recommendation. Information Systems, 102:00 2022.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, et al. Augmented language models: a survey. arXiv preprint SXXiv22002.07842 2023.
Sewon Min, Weijia Shi, Mike Lewis, Xilun Chen, Wen-tau Yih, Hannaneh Hajishirzi, and Luke Zettlemoyer. Nonparametric masked language modeling. arXiv preprint arXiv:22/2.013349 2022.
Sewon Min, Suchin Gururangan, Eric Wallace, Hannanch Hajishirzi, Noah A Smith, and Luke Zettlemoyer. Silo language models: Isolating legal risk in a nonparametric datastore. arXiv preprint ar0iv:2308. 34340.3008.04400 2023.
13

---

Published as a conference paper at ICLR 2025
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Heráández Åbrego, Ji Ma, Vincent Y Zhao, Yi Luan, Keith B Hall, Ming- Wei Chang, et al. Large dual encoders are generalizable retrievers. arXiv preprint arXiv:2112.078888
2021.
Ori Ram, Liat Bezalel, Adi Zicher, Yonatan Belinkov, Jonathan Berant, and Amir Globerson. What are you token about? dense retrieval as distributions over the vocabulary. arXiv preprint arXiv:2212. 10380, 2022.
Juan Ramos et al. Using tf-idf to determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning, volume 242, pp. 29-48. Citeseer, 2003,
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends@ in Information Retrieval, 3(4):333-389, 2009..
Joshua David Robinson, Ching- Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In International Conference on Learning Representations, 2021. URL
https: http:////ppeeeeeeeeeeeeeeeeeeeee net/ / forum? i decccII0OOUUT..
Rulin Shao, Jacqueline He, Akari Asai, Weijia Shi, Tim Dettmers, Sewon Min, Luke Zettlemoyer, and Pang Wei Koh. Scaling retrieval-based language models with trillion-token datastore. arXiv preprint arXiv:2407. 12854,, 2024.
Tao Shen, Xiubo Geng, Chongyang Tao, Can Xu, Xiaolong Huang, Binxing Jiao, Linjun Yang, and Daxin Jiang. Lexmae: Lexicon-bottleneeked pretraining for large-scale retrieval. arXiv preprint arXiv:2208. 14754, 2022.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Sco, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen-tau Yih. Replug: Retrieval-augmenned black-box language models. arXiv preprint arXiv:2301.12652. 2023.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2 2021.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, et al. Survey on factuality in large language models: Knowledge, retrieval and domain--pecificctty arXiv preprint arXiv:2310.07521. 2023.
Lei Wang, Chen Ma, Xucyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang. Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):1-26, 2024.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-superrvee contrastive pre-training. arXiv preprint arXiv:22/2.03553. 2022.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Amold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00808 2020.
Ikuya Yamada, Akari Asai, and Hannanch Hajishirzi. Efficient passage retricval with hashing for open-domain question answering. arXiv preprint arXiv:2106.0088 000820 2021.
Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu. Qingyun Wang. Heng Ji, and Meng Jiang. A
survey of chowledge-enhaneee text generation. ACM Computing Surveys, 14(11s):1338 2022.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. Optimizing dense retrieval model training with hard negatives. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1503-1512, 2021.
14

---

Published as a conference paper at ICLR 2025
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's 's song in the ai ocean: A survey on hallucination in large language models. arXiv preprint arXhv:2309.01219 2023.
Tiancheng Zhao, Xiaopeng Lu, and Kyusong Lee. Sparta: Efficient open-domain question answer- ing via sparse transformer matching retrieval. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 565-575, 2021.
Zexuan Zhong, Tao Lei, and Danqi Chen. Training language models with memory augmentation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 18653/v 00533//1/2222.e// h200//emmmmmmmmi........ URL https: / a antho loyy wrgg/022. 2022. emn p-lmlppmm--- i m.. 382.
Jiawei Zhou, Xiaoguang Li, Lifeng Shang. Lan Luo, Ke Zhan, Enrui Hu, Xinyu Zhang, Hao Jiang. Zhao Cao, Fan Yu, et al. Hlyperiin--nduuee pre- tre-training for passage retrieval in open- open-domain question answering. arXiv preprint arXiv:2203. 201v:2203.06942, 2022.
Jiawei Zhou, Xiaoguang Li, Lifeng Shang. Xin Jiang, Qun Liu, and Lei Chen. Retrieval- disen- tangled representation learning with natural language supervision. In The Twelfth International Conference on Learning Representations, 2024. URL https: http///ppnneeeeeeeeeeeeee net. forrm Id=z1QRR1 Fmq7Y.
Fengbin Zhu, Wenqiang Lei, Chao Wang. Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. Retrieving and reading: A comprehensive survey on open-domain question answering. arXiv 2021.
preprint 1rriv:2101.000
Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji- Rong Wen. Large language models for information retrieval: A survey. arXiv preprint 1XXx::2308.07007 2023.
Shengyao Zhuang and Guido Zuccon. Fast passage re-ranking with contextualized exact term matching and efficient passage expansion. arXiv preprint arXiv:2108. 08511,, 2021a.
Shengyao Zhuang and Guido Zuccon. Tilde: Term independent likelihood model for passage re- ranking. In Proceedings of the 44th International ACM SIGIR Conference Research and Development Information Retrieval, pp. 1483-1492, 2021b.
A TAXONOMY OF NEURAL. RETRIEVAL.
In this section, outline the taxonomy of existing neural retrievers, discussing distinctions such as disentangled versus entangled, dense versus sparse, and parametric versus semi-parametric. This classification aims clarify the concepts discussed throughout our paper.
Entangled vs. Disentangled Retriever differ in the type of representations they use for search:
Entangled retriever typically employs latent representations with dimensions such 512, 768, 1024, or 2048 for inner-product search. These representations can be learned flexibly and effectively without prior knowledge, as they are entangled. An entangled retriever can also sparse latent representations, such as the BPR method (Yamada et al., 2021), which employs a learned hash function on 768-dimensionnl latent vector for efficient searching.
Disentangled retriever, on the other hand, utilizes disentangled representations with much larger dimensionality, spanning from tens of thousands, representing the language model's 's vocabulary, to millions, typical of BM25 vocabulary. Each dimension corresponds to the importance of specific token within the vocabulary. Disentangled retrieval methods rely on disentangled representations prior, which leads to desirable properties such as interpretability. controllability, robustness, and, as discussed in this paper, accelerated indexing. Previous work (Zhou et al., 2024; Formal al., 202 la; Shen et al., 2022) has also suggested that using disentangled representations improves embedding quality, as foundational models are optimized in this disentangled representation space predict mask tokens next tokens, rather than in the entangled representation space. Note
15

---

Published as a conference paper at ICLR 2025
that disentangled retrievers also dense (fully activated) representations. For example, VDR (Zhou et al., 2024) utilizes disentangled representations that can be fully activated for search.
Thus, the classification of entangled vs. disentangled is independent of whether the representation dense or sparse. Generally, entangled retrieval methods tend to use dense representations, while disentangled retrieval methods often sparse ones, but this is not strict requirement.
Dense vs. Sparse Retriever differ based on whether their representations are fully activated or have been sparsified. Typically, entangled representations are fully activated, while disentangled representations are often sparsified to reduce storage size and to facilitate the construction of an inverted index for efficient searching. Consequently, dense retrieval is commonly associated with entangled representations and sparse retrieval with disentangled ones. However, the relationship between entangled/disentanglle and dense/sparse retrieval not rigid. Exceptions exist where entangled representations can be sparsified (Yamada et al., 2021) and disentangled representations can be fully activated (Zhou al., 2024).
Full Parametric vs. Semi-parametriy Retrieval These approaches differ based on the utilization of neural parameters for encoders. Full parametric retrieval systems employ neural parameters for both encoders. In contrast, semi-parametric retrieval systems use one neural encoder alongside one non-par con-parametric encoder, typically involving tokenization-biis based representations. To our knowledge, existing semi-paramiee parametric systems predominantly engage in disentangled retrieval, as they all utilize tokenization- based non-parametric representations that operate on vocabulary space. Notable examples include TILDE (Zhuang & Zuccon, 2021b), TILDE-v2 (Zhuang & Zuccon, 2021a), and SPARTA (Zhao et al., 2021), which implement tokenization- based representations on the query side for efficient online query processing. Our work, SIDR, represents unique approach within this category, offering semi-parametric retrieval that utilizes binary bag-of-tokens representations on the index side for emerging scenarios.
B DETAILS OF LATENCY EVALUATION
During the indexing stage, Eg and T operate with batch size of 32 and maximum text length of 256. In the search stage, the computation of f(g. 'D includes both inner product calculation and sorting of the top-k passages. Queries are processed in batches of 32, with passage embeddings stored in either CPU or GPU memory using half-precision floating-point (FP16) to optimize memory usage. Our analysis excludes the time spent on I/O and data type conversion between CPU and GPU, assuming sufficient processing resources are available.
For BM25, we utilize Pyserini (Lin et al., 2021), library based on a Java implementation developed around Lucene. For neural retrieval, our implementation is in Python, leveraging PyTorch's 's sparse module! for efficient inner product computation, without building an inverted index. Timing measure- ments are performed by running each operation 10 times and reporting the average after excluding maximum and minimum values. To avoid out-of-meneen of-emmorroor issues on our devices, we perform searches in batches on I-milionn document corpus and accumulate the latencies. Note that inverted indexes rely heavily on memory access and integer operations, which are generally inefficient on GPU architectures.
C IN-TRAINING RETRIEVAL DETAILS
Latency (ms)
Storage (GB)
GPU (GB)
Index Density
a=256
0.20
2.8
6.9
We conducted simulation test to evaluate factors af- fecting in-training retrieval, summarized in Table Initially, built binary token index with dimensions of 30k and a sample size of 500m, where each vector consists of 256 dimensional activated. We then varied the density of passage representations by adjusting the activation number from 256 512 and 1024. As the activation number increased, storage and GPU alloca-
a=512
0.21
4.9
23.5
a=1024
0.21
9,0
46.7
Query Batch Size
bs=32
0.20
bs=128
0.21
bs=512
0.24
Table 6: Retrieval latency, index storage size, and GPU allocation for SIDR
htttps::/ppttrrch..rgg//// /ttabl e/ e/sparsaaae wparse.e...... htm1
16

---

Published as a conference paper at ICLR 2025
tion also increased, while latency remained largely unchanged. Additionally, we found query batch size had minimal impact on latency.
D ANALYSIS ON TERM WEIGHTING AND EXPANSION
To systematically compare our method with BM25, we control for vocabulary differences by using BM25 with the same BERT-base- uncased vocabulary ours. This ensures both methods share the same dimensionality for sparse representation, differing only in two key aspects:
Term expansion: BM25 uses only lexical tokens, while SiDR allows for term expansion.
Term weighting: BM25 relies on statistical-bass term weights, whereas SiDR learns contextualized weights.
In this section, we empirically demonstrate how these factors - term expansion and term weighting affect the outcome. Below, we introduce two additional representation forms:
Lexical Parametric Representation (x) This representation activates only the lexical tokens present in the input text æ. leverages learned term weights but does not permit term expansion:
Vi (x) VBOT (z
Binary Parametric Representation Vahnn (1:) This representation activates the number of tokens but assigns uniform weights (set to one), removing learned scalar term weighting. It allows term expansion but does not apply term weights:
Vilnn (x) = Binarize Vo(
where Binarize is function mapping non-zero values to one.
To independently assess the impact of term expansion and term weighting, we propose several variants of SiDRan SiDRiau (w/o weight at doc) and SiDR (w/o expand at doc), which utilize V (p) and (p) on the document side during inference. Additionally, we introduce variants of SiDR;: SiDR, (w/o weight at query) and SiDR, (w/o expand at query), which employ V (4) and (4) on the query side at inference. Furthermore, we propose SiDR full (w/o expand doc, training), which trained with (p) replaced by (p(( ensure consistency between training and inference phases. All these models are compared against BM25 that utilizes the same bert-base- uncased tokenization and vocabulary. This controls for vocabulary differences, isolating the effects of term selection and term weighting. We also include an extreme baseline that uses a bag-of-tokens representation for both the query and the passage, referred as BoT overlap.
Table 7: Ablation study on learned term weighting and term expansion, with results reported as top-1 accuracy on NQ test splits.
Query
Document
Model
Expand
Weight
Expand
Weight
Accuracy
BM25 (bert-basc-uncasec)
21.9
Ablation of SiD on doc side
SiDR
/
49.1
SiDR (wlo weight doe)
33.1
SiDRua (wlo expand doc)
38.9
SiDRaa (wlo expand doc, training)
V
43.1
SiDR
x
39.8
Ablation of SiDR query side
SiDRrea (w/o weight al yuery)
14,5
SiDRaa (w/o expand at query)
34.3
BoT overlap
14.2
When using a parametric index, we conduct an ablation study on SiDRial to assess the impact of removing term weighting and term expansion the document side. From top to bottom,
17

---

Published as a conference paper at ICLR 2025
systematically remove term weight or expansion, simplifying the index of SiDR assess their individual contributions. Our results indicate that removing either term weight or term expansion leads to worse outcomes than removing both (i.e., ,) This is because is specifically designed to align query embeddings with the BoTI' index, rather than these variations. Furthermore, we find that if the training is adjusted accommodate these variations, such as document representations without term expansion, these variations can outperform the BoT index. This demonstrates that neural bi-encoders have great learning potential, with improvement stemming from not only the training itself but also how well the training aligns with inference.
training objective
When using bag-of- index, we assess the impact of term weight and expansion on the query side. Starting from a baseline uses unweighted term overlap, referred to as "BoT overlap". applying BM25's term weights to both queries and documents yields a 7.7% improvement. In comparison, our method's 's learned query term weights achieve 20.15 % improvement, while learned term expansion provides minimal additional gain. Combining term weights and expansion on the query side results in 25.6% improvement, which is SiDR.
In conclusion, when using an embedding index, we demonstrate that both learned term weighting and term expansion on the document side are crucial. Conversely, when using a bag-of-tokent index, the improvements primarily come from term weighting on the query side rather than expansion. Furthermore, ensuring consistency between training and inference representations is essential. A parametric index can underperform compared to non-parametric ones if the representations used during inference do not align with those used during training.
E COST-EIFECCTVVEEES ANALYSIS
Table 8: Additional late- aate-parametrc baselines and cost-effectiveness analysis performed on retrieval task using 3.6k NQ test queries across 21 million Wikipedia corpus. Costs are determined by the number of text chunks (including both queries and passages) that require embedding by neural encoders. Parentheses indicate the ratio of text chunks embedded to the total retrieval corpus.
Performance
Cost
Non-parametric Index BM25
22.7
0
SIDRg
39.8
3.6k (0.01%)
1 Iate-purametic with top-100 rerank
BM25 + VDR
41.6
364k (1.73%)
BM25 + SiDR
44.0
364k (1.73%)
BM25 + Contriever
39.3
364k (1.73%)
BM25 + E5
50.4
364k (1.73%)
SiDR; (m=100)
50.3
364k (1.73%)
SIDR; ; VDR
43.2
367k (1.74%)
SIDR,; ; Contriever
42.9
367k (1.74%)
SIDR; ; ES
57.7
367k (1.74%)
Parametric Index
StDRau
49.1
21m (100.01 %)
Contriever
41.5
21m (100.01 (b)
ES
57.9
21m (100.01 %%)
Late parametric retrieval aims to provide quick-start and low-cost search initialization through non- parametric index, while simultaneously building a parametric index during the search service, eventually transitioning a fully parametric index for searching. To fulfill this requirement, the first- stage utilizes retriever that supports a non-parametric index, while the second-stage retriever can be any parametric bi-encoder. This method can be seen as subset of hybrid retrieval systems (Leonhardt et al., 2022; Gao et al., 2021b), with specific choices constrained to the two stages.
We introduce various combinations of BM25 and SiDR; as the first-stage retriever, paired with more advanced retrievers in the second stage to demonstrate their effectiveness. The results of these combinations are presented in Table 8. Moreover, we assess the cost-effectiveness of these frameworks, particularly in scenarios where data has not been indexed. In such cases, the primary cost arises from the neural model's forward pass for text embedding. Therefore, we measure cost by counting the number of text chunks (both query and passage) that require embedding.
18

---

Published as a conference paper at ICLR 2025
For BM25, no neural embedding is required. While SiDR, employs bag-of-tokens index, waiving the indexing cost, requires embedding 3.6k queries (0.01% % of the corpus) to complete the retrieval task. Despite this, it offers significant performance improvement of 17.1% in accuracy over BM25. For various late parametric baselines, an additional embedding of 100 passages per query is needed approximately 1.7% of the corpus yet this results in further improvements BM25, Conversely, the conventional retrieval pipeline, which requires embedding the entire corpus achieve performance comparable to that of late parametric models with top- 100 reranking. This analysis shows that semi-pan comi-parmettrc models provide more cost-effective solution by balancing retrieval performance with computational efficiency.
Among various late- parammerrettc baselines, SiDR, consistently outperforms BM25 as the first-stage retriever. However, this advantage requires embedding an additional 3.6k queries other models are employed as the second-stage retriever. In exploring second-stage retrievers, we have tested state-of-the-ar models like E5 and Contriever. Our results indicate that stronger retrievers lead to better overall late-parametric performance. Notably, in all our tests, SiDR, combined with any second-stage model consistently outperforms BM25 paired with the same model. Furthermore, when SiDR,, serves the first-stage retriever and re-ranks the top-100 passages, its performance is comparable to, and often exceeds, that of full parametric search with these retrievers.
19


Reviews:
[{"review_id": "oj7YhgxpMd", "reviewer": "Reviewer_PzGy", "summary": "This paper main contribution is a retrieval model where documents are represented by the presence (boolean) of tokens (i.e. a binary bag of tokens). The advantage of such an approach is that the index has to be computed just once (it does not depend on parameters). However, the precision of such a model being low, the authors show that it can be successfully trained with a dense model (used as a second-stage ranker) - and combined with it at inference time. Experiments are conducted on the Wiki21m and BEIR benchmarks and retrieval latency is evaluated - showing that this approach offers a good compromise between effectiveness and efficiency.", "questions": "- what is the purpose of section 3.1? I guess the argument is around the relationship between MLM and vocabulary expansion - see e.g. https://dl.acm.org/doi/10.1145/3634912\n\n- Unless I miss something, there is no justification of the second par of $L_{semi-para}$, i.e. $L(V_{BoT}(q), V_\\theta(p))$ since it is never used. And no ablation shows the importance of this factor\n\n- What is $VDR_{\\beta}$ in table 1? I could not find this model in the VDR paper.\n\n- I do not agree on the difference in efficiency (lines 499-506) between your model and BM25. While it is true that leveraging GPU is useful in your case, BM25 could actually be implemented on GPU, so the reported speedup does not mean much.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The main strength of the paper lies in the static representation of documents. As far as I know, this is the first model that relies on a simple index structure, that of representing the documents as a binary bag-of-token. This allows for potentially fast retrieval (although this can be debated, see weaknesses) of potential candidates that have then to be re-ranked.", "weaknesses": "One of the main weaknesses is related to the number of collisions (i.e. number of documents for one token) increases with the size of the collection. It is unclear how this approach performs when the number of documents, their length, or both, increase. It would be important thus to investigate using the model on a larger collection (e.g. MS-Marco).\n\nAnother point is that the authors state (l. 329) that SOA training \"techniques are orthogonal to the retrieval model and have not been applied in our works.\". It is not clear if this is truly the case here, and no experiments have been conducted to check the potential of e.g. simple techniques like knowledge distillation. \n\nOther things:\n- The authors propose to use the elu1p (p. 4) - it is not clear why softplus has not been used, the function is quite close to this and more \"standard\".\n- The negative mining proposed for the Wikipedia collection is very specific to this collection (l. 292-293). In my opinion, this invalidates the results reported on the Wikipedia collection (table 1 p. 7) and would justify the use of the dismissed SOA training techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DMeI3nJoGP", "reviewer": "Reviewer_MXa4", "summary": "This paper extends the VDR encoder (which also seems to be under review), aligning masked term prediction and bag of tokens representations. The paper shows that hybrid retrieval outperforms either vector based or token based retrieval alone, and the same parametric representation used in vector based retrieval can be \"aligned\" with bag of tokens to perform non-parametric retrieval.", "questions": "BM25 as a token retrieval baseline has stood the test of time, being competitive for three decades. Why do the authors think SiDR_beta handily outperform BM25? What can we learn from this? Could this be because it's not an apples to apples comparison? Are there other underlying factors? \n\nWhy did the authors not consider hybrid retrieval systems a more natural baseline?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well motivated. The problems identified with constructing and refreshing vector based document indices are consistent with prevalent challenges in the industry. While most SOTA retrieval systems, especially those used in the industry, are hybrid, re-using representations across the two forms of retrieval seems novel. There is a good set of baselines, though the most relevant baselines are those that perform hybrid retrieval with vector based and token based approaches. The paper will also benefit from inclusion of more qualitative analysis, especially on how the aligned BoT query representations tackle classical IR problems like polysemy and synonymy.", "weaknesses": "One of the strengths of bag-of-token based retrieval is that it is easily interpretable. The paper misses an opportunity to demonstrate how going from a parametric representation (which is considered semantic retrieval) allows us to tackle the standard problems in IR such as polysemy and synonymy. How do the lack of weights (a la BM25) on the document side affect retrieval? \n\nA nit: since the paper uses a hybrid retrieval system, comparison against either vector retrieval alone is not an apples to apples comparison. For example, in Table 1, this system seems to handily outperform BM25. But BM25 doesn't have the benefit of query side synonyms. While the index is non-parametric, the query is not (it's an \"aligned version\" of the parametric embedding). This doesn't invalidate the results. Rather it provides a channel for further investigation into the kinds of representations that are produced.  \n\nComparison against other hybrid retrieval systems (Gao et al 2021, Kuzi et Al 2020) would make the paper a lot stronger. (These are ones that come to mind, but there may be more). \n\nLuyu Gao, Zhuyun Dai, Tongfei Chen, Zhen Fan, Benjamin Van Durme, and Jamie Callan. Complement lexical retrieval model with semantic residual embeddings. In European Conference on Information Retrieval, 2021. URL https://api.semanticscholar.org/CorpusID:232423090\n\nSaar Kuzi, Mingyang Zhang, Cheng Li, Michael Bendersky, and Marc Najork. Leveraging semantic and lexical matching to improve the recall of document retrieval systems: A hybrid approach, 2020. URL https://arxiv.org/abs/2010.01195.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QONhrnlFsK", "reviewer": "Reviewer_14wh", "summary": "The paper focuses on sparse neural retrieval using a SPLADE-like model. It proposes to train a neural sparse bi-encoder V for both queries Q and documents D in such a way that it works well when computing similarity in the standard bi-encoder fashion, i.e., as the inner product of V(Q) and V(D) as well as through the inner product of V(Q) and the binary bag-of-tokens representation of a document.\n\nThis can be quite useful since it permits using a neural encoder together with non-parametric document representations (that do not required encoding) for effective and efficient retrieval.\n\nInitially I had several concerns regarding the fairness of comparison to BM25 (multi vs single-thread and running BM25 on GPU) as well as to a possibility of re-ranking using a stronger model. However, additional experiments and clarifications resolved the issues. \n\nThere are also some clarity issues (see the weaknesses section), which I think could be resolved or mitigated in the final version. The paper is somewhat hard to read, but it's understandable with some effort.\n\nI would like to emphasize that the final version of authors need to compare against the fast implementation of BM25 (i.e., using PISA or similar) and make sure they make it clear that the comparison is vs single-thread BM25 (this is only for efficiency evaluation, no need to re-run all experiments).\n\nLast, but not least, that BM25 with re-ranking is a strong baseline is definitely worth highlighting and connecting it to prior work, e.g.:\nLeonhardt, Jurek, et al. \"Efficient neural ranking using forward indexes.\" Proceedings of the ACM Web Conference 2022. 2022.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "See the summary for more details.\n* An interesting approach\n* A semi-parameteric index retrieval is easy to carry out on a GPU\n* A substantial evaluation using the BEIR datasets (plus additional QA datasets)\n* Promising results", "weaknesses": "After discussion with authors, I have come to a conviction that the paper is generally solid, but presentation can be improved. \n\nThere are several examples of where the paper is hard to read:\n1. No explanation for the need of VDR (I still didn't quite get your explanations in the rebuttal).\n2. The whole section of revisiting MLM is very confusing. You say \"We provide insights into the consistencies between semi-parametric alignment and masked language\nmodel pre-training\", but this is already a pretty confusing phrase. To begin with you probably meant to say \"connections\". This is a small things, but a lot of such small things here and there complicate reading. Compare, e.g., how this was introduced in the SPLADE paper: We describe in Section 3.1 how the Masked Language Modeling (MLM) head of Pre-trained Language Models can be used to represent tokens in a sequence as vectors in the vocabulary space.\n3. Then the whole section \"PARAMETRIC AND NON-PARAMETRIC REPRESENTATION\" is a bit of a slog to read. You need to explain what you mean by \"parametric\" vs \"non-parameteric\" vs \"semi-parametric\" representations. Moreover, when you talk about parametric representations it's IMHO better talk about **learned* token weights vs token computed using a hand-crafted rule (as in BM25). Don't get me wrong parametric vs non-parametric isn't wrong per se, but the distinction between parametric and non-parametric ML is a bit blurry. Plus, in the context of learned representations it is not frequently used IMHO.\n\nBoT appears almost out of the blue. I think it was only mentioned in the MLM-section. IMHO, it's better to start by explaining that a document/query can be represented by a bag of words or tokens. In some cases these tokens are actual document/query tokens and in some cases they are \"predicted\" by the encoder model. Moreover, tokens can be weighted and non-weighted. Weights can come either from a model or from token-counting (as in BM25). This would have been a much less confusing explanation IMHO. Then, you can explain how weights are coming from an MLM head of a BERT model. This would have been much more logical, IMHO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2YsPNukt90", "reviewer": "Reviewer_YeDM", "summary": "This paper introduces a Semi-parametric Disentangled Retrieval (SiDR) strategy that integrates the advantages of non-parametric and parametric retrieval, achieving competitive performance through the late parametric mechanism while maintaining high efficiency.", "questions": "1. How does SiDR compare with other late parametric methods, and what are the reasons for its success in the late parametric scenario?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed SiDR integrates the advantages of non-parametric and parametric retrieval, achieving competitive performance while maintaining high efficiency.\n2. The paper provides clear explanations of the SiDR method, experiments, and analysis, making it easy to understand.", "weaknesses": "1. In Table 1, the performance of SiDR in parametric retrieval scenario does not surpass ANCE, even with the late parametric mechanism, yet the authors still report it as the best result with the bolded values.\n2. In Table 2, under the non-parametric retrieval scenario, SiDR fails to surpass BM25 in most of datasets.\n3. The late parametric mechanism is not novel, and the experiments should include more comparisons to various methods in this category.\n4. Because of the comparison results with SiDR and ANCE/BM25, the experimental analysis should include an examination of the reasons for SiDR's better performance in the late parametric scenario.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)