Semi-Parametric Retrieval via Binary Bag-of-Tokens Index

## Abstract

Abstract The landscape of information retrieval has broadened from search services to a critical component in various advanced applications, where indexing efficiency, cost-effectiveness, and freshness are increasingly important yet remain less explored.
To address these demands, we introduce S emi-parametric V ocabulary D isentangled R etrieval ( SVDR SVDR \rm{SVDR} roman_SVDR ). SVDR SVDR \rm{SVDR} roman_SVDR is a novel semi-parametric retrieval framework that supports two types of indexes: an embedding-based index for high effectiveness, akin to existing neural retrieval methods; and a binary token index that allows for quick and cost-effective setup, resembling traditional term-based retrieval.
In our evaluation on three open-domain question answering benchmarks with the entire Wikipedia as the retrieval corpus, SVDR SVDR \rm{SVDR} roman_SVDR consistently demonstrates superiority.
It achieves a 3% higher top-1 retrieval accuracy compared to the dense retriever DPR when using an embedding-based index and an 9% higher top-1 accuracy compared to BM25 when using a binary token index.
Specifically, the adoption of a binary token index reduces index preparation time from 30 GPU hours to just 2 CPU hours and storage size from 31 GB to 2 GB, achieving a 90% reduction compared to an embedding-based index. 1 1 1 Our code is available at: https://github.com/jzhoubu/VDR

## 1 Introduction

Over the past decades, the field of information retrieval (IR) has undergone significant evolution.
Initially focused on end-to-end information search services [], IR has expanded to become a fundamental component in a wide array of applications including question answering [, ], classification [], and recommendation [, ].
This expansion has been further accelerated by the advent of large language models (LLMs), which have popularized the paradigm of retrieval-augmented generation (RAG) [, , , ].
RAG improves LLMs by integrating external data sources, thereby addressing problems like hallucination [, ], obsolescence [], and privacy concerns [].

While promising, RAG introduces new challenges for IR, particularly in indexing.
Traditional bi-encoder retrieval follows an index-then-search pipeline where the index is constructed offline and typically remains static.
Historically, this has led IR research to prioritize minimizing online search latency [, ], often resulting in the oversight of the indexing phase itself. However, as the demands and complexity of RAG systems grow, it becomes crucial to reassess these indexing practices.

Existing neural retrieval frameworks largely depend on embedding-based indexes, which are both time-consuming and resource-intensive to build. These limitations become particularly evident in the context of RAG, underscoring the need for more innovative indexing solutions to accommodate varied application needs.
Firstly, there is a critical need for efficient online indexing, as more intelligent agents require access to dynamic or interactive data sources for optimal functionality. For example, role-playing agents [, ] need immediate access to user-uploaded content for effective character simulation and interaction, while financial agents [] require real-time internet information for decision-making. Current neural retrieval frameworks, with their lengthy index construction times, struggle to meet these demands.
Secondly, the requirement for low-resource indexing is becoming increasingly evident as RAG becomes more widespread and practical. Smaller entities, such as startups and individuals, are increasingly integrating online LLM APIs [] with their business data to enhance services. Driven by data privacy [, ] and licensing concerns [], these groups often attempt to establish their own local retrieval systems. However, they face significant challenges due to the substantial GPU resources and disk storage required to build and maintain large-scale indexes.
Lastly, recent research efforts have focused on training retrieval-based models [, ] end-to-end to capture effective retrieval relevance for the entire framework. The training of these models [, , ] necessitates frequent re-indexing, as the embedding-based index is coupled with the retriever’s parameters.
The development of a parameter-agnostic index would greatly benefit the training of retrieval-based models by eliminating the need for frequent re-indexing. Yet, this area remains less explored within neural retrieval frameworks.

Figure: Figure 1: Left: a comparison of storage requirements (2 GB vs. 31 GB) and time/resource costs (2 CPU hours vs. 30 GPU hours) for setting up a semi-parametric retrieval pipeline with a binary token index versus a conventional neural retrieval pipeline with an embedding-based index. Right: decision-making criteria and scenarios guiding the selection between the two indexing types.
Refer to caption: extracted/2405.01924v1/figures/svdr-image1.png

To address these challenges, we propose Semi-parametric Vocabulary Disentangled Retrieval ($\rm{SVDR}$), a sparse lexical retrieval framework that accommodates both an embedding-based index and a binary token index.
As shown in Figure 1, the binary token index is efficiently and cost-effectively constructed through tokenization, operating independently from retrieval parameters.
The core of $\rm{SVDR}$ lies in learning parametric representations within a disentangled representation space where nonparametric representations can be straightforwardly defined and constructed.
Inspired by Zhou et al. [], we use tokenizer-defined vocabulary as this disentangled representation space, and employ a binary bag-of-tokens vector as the nonparametric representation of textual data.
Our framework aligns parametric and nonparametric representations during training, effectively measuring the relevance between these two forms and thereby ensuring the use of both an embedding-based index and a binary token index.
Combining the contextual understanding of neural retrieval with the efficient index construction of term-based methods, $\rm{SVDR}$ provides a balanced solution in terms of effectiveness, efficiency, resource utilization, making it a versatile choice for diverse retrieval-based applications.

Our evaluation covered three established benchmarks for open-domain question answering, using a retrieval corpus of 21 million Wikipedia passages. We compared our method with the dense retriever DPR and the widely-used term-based retriever BM25. We benchmarked our method against the dense retriever DPR and the widely-utilized term-based retriever BM25. Our model significantly surpassed the baseline models in retrieval accuracy across both embedding-based index and binary token index scenarios. Notably, the adoption of binary token index yielded significant savings in both time and resources, cutting down the indexing time (from 30 GPU hours to 2 CPU hour) and storage requirements (from 31 GB to merely 2 GB) by over 90%, in comparison to the embedding-based index.

Our contributions are twofold.
First, we introduce $\rm{SVDR}$, a semi-parametric neural retrieval framework that can support binary token index. This index is efficient and low-cost to construct, making it suitable for a wide range of retrieval-augmented applications and scenarios.
Second, our approach enables in-training retrieval without the need for costly updates to the retrieval index, providing a potential solution to a common challenge in current research on training retrieval-based models.

## 2 Related Work

#### Sparse lexical retrieval.

Previous work [, , , , ] has demonstrated that projecting contextualized token representations into a high-dimensional vocabulary space is an effective way for text matching. Subsequent research [, , ] has leveraged the inherent lexical sparsity to optimize search efficiency through the use of inverted index [, ].
More recent advancements have been in developing sparse-dense hybrid systems [, , ], which aim to strike a balance between efficiency and effectiveness, though their primary focus remains on reducing online search latency.

#### Binary representations.

Binary representations have been extensively studied across various fields, including text retrieval [], semantic matching [, ], image retrieval [, ], and reading comprehension []. Much of this research involves applying semantic hashing [] on neural embeddings to generate binary representations. While these methods significantly reduce the storage requirements of indexes and online search latency, they do not necessarily enhance the indexing efficiency.

#### In-training retrieval.

Retrieving data during the training of retrieval models is an emerging yet challenging practice that serves several critical purposes.
This includes acquiring negative samples for contrastive learning [, ], sourcing relevant instances for data augmentation [, ], and facilitating the training of retrieval-based models [] in an end-to-end manner.
However, this process is complicated due to the need for frequent re-indexing of the corpus as the training of the retriever progresses.
Recent research has explored strategies like asynchronous index updates [, , , ] or building temporary indexes on-the-fly from the current training batch [, ].
Our work introduces a semi-parametric retrieval framework that employs a binary token index, which is independent of the retriever’s parameters, allowing for efficient in-training retrieval without the need to rebuild the index.

## 3 Preliminary: Information Retrieval

Information retrieval [] aims to find the most relevant passage $p$ from a large corpus $\mathcal{D}$ based on the given query $q$.
This is typically achieved by utilizing the bi-encoder framework, which employs two independent encoders to embed the queries and passages into vectors.
The retrieval process can be formulated as follows:

$$ $\displaystyle\hat{p}={\mathrm{argmax}}\,f(q,\mathcal{D})=\underset{\forall p \in\mathcal{D}}{\mathrm{argmax}}\,f(q,p)$ $\displaystyle f(q,p)=\langle E(q),E(p)\rangle$ $$

In this equation, $\hat{p}$ is the retrieved passage, and the function $f$ measures the relevance between $q$ and $p$, typically determined by the dot product of their representations.

### 3.1 Nonparametric Term-based Retrieval

Traditional term-based retrieval methods, such as BM25 [] and TF-IDF [], measure relevance between a query $q$ and a document collection $\mathcal{D}$ based on term overlap.
This relevance is quantified using the inner product of their term-based vectors:

$$ $\displaystyle f(q,\mathcal{D})$ $\displaystyle=\langle T(q),T(\mathcal{D})\rangle$ $$

The tokenization-based function $T:x\rightarrow\mathbb{R}^{|\mathbb{V}|}$ maps input $x$ to a term-based vector, where $|\mathbb{V}|$ is the vocabulary size.
Although these methods employ statistical parameters such as term frequency and inverse document frequency, they are classified as nonparametric because they do not rely on learned parameters. This distinction is consistent with prior studies [, ] and highlights their relatively low computational complexity.
Due to the efficiency and cost-effectiveness in constructing the nonparametric term-based index $T(\mathcal{D})$, these methods facilitate rapid search initiation and are widely used in industry.

### 3.2 Parametric Neural Retrieval

Neural retrieval systems have emerged as a powerful approach in information retrieval [, ].
These methods concentrate on learning embeddings for queries and documents, aiming to maximize the relevance of positive query-document pairs while simultaneously minimizing the relevance of negative pairs through contrastive learning [].
The relevance between $q$ and $p$ is measured by the inner product of their embeddings, which are generated by neural encoders parameterized by $\theta$.
The relevance can be expressed as follows:

$$ $\displaystyle f(q,\mathcal{D})$ $\displaystyle=\langle E_{\theta}(q),E_{\theta}(\mathcal{D})\rangle$ $$

The neural encoder $E_{\theta}:x\rightarrow\mathbb{R}^{|d|}$ embeds input $x$ into a $d$-dimensional latent vector.
While neural retrievers have proven to be highly effective with ample training data, the construction of the embedding-based index $E_{\theta}(\mathcal{D})$ introduces substantial computational resources and latency, which limits their wide adoption.

Figure: Figure 2: Overview of training frameworks (left) and various retrieval pipelines (right) of $\rm{SVDR}$.
Refer to caption: extracted/2405.01924v1/figures/svdr-image2.png

## 4 Methodology

In this section, we introduce $\rm{SVDR}$, a neural bi-encoder that supports the use of both parametric embedding-based index and nonparametric binary token index.
During training, $\rm{SVDR}$ learns parametric term weighting within the vocabulary space and aligns them with nonparametric binary token representations through semi-parametric contrastive learning. At inference, $\rm{SVDR}$ can utilize either neural embeddings or binary token representations to construct the retrieval index.

Our technical contributions include the introduction of semi-parametric training objective (§4.2), in-training retrieval for negative sampling (§4.3), and the semi-parametric beta search functionality (§4.4).

### 4.1 Parametric and Nonparametric Representation

Our approach involves two types of representations: parametric representations derived from a neural encoder, symbolized as $E_{\theta}(x)$, and non-parametric representations obtained through tokenization, denoted by $T(x)$.

#### Parametric representation.

We utilize the VDR encoder [], which leverages pre-trained masked language models (MLMs) [] to generate sparse lexical representations. This process begins by encoding the textual input $x$ into a sequence of token-wise representations using pre-trained MLMs, expressed as $E_{MLM}:x\rightarrow\mathbb{R}^{|\mathbb{V}|\times L}$, where $L$ represents the length of $x$. These token representations undergo processing through an $elu1p$ activation function:

$$ $elu1p(x)=\begin{cases}x+1&\text{if $x>=0$}\\ e^{x}&\text{otherwise}\end{cases}$ (1) $$

and max pooling, aggregating them into a non-negative representation of dimension $|\mathbb{V}|$. The final step involves top-$k$ sparsification, activating only the top $k$ largest dimensions within the representation. As a result, the embedding process is formalized as $E_{\theta}:x\rightarrow\mathbb{R}^{|\mathbb{V}|}$.

#### Nonparametric representation.

We start by tokenizing input text $x$ into a set of tokens, denoted as $\rm{Token}(x)$. Then, we use a binary bag-of-tokens vector for nonparametric representation, defined as follows:

$$ $\displaystyle T(x)[i]=\begin{cases}1&\text{if $V[i]\in$ $\rm{Token}(x)$}\\ 0&\text{otherwise}\end{cases}$ (2) $$

Here, $T(x)$ is the nonparametric binary token representation of $x$. The $i$-th dimension in $T(x)$ is assigned a value of 1 or 0 based on whether the corresponding token $V[i]$ is present in $\text{Token}(x)$.

### 4.2 Semi-parametric Objective

In a batch containing $N$ instances, each instance consists of a query $q_{i}$, a positive passage $p_{i}$, and a set of of negative passages.
Our training objective is based on contrastive learning [], which aims to maximize the similarity of positive pairs $f(q_{i},p_{i})$ for all instances $i$, while minimize the similarity of all negative pairs, denoted as $f(q_{i},p_{j})$ for all $j\neq i$.
The loss function is defined as follows:

$$ $\scriptsize\begin{split}L(q,p)=&-\sum_{i=1}^{N}(\log{\underbrace{\frac{e^{{f(q _{i},p_{i})}}}{\sum_{\forall p\in B}e^{f(q_{i},p)}}}_{\text{q-to-p}}}+\log{ \underbrace{\frac{e^{f(p_{i},q_{i})}}{\sum_{\forall q\in B}e^{f(p_{i},q)}}}_{ \text{p-to-q}}})\end{split}$ $$

This results in a final loss that integrates both parametric and semi-parametric components:

$$ $\scriptsize\begin{split}L_{p}(q,p)&=L(E_{\theta}(q),E_{\theta}(p))\\ L_{sp}(q,p)&=L(E_{\theta}(q),T(p))/2+L(T(q),E_{\theta}(p))/2\\ L_{final}(q,p)&=L_{p}(q,p)+L_{sp}(q,p)\end{split}$ $$

The parametric contrastive loss $L_{p}$ aims to align the parametric representations of $q$ and $p$, a common objective for retrieval training.
On the other hand, the semi-parametric contrastive loss $L_{sp}$ ensures interaction between the non-parametric and parametric representations, which forms the foundation of our model to support a binary token index.

### 4.3 In-training Negative Retrieval

During training, $\rm{SVDR}$ dynamically search hard negative passages from the extensive Wikipedia corpus in real time via a binary token index, differing from traditional methods that rely on static pre-mined negatives. Crucially, this index $T(\mathcal{D})$ is independent to retriever parameter $\theta$, thereby eliminating the need for re-indexing during training. In practice, for each query, $\rm{SVDR}$ retrieves the top-$m$ passages and determines whether they are negative based on exact matches with answer strings, following methodologies from prior works [, ]. For each instance, one negative is randomly selected from the set of identified negatives. The parameter $m$ is adjustable and serves to indirectly control the difficulty level of the negatives used in training.

### 4.4 Search Functions

Our approach supports diverse search functions, each tailored to optimize the balance between effectiveness and efficiency at different stages of the retrieval pipeline.

Full parametric search ($\rm{SVDR_{full}}$) utilizes embedded queries and an embedding-based index: $f(q,\mathcal{D})=\langle E_{\theta}(q),E_{\theta}(\mathcal{D})\rangle$, akin to standard bi-encoder retrieval systems.
This approach maximizes effectiveness but incurs substantial initialization time due to delays in building the embedding-based index.

Alpha search ($\rm{SVDR_{\alpha}}$) utilizes binary token query representations alongside an embedding-based index: $f_{\alpha}(q,\mathcal{D})=\langle T(q),E_{\theta}(\mathcal{D})\rangle$.
This approach eliminates the need for query embedding, significantly optimizing the efficiency in search stage.

Beta search ($\rm{SVDR_{\beta}}$) uses embedded queries and a binary token index: $f_{\beta}(q,\mathcal{D})=\langle E_{\theta}(q),T(\mathcal{D})\rangle$.
This approach eliminates the need to embed the entire corpus prior to search, thus minimizing the latency in indexing stage.

Beta search with re-rank ($\rm{SVDR_{\beta}}$ (m)) uses a binary token index: $f_{\beta}(q,\mathcal{D})=\langle E_{\theta}(q),T(\mathcal{D})\rangle$ to retrieve top-$m$ passages, noted as $\mathcal{D}_{m}$, which are then embedded and re-ranked on-the-fly: $f(q,\mathcal{D}_{m})=\langle E_{\theta}(q),E_{\theta}(\mathcal{D}_{m})\rangle$.
This approach allows for simultaneous construction of the parametric index and execution of the search.

## 5 Experimental Setup

### 5.1 Datasets

We employ the same three QA datasets, along with the training/dev/testing splitting method and Wikipedia retrieval corpus as outlined in previous work [, ].

Natural Questions (NQ) [] is a widely recognized QA dataset comprised of real queries sourced from Google Search, with the answers annotated from Wikipedia content.

TriviaQA (TQA) [] contains a collection of question-answer pairs scraped from trivia websites, offering a diverse range of topics.

WebQuestions (WQ) [] consists of questions generated via the Google Suggest API with entity-level answers derived from Freebase.

Wikipedia Corpus is used as the retrieval corpus for the aforementioned three datasets.
It comprises over 21 million passages, with each passage being a 100-word non-joint segment sourced from Wikipedia.

### 5.2 Implementation Details

All of our models undergo training on the respective training sets for 80 epochs, employing a batch size of 128 and an AdamW optimizer [] with a learning rate of $2\times 10^{-5}$.
For our parametric representations, we maintain top-$k$ sparsification with $k=768$, which aligns with the dimension of DPR for fair comparison.
In the case of in-training retrieval, we initially use the original BM25 negatives for the first 50% of the training progress to ensure learning relevance for the downstream task.
Subsequently, we employ $\beta$ search on the entire Wikipedia corpus to dynamically retrieve adversarial negatives during training.
Our devices are equipped with NVIDIA A800 and A100 GPUs and Intel Xeon Platinum 8358 CPUs.

### 5.3 Baselines

We benchmark $\rm{SVDR}$ against DPR for neural retrieval, which employs an embedding-based index, and against BM25 for term-based retrieval, which utilizes a term-based index. We chose to compare with DPR due to its widespread usage and its close alignment with our approach. Both DPR and our method utilize the same pre-trained MLMs and maintain an identical parameter count, ensuring the validity of the comparison.
On the other hand, BM25 represents the benchmark for term-based retrieval.
Both our method and BM25 leverage tokenization for efficient index construction, but BM25 differs by using heuristic-based term weighting, whereas our approach utilizes binary values. We classify both as tokenization-based indexes.

## 6 Experiments

**Table 1: Top-k retrieval accuracy on test sets (i.e., the percentage of questions for which the answer is found in the top-k retrieved passages); $E_{\theta}(q)$: requires neural embedding for query representation; $m$: number of retrieved passages been further re-ranked.**
|  |  | NQ | TQA | WQ |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | $E_{\theta}(q)$ | top1 | top5 | top20 | top1 | top5 | top20 | top1 | top5 | top20 |
|  |  | Embedding-based Index |  |  |  |  |  |  |  |  |
| $\rm{DPR}$ | ✔ | 46.0 | 68.9 | 80.2 | 54.1 | 71.5 | 80.0 | 37.4 | 59.7 | 73.2 |
| $\rm{VDR}_{\alpha}$ | ✘ | 44.0 | 67.9 | 79.7 | 52.4 | 71.2 | 79.5 | 35.5 | 59.0 | 72.1 |
| $\rm{SVDR_{\alpha}}$ | ✘ | 43.6 | 66.1 | 79.0 | 52.1 | 70.4 | 79.2 | 33.7 | 55.3 | 68.9 |
| $\rm{VDR}$ | ✔ | 43.8 | 68.0 | 79.9 | 52.9 | 71.3 | 79.3 | 37.1 | 58.7 | 72.5 |
| $\rm{SVDR_{full}}$ | ✔ | 49.1 | 69.3 | 80.7 | 56.2 | 73.0 | 80.5 | 40.2 | 61.0 | 73.2 |
|  |  | Tokenization-based Index |  |  |  |  |  |  |  |  |
| BM25 | ✘ | 22.7 | 43.6 | 62.9 | 48.2 | 66.4 | 76.4 | 19.5 | 42.6 | 62.8 |
| $\rm{VDR}_{\beta}$ | ✔ | 12.3 | 30.0 | 46.8 | 16.9 | 31.6 | 45.9 | 7.7 | 22.4 | 39.2 |
| $\rm{SVDR_{\beta}}$ | ✔ | 35.5 | 60.6 | 76.4 | 50.4 | 70.7 | 79.5 | 32.1 | 54.1 | 69.8 |
| $\rm{BM25+DPR~{}(m=5)}$ | ✔ | 32.2 | 43.6 | 62.9 | 54.8 | 66.4 | 76.4 | 28.0 | 42.6 | 62.8 |
| $\rm{BM25+DPR~{}(m=20)}$ | ✔ | 39.4 | 55.5 | 62.9 | 55.4 | 71.0 | 76.4 | 34.6 | 53.2 | 62.8 |
| $\rm{BM25+DPR~{}(m=100)}$ | ✔ | 44.4 | 63.6 | 73.5 | 56.6 | 72.3 | 80.5 | 39.9 | 59.2 | 70.2 |
| $\rm{SVDR_{\beta}}$ $(m=5)$ | ✔ | 44.9 | 60.6 | 76.4 | 54.9 | 70.7 | 79.6 | 38.0 | 54.1 | 69.8 |
| $\rm{SVDR_{\beta}}$ $(m=20)$ | ✔ | 49.5 | 68.2 | 76.4 | 56.7 | 72.9 | 79.5 | 39.6 | 59.5 | 69.8 |
| $\rm{SVDR_{\beta}}$ $(m=100)$ | ✔ | 50.3 | 70.7 | 80.6 | 56.8 | 73.3 | 81.3 | 41.5 | 62.0 | 73.5 |

### 6.1 Effectiveness

Table 1 presents the retrieval accuracy of various baseline models across three QA datasets, comparing performances using an embedding-based index and a tokenization-based index separately.

#### Embedding-based index.

As illustrated in the table, $\rm{SVDR_{full}}$ demonstrates notable enhancements in top-1 retrieval accuracy, outperforming $\rm{DPR}$ and $\rm{VDR}$ by 2.6% and 3.8%, respectively.
These improvements indicate that our method not only extends the functionality but also strengthens its effctiveness in standard parametric search.
For semi-parametric $\alpha$ search, $\rm{SVDR_{\alpha}}$ experiences a performance drop of approximately 5.3%, whereas $\rm{VDR}_{\alpha}$ exhibits only a minor decrease.
This difference in performance is attributed to the semi-parametric approach, enabling interactions between parametric queries and non-parametric passages and vice versa. Conversely, $\rm{VDR}$ is tailored for one-directional interactions from non-parametric queries to parametric passages, explaining its improved performance in $\alpha$ search but also highlighting its limitations in standard and $\beta$ search contexts.

#### Tokenization-based index.

In scenarios utilizing tokenization-based indexes, $\rm{SVDR_{\beta}}$ markedly surpasses $\rm{VDR}_{\beta}$ and BM25 in top-1 accuracy by 27.0% and 9.2%, respectively.
This superiority of $\rm{SVDR_{\beta}}$ over $\rm{VDR}_{\beta}$ arises from the semi-parametric training objective, which enables the interaction between parametric queries and non-parametric passages, making it compatible with the binary token index.
Moreover, despite BM25 relies on statistical parameters and heuristic rules developed through extensive empirical research, it falls short of the performance of $\rm{SVDR_{\beta}}$.
These findings underscore the adaptability and potential of neural retrieval approaches.

Furthermore, our findings indicate that using parametric representations to re-rank the top-$m$ retrieved passages leads to a significant performance improvement.
Specifically, re-ranking the top-5/20/100 passages retrieved by $\rm{SVDR_{\beta}}$ achieves over 90% of the performance of $\rm{SVDR_{full}}$ across top-1/5/20 accuracy metrics, respectively. Remarkably, re-ranking the top-100 passages can recover more than 98% of the effectiveness of parametric search, and in some cases, even surpass its performance.
In addition, we demonstrate that $\rm{SVDR_{\beta}}$ with re-ranking consistently outperforms the combination of using BM25 for search followed by $\rm{DPR}$ re-ranking.
This integration enables rapid retrieval startup and on-the-fly construction of an embedding-based index.

### 6.2 Retrieval Latency

We evaluated the latency of various retrieval stages across different retrieval systems, with results summarized in Table 2.
The retrieval pipeline typically comprises two main stages: indexing and search.
The indexing stage establishes the setup cost and duration necessary for the retrieval system to execute searches. In this stage, a substantial corpus of documents is processed and converted into a format that can be readily searched.
Subsequently, the search stage handles incoming queries and retrieves relevant items from the indexed data.
Further details on our evaluation methodology can be found in Appendix B.

**Table 2: Computational time of each stage within the retrieval pipeline, measured using queries from NQ test set and Wikipedia corpus. ${{\dagger}}$: computation on GPU. $E_{\theta}(p)$: embedding passages for re-ranking.**
| Model | Indexing | Search | Total |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\rm{T(\mathcal{D})}$ | $\rm{E_{\theta}(\mathcal{D})}$ | $\rm{E_{\theta}(q)}$ | $\rm{T(q)}$ | $\rm{f(q,\mathcal{D})}$ | $\rm{E_{\theta}(p)}$ | Total |  |  |
| $\rm{BM25}$ | 0.8h | / | / | 2m | / | 2m | 0.8h |  |
| $\rm{DPR}$ | / | 20.3h^† | 12s^†(2m) | / | 41ms^†(2m) | / | 12s | 20.3h |
| $\rm{VDR}$ | / | 23.7h^† | 15s^†(2m) | / | 130ms^†(20m) | / | 15s | 23.7h |
| $\rm{SVDR_{full}}$ | / | 23.7h^† | 15s^†(2m) | / | 130ms^†(20m) | / | 15s | 23.7h |
| $\rm{SVDR_{\alpha}}$ | / | 23.7h^† | / | 0.4s | 130ms^†(20m) | / | 0.5s | 23.7h |
| $\rm{SVDR_{\beta}}$ | 1.1h | / | 15s^†(2m) | / | 30ms^† (3m) | / | 15s | 1.1h |
| $\rm{SVDR_{\beta}}$ $(k=20)$ | 1.1h | / | 15s^†(2m) | / | 30ms^†(3m) | 4m^† | 4m | 1.2h |
| $\rm{SVDR_{\beta}}$ $(k=100)$ | 1.1h | / | 15s^†(2m) | / | 30ms^†(3m) | 20m^† | 20m | 1.4h |

#### Latency of indexing.

In the indexing stage, both $\rm{SVDR_{\beta}}$ and BM25 construct their indexes through tokenization, resulting in relatively short indexing periods of approximately 1 hour on a CPU. This is notably faster compared to the 20 GPU hours needed to construct an embedding-based index for neural retrieval systems.
Specifically, $\rm{VDR}$ and $\rm{SVDR_{full}}$ take 16% more time for indexing than $\rm{DPR}$ as they need to project the hidden state to a higher-dimensional vocabulary space.
Furthermore, since the indexing process benefits greatly from parallelization, the utilization of CPU resources by BM25 and $\rm{SVDR_{\beta}}$ makes them more cost-effective to scale.
The importance of efficient indexing cannot be overstated, as it determines the initial setup time for the retrieval pipeline.
This aspect is frequently overlooked in other studies, which assume that the retrieval index can be constructed offline in advance. Despite being a one-time computation, indexing represents a substantial portion of the time and cost involved in the entire retrieval pipeline.

#### Latency of search.

In the search stage, latency can vary depending on how the query is represented and interacts with passage representations.
For standard neural retrieval methods like $\rm{DPR}$, $\rm{VDR}$, and $\rm{SVDR_{full}}$, latency arises from the time taken to embed the query and perform inner product computations between the embeddings. Utilizing GPUs for these processes can significantly reduce the search latency [].
For semi-parametric search, $\rm{SVDR_{\alpha}}$ eliminates the need for query embedding by employing non-parametric queries.
On the other hand, $\rm{SVDR_{\beta}}$ utilizes a binary token index that can be loaded with manageable GPU memory, obviating the need for batch inner product computations, which leads to a substantial reduction in computational time, compared to $\rm{VDR}$.

## 7 Analysis

### 7.1 Ablation Study

Our ablation study, detailed in Table 3, assesses the impact of removing various components from our method and the retrieval performance under different configurations for retrieving negatives. Details can be found in Appendix C.

#### Impact of our proposed components.

Our ablation confirms the significance of each component in our approach.
Excluding in-training retrieved negatives and relying solely on official BM25 negatives results in a decrease in top-1 accuracy: 4.2% for $\rm{SVDR_{full}}$ and 7.5% for $\rm{SVDR_{\beta}}$.
Furthermore, removing the semi-parametric loss leads to a drop in accuracy of 5.3% for $\rm{SVDR_{full}}$ and a substantial 31.5% for $\rm{SVDR_{\beta}}$, rendering it non-functional.

#### Effect of negative sample hardness.

We explore how the difficulty of retrieved negatives affects model performance. The parameter $m$ indicates the size of the retrieved passage pool for negative identification. Lower values of $m$ lead to the selection of more challenging negatives, potentially improving contrastive learning but increasing the risk of incorrectly misclassifying weak positive passages as negatives.
The reliability depends on the ability to accurately distinguish negatives from weak positives, a task made challenging by ambiguous or missing answers in the dataset. Our findings show that adjusting $m$ to 1 or 100 from the baseline of 20 negatively impacts the parametric search performance. Consequently, an $m$ value of 20 is deemed most effective for maintaining a balance between selecting challenging negatives and minimizing the risk of misclassification.

#### Effect of negative sample source.

Our results indicate that switching the source of negative samples from the Wikipedia corpus to the MS MARCO [], which comprises 8.8 million passages, leads to a notable drop in performance.
A parallel experiment with a same sized Wikipedia corpus yielded performance levels akin to our baseline, suggesting that corpus size alone does not drive the observed performance decline.
Instead, the source of negatives plays a critical role in the performance.
The observed disparity is primarily attributed to the variances in styles and patterns unique to each corpus. This variation induces a superficial learning bias in the model, diverting its focus towards identifying positives and negatives based on corpus-specific nuances rather than on a substantive understanding of the content.

#### Impact of text length.

Since the binary token index lacks term weighting like BM25, longer texts usually possess more activated dimensions, potentially leading to higher inner product scores.
Thus, we examined how varying text lengths affect the performance of $\rm{SVDR_{\beta}}$ during inference.
We re-segmented the Wikipedia corpus into passages ranging from 50 to 200 words while maintaining the same overall number of passages.
Our results show that $\rm{SVDR_{\beta}}$ experiences a performance decrease with these variable-length passages, with top-1 accuracy dropping from 39.8% to 37.5%, which is not significant compared to other ablation tests.
This suggests that text length does not substantially affect performance. This observation can be attributed to two main factors. Firstly, as text length increases linearly, the number of unique tokens within a text grows at a sub-linear rate. Secondly, given the high sparsity of these representations (0.3% activation), even doubling the activation randomly has only a slight chance of enhancing relevance.

Figure: Figure 3: Ablation study of $\rm{SVDR_{full}}$ and $\rm{SVDR_{\beta}}$ on NQ dataset.

### 7.2 In-training Retrieval

We compare the feasibility of different methods for in-training retrieval during training, as detailed in Table 3.
When it comes to large corpus, the latency of BM25 proves inferior to neural retrieval techniques leveraging GPU resources.
Our results show that $\rm{SVDR_{\beta}}$ is more efficient than BM25 and requires less GPU allocation and index storage compared to conventional dense retrieval, while maintaining efficiency. Notably, the binary token index do not need updates alongside retriever training.
We further conducted a simulation test for in-training retrieval. Our results, detailed in Appendix D, indicate that the density of passage representation exhibits a monotonically increasing effect on index storage size and GPU allocation size but has minimal impact on latency. Moreover, the batch size of queries also has little effect on latency.

Our approach stands out by offering two key advantages that streamline the in-training retrieval.
Firstly, the binary token index exhibits substantial sparsity, with an observed rate of approximately 0.2% density for the Wikipedia corpus. This notably reduces both storage requirements and GPU resource demands during training.
Secondly, while the need to update the large corpus index poses a considerable challenge in training retrieval-augmented models [, ], our semi-parametric approach eliminates the necessity for retrieval index updates during training.

**Table 3: In-training retrieval latency for a batch, and the storage size and GPU memory allocation for the corresponding index.**
| Method | Latency | Storage | GPU |
| --- | --- | --- | --- |
| BM25 | 3s | 2.3GB | / |
| DPR | <1ms | 31.5GB | 31GB |
| $\rm{SVDR_{\beta}}$ | <1ms | 2.7GB | 10GB |

## 8 Conclusions

In this paper, we introduces $\rm{SVDR}$, a semi-parametric bi-encoder retrieval framework.
It enables swift retrieval setup when employing a binary token index, and preserves effectiveness and adaptability for standard retrieval pipeline when utilizing an embedding-based index. Our approach illustrates that with a well-designed training objective, existing lexical retrieval can seamlessly adapt to the utilization of a binary token index.
This versatility allows our framework to optimize the effectiveness-efficiency trade-off for various retrieval scenarios, potentially shedding light on emerging retrieval-augmented applications.

## Limitations

#### Multi-modality.

While $\rm{VDR}$ [] has proven effective in both text-to-text and cross-modal retrieval settings, our work builds upon it with a focus on semi-parametric objectives. However, this design limitation prevents our framework from extending to cross-modal scenarios. This limitation arises from our reliance on binary token representation for both query and passage side, which is inherent to textual data only.

#### Vocabulary.

Our lexical retrieval harnesses the effectiveness of masked language model (MLMs), with the vocabulary predefined by the tokenizer and the MLM itself. While our work utilizes a BERT-based model as the encoder, which typically has a relatively smaller vocabulary size of 30k, multilingual encoders often have a larger vocabulary. This can increase latency due to the computational overhead of projection to the vocabulary space, as observed in the latency gap between our approach and $\rm{DPR}$.

## Appendix A BEIR Benchmark

**Table 4: Retrieval performance on MS MARCO (MRR@10) and BEIR benchmark (NDCG@10).**
| Model | MS MARCO | ArguAna | Climate-FEVER | DBPedia | FEVER | FiQA | HotpotQA | NFCorpus | NQ | SCIDOCs | SciFact | TREC-COVID | Touché-2020 | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Embedding-based Index |  |  |  |  |  |  |  |  |  |  |  |  |  |
| $\rm{DPR}$ | 30.2 | 40.8 | 16.2 | 30.4 | 63.8 | 23.7 | 45.2 | 26.1 | 43.2 | 10.9 | 47.4 | 60.1 | 22.1 | 35.8 |
| $\rm{VDR}$ | 34.3 | 46.5 | 16.9 | 38.9 | 73.9 | 28.4 | 65.0 | 32.5 | 47.2 | 15.3 | 66.6 | 66.2 | 29.4 | 43.9 |
| $\rm{SVDR_{full}}$ | 34.5 | 52.0 | 16.6 | 39.3 | 70.8 | 29.7 | 65.4 | 33.0 | 47.6 | 15.1 | 65.9 | 68.0 | 28.6 | 44.3 |
|  | Tokenization-based Index |  |  |  |  |  |  |  |  |  |  |  |  |  |
| BM25 | 18.7 | 31.5 | 21.3 | 31.3 | 75.3 | 23.6 | 60.3 | 32.5 | 32.9 | 15.8 | 66.5 | 65.6 | 36.7 | 41.1 |
| $\rm{VDR}_{\beta}$ | 6.1 | 14.1 | 6.1 | 7.9 | 28.4 | 6.4 | 5.7 | 23.9 | 6.8 | 8.1 | 54.5 | 21.9 | 9.2 | 16.1 |
| $\rm{SVDR_{\beta}}$ | 19.0 | 38.6 | 10.8 | 20.8 | 46.5 | 19.8 | 49.4 | 27.9 | 25.3 | 11.1 | 64.2 | 53.5 | 23.7 | 32.5 |
| $\rm{SVDR_{\beta}}$ $(m=10)$ | 26.3 | 44.0 | 12.1 | 24.9 | 57.3 | 22.8 | 55.4 | 30.2 | 33.1 | 12.2 | 65.5 | 54.9 | 25.2 | 36.5 |
| $\rm{SVDR_{\beta}}$ $(m=20)$ | 29.2 | 48.0 | 13.8 | 30.6 | 62.3 | 25.7 | 58.7 | 32.3 | 38.4 | 13.8 | 65.6 | 59.1 | 27.3 | 39.6 |
| $\rm{SVDR_{\beta}}$ $(m=100)$ | 32.9 | 51.5 | 16.6 | 37.8 | 69.2 | 29.3 | 63.3 | 33.2 | 44.7 | 14.8 | 65.7 | 65.9 | 29.0 | 43.4 |

We conducted additional evaluations on the BEIR benchmark, a common standard for evaluating retrieval systems.
We did not include these results in the main sections as many BEIR datasets exhibit a lexical bias, which is acknowledged in the BEIR paper itself [].
Specifically, these datasets have often been constructed using a BM25 retrieval system to retrieve candidates for subsequent human annotation. This setup inadvertently provides unfair advantages to BM25.

#### Training.

Our model was trained on the MS MARCO [] ranking dataset, which includes 8.8 million passages and around 500 thousand queries.
During training, we used a batch size of 256, a learning rate of 2e-5, and trained for 20 epochs. Each query was paired with one negative passage from the MS MARCO dataset.
It is important to highlight that MS MARCO facilitates passage-level retrieval, which does not provide answer spans for distinguishing weak positives from hard negatives. Consequently, we opted not to engage in in-training retrieval for $\rm{SVDR_{full}}$, instead relying on the predetermined BM25 negatives throughout training.

#### Effectiveness.

Our results in Table 4 demonstrate that $\rm{SVDR_{full}}$ surpasses $\rm{VDR}$ and $\rm{DPR}$ when employing an embedding-based index.
This suggests that the integration of a semi-parametric objective enhances the effectiveness of standard parametric search.
On the other hand, when using a binary token index, $\rm{SVDR_{\beta}}$ significantly surpasses $\rm{VDR}_{\beta}$ and demonstrates comparable results to $\rm{DPR}$, underscoring its functionality.
Although $\rm{SVDR_{\beta}}$ outperforms BM25 in domain, it fall behind BM25 in zero-shot scenarios within the BEIR benchmark.
Further improvements are observed when the top-10 passages retrieved from the binary token index are re-ranked, with $\rm{SVDR_{\beta}}$ $(m=10)$ surpassing the dense retrieval performance of DPR in the BEIR benchmark.
Additionally, by re-ranking the top-100 passages, $\rm{SVDR_{\beta}}$ $(m=100)$ achieves 97% of the effectiveness of $\rm{SVDR_{full}}$, outperforming BM25 by a margin of 2.3%.

## Appendix B Details of Latency Evaluation

The evaluation process is outlined as follows.
For BM25 retrieval, we utilize Pyserini [], a library based on a Java implementation developed around Lucene.
In the context of neural retrieval, our implementation is based on Python. For efficient inner dot product computation, we simply utilize the sparse module from PyTorch(^2^22https://pytorch.org/docs/stable/sparse.html), without implementing an inverted index.
Timing measurements are conducted by executing each operation 10 times, then calculating the average after excluding the maximum and minimum values for reporting.

In the indexing stage, $E_{\theta}$ and $T$ utilizes a batch size of 32 and a maximum text length of 256, operating on either a single GPU or CPU.
For the search stage, the computation of $\rm{f(q,D)}$ includes the similarity computation and the subsequent sorting of top-$k$ passages.
In this stage, a batch of 32 is utilized for queries, with the entire passage embeddings allocated into CPU or GPU memory.
Specifically, we use half-precision floating-point data type (fp16) to save GPU allocation. The embedding-based index of DPR take about 30GB GPU allocation, while the embedding-based index of $\rm{SVDR_{full}}$ take 120GB GPU allocation, which exceeds the capacity of our GPU. Therefore, we divide it into 4 shards and compute the search sequentially, summarizing the computational time.
For $\rm{SVDR_{\beta}}$, the binary token index takes about 10GB, which is entirely loaded into GPU memory for computation.
Our analysis excludes the time spent on I/O and data type conversion, assuming that there are adequate resources for processing.

## Appendix C Details of Ablation Study

In this section, we detail the methodology behind our ablation study.
First, we assess the impact of removing specific components we proposed on overall performance.
Initially, we explore the effects of excluding the in-training negative retrieval, opting instead to use the standard BM25 negatives for the entirety of the training process.
This configuration is referred to as “w/o retrieved neg”.
We further investigate the influence of omitting additional negatives. To maintain the same number of passages in each mini-batch despite this removal, we double the batch size. While this approach still allows for in-batch negatives to be used for contrastive learning, it essentially equates to employing random negative samples for each query. We term this setting as “w/o neg”.
While exclude the semi-parameter objective, the $\beta$ search become non-functional, we reference the performance of the $\rm{VDR}$ as an indicator, and term this setting as “w/o SP objective”.

We also assess the effect of the configuration during in-training retrieval.
We investigate the impact of negative sample hardness on model performance. To this end, we manipulate the value of $m$, referred to as “w/ retrieved neg (m)”.
Our negative samples are randomly selected from the top-$m$ passages identified by $\rm{SVDR_{\beta}}$, with the value of $m$ indirectly determining the difficulty of these negatives. A smaller $m$ indicates a higher difficulty of the negative samples retrieved by $\rm{SVDR_{\beta}}$. We conduct evaluations at $m=1$ and $m=100$ to assess how varying the difficulty of negative samples influences retrieval performance.
Moreover, we explore the influence of the source of retrieved negatives by switching the retrieval corpus from Wikipedia to MS MARCO, denoted as “w/ retrieved neg (MARCO)”. We also conduct an ablation study where the size of the Wikipedia corpus is reduced to 8.8 million passages, matching the size of MS MARCO, to isolate the effects of corpus size on retrieval performance. This setting is referred to as “w/ retrieved neg (WIKI 8m)”.

## Appendix D In-training Retrieval Simulation

**Table 5: Retrieval latency, index storage, and index GPU allocation for $\rm{\mathbb{V}triever_{\beta}}$ across varying binary token index density and query batch size.**
|  | Latency (ms) | Storage (GB) | GPU (GB) |
| --- | --- | --- | --- |
|  | Index Density |  |  |
| a=256 | 0.20 | 2.8 | 6.9 |
| a=512 | 0.21 | 4.9 | 23.5 |
| a=1024 | 0.21 | 9.0 | 46.7 |
|  | Query Batch Size |  |  |
| bs=32 | 0.20 | / | / |
| bs=128 | 0.21 | / | / |
| bs=512 | 0.24 | / | / |

We conducted a simulation test to evaluate various factors affecting in-training retrieval, with results summarized in Table 5. Initially, we constructed a binary token index with dimensions of 30k and a sample size of 500m, where each passage vector consists of 256 dimensional activated. Subsequently, we varied the density of passage representations by adjusting the activation number from 256 to 512 to 1024. Notably, we observed a monotonically increase in the storage and GPU allocation of the index as the activation number increased, yet the latency showed minimal change.
Furthermore, the latency exhibited minimal variation in response to changes in the query batch size.

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 5
presentation: 4
questions: - what is the purpose of section 3.1? I guess the argument is around the relationship between MLM and vocabulary expansion - see e.g. https://dl.acm.org/doi/10.1145/3634912

- Unless I miss something, there is no justification of the second par of $L_{semi-para}$, i.e. $L(V_{BoT}(q), V_\theta(p))$ since it is never used. And no ablation shows the importance of this factor

- What is $VDR_{\beta}$ in table 1? I could not find this model in the VDR paper.

- I do not agree on the difference in efficiency (lines 499-506) between your model and BM25. While it is true that leveraging GPU is useful in your case, BM25 could actually be implemented on GPU, so the reported speedup does not mean much.
soundness: 3
strengths: The main strength of the paper lies in the static representation of documents. As far as I know, this is the first model that relies on a simple index structure, that of representing the documents as a binary bag-of-token. This allows for potentially fast retrieval (although this can be debated, see weaknesses) of potential candidates that have then to be re-ranked.
summary: This paper main contribution is a retrieval model where documents are represented by the presence (boolean) of tokens (i.e. a binary bag of tokens). The advantage of such an approach is that the index has to be computed just once (it does not depend on parameters). However, the precision of such a model being low, the authors show that it can be successfully trained with a dense model (used as a second-stage ranker) - and combined with it at inference time. Experiments are conducted on the Wiki21m and BEIR benchmarks and retrieval latency is evaluated - showing that this approach offers a good compromise between effectiveness and efficiency.
weaknesses: One of the main weaknesses is related to the number of collisions (i.e. number of documents for one token) increases with the size of the collection. It is unclear how this approach performs when the number of documents, their length, or both, increase. It would be important thus to investigate using the model on a larger collection (e.g. MS-Marco).

Another point is that the authors state (l. 329) that SOA training "techniques are orthogonal to the retrieval model and have not been applied in our works.". It is not clear if this is truly the case here, and no experiments have been conducted to check the potential of e.g. simple techniques like knowledge distillation. 

Other things:
- The authors propose to use the elu1p (p. 4) - it is not clear why softplus has not been used, the function is quite close to this and more "standard".
- The negative mining proposed for the Wikipedia collection is very specific to this collection (l. 292-293). In my opinion, this invalidates the results reported on the Wikipedia collection (table 1 p. 7) and would justify the use of the dismissed SOA training techniques.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: BM25 as a token retrieval baseline has stood the test of time, being competitive for three decades. Why do the authors think SiDR_beta handily outperform BM25? What can we learn from this? Could this be because it's not an apples to apples comparison? Are there other underlying factors? 

Why did the authors not consider hybrid retrieval systems a more natural baseline?
soundness: 3
strengths: The paper is well motivated. The problems identified with constructing and refreshing vector based document indices are consistent with prevalent challenges in the industry. While most SOTA retrieval systems, especially those used in the industry, are hybrid, re-using representations across the two forms of retrieval seems novel. There is a good set of baselines, though the most relevant baselines are those that perform hybrid retrieval with vector based and token based approaches. The paper will also benefit from inclusion of more qualitative analysis, especially on how the aligned BoT query representations tackle classical IR problems like polysemy and synonymy.
summary: This paper extends the VDR encoder (which also seems to be under review), aligning masked term prediction and bag of tokens representations. The paper shows that hybrid retrieval outperforms either vector based or token based retrieval alone, and the same parametric representation used in vector based retrieval can be "aligned" with bag of tokens to perform non-parametric retrieval.
weaknesses: One of the strengths of bag-of-token based retrieval is that it is easily interpretable. The paper misses an opportunity to demonstrate how going from a parametric representation (which is considered semantic retrieval) allows us to tackle the standard problems in IR such as polysemy and synonymy. How do the lack of weights (a la BM25) on the document side affect retrieval? 

A nit: since the paper uses a hybrid retrieval system, comparison against either vector retrieval alone is not an apples to apples comparison. For example, in Table 1, this system seems to handily outperform BM25. But BM25 doesn't have the benefit of query side synonyms. While the index is non-parametric, the query is not (it's an "aligned version" of the parametric embedding). This doesn't invalidate the results. Rather it provides a channel for further investigation into the kinds of representations that are produced.  

Comparison against other hybrid retrieval systems (Gao et al 2021, Kuzi et Al 2020) would make the paper a lot stronger. (These are ones that come to mind, but there may be more). 

Luyu Gao, Zhuyun Dai, Tongfei Chen, Zhen Fan, Benjamin Van Durme, and Jamie Callan. Complement lexical retrieval model with semantic residual embeddings. In European Conference on Information Retrieval, 2021. URL https://api.semanticscholar.org/CorpusID:232423090

Saar Kuzi, Mingyang Zhang, Cheng Li, Michael Bendersky, and Marc Najork. Leveraging semantic and lexical matching to improve the recall of document retrieval systems: A hybrid approach, 2020. URL https://arxiv.org/abs/2010.01195.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 2
questions: N/A
soundness: 3
strengths: See the summary for more details.
* An interesting approach
* A semi-parameteric index retrieval is easy to carry out on a GPU
* A substantial evaluation using the BEIR datasets (plus additional QA datasets)
* Promising results
summary: The paper focuses on sparse neural retrieval using a SPLADE-like model. It proposes to train a neural sparse bi-encoder V for both queries Q and documents D in such a way that it works well when computing similarity in the standard bi-encoder fashion, i.e., as the inner product of V(Q) and V(D) as well as through the inner product of V(Q) and the binary bag-of-tokens representation of a document.

This can be quite useful since it permits using a neural encoder together with non-parametric document representations (that do not required encoding) for effective and efficient retrieval.

Initially I had several concerns regarding the fairness of comparison to BM25 (multi vs single-thread and running BM25 on GPU) as well as to a possibility of re-ranking using a stronger model. However, additional experiments and clarifications resolved the issues. 

There are also some clarity issues (see the weaknesses section), which I think could be resolved or mitigated in the final version. The paper is somewhat hard to read, but it's understandable with some effort.

I would like to emphasize that the final version of authors need to compare against the fast implementation of BM25 (i.e., using PISA or similar) and make sure they make it clear that the comparison is vs single-thread BM25 (this is only for efficiency evaluation, no need to re-run all experiments).

Last, but not least, that BM25 with re-ranking is a strong baseline is definitely worth highlighting and connecting it to prior work, e.g.:
Leonhardt, Jurek, et al. "Efficient neural ranking using forward indexes." Proceedings of the ACM Web Conference 2022. 2022.
weaknesses: After discussion with authors, I have come to a conviction that the paper is generally solid, but presentation can be improved. 

There are several examples of where the paper is hard to read:
1. No explanation for the need of VDR (I still didn't quite get your explanations in the rebuttal).
2. The whole section of revisiting MLM is very confusing. You say "We provide insights into the consistencies between semi-parametric alignment and masked language
model pre-training", but this is already a pretty confusing phrase. To begin with you probably meant to say "connections". This is a small things, but a lot of such small things here and there complicate reading. Compare, e.g., how this was introduced in the SPLADE paper: We describe in Section 3.1 how the Masked Language Modeling (MLM) head of Pre-trained Language Models can be used to represent tokens in a sequence as vectors in the vocabulary space.
3. Then the whole section "PARAMETRIC AND NON-PARAMETRIC REPRESENTATION" is a bit of a slog to read. You need to explain what you mean by "parametric" vs "non-parameteric" vs "semi-parametric" representations. Moreover, when you talk about parametric representations it's IMHO better talk about **learned* token weights vs token computed using a hand-crafted rule (as in BM25). Don't get me wrong parametric vs non-parametric isn't wrong per se, but the distinction between parametric and non-parametric ML is a bit blurry. Plus, in the context of learned representations it is not frequently used IMHO.

BoT appears almost out of the blue. I think it was only mentioned in the MLM-section. IMHO, it's better to start by explaining that a document/query can be represented by a bag of words or tokens. In some cases these tokens are actual document/query tokens and in some cases they are "predicted" by the encoder model. Moreover, tokens can be weighted and non-weighted. Weights can come either from a model or from token-counting (as in BM25). This would have been a much less confusing explanation IMHO. Then, you can explain how weights are coming from an MLM head of a BERT model. This would have been much more logical, IMHO.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. How does SiDR compare with other late parametric methods, and what are the reasons for its success in the late parametric scenario?
soundness: 3
strengths: 1. The proposed SiDR integrates the advantages of non-parametric and parametric retrieval, achieving competitive performance while maintaining high efficiency.
2. The paper provides clear explanations of the SiDR method, experiments, and analysis, making it easy to understand.
summary: This paper introduces a Semi-parametric Disentangled Retrieval (SiDR) strategy that integrates the advantages of non-parametric and parametric retrieval, achieving competitive performance through the late parametric mechanism while maintaining high efficiency.
weaknesses: 1. In Table 1, the performance of SiDR in parametric retrieval scenario does not surpass ANCE, even with the late parametric mechanism, yet the authors still report it as the best result with the bolded values.
2. In Table 2, under the non-parametric retrieval scenario, SiDR fails to surpass BM25 in most of datasets.
3. The late parametric mechanism is not novel, and the experiments should include more comparisons to various methods in this category.
4. Because of the comparison results with SiDR and ANCE/BM25, the experimental analysis should include an examination of the reasons for SiDR's better performance in the late parametric scenario.

Meta-review: This paper introduces a new, efficient retrieval method, namely “semi-parametric disentangled retrieval framework (SiDR)”. The key idea is to have two encoders: (1) a VDR-based neural encoder (called parametric), and (2) a simple bag-of-words representation (called nonparametric). The neural encoder is trained in a way that these two encoders would align to each other. At test time, retrieval is done by using parametric encoder for the query and nonparametric encoder for the passage, optionally with reranking using parametric encoder for both. This brings benefits of both worlds, with indexing as efficient as nonparametric retrieval like BM25 and expressivity that is comparable to neural retrieval. Results on open-domain QA datasets and BEIR show the effectiveness of the method.

All reviewers acknowledged that this is the first model where the index building is as efficient as BM25 while the expressivity and performance is comparable to dense retrieval, and the benefit of this model is clear given the widely known importance of constructing and refreshing retrieval indexes.

Minor concerns were raised on the writing (e.g., method description and the section revisiting MLMs) and justification of certain modeling choices such as the use of elu1p for the parametric encoder.

Decision: Accept (Poster)