# Towards Effective and Efficient Sparse Neural Information Retrieval

Thibault Formal, Carlos Lassance, Benjamin Piwowarski, Stéphane Clinchant

<figure><img src="image_1.png" /></figure>

## To cite this version:

Thibault Formal, Carlos Lassance, Benjamin Piwowarski, Stéphane Clinchant. Towards Effective and Efficient Sparse Neural Information Retrieval. ACM Transactions on Information Systems, 2024, 42 (5), pp.1-46. 10.1145/3634912. hal-04787990

HAL Id: hal-04787990

https://hal.sorbonne-universite.fr/hal-04787990v1

Submitted on 3 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Towards Effective and Efficient Sparse Neural Information Retrieval

THIBAULT FORMAL*, Naver Labs Europe, France

CARLOS LASSANCE*, Naver Labs Europe, France

BENJAMIN PIWOWARSKI, Sorbonne Université, CNRS, ISIR, France

STÉPHANE CLINCHANT, Naver Labs Europe, France

Sparse representation learning based on Pre-trained Language Models has seen a growing interest in Information Retrieval. Such approaches can take advantage of the proven efficiency of inverted indexes, and inherit desirable IR priors such as explicit lexical matching or some degree of interpretability. In this work, we thoroughly develop the framework of sparse representation learning in IR, which unifies term weighting and expansion in a supervised setting. We then build on SPLADE – a sparse expansion-based retriever – and show to which extent it is able to benefit from the same training improvements as dense bi-encoders, by studying the effect of distillation, hard negative mining as well as the Pre-trained Language Model’s initialization on its *effectiveness* – leading to state-of-the-art results in both in- and out-of-domain evaluation settings (SPLADE++). We furthermore propose *efficiency* improvements, allowing us to reach latency requirements on par with traditional keyword-based approaches (Efficient-SPLADE).

Additional Key Words and Phrases: Information Retrieval, Sparse Representations, Effectiveness, Efficiency

**ACM Reference Format:**

Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2025. Towards Effective and Efficient Sparse Neural Information Retrieval. 1, 1 (December 2025), 45 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn

## 1 INTRODUCTION

Information Retrieval (IR) models based on keyword-matching – such as BM25 [90, 91] – have withstood the test of time and continue to power modern-day search systems. They are simple, efficient, and yet effective. These approaches are, however, limited to simple statistical properties of words and cannot accurately model their complex relationships. Going beyond keyword matching requires having a proper understanding of the meaning (or semantics) of queries and documents, which is more in line with how humans think and process information. The recent breakthroughs brought by Deep Learning in the field of Natural Language Processing (NLP) have nurtured the hope of a new generation of semantic search systems, where models based on neural networks would be able to faithfully represent the content of documents and queries beyond the words they use. Despite their promises, word embeddings like word2vec [73] have shown limited success in the context of IR [53, 117].

We had to wait for Pre-trained Language Models (PLM) to truly witness the rise of neural IR. These models, based on the transformer architecture [106], acquire general *contextualized* knowledge about language through unsupervised *pre-training* on large text corpora, and only need to be *fine-tuned* on downstream tasks with a limited amount of

Authors' addresses: Thibault Formal, Naver Labs Europe, Meylan, France, thibault(formal@naverlabs.com; Carlos Lassance, Naver Labs Europe, Meylan, France, carlos.lassance@naverlabs.com; Benjamin Piwowarski, Sorbonne Université, CNRS, ISIR, Paris, France, benjamin@piwowarski.fr; Stéphane Clinchant, Naver Labs Europe, Meylan, France, stephane.clinchant@naverlabs.com.

<figure><img src="image_3.png" /></figure>

This work is licensed under a Creative Commons Attribution 4.0 International License.

© 2025 Copyright held by the owner/author(s).

Manuscript submitted to ACM

<footnote>*Both authors contributed equally.</footnote>
training data. Models like BERT [21] have irreversibly impacted IR – way beyond what the community thought would be possible a few years back – by completely shifting the state of the art on traditional retrieval benchmarks [17, 65, 80]. Initially employed as re-rankers in multi-stage ranking pipelines, the real paradigm shift, however, has occurred with the emergence of a new generation of neural models tackling the retrieval step in multi-stage ranking pipelines – holding the promise to replace well-established IR models in modern search engines. These approaches aim to address the inherent limitations of keyword matching by directly retrieving, from the complete collection, relevant documents that would have been missed otherwise. Due to the scale of the problem, they have to cope with strict efficiency requirements to provide users with a smooth search experience. Improving the accuracy of search results is, therefore, not enough for systems to be useful, and efficiency constraints have also driven model design. Both aspects – effectiveness and efficiency – are central to this work.

Within the last three years, many approaches have been proposed for the retrieval task, leading to unprecedented results on various benchmarks. They can roughly be categorized into dense and sparse approaches. Models based on dense representations combined with Approximate Nearest Neighbor (ANN) algorithms have proven to be both highly effective and efficient [29, 39, 41, 44, 57, 89]. By directly modeling relevance as distances in a continuous semantic space, they tend to move away from sparse signals and the long-standing vocabulary mismatch problem [28]. In the meantime, several works have been dedicated to bringing lexical models up to date by taking advantage of PLM in various manners [2, 19, 26, 55, 69, 78, 118, 122]. Such sparse approaches, for instance, learn contextualized term weights, query or document expansion, or both mechanisms jointly. They inherit the good properties of Bag-of-Words (BoW) models, such as the exact matching of terms or some degree of interpretability. Motivated by these advantages, we propose several improvements to the SPLADE model [26] in order to increase both its effectiveness and efficiency. More specifically, our contributions are:

* A thorough formalization of sparse representation learning in the context of Information Retrieval (Section 3).

* A detailed description of the SPLADE model (Section 4), including additional experimental results and analyses (Section 6).

* Several improvements in order to increase its effectiveness in both in- and out-of-domain (OOD) evaluation settings (SPLADE ++, Section 7).

* Several improvements in order to increase its efficiency (Efficient-SPLADE, Section 8).

## 2 RELATED WORKS

We review the two main lines of research related to first-stage ranking – namely dense (Section 2.2) and sparse (Section 2.3) neural ranking models. We additionally cover various strategies used to train the former in Section 2.4 – which are the basis for the effectiveness improvements discussed in Section 7.

### 2.1 A Representational View

Although based on radically different paradigms, many retrieval models can be viewed under the same common framework. Following [34, 103], a ranking function can be abstracted as:

$$s(q, d) = f(\phi(q), \psi(d), \eta(q, d)) \qquad (1)$$
where $\phi$ and $\psi$ are the functions mapping queries and documents to their representation spaces, while $\eta$ extracts features from their joint representation – or interaction. $f$ represents any function deriving relevance from such features<fnref n="1" />. In the remainder of this document, we focus on *representation-based* approaches that discard the interaction component $\eta$, and can be written as follows:

$$s(q, d) = f(\phi(q), \psi(d)) \qquad (2)$$

where the representation functions – usually called encoders – map queries and documents in the same vector space, and $f$ usually corresponds to a similarity such as dot product, i.e., $s(q, d) = \phi(q)^T \psi(d)$. In this setting, approaches only differ in the way they *represent* queries and documents, from unsupervised to supervised approaches. In the specific case where $\phi = \psi$, models are usually referred to as bi-encoders, dual encoders, or Siamese networks [7]. Such a formulation of the ranking problem was recently brought up to date in the conceptual framework for a representational approach to IR introduced in Lin [54]. Models following Eq. 2 allow to pre-compute document representations offline, making them usually extremely attractive for Information Retrieval – for which efficiency is a critical aspect.

## 2.2 Dense Approaches

In the dense setting, representations belong to a low-dimensional latent space $\mathbb{R}^d$, where $d$ is small (e.g., $d = 768$ for BERT-based approaches)<fnref n="2" />. These representations are usually obtained by pooling contextualized term representations of the input sequence – for instance, by averaging, or by simply considering the [CLS] token in the case of BERT. Such models – including, for instance, DPR [44], ANCE [114] or TAS-B [39] – are based on the same structural design but rely on different training strategies, including distillation, hard negative mining, or even pre-training – which are further detailed in Section 2.4. Taking a different perspective, ColBERT [45, 94] avoids this pooling mechanism, by representing every token of the input (query or document) as a dense vector. By postponing the interaction of query-document tokens to the very end, it thus becomes possible to pre-compute document representations offline, while still modeling fine-grained relevance information – bridging the gap between interaction- and representation-based approaches. Despite its simple design, ColBERT remains, as of today, one of the most effective IR models. Retrieving from large collections of dense vectors requires dedicated indexing structures and efficient similarity search algorithms – usually relying on *approximate* versions of the search problem – dubbed Approximate Nearest Neighbor (ANN) search<fnref n="3" />.

## 2.3 Sparse Approaches

While dense approaches have witnessed an increasing interest in the IR and NLP communities, they rely on real-time query embedding as well as ANN techniques – usually increasing query latency compared to traditional methods based on inverted indexes and efficient query processing algorithms<fnref n="4" />. Therefore, another research direction simultaneously emerged in the field, dedicated to bringing traditional IR approaches up to date. These methods falling under the “sparse umbrella” tackle the problem through different views, leading to various approaches and architectures. They aim to extend traditional term-based approaches with Pre-trained Language Models, by either learning *term-weighting*

<footnote n="1">¹Note that these functions can be parametrized, e.g., $\phi=\phi_\theta$.</footnote>

<footnote n="2">²As opposed to the *sparse* approaches which represent queries and documents in large vocabulary spaces $\mathbb{R}^{|V|}$, $|V| \gg 10k$, with a sparse structure – see Section 2.3.</footnote>

<footnote n="3">³For instance, FAISS [43] provides most of the state-of-the-art ANN algorithms and has become a common ground for IR researchers working with dense retrieval.</footnote>

<footnote n="4">⁴Note that this has to be taken very cautiously. Comparisons between different types of systems are rather difficult, as they depend on different hardware (GPU vs CPU) and software design choices (e.g., multi- vs mono-CPU). In any case, in large industry-scale systems, traditional approaches have withstood the test of time and remain the core component of modern search systems. We further discuss such efficiency aspects in Section 8.</footnote>
functions, *expansion* methods, or a combination of both. We can thus think of these methods as aiming to implicitly or explicitly learn sparse representations of queries and documents that can efficiently be compared with sparse dot products. While initially motivated by efficiency concerns, other advantages are brought by sparse modeling. For instance, scoring functions become more explainable, as we can explicitly interpret each dimension of the representations – i.e., terms of the vocabulary – contrary to the rather opaque nature of dense embedding spaces. Moreover, such a design explicitly enforces lexical matching, which is known to be critical for IR systems [33]. They can also – in some cases – seemingly be integrated into standard software stacks, which usually only require mono-CPU environments – compared to dense methods based on ANN that might require GPUs and multi-threading.

### 2.3.1 Expansion.
Expansion techniques in IR date back decades and have originally been developed to tackle the vocabulary mismatch problem, where query terms can be lexically different from those used in relevant documents [28]. The idea behind expansion is to enrich (or expand) queries and/or documents to better align them in terms of the vocabulary they use. Traditionally, such approaches have been relying on techniques like Pseudo-Relevance Feed-back [50], but have been significantly impacted by PLM. doc2query [78, 79] is the first of such approaches, and is based on a simple idea: for any given document in a collection *C*, a seq2seq model is used to *generate* queries for which that document might be relevant. The model is trained on pairs of queries and relevant documents (*q*, *d*<sup>+</sup>) from MS MARCO [3], and is used to generate *N* queries for each document using top-*k* random sampling [23], which are simply concatenated to original documents. Then, the newly “expanded collection” *C*′ can be indexed and queried using standard term-based approaches like BM25. This expansion approach has two main effects: it either *i*)* adds new terms to documents (expansion terms), that are likely to bridge the gap with semantically relevant queries, *ii*)* boosts important terms from the document, by increasing their frequency. By modeling query likelihood *P(q|d*)* based on BERT, the TILDE model [124] has also been used as a light document expansion model in TILDEv2 [123].

From a different perspective, other works have been tackling query expansion using PLM. For instance, CEQE [76] extends the traditional relevance feedback model from Lavrenko and Croft [50] by incorporating BERT contextual representations to better select expansion terms. Claveau [14] conditions query expansion based on GPT-2 [86] with constrained generation – using the query as the seed for generation. Similarly to doc2query, the large number of generations encapsulates both aspects of expansion and implicit term weighting.

### 2.3.2 Term Weighting.
Expansion can implicitly boost important terms by increasing their frequency in queries or documents. This is, however, quite inefficient, as the number of generations has to be large for the approach to be effective – up to 80 generated queries per document for doc2query-T5 [78]. Additionally, this mechanism is not learned, and is merely a byproduct of the large number of generations – it, therefore, suffers from its inability to estimate term importance precisely. Such limitations thus motivated approaches that can better estimate *contextualized* term importance in a principled way, to replace traditional weighting schemes estimated from term statistics – which are effective yet imperfect proxies for term importance.

DeepCT [16, 19] aims to map BERT contextualized passage term embeddings ($h_d$)$_j = [h_0, h_1, ..., h_N]$, with a linear projection ($W, b) \in (\mathbb{R}^d, \mathbb{R})$, to scalar values that can be interpreted as term importance<fnref n="5" />. Instead of explicitly optimizing ranking metrics, DeepCT is trained with a per-token regression loss, aiming to assign each token to an “ideal” term importance in the form of Query Term Recall $\in [0, 1]$, estimated from query statistics. It relies on the assumption that important terms should appear in queries for which the given document is relevant. Once the model has been

<footnote n="5">⁵Note that Dai and Callan also introduce a query weighting mechanism for “long” queries.</footnote>
trained, the collection can be indexed by re-scaling and quantizing (for instance to integer values between 0 and 100) document term weights, such that they can be stored in a standard inverted index<fnref n="6" />. Finally, retrieval on the augmented index is performed with BM25 – with no additional online inference needed, as queries are kept unchanged. This is a major advantage in terms of efficiency, compared to dense models, for instance, which require online query encoding. HDCT [18] later extended DeepCT to tackle long document ranking.

DeepCT does not include any expansion mechanism and is thus merely operating as a term re-weigher. Consequently, it does not address the vocabulary mismatch and suffers from limited recall gains. DeepImpact [69] combines the best of both worlds, by re-weighting terms from documents expanded with doc2query-T5. The core idea remains the same, but some mechanisms slightly differ: i) term scoring relies on a two-layer MLP instead of a linear layer, ii) the first occurrence of each unique term is provided to the scorer – instead of considering the max score for DeepCT, iii) the model is trained with a pairwise ranking loss, iv) and finally, the ranking score $s(q, d)$ is simply obtained by summing the importance of query terms appearing in documents – thus not relying on, e.g., BM25. DeepImpact quantizes weights into 8-bit integers – called impact scores – that are directly stored in an inverted index<fnref n="7" />.

Similarly, TILDEv2 [123] learns term weighting on top of expanded documents. However, the model uses a single-layer MLP to compute scores, and represents queries and documents in BERT’s WordPiece vocabulary, instead of the full vocabulary of terms. This has the effect of reducing the index to a fixed small number $|V| = 30k$ – implying longer posting lists. Finally, authors take advantage of their previously introduced TILDE re-ranking model [124] that can cheaply provide expansion terms for each document at almost no effectiveness cost when compared to doc2query-T5 expansion. Note, however, that TILDEv2 is solely evaluated as an efficient re-ranker.

Taking a different perspective, inspired by ColBERT [94], COIL-tok [30] rather learns a low-dimensional dense vector (instead of a weight) for each query and document term, providing effective contextualized term matching by means of the dot product, at the expense of increased storage and retrieval cost<fnref n="8" />. UniCOIL [55] further extends COIL by i) relying on doc2query-T5 expanded documents, ii) projecting term representations as single scores instead of vectors. The approach is thus similar to DeepImpact, but additionally includes query term weighting, which largely boosts the overall effectiveness, while requiring additional online computation.

**Remark.** Previously introduced models, which learn expansion and/or term weighting functions, perform representation learning in a somewhat implicit manner. For instance, DeepImpact can be seen as estimating $s(q, d)$ as a sparse dot product between a binary sparse query vector, and the learned sparse document representation. However, these approaches do not learn expansion and term weighting in a joint manner, and are, to this end, sub-optimal. For instance, DeepImpact relies on the fixed doc2query-T5 expansion, which has not explicitly been trained for the ranking task. In Section 3, we thoroughly develop the general framework of *sparse representation learning*, which unifies term-weighting and expansion in a supervised setting.

## 2.4 Training Retrieval Models

### 2.4.1 Standard Setting.
To train IR models, we usually have access to samples in the form of i) a query $q$, ii) a corresponding relevant document $d^+$, iii) a set of $m$ negative documents $\{d_1^-, \dots, d_m^-\}$. Many losses have been proposed in the Learning-to-Rank literature to optimize ranking models from this data, such as Hinge Loss or ListNet [59]. Driven

<footnote n="6">⁶In practice, a new “fake” collection is created, by repeating terms according to their quantized term frequency. This collection is further processed and indexed in the standard way.</footnote>

<footnote n="7">⁷For that, Mallia et al. rely on the PISA search engine: https://github.com/pisa-engine/pisa.</footnote>

<footnote n="8">⁸Note that the full COIL model relies on the matching between query and document dense [CLS] vectors (after a downsizing projection), and can thus be considered as an “hybrid” method.</footnote>
by their recent success in Computer Vision [10, 36], contrastive approaches have effectively been applied in Information Retrieval to learn effective query and document representations. In its simplest form, it relies on minimizing the negative log-likelihood of the positive passage ($d^+$) against the set of m negative ones ($d_i^-$):

$$ \mathcal{L} = -\log(\mathcal{P}_{\theta}(q|d^+)) \qquad (3) $$

where:

$$ \mathcal{P}_{\theta}(q|d^+) = \frac{\exp(s(q, d^+)/\tau)}{\exp(s(q, d^+)/\tau) + \sum_{i=1,...,m} \exp(s(q, d_i^-)/\tau)} \qquad (4) $$

where $\tau$ is the softmax temperature<fnref n="9" />. In an ideal setting, all the negative documents in the collection C should be taken into account. However, this is computationally infeasible, as the collection contains millions – or more – documents for which it would be required to compute $s(q, d)$. Hence, m is practically set to a small number, such that we approximate the true distribution over the collection – following the Noise Contrastive Estimation literature [35]. This requires clever strategies to sample negative documents that we discuss next.

### 2.4.2 Hard Negative Mining.
The way negatives are selected to build training examples is thus critical and has recently been the focus of several works in neural IR<fnref n="10" />. One obvious way consists in randomly sampling negatives from the collection C. In-Batch Negatives (IBN) is another option that considers positive documents from other queries in the current batch as negative documents. Both types of approaches, however, suffer from slow training convergence [114], as they end up providing non-informative samples<fnref n="11" />. Thus, several approaches have focused on providing more informative – or hard – negatives to train models. They are usually based on a retrieval model that can identify negative documents close enough to the relevant ones. As such, Karpukhin et al. [44] propose to use BM25 as a way to select negatives – combined with local IBN. This procedure remains the “standard” way to train neural retrievers. Other works have proposed to rely on the models themselves to select even more informative negatives, typically in a multi-step or dynamic training strategy [85, 89, 114, 120]. For instance, ANCE [114] periodically indexes – at training time – the collection with the current model. Such an index is used to dynamically retrieve negative documents for subsequent training queries. Finally, some works have looked into maintaining a global cache of generic negatives that are the most informative for any query [58].

### 2.4.3 Distillation.
Other developments have focused on distillation strategies. Traditionally, distillation is employed to distill a large model into a smaller one with the same overall architecture [38]. However, in Information Retrieval, we rather aim to distill an effective – but costly – model like a cross-encoder [80] into an efficient one, such as a bi-encoder – sometimes referred to as cross-architecture distillation. Hence, Hofstätter et al. [40] first proposed the marginMSE loss, which minimizes, with Mean Squared Error, the positive-negative margins between a cross-encoder teacher T and various student models S:

$$ \mathcal{L} = ((s_T(q, d^+) - s_T(q, d^-)) - (s_\theta(q, d^+) - s_\theta(q, d^-))^2 \qquad (5) $$

Similarly, Lin et al. [57] minimize the Kullback-Leibler divergence between $\mathcal{P}_T(q|d^+)$ and $\mathcal{P}_\theta(q|d^+)$, for a light ColBERT teacher, enabling the use of dynamic In-Batch Negatives. Finally, Hofstätter et al. [39] combine both types of distillation.

<footnote n="9">⁹In the remainder, we set $\tau$ to 1.</footnote>

<footnote n="10">¹⁰Note that this problem is rather new, as it is directly linked to training first-stage models. Re-rankers do not suffer from such issues, as samples are simply built from the candidate generator – for instance BM25.</footnote>

<footnote n="11">¹¹Such negatives can be seen as too “easy” for the model. Consequently, they usually provide near-zero loss values and, thus, near-zero gradients.</footnote>
Overall, cross-architecture distillation has become quite standard and is a simple and effective way to boost the performance of bi-encoders.

# 3 SPARSE REPRESENTATION LEARNING IN INFORMATION RETRIEVAL

In this Section, we introduce a thorough formalization of sparse representation learning in the context of Information Retrieval. Here, models explicitly aim to learn sparse representations for queries and documents<fnref n="12" />. We describe in Section 3.1 how the Masked Language Modeling (MLM) head of Pre-trained Language Models can be used to represent tokens in a sequence as vectors in the vocabulary space. In Section 3.2, we present models that aggregate such token-level representations into query and document vectors – setting the ground for the SPLADE model described in Section 4.

## 3.1 Representing Tokens in the Vocabulary Space

In sparse representation learning for IR, term importance is usually predicted in BERT's WordPiece vocabulary – where |V| = 30522 – based on the logits of the MLM head<fnref n="13" />. More precisely, let us consider an input query or document sequence after WordPiece tokenization t = (t₀, t₁, ..., tN) = ([CLS], t₁, ..., tN-1, [SEP]). BERT outputs contextualized embeddings (h₀, h₁, ..., hN) for each token<fnref n="14" /> in t. The MLM head projects each embedding hi back to the vocabulary space ℝ^|V|. More specifically, the importance (logit) wiv of the token v (of the vocabulary) for a token i (of the input sequence) is given by:

$$w_{iv} = \text{transform}(h_i)^T E_v + b_v, \quad v \in \{1, \dots, |V|\} \qquad (6)$$

where E_v denotes the BERT input embedding for token v, b_v is a token-level bias, and transform(.) is a linear layer with GeLU activation [37] and LayerNorm [1]. Note that the projection matrix is tied to the input embedding matrix E_{|V|×d}.

While nothing a priori prevents having a separate output projection matrix W, tying the input and output embedding matrices pre-dates BERT and has shown to be both an effective and efficient design choice – by reducing the number of training parameters [83] – thus becoming standard for PLM.

The weights w_i correspond to an unnormalized log-probability distribution over the vocabulary, for token t_i, where each output dimension v ∈ V is actually associated with the token it represents – therefore providing an interpretable representation basis for IR models. MLM pre-training relies on a fairly simple idea: given an input sequence, randomly mask some fraction of the input tokens, that the model must predict given their context. More specifically, a softmax is applied to the term importance w_i corresponding to a [MASK] tokens (through Eq. 6), which are then used to train the models with Cross-Entropy to predict the actual masked tokens. The MLM head is usually discarded during fine-tuning – only the contextualized embeddings h_i are used for downstream tasks. However, it provides a well-calibrated initialization point for representing queries and documents in the vocabulary space by implicitly learning term expansion.

We illustrate this phenomenon for two pre-trained models in Table 1. As the task requires predicting masked tokens from their context, expansion thus naturally occurs. For instance, synonyms of a given word are usually associated with high logits, as they could likely occur in the same context – for instance “cancer”→“tumor” in Table 1. In IR, we can see this as propagating importance to similar tokens. We thus see that such vocabulary-based representations offer two nice properties, namely i) expansion and ii) term-weighting, as tokens “close” to an input token t_i have higher values (logits) than non-related ones.

<footnote n="12">¹²Recall that in many approaches (for instance DeepCT), representation learning is done in a somewhat implicit manner.</footnote>

<footnote n="13">¹³Note that most approaches, including SPLADE, are originally based on BERT; we show in Section 8.1.1 how it is possible to use various PLM models, as long as they provide a pre-trained MLM head.</footnote>

<footnote n="14">¹⁴Note that we consider tokens to be tokenized units of the WordPiece vocabulary.</footnote>
<table><caption>Table 1. MLM prediction for the Robust04 query "prostate cancer detection treatment". We show for each token the top-3 predicted tokens, i.e., with the highest MLM logits. RoBERTa relies on a cased Byte-Pair Encoding (BPE) [96] tokenizer, where G indicates the start of a new token. All the other methods rely on WordPiece [112]. Note that only the generator part of ELECTRA has been pre-trained with MLM. We also include for comparison the prediction for one of our SPLADE++ models from Section 7.</caption><thead><tr><th></th><th></th></tr></thead><tbody><tr><td colspan="2">» distilbert-base-uncased [93]</td></tr><tr><td>prostate</td><td>→ (prostate, ., the)</td></tr><tr><td>cancer</td><td>→ (cancer, tumor, ##mour)</td></tr><tr><td>detection</td><td>→ (detection, identification, screening)</td></tr><tr><td>treatment</td><td>→ (., .; treatment)</td></tr><tr><td colspan="2">» google/electra-base-generator [13]</td></tr><tr><td>prostate</td><td>→ (prostate, ##iate, mal)</td></tr><tr><td>cancer</td><td>→ (cancer, cancers, tumor)</td></tr><tr><td>detection</td><td>→ (detection, detecting, screening)</td></tr><tr><td>treatment</td><td>→ (treatment, therapy, treatments)</td></tr><tr><td colspan="2">» distilroberta-base [60]</td></tr><tr><td>pro</td><td>→ (pro, Pro, per)</td></tr><tr><td>state</td><td>→ (state, Gstate, GState)</td></tr><tr><td>Gcancer</td><td>→ (Gcancer, Gtumor, GCancer)</td></tr><tr><td>Gdetection</td><td>→ (Gdetection, Gdiagnosis, Gdetect)</td></tr><tr><td>Gtreatment</td><td>→ (Gtreatment, Gtherapy, Gtreatments)</td></tr><tr><td colspan="2">» naver/splade-cocondenser-ensembledistil [26]</td></tr><tr><td>prostate</td><td>→ (prostate, bp, ur)</td></tr><tr><td>cancer</td><td>→ (cancer, tumor, disease)</td></tr><tr><td>detection</td><td>→ (detection, detect, test)</td></tr><tr><td>treatment</td><td>→ (treatment, therapy, treated)</td></tr></tbody></table>

**Remark.** This approach conceptually differs from term-weighting models such as DeepImpact [69] (Section 2.3.2), which learn scoring functions mapping contextualized embeddings to a score in $\mathbb{R}$. In SPLADE [26] and related models, each embedding is mapped to a vocabulary-sized representation in $\mathbb{R}^{|V|}$.

## 3.2 From Token to Sequence-Level Representations

In Section 3.1, we have shown how we can represent tokens as vectors in the vocabulary space, therefore explicitly modeling expansion and term weighting. Different approaches have been proposed to derive query and document views from the token representations by aggregating importance vectors $w_i$ for tokens $t_i$ in input sequences. Importantly, representations are located in a high-dimensional space $\mathbb{R}^{|V|}$, with $|V| \gg 10k$. To perform efficient retrieval from large collections with inverted indexes, such representations must be sparse. Thus, methods also differ in the way they enforce such sparsity. The overall scheme of sparse representation learning based on the output of the MLM head is represented in Figure 1. It follows the representation paradigm from Eq. 2, which aims to represent queries and documents as sparse vectors $w$. Depending on the approach, query and document encoders can be different or symmetric. We note $w^{(q)}$ (resp. $w^{(d)}$) vectors for queries (resp. documents). We also denote selection with subscripts:
$w_{iv}$ corresponds to the v-th token of the i-th position of the input sequence, while $w_v$ simply corresponds to the v-th dimension of a query or document vector.

<figure><img src="image_4.png" /><figcaption>Fig. 1. Sparse representation learning based on the MLM head. Ranking models rely on i) an aggregation mechanism, ii) a method to make representations sparse.</figcaption></figure>

### 3.2.1 EPIC
EPIC [64] (for Expansion via Prediction of Importance with Contextualization) is a lightweight re-ranking module based on vocabulary-grounded query and document representations. It is the first approach proposing to perform passage expansion based on a mechanism similar to Eq. 6. In contrast, queries are treated similarly to term-weighting approaches like DeepCT [16, 19], i.e., by mapping each query token $t_i$ to a single score $w^{(q)}(t_i)$ as follows:

$$w^{(q)}(t_i) = \log \left( 1 + \text{softplus} \left( \theta_1^T h_{q_i} \right) \right) \qquad (7)$$

where $\theta_1 \in \mathbb{R}^d$. The query is represented as a sparse vector $\phi(q) = w^{(q)}$ in $\mathbb{R}^{|V|}$ by setting to 0 all the other dimensions (tokens) in the vocabulary – it thus does not perform expansion. The softplus function prevents terms from having a negative importance while enforcing a saturation effect through the logarithm.

Contrary to queries, documents are represented as dense vectors in the vocabulary space $\mathbb{R}^{|V|}$ by means of three components. More precisely, document terms $t_j$ are first projected to the vocabulary $\mathcal{V}$, i.e., $w_j = \theta_2 h_{d_j}$, $V_j$. Note that while $\theta_2$ is initialized to the pre-trained MLM projection matrix (or equivalently, the input embedding $E$), it is, however, not tied to it. Such projection is akin to what is done in Eq. 6 – although not strictly equivalent. In addition, term importance is taken into account, similarly to Eq. 7:

$$w^{(d)}(t_j) = \log \left( 1 + \text{softplus} \left( \theta_3^T h_{d_j} \right) \right)$$

Finally, a global feature for the passage is computed, based on the representation of the [CLS] token: $c(d) = \sigma(\theta_4^T h_{d_{[CLS]}})$, where $\sigma$ is the sigmoid function. The importance of each token $v$ of the vocabulary $\mathcal{V}$ is finally given by:
$$w_v = c(d) \max_{j \in d} (w^{(d)}(t_j) w_{jv}), \quad v \in \{1, \dots, |V|\} \qquad (8)$$

While queries are *sparse* by design, Eq. 8 implies that documents are *dense* vectors in the vocabulary space. MacAvaney et al. indeed focus on the re-ranking task, for which exhaustive computation of large dot products is feasible – i.e., without requiring fast query processing over inverted indexes. The approach is efficient, in the sense that document representations can be computed offline – at the expense of a large index ($d \in \mathbb{R}^{|V|}$). The authors further study the role of pruning, by observing that a large proportion of the values in document vectors are actually low. Documents representations can be reduced up to 1000 non-zero-values, by simple top-k pooling (after training), without any loss of effectiveness<fnref n="15" />.

**Remark.** Even though EPIC is not strictly relying on the MLM head to represent queries and documents, and only tackles re-ranking, it still has set the basis for grounding query and document representations in the PLM vocabulary space in neural IR.

### 3.2.2 SPARTA.
In the context of Open-Domain Question Answering, SPARTA [122] (for Sparse Transformer Matching) addresses the problem from a different angle, by introducing a matching function similar to ColBERT, but where query tokens are *not* contextualized. More specifically, given the *input* BERT embeddings $\mathcal{E}(q) = (e_q)_i$, and the *output* BERT embeddings $(h_d)_j$ for respectively the query $q$ and document $d$, the relevance score is computed as:

$$y_i = \max_{j \in d} \cos(e_{q_i}, h_{d_j}) \qquad (9)$$

$$s(q, d) = \sum_{i \in q} \log(\text{ReLU}(y_i + b) + 1) \qquad (10)$$

where $b$ is a trainable bias. Because query token embeddings are non-contextual, it thus becomes possible to compute, prior to indexing, term matching scores $y_v = \max_{j \in d} \cos(e_v, h_{d_j})$, and this for every term $v$ in the WordPiece vocabulary. This results in viewing documents as vectors $w^{(d)}$ in the vocabulary space<fnref n="16" />, with a formulation *implicitly* close to the MLM one. The ReLU function, coupled with $b$, enforces sparsity such that documents only have a handful of non-zero term matching scores $y_v$. In practice, however, there is no explicit control of the sparsity, and SPARTA representations are quite large – up to several thousand terms. Zhao et al. further study the effect of a post-processing top-k pooling strategy – similar to EPIC – and manage to reduce vectors up to a few hundred dimensions without significant drops in effectiveness.

**Remark.** SPARTA incidentally bridged the gap between interaction-based models like ColBERT and sparse bi-encoders based on the MLM head. Even if the latter aim to represent queries and documents as single vectors, they are, to some extent, computing interactions with a *fixed* vocabulary basis.

### 3.2.3 SparTerm.
SparTerm [2] (for Term-based Sparse Representations) is the first sparse bi-encoder designed to explicitly tackle first-stage ranking. It relies on the term-level, vocabulary-grounded representations from the MLM head expressed in Eq. 6. The final query or document vector is then obtained by summing importance predictors over

<footnote n="15">¹⁵Note that 1000 is still quite large – $10^8$ weights for 1M documents – but further reducing $k$ (e.g., $k = 100$) leads to performance drops.</footnote>

<footnote n="16">¹⁶Queries in the meantime can be seen as sparse binary BoW vectors.</footnote>
the input sequence $t$, after applying ReLU to ensure that term weights are positive:

$$w_v = g_v \times \sum_{i \in t} \mathrm{ReLU}(w_{iv}), \quad v \in \{1, \dots, |V|\} \qquad (11)$$

where $g_v$ is a binary mask (or gating controller) that controls which terms to include in the representation, by turning off certain dimensions $v$ of the vocabulary $V$. Vectors can be made sparse, thus allowing efficient retrieval from an inverted index. Bai et al. introduce two masking mechanisms:

* lexical-only is a BoW masking, i.e., $g_v = 1$ if token $v$ appears in $t$, and 0 otherwise;

* expansion-aware is a lexical-expansion-aware binary gating mechanism based on BERT, which learns to select the terms to appear in the final representation. To preserve the original input, it is forced to 1 if the token $v$ appears in $t$.

The binary nature of the gating controller $g$ prevents training the whole model in an end-to-end manner. Thus, the controller is first trained on four auxiliary tasks to learn how to expand. It is then fixed, while fine-tuning the term-weighting model with negative log-likelihood, on samples built from BM25 (cf. Section 2.4). A threshold value is chosen for the binarizer, such that representations are sparse enough.

**Remark.** SparTerm expansion-aware gating is somewhat intricate. The model cannot be trained end-to-end, therefore preventing it to learn the optimal pruning strategy for the ranking task. Moreover, the two lexical and expansion-aware strategies do perform almost equally well, questioning the actual benefits of expansion. Finally, including all the tokens appearing in the input is also limiting since, e.g., stopwords are included, which is detrimental to ensure sparsity<fnref n="17" />.

# 4 SPLADE

SPLADE [26] (for SParse Lexical AnD Expansion) was born from these preliminary works. Slight – yet essential – changes to the SparTerm model allow to dramatically improve both its effectiveness and efficiency, making SPLADE the first sparse model to outperform dense approaches<fnref n="18" />.

## 4.1 Saturated Term Weighting

The importance estimation from Eq. 11 is modified by introducing a log-saturation effect which prevents some terms from dominating:

$$w_v = \sum_{i \in t} \log (1 + \mathrm{ReLU}(w_{iv})), \quad v \in \{1, \dots, |V|\} \qquad (12)$$

It is inspired by sublinear term frequency scaling in traditional IR models, which hypothesizes that relevance should not be linear w.r.t. $tf$. Such ideas were formally developed in axiomatic approaches to IR [24], and models based on $log(tf)$ are usually more effective than their unscaled counterparts. This type of saturation has proven effective in various neural IR models, including ones based on BERT, such as EPIC (Section 3.2.1) or SPARTA (Section 3.2.2). Finally, replacing the sum aggregation by a max – illustrated in Figure 2 – actually results in much more effective models (see Section 6)<fnref n="19" />:

<footnote n="17">¹⁷In Section 6.3 (Figure 5) we show how frequent terms from the input are likely to be removed in SPLADE vectors.</footnote>

<footnote n="18">¹⁸In equivalent training settings – see Section 7.2.2.</footnote>

<footnote n="19">¹⁹The sum formulation was part of the first SPLADE publication [26], but was quickly replaced by the max starting from [25].</footnote>
$$w_v = \max_{i \in t} \log (1 + \text{ReLU}(w_{iv})), \quad v \in \{1, \dots, |V|\} \qquad (13)$$

<figure><img src="image_5.png" /><figcaption>Fig. 2. High-level overview of SPLADE-max. Red dots correspond to the max values over the sequence. With this formulation, the model selects, for each dimension v, the most salient contribution among input tokens.</figcaption></figure>

## 4.2 Learning Sparse Representations

The idea of learning sparse representations for first-stage retrieval dates back to SNRM [118] – in the pre-BERT era of neural IR. SNRM is based on an encoder $\phi$ that maps queries and documents into a high-dimensional latent space. At training time, an $\ell_1$ regularization is applied to query and document representations, and the model is trained by jointly optimizing ranking and regularization losses:

$$\mathcal{L} = \mathcal{L}_{\text{rank}}(q, d^+, d^-) + \lambda (\ell_1(\phi(q)) + \ell_1(\phi(d^+)) + \ell_1(\phi(d^-))) \qquad (14)$$

where $\ell_1(w) = \sum_v |w_v|$ is a smooth, differentiable proxy of the $\ell_0 = \sum_v |w_v|^0$ loss – the true non-differentiable objective to minimize, which corresponds to the number of non-zero elements in w. The strength of the regularization is given by the hyperparameter $\lambda$. After training, SNRM’s representations are sparse enough to be stored in an inverted index, and retrieval from the collection C is performed with standard query processing techniques [104]. However, the effectiveness of the approach remained quite limited at that time.

In the context of learning sparse image representations in Computer Vision, Paria et al. [81] pointed out that minimizing the $\ell_1$ norm of high-dimensional representations does not result in the most efficient index, as nothing ensures that posting lists are evenly distributed. To obtain a well-balanced index, Paria et al. introduce the FLOPS regularizer, a smooth relaxation of the average number of floating-point operations necessary to compute the dot product between two vectors – hence directly related to the retrieval time. More precisely, let us consider the activation probability $p_v$ for token (or dimension) $v \in V$, and its empirical estimation $\bar{p}_v$ over a batch $\mathcal{B}$ of b document (and equivalently on the query side):

$$\bar{p}_v = \frac{1}{b} \sum_{i=1}^{b} 1[w_v^{(d_i)} \neq 0] \qquad (15)$$
where $1[.]$ denotes the indicator function. The mean-FLOPS-per-row, i.e., the expected number of floating-point operations when computing the dot product between two random document vectors, is given by$^{20}$:

$$ \mathcal{F} = \sum_{v \in \mathcal{V}} p_v^2 \qquad (16) $$

It is estimated over a batch by $\bar{\mathcal{F}} = \sum_{v \in \mathcal{V}} \bar{p}_v^2$. For a fixed amount of sparsity $\sum_{v \in \mathcal{V}} p_v = |\mathcal{V}|p$, such a quantity is minimized when $p_v = p, \forall v$, i.e., when distributions are uniform across posting lists. Intuitively, its goal is to balance each dimension (or token) in the representation space, such that they are equally used. In other words, we want to spread as much as possible the representations in the embedding space. As $\bar{\mathcal{F}}$ is not continuous, and cannot be optimized with backpropagation, Paria et al. further introduce a smooth relaxation of the (empirical) activation probabilities as $\bar{a}_v = \frac{1}{b} \sum_{i=1}^b |w_v^{(d_i)}|$ – which corresponds to the $l_1$ norm of the activation (scaled by $1/b$)$^{21}$. This gives the following relaxation, estimated over a batch:

$$ \tilde{\mathcal{F}} = l_{\text{FLOPS}} = \sum_{v \in \mathcal{V}} \bar{a}_v^2 = \sum_{v \in \mathcal{V}} \left( \frac{1}{b} \sum_{i=1}^{b} w_v^{(d_i)} \right)^2 \qquad (17) $$

This differs from the $l_1$ regularization used in SNRM, where the $\bar{a}_v$ are not squared – as the $l_1$ estimated over a batch can be written as:

$$ l_1 = \frac{1}{b} \sum_{i=1}^{b} \sum_{v \in \mathcal{V}} w_v^{(d_i)} \qquad (18) $$

$$ = \sum_{v \in \mathcal{V}} \frac{1}{b} \sum_{i=1}^{b} w_v^{(d_i)} = \sum_{v \in \mathcal{V}} \bar{a}_v \qquad (19) $$

Contrary to $l_1$, $l_{\text{FLOPS}}$ tends to sparsify the denser dimensions at a faster rate. Paria et al. furthermore show that, in the case of normalized representations $w/||w||$, $\tilde{\mathcal{F}}$ is minimized when vectors are orthogonal – such a loss tends to make the representations occupy the space in a uniform manner. It bears a resemblance to a thread of work dedicated to learning image representations that are uniformly distributed over the unit sphere [32, 92]. Note that in practice, the non-linear positive activation is crucial to learn sparse vectors: the bias in Eq. 6 directly acts as a knob that can turn off specific dimensions.

**Remark.** The FLOPS measure [81] estimates the average number of floating point operations needed to compute the score (i.e., the dot product) between two vectors – not to be confused with the FLOPS measure commonly used in Deep Learning, which corresponds to the total number of floating-point operations performed by a model’s forward pass.

## 4.3 Summing-Up

SPLADE combines vocabulary-based representations from the MLM head with sparse regularization for end-to-end training of sparse expansion-aware representations of documents and queries. More specifically, the log-saturated model (Eq. 13) is trained by jointly optimizing ranking and regularization losses:

$$ \mathcal{L} = \mathcal{L}_{\text{rank}} + \lambda q \ell_{\text{reg}}^q + \lambda d \ell_{\text{reg}}^d \qquad (20) $$

<footnote>$^{20}$We refer to Paria et al. for the rationale and the complete derivation.</footnote>

<footnote>$^{21}$ $\bar{a}_v = \frac{1}{b} \sum_{i=1}^{b} w_v^{(d_i)}$ in our case, as SPLADE activations are positive.</footnote>
where $l_{reg}$ is a sparse regularization (such as $l_1$ or $l_{FLOPS}$). Two distinct regularization weights ($\lambda_d$ and $\lambda_q$) are used for queries and documents, making it easier to put more pressure on the query sparsity – a critical aspect for fast retrieval (see Section 8). SPLADE avoids resorting to approximate search – contrary to ANN methods for dense bi-encoders, whose impact on IR metrics has not been fully evaluated yet$^{22}$. Also, note that SPLADE explicitly optimizes an efficiency proxy – while most of the ANN methods for dense retrieval are *unsupervised* and applied after model training. A few works have, however, recently tackled end-to-end optimization of dense encoders and quantization methods. For instance, Yamada et al. [115] integrate a learning-to-hash module into DPR training to represent passages with compact binary codes rather than continuous vectors. Similarly, Zhan et al. [119] jointly train a dense encoder and Product Quantization.

**Remark.** Due to its simplicity, as well as its competitive results, we build on SPLADE [25, 26], and propose improvements to its effectiveness and efficiency in respectively Section 7 (SPLADE++) and Section 8 (Efficient-SPLADE). All in all, we list in Table 2 existing sparse approaches with their main characteristics. Similar unified views of sparse methods are also introduced in Nguyen et al. [77] and Thakur et al. [102].

<table><caption>Table 2. A cartography of existing sparse approaches for first-stage ranking. QE, DE, TW, and e2e refer respectively to Query Expansion, Document Expansion, explicit Term Weighting and end-to-end, which are the main characteristics differentiating models. We notice that the end-to-end nature of SPLADE allows us to seemingly apply the same training techniques introduced for dense approaches such as distillation or negative sampling (Section 2.4). Also, note that each model relies on additional specifics and implementation details. Our contributions are highlighted in gray.</caption><thead><tr><th>model</th><th>QE</th><th>DE</th><th>TW</th><th>e2e</th></tr></thead><tbody><tr><td>doc2query-T5 [78] (2019)</td><td>✗</td><td>✓</td><td>✗</td><td>✗</td></tr><tr><td>DeepCT [19] (2020b)</td><td>✗</td><td>✗</td><td>✓</td><td>✗</td></tr><tr><td>DeepImpact [69] (2021)</td><td>✗</td><td>✓</td><td>✓</td><td>✗</td></tr><tr><td>TILDEv2 [123] (2021a)</td><td>✗</td><td>✓</td><td>✓</td><td>✗</td></tr><tr><td>COIL-tok [30] (2021a)</td><td>✗</td><td>✗</td><td>✓</td><td>✓</td></tr><tr><td>UniCOIL [55] (2021)</td><td>✓</td><td>✓</td><td>✓</td><td>✗</td></tr><tr><td>SPARTA [122](2021)</td><td>✗</td><td>✗</td><td>✓</td><td>✗</td></tr><tr><td>SparTerm [2] (2020)</td><td>✗</td><td>✓</td><td>✓</td><td>✗</td></tr><tr><td>SpaDE [12] (2022)</td><td>✗</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>SPLADE-doc [25] (2021a)</td><td>✗</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>SPLADE [26] (2021b)</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>SPLADE++ (Section 7)</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Efficient-SPLADE (Section 8)</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr></tbody></table>

# 5 EXPERIMENTAL SETTING

In this Section, we describe the experimental setting we follow for training and evaluating SPLADE models, from early iterations (SPLADE, Section 6) to more effective (SPLADE++, Section 7) and efficient (Efficient-SPLADE, Section 8) models.

<footnote>22Some studies like Izacard et al. [42] however showed how dimension reduction or quantization techniques could be used with minimal performance loss on two Question Answering datasets.</footnote>
## 5.1 Data

We train and evaluate our models on the MS MARCO passage ranking dataset [3]. We rely on the official MS MARCO training pairs<sup>23</sup>, which contain samples based on a query *q*, an annotated positive document *d*<sup>+</sup>, and a (pseudo) negative document *d*<sup>-</sup> sampled from the top-1000 documents retrieved by BM25<sup>24</sup>, for query *q*. We further rely on IBN [44, 57, 85] (Section 2.4). Specifically, we augment the number of negatives for a given sample by considering every other positive document in the batch *B* as negative. Thus, a training sample becomes (*q*, *d*<sup>+</sup>, *d*<sup>-</sup>, *d*<sub>1</sub><sup>-</sup>, ..., *d*<sub>|B|-1</sub><sup>-</sup>), and the model can further be optimized with Eq. 3. We use the MS MARCO development set, which contains 6980 queries with shallow annotation for evaluation. We also evaluate on TREC DL 2019 [15], which contains fine-grained annotations from human assessors for a set of 43 queries. To lighten the analysis, we voluntarily omit results on TREC 2020 – which tend to align with the ones from 2019.

**Remark.** The experimental setting introduced here can be seen as the “default” one for training and evaluating SPLADE on MS MARCO. In Section 7, we show how SPLADE training can be improved with better sampling strategies or distillation techniques that have previously been shown to be effective for dense bi-encoders. We further extend the evaluation procedure in Section 7 by including zero-shot experiments on the BEIR benchmark.

## 5.2 Training, Indexing and Retrieval

Models are trained based on PyTorch [82] and HuggingFace transformers [111]. We consider DistILBERT [93] – which contains around 66M parameters – to be the default initialization checkpoint for SPLADE models<sup>25</sup>. We show in Section 8.1.1 that different architectures can be used as a backbone, as long as they provide the pre-trained MLM head. Models are trained for 150k iterations with the ADAM optimizer [47], with a batch size of 124, using a learning rate of 2e-5 with linear scheduling and a warmup of 6000 steps. We keep the best model checkpoint based on MRR@10 computed on an approximate retrieval validation set every 10k iteration, in a similar fashion to Hofstätter et al. [39] and Kim et al. [46]. Indeed, our end goal is to optimize first-stage retrieval for the collection *C*. However, indexing the complete collection (8.8M documents) based on the SPLADE encoder $\phi_{(s)}$ at each validation step *s* is costly, and we need to resort to some approximation. Thus, we first select a set of 1600 queries from the larger set of MS MARCO dev queries – which are not part of the official evaluation dev queries. For each of such queries, we retrieve 100 documents from *C* with *i*) BM25 *ii*) and a simple dense bi-encoder model. By considering each unique retrieved document, and adding all the missing relevant ones, we end up with a collection sample $C_s$ of approximately 300k documents, that can be easily indexed in a few minutes at training time, providing us a proxy of the true retrieval performance. We consider a maximum length of 256 for input sequences – larger ones are truncated. This is quite reasonable for MS MARCO passages which are usually shorter – 75 WordPiece tokens on average. To mitigate the contribution of the regularizer at the early stages of training, we follow Paria et al. [81] and use a scheduler that quadratically increases $\lambda = (\lambda_q, \lambda_d)$ at each training iteration until a given step (50k in our case), from which it remains constant<sup>26</sup>. Typical values for $\lambda$ fall between 1e-1 and 1e-5.

Sparse representations allow us to take advantage of inverted indexes and query processing techniques. We use a custom inverted index implementation based on Python arrays and term-at-a-time processing, which relies on

<footnote n="23"><sup>23</sup>https://microsoft.github.io/msmarco/Datasets.

<sup>24</sup>Note that more negatives could be used. Gao et al. [31] show some improvements when training cross-encoders.

<sup>25</sup>distilbert-based-uncased on HuggingFace. Note that many works (such as SparTerm or EPIC) use BERT as a backbone. As we show in Section 8.1.1, results are usually similar on MS MARCO (in-domain), but tend to degrade with smaller checkpoints on out-of-domain evaluation – not shown here. Nevertheless, models are much faster to train, and this constitutes our base setting.

<sup>26</sup>Although it remains our default setting, some experiments we recently conducted didn’t conclude in a huge effect of the scheduler.</footnote>
Numba [49] to parallelize retrieval. Contrary to standard software implementations of inverted indexes that usually store compressed integer values for terms such as tf, our index can directly store the float values in SPLADE vectors. We further devise a quantization strategy inspired by DeepCT. At indexing time, we create quantized pseudo-tf for each token (or dimension) v of SPLADE vectors, by simple scaling and rounding: tf(v) = round(w_v(d) * 100). We can thus create a pseudo text collection by repeating, for each document, each term of the WordPiece vocabulary tf(v) times. Afterward, such a collection can be processed, indexed, and retrieved using standard retrieval software stacks<sup>27</sup> – provided the same treatment is applied to queries. In the end, we have two different ways to perform indexing and retrieval, which, however, lead to the same results – despite the quantization in the latter case.

We conducted experiments with two main regularization losses, $\ell_1$ (Eq. 18) and $\ell_{FLOPS}$ (Eq. 17), which have different effects (see for instance Section 6.2). Finally, we rely on 4 Tesla V100 GPUs with 32GB memory<sup>28</sup> to train and index models – retrieval can trivially be performed on a single GPU or even on CPU-based environments. Models are trained and evaluated with half-precision (fp16) [72]. We open-source the code, available at https://github.com/naver/splade. We further release several model checkpoints on HuggingFace, available at https://huggingface.co/naver.

## 5.3 Evaluation

We report R@1000 for all datasets, as well as the official metrics MRR@10 and nDCG@10 for MS MARCO dev set and TREC DL, respectively. As SPLADE allows various effectiveness-efficiency trade-offs<sup>29</sup> – depending on the regularization strength $\lambda_q$ and $\lambda_d$ – we resort to the FLOPS measure to compare efficiency between different models. It gives an estimation of the average number of floating point operations needed to compute the score (i.e., the dot product) between a query and a document vector. It is defined as the expectation $E_{q \sim Q, d \sim D} [\sum_{v \in V} p_v^{(q)} p_v^{(d)}]$ where $p_v$ is the activation probability for token v in a document d or a query q. It is empirically estimated based on the representations obtained from the complete set of 100k MS MARCO dev queries, as well as the full collection C. Overall, the FLOPS gives an indication of i) how sparse representations are, ii) how balanced the dimensions in query and document representations are. Note that it does not account for the computations needed to obtain the representations (model inference). Other efficiency metrics, closer to practical considerations – such as query latency – could also be used. However, they can be hard to evaluate properly, as different models can rely on different hardware and software stacks (e.g., CPU or GPU, mono-CPU or multi-threaded, etc.). When we compare the efficiency of SPLADE models, the FLOPS is, therefore, informative enough. We, however, discuss such aspects in Section 8.

## 5.4 Baselines

We present preliminary experimental results and analyses for SPLADE in Section 6. Section 7 introduces various effectiveness improvements for SPLADE – dubbed SPLADE++. Finally, Section 8 tackles efficiency aspects and introduces more efficient extensions – dubbed Efficient-SPLADE. When reporting results in Tables, we consider the best performance we can achieve – in terms of MRR@10 on our approximate validation set – with a FLOPS value inferior to 3. It thus does not necessarily correspond to the best performance we can obtain, but to a more realistic trade-off between effectiveness and efficiency. We further analyze such trade-offs in Sections 6.2 and 8, and provide additional analyses in Section 6.3.

<footnote n="27"><sup>27</sup>Such as Anserini [116], which is built on Lucene. The approach, which requires a few adaptations (such as dot product scoring instead of BM25), is described here: https://github.com/castorini/anserini/blob/master/docs/regressions-msmarco-passage-distill-splade-max.md.</footnote>

<footnote n="28"><sup>28</sup>However, effective models can also be trained on a single GPU, given appropriate hyperparameters adaptation – we provide such configurations in our repository.</footnote>

<footnote n="29"><sup>29</sup>We refer to *effectiveness* (or performance) as IR metrics such as MRR or nDCG. *Efficiency*, in the meantime, refers to how “light” a model can be, for instance, in terms of query latency. See Section 6.2 and Section 8.</footnote>
Since we are essentially interested in *retrieval* – and not *re-ranking* – we do not consider cross-encoders, and we only
compare our approach to first-stage rankers – either dense or sparse<fnref n="30" />. We consider two main families of approaches:
i) sparse, which include models like BM25 [91], doc2query-T5 [78] or DeepImpact [69]; ii) dense which include models
like ANCE [114], TAS-B [39] or ColBERT [45]. Table 7 provides an extensive comparison to various baselines.

We report results from corresponding papers and reproduce some of them. While adding a cross-encoder re-ranker
could obviously increase the effectiveness of considered IR systems, we are solely interested in the retrieval step of
multi-stage ranking stacks. Thus, gains in Recall are of primary interest in such a setting. However, in the case of a
sufficiently effective model, we could imagine bypassing the (costly) re-ranking step – the system would only perform a
single *retrieve-and-rank* step. Thus MRR or nDCG at early ranks also constitute our metrics of interest. Note that this
way, we also simplify our evaluation process, as the interaction between first-stage and re-ranking models has to be
taken into account to properly compare approaches [31]<fnref n="31" />. We furthermore report significance paired *t*-tests, using the
ranx library [4], to assess improvements of different SPLADE models<fnref n="32" />.

# 6 SPLADE: EXPERIMENTAL RESULTS

This Section contains parts of the experiments in Formal et al. [25, 26], as well as additional experimental results and
analyses.

## 6.1 Preliminary Results

The sum aggregation in SPLADE (Eq. 12) was initially borrowed from SparTerm. By drawing some parallels with
EPIC and SPARTA, and to some extent ColBERT, we introduce the max-pooling strategy (Eq. 13). Such a mechanism
allows the selection of the most salient contribution among input tokens for every dimension $v$ in the vocabulary space.
Additionally, the magnitude of the weights becomes more invariant to the input length. This simple modification already
leads to significant boosts in effectiveness – around $↑+2.3$ MRR@10 and $↑+1$ R@1000, see Table 3. Improvements are also
statistically significant. SPLADE outperforms models like DeepImpact and becomes competitive with approaches like
COIL-tok – for a much lower indexing and inference cost – but, however, fails to outperform UniCOIL (w/ doc2query-T5
expansion). We have also experimented with other simple aggregation strategies, which didn’t perform as well. For
instance, taking the average instead of the sum – similar to what is done for dense bi-encoders – leads to very poor
results. This might be due to the length penalty being too strong, especially given the log activations.

## 6.2 The Effectiveness-Efficiency Trade-Off

The IR community has been thinking about the effectiveness-efficiency tradeoffs for decades [109]. It has recently
received renewed attraction due to the increasing size of models based on Pre-trained Language Models [95]. Complex
methods – such as cross-encoders based on BERT – perform extremely well but run orders of magnitude more slowly
than term-based approaches. One usually has to sacrifice performance to reduce the time required to present query
results to users.

By jointly optimizing both ranking and regularization losses (Eq. 20), SPLADE can inherently adjust such a trade-off,
by means of the $\lambda = (\lambda_q, \lambda_d)$ hyperparameter. Thus, SPLADE explicitly encompasses efficiency as part of its training –

<footnote n="30">³⁰Results reported on the MS MARCO leaderboard (https://microsoft.github.io/MSMARCO-Passage-Ranking-Submissions/leaderboard/) – for which top systems rely on a multi-stage ranking pipeline – are thus not entirely comparable to the results presented here.</footnote>

<footnote n="31">³¹In Section 7, we, however, rely on cross-encoders for distillation purposes.</footnote>

<footnote n="32">³²Note that we only conduct statistical tests between SPLADE models to validate proposed improvements. We leave for future work a thorough comparison with other sparse methods.</footnote>
<table><caption>Table 3. Performance of SPLADE on MS MARCO (dev) and TREC DL 19. We report results for the most competitive sparse baselines and report results for effective dense approaches as comparison points. <sup>ab</sup> denote significant improvements over the corresponding rows, for a paired *t*-test with *p*-value=0.01.</caption><thead><tr><th rowspan="2">model</th><th colspan="2">MS MARCO dev</th><th colspan="2">TREC DL 2019</th></tr><tr><th>MRR@10</th><th>R@1000</th><th>nDCG@10</th><th>R@1000</th></tr></thead><tbody><tr><td><strong>Dense bi-encoders</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>Bi-encoder (ours)</td><td>31.2</td><td>94.1</td><td>63.7</td><td>71.1</td></tr><tr><td>ANCE [114]</td><td>33.0</td><td>95.9</td><td>64.8</td><td>-</td></tr><tr><td>TCT-ColBERTv2 [57]</td><td>35.9</td><td>97.0</td><td>71.9</td><td>76.0</td></tr><tr><td>TAS-B [39]</td><td>34.7</td><td>97.8</td><td>71.7</td><td>84.3</td></tr><tr><td>RocketQAv2 [89]</td><td>38.8</td><td>98.1</td><td>-</td><td>-</td></tr><tr><td><strong>Sparse</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>BM25 [91]</td><td>18.4</td><td>85.3</td><td>50.6</td><td>74.5</td></tr><tr><td>DeepCT [16]</td><td>24.3</td><td>91.3</td><td>55.1</td><td>75.6</td></tr><tr><td>doc2query-T5 [78]</td><td>27.7</td><td>94.7</td><td>64.2</td><td>82.7</td></tr><tr><td>SparTerm [2]</td><td>27.9</td><td>92.5</td><td>-</td><td>-</td></tr><tr><td>UniCOIL (w/ doc2query-T5) [55]</td><td>35.2</td><td>95.8</td><td>70.2</td><td>82.9</td></tr><tr><td>DeepImpact [69]</td><td>32.6</td><td>94.8</td><td>69.5</td><td>-</td></tr><tr><td>COIL-tok [30]</td><td>34.1</td><td>94.9</td><td>66.0</td><td>-</td></tr><tr><td>COIL-full [30]</td><td>35.5</td><td>96.3</td><td>70.4</td><td>-</td></tr><tr><td>SPLADE-sum (a)</td><td>32.2</td><td>95.5</td><td>66.5</td><td>81.3</td></tr><tr><td>SPLADE-max (b)</td><td>34.5<sup>a</sup></td><td>96.5<sup>a</sup></td><td>67.6<sup>a</sup></td><td>81.1</td></tr></tbody></table>

contrary to standard dense bi-encoders, where ANN algorithms are usually applied in an unsupervised manner once the models are trained. Increasing $\lambda$ generally pushes models to learn sparser representations – therefore leading to more efficient models – usually at the expense of a performance loss. Up to now, we have shown single performance points for different SPLADE models (Table 3), that correspond to the best performance we can obtain, and which usually correspond to lower $\lambda$ (i.e., “denser” models). Figure 3 illustrates such a trade-off between effectiveness (MRR@10 on MS MARCO) and efficiency (as measured by FLOPS), when we vary $\lambda_q$ and $\lambda_d$, for different versions of SPLADE. In the remainder of this document, for each such trade-off plot, we additionally plot the Pareto front (dotted line).

Overall, the MRR@10 tends to increase as the strength of the regularization $\lambda$ diminishes – leading to higher FLOPS. On the contrary, if $\lambda$ is high, models must learn compact representations by selecting a handful of terms in the embedding space. If the compression becomes too important, some key terms might be dropped, thus hurting performance to some extent. Note that expansion saturates at some point: further decreasing $\lambda$ (leading to higher FLOPS, not shown here) does not lead to better performance and usually even worsens the results. As such, the most effective model (MRR@10 = 34.5) is obtained for a middle-valued $\lambda$ (FLOPS=1.23). Interestingly, strongly regularized models still show competitive performance. For instance, a model resulting in an average query (resp. document) length $\ell_0(q) = 11$ (resp. $\ell_0(d) = 33$) reaches 32.6 MRR@10 – results on par with models like DeepImpact, which relies on heavily expanded passages<fnref n="33" />. When comparing different versions of the model, we first observe that SPLADE trained with $\ell_{FLOPS}$ defines the Pareto front in Figure 3. The regularization effect brought by $\ell_{FLOPS}$ compared to $\ell_1$ is clear: for the same efficiency level, the latter’s performance is always lower. In addition to optimizing efficiency, $\ell_{FLOPS}$ implicitly adds an entropy-based regularization, which seems to be beneficial to learning effective representations. We also notice

<footnote n="33">³³Note that for comparison, $\ell_0(d) = 75$ for original documents tokenized with WordPiece.</footnote>
<figure><img src="image_6.png" /><figcaption>Fig. 3. effectiveness-efficiency trade-off on MS MARCO dev set (MRR@10). Models on the right correspond to high $\lambda = (\lambda_q, \lambda_d)$.</figcaption></figure>

the log-saturation effect’s beneficial effect – boosting both effectiveness and efficiency. Finally, it is pretty clear how SPLADE-sum lags behind its max counterpart.

Finally, we plot in Figure 4 the sorted number of passages in each posting list, for different models – for the 8.8M MS MARCO passages. We first observe that a SPLADE model trained *without* regularization is still somewhat sparse<fnref n="34" />, but has overall much larger posting lists – increasing index size and latency. On the other hand, we provide the same distribution for original text documents, when tokenized with WordPiece<fnref n="35" /> – which has a Zipfian profile. The blue and red dotted lines correspond to two SPLADE models with similar final document length $l_0(d)$, trained with respectively $l_{FLOPS}$ and $l_1$. We observe that the FLOPS tends to distribute the posting lists more evenly – thus acting as an entropy regularizer. We also see how decreasing $\lambda$ increases the size of posting lists overall (solid purple line).

## 6.3 Interpretable Representations

By design, SPLADE BoW representations are grounded in the vocabulary. This is ensured by the fact that the weights of the output projection of the MLM head are tied to the input embeddings of the PLM. Thus, each dimension of the representation space actually corresponds to a token of the vocabulary. This results in a valuable byproduct: representations become *interpretable*, and we can – to some extent – provide explanations for the observed rankings. For instance, it is possible to identify terms with the highest weights in a document vector – those contribute the most to the relevance of the document to a given query<fnref n="36" />. Such a property can also be useful for failure analysis, as it becomes easier to understand the causes of low-performing queries. Dense representations – for which the learned latent space is more opaque – do not enjoy such direct benefits. We provide in Table 4 and Table 5 examples of predictions for queries and documents, respectively. We observe four main effects:

* Term-weighting Tokens in the input are given weights according to their estimated *contextual* importance – usually high for what seems to be “important” tokens. The effect of context can also be observed: for instance,

<footnote n="34">³⁴Otherwise, posting lists would have a constant size of $|C|$.</footnote>

<footnote n="35">³⁵Such an index was built for a BM25 model based on WordPiece tokens – instead of actual terms – for which effectiveness only decreases a bit (MRR@10=17.5 instead of 18.4).</footnote>

<footnote n="36">³⁶While we haven’t explicitly worked on explainability aspects of SPLADE, we acknowledge its benefit for IR, as users, as well as political authorities, tend to require more transparency from search and recommendation systems.</footnote>
<figure><img src="image_7.png" /><figcaption>Fig. 4. Index distribution for subterm-based representations – top-2k and top-25k dimensions. We see the regularization effect on the distribution: the lower the $\lambda$, the “denser” the posting lists. Note that around 2k (out of the 30k of the WordPiece vocabulary) dimensions are actually not used by the model (i.e., appear in no document representations after indexing the collection.) – not shown here. These usually correspond to rare or unseen terms on MS MARCO, as well as other special characters of the BERT tokenizer. For SPLADE models, large posting lists also correspond to frequent terms: “is”, “do” and “the”, with respectively 3.2M, 2.9M and 2.4M documents for the blue line.</figcaption></figure>

<table><caption>Table 4. SPLADE BoW prediction for six TREC DL 2019 queries. We highlight in color the predictions for terms in the input (lexical part), and strike-through terms which are not in the final presentation, i.e., terms whose weights are zero (compression). Other terms correspond to expansion. We selected a relatively sparse model ($t_0(q) = 11$ and $t_0(d) = 33$) – see Table 12.</caption><thead><tr><th colspan="2">SPLADE BoW predictions</th></tr></thead><tbody><tr><td>query (id: 1127893) → “ben foster footballer net worth”</td><td>► (foster, 2.65), (ben, 1.89), (worth, 1.78), (footballer, 1.39), (wealth, 0.96), (player, 0.63), (football, 0.56), (benjamin, 0.37), (net, 0.24)</td></tr><tr><td>query (id: 478605) → “port orange what county”</td><td>► BoW → (orange, 2.48), (port, 2.24), (county, 1.67), (counties, 0.03)</td></tr><tr><td>query (id: 1124145) → “truncating meaning”</td><td>► (tr, 2.26), (##cating, 2.24), (##un, 1.89), (##cation, 1.53), (##cate, 1.43), (##cated, 1.09), (meaning, 0.49), (., 0.23), (##cy, 0.13)</td></tr><tr><td>query (id: 646207) → “what does production design entail”</td><td>► (production, 2.46), (design, 2.18), (en, 1.85), (##tail, 1.79), (producer, 1.03), (designer, 0.77), (designed, 0.68), (factory, 0.18), (product, 0.05)</td></tr><tr><td>query (id: 1037798) → “who is robert gray”</td><td>► (gray, 2.61), (robert, 2.31), (grey, 1.59), (who, 1.04), (strange, 0.8), (he, 0.73), (kent, 0.25), (bobby, 0.09)</td></tr><tr><td>query (id: 104861) → “cost of interior concrete flooring”</td><td>► (concrete, 2.16), (interior, 2.11), (floor, 1.83), ($, 1.26), (cost, 1.01), (cement, 0.73), (inside, 0.64), (price, 0.62), (internal, 0.14)</td></tr></tbody></table>

the term “concrete” (Table 5) is given a high score (2.08) for a document about floor materials, while it has a low score (0.03) in the last document about another topic. We also notice that some apparently non-important tokens actually have high weights (e.g., “who” for the query “who is robert gray”) – Table 4. PLM are therefore not only useful to bridge the vocabulary gap but also better estimate the importance of terms in IR models. Overall,
<figure><img src="image_8.png" /><figcaption>Fig. 5. Probability *P* of a token to be “dropped” – i.e., appearing in a document but not its SPLADE vector – estimated on a sample of 20k MS MARCO passages, against decreasing document frequency (top-2k terms) – x-axis binned. Note how very frequent terms (bins (0, 100], (100, 200]) have a higher probability of being filtered out.</figcaption></figure>

such a mechanism conditions the model to rely on *exact matching* – which might be harder to learn for dense
bi-encoders.

* **Compression** Some input tokens are discarded – the model predicts a zero weight for this dimension. This effect is due to the sparsity regularization, which identifies irrelevant tokens to remove. This is usually the case for stopwords, similar to standard practices in term-based approaches – but not only. Thus, we see for the first document in Table 5, the birth date ("*May 10, 1755*") is actually deemed non-important by the model. This effect is also stronger on documents, as removing terms from queries – which are already rather short – is riskier and can greatly impact performance. The filtering effect on frequent terms is further illustrated in Figure 5.

* **Expansion** Tokens not in the original input are predicted non-zero weights. This usually corresponds to related tokens, such as synonyms, that allow bridging the vocabulary gap between queries and documents. For instance, the tokens "price" or "cement" are both expanded on the last query in Table 4, and the corresponding relevant document in Table 5. Note that expansion is not always accurate: we see how tokens like "grey" or "bobby" are wrongly predicted for the query "who is robert gray". This is a side effect of expansion, which happens to predict "wacky" weights [67].

* **Stemming** This is a specific side effect of expansion, where linguistic variations of a token are predicted – as a result of the WordPiece-based representation space. Stemming – which has a long-standing history in IR – is thus (partially) taken into account in the model, and *learned* at training time. It is shown in the third row of Table 4, where various suffixes of *"trun"* are predicted by SPLADE (*"truncating"* → *"truncation"*, *"truncate"*, *"truncated"*).

We further illustrate in Figure 6 how the model provides some level of explanation for the ordering of documents – including failure cases. Relevant documents are placed on top of the ranking, as the model correctly weighs important query subwords (“*con*”, “##tour”, “*pl*”, “###owing”, “###ow”). The first document with *rel* < 2 (*rank*=13) gives less importance to those words, and does not predict any word around the concept of “*reducing*” (highlighted in orange). In the meantime, the document at *rank*=22 is irrelevant, as the model (correctly though) gives high weights to “*con*” and “##tour”, as well as query words related to “*reduce*”.
<table><caption>Table 5. SPLADE BoW prediction for three documents. The first two are respectively relevant for the two last queries in Table 4. Compared to queries, we see how documents are far more compressed, due to the regularization constraint. We use the same model as the one in Table 4.</caption><thead><tr><th>SPLADE BoW predictions</th></tr></thead><tbody><tr><td>document (id: 8760871) → “Atlantic Ocean, United States. Robert Gray, (born May 10, 1755, Tiverton, R.I. died summer 1806 at sea near eastern U.S. coast) captain of the first U.S. ship to circumnavigate the globe and explorer of the Columbia River.”<br>► (gray, 2.55), (robert, 2.19), (atlantic, 2.17), (ocean, 1.74), (grey, 1.73), (##vert, 1.59), (globe, 1.53), (first, 1.4), (captain, 1.37), (who, 1.31), (ship, 1.25), (died, 1.2), (explorer, 1.12), (columbia, 1.0), (born, 1.0), (sea, 0.96), (he, 0.77), (##vi, 0.66), (states, 0.65), (death, 0.64), (##rc, 0.63), (discovery, 0.59), (commanded, 0.55), (##gate, 0.45), (served, 0.44), (strange, 0.4), (river, 0.39), (coast, 0.38), (invented, 0.36), (united, 0.34), (us, 0.24), (boat, 0.22), (ti, 0.21), (american, 0.16), (##on, 0.06), (deceased, 0.02)</td></tr><tr><td>document (id: 1017092) → “Some things that may add to that cost are: site and sub-base preparation, site access, small floors under 500 sq. ft., and thicker concrete. Integral colored concrete floor cost: $3.75 dollars per square foot, that includes the basi concrete floor package and adding color to the concrete.”</td></tr><tr><td>► ($, 2.13), (concrete, 2.08), (integral, 2.02), (floor, 1.89), (cost, 1.83), (add, 1.43), (square, 1.38), (color, 1.22), (small, 1.16), (foot, 1.06), (package, 1.01), (price, 0.97), (per, 0.82), (sub, 0.73), (paint, 0.7), (site, 0.67), (cement, 0.64), (preparation, 0.46), (thickness, 0.31), (thick, 0.21), (base, 0.07), (building, 0.06), (large, 0.01)</td></tr><tr><td>document (id: 8530) → “An hypothesis is a specific statement of prediction. It describes in concrete (rather than theoretical) terms what you expect will happen in your study. Not all studies have hypotheses. Sometimes a study is designed to be exploratory (see inductive research). There is no formal hypothesis, and perhaps the purpose of the study is to explore some area more thoroughly in order to develop some specific hypothesis or prediction that can be tested in future research.”</td></tr><tr><td>► (hypothesis, 2.64), (theory, 1.85), (study, 1.83), (prediction, 1.75), (statement, 1.59), (predict, 1.33), (##oth, 1.21), (specific, 1.07), (research, 1.03), (candidate, 0.92), (., 0.9), (##yp, 0.88), (science, 0.86), (an, 0.86), (expect, 0.78), (in, 0.76), (##ora, 0.69), (purpose, 0.67), (studies, 0.65), (theoretical, 0.6), (##uc, 0.37), (mathematical, 0.37), (scientific, 0.36), (formal, 0.35), (##tory, 0.34), (designed, 0.25), (predicted, 0.2), (contrast, 0.18), (expected, 0.18), (ind, 0.12), (h, 0.11), (assumptions, 0.08), (##eses, 0.07), (terms, 0.04), (concrete, 0.03), (describing, 0.01)</td></tr></tbody></table>

# 7 TOWARDS EFFECTIVE MODELS

Neural retrievers based on dense representations combined with ANN search have recently received a lot of attention, owing their success to distillation and/or a better sampling of examples for training. While sparse approaches have also gained increasing attraction, a lesser effort has been put into the training of such models, making it unclear whether they would experience the same boosts as dense architectures. We thus wonder if these improvements are *additive*, in the sense that if a model performs better than another in a “base” training setting, would we still observe the same hierarchy with distillation? Answering such a question would allow decoupling architectures from training innovations when comparing neural retrievers.

In this Section, we therefore extensively study the influence of various training strategies on SPLADE *effectiveness*. More precisely, we study the effect of the three main innovations, namely i) distillation, ii) hard negative mining, iii) as
<table><caption>Table 6. SPLADE BoW prediction for *ranked* documents for the TREC DL 2019 query "*what can contour plowing reduce*". We show the first two documents (*rel*=2), as well as the first document with *rel*<2 (at *rank*=13) and the first non-relevant document (at *rank*=22). We use the same model as the one in Table 4</caption><thead><tr><th></th><th>query (id: 1121402) → "what can contour plowing reduce"</th></tr></thead><tbody><tr><td>► (##tour, 2.38), (pl, 1.93), (reduce, 1.87), (con, 1.84), (benefit, 1.06), (##owing, 1.02), (##ow, 0.87), (reduction, 0.84), (lower, 0.83), (help, 0.8), (prevent, 0.46), (remedy, 0.35), (reducing, 0.34)</td></tr><tr><td>↓ rank</td><td>document and SPLADE BoW prediction</td></tr><tr><td>1 rel=2</td><td>document (id: 188213) → "Contour plowing is the act of farming on a hill or a contoured area. The plow follows the contours of the land horizontally helping to reduce runoff of water. This practice helps to prevent soil erosion in hilly and contoured areas by capturing the water runoff using water breaks to keep water contained. Contour plowing is also known as contour farming, and is a sustainable form of agriculture."<br>► (##tour, 2.51), (pl, 2.32), (con, 2.26), (##ow, 1.91), (##owing, 1.6), (hill, 1.52), (sustainable, 1.45), (follow, 1.43), (farmer, 1.2), (land, 1.14), (farming, 1.0), (following, 0.98), (follows, 0.92), (farm, 0.83), (agricultural, 0.77), (ridge, 0.76), (##ed, 0.76), (runoff, 0.65), (erosion, 0.64), (soil, 0.62), (break, 0.58), (slate, 0.55), (hilly, 0.55), (capture, 0.55), (capturing, 0.54), (is', 0.5), (vertical, 0.43), (terrain', 0.42), (horizontally, 0.34), (vertically, 0.3), (reduce, 0.28), (act, 0.22), (along, 0.09), (agriculture, 0.08), (area, 0.03)</td></tr><tr><td>2 rel=2</td><td>document (id: 8052620) → "2016-09-28T08:13:26-04:00. Farmers can reduce soil erosion by contour plowing. The correct option among all the options that are given in the question is the second option or option B. The other choices given in the question are incorrect and can be neglected. I hope that this is the answer that has actually come to your desired help."<br>► (erosion, 2.1), (##tour, 1.84), (option, 1.77), (pl, 1.76), (soil, 1.69), (reduce, 1.65), (con, 1.58), (farmer, 1.56), (can, 1.29), (question, 1.16), (prevent, 1.09), (farm, 0.93), (correct, 0.89), (##owing, 0.89), (benefit, 0.78), (reduction, 0.76), (farmers, 0.61), (options, 0.6), (prevention, 0.6), (choice, 0.55), (##ow, 0.52), (help, 0.51), (grass, 0.46), (crop, 0.44), (remedy, 0.41), (alternatives, 0.41), (second, 0.4), (best, 0.33), (agricultural, 0.32), (should, 0.3), (bearing, 0.22), (avoid, 0.1), (lower, 0.05), (increase, 0.05), (wrong, 0.05)</td></tr><tr><td>13 rel=1</td><td>document (id: 188217) → "Definitions for Contour plowing. Contour plowing. Contour ploughing or contour farming is the farming practice of ploughing and/or planting across a slope following its elevation contour lines."<br>► (##tour, 2.54), (con, 2.11), (pl, 1.9), (##ough, 1.35), (farmer, 1.15), (##ing, 1.12), (elevation, 1.0), (##owing, 0.83), (farming, 0.82), (slate, 0.7), (##ow, 0.7), (farm, 0.58), (harvest, 0.45), (agricultural, 0.33), (crop, 0.32), (slope, 0.28), (practice, 0.13), (ridge, 0.09), (reed, 0.04)</td></tr><tr><td>22 rel=0</td><td>document (id: 1020003) → "Tweet. Liposuction can improve your body contours by removing excess fat from thighs, hips, buttocks, abdomen, waist, upper arms, back, chest, face, and neck."<br>► (lip, 2.58), (##tour, 2.08), (improve, 1.83), (fat, 1.8), (increase, 1.66), (help, 1.58), (waist, 1.47), (con, 1.42), (thighs, 1.4), (body, 1.35), (##uc, 1.31), (chest, 1.29), (face, 1.24), (benefit, 1.24), (##ocks, 1.2), (improvement, 1.09), (back, 1.01), (hip, 0.96), (facing, 0.95), (remove, 0.95), (your, 0.91), (upper, 0.82), (reduce, 0.81), (assist, 0.8), (neck, 0.75), (straight, 0.67), (fatty, 0.61), (butt, 0.56), (##os, 0.55), (good, 0.55), (thigh, 0.52), (clean, 0.49), (can, 0.48), (remedy, 0.46), (restore, 0.45), (hips, 0.45), (affect, 0.42), (boost, 0.4), (stomach, 0.38), (exercise, 0.38), (rid, 0.37), (teach, 0.37), (enhance, 0.32), (effect, 0.3), (excess, 0.3), (removal, 0.28), (posture, 0.27), (better, 0.26), (excessive, 0.23), (arm, 0.22), (health, 0.2), (prevent, 0.2), (cure, 0.19), (torso, 0.19), (alignment, 0.16), (for, 0.13), (assistance, 0.08), (prevention, 0.05), (relieve, 0.05), (arms, 0.04), (##tion, 0.01), (recover, 0.01), (heal, 0.01), (change, 0.01)</td></tr></tbody></table>
well as the PLM initialization. We furthermore analyze the link between effectiveness and efficiency and show how findings from Section 6.2 still apply to the considered training settings.

## 7.1 Setting

Let us consider the “default” SPLADE model, defined by Eq. 13 and trained with Eq. 20, where $\mathcal{L}_{rank}$ is the contrastive InfoNCE loss, based on BM25 and IBN negatives. To further distinguish training settings, we thus note:

$$ \mathcal{L} = \mathcal{L}_{InfoNCE,BM25} + \lambda_q \ell_{FLOPS}^q + \lambda_d \ell_{FLOPS}^d \quad (21) $$

In the following, we detail several training tricks previously introduced for dense models that can seamlessly be applied to SPLADE. All the extensions rely on a simple modification of Eq. 21, either by modifying the ranking loss, the source of hard negatives, or a combination of both. We also discuss the initialization strategy, where models can further be improved by relying on PLM checkpoints that have been pre-trained on retrieval-oriented tasks.

### 7.1.1 Distillation.
Distillation of (costly) well-performing models like cross-encoders has been shown to greatly improve the effectiveness of various more efficient neural ranking architectures [39, 40, 57, 94]. We rely on the MarginMSE loss [40] – Mean Squared Error between the positive-negative margins of a cross-encoder teacher and the student – and train SPLADE by optimizing:

$$ \mathcal{L} = \mathcal{L}_{MarginMSE,BM25} + \lambda_q \ell_{FLOPS}^q + \lambda_d \ell_{FLOPS}^d \quad (22) $$

MarginMSE distillation constitutes our default strategy for all the settings introduced below.

### 7.1.2 Mining Hard Negatives.
The standard setting using BM25 negatives is limited, and the benefit of using more informative negatives has been highlighted in several prior works [85, 89, 114, 120]. Note that by considering MarginMSE distillation as the basis for hard negative mining, we remove the need to resort to denoising, i.e., filtering noisy false negatives [85, 89].

**Self-mining** Following ANCE [114], which dynamically samples negatives from the model that is being trained, we propose to follow a simpler two-step strategy that has also been adopted in prior works (e.g., Lin et al. [57]):

* (step 1) we initially train a SPLADE model, as well as a cross-encoder re-ranker, in the previously introduced distillation setting;

* (step 2) we then generate triplets using the SPLADE model trained in (step 1), and use the cross-encoder to generate the scores needed for MarginMSE, and go for another round of training.

This simply leads to a distillation strategy where mined pairs are supposed to be of better “quality” compared to BM25. The model is trained by optimizing:

$$ \mathcal{L} = \mathcal{L}_{MarginMSE, self} + \lambda_q \ell_{FLOPS}^q + \lambda_d \ell_{FLOPS}^d \quad (23) $$

**Ensemble-mining** While the self-mining setting provides a better sampling strategy compared to BM25, it might be limited, as it only considers a single model, and one could wonder if using various types of models to mine negatives for (step 2) could be beneficial. We thus rely on a dataset containing for each query: i) the top-50 hard negatives mined from BM25 and each of 12 various dense retrievers, ii) the scores coming from a cross-encoder for each available $(q, d^+, d^-)$ to perform MarginMSE knowledge distillation. Our training loss thus becomes:
$$ \mathcal{L} = \mathcal{L}_{\text{MarginMSE,ensemble}} + \lambda_q f_{\text{FLOPS}}^q + \lambda_d f_{\text{FLOPS}}^d \quad (24) $$

### 7.1.3 Pre-training
NLP has recently borrowed ideas from contrastive learning techniques in Computer Vision, with the goal of learning high-quality sentence or document representations without annotation [29, 41, 87, 107, 113]. The general idea consists in designing pre-training tasks, that are better suited for subsequently training neural retrievers. Most approaches in IR are very similar to CoCondenser [29], which contrastively learns the embedding space from spans of documents – two spans from the same document are considered as positives, the other documents in the batch as negatives. We thus simply consider using such pre-trained checkpoint to initialize SPLADE, in the scenarios described above. Note that while CoCondenser is not directly optimized for sparse retrieval, its learned embedding space might contain more informative knowledge for retrieval compared to models pre-trained with Masked Language Modeling.

## 7.2 Experiments

We describe our experimental setting in Section 7.2.1, and present the results of our approach in Section 7.2.2.

### 7.2.1 Experimental Setting
We conduct an extensive experimental study, by training and evaluating several models, for each setting introduced in Section 7.1. As we provide improvements over SPLADE, we refer to all our strategies under the same alias, coined SPLADE++. Overall, we follow the training and evaluation workflow described in Section 5. Especially, we use the same hyperparameters, as well as the validation strategy, but modify the components introduced in Section 7.1 – accordingly the ranking loss for distillation, the set of (hard) negatives, and/or the PLM initialization. We thus consider the following settings: SPLADE, which corresponds to the original setting (Eq. 21); DistilMSE which relies on Eq. 22 for training<fnref n="37" />; SelfDistil where models are trained using Eq. 23; EnsembleDistil<fnref n="38" /> which rather makes use of Eq. 24; and finally CoCondenser-SelfDistil and CoCondenser-EnsembleDistil which correspond to the two latter settings, where the model has additionally been initialized from a pre-trained CoCondenser checkpoint<fnref n="39" />.

For each setting, we train five models, corresponding to different values of the regularization magnitude ($\lambda_q, \lambda_d$) in, e.g., Eq. 21 – thus providing various trade-offs between effectiveness and efficiency. Note that, as each procedure either modifies the loss or the training pairs, the range taken by loss values slightly differs. We, therefore, need to adapt $\lambda$ in each case; we simply rely on grid-search and keep five configurations that cover a broad range of effective and efficient models. Note that, as we only compare the efficiency of SPLADE models, we only report the FLOPS and not other measures such as query latency.

### 7.2.2 Results

**Overall results.** Results on MS MARCO dev and TREC DL 2019 are given in Table 7. For each, we report the best model (among the five that are trained) with a FLOPS value inferior to 3 – similar to results reported in Section 6. The interplay between effectiveness and efficiency is given in Fig. 6: we report MRR@10 on MS MARCO dev vs FLOPS, for the five configurations in each training setting. Overall, we observe that:

(1) SPLADE is able to take advantage of various training strategies to increase its effectiveness;

<footnote n="37">³⁷We rely on the cross-encoder teacher scores provided at: https://github.com/sebastian-hofstaetter/neural-ranking-kd.</footnote>

<footnote n="38">³⁸We rely on the msmarco-hard-negatives dataset: https://huggingface.co/datasets/sentence-transformers/msmarco-hard-negatives, available in the Sentence Transformers library [88].</footnote>

<footnote n="39">³⁹Available via HuggingFace: https://huggingface.co/Luyu/co-condenser-marco.</footnote>
<table><caption>Table 7. Evaluation on MS MARCO passage retrieval (dev set) and TREC DL 2019. We distinguish methods between simple training and training that benefit from various improvements. Note that at the time when experiments were conducted, other sparse methods were trained in a simple training setting. We refer to [77] for a detailed analysis of the role of distillation and hard negatives on various sparse rankers. abcdef denote significant improvements over the corresponding rows for a paired t-test with p-value=0.01.</caption><thead><tr><th rowspan="2">model</th><th colspan="2">MS MARCO dev</th><th colspan="2">TREC DL 2019</th></tr><tr><th>MRR@10</th><th>R1000</th><th>nDCG@10</th><th>R1000</th></tr></thead><tbody><tr><td colspan="5"><strong>► Simple training</strong></td></tr><tr><td>BM25 [91]</td><td>18.4</td><td>85.3</td><td>50.6</td><td>74.5</td></tr><tr><td>DeepCT [16]</td><td>24.3</td><td>91.3</td><td>55.1</td><td>75.6</td></tr><tr><td>doc2query-T5 [78]</td><td>27.7</td><td>94.7</td><td>64.2</td><td>82.7</td></tr><tr><td>SparTerm [2]</td><td>27.9</td><td>92.5</td><td>-</td><td>-</td></tr><tr><td>UniCOIL (w/ doc2query-T5) [55]</td><td>35.2</td><td>95.8</td><td>70.2</td><td>82.9</td></tr><tr><td>DeepImpact [69]</td><td>32.6</td><td>94.8</td><td>69.5</td><td>-</td></tr><tr><td>COIL-tok [30]</td><td>34.1</td><td>94.9</td><td>66.0</td><td>-</td></tr><tr><td>COIL-full [30]</td><td>35.5</td><td>96.3</td><td>70.4</td><td>-</td></tr><tr><td>Bi-encoder (ours)</td><td>31.2</td><td>94.1</td><td>63.7</td><td>71.1</td></tr><tr><td>ColBERT [56]</td><td>36.8</td><td>96.9</td><td>-</td><td>-</td></tr><tr><td>SPLADE (a)</td><td>34.5</td><td>96.5</td><td>67.6</td><td>81.1</td></tr><tr><td colspan="5"><strong>► Distillation, negative mining or pre-training</strong></td></tr><tr><td>ANCE [114]</td><td>33.0</td><td>95.9</td><td>64.8</td><td>-</td></tr><tr><td>TCT-ColBERTv2 [57]</td><td>35.9</td><td>97.0</td><td>71.9</td><td>76.0</td></tr><tr><td>TAS-B [39]</td><td>34.7</td><td>97.8</td><td>71.7</td><td>84.3</td></tr><tr><td>RocketQAv2 [89]</td><td>38.8</td><td>98.1</td><td>-</td><td>-</td></tr><tr><td>CoCondenser [29]</td><td>38.2</td><td>98.4</td><td>-</td><td>-</td></tr><tr><td>Contriever [41]</td><td>34.1</td><td>97.9</td><td>67.6</td><td>84.3</td></tr><tr><td>AR2 [121]</td><td>39.5</td><td>98.6</td><td>-</td><td>-</td></tr><tr><td>ColBERTv2 [94]</td><td>39.7</td><td>98.4</td><td>-</td><td>-</td></tr><tr><td colspan="5"><strong>► Our methods: SPLADE++</strong></td></tr><tr><td>DistilMSE (b)</td><td>35.8<sup>a</sup></td><td>97.8<sup>a</sup></td><td>72.9</td><td>85.9</td></tr><tr><td>SelfDistil (c)</td><td>36.7<sup>ab</sup></td><td>98.0<sup>a</sup></td><td>72.6</td><td>87.9<sup>a</sup></td></tr><tr><td>EnsembleDistil (d)</td><td>36.8<sup>ab</sup></td><td>97.8<sup>a</sup></td><td>70.7</td><td>86.7</td></tr><tr><td>CoCondenser-SelfDistil<sup>†</sup> (e)</td><td>37.6<sup>abcd</sup></td><td>98.3<sup>abcd</sup></td><td>73.2<sup>a</sup></td><td>87.7<sup>ab</sup></td></tr><tr><td>CoCondenser-EnsembleDistil<sup>‡</sup> (f)</td><td>38.0<sup>abcd</sup></td><td>98.2<sup>abd</sup></td><td>72.7<sup>a</sup></td><td>87.1<sup>a</sup></td></tr></tbody></table>

(2) performance boosts are additive;

(3) our settings lead to competitive results on MS MARCO (in-domain evaluation), e.g., CoCondenser-EnsembleDistil reaches 38 MRR@10;

(4) model effectiveness is linked to efficiency for all the considered settings (the sparser, the less effective).

More specifically, we observe from Table 7 that the use of distillation (DistilMSE) offers the largest boost in effectiveness across all settings (↑+1.3 MRR@10). When combining distillation with hard negative mining, we note that the SelfDistil case seems to be the most effective. However, when changing the initialization checkpoint to CoCondenser, we observe the reverse trend, where CoCondenser-EnsembleDistil is able to outperform its counterpart. Distillation with hard negatives is beneficial, but the exact “optimal” setting depends on the pre-training strategy. We also note that improvements are less clear when inspecting other metrics like R@1000 or nDCG@10 on TREC DL 2019.
In Figure 6, we analyze the interplay between effectiveness and efficiency (in terms of FLOPS) on MS MARCO dev set. There is an overall trend that more expressive models tend to be more effective across all training settings. We also observe additivity in improvements, with the best model on MS MARCO (CoCondenser-EnsembleDistil) taking advantage of distillation, ensemble mining, and pre-trained checkpoints altogether.

<figure><img src="image_9.png" /><figcaption>Fig. 7. Additivity study: each subplot corresponds to a given training setting, and the dense bi-encoder performance is shown with the dotted black line. It reaches 31.3, 33.5, 35.3, and 35.5 MRR@10 on MS MARCO for, respectively, the “standard”, DistilMSE, SelfDistil, and EnsembleDistil training settings.</figcaption></figure>

<figure><img src="image_10.png" /></figure>

**Additivity.** To further demonstrate the “additivity” of training settings, we train for each setting$^{40}$, a dense bi-encoder with the same DistilBERT backbone model. We rely on mean pooling of the contextualized embeddings and follow standard practices to train them. We plot in Figure 7 the MRR@10 achieved by the dense bi-encoders, alongside

<footnote>$^{40}$Except the ones based on CoCondenser.</footnote>
different trade-offs for SPLADE models. Results suggest that SPLADE can always outperform its dense counterpart – and this, regardless of the training setting – given enough expansion power, i.e., up to a given FLOPS for the last two. This suggests that SPLADE architectural prior, based on sparse expansion and term-matching, is somewhat well suited for IR, when compared to the dense matching of semantic vectors. This study, however, needs to be complemented with the various pre-training techniques introduced for dense models: would we still observe the same gap? This further questions how to adapt pre-training techniques for models such as SPLADE – which is not as straightforward [97].

**Zero-Shot Evaluation.** In Information Retrieval, the question of generalization has often been eluded, due to the robust and long-standing performance of term-based approaches. However, with the recent advent of neural retrieval based on PLM, the generalization issue has become as relevant as ever. Initially evaluated on in-domain settings (like MS MARCO), where train and test queries follow the same distribution, conclusions became more contrasted when Thakur et al. released the zero-shot BEIR benchmark [101] – in which some models like DPR achieve lower overall performance compared to unsupervised term-based approaches like BM25. Knowing that many production systems use (or will use) models based on PLM (e.g., Xiong et al. [114]), while being exposed to new documents and queries every day, robustness is thus a critical aspect that must be assessed.

The BEIR benchmark consists of a test suite of 18 datasets – each containing documents, queries, and corresponding *qrels*. They are used to evaluate models in a *zero-shot* setting, i.e., without any sort of training based on those datasets.

We thus assess the zero-shot performance of our models on the BEIR benchmark<fnref n="41" />. For comparison with other approaches, we rely on the subset of 13 datasets that are readily available. Thus, we do not consider CQADupstack, BioASQ, Signal-1M, TREC-NEWS and Robust04; otherwise, we evaluate our models on the complete benchmark (18 datasets). We report the results from the models introduced in Section 7, and show the effect of various training strategies on generalization capabilities. We furthermore analyze the link between effectiveness and efficiency in Figure 8. Please note that for BEIR, averaging metrics over multiple datasets is questionable [100] – but the observed trends are, however, useful to compare various instances of SPLADE models. We nevertheless provide an evaluation on every dataset in Table 10.

<figure><img src="image_11.png" /><figcaption>Fig. 8. effectiveness-efficiency trade-off on BEIR (18 datasets – results are thus not comparable to Table 8). We report mean nDCG@10.</figcaption></figure>

<footnote n="41">⁴¹We are aware of other datasets that serve the same purpose, such as the LoTTE benchmark [94], specifically tailored for OOD retrieval of natural search queries over long-tail topics.</footnote>
<table><caption>Table 8. Mean nDCG@10 on the subset of 13 BEIR datasets. SPLADE++$^{‡,†}$ respectively correspond to our best SPLADE models CoCondenser-EnsembleDistil$^{‡}$ and CoCondenser-SelfDistil$^{†}$.</caption><thead><tr><th></th><th>BM25</th><th>TAS-B</th><th>Contriever</th><th>ColBERTv2</th><th>SPLADE</th><th>SPLADE++<sup>‡</sup></th><th>SPLADE++<sup>†</sup></th></tr></thead><tbody><tr><td>nDCG@10</td><td>43.7</td><td>43.7</td><td>47.5</td><td>49.7</td><td>45.7</td><td>50.4</td><td>50.7</td></tr><tr><td></td><td colspan="4" style="text-align: center;">SPLADE++<sup>‡</sup> + BM25 SPLADE++<sup>†</sup> + BM25</td><td></td><td></td><td></td></tr><tr><td>nDCG@10</td><td colspan="3" style="text-align: center;">52.1</td><td colspan="3" style="text-align: center;">52.1</td><td></td></tr></tbody></table>

<caption>Table 9. Mean nDCG@10 on the subset of 13 BEIR datasets for a simple combination (sum) of SPLADE++ and BM25.</caption>

Overall, we observe that:

(1) the SPLADE models introduced in Section 7.1 lead to state-of-the-art results on zero-shot evaluation;

(2) the link between effectiveness and efficiency still applies in out-of-domain evaluation;

(3) out-of-domain improvements are consistent with the ones observed on MS MARCO.

In Figure 8, we indeed observe a trend similar to the one observed in Section 7, where effectiveness comes with higher complexity. However, the SelfDistil settings seem to be better suited for generalization – they both respectively outperform their EnsembleDistil counterparts, also see Table 8 – contrary to what is observed on MS MARCO. One reason for this behavior might be the use of only dense retrievers in the latter case. As such models have been shown to overfit on MS MARCO, the mined negatives might be too specialized on this dataset, thus hurting the generalization capabilities of subsequently trained models.

We additionally report in Table 8 the BEIR results for our two CoCondenser settings, reaching state-of-the-art results on zero-shot evaluation$^{42}$, strengthening the observation that sparse retrieval models seem to be better able to generalize [27, 74, 101]. Note that the “standard” SPLADE is already competitive, while our SPLADE CoCondenser-SelfDistil is the best overall on five datasets (out of the 13), among other state-of-the-art baselines. Note that BM25 still performs well, and many baselines reported in Thakur et al. [101] achieve lower performance. If we look at individual results, we see that SPLADE models largely outperform BM25 on datasets like Robust04 or TREC-COVID.

We also report in Table 9 the results of the two previous approaches combined with BM25 – with a simple sum. We use a simple score combination inspired by Lin et al. [57], where i) documents not present in the top-100 of a model are assigned its smallest score, ii) and normalization is based on the maximum and minimum scores of the method. Finally, we simply sum the scores of the two methods, assigning equal weight to both. Additional gains can be obtained, showing that pure lexical approaches are still somehow complementary to sparse neural models, especially in a zero-shot setting. While such results have been shown for dense bi-encoders [8, 11, 52, 61, 63, 110], this is more surprising for sparse models.

# 8 TOWARDS EFFICIENT MODELS

Section 7 shows how SPLADE can be made *effective*. We now turn our attention to *efficiency* considerations. In Section 8.1, we discuss two main issues inherent to SPLADE that are central to achieving truly efficient representations (Section 8.2).

<footnote>$^{42}$We acknowledge that recent works also achieve impressive zero-shot performance [20, 84]. Additionally, domain adaptation techniques have been shown to be effective on the BEIR benchmark [108].</footnote>
## 8.1 SPLADE Limitations

8.1.1 *The Impact of the MLM Head.* So far, our experiments have been mostly based on a DistilBERT backbone model (66M parameters, |V| = 30k). We provide additional experiments in Table 11, for which we change the base PLM model, including i) BERT (110M parameters, |V| = 30k), ii) RoBERTa [60] (125M parameters, |V| = 50k, and iii) ELECTRA [13] (33M parameters, |V| = 30k).

We first observe that SPLADE based on the ELECTRA generator, with a number of parameters twice lower as the base SPLADE, already outperforms models like DeepImpact. Additionally, increasing the number of parameters (by switching from DistilBERT to BERT) doesn't seem to help on MS MARCO – with performance actually lower overall, which might be caused by a slight overfitting<sup>43</sup>. We further show in Figure 9 the same type of trade-off shown in earlier Sections – obtained by training models with varying λ.

Finally, we note how SPLADE is not able to take advantage of the RoBERTa checkpoint. The models are harder to train – and sometimes even diverge. This is mostly due to the ill-conditioned distribution of the MLM logits for SPLADE, as noted by Nair et al. [75]. Indeed, as we aim to learn sparse representations, the distribution of the MLM weights at initialization plays a key role in both the convergence and performance of SPLADE. In order to pre-condition the usage of any PLM with SPLADE, we propose an intermediate unsupervised pre-training<sup>44</sup> step – i.e., before fine-tuning – which trains the model with a combination of the Mask Language Modeling task and the FLOPS regularization (Eq. 17) applied on the logits of the MLM head – dubbed MLM-FLOPS. More specifically, the MLM logits first go through

<table><caption>Table 10. nDCG@10 on BEIR for all the datasets (18). For comparison, we report results directly from corresponding papers, where the evaluation is generally done on the subset of 13 readily available BEIR datasets.</caption><thead><tr><th>Model (→)</th><th>BM25</th><th>TAS-B</th><th>Contriever</th><th>ColBERTv2</th><th>SPLADE</th><th>SPLADE++‡</th><th>SPLADE++†</th></tr></thead><tbody><tr><td>Dataset (↓)</td><td colspan="7">nDCG@10</td></tr><tr><td>TREC-COVID</td><td>65.6</td><td>48.1</td><td>59.6</td><td><strong>73.8</strong></td><td>62.9</td><td>72.1</td><td>72.5</td></tr><tr><td>BioASQ</td><td>46.5</td><td>38.3</td><td>-</td><td>-</td><td>43.4</td><td>50.5</td><td><strong>50.8</strong></td></tr><tr><td>NFCorpus</td><td>32.5</td><td>31.9</td><td>32.8</td><td>33.8</td><td>31.9</td><td><strong>34.6</strong></td><td>34.5</td></tr><tr><td>NQ</td><td>32.9</td><td>46.3</td><td>49.8</td><td><strong>56.2</strong></td><td>45.9</td><td>53.5</td><td>53.3</td></tr><tr><td>HotpotQA</td><td>60.3</td><td>58.4</td><td>63.8</td><td>66.7</td><td>65.4</td><td>68.4</td><td><strong>69.3</strong></td></tr><tr><td>FiQA-2018</td><td>23.6</td><td>30.0</td><td>32.9</td><td><strong>35.6</strong></td><td>27.1</td><td>34.7</td><td>34.9</td></tr><tr><td>Signal-1M (RT)</td><td><strong>33.0</strong></td><td>28.9</td><td>-</td><td>-</td><td>29.6</td><td>30.2</td><td>30.9</td></tr><tr><td>TREC-NEWS</td><td>39.8</td><td>37.7</td><td>-</td><td>-</td><td>36.3</td><td><strong>42.3</strong></td><td>41.9</td></tr><tr><td>Robust04</td><td>40.8</td><td>42.7</td><td>-</td><td>-</td><td>41.0</td><td>46.0</td><td><strong>48.5</strong></td></tr><tr><td>ArguAna</td><td>31.5</td><td>42.9</td><td>44.6</td><td>46.3</td><td>46.7</td><td>51.3</td><td><strong>51.8</strong></td></tr><tr><td>Touché-2020</td><td><strong>36.7</strong></td><td>16.2</td><td>23.0</td><td>26.3</td><td>22.0</td><td>25.5</td><td>24.2</td></tr><tr><td>CQADupStack</td><td>29.9</td><td>31.4</td><td>34.5</td><td>-</td><td>32.5</td><td>33.4</td><td><strong>35.4</strong></td></tr><tr><td>Quora</td><td>78.9</td><td>83.5</td><td><strong>86.5</strong></td><td>85.2</td><td>81.1</td><td>83.4</td><td>84.9</td></tr><tr><td>DBPedia</td><td>31.3</td><td>38.4</td><td>41.3</td><td><strong>44.6</strong></td><td>36.4</td><td>43.7</td><td>43.6</td></tr><tr><td>SCIDOCs</td><td>15.8</td><td>14.9</td><td><strong>16.5</strong></td><td>15.4</td><td>14.2</td><td>15.8</td><td>16.1</td></tr><tr><td>FEVER</td><td>75.3</td><td>70.0</td><td>75.8</td><td>78.5</td><td>76.2</td><td>79.2</td><td><strong>79.6</strong></td></tr><tr><td>Climate-FEVER</td><td>21.3</td><td>22.8</td><td>23.7</td><td>17.6</td><td>18.6</td><td>22.9</td><td><strong>23.7</strong></td></tr><tr><td>SciFact</td><td>66.5</td><td>64.3</td><td>67.7</td><td>69.3</td><td>65.7</td><td>70.5</td><td><strong>71.0</strong></td></tr><tr><td>best on (out of 13)</td><td>1</td><td>0</td><td>2</td><td>4</td><td>0</td><td>1</td><td>5</td></tr></tbody></table>

<footnote n="43"><sup>43</sup>Larger checkpoints are, however, usually helpful for zero-shot transfer – not shown here.</footnote>

<footnote n="44"><sup>44</sup>Note that it is actually more accurate to refer to this as middle-training, as we already start from a pre-trained model such as DistilBERT.</footnote>
<table><caption>Table 11. Performance of SPLADE on MS MARCO (dev) and TREC DL 19 for various PLM initializations. <sup>abcde</sup> denote significant improvements over the corresponding rows, for a paired *t*-test with *p*-value=0.01. Note that none of the improvements are significant on TREC DL 19.</caption><thead><tr><th rowspan="2">model</th><th colspan="2">MS MARCO dev</th><th colspan="2">TREC DL 2019</th></tr><tr><th>MRR@10</th><th>R@1000</th><th>nDCG@10</th><th>R@1000</th></tr></thead><tbody><tr><td>SPLADE (a)</td><td>34.5<sup>bcd</sup></td><td>96.5</td><td>67.6</td><td>81.1</td></tr><tr><td>SPLADE (RoBERTa) (b)</td><td>30.9</td><td>96.0</td><td>67.2</td><td>79.4</td></tr><tr><td>SPLADE (ELECTRA) (c)</td><td>33.0<sup>b</sup></td><td>96.7<sup>b</sup></td><td>67.1</td><td>82.4</td></tr><tr><td>SPLADE (BERT-base) (d)</td><td>33.7<sup>b</sup></td><td>96.7<sup>b</sup></td><td>70.0</td><td>81.1</td></tr><tr><td>MLM-FLOPS (e)</td><td>35.7<sup>abcd</sup></td><td>97.2<sup>abcd</sup></td><td>67.9</td><td>80.2</td></tr></tbody></table>

the SPLADE activation function (i.e., $log(1 + ReLU(w_{iv}))$), and are fed to a softmax which defines an MLM loss over sparse vectors ($\mathcal{L}_{MLM-SPLADE}$). The FLOPS regularization is applied to sequence-level representations obtained with max-pooling – similar to SPLADE – to achieve sparsity. The total loss is simply given by:

$$ \mathcal{L} = \mathcal{L}_{\text{MLM}} + \mathcal{L}_{\text{MLM-SPLADE}} + \lambda t_{\text{FLOPS}} \qquad (25) $$

Models are trained for 10 epochs on MS MARCO passages, with $\lambda = 0.001$. Thus, representations are already pre-conditioned to be sparse, which greatly helps SPLADE to converge, leading to significant performance boosts (e.g., $↑+1.2$ MRR@10 – as shown in Table 11). It is also key to obtaining efficient models (Section 8.2).

More generally, the effect of initialization (of the MLM head) on SPLADE remains yet to be explored. Early experiments with random projection heads were overall very unstable, leading to less effective models. We leave for future work the exploration of other PLM, with different pre-training objectives such as Next Token Prediction (e.g., GPT-2 [86]), for SPLADE models$^{45}$.

<figure><img src="image_12.png" /><figcaption>Fig. 9. effectiveness-efficiency trade-off on MS MARCO dev set for different base Pre-trained Language Models.</figcaption></figure>

### 8.1.2 Evaluating Query Latency.
Up to now, we have characterized SPLADE’s efficiency by the sparsity of its learned representations (or its FLOPS). In Information Retrieval, we are usually interested in measuring query latency – the average time needed to serve a query.

<footnote>$^{45}$However, such approaches are usually encoder-decoder or decoder-only models, which require slight adaptations for retrieval models, see for instance Soares et al. [99].</footnote>
**Challenges in Measuring Query Latency for Learned Sparse Models.** The FLOPS regularization introduced in Paria et al. [81] (Section 4.2) was motivated by the implied efficiency of sparse representations coupled with inverted indexes, that usually do not require approximations<sup>46</sup> – contrary to ANN techniques for dense representations. In their setting, retrieval is cast as a sparse matrix multiplication between query and document vectors, that suppose *intra-query* parallelism, i.e., that posting lists can be traversed in parallel in a multi-threaded setting. Thus, the FLOPS measure is directly related to retrieval time. By relying on the same regularization, SPLADE is also optimized for multi-threaded retrieval.

However, the standard retrieval software stacks usually only assume *inter-query* parallelism: as such, a single query is processed by a single CPU core<sup>47</sup>. In this setting, increasing the number of posting lists to traverse at inference time usually leads to huge efficiency drops, which is particularly hurtful for query expansion methods – including SPLADE. Moreover, document expansion can also impact efficiency, as it increases the size of the posting lists. One subtle additional point is the distribution of weights in the posting themselves. Standard query processing algorithms such as MaxScore [105] or WAND [6] assume a certain weight distribution, derived from traditional term-based scoring functions such as BM25 and the statistical properties of natural language. The new sparse approaches based on PLM can therefore learn term distributions for which standard algorithms are not suited. For instance, Mallia et al. [69] show how query latency is significantly higher for DeepImpact, despite the absence of query expansion. Mackenzie et al. [67] show how SPLADE generates “wacky” weights, that reduce the opportunities for skipping and early exiting optimizations that lie at the core of standard *document-at-a-time* techniques. Indeed, traditional models like BM25 assume that frequent terms have low importance scores (IDF effect) – which is not necessarily the case for learned sparse models (as shown in Section 6.3). As a consequence, a relatively new research branch emerged, with the goal of speeding up learned sparse retrieval. Mallia et al. [70] propose to guide the traversal of (learned) posting lists with BM25. Mackenzie et al. [66] recently introduced *postings clipping* – adapting MaxScore and WAND for learned importance schemes<sup>48</sup>.

**Measuring SPLADE Latency: Case Study.** Generally speaking, measuring and comparing the latency of retrieval systems is intricate, as different approaches rely on different assumptions. As such, dense retrieval usually assumes having access to multiple CPUs – or even GPUs – to perform search. In the meantime, sparse retrieval based on inverted indexes is usually based on mono-core implementations. In a first experiment, we propose to measure SPLADE efficiency in two settings to illustrate the issues mentioned above<sup>49</sup>. First, we rely on Anserini [116], an IR toolkit built on the open-source Lucene search library. We follow the quantization approach described in Section 5 to index our collection. We keep the Anserini default setting, which relies on BMW [22] (*document-at-a-time*) query evaluation. To measure latency with Anserini, we place ourselves in a mono-CPU setting<sup>50</sup>. Additionally, we propose to evaluate latency based on our custom inverted index, which relies on a multi-threaded Numba *term-at-a-time* query processing – thus being more aligned with SPLADE training objective. Queries are still processed sequentially, but we benefit from intra-query parallelism – which is not supported by Anserini. We conduct experiments on a CPU-based environment, using a server with 64 2GHz Intel(R) Xeon(R) Gold 6338 CPUs and 256GB of RAM. We use 1 and 12 CPU cores to measure latency with Anserini and our index, respectively. We evaluate the mean query latency on the set of 6980 dev queries.

<footnote n="46"><sup>46</sup>Although many efficient query processing algorithms do rely on approximations.</footnote>

<footnote n="47"><sup>47</sup>The rationale is that production systems should deal with many queries concurrently – the bottleneck is thus not within a single query.</footnote>

<footnote n="48"><sup>48</sup>They, however, do not consider SPLADE as part of the study.</footnote>

<footnote n="49"><sup>49</sup>More thorough latency evaluations are performed in Section 8.2.</footnote>

<footnote n="50"><sup>50</sup>By setting the option -parallelism 1.</footnote>
<table><caption>Table 12. Experimental results on MS MARCO dev set. $l_0(q)$ and $l_0(d)$ denote the average query and document sizes, estimated respectively from the 6980 dev queries and the full collection C after training. We measure latency (in milliseconds) for BM25 (47.85ms in the same evaluation environment), and report numbers for DeepImpact from [67] for comparison – acknowledging the differences between experimental settings.</caption><thead><tr><th>model</th><th>MRR@10</th><th>l<sub>0</sub>(q)</th><th>l<sub>0</sub>(d)</th><th>FLOPS</th><th>Latency (ms)</th><th>Index Size (GB)</th></tr><tr><th></th><th></th><th></th><th></th><th></th><th>Anserini</th><th>Numba</th></tr></thead><tbody><tr><td>BM25</td><td>18.4</td><td>-</td><td>-</td><td>0.13</td><td>47.8</td><td>0.6</td></tr><tr><td>DeepImpact</td><td>32.5</td><td>-</td><td>-</td><td>-</td><td>244.1</td><td>1.4</td></tr><tr><td>(λ<sub>q</sub>, λ<sub>d</sub>)</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>(8e-5, 6e-5)</td><td>34.2</td><td>22</td><td>166</td><td>4.69</td><td>2494.3</td><td>3.4</td></tr><tr><td>(3e-4, 1e-4)</td><td>34.4</td><td>19</td><td>148</td><td>2.61</td><td>1434.2</td><td>3.2</td></tr><tr><td>(8e-4, 6e-4)</td><td>34.5</td><td>16</td><td>90</td><td>1.23</td><td>828.37</td><td>2.3</td></tr><tr><td>(3e-3, 1e-3)</td><td>33.9</td><td>15</td><td>74</td><td>0.74</td><td>486.53</td><td>2.1</td></tr><tr><td>(8e-3, 6e-3)</td><td>32.7</td><td>14</td><td>44</td><td>0.31</td><td>286.96</td><td>1.6</td></tr><tr><td>(3e-2, 1e-2)</td><td>32.6</td><td>11</td><td>33</td><td>0.15</td><td>119.20</td><td>1.4</td></tr><tr><td>(8e-2, 6e-2)</td><td>29.6</td><td>9</td><td>18</td><td>0.05</td><td>69.34</td><td>1.1</td></tr></tbody></table>

We report in Table 12 effectiveness, as well as several efficiency measures, for a set of SPLADE models trained with varying $\lambda$ – corresponding to the ones in Figure 3. First, we provide Anserini index sizes which highlight the effect of regularization: the higher the $\lambda$, the smaller the memory requirements for the index$^{51}$. Overall, we see that latency decreases with sparsity (and the FLOPS measure). More specifically, latency measurements on a mono-CPU environment (with Anserini) are in line with the ones reported in Mackenzie et al. [67]. They are especially large for less regularized SPLADE models (e.g., 24954ms for a FLOPS value of 4.7). However, for a sufficiently large $\lambda$, we are able to reach efficiency levels similar to BM25 by sacrificing effectiveness – e.g., 69.3ms for a model with almost 30 MRR@10. When switching to a multi-threaded setting based on Numba parallelism, query latency largely decreases – reaching efficiency levels on par with BM25 (Anserini) for the most effective models (44.1ms for 34.5 MRR@10). This highlights the efficiency issues of SPLADE in standard retrieval stacks – which is bound by the size of query vectors.

## 8.2 Efficient SPLADE models

As already discussed in Section 8.1.2, efficiency is a critical aspect of IR systems. Reducing measures such as query latency can greatly improve the “usability” of models for production scenarios. Minimizing $l_{FLOPS}$ (or $l_1$) only optimizes for a certain aspect of efficiency, that is not entirely aligned with what makes a model truly efficient. In particular, i) standard IR software stacks do not usually perform intra-query parallelism, and thus require small queries to process a low number of posting lists, ii) query inference time, i.e., the time needed to compute query representations – based on Pre-trained Language Models or not – should be low. We propose to deal with both aspects in the following.

We propose four adaptations to the SPLADE model, namely i) the separation of query and document encoders, ii) the change of query regularization to $l_1$, iii) the FLOPS-regularized PLM initialization (Section 8.1.1), iv) the use of a smaller PLM on the query side, which are detailed in the following.

### 8.2.1 Searching for Appropriate Hyperparameters.
First, we investigate how a careful tuning of hyperparameters can lead to more efficient SPLADE models. Notably, we replace the MarginMSE distillation loss with the traditional

<footnote>$^{51}$Note that a standard flat index obtained with FAISS [43] for a dense bi-encoder weighs around 22GB for the whole MS MARCO collection.</footnote>
KL-Divergence [38, 48] – improving the overall training stability<fnref n="52" />. We also carefully search for a set of $\lambda = (\lambda_q, \lambda_d)$ in order to have more control over the final query and document sizes. We choose to keep a set of three configurations: Small (S), Medium (M), and Large (L). (S) is the “sparsest” configuration (i.e., higher $\lambda$), (M) has the same level of final query sparsity as (S), but larger document sizes, while (L) has larger queries than (S) and the same document sparsity as (M). The second improvement comes from the data – we directly consider the effectiveness-oriented EnsembleDistil setting from Section 7 in order to focus solely on efficiency considerations. We refer to these overall improvements as $I) + II)$.

### 8.2.2 Separating Document and Query Encoders.
We first notice that it is hard to target smaller queries – even for very large differences between $\lambda_d$ and $\lambda_q$ – due to the shared nature of SPLADE encoder. Therefore, we propose having two distinct encoders for queries and documents, thus relieving the model to find an optimal trade-off for documents and queries within a single model. We refer to this setting as III).

### 8.2.3 $t_1$ Query Regularization.
We simply propose to regularize query representations with $t_1$ instead of $t_{FLOPS}$. Indeed, while $t_{FLOPS}$ leads to more distributed representations (Figure 4), $t_1$, however, has a stronger “sparsity” effect overall. Thus, we propose to optimize SPLADE with:

$$ \mathcal{L} = \mathcal{L}_{\text{rank}} + \lambda_q t_1^q + \lambda_d t_{\text{FLOPS}}^d \quad (26) $$

We refer to this setting as IV).

### 8.2.4 Middle-Training with MLM-FLOPS.
As observed in Section 8.1.1, the impact of initialization is critical for SPLADE convergence. In particular, the MLM-FLOPS middle-training already pre-conditions SPLADE representations to be highly sparse, leading to more efficient models overall. We refer to this setting as V).

### 8.2.5 Smaller PLM-based Query Encoder.
While previous extensions allow for reducing query evaluation time by lowering the number of posting lists to traverse, they are still limited by the inference time of the PLM needed to get the query representation. We first rely on a more efficient PLM for the query encoder – namely BERT-tiny [5]<fnref n="53" />. We additionally consider a document-only version of SPLADE – referred to as SPLADE-doc [25]. In this case, there is neither query term weighting nor expansion, and the ranking score is simply given by:

$$ s(q, d) = \sum_{v \in q} w_v^{(d)} \quad (27) $$

Here, we can see the query as a sparse binary BoW vector, as shown in Figure 10. Such a formulation is akin to traditional term-based approaches in IR, and offers an interesting efficiency boost. Because the ranking score solely depends on the document term weights, everything can be pre-computed offline, reducing inference cost. It also enables GPU-free inference. Thus, the model becomes more comparable with approaches like DeepImpact [69] or TILDEv2 [123]. However, document expansion is part of the training loop – contrary to the aforementioned approaches, which rely on fixed expansion techniques (doc2query-T5 and TILDE, respectively).

<footnote n="52">⁵²Note that this contrasts with the losses used in Section 7 which usually performed equally or better in terms of effectiveness.</footnote>

<footnote n="53">⁵³Another possibility would be to perform quantization and/or compression of the PLM – we leave exploring this to future work.</footnote>
<figure><img src="image_13.png" /><figcaption>Fig. 10. SPLADE-doc model. Here, we can see queries as binary BoW vectors in the WordPiece vocabulary.</figcaption></figure>

## 8.3 Experiments

8.3.1 *Experimental Setting.* Overall, we follow the training and evaluation workflow described in Section 5. Especially, up to some slight variations$^{54}$, we use the same hyperparameters, as well as the validation strategy, but modify the components introduced in Section 8.2. We initialize models with DistILBERT or BERT-tiny. For SPLADE-doc, early experiments indicated that fewer training steps are actually needed for the models to converge – training for longer usually degrades the performance. We hypothesize that joint query and document expansion needs careful calibration, while removing query expansion simplifies the learning problem, by inducing an explicit lexical prior. We thus train such models for 50k iterations. Middle-training is performed using default MLM parameters, with an added FLOPS regularization of $\lambda = 0.001$, for 10 epochs on MS MARCO passages. For settings *I*), *II*), *III*), models use the same hyperparameters – $\lambda = (0.1, 5e-3)$, $\lambda = (0.1, 5e-4)$, $\lambda = (0.01, 5e-4)$ for respectively (S), (M) and (L). For settings *IV*), *V*) and *VI*), we use $\lambda = (5e-3, 5e-3)$, $\lambda = (5e-4, 5e-4)$ and $\lambda = (5e-4, 5e-4)$ for respectively (S), (M) and (L).

In order to compute efficiency measures, all experiments are performed on the same machine, with an Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz CPU and sufficient RAM memory to preload indexes, models, and queries before starting the experiment. For retrieval, we test on both Anserini [116] and PISA [71]. We measure encoding inference time for PyTorch using the benchmarking tool from HuggingFace Transformers [111]. Latency is computed as a simple addition of query encoding and retrieval time. The DistILBERT query encoder has an average latency of 45.3ms, while BERT-tiny has an average latency of 0.7ms.

8.3.2 *Results.* We represent the effectiveness-efficiency trade-off in Figure 11 and Figure 12 for, respectively, PISA and Anserini. Note that latency from models in Section 7 – usually having much higher latencies (as observed in Table 12) – are not reported here.

* First, we note that carefully setting regularization hyperparameters allows us to decrease the PISA latency of a “standard” SPLADE by almost 10 times while keeping similar – or even improved – performance on MS MARCO. Results are overall similar for Anserini. This is expected, as such models are not explicitly optimized to reduce query latency. We also see that all the considered models lead to close to state-of-the-art results in first-stage retrieval – with at most a 10% reduction in MRR@10 performance.

* All the proposed improvements yield additive gains: we observe that improvements *III*) to *V*) successfully improve latency while keeping similar levels of effectiveness on MS MARCO. Nevertheless, each setting seems to bring diminishing gains and is indeed heavily impacted by the inference latency of DistILBERT. For instance, when considering PISA (Figure 11), the sparsest *V*) (left-side) has a gain of 1.2ms compared to the sparsest *IV*), which represents an approximate 20% reduction of retrieval time, but an overall reduction of just 2% (i.e., PyTorch + PISA inference).

<footnote n="54">⁵⁴For instance, we notice that training models for longer, i.e., up to 250k steps, was generally helpful to improve the effectiveness-efficiency trade-off.</footnote>
<figure><img src="image_14.png" /><figcaption>Fig. 11. Latency comparison (PISA) between all proposed improvements. † indicates that we remove stopwords from queries.</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>Fig. 12. Latency comparison (Anserini) between all proposed improvements. † indicates that we remove stopwords from queries.</figcaption></figure>

* To reduce DistilBERT inference latency, using Bert-tiny [VI] is needed to further speed up query encoding. In such a setting, we note a trade-off: at the cost of a slight effectiveness loss (around 1.0 MRR@10 on MS MARCO), we are able to greatly reduce the latency of the sparsest considered SPLADE models (around 10-fold for PISA and 2-fold for Anserini). For the query encoder choice, BERT-tiny has a slight advantage over SPLADE-doc in terms of effectiveness – showing the importance of the encoder, even if small.

We further compare the proposed approaches with respect to other sparse retrieval methods. To this end, we compare a subset of our systems *V*) and *VI*) to *i*) BM25, *ii*) doc2query-T5, *iii*) DeepImpact, *iv*) UniCOIL (TILDE)<fnref n="55" />. Results are shown in Figure 13 for PISA – similar observations can be made for Anserini (not shown here). We can see that, compared to all the PLM-based techniques, we are able to train models that are **both more efficient and effective on MS MARCO in-domain**<fnref n="56" />. When compared to BM25, we achieve similar efficiency, with a 2× gain on effectiveness.

<footnote n="55">⁵⁵Similar to SPLADE, UniCOIL also performs query inference – we, therefore, add DistilBERT inference latency to the retrieval time.</footnote>

<footnote n="56">⁵⁶Note that some of the improvements proposed here could be applied to these systems – we leave such study for future works.</footnote>
<figure><img src="image_16.png" /><figcaption>Fig. 13. Latency comparison between the proposed adaptations and sparse methods. †: queries without stop words.</figcaption></figure>

**Zero-Shot Evaluation** we study the effect of our efficiency improvements on zero-shot evaluation in Table 13. Unfortunately, the gains in efficiency and in-domain effectiveness lead to a drop in performance when evaluating models in a zero-shot setting. While our efficient approaches still have decent effectiveness compared to BM25 – which is not the case for most dense models [101] – they still lose a lot when compared to effective models introduced in Section 7. By investigating performance on a per-dataset basis (not shown here), we notice sharp losses on some datasets, especially on the Quora dataset – where nDCG@10 falls from around 0.84 to 0.46 for e.g., *BT*-SPLADE-S. It is however expected, as this dataset relies on queries (questions) for both documents *and* queries, and is thus more impacted by aggressive query pruning. Similarly to what was observed for effective models (in Table 9), it is possible to mitigate this overall loss of effectiveness by fusing results with the ones from BM25 (BEIR\* column). For the more efficient models, combining our method with BM25 allows us to outperform doc2query-T5 on BEIR, with similar levels of latency (11ms).

<table><caption>Table 13. Results on in- and out-of-domain (MS MARCO and BEIR respectively). BEIR\* is the combination (sum-fusion) of BM25 and the corresponding row. *BT* refers to BERT-tiny query encoder. † indicates that we remove stopwords from queries.</caption><thead><tr><th>Model</th><th>Latency</th><th>MS MARCO dev MRR@10</th><th>TREC DL 19 nDCG@10</th><th>BEIR nDCG@10</th><th>BEIR*</th></tr></thead><tbody><tr><td colspan="6"><strong>► Baselines</strong></td></tr><tr><td>BM25<sup>†</sup></td><td><strong>4</strong></td><td>19.7</td><td>50.6</td><td>43.0</td><td>-</td></tr><tr><td>doc2query-T5</td><td>11</td><td>27.6</td><td>64.2</td><td>44.1</td><td>-</td></tr><tr><td colspan="6"><strong>► Our methods</strong></td></tr><tr><td>VI) BT-SPLADE-S</td><td><u>7</u></td><td>35.8</td><td>67.2</td><td>39.2</td><td>45.9</td></tr><tr><td>VI) BT-SPLADE-M</td><td>13</td><td><u>37.6</u></td><td>69.4</td><td>42.1</td><td>47.1</td></tr><tr><td>VI) BT-SPLADE-L</td><td>32</td><td><strong>38.0</strong></td><td><u>70.3</u></td><td><u>44.5</u></td><td><u>48.0</u></td></tr></tbody></table>

### 8.3.3 Comparison Against Dense Models.
We now compare a state-of-the-art (mono-representation) dense model – CoCondenser – with our efficient extensions. In order to study different settings with different trade-offs, we consider: i) measuring latency under mono-CPU using one query at a time, ii) measuring QPS (Queries-Per-Second) under multi-threaded CPU using batched queries iii) measuring QPS under multi-threaded CPU disregarding inference time
with *batched queries*. For CoCondenser, we rely on FAISS [43] to perform indexing and retrieval. We either perform a brute-force search, or an ANN search using HNSW [68] and IVF [98] without any quantization<fnref n="57" />. We acknowledge the difficulties in properly comparing latency measurements for such systems, and while the three settings are not completely fair to fully characterize both approaches<fnref n="58" />, they still represent some of the different scenarios that one would consider for deploying such systems. Note that CoCondenser already starts with a “disadvantage”, as it relies on a larger encoder (i.e., BERT, which has around two times the computational cost of DistilBERT)<fnref n="59" />. In order to consider a “DistilBERT” version of CoCondenser, we assume a *perfect* distillation of the BERT model into DistilBERT, as well as a scenario where inference is disregarded. Results are shown in Figures 14, 15, and 16.

<figure><img src="image_17.png" /><figcaption>Fig. 14. Latency comparison between the proposed adaptations and a dense method with ANN. x-axis is in $log_2$ scale. CC refers to CoCondenser, and DB-CC represents an ideal distillation of CoCondenser from BERT to DistilBERT. Better results → up-left.</figcaption></figure>

<figure><img src="image_18.png" /><figcaption>Fig. 15. QPS comparison between the proposed adaptations and a dense method with ANN, considering inference time. x-axis is in $log_4$ scale. DB-CC is omitted as the difference with CC is too small. Better results → up-right.</figcaption></figure>

In mono-CPU latency (Figure 14), the brute force approach lags behind all the others – more than 200ms – and is therefore not shown. We notice that HNSW allows dense models to be very close to efficient SPLADE models. However,

<footnote n="57">⁵⁷ Note that we also tested with Product Quantization, which led to worse effectiveness-efficiency trade-offs.</footnote>

<footnote n="58">⁵⁸ For instance, they fail to account for index size, maximum RAM, as well as approximate methods for sparse retrieval [67].</footnote>

<footnote n="59">⁵⁹ As far as we are aware, there are no state-of-the-art dense implementations relying on BERT-tiny.</footnote>
<figure><img src="image_19.png" /><figcaption>Fig. 16. QPS comparison between the proposed adaptations and a dense method with ANN, disregarding inference time. x-axis in log<sub>4</sub> scale. Better results → up-right.</figcaption></figure>

HNSW comes with a large increase in index size – in our case, an extra 256 bytes per passage, or around 2GB for the entire MS MARCO<sup>60</sup>. If we consider methods that do not increase index size by much – such as IVF – the difference is still quite large<sup>61</sup>. With multiple CPU threads, we still observe a slight advantage for models presented in this Section – even when compared to a BERT-based CoCondenser (Fig 15) or by disregarding inference time (Fig 16). These results further show the interest in studying sparse representations, which can achieve equal or better effectiveness-efficiency trade-offs with smaller indexes – and this under various evaluation settings.

9 CONCLUSION

In this work, we have shown how SPLADE can be made both *effective* and *efficient*. By unifying expansion and term weighting in a simple framework, SPLADE can control the effectiveness-efficiency trade-off in an end-to-end fashion, making it able to fully take advantage of improved training techniques such as distillation or hard negative mining. Thus, SPLADE++ is the first sparse retriever able to truly compete with dense bi-encoders on MS MARCO – while reaching state-of-the-art results in zero-shot evaluation settings – highlighting its *robustness* when facing distribution shifts. In addition, efficient extensions of SPLADE can be made as fast as traditional keyword-based approaches.

All the advantages of SPLADE have made it an appealing candidate in retrieval scenarios that go beyond ad-hoc IR – including, for instance, Conversational Search [51], Cross-Lingual IR [75] or even Cross-Modal retrieval [9, 62].

REFERENCES

[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer Normalization. https://doi.org/10.48550/ARXIV.1607.06450

[2] Yang Bai, Xiaoguang Li, Gang Wang, Chaoliang Zhang, Lifeng Shang, Jun Xu, Zhaowei Wang, Fangshan Wang, and Qun Liu. 2020. SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval. https://doi.org/10.48550/ARXIV.2010.00768

[3] Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268 (2016).

[4] Elias Bassani. 2022. ranx: A Blazing-Fast Python Library for Ranking Evaluation and Comparison. In *Advances in Information Retrieval - 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10-14, 2022, Proceedings, Part II (Lecture Notes in Computer Science, Vol. 13186)*, Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Krisztian Balog, Kjetil Nørvåg, and Vinay Setty (Eds.). Springer, 259–264. https://doi.org/10.1007/978-3-030-99739-7_30

<footnote n="60"><sup>60</sup>Note that the entire PISA indexes for SPLADE weigh around 1GB, 2GB, and 4GB for, respectively (S), (M) and (L).</footnote>

<footnote n="61"><sup>61</sup>Note that unquantized dense indexes weigh 25GB and 12GB for respectively `float32` and `float16`.</footnote>
[5] Prajjwal Bhargava, Aleksandr Drozd, and Anna Rogers. 2021. Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics. arXiv:2110.01518 [cs.CL]

[6] Andrei Z. Broder, David Carmel, Michael Herscovici, Aya Soffer, and Jason Zien. 2003. Efficient Query Evaluation Using a Two-Level Retrieval Process. In *Proceedings of the Twelfth International Conference on Information and Knowledge Management* (New Orleans, LA, USA) (CIKM '03). Association for Computing Machinery, New York, NY, USA, 426–434. https://doi.org/10.1145/956863.956944

[7] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. 1993. Signature Verification Using a "Siamese" Time Delay Neural Network. In *Proceedings of the 6th International Conference on Neural Information Processing Systems* (Denver, Colorado) (NIPS '93). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 737–744.

[8] Sebastian Bruch, Siyu Gai, and Amir Ingber. 2022. An Analysis of Fusion Functions for Hybrid Retrieval. https://doi.org/10.48550/ARXIV.2210.11934

[9] Chen Chen, Bowen Zhang, Liangliang Cao, Jiguang Shen, Tom Gunter, Albin Madappally Jose, Alexander Toshev, Jonathon Shlens, Ruoming Pang, and Yinfei Yang. 2023. STAIR: Learning Sparse Text and Image Representation in Grounded Tokens. arXiv:2301.13081 [cs.CV]

[10] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2022. A Simple Framework for Contrastive Learning of Visual Representations. In *Proceedings of the 37th International Conference on Machine Learning (ICML'20)*. JMLR.org, Article 149, 11 pages.

[11] Tao Chen, Mingyang Zhang, Jing Lu, Michael Bendersky, and Marc Najork. 2022. Out-of-Domain Semantics to the Rescue! Zero-Shot Hybrid Retrieval Models. In *Advances in Information Retrieval - 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10-14, 2022, Proceedings, Part I (Lecture Notes in Computer Science, Vol. 13185)*, Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Krisztian Balog, Kjetil Norvåg, and Vinay Setty (Eds.). Springer, 95–110. https://doi.org/10.1007/978-3-030-99736-6_7

[12] Eunseong Choi, Sunkyung Lee, Minijn Choi, Hyeseon Ko, Young-In Song, and Jongwuk Lee. 2022. SpaDE: Improving Sparse Representations Using a Dual Document Encoder for First-Stage Retrieval. In *Proceedings of the 31st ACM International Conference on Information & Knowledge Management* (Atlanta, GA, USA) (CIKM '22). Association for Computing Machinery, New York, NY, USA, 272–282. https://doi.org/10.1145/3511808.3557456

[13] Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. In *International Conference on Learning Representations*. https://openreview.net/forum?id=r1xMH1BtvB

[14] Vincent Claveau. 2021. Neural Text Generation for Query Expansion in Information Retrieval. In *IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology* (Melbourne, VIC, Australia) (WI-IAT '21). Association for Computing Machinery, New York, NY, USA, 202–209. https://doi.org/10.1145/3486622.3493957

[15] Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2020. Overview of the TREC 2019 deep learning track. CoRR abs/2003.07820 (2020). arXiv:2003.07820 https://arxiv.org/abs/2003.07820

[16] Zhuyun Dai and Jamie Callan. 2019. Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval. https://doi.org/10.48550/ARXIV.1910.10687

[17] Zhuyun Dai and Jamie Callan. 2019. Deeper Text Understanding for IR with Contextual Neural Language Modeling. In *Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019*, Benjamin Piwowarski, Max Chevalier, Éric Gaussier, Yoelle Maarek, Jian-Yun Nie, and Falk Scholer (Eds.). ACM, 985–988. https://doi.org/10.1145/3331184.3331303

[18] Zhuyun Dai and Jamie Callan. 2020. Context-Aware Document Term Weighting for Ad-Hoc Search. In *Proceedings of The Web Conference 2020* (Taipei, Taiwan) (WWW '20). Association for Computing Machinery, New York, NY, USA, 1897–1907. https://doi.org/10.1145/3366423.3380258

[19] Zhuyun Dai and Jamie Callan. 2020. Context-Aware Term Weighting For First Stage Passage Retrieval. In *Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval* (Virtual Event, China) (SIGIR '20). Association for Computing Machinery, New York, NY, USA, 1533–1536. https://doi.org/10.1145/3397271.3401204

[20] Zhuyun Dai, Vincent Y. Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming-Wei Chang. 2022. Promptagator: Few-shot Dense Retrieval From 8 Examples. https://doi.org/10.48550/ARXIV.2209.11755

[21] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*. Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186. https://doi.org/10.18653/v1/N19-1423

[22] Shuai Ding and Torsten Suel. 2011. Faster Top-k Document Retrieval Using Block-Max Indexes. In *Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval* (Beijing, China) (SIGIR '11). Association for Computing Machinery, New York, NY, USA, 993–1002. https://doi.org/10.1145/2009916.2010048

[23] Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical Neural Story Generation. In *Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*. Association for Computational Linguistics, Melbourne, Australia, 889–898. https://doi.org/10.18653/v1/P18-1082

[24] Hui Fang, Tao Tao, and ChengXiang Zhai. 2004. A Formal Study of Information Retrieval Heuristics. In *Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval* (Sheffield, United Kingdom) (SIGIR '04). Association for Computing Machinery, New York, NY, USA, 49–56. https://doi.org/10.1145/1008992.1009004

[25] Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2021. "SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval". https://doi.org/10.48550/ARXIV.2109.10086
[26] Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking. In *Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval* (Virtual Event, Canada) (SIGIR '21). Association for Computing Machinery, New York, NY, USA, 2288–2292. https://doi.org/10.1145/3404835.3463098

[27] Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2022. Match Your Words! A Study of Lexical Matching in Neural Information Retrieval. In *Advances in Information Retrieval: 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10–14, 2022, Proceedings, Part II* (Stavanger, Norway). Springer-Verlag, Berlin, Heidelberg, 120–127. https://doi.org/10.1007/978-3-030-99739-7_14

[28] G. W. Furnas, T. K. Landauer, L. M. Gomez, and S. T. Dumais. 1987. The Vocabulary Problem in Human-System Communication. *Commun. ACM* 30, 11 (nov 1987), 964–971. https://doi.org/10.1145/32206.32212

[29] Luyu Gao and Jamie Callan. 2022. Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval. In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*. Association for Computational Linguistics, Dublin, Ireland, 2843–2853. https://doi.org/10.18653/v1/2022.acl-long.203

[30] Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021. COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List. In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*. Association for Computational Linguistics, Online, 3030–3042. https://doi.org/10.18653/v1/2021.naacl-main.241

[31] Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021. Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline. In *Advances in Information Retrieval: 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 – April 1, 2021, Proceedings, Part II*. Springer-Verlag, Berlin, Heidelberg, 280–286. https://doi.org/10.1007/978-3-030-72240-1_26

[32] Chuan Guo, Ali Mousavi, Xiang Wu, Daniel Holtmann-Rice, Satyen Kale, Sashank Reddi, and Sanjiv Kumar. 2019. *Breaking the Glass Ceiling for Embedding-Based Classifiers for Large Output Spaces*. Curran Associates Inc., Red Hook, NY, USA.

[33] Jiafeng Guo, Yixing Fan, Qingyao Ai, and W. Bruce Croft. 2016. A Deep Relevance Matching Model for Ad-Hoc Retrieval. In *Proceedings of the 25th ACM International on Conference on Information and Knowledge Management* (Indianapolis, Indiana, USA) (CIKM '16). Association for Computing Machinery, New York, NY, USA, 55–64. https://doi.org/10.1145/2983323.2983769

[34] Jiafeng Guo, Yixing Fan, Liang Pang, Liu Yang, Qingyao Ai, Hamed Zamani, Chen Wu, W. Bruce Croft, and Xueqi Cheng. 2020. A Deep Look into neural ranking models for information retrieval. *Inf. Process. Manag.* 57, 6 (2020), 102067. https://doi.org/10.1016/j.ipm.2019.102067

[35] M. Gutmann and A. Hyvärinen. 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In *Proc. Int. Conf. on Artificial Intelligence and Statistics (AISTATS)* (JMLR W&CP, Vol. 9), Y.W. Teh and M. Titterington (Eds.). 297–304.

[36] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2019. Momentum Contrast for Unsupervised Visual Representation Learning. https://doi.org/10.48550/ARXIV.1911.05722

[37] Dan Hendrycks and Kevin Gimpel. 2016. Gaussian Error Linear Units (GELUs). https://doi.org/10.48550/ARXIV.1606.08415

[38] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the Knowledge in a Neural Network. https://doi.org/10.48550/ARXIV.1503.02531

[39] Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021. Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling. In *Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval* (Virtual Event, Canada) (SIGIR '21). Association for Computing Machinery, New York, NY, USA, 113–122. https://doi.org/10.1145/3404835.3462891

[40] Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2020. Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation. https://doi.org/10.48550/ARXIV.2010.02666

[41] Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised Dense Information Retrieval with Contrastive Learning. *Transactions on Machine Learning Research* (2022). https://openreview.net/forum?id=jKN1pXi7b0

[42] Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola de Cao, Sebastian Riedel, and Edouard Grave. 2021. A Memory Efficient Baseline for Open Domain Question Answering. (Dec. 2021). https://hal.archives-ouvertes.fr/hal-03463488 working paper or preprint.

[43] Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. *IEEE Transactions on Big Data* 7, 3 (2019), 535–547.

[44] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*. Association for Computational Linguistics, Online, 6769–6781. https://doi.org/10.18653/v1/2020.emnlp-main.550

[45] Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. In *Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval* (Virtual Event, China) (SIGIR '20). Association for Computing Machinery, New York, NY, USA, 39–48. https://doi.org/10.1145/3397271.3401075

[46] Gyuwan Kim, Jinhyuk Lee, Barlas Oguz, Wenhan Xiong, Yizhe Zhang, Yashar Mehdad, and William Yang Wang. 2022. Bridging the Training-Inference Gap for Dense Phrase Retrieval. In *ICML 2022 Workshop on Knowledge Retrieval and Language Models*. https://openreview.net/forum?id=vOrgYjli9R

[47] Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In *3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings*, Yoshua Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1412.6980

[48] Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. *The annals of mathematical statistics* 22, 1 (1951), 79–86.

[49] Siu Kwan Lam, Antoine Pitrou, and Stanley Seibert. 2015. Numba: A llvm-based python jit compiler. In *Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC*. 1–6.

[50] Victor Lavrenko and W. Bruce Croft. 2001. Relevance Based Language Models. In *Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval* (New Orleans, Louisiana, USA) (SIGIR '01). Association for Computing Machinery, New York, NY, USA, 120–127. https://doi.org/10.1145/383952.383972
[51] Nam Le Hai, Thomas Gerald, Thibault Formal, Jian-Yun Nie, Benjamin Piwowarski, and Soulier Laure. 2023. CoSPLADE: Contextualizing SPLADE for Conversational IR. In *Advances in Information Retrieval: 45th European Conference on IR Research, ECIR 2023, Dublin, Ireland, April 2-6, 2023, to appear* (Dublin, Ireland).

[52] Hang Li, Shuai Wang, Shengyao Zhuang, Ahmed Mourad, Xueguang Ma, Jimmy Lin, and Guido Zuccon. 2022. To Interpolate or Not to Interpolate: PRF, Dense and Sparse Retrievers. In *Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval* (Madrid, Spain) (*SIGIR '22*). Association for Computing Machinery, New York, NY, USA, 2495–2500. https://doi.org/10.1145/3477495.3531884

[53] Jimmy Lin. 2019. The Neural Hype and Comparisons Against Weak Baselines. *SIGIR Forum* 52, 2 (jan 2019), 40–51. https://doi.org/10.1145/3308774.3308781

[54] Jimmy Lin. 2022. A Proposed Conceptual Framework for a Representational Approach to Information Retrieval. *SIGIR Forum* 55, 2, Article 4 (mar 2022), 29 pages. https://doi.org/10.1145/3527546.3527552

[55] Jimmy Lin and Xueguang Ma. 2021. A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniques. https://doi.org/10.48550/ARXIV.2106.14807

[56] Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. 2020. Distilling Dense Representations for Ranking using Tightly-Coupled Teachers. https://doi.org/10.48550/ARXIV.2010.11386

[57] Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. 2021. In-Batch Negatives for Knowledge Distillation with Tightly-Coupled Teachers for Dense Retrieval. In *Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021)*. Association for Computational Linguistics, Online, 163–173. https://doi.org/10.18653/v1/2021.repl4nlp-1.17

[58] Erik Lindgren, Sashank J. Reddi, Ruiqi Guo, and Sanjiv Kumar. [n. d.]. Efficient Training of Retrieval Models using Negative Cache. https://openreview.net/forum?id=824xC-SgWgU&referrer=%5BReviewer%20Console%5D(%2Fgroup%3Fid%3DNeurIPS.cc%2F2021%2FConference%2FReviewers%23assigned-papers)

[59] Tie-Yan Liu. 2011. *Learning to Rank for Information Retrieval*. Springer. I–XVII, 1–285 pages.

[60] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. https://doi.org/10.48550/ARXIV.1907.11692

[61] Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021. Sparse, Dense, and Attentional Representations for Text Retrieval. *Transactions of the Association for Computational Linguistics* 9 (2021), 329–345. https://doi.org/10.1162/tacl_a_00369

[62] Ziyang Luo, Pu Zhao, Can Xu, Xiubo Geng, Tao Shen, Chongyang Tao, Jing Ma, Qingwen lin, and Daxin Jiang. 2023. LexLIP: Lexicon-Bottlenecked Language-Image Pre-Training for Large-Scale Image-Text Retrieval. arXiv:2302.02908 [cs.CV]

[63] Xueguang Ma, Kai Sun, Ronak Pradeep, and Jimmy Lin. 2021. A Replication Study of Dense Passage Retriever. CoRR abs/2104.05740 (2021). arXiv:2104.05740 https://arxiv.org/abs/2104.05740

[64] Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, Nazli Goharian, and Ophir Frieder. 2020. Expansion via Prediction of Importance with Contextualization. In *Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval* (Virtual Event, China) (*SIGIR '20*). Association for Computing Machinery, New York, NY, USA, 1573–1576. https://doi.org/10.1145/3397271.3401262

[65] Sean MacAvaney, Andrew Yates, Arman Cohan, and Nazli Goharian. 2019. CEDR: Contextualized Embeddings for Document Ranking. In *Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval*. 1101–1104. https://doi.org/10.1145/3331184.3331317

[66] Joel Mackenzie, Antonio Mallia, Alistair Moffat, and Matthias Petri. 2022. Accelerating learned sparse indexes via term impact decomposition. In *EMNLP 2022*. https://www.amazon.science/publications/accelerating-learned-sparse-indexes-via-term-impact-decomposition

[67] Joel Mackenzie, Andrew Trotman, and Jimmy Lin. 2021. Wacky Weights in Learned Sparse Representations and the Revenge of Score-at-a-Time Query Evaluation. https://doi.org/10.48550/ARXIV.2110.11540

[68] Yu A. Malkov and D. A. Yashunin. 2020. Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. *IEEE Trans. Pattern Anal. Mach. Intell.* 42, 4 (apr 2020), 824–836. https://doi.org/10.1109/TPAMI.2018.2889473

[69] Antonio Mallia, Omar Khattab, Torsten Suel, and Nicola Tonellotto. 2021. Learning Passage Impacts for Inverted Indexes. In *Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval* (Virtual Event, Canada) (*SIGIR '21*). Association for Computing Machinery, New York, NY, USA, 1723–1727. https://doi.org/10.1145/3404835.3463030

[70] Antonio Mallia, Joel Mackenzie, Torsten Suel, and Nicola Tonellotto. 2022. Faster Learned Sparse Retrieval with Guided Traversal. In *Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval* (Madrid, Spain) (*SIGIR '22*). Association for Computing Machinery, New York, NY, USA, 1901–1905. https://doi.org/10.1145/3477495.3531774

[71] Antonio Mallia, Michal Siedlaczek, Joel Mackenzie, and Torsten Suel. 2019. PISA: Performant Indexes and Search for Academia. In *Proceedings of the Open-Source IR Replicability Challenge co-located with 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, OSIRRC@SIGIR 2019, Paris, France, July 25, 2019*, 50–56. http://ceur-ws.org/Vol-2409/docker08.pdf

[72] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. Mixed Precision Training. In *International Conference on Learning Representations*. https://openreview.net/forum?id=r1gs9JgRZ

[73] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed Representations of Words and Phrases and Their Compositionality. In *Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2* (Lake Tahoe, Nevada)
(NIPS'13). Curran Associates Inc., Red Hook, NY, USA, 3111-3119.

[74] Iurii Mokrii, Leonid Boytsov, and Pavel Braslavski. 2021. A Systematic Evaluation of Transfer Learning and Pseudo-labeling with BERT-based Ranking Models. In *Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval*. ACM. https://doi.org/10.1145/3404835.3463093

[75] Suraj Nair, Eugene Yang, Dawn Lawrie, James Mayfield, and Douglas Oard. 2022. Learning a Sparse Representation Model for Neural CLIR. In *Proceedings of the 2022 Design of Experimental Search and Information Retrieval Systems* (San Jose).

[76] Shahrzad Naseri, Jeffrey Dalton, Andrew Yates, and James Allan. 2021. CEQE: Contextualized Embeddings for Query Expansion. In *Advances in Information Retrieval*, Djoerd Hiemstra, Marie-Francine Moens, Josiane Mothe, Raffaele Perego, Martin Potthast, and Fabrizio Sebastiani (Eds.). Springer International Publishing, Cham, 467-482.

[77] Thong Nguyen, Sean MacAvaney, and Andrew Yates. 2023. A Unified Framework for Learned Sparse Retrieval. In *Advances in Information Retrieval*, Jaap Kamps, Lorraine Goeuriot, Fabio Crestani, Maria Maistro, Hideo Joho, Brian Davis, Cathal Gurrin, Udo Kruschwitz, and Annalina Caputo (Eds.). Springer Nature Switzerland, Cham, 101-116.

[78] Rodrigo Nogueira and Jimmy Lin. 2019. From doc2query to docTTTTTquery.

[79] Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019. Document Expansion by Query Prediction. https://doi.org/10.48550/ARXIV. 1904.08375

[80] Rodrigo Frassetto Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT. CoRR abs/1901.04085 (2019). arXiv:1901.04085 http://arxiv.org/abs/1901.04085

[81] Biswajit Paria, Chih-Kuan Yeh, Ian E.H. Yen, Ning Xu, Pradeep Ravikumar, and Barnabás Póczos. 2020. Minimizing FLOPs to Learn Efficient Sparse Representations. In *International Conference on Learning Representations*. https://openreview.net/forum?id=SygpC6Ntvr

[82] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In *Advances in Neural Information Processing Systems 32*. Curran Associates, Inc., 8024-8035. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf

[83] Ofir Press and Lior Wolf. 2017. Using the Output Embedding to Improve Language Models. In *Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers*. Association for Computational Linguistics, Valencia, Spain, 157-163. https://aclanthology.org/E17-2025

[84] Yujie Qian, Jinhyuk Lee, Sai Meher Karthik Duddu, Zhuyun Dai, Siddhartha Brahma, Iftekhar Naim, Tao Lei, and Vincent Y. Zhao. 2022. Multi-Vector Retrieval as Sparse Alignment. https://doi.org/10.48550/ARXIV.2211.01267

[85] Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering. In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*. Association for Computational Linguistics, Online, 5835-5847. https://doi.org/10.18653/v1/2021.naacl-main.466

[86] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. (2019).

[87] Ori Ram, Gal Shachaf, Omer Levy, Jonathan Berant, and Amir Globerson. 2022. Learning to Retrieve Passages without Supervision. In *Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*. Association for Computational Linguistics, Seattle, United States, 2687-2700. https://doi.org/10.18653/v1/2022.naacl-main.193

[88] Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In *EMNLP/IJCNLP (1)*, Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, 3980-3990. http://dblp.uni-trier.de/db/conf/emnlp/emnlp2019-1.html#ReimersG19

[89] Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021. RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking. In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 2825-2835. https://doi.org/10.18653/v1/2021.emnlp-main.224

[90] Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. *Found. Trends Inf. Retr.* 3, 4 (apr 2009), 333-389. https://doi.org/10.1561/1500000019

[91] Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. 1994. Okapi at TREC-3.. In *TREC (NIST Special Publication, Vol. 500-225)*, Donna K. Harman (Ed.). National Institute of Standards and Technology (NIST), 109-126. http://dblp.uni-trier.de/db/conf/trec/trec94.html#RobertsonWJHG94

[92] Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. 2019. Spreading vectors for similarity search. In *International Conference on Learning Representations*. https://openreview.net/forum?id=SkGuG2R5tm

[93] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. In *5th Workshop on Energy Efficient Machine Learning and Cognitive Computing @ NeurIPS 2019*. arXiv:1910.01108 http://arxiv.org/abs/1910.01108

[94] Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022. ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. In *Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics*:
*Human Language Technologies*. Association for Computational Linguistics, Seattle, United States, 3715–3734. https://doi.org/10.18653/v1/2022.naacl-main.272

[95] Keshav Santhanam, Jon Saad-Falcon, Martin Franz, Omar Khattab, Avirup Sil, Radu Florian, Md Arafat Sultan, Salim Roukos, Matei Zaharia, and Christopher Potts. 2022. Moving Beyond Downstream Task Accuracy for Information Retrieval Benchmarking. https://doi.org/10.48550/ARXIV.2212.01340

[96] Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In *Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*. Association for Computational Linguistics, Berlin, Germany, 1715–1725. https://doi.org/10.18653/v1/P16-1162

[97] Tao Shen, Xiubo Geng, Chongyang Tao, Can Xu, Xiaolong Huang, Binxing Jiao, Linjun Yang, and Daxin Jiang. 2022. LexMAE: Lexicon-Bottlenecked Pretraining for Large-Scale Retrieval. https://doi.org/10.48550/ARXIV.2208.14754

[98] Sivic and Zisserman. 2003. Video Google: a text retrieval approach to object matching in videos. In *Proceedings Ninth IEEE International Conference on Computer Vision*. 1470–1477 vol.2. https://doi.org/10.1109/ICCV.2003.1238663

[99] Livio Baldini Soares, Daniel Gillick, Jeremy R. Cole, and Tom Kwiatkowski. 2023. NAIL: Lexical Retrieval Indices with Efficient Non-Autoregressive Decoders. arXiv:2305.14499 [cs.CL]

[100] Ian Soboroff. 2018. Meta-Analysis for Retrieval Experiments Involving Multiple Test Collections. In *Proceedings of the 27th ACM International Conference on Information and Knowledge Management* (Torino, Italy) (CIKM '18). Association for Computing Machinery, New York, NY, USA, 713–722. https://doi.org/10.1145/3269206.3271719

[101] Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. In *Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)*. https://openreview.net/forum?id=wCu6T5xfjeJ

[102] Nandan Thakur, Kexin Wang, Iryna Gurevych, and Jimmy Lin. 2023. SPRINT: A Unified Toolkit for Evaluating and Demystifying Zero-shot Neural Sparse Retrieval. arXiv:2307.10488 [cs.IR]

[103] Nicola Tonellotto. 2022. Lecture Notes on Neural Information Retrieval. https://doi.org/10.48550/ARXIV.2207.13443

[104] Nicola Tonellotto, Craig Macdonald, and Iadh Ounis. 2018. Efficient Query Processing for Scalable Web Search. *Foundations and Trends*® in Information Retrieval 12, 4-5 (2018), 319–500. https://doi.org/10.1561/1500000057

[105] Howard Turtle and James Flood. 1995. Query Evaluation: Strategies and Optimizations. *Inf. Process. Manage*. 31, 6 (nov 1995), 831–850. https://doi.org/10.1016/0306-4573(95)00020-H

[106] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In *Advances in Neural Information Processing Systems*, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

[107] Kexin Wang, Nils Reimers, and Iryna Gurevych. 2021. TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning. In *Findings of the Association for Computational Linguistics: EMNLP 2021*. Association for Computational Linguistics, Punta Cana, Dominican Republic, 671–688. https://doi.org/10.18653/v1/2021.findings-emnlp.59

[108] Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2022. GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval. In *Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*. Association for Computational Linguistics, Seattle, United States, 2345–2360. https://doi.org/10.18653/v1/2022.nacl-main.168

[109] Lidan Wang, Jimmy Lin, and Donald Metzler. 2010. Learning to Efficiently Rank. In *Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval* (Geneva, Switzerland) (SIGIR '10). Association for Computing Machinery, New York, NY, USA, 138–145. https://doi.org/10.1145/1835449.1835475

[110] Shuai Wang, Shengyao Zhuang, and Guido Zuccon. 2021. BERT-Based Dense Retrievers Require Interpolation with BM25 for Effective Passage Retrieval. In *Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval* (Virtual Event, Canada) (ICTIR '21). Association for Computing Machinery, New York, NY, USA, 317–324. https://doi.org/10.1145/3471158.3472233

[111] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*. Association for Computational Linguistics, Online, 38–45. https://doi.org/10.18653/v1/2020.emnlp-demos.6

[112] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. https://doi.org/10.48550/ARXIV.1609.08144

[113] Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020. CLEAR: Contrastive Learning for Sentence Representation. https://doi.org/10.48550/ARXIV.2012.15466

[114] Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. In *9th International Conference on Learning Representations, ICLR 2021, Virtual Manuscript submitted to ACM*
Event, Austria, May 3-7, 2021. OpenReview.net. https://openreview.net/forum?id=zeFrfgyZln

[115] Ikuya Yamada, Akari Asai, and Hannaneh Hajishirzi. 2021. Efficient Passage Retrieval with Hashing for Open-domain Question Answering. In *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)*. Association for Computational Linguistics, Online, 979–986. https://doi.org/10.18653/v1/2021.acl-short.123

[116] Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the Use of Lucene for Information Retrieval Research. In *Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval* (Shinjuku, Tokyo, Japan) (SIGIR '17). Association for Computing Machinery, New York, NY, USA, 1253–1256. https://doi.org/10.1145/3077136.3080721

[117] Wei Yang, Kuang Lu, Peilin Yang, and Jimmy Lin. 2019. Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models. In *Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval* (Paris, France) (SIGIR '19). Association for Computing Machinery, New York, NY, USA, 1129–1132. https://doi.org/10.1145/3331184.3331340

[118] Hamed Zamani, Mostafa Dehghani, W. Bruce Croft, Erik Learned-Miller, and Jaap Kamps. 2018. From Neural Re-Ranking to Neural Ranking: Learning a Sparse Representation for Inverted Indexing. In *Proceedings of the 27th ACM International Conference on Information and Knowledge Management* (Torino, Italy) (CIKM '18). Association for Computing Machinery, New York, NY, USA, 497–506. https://doi.org/10.1145/3269206.3271800

[119] Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2022. Learning Discrete Representations via Constrained Clustering for Effective and Efficient Dense Retrieval. In *Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining* (Virtual Event, AZ, USA) (WSDM '22). Association for Computing Machinery, New York, NY, USA, 1328–1336. https://doi.org/10.1145/3488560.3498443

[120] Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. 2020. Learning To Retrieve: How to Train a Dense Retrieval Model Effectively and Efficiently. https://doi.org/10.48550/ARXIV.2010.10469

[121] Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen. 2022. Adversarial Retriever-Ranker for Dense Text Retrieval. In *International Conference on Learning Representations*. https://openreview.net/forum?id=MR7XubKUFB

[122] Tiancheng Zhao, Xiaopeng Lu, and Kyusong Lee. 2021. SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval. In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*. Association for Computational Linguistics, Online, 565–575. https://doi.org/10.18653/v1/2021.nacl-main.47

[123] Shengyao Zhuang and Guido Zuccon. 2021. Fast Passage Re-ranking with Contextualized Exact Term Matching and Efficient Passage Expansion. https://doi.org/10.48550/ARXIV.2108.08513

[124] Shengyao Zhuang and Guido Zuccon. 2021. TILDE: Term Independent Likelihood Model for Passage Re-Ranking. In *Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval* (Virtual Event, Canada) (SIGIR '21). Association for Computing Machinery, New York, NY, USA, 1483–1492. https://doi.org/10.1145/3404835.3462922