Learning Retrieval Models with Sparse Autoencoders
Authors: Thibault Formal
Maxime Louis
Hervé Déjean
Stéphane Clinchant
Venue: ICLR 2026 Poster (2026)
Keywords: text embedding
sparse autoencoders
sparse retrieval
large language models
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We introduce a novel competitive retrieval model built on sparse autoencoders that generates generalizable, multilingual sparse latent embeddings.

Abstract: Sparse autoencoders (SAEs) provide a powerful mechanism for decomposing the dense representations produced by Large Language Models (LLMs) into interpretable latent features. We posit that SAEs constitute a natural foundation for Learned Sparse Retrieval (LSR), whose objective is to encode queries and documents into  high-dimensional sparse representations optimized for efficient retrieval. In contrast to existing LSR approaches that project input sequences into the vocabulary space, SAE-based representations offer the potential to produce more semantically structured, expressive, and language-agnostic features. By leveraging recently released open-source SAEs, we show that their latent features can serve as effective indexing units for representing documents and queries for sparse retrieval. Our experiments demonstrate that SAE-based LSR models consistently outperform their vocabulary-based counterparts in multilingual and out-of-domain settings. Finally, we introduce SPLARE, a 7B-parameter multilingual retrieval model capable of producing generalizable sparse latent embeddings for a wide range of languages and domains, achieving top results on MMTEB’s multilingual and English retrieval tasks. We also release a more efficient 2B-parameter variant, offering strong performance with a significantly lighter footprint.

Published as a conference paper at ICLR 2026
LEARNING RETRIEVAL MODELS WITH SPARSE AUTOENCODERS
Thibault Formal
Maxime Louis"
Naver Labs Europe
Naver Labs Europe
thibault formal gma com
maxime, louis s. x201 28gmai coom
Hervé Déjean
Stéphane Clinchant
Naver Labs Europe herve. den.eeennneeeeelbabrr com
Naver Labs Europe st stephane. clinchantehaverllabee
ABSTRACT
Sparse autoencoders (SAEs) provide a powerful mechanism for decomposing the dense representations produced by Large Language Models (LLMs) into inter- pretable latent features. We posit that SAEs constitute a natural foundation for Learned Sparse Retrieval (LSR), whose objective is encode queries and doc- uments into high-dimensional sparse representations optimized for efficient re- trieval. In contrast existing LSR approaches that project input sequences into the vocabulary space, SAE-based representations offer the potential to produce more semantically structured, expressive, and language-agnostic features. Build- ing on this insight, introduce SPLARE, method to train SAE-based LSR models. Our experiments, relying on recently released open-source SAEs, demon- strate that this technique consistently outperforms vocabulary baased LSR in mul- tilingual and out-of-domain settings. We developed SPLARE-7B. multilingual retrieval model capable of producing generalizable sparse latent embeddings for wide range of languages and domains, achieving top results on MMTEB's multi- lingual and English retrieval tasks. We also propose a 2B-parameter variant with a significantly lighter footprint.
INTRODUCTION
Emhedding models have become pivotal tool for search systems, enabling the better capture of semantic relationships between queries and documents across various domains and modalities. This trend has been further accelerated by the advent of Detrieval-Augmenter Generation (RAG) Lewis et al. (2020) and agent-based systems, which impose even higher demands on retrieval performance and robustness. Recently, dense embedding models Reimers & Gurevych (2019), Karpukhin et al. (2020), which map inputs into single dense vectors, have demonstrated impressive performance on the (M)MTEB benchmark (Muennighoff et al.) [2023 Enevoldsen et al. 2025). Specifically, embed- ding models relying large (V)LLM backbones have become the de facto approach for generalist multilingual Lee al. (2025b); Zhang et al. (2025); WWag et al. (2024a); [Lee et al. (2025a); Li modti models Günther et al. (2025), Faysse al. (2025); Xu et al. (2025a) marking a shift away from encoder-only language models which have defined the state of
ct al. (2023b) or
the art for years (Izacard et al. 2022 Karpukhin al. 2020 Xiong al. 2020).
Learned Sparse Retrieval (LSR) methods Formal et al. 2021]: Mallia et al. (2021) Nguyen et al. (2023); Kong et al.] (2023) have achieved ofate-of-the-ooe art performance widely used English- centric benchmarks Thakur et al. (2021) )); [Bajaj er al. (2018): Craswell et al. (2021) and have demon- strated strong generalization when compared dense embedding models (Formal et al. 2022b Lupart et al. 2023 Déjean et al. (002). Beyond their efficiency, these approaches provide level of interpretability that particularly valuable in production systems. Models such as SPLADE For-
mal et al. 2022 2022a): Lassance et al. (2024) operationalizex this idea by representing documents
"Contributed equally. Corresponding authors.

---

Published as a conference paper at ICLR 2026
and queries sparse, weighted Dag-of-worde over the vocabulary space of their backbone model. While originally developed for encoder-only architectures such as BERT Devlin et al. (2019). recent work has explored adapting SPLADE to LLM backbones (Qiao et al. 2025 Doshi al. [2224] Xu
et al. 2025b Zeng et al., 2025; Soares et al. 2023 Ma et al., 2025). However, these models remain limited English-centrie contexts and struggle to match state-of-the-an performance on more com- prehensive benchmarks like MMTEB which place greater emphasis on generalization across novel domains and languages. Unlike dense retrieval, which models relevance within continuous em- bedding space, LSR methods are inherently constrained by the fixed vocabulary of their underlying backbone, which incurs issues such as tokenization redundancy (Lei et al. 2025). This limitation also makes it significantly harder to handle multilingual or cross-lingual retrieval Nair et al. (2023)
2022:
Lassance ) -and even more so when extending to multimodal settings (Nguyen et al,,
2024 We hypothesize that this is a key reason why LSR models have recently fallen behind dense
approaches
In the context of LLMs, Sparse Autoencoders (SAEs) Makhzani & Frey 2013); Huben et al. (024): Bricken et al. (2023) decompose dense token representations into sparse vectors of la- tent features. These features have been shown to exhibit desirable properties: they are largely mono-semantic (most features correspond to a single interpretable concept), multilingual (remain- ing largely language-agnostic) and even multimodal (generalizing across modalities in multimodal LLMs) (Bricken et al., 2023] "Tmmpletn et al. 2024 Lieberum et al.] [024] Huben et al. [024 He
al. 2024; Deng al., (025). While SAEs have generated significant excitement for mechanistic interpretability, recent work has also highlighted their limitations, showing that they can struggle to
transfer effectively certain downstream tasks (Kantamneni et al. 2025 Smith al. 2023).
In this work, we argue and empirically demonstrate that SAEs are a natural fit for LSR models: their representation space for sparse retrieval
learned latent features provide a which is particularly advantageous in domains or languages where vocabulary- may underperform. To this end, we propose a new LSR approach that represents queries and doeu- ments as sparse vectors over a latent vocabulary space, by replacing the standard language modeling (LM) head with pre-trained SAEs such as Llama Scope (He et al.) 2024). More specifically, our contributions are as follows:
approaches
We introduce SPLARE for SParse LAtent REtricval-a new LSR approach relying on pre-trained SAEs;
We conduct a systematic investigation of the advantages of using a latent vocabu- wuy-comparedd to the standard LLM vocabulary-acreee across comprehensive set of bench- marks spanning diverse tasks, domains, and languages;
Finally, we introduce a new 7B multilingual latent sparse retriever that supports 100+ lan- guages through cross-lingual transfer and achieves competitive results the MMTEB retrieval benchmark. SPLARE is the first LSR model to rival state-of-the-art dense ap- proaches on MMTEB, by fine-tuning only on a large open-source dataset, without addi- tional pretraining or data augmentation. We additionally develop a compact and efficient 2B counterpart.
BACKGROUND
We first provide some background on Sparse Autoencoders well as Learned Sparse Retrieval. SPLARE can be understood as synthesizing these two research directions into a unified framework.
2.1 SPARSE AUTOENCODERS
Given activations x E Rd from a language model, Sparse Autoencoder (SAE) single hidden layer model, comprising an encoder and decoder:
f(Wenc ene + bene), f == Wdoc base
where & E RR IWI, with W >> d corresponding to the width of SAE, i.c., the number of features in the latent space. SABs, class of autoencoders, are trained using standard reconstruction
As of the time of writing, no sparse retrieval model listed the MTTE(MMllilnnuull, v2) leaderboard.
2

---

Published as a conference paper at ICLR 2026
transformer backbone
R WW]
internuts to stecroaapet cemporets their jereects
refermnce to aiopriaccee and reterteed beteet
20
inference
[=32
Top-K
SPLADE pool
1=22
LoRA
SAE
SPLARE index (offline)
retrieval
S( ( enci Denk)
t1
13
magnetic flux
magnétique
X
Figure 1: Overview of SPLARE. A pre-trained SAE can be inserted at any layer I of the LLM to sparse latent representations of input tokens. These token-level representations are then aggregated into a single sparse vector using pooling mechanism analogous to SPLADE. During training, we only fine-tune the L.M parameters (via LoRA adapters) while keeping the SAE frozen. We show the top-5 activated features Lin (2023) for the English query-foo for our final SPLARE model.
objective L | x| Sparsity in the decomposition is induced through suitable activation functions / such as ReLU Bricken et al. (2023), Top-K Makhzani Frey (2013); Gao et al. (2025)
or JumpReLU Rajamanoharan et al. (2024), and/or regularization penalties such l1. Several works have demonstrated that SAEs can recover highly monosemantic features, many of which the
concepts across languages -ndd, in some cases, even multimodal (Huben et al.] 20244 Bricken et al.] [023) Templeton et al., 2024 Lieberum al., 2024 He et [2024] Cunningham
language-agnostic- aaaguugg-ggnottic--reonnnidiiiio responding consistently
Conerly, [024 Deng al.] 2025). Large Sparse Autoencoders are also notoriously hard and costly to train. Recently, high-quality large scale open-source SAEs have become available to the research community. In particular,
rely in this work on the Llama Scope series of models He et al. (2024) which offers SAEs trained on Llama-3.1- 8B and the Gemma Scope suite Lieberum et (2024) which offers SAEs trained Gemma-2-2B, 9B and 27B models.
2.2 SPLADE
Learned Sparse Retrieval (LSR) models aim map input sequences into high-dimensional sparse representations for efficient retrieval. Among these approaches, the SPLADE family of ap- proaches Formal et al. (2021 2022a): Lassance al. (2024) has emerged method, achieving performance comparable to or exceeding that of dense embedding models in many settings. Given an input sequence tokenized f (1 12 .,. (n) and fed through all the layers of the transformer, SPLADE generates sequence of logits (V), U2,..., vn) by projecting each final hidden state (h1, h2, f2...... ... hn) onto the vocabulary space V using the language modeling
the state-of-the-an
head. The weights (Vir)æev correspond to an unnormalized proobabiiyyy distribution over V for token t, where each output dimension j is actually associated with the token it represents. To ob- tain single sequence-level representation, SPLADE first applies a term saturation function, before max- max-pooling over the sequence:
= max log (1 ReLU(()))) =I....
Given these sparse representations u E R for queries and documents, relevance scores are com- puted as sparse dot product s(g. d) =< u², uº >. This operation can be efficiently supported using inverted index structures together with specialized query processing techniques (Tonellotto et al. 2018 Bruch al. 2024c Zobel & Moffat 2006).

---

Published as a conference paper at ICLR 2026
3 METHOD
3.1 SPLARE
Conceptually, SPLARE closely parallels SPLADE but operates in the latent representation space. Rather than projecting the final hidden states of the language model onto the vocabulary space via the LM head, SPLARE employs sparse autoencoders to transform representations from a selected layer into sparse latent space, which can be interpreted as a latent vocabulary.
enc, benc) in Eq. denote the SAE's encoder parameters at a given layer I of the transforme F Similarly to SPLADE. we can obtain sequences of sparse latent logits (w1, W2,.... Wn) by mapping the hidden states at layer/ with the SAE encoder as illustrated in Figure | The weights (wis))Ew) e
Let (Wen:
RIWI contain the sparse list of latent features associated with token in the input sequence. can be used in place of the vocabulary decomposition to compute sequence-level representations for input
queries or documents into sparse set of latent features using the same type of pooling mechanism which we refer as SPLADE-pon -pool in Figure
as in Eq. 2
3.2 TRAINING
Training LSR Models. The training procedure for LSR models mirrors that of dense embedding models. While contrastive learning Oord et al. 20 Chen et al. (2020) is the de-facto approach to train state-of-the-arr dense models Lee et al. (2025b); Zhang al. (2025), we instead adopt distillation-basee approach using cross-encoder teacher model NNgueirr & Cho (2020) to train sparse embeddings. Specifically, we optimize the Kullback-Leibler divergence between the teacher and student relevance distributions (Lin et al., (020). Given a query g. (d1, d2,... dm) which contains positive document and pool of hard negatives, (2,, 8m) the corresponding teacher scores for documents d; with respect to q. and T a temperature parameter, the training loss given by:
(an,ii// LK.. Σ Ps (log logg:) Ds E, ealadd)/+
P
E5es
Distillation vs Contrastive Learning. Distillation is common toolbox to train retrieval mod- els Hofstätter et al. (2020): [in al.) (2020), but has been overlooked in the context of LLM-based embeddings. State-of-the-art embedding models typically rely contrastive learning, which is generally effective but suffers from well-known issues such as false negatives. As result, many recent systems incorporate filtering mechanisms for negative samples-ofie often using cross-encoders or LLMMs-hich can themselves be viewed as implicit forms of distillation (Lee et al. 2025b] de Souza P. Moreira al. 2025). In addition, since retrieval model mirrors SPLADE (with the vocabulary projection being the only difference), we follow the established training practices for SPLADE (Formal et al. 2022a Lassance et al. 2024).
Sparsity. To encourage sparsity in query and document representations, LSR models are typically trained with a sparsity-inducing regularization term, analogous to that used in SAFs. We use the FLOPS loss Paria et al. (2020) employed in SPLADE. The final loss is:
L = LKI. + X PLoPs Aae
(4)
The sparsity of LSR approaches plays crucial role in determining both effectiveness and computa- tional efficiency on retrieval benchmarks. However, the sparsity induced by L can vary significantly depending the model configuration, backbone architecture, SAE suite. and dataset characteris- tics. Achieving a desired target sparsity would require continuous adjustment of Xd.q- To mitigate this challenge and establish more robust training setup. additionally apply Top-K pooling at inference time, as illustrated in Figure [ This strategy allows us to train a single model with mod- erate sparsity-using fixed, conservative values of Ad. while systematically studying the effect of pooling without the need for re- re-training Although some prior works have entirely replaced explicit
2 Note that we only rely the encoder parameters, as only aim to extract sparse features from represen- tations. Also that consider SAFs trained on the residual streams of the transformer.
4

---

Published as a conference paper at ICLR 2026
sparsity regularization with Top-K pooling (Lassance et al. 2023 Doshi et al. periments with this approach yielded inferior results. Note that Top- acts as a strict upper bound: depending the dataset, queries and documents may contain fewer active dimensions due to the inherent sparsity of the representations.
)024), our initial ex-
produces sparse token- sparsity at initialization remains relatively high (e.g.. a few thousands non-zero values). As a result, additional sparsity regularization is required to ensure the model achieves the desired efficiency. It is also worth noting that LSR models are usually hard to train and require a careful initialization of the projection head. While the LM head or a SAE can provide a suitable initialization, training an LSR model entirely from scratch is highly difficult and consistently results in much lower performance- when converging.
Finally, note that while SPLARE is initialized with an level cerrrsemtationsssssseneeeeeeeeee
4 EXPERIMENTAL SETUP
Training Data. We conduct two large sets of experiments: contains various ablations and analyses for models trained on English data on the MS MARCO dataset (Bajaj et al., 2018). In $ 6 we further extend training to a larger set of publicly available data, including multilingual datasets. We do not prepend any special instructions or prefix to our input sequences- -wichh could only likely yield further improvements. To ease reproducibility, we also refrain from any form of pre-finetuning or synthetic data generation Lee al. (2025b); Günther al. (2025); Zhang al. (2025), both of which have recently become common practice for achieving top results on the MTEB benchmark. We detail in Appendix A our two training settings.
Evaluation. MTEB Muennighoff et al. (023) and MMTEB Enevoldsen et al. 2025) are the most widely adopted benchmarks for evaluating embedding models. Our evaluation focuses only on the retrieval subsets of these benchmarks, excluding other task categories. In addition the En- glish and Multilingual splits, we also report results on domain-speciio Code, Medical, Law, and Chemical domains. Given SPLARE's s strong performance in multilin- gual settings, we further place particular emphasis on this aspect by including language specific splits of MMTEB for five languages, as well as evaluations on the MIRACI /haan al. (2023) and XTREME-UP|Rude et al. (2023) datasets. The latter introduces challenging cross-lingual retrinnn retrieval task, requiring retrieval from an English corpus using queries from low-resource languages. We also
subsets of MTEB, including
report results on MS MARCO Bajaj et al. (2018) and BEIR Thakur et al. (2021) (Appendix C
While our approach broadly applicable to any pre-trained SAE, we conduct the majority of our experiments using the Llama Scope model suite He et al. (2024). built on Llama-3.1 3.1-8B (et al., 2024). During training, we fine-tune the backbone with LoRA adapters Hu et al.) (2022) while keep- ing SAE parameters frozen. Preliminary experiments indicated that this strategy not only improves performance but also simplifies training. Moreover, it preserves the interpretability of the latent fea- ture space (Lin) 2023). As in prior work Zeng et al. (2025): [BehnamGhader al. (2024): Lei et al. 2025), we enable bidirectional attention across all backbones and pretrain them with Masked Next Token Prediction. Following the exact procedure of ofZeng et al. al. (2025), we mask 20% of tokens the MS MARCO corpus and train for 10k steps which takes about five hours. Bidirectional attention is particularly important for LSR models since pooling occurs at every position of the input sequence, unlike dense models that rely on the <EOS> token. Full details of our experimental hyperparameters are provided Appendix B Unless stated otherwise, retrieval evaluation is performed using Top-K pooling, with default values of A 40 for queries and A 400 for documents. For our multilingual models ($ (6. we additionally rely on model averaging Wortsman et al. (2022) from several training runs, which boosts generalization performance (Lee al. 20256 Zhang et al. 2025).
We are mainly interested in comparing SPLARE to current -fate-of-the-art LSR methods, which all vocabulary-based. To this end, we perform controlled comparisons with SPLADE model built on the same Llama-3.1 1-B backbone- backbone-ffolowingg the methodology of Doshi et al. (022)); Zeng al. 2025) and trained under identical settings. We refer to this baseline 3PLADE-Laamm
5

---

Published as a conference paper at ICLR 2026
50.0
49.5
MTEB(eng, v2)
49.0
45
MTnB(eng.
40
Liama- |W] 131k
47.5
Llama-Scope, [W] 32k
Gemma-2B, Layer 12
35
47.0
Gemma-Scope [W] 65 k
Gemma-2B, Layer 19
10
15 Layer
20
25
30
10a
10f
SAE width
Figure 2: (Left) Performance across layers on Llama Scope (Llama-3.1-8B) and Gemma Scope (Gemma-2-2B). (Right) Performance with increasing SAE width Gemma-2. Evaluation done with Top-K (40,400)
5 ANALYSIS AND DESIGN CHOICES FOR SPLARE MODELS
We first conduct series of ablation studies in controlled, English-only setting. At this stage, our primary objective is to compare SPLARE's latent representations with traditional vocabulary- based approaches (i.e., SPLADE-Llama baseline). Specifically,
aim to address the following research questions: (i) At which transformer layer depth do we obtain the most effective sparse latent representations for retrieval? (ii) How does the width of the SAE affect retrieval performance? (iii) What are the efficiency-ffeettiveess trade-offs introduced by the latent vocabulary? (iv) Do the sparse latent features learned by the SAE yield improvements over equivalent SPLADE models?
Performance and Layer Depth. We train SPLARE models at varying depths on Llama-3.1-88 3-18BB using SAEs from Llama Scope with two widths [W] E (32k, 131k), and on Gemma-2-2B, using Gemma Scope with width [W]] = 65k, and report the average MTEB (English, v2) performance in Figure 2 (Lef). Interestingly, the highest performance is consistently achieved at about two-thirds of the model depth, i.c., around layer 20 (out of 32) for Llama Scope and 16 (out of 26) for Gemma Scope. These findings are consistent with prior work suggesting that intermediate transformer layers often yield richer representations for retrieval tasks (Skean al. 2025 Zhuang et al.) 2025 Wang jet al. 2025). A further advantage of using intermediate layers is the reduction in retriever size and, consequently, inference latency-an improvement over SPLADE models, which require processing through all layers of the LLM (see Appendix F
While the best performance often appears relatively deep layers, this choice is not critical: per- formance at carlier layers remains strong, and the selection ultimately reflects an effectiveness- efficiency trade-off. A practical rule of thumb to select a layer around two-thirds of the model depth to maximize effecttivnnsss For the remainder of the paper, our main SPLARE models are trained layer 26 of Llama-3.1-8B, yielding 7B-parameter model (including the SAE parame- ters), but we also train strong 2B model (layer 6) in $ $
How does the Width of the SAE Affeet Retrieval Performance? Unlike SPLADE models, the dimensionality of SPLARE's feature space-determinn determined by the SAE width W not constrained by the LLM's vocabulary size. To study the effect of SAE width on retrieval effectiveness, we train multiple SPLARE models using Gemma Scope, which offers broader range of SAE configurations. Especially, we consider SAES at layers 12 and 19 of Gemma-2-2B with widths [W] E {224 so 16k, 225 220 1M). We report the resulting average MTEB (English, v2) performance in Figure 2 Right). (Right) Our results show roughly log- lin-linear relationship between SAE width and retrieval effectiveness, providing scaling mechanism for improved performance something not possible with SPLADE's fixed vocabulary size. Prior work has shown that SAEs can scale to widths as large as 14M on very large L.MMs Templeton et (024), though such models remain proprietary. Llama Scope, while limited to [W[ E {32k, 1311 exhibits the same scaling effect consistently
Unless otherwise specified, SPLARE refers SPLARE-7B.

---

Published as a conference paper at ICLR 2026
10'
50
Number of documents
10
SPLADE-Llame
SPLARE
MTEB(eng: v2)
10a
45
10
40
10a
35
10
SPLARE
101
30
SPLADE- Liama
100
0
50
100
150
200
250
300 350
400
20000
40000 60000 80000 100000 120000
100000120000
Top-K document
Sorted dimensions
Figure 3: (Left) Impact of pruning documents with Top-K (with k 40 for queries). (Right) MS MARCO index distribution for SPLARE and SPLADE (8.8M documents).
across layers (Figure (Left)). These experiments also highlight that the approach is transferable across different backbone architectures. Despite the availability of much wider SAEs in Gemma Scope, we observe that Llama Scope models achieve superior overall performance. Consequently, we report results using this model (with [W] k31k) for all subsequent experiments.
Srrectiveenss-Eftiiinneee Trade-Ofr. Sparse retrieval methods achieve efficiency through the use of dedicated inverted index structures and exact Zobel & Moffat (2066) or approximate Bruch et al. (2024a) query processing algorithms. In all cases, obtaining highly sparse representations critical for achieving low-latency retrieval. While SPLADE has been successfully adapted to LLM back- bones, efficiency considerations have generally been overlooked. As discussed in $3.22 LSR models can easily become "dense" in practical scenarios, which undermines their efficiency.
We study the relationship between SPLARE performance and sparsity by capping, at inference time, the number of activated features for document vectors using Top-K pooling. Results are shown in Figure
3 (Left). SPLARE exhibits substantially greater robustness to document pruning: when in- dexing only Top-K 100 document features, its performance drops by merely 2%, compared to over 6% for SPLADE. This difference can be partially attributed to SPLARE's more compact and structured latent feature space as well as the fact that SPLADE models based on LLMs are inher- ently harder to sparsify. As we show in Appendix E this difference translates into lower query latency at given accuracy level, when evaluated using Seismic (Bruch al. 2024a by 2025). For reference, performing retrieval with SPLARE (Top-K- (40, 400) on MS MARCO (8.8M docu- ments) requires about 5ms per query only accounting model inference. Figure (Right) further illustrates the distributions of activated features after training. Notably. SPLARE utilizes a much larger portion of the available feature space, activating nearly all dimensions, in contrast to SPLADE, which relies on fewer than 100k dimensions (out of 128k). Moreover, SPLARE ex- hibits more balanced activation distribution across features. By comparison, SPLADE tends to over-activate small subset of dimensions (Mackenzie et al. 2023 Lei et al. 2025).
Comparison of Lexical and Latent Features. Finally, we compare the performance of SPLARE with existing top LSR methods trained the English MS MARCO dataset. In particular, Lion-SP- 8B Zeng et al. (2025) represents the most effective contemporary SPLADE adaptation for LLM- based retrieval. We show the results on Table [ for various splits of MTEB (English Models). First, notice that SPLADE-Llama (our baseline) aTready significantly outperforms Lion-SP-8B (e.g. +4.4 on MTEB English). We further observe that SPLARE consistently outperforms competing methods on both multilingual and several out-of-domain evaluation sets. In particular, it achieves an improvement of roughly two points on the multilingual split and shows superior performance on the Law and Medical retrieval benchmarks- Chemical splits. The observed multilingual generalization from English-only training is unsurpris- nature of SAE features. Meanwhile, the performance drop on the Code tasks is likely due to the highly cammain-specifi nature code retrieval, which does not align well with the features learned by the SAE (a trend that is further supported by observations in S (). To illustrate this behavior, we provide in Appendix G (Figures [5 several examples where SPLARE underperforms compared to SPLADE on MTEB Code. In these cases, the top activated
its advantage diminishes on the Code and
ing, given the anguage-agnostte
7

---

Published as a conference paper at ICLR 2026
Table Average performance on various MTEB splits. English models are trained on MS MARCO only 5.. Multilingual models are trained on lcage-scale multilingual training set ($ Evaluation done with Top-K (40, 400).
English
Multilingual
Code
Medical
Law
ChemTEB
English Models SPLADE-v3 (Lassance 2024
50.7
38.1
44.5
44.2
40.4
75.6
Lion-SP-8B (Zeng et al. 2025
50.0
53.3
54.4
48.5
71.1
SPLADE-Llama
$2.9
54.3
57.3
61.0
49.0
75.9
SPLARE
52.9
56.3
55.1
62.9
51.2
70.0
Multilingual Models SPLADE-Llama
58.9
61.7
64.3
67.6
60.7
77.4
SPLARE
59.3
62.3
63.0
67.7
60.8
78.1
Table 2: Multilingual comparison of SPLARE and SPLADE (Top-K 400)).
indic
sca
deu
fra
kor
XTREME-UP
MIRACL
SPLADE-Llame 91.9
70.4
57.3
65.6
74.8
56.2
69.9
SPLARE
92.3
70.8
57.1
64.8
76.0
58.6
71.7
features appear overly generic rather than specialized to code semanties. This suggests that for highly samain-specific scenarios such as code retrieval, dedicated SAEs trained on code-focused corpora may be more appropriate. We leave this direction for future work.
MULTILINGUAL MODELS
In $ we showed how the latent feature space of the SAE offers some advantages for LSR mod- els when compared to the vocabulary space-inh a controlled English-based setting. In $ $6.1 we further extend those findings for multilingual models, by training models on large-scale multilin- gual dataset and broadening the evaluation to cover more diverse of benchmarks, as detailed in In $ 6.2 we compare SPLARE to concurrent models on (M)MTEB and XTREME-UP.
COMPARING LATENT MODELS TO LEXICON-BASEE BASED APPROACHES
Table T (Multilingual models) reports the average performance of multilingual SPLARE and SPLADE across the various MTEB splits, with complete results available in Appendix D Overall, SPLARE consistently outperforms its vocabulary-based counterpart, except on the Code split-an an outcome aligned with the discussion in $ Notably, the performance gap is larger on the Multilin- gual split of MTEB (+0.6 average nDCG@ @ 10). Examining the mulrilinguul-ont datasets within this split confirms that SPLARE systematically surpasses SPLADE (Table 8 +0.9 points). This trend is confirmed by Table 2 which underscores the advantage of latent-based LSR models multilingual scenarios (e.g., +2.4 on XTREME-UP and +1.8 on MIRACL). Comprehensive results for MIR ACL and XTREME-UP. together with comparisons to concurrent methods, are presented in Appendix D and Table 3 respectively. SPLARE achieves particularly strong performance the hidden test sets of MIRACL (Table 11 de and yo) as well as on the low-resource languages of XTREME-UP.
6.2 COMPARING TO TOP MODELS
Finally, we compare SPLARE to top models from the MTEB leaderboard in Table SPLARE reaches an average score of 62.3 (for the pooled version), making among the top 10 models on MTEB(Mulillnguudl v2) retrieval and the top-1 LSR model. Notably, these results are achieved without relying on private or synthetic data and without any pre-finetuning. This is also particu- larly interesting, open models like gte-Qwen2-78B 7B instruct or NV-Embed-v2 rely 3584-d (resp. 4096-d) dense vectors to encode queries and documents, while SPLARE only needs 40 features (resp. 400) encode queries (resp. documents) in its high-dimensional feature space reach high
8

---

Published as a conference paper at ICLR 2026
Table Average MTEB retrieval performance of SPLARE against top models. Multilingual (resp. Eng) refers to MTEB(Mulillnggaa, v2) (resp. MTEB(eng, v2)). At the time of writing (February 26, 2026), SPLARE ranks in the top-10 models on MTEB(Muliilngull, v2) retrieval. For XTREME- UP. we report results from (Lee al. 2025b). Unless specified, evaluation for SPLARE is done with Top-K (40, coresponnn to our default model SPLARE or SPLARE-2B.
English
Multilingual
XTREME-UP
Top Open Source models e5-mistral-7b-inite (Wang al. 2024a
57.6
55.8
NV-Embed-v2 [ee et al. 2025a)
62.8
56.7
Buttliinuul-eS--lrge-nnne
(Wang al.
2024b
53.5
57.1
18.7
Grit (Muennighoff et al. 2024)
55.0
58.3
SFR-Embeddngg-Win (Meng al.
2024
59.3
59.4
Ling-Embed-Mistral (Kim et al.
2024
60.1
58.7
24.6
gte-Qwen2-7B-innsruei (Li al.
2023b)
58.1
60.1
17.4
voyage-3-large (Al
2025)
53.5
66.1
39.2
jina-embeddings.ve Gunther
2025
56.2
66.4
inf-retriever-vll (Yang et al. 2025
64.1
66.5
Qwen-3-Embeddmng-e (Zhang et al.
2025
69.4
70.9
Commercial models
Cobeen-enthecd-mttiinnnul---- (Cohere
2023)
55.7
59.2
text-embeedding---largee OpenAl
2024)
58.0
59.3
18.8
gemini-embeeding-000 (Lee et 2025b
64.4
67.7
64.3
SPLARE
59.3
62.3
58.6
SPLARE no-pooling
61.4
63.8
61.4
SPLARE Top-K = (20, 200)
55.9
59.9
53.8
SPLARE Top-K- (10, 100)
50.1
56.0
46.5
SPLARE-2B
55.9
59.1
41.6
effectiveness (Top-K at inference time). We also observe an average gain +1.5 points for the non- pooled version, albeit at the cost of higher retrieval complexity. On the other hand, extremely sparse models (Top-K = (10, 100)) still offer competitive performance. Note that in practical retrieval sce- marios, dense embeddings often require dimensionality redduction techniques Kusupati al. (2022) and/or approximate nearest-neighbor search algorithms Johnson et al. (2019) 2 -whose performance degradation is rarely reported standard benchmarks. In contrast, sparse retrieval methods tively support efficient exact search without incurring such compromises. Finally, we also report results for SPLARE model trained at layer (SPLARE-2B). Although its performance some- what lower than that of the full SPLARE model (7B parameters), it remains strong- crang-patticularlyy on the XTREME-UP dataset. Importantly, this model is substantially more efficient and therefore offers different, and often attractive, point on the ceattiiveesssffffcfinnee trade-off curve.
6.3 INTERPRETABILITY MECHANISTIC INTERPRETATION OF SPLARE
Finally, we provide interpretability insights for SPLARE. We leverage Neuronpedia (Ln 2023
obtain explanations for individual SAE features which, as a reminder, remain frozen during fine-tuning- and list the top features contributing to document's relevance with respect to a given query. For SPLADE, by contrast, we report the tokens with the highest relevance contributions. Figure T Filluustrates a cross-lingual example from XTREME-UP from Tamil to English. The features activated by SPLARE align well with meaningful concepts present in both the query and document. They correspond to coherent, language-agnostic concepts which combine into comprehensive de- scription the data point. In contrast, SPLADE exhibits a higher degree of redundancy (e.g., separate activations for "Indian" and "indian"") and predominantly relies on Latin-script tokens effectively defaulting to English subword representations which provide less informative signals in this setting. Further examples given in Appendix G

---

Published as a conference paper at ICLR 2026
Figure 4: Retrieval example from XTREME-UP: Tamil English lish
Query: anddiaawjaa quLdud arnorfluaa viggav @ijdujaa Apšpeni
Translation: On average, how many Indians died under British rule? Positive document:
Indian Army during World War Il: The British Indian Army fought in Ethiopia against the Italian Army, in Egypt, Libya, Tunisia and Algeria against both the Italian and German Army, and, after the Italian surrender, against the German Army in Italy. [...]
SPLARE top Features (doc rank 4)
SPIADE
tokena (doc rank 23)
Exp Lon (Eron Neusonpedla) Lin 2025
elements related to historical or cultural contexts
10.5
.Indian
12.6
mentions of India and its relation to various contexts
8.7
.Indians
11.2
descriations that contrast Traditional experiences with unique local accommodation
7.5
_>asualtiea
9.0
powers, specifically Britain references milttary casueltes and losses quantitative statistics and casuaitis related to wars and conflicts information related to economic data connectivily issues India protests references historical events and polltical movements references corporate structure and business dynamics
.India
6.5
.indian
7.8
5,9
_Britihh
7.5
5,2
6.8
rights movements
ifatalitics
-india
4.7
4.4
_Ra
4.5
RELATED WORKS
LLMs and Retrieval. Dense embedding models derived from LLMs have demonstrated substan-
tial gains over traditional BERT-style encoders (Lee et al. 2025b Zhang et al. (022).. Recent approaches such as LLM2Vec [BehnamGhader al. (2024) GriLL Muennighoon
al. (2024)
highlight how LLMs can be effectively adapted into powerful text encoders by incorporating bi- directional attention. Beyond providing stronger backbone architectures. LLMs have also signif- icantly advanced retrieval model training, enabling the generation of high-quality synthetic data and improved filtering of training samples (Wang et al. 2024a Lee et al., 2025a b) /Zhang et al.) 2025 Dai et al. 2023). Nonetheless, despite the impressive progress of dense embeddings, trolled evaluations have shown that they can still be outperformed by alternative architectures such as multi-vector models sparse retrievers (Zeng et
2025 Faysse et al. 2025 Chen et al. 2024a).
Sparse Autoencoders and Retrieval. Sparse autoencoders have primarily been employed in In- formation Retrieval (IR) to approximate dense representations for efficient neighot search. Given a dense embedding model, these approaches learn to map query and document vectors into sparse latent representations that preserve the structure of the original embedding space (Lassance et al. 2021: Borges et al. 2023 Park et al. 20255] Kang et al. 2025 Wen et al. (025). SAES have also been used to interpret dense embeddings
both IR OO'Neill et al. (2024) and Recommender Systems (Kasalický et al.) 2025: Klenitskiy et al. (0255) Most closely related to our work is Park cet al. (2025), which shows that SAE-derived features can serve as effective indexing units. However, all prior studies train SAES on top of an already-truined dense relriever. In contrast, our approach leverages pre-trained SAEs on the base LLM and fine-tunes an LSR model directly in SPLADE- like fashion, allowing for tighter integration of relevance and sparsity.
CONCLUSION
In this work, we investigated two complementary research directions: Sparse autoencoders and Learned Sparse Retrieval models. We demonstrated that SAEs provide natural foundation for LSR by yielding semantically rich and multilingual latent features that overcome the vocabulary de- pendence of traditional LSR approaches. Our experiments show that SAE-based LSR models con- sistently outperform vocabulary-baae counterparts, particularly in multilingual and out-of-domain scenarios. Finally, we introduced SPLARE, a competitive 7B 7B-parameter multilingual model capa- ble of producing generalizable sparse latent embeddings, thereby paving the way for more robust, versatile, and cross-lingual retrieval across diverse domains and modalities.
REFERENCES
Voyage AL. Voyage-3 large.
voyage-33 3- 3- arree/ January 2025. Accessed: 2025-09-24.
10

---

Published as a conference paper at ICLR 2026
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. Ms marco: A human generated machine reading comprehension dataset, 2018. URL htt..htps://arxxi.oorg/
61 09268
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapa- dos, and Siva Reddy. L.M22ec: Large language models are secretly powerful text encoders. In First Conference Language Modeling, 2024. WRLLhttps://opeereeeee
ner/forum?
id IR1PR7vEBE.
Luís Borges, Bruno Martins, and Jamie Callan. Kale: Using k-sparse projector for lexical pansion. In Proceedings of the 2023 ACM SIGIR International Conference on Theory of In- formation Retrieval, ICTIR '23, pp. 13-22, New York, NY, USA, 2023. Association for Com-
puting Machinery. ISBN 9798400700736.
doi: 10.1145/3377833.36651111
URL https:
//doi.org/10 1145/35783333 3ww.:8337336665133
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby. Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https//transffrmeer- http:////wmmooommmmmatttttrree//
Sebastian Bruch, Franco Maria Nardini, Cosimo Rulli, and Rossano Venturini. Efficient inverted indexes for approximate retrieval over learned sparse representations. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 152 162, 2024a.
Sebastian Bruch, Franco Maria Nardini, Cosimo Rulli, and Rossano Venturini. Pairing clustered inverted indexes with k-nn graphs for fast approximate retrieval learned sparse represen- tations. In Proceedings of the 33rd International ACM Conference on Information and Knowl- edge Management (CIKM), pp. 3642-3646. ACM, 2024b. doi: 10.1145/322767333779977 URL http:////ooo...11.111111.
Sebastian Bruch, Franco Maria Nardini, Cosimo Rulli, and Rossano Venturini. Efficient inverted indexes for approximate retrieval over learned sparse representations. In Proceedings of the 47th International ACM SIGIR Conference Research and Development in Information Retrieval (SIGIR), pp. 152-162. ACM, 2024c. doi: 10.1145/666677233577669 URL https:://di.. Dxc/10.1145/3626777 9
Sebastian Bruch, Franco Maria Nardini, Cosimo Rulli, Rossano Venturini, and Leonardo Venuta. Investigating the scalability of approximate sparse retrieval algorithms massive datasets. Ad- vances in Information Retrieval, pp. 437-445, Springer Nature Switzerland, 2025. doi: 10.1007/
978-3-031-88714-7.22
URL /ttp::///oo...rr//0000000///393. 0t1.://..110007777--3---11--1--1
/ooi
43
Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi-functionality. multi-granularity text embeddings through seff-knowledge dis- tillation, 2024a.
Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. M3-embedding: Multi-linguality, mulii-functionaaity, multi-granularity text embeddings through self-Knowledge distillation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the As- sociation for Computational Linguistics: ACL 2024, pp. 23355, Bangkok, Thailand, August 2024b. Association for Computational Li Linguisties. doi: 10. 11111186633122244 http:////222.....nnns.ooo. ac1. URL fittps://aat antho h wenttholoyyy log..oy....o wgw..og...............o......ooa
findings- 1377
Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning,
pp. 1697-1607. PmL.R, 2020.
11

---

Published as a conference paper at ICLR 2026
Cohere.
Introducing embed http:///////oaaaccca///////raaa/
November 2023. Accessed: 2025-09-24.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. Overview of the trec 2020 deep learning track, 2021.
wttp://wwww..gg////////.///// 07662
Hoagy Cunningham and Tom Conerly. Comparing topk and gated saes standard saes. Trans- former Circuits Thread, 2024.
/ttp://c......oaao..oo.ooooooo// pu2/2024
jure-update/ index html
Zhuyun Dai, Vincent Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu,
Keith Hall, and Ming-Wa wei Chang. Promptagator: Few-shot dense retrieval from 8 examples. In The Eleventh International Conference on Learming Representations, 2023. URL https:
//openzeview.o
Gabriel de Souza P. Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge. Ny-retriever: Improving text embedding models with effective hard-negative mining, 2025.
Hervé Déjean, Stephane Clinchant, Carlos Lassance, Simon Lupart, and Thibault Formal. Bench- marking middle-trained language models for neural search. In Proceedings of the 46th Interna- tional ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '23, pp. 1848-1852, New York, NY, USA, 2023, Association for Computing Machinery. ISBN 9781450394086. doi: 10.1145/33396183551956
URL https: httgs:://dci.ogg/// 1145/
3539618 0a.658880 3591956
Boyi Deng, Yu Wan, Baosong Yang, Yidan Zhang, and Fuli Feng. Unveiling language-specific features in large language models via sparse autoencoders. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1563-46090 Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979- 8-8976- 251-0. doi: 10. 10.18653/11/2225.aaa acl woo.nn... 2297
URL
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume / (Long and Short Papers), pp. 4171-4186 Minneapolis, Minnesota, June 2019. Association for Com- putational Linguistics. doi: 10.18653/1/N119-4222 URL attps::/aclanttee N19 9-123/
org/
Meet Doshi, Vishwajeet Kumar, Rudra Murthy, Vignesh P, and Jaydeep Sen. Mistral-splade: Lims for better learned sparse retrieval, 2024. http:ttsss/axxiv.oogpp/oo
Hervé Déjean, Stéphane Clinchant, and Thibault Formal. A thorough comparison crosseecodersss and Iims for reranking splade, 2024.
Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiński, Genta Indra Winata, Saba Sturua, Saiteja Ut- pala, Mathieu Ciancone, Marion Schaeffer, Diganta Misra, Shreeya Dhakal, Jonathan Rystrøm, Roman Solomatin, Omer Veysel Çağatan, Akash Kundu, Martin Bernstorff, Shitao Xiao, Ak- shita Sukhlecha, Bhavish Pahwa, Rafal Pošwiata. Kranthi Kiran GV, Shawon Ashraf, Daniel Auras, Björn Plüster, Jan Philipp Harries, Loic Magne, Isabelle Mohr, Dawei Zhu, Hippolyte Gisserot-Boukhie Tom Aarsen, Jan Kostkan, Konrad Wojtasik, Taemin Lee, Marek Suppa, Crystina Zhang, Roberta Rocca, Mohammed Hamdy, Andrianos Michail, John Yang. Manuel Faysse, Aleksei Vatolin, Nandan Thakur, Manan Dey, Dipam Vasani, Pranjal Chitale. Simone Tedeschi, Nguyen Tai, Artem Snegirev, Mariya Hendriksen, Michael Günther, Mengzhou Xia, Weijia Shi, Xing Han LU, Jordan Clive, Gayatri K, Maksimova Anna, Silvan Wehrli, Maria Tikhonova, Henil Shalin Panchal, Aleksandr Abramov, Malte Ostendorff, Zheng Liu, Simon Clematide, Lester James Validad Miranda, Alena Fenogenova, Guangyu Song, Ruqiya Bin Safi, Wen-Ding Li, Alessia Borghini, Federico Cassano, Lasse Hansen, Sara Hooker, Chenghao Xiao,
12

---

Published as a conference paper at ICLR 2026
Vaibhav Adlakha, Orion Weller, Siva Reddy, and Niklas Muennighoff. MMTEB: Massive mul- tilingual text embedding benchmark. In The Thirteenth International Conference on Learning Representations, 2025. /ecttps//ppeneeveeee.eee//e
Aaron Grattafiori et al. The llama 3 herd of models, 2024. URL htttp:://arii..rr//// 2407.21 21783
Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, CELINE HUDELOT, and Pierre Colombo. Colpali: Efficient document retrieval with vision language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openeeview. www. http:///ooooooodddogoggo.oo...o
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. Splade: Sparse lexical and ex- pansion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, pp. 2288-2292, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781 9781450380379, doi: 10.1145/340833.3463099
httttt::///// wrg./0..//oo. 11tp///001114434444333 3466998
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. From distillation to hard negative sampling: Making sparse neural ir models more effective. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information
Retrieval, SIGIR '22, pp. 2353-2359, New York, NY, USA, 2022a. Association for Computing Machinery. ISBN 4781450387333, org/ 100.1114/3377455 353187
doi: 10. 10.1145/347445533311555
URL littps://doi.
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. Match your words! a study of lexical matching in neural information retrieval. In Advances in Information Retrieval: 44th
European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10-14, 2022, Proceedings, Part 11, pp. 120-127, Berlin, Heidelberg. 2022b. Springer- Verlag. ISBN 978-3- 030-99738-0. doi: 10. 100.1007777833300 0-0079977----00--779-----7 9788 3 030 99739-9997799 14
URL https://8oi.org/1100
Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In The Thirteenth Inter- national Conference on Learning Representations, 2025. URL d=tcs7t frrummid=tcs.... 97NKD
forum?
Michael Günther, Saba Sturua, Mohammad Kalim Akram, Isabelle Mohr, Andrei Ungureamu, Bo Wang, Sedigheh Eslami, Scott Martens, Maximilian Werk, Nan Wang, and Han Xiao. jina-embeddings-vke Universal embeddings for multimodal multilingual retrieval, 2025. URL https::/aaxxls
orgg wogg/abs//ee 2555 18902
Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. Llama scope: Extracting millions of features from Ilama-3. 1.188bb with sparse autoencoders, 2024. URLhttos. //arxiv.orq/antt www./orr//bb.///o.eeo..oo 20522
Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. Im- proving efficient neural ranking models with cross-architecture knowledge distillation. arXiv preprint arXiv:2010.02666, 2020.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL https: openreview net forum?
id=nZevKeeFFEE
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelftth International Conference on Learning Representations, 2024, URLhttps: http:::///////eeee/ee// net forum?
id=F7 6bwwS1.e
13

---

Published as a conference paper at ICLR 2026
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learn- ing. Transactions Machine Learning Research, 2022. ISSN 2835-8856.
h
//openreview. net./ formmm j pxi pxi760
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. JEEE Transactions Big Data, 7(3):535- 2019.
Hao Kang, Tevin Wang, and Chenyan Xiong. Interpret and control dense retrieval with sparse latent features. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume 2: Short Papers). pp. 709, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 0777---8997660 190-29922. doi: 10.15//3/v1/ 2025.naacl wwwaa--aaat.t..a5 URL htLLhttpss//accaaooloooog orq/ 2025. www.aaaaccclaaaa short 58/
Subhash Kantamneni, Joshua Engels, Senthooran Rajamanoharan, Max Tegmark, and Neel Nanda. Are sparse autoencoders useful? case study in sparse probing. In Forty-second International Conference on Machine Learning, 2025. URL hettps://openrevieee net forum? dd==
ENfzIBYkgC
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 6769 16681, Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10. 1t1166///112220mmmmmplooo. URL ptttps://aclaathholoo.oooo org/2020 wttp:/222.emmlppppp. wmnlp-mnnmm..eee mai-main.... 550/
Petr Kasalický, Martin Spišák, Vojtëch Vančura, Daniel Bohunëk, Rodrigo Alves, and Pavel Kordík. The future is sparse: Embedding compression for scalable retrieval in recommender systems. In Proceedings of the Nineteenth ACM Conference on Recommender Systems, RecSys '25, pp. 1099-1103, New York, NY, USA. 2025, Association for Computing Machinery. ISBN 9798400713644. doi: 041145/37053288.7488177 URL https:://cc..nn http:///cci.or//////// org/10........0 11tg//1..111/////
3705328.374817
Junseong Kim, Seolhwa Lee, Jihoon Kwon, Sangmo Gu, Yejin Kim, Minkyung Cho, Jy yong Sohn, and Chanyeol Choi. htte--embedd-issstllllleeaaee text retrieval with improved gpt data through task-specific control and quality refinement. Linq AI Research Blog, 2024. URL https:// embed mistral/
embeeee
Anton Klenitskiy, Konstantin Polev, Daria Denisova, Alexey Vasilev, Dmitry Simakov, and Gleb Gusev. Sparse autoencoders for sequential recommendation models: Interpretation and flexible control, 2025.
12202
Weize Kong, Jeffrey M. Dudek, Cheng Li, Mingyang Zhang, and Michael Bendersky. Sparscembed: Learning sparse lexical representations with contextual embeddings for retrieval. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 233, pp. 2399-2403 New York, NY, USA, 2023. Association for Computing Machinery, ISBN 4781450394086 3592065
doi: 10.1145/5339118.5592000
URL https:://oii.
OIq/10.1145/3552
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation learning. In Advances in Neural Information Processing Systems, December 2022.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wo -ei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019, doi: 10.1162/ URL
org. Q19 1026/
Carlos Lassance Extending english ir methods to multi- multi-lingual ir, 2023. URL https:://aat
arg/abs/200 /202222222... 14723
14

---

Published as a conference paper at ICLR 2026
Carlos Lassance, Thibault Formal, and Stéphane Clinchant. Composite code sparse autoencoders for first stage retrieval. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, pp. 2136-2140, New York, NY. USA, 2021. Association for Computing Machinery. ISBN 47814533...7 doi: 10.1145/3404855. 3463066. URL httpps//:/// .rg. /10 1011111444443355 3463066
Carlos Lassance, Simon Lupart, Hervé Déjean, Stéphane Clinchant, and Nicola Tonellotto. A static pruning study on sparse neural retrievers. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '23, pp. 1771-1775, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450394086. doi: (5911441]
URL.tttps://dol org/10. 11111
Carlos Lassance, Hervé Déjean, Thibault Formal, and Stéphane Clinchant. Splade-v3: New base- lines for splade. arXiv preprint 200.:22403.00689222222
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping. NV-embed: Improved techniques for training LLMs as generalist embedding models. In The Thirteenth International Conference on Learning Representations, 2025a. URL https:/ ftttp:://cpenreet /eppereeevveeeeeeeeeeeeeeeeeeeee ew net/ wwt.:nttttrtaa//e www.ummiddlggee 1sy..SSDe
Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer, Madhuri Shanbhogue, Iftekhar Naim, Gus- tavo Hernández Ábrego, Zhe Li, Kaifeng Chen, Henrique Schechter Vera, Xiaoqi Ren, Shanfeng Zhang, Daniel Salz, Michael Boratko, Jay Han, Blair Chen, Shuo Huang, Vikram Rao, Paul Suganthan, Feng Han, Andreas Doumanoglou, Nithi Gupta, Fedor Moiseev, Cathy Yip, Aashi Jain, Simon Baumgartner, Shahrokh Shahi, Frank Palma Gomez, Sandeep Mariserla, Min Choi, Parashar Shah, Sonam Goenka, Ke Chen, Ye Xia, Koert Chen, Sai Meher Karthik Duddu, Yichang Chen, Trevor Walker, Wenlei Zhou, Rakesh Ghiya, Zach Gleicher, Karan Gill, Zhe Dong. Mojtaba Seyedhosseini, Yunhsuan Sung, Raphael Hoffmann, and Tom Duerig. Gemini embedding: Gener- alizable embeddings from gemini, 2025b. URL httph:ttt//eeet http:///xxvvv.rr..o.
2200 07891
Yibin Lei, Tao Shen, Yu Cao, and Andrew Yates. Enhancing lexicon-based text embeddings with large language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Moham- mad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Com- Vienna, Austria, July 2025. doi: 10. 10.18653/112025.. log.. 930/
putational Linguistics (Volume 1: Long Papers), pp. Association for Computational Linguistics. ISBN 979 979-8-89176-251-00
acl-long.930 URL ftttss:t:://// //act antho logy. wwg..oo...//a.oo..o..
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih. Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmmette generation for knowledge-intensive nip tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 978171 11132256..
Chaofan Li, Zheng Liu, Shitao Xiao, and Yingxia Shao. Making large language models a better foundation for dense retrieval, 2023a.
Chaofan Li, Minghao Qin, Shitao Xiao, Jianlyu Chen, Kun Luo, Defu Lian, Yingxia Shao, and Zheng Liu. Making text embedders few-shot learners. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL httttt:t////tt httt:ss://opereview.eeee. net mettfforrmiii wf LfLuiD000
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning, 2023b. URL httpttttp///riiti
orq/ hbs//bb/225 2308. 03281
Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, Janos Kramar, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen (eds.), Proceedings of the 7th Black- boxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 278 308-000,,000 Miami, Florida, US, November 2024. Association for Computational Linguistics. doi: 10. 10.18653/v1/0224 blackboxnlp- 1. URL https::tp///........ / //tp:::::///aaaee h antho logy. ogg.oorg/002.... acw.bom
15

---

Published as a conference paper at ICLR 2026
Johnny Lin. Neuronpedia: Interactive reference and tooling for analyzing neural networks, 2023.
URL https:://ww.neuronnngto/ www.nneuroopedia..cm org] Software available from neuronpedia.org
Sheng-Chiee Lin, Jheng-Hong Yang, and Jimmy Lin. Distilling dense representations for ranking using tightly-coupled teachers. arXiv preprint arXiv:2010. 1386, 2020.
Simon Lupart, Thibault Formal, and Stéphane Clinchant. Ms-shift: An analysis of ms marco
distribution shifts neural retrieval. In Advances in Information Retrieval: 45th Euro-
pean Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2-6, 2023, Proceedings, Part pp. 636-652, Berlin, Heidelberg, 2023. Springer-Verlag. ISBN 978-3- 031-28243-0. doi: 00000779788--01-28244---44 978-3-031-28244-72 40]
URL https: http://oo.....11...00 1007/
Guangyuan Ma, Yongliang Ma, Xuanrui Gou, Zhenpeng Su, Ming Zhou, and Songlin Hu. Lightre- triever: A llm-based hybrid retrieval architecture with 1000x faster query inference, 2025. URL
https://arxiv. org/ abs 2505. 12260
Joel Mackenzie, Andrew Trotman, and Jimmy Lin. Efficient document-at-a-time and score-at-a-time query evaluation for learned sparse representations. ACM Trans. Inf. Syst. 41(4), March 2023. ISSN 1046-8188. doi: 10. (0.1145/357692. URL
Alireza Makhzani and Brendan Frey. K-sparse autoencoders. arXiv preprint arXiv: 2013.
Antonio Mallia, Omar Khattab, Torsten Suel, and Nicola Tonellotto. Learning passage impacts for inverted indexes. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1723-1727, 2021.
Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfr- embedding embedding-mmittraleent mistraleenaance text retrieval with transfer learning. Salesforce Al Research Blog, /tt.s/feddeeegggg//////o
2024. URL wwwp//wwwww./ wwweww.alls............. www.aslsssreee...oooooeee wwt.booo///ooooooooooooooooooooo http///oooo/sffffeedtieeee
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: Massive text em- bedding benchmark. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2014- 2037, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 18.13/53/ v1/2023. 1.0003.eaal-main. 148. Utt https///aaaaattoooooooo /tt.a/////ittta///aaaaaaoooooooo
org/ 2023 eacl main. 148/
Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang. Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative Representational Instruction Tuning, February 2024.
09906 arXiv:2402.0988 09906 [cs].
Suraj Nair, Eugene Yang, Dawn J. Lawrie, James Mayfield, and Douglas W. Oard. Learning a sparse representation model for neural clir. In DESIRES, pp. 53-64, 2022. URL http::ttt::c/cttt -weerwww...
htg/Vol-3440/papee pdf
Suraj Nair, Eugene Yang, Dawn Lawrie, James Mayfield, and Douglas W. Oard. Blade: Combin- ing vocabulary pruning and intermediate pretraining for scaleable neural clir. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Informa- tion Retrieval, SIGIR '23, pp. 1219-1229, New York, NY, USA, 2023. Association for Com- puting Machinery. ISBN 9781450394086.
doi: 10. 10.1145/35396188355111
URL https:
//doi /rg/1orr/1.. 1145/ / 3539618 3591644
Thong Nguyen, Sean MacAvaney, and Andrew Yates. A unified framework forlearned sparse re- trieval. In Advances in Information Retrieval: 45th European Conference Information Re- trieval, ECIR 2023, Dublin, Ireland, April 2-6, 2023, Proceedings, Part III. pp. 101-116, Berlin, Heidelberg, 2023. Springer- ISBN 978-3-03---24244000 doi: 10. 10.1007/978-3-001--824111 7. URL httttttt:::/ /ttetts:://oo..oo 1ttp://///.111000000-00000001111
Thong Nguyen, Mariya Hendriksen, Andrew Yates, and Maarten de Rijke. Multimodal learned sparse retrieval with probabilistic expansion control. In Advances in Information Retrieval: 46th
European Conference on Information Retrieval, ECIR 2024, Glasgow, UK, March 24-28, 2024, Proceedings, Part II, pp. 031-56059-0. doi: 10. --0007797788-01-5-606----- 29. URL actps: // wwwpoo./...0.........ooo 1007/ 978-3-031-56060-2
Berlin, Heidelberg, 2024. Springer- Verlag. ISBN 978-3-
16

---

Published as a conference paper at ICLR 2026
Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with bert, 2020. URL https://
axxiv.org/ao abtp/ wttpv/..rr/bb//11...0oo. 04085
Charles O'Neill, Christine Ye, Kartheik G. Iyer, and John F Wu. Towards interpretable scientific foundation models: Sparse autoencoders for disentangling dense embeddings of scientific con- cepts. In Neurips 2024 Workshop Foundation Models for Science: Progress, Opportunities, and Challenges, 2024. /tRLhhttp:///ppeee
net wwwteoorweeieeeeeeeeeeeeee www.ooomddmmmmaaeeeeetteoo
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprini urXiv:1807.03348. 2018.
OpenAl. text-embeddiing--llarr and new embedding models, index/ http:///wwwwaeoooooooeooeeoeaaaa
Settps://openaz com/
January 2024. Accessed:
2025-09-25.
Biswajit Paria, Chih-Kuan Yeh, Ian EH Yen, Ning Xu, Pradeep Ravikumar, and Barnabás Póczos. Minimizing flops to learn efficient sparse representations. arXiv preprint arXiv:2004. 05665, 2020.
Seongwan Park, Taeklim Kim, and Youngjoong Ko. Decoding dense embeddings: Sparse autoen- coders for interpreting and discretizing dense retrieval, 2025. URL https::/arxxiv org/
abb/2506. 00041
Jingfen Qiao, Thong Nguyen, Evangelos Kanoulas, and Andrew Yates. Leveraging decoder archi- tectures for learned sparse retrieval, 2025. URL
Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Commy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders, 2024. httpst::::/ /tt.pp:://xxi..oor/aa//////// 14435
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguisties, 11 2019. URL https://arxix:
org/ abs//bb//// 1908. 10084
Sebastian Ruder, Jonathan H. Clark, Alexander Gutkin, Mihir Kale, Min Ma, Massimo Nicosia, Shruti Rijhwani, Parker Riley, Jean-Michel A Sarr, Xinyi Wang, John Wieting, Nitish Gupta, Anna Katanova, Christo Kirov, Dana L. Dickinson, Brian Roark, Bidisha Samanta, Connie Tao, David I. Adelani, Vera Axelrod, Isaac Caswell, Colin Cherry, Dan Garrette, Reeve Ingle, Melvin Johnson, Dmitry Panteleev, and Partha Talukdar. XTREME-UP: A user-centric scarce- data benchmark for under-represented languages. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1856-1885 1884, Singapore, December 2023, Association for Computational Linguistics. doi: 16633//2221/223. htwpl/ooyyrogg22203 wttg//23333000 fin.iiiinninie -nn
Andings-emmlp.122 125.. URL
125/
Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Nikul Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations language models. In Forty- second International Conference on Machine Learning, 2025. URL http::tt////peerevieew
WGXb WGXXTUUVTX
Lewis Smith, Senthooran Rajamanoharan, Arthur Conmy, Callum McDougall, Tom Lieberum, János Kramár, Rohin Shah, and Neel Nanda. Negative results for saes on down- stream tasks and deprioritising research (gdm mech interp team progress up- date 2). https::/www.. www.:ttttiiiimnnnnnoomommmo
sae
-draft 2025.
Livio Soares, Daniel Gillick, Jeremy Cole, and Tom Kwiatkowski. NAIL: Lexical retrieval in- dices with efficient non-autoregressive decoders. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2574-2583 Singapore, December 2023. Association for Computational Linguis tics. doi: 10.18653/v 10.18653/v120233 http://w11122222m2mmmmmme www.ee-man.nnn..nmeem m5mn....
URL ht U http:///aaaaattttaaoaee wwt.h/oooooyy.../22.....3
eemn 1 ma i 1567
17

---

Published as a conference paper at ICLR 2026
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Trans- former Circuits Thread, 2024, URL
wewaaasoooee--rrrisssssseeee- pub//222
scalingg monosemanticityy Seriatyyinndeee index. html
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL
Nicola Tonellotto, Craig Macdonald, Iadh Ounis, et al. Efficient query processing for scalable web search. Foundations and Trends@ in Information Retrieval, 12(4-5):319-500, 2018.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Im- proving text embeddings with large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 11897-11916, Bangkok, Thailand, August URL
2024a. Association for Computational Linguistics. doi: 10. 111166///12222..clloooggg
org/ 202pg//2224cco 201. 6427|
Liang Wang, Nan Yang, Xiaolong Huang. Linjun Yang, Rangan Majumder, and Furu Wei. Multi- lingual e5 text embeddings: technical report. arXiv preprint 2402:::40000067... 2024b.
Shuai Wang, Shengyao Zhuang, Bevan Koopman, and Guido Zuccon. 2d matryoshka train- ing for information retrieval. In Proceedings of the 48th International ACM SIGIR Confer- ence Research and Development in Information Retrieval, SIGIR '25, pp. 3125-3134 New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400715921. doi: 10.1145/3726302.3730032 URL https:ttp//............t http:::/dooi..ggg..
1 3730333
Tiansheng Wen, Yifei Wang, Zequn Zeng, Zhong Peng, Yudi Su, Xinyang Liu, Bo Chen, Hongwei Liu, Stefanie Jegelka, and Chenyu You. Beyond matryoshka: Revisiting sparse coding for adap- tive representation. In Forty- International Conference Machine Learning, 2025. URL httpp::/opennrvview net/ fttt://foommmm=111000000 9BB2F
Mitchell Wortsman, Gabriel Ilharco. Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves ac- curacy without increasing inference time, 2022. URL httpss:/aaxxi..rrrt wwwpxxwww.r//bb//222..3 05482]
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00800 2020.
Mengyao Xu, Gabriel Moreira, Ronay Ak, Radek Osmulski, Yauhen Babakhin, Zhiding Yu, Benedikt Schifferer. and Even Oldridge. Llama nemoretriever colembed: perforrfn text- image retrieval model, 2025a. URL https: http://www.....o.////////////// 05513
Zhichao Xu, Aosong Feng, Yijun Tian, Haibo Ding, and Lin Lee Chcong. Csplade: Learned sparse retrieval with causal language models, 2025b. URL
10816
Junhan Yang, Jiahe Wan, Yichen Yao, Wei Chu, Yinghui Xu, and Yuan Qi. inf-rerriever-vvl (revision 5f469d7). 2025. httth:tt////// http::s:////ngggggaaee.ooo http:///ww..lllnnfeeteeeeeeeeeee
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpoiQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369- 2380, Brussels, Belgium, Novebbrr 2018. Association for Computational Linguistics. doi: 10.18653/v/DD8--122
htRl.tttts: pectts://ert //ccl antho ogy. 00y.orr//D1B-11255/
18

---

Published as a conference paper at ICLR 2026
Hansi Zeng, Julian Killingback, and Hamed Zamani. Scaling sparse and dense retrieval in decoder-
only llms. In Proceedings of the 48th International ACM SIGIR Conference on Research and
Development in Information Retrieval, SIGIR '25, pp. 2679-2684, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 19914490711991 doi: 10. 10.1145//72630023333255.
URL htt.hhtps://doi.. wtmpioorgg11.00 2ttpr//00..11133222222222...3. 3730222
Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. Mr. TyDi: A multi-lingual bench- mark for dense retrieval. In Duygu Ataman, Alexandra Birch, Alexis Conneau, Orhan Fi- rat, Sebastian Ruder, and Gozde Gul Sahin (eds.), Proceedings of the 1st Workshop on Mul- tilingual Representation Learning, pp. 127-137, Punta Cana, Dominican Republic, Novem- ber 2021. Association for Computational Linguistics. doi: 10.18653/ https: httpes//Aelarthhoooo..or//////// m1.11.1m11111111
URL
Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso- Hernnl,, Xi- aoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. MIRACL: multilingual retrieval dataset covering 18 diverse languages. Transactions of the Association for Computational Lin- guistics, 11:1114-111 2023, doi: 10.1162/tacl URL httttss//aclaatholl/l org/2023 tacl 637
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang. Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models, 2025. URL https:
wlaxxii.rr/abb///222. 22500
Shengyao Zhuang, Shuai Wang, Fabio Zheng, Bevan Koopman, and Guido Zuccon. Starbucks-v2: Improved training for 2d matryoshka embeddings,
2410 13230
Justin Zobel and Alistair Moffat. Inverted files for text search engines. ACM Comput. Surv,, 38 (2)66 es, July 2006. ISSN 0360-0300. doi: 10.1145/1 132956.1 1332599. URL https://doi.
0rc/10.1145/112298 1132955
19

---

Published as a conference paper at ICLR 2026
A EXPERIMENTAL SETTING
We detail below the training sets used for the English and Multilingual settings.
English Setting. For our ablation study, we restrict training to the MS MARCO dataset, given the computational cost associated with training 7B-parameter models. Our experimental setup closely follows that of SPLADE-v3 (Lassance et al. 2024). For each training query, we mine hard negatives using a SPLADE model and derive distillation targets from reranking scores produced by open- source DeBERTa-v3 reranker (Déjean et al. 2024). This controlled setting is designed to enable direct and fair comparison between SPLARE and its vocabulary- counterpart, SPLADE- Llama.
Multilingual Setting, In this more compute-intensive setting, we use the same training set em- ployed for the bge-multilnguallgeemmet
model (Li et al., 2025 This corpus includes several English-centric public datasets (e.g.. MS MARCO Bajaj et al. (2018), NQ/Kwiatkosski et al. (2019), and HotPotQA Yang al.) (2018)). a large collection of Chinese retrieval datasets, and two multi- lingual benchmarks: MIRACI Zhang et al. (2023) and Mr.TyDi (Zhang et al. (021). Since we rely on distillation for training. we only keep samples from this dataset which were annotated using the BGE multilingual reranker (Chen et al. 2024b, Li et al. 2023a After filtering, the final training set comprises approximately 1.3M queries with hard negatives. Notably, some of these datasets correspond to training splits of several MTEB benchmark tasks. While this may constrain the strict evaluation of generalization, this practice has become standard in prior work on seneral-purposs embedding models (Lee al. 2025a Wang et al. 2024a BehnamGhader al. (0244).
B PIEER-PARAMETERS
Table 4 gives the hyper-parameters used to train and evaluate SPLARE models and other baselines. Note that the temperature parameter is critical and needs to be adapted to each SAE suite. For instance, the optimal
different between Llama Scope or Gemma Scope. This depends on the
scale of the logits and the initial sparsity of the SAE. For ill-suited actually diverge- for instance, collapse of the lg. To determine the optimal temperature, we ran grid search the values {1. 10, 20, 40, 50, 80, 100), and used NanoBEIR's nDCG@ @ 10 as evaluation criterion for all model
it can happen that models
SAE Choice. Gemma Scope contains multiple SAES for the same layer and width, but with dif- ferent l.- In practice, we observed that the initial SAE's fy had critical effect on final perfor- mance most likely because we tue-tune the backbone LLM. We use SAEs with Ey closest to 100 throughout the paper. Additionally, Llama and Gemma Scope contain residual SAEs as well as MLP and attention stream SAEs-we we only use residual SAES in this study.
C ENGLISH-ONLY SPLARE FULL RESULTS
We evaluate models from Section (trained on English data only) on several benchmarks, and provide results Table Table 5 additionally reports evaluation results comparing SPLARE with SPLADE-Llama. We report MRR @ 10 MS MARCOBaja et (2018) and nDCG@ 10 on TREC DL datasets Craswell et al. (221)) as well as on all BEIR datasets (Thakur et al. 2021).
D FULL RESULTS
Tables 6 10
provide the full results of several MTEB datasets: English, Multilingual, and various
domains languages.
Baa MeAAl//qen = reranker
httepp:///oogginggaacccoo/oo///o http:///ao.a.at..ooooooooooooaaa n hanebeesrr
20

---

Published as a conference paper at ICLR 2026
Table 4: Hyper-parameters
Component
Value
LoRA rank
64
Max training sequence length (english models)
128
Max training sequence length (multilingual models) Epochs
512
1
Batch size (w/ gradient accumulation)
128
Learning rate
10-5
Warmup ratio
0.01
Nb negatives per query
8 0.0001
Ad
Xg
0.0001
SPLARE Llama Scope
80
SPLARE Gemma Scope
50
SPLADE-Llama
10
Evaluation max context size
1024
Adam As
0.9, 0.999
Dataset
SPLARE
OPLADE-Llame
Arguana
16.0
16.2
Climatc-FEVER
18.3
18.0
DBPedia
44.3
44.8
FEVER
76.0
75.8
FiQA-2018
42.4
42.3
HotpotQA
66.8
67.6
NFCorpus
37.3
36.4
NQ
61.6
61.2
Quora
87.3
87.9
SCIDOCS
17.5
17.3
SciFact
72.5
72.9
TREC-COVID
84.7
82.4
Touché-2020
27.2
26.9
Average
50.2
50.0
MS MARCO (MRR @ 10)
40.8
40.0
TREC DL '19
77.4
76.3
TREC DL '20
77.3
75.9
Table 5: Full results (nDCG@ @ 10 unless specified) on BEIR, MS MARCO and TREC DL for English-based SPLARE and SPLADE-Llama models. Evaluation done with Top-K (40, 400).
Table 11 compares the SPLARE results on the MIRACL dataset with top multilingual dense re- trievers baseline results taken from Chen et al. 2024b). On this benchmark, SPLARE obtains an average score of 71.7, +0.2 points above M3-embeddings (hybrid: dense + sparse) Chen et al. (024a). Notably, SPLARE state-of-the-an in English, Finnish, Hindi, Japanese, Russian, Swahili, Genman and Yoruba, once again indicating its ability to generalize to diverse languages. Note in par- ticular that German and Yoruba are the "secret" languages of MIRACL which were released later without associated training duta.
E LATENCY MEASURES
We provide per-query retrieval latency measured MS MARCO (retrieval from collection of 8.8.M documents) for SPLARE and SPLADE-Llame in Figure 5 To measure this, we first index the collection using Seismic Bruch al. 2024c). and then perform single-threaded retrieval on the
21

---

Published as a conference paper at ICLR 2026
Task Name
SPLARE
SPLADE-Llama
ArguAna
67.4
65.9
COADupstacKGamingReeni
59.6
59.3
CQADuustastUmisReriival
43.7
44.9
ClmnacEEVERRAadNNeeatiis
33.4
35.9
FEVERHardNegatives
90.6
91.1
FiQA2018
57.8
57.2
76.0
73.2
SCIDOCS
20.8
20.3
TRECCOVID
83.4
82.6
Touche20000Rerreval..33
60.8
58.4
Average
59.3
58.9
Table 6: Full results of SPLARE and SPLADE-Llama MTEB(Eng, v2).
Evaluation done with
Top-K = (40, 400).
Task Name
SPLARE
SPLADE-Llame
AlLAStatutes
38.9
36.3
ArguAna
67.4
65.9
BelebelcRetrieval
83,9
83.8
CovidRetrieval
83.3
81.5
HagridRetrieval
98.9
98.8
LEMBPasskeyRerrievall
48.2
48.2
LegaBBenchhCorrprtteLLbbbyy
95.1
94.9
MIRAACCRReriieallladdggggiss
72.4
70.5
MLQARetrieval
83.8
81.5
SCIDOCS
20.8
20.3
SpartQA
5.3
5.2
StackOverflowOA
88.4
90.0
StacanDDaloguueDattsteeettti
30.8
30.5
TRECCOVID
83.4
82.6
TempReasonLi
2.4
3,8
TwiiterfijermeRettivvall
73.2
73.4
WikipediaRetiivaaMullliggggg
91.7
90.6
WinoGrande
53.1
52.8
Average
62.3
61.7
Table Full results of SPLARE and SPLADE-LLams on MTTEBMultilinguall, v2). Evaluation done with Top-K (40,400).
saved index. Building a highly optimized sparse retrieval setup is difficult in general; here we use the Seismic library with default hyperparameters-
in Table 14
In this simple setup, retrieval takes around 5ms per query with maximal accuracy for SPLARE. In low-latency regime (<: 4ms), SPLARE can be used with higher accuracy compared to SPLADE.
F SPLADE LAYER ABLATION
In $ we showed that SPLARE models are typically more effective at intermediate layers, yielding latency advantage SPLADE. In principle, however, SPLADE models also be trained using intermediate representations by simply applying the LM head to these layers. Table 13 reports results obtained with this training procedure.
22

---

Published as a conference paper at ICLR 2026
Task Name
SPLARE
SPLADE-Llama
BelebelcRetrieval
83,9
83.8
MIlAACI.RrrievalHaardNeeee
72.4
70.5
MIQQARetrieval
83.8
81.5
SuacanDDaalguuuDDuustettttealll
30,8
30.5
TwiiterffjerneeRetriival
73.2
73,4
WikkpedaaReriivvllllllllggga
91.7
90.6
Average
72.6
71.7
Table 8: Full results of SPLARE and SPLADE-Llama the multilingual-only datasets of
(MUtlluliln v2). Evaluation done with Top-K = 400).
MTEB(eng. v2)
MTEB(eng, v2)
45
35
SPLARE SPLADE Liama
35
SPLARE
30
SPLADE- Liama
2
2
3
Latency (ms)
Latency (ms)
Figure 5: Retrieval Latency (ms) when pooling documents (Lefft or query (Right) representations
with Top-K. In low-latency settings, SPLARE enables higher accuracy for given level of latency.
Figure 6: Retrieval example from BEIR/ SetR//cifeen fact
Flexible molecules experience greater steric hindrance in the tumor microenviroment than rigid molecules.
Query:
Positive document: A solid tumor is an organ composed of cancer and host cells embedded in extracellular matrix and nourished by blood vessels. A prerequisite to understanding tumor pathophysiology is the ability to distinguish and monitor each component in dynamic studies. Standard fluorophores hamper simultaneous intravital imaging of these components. Here, used multiphoton microscopy techniques and transge [...]
(doc
top
rank
Exp lanat
perd Lin 2023
Token
references tumors and their rolated biological processes
8.6
_TURO"
12.7
terms related to drug delivery and collular mechanisms
7.5
_TuBora
8.1
related
research and
5.5
7.3
conditions diseases, particularly types cancer characteristics
J1.0m
concepts related to llexiblily in various contexts
4.8
Jhenopartieles
5,3
terms related to cellular processes and immune system lunctions
4,2
_Cancez
4,9
references expe experimental methods and cell-related terminoloyy
4.0
4,4
lerms related to microscapy
scientific analysis
Jhän
variations word "lumble" or its related lorms terms related to cancer and tumors
3.6
.solid
2.6
3.5
mellgnnan
2.6
G RETRIEVAL EXAMPLES
We provide in Figures 6 14 multiple examples of scores and explanations obtained for positive documents for some queries English, Multilingual and multi-domain datasets. We also pro- vide examples on the code domain (Figures 15 (7), which highlight some of the limitations of SPLARE on specific domains which might require dedicated SAEs. Notably, in Figure 14 which shows a Tamil example, the document and query representations coincide for only tokens, fur- ther highlighting SPLADE multilingual limitations. Note that the explanations we used, taken from Neuronpedia, are mostly annotated by LLMs provided with examples of context with features acti- vations. As such, these explanations may remain inaccurate or incomplete.
23

---

Published as a conference paper at ICLR 2026
Task Name
SPLARE
SPLADE -lama
Code
AppsRetrieval
28.2
30.7
CoIRCodeSeachhNeeRenieeaal
62.1
70.0
CodeEdiiSeurchRettivall
74.2
75.0
Cod
56.3
55.5
Codckeedbac
78.0
78.8
CodeSSeachhNeCCCRenie
61.3
64.4
CodeeSearchNetRRetiieaalll
85.3
86.8
CodeTrannsOccanCCnttet
86.7
88.5
Trad
36.3
33.4
CosQA
31.1
30.7
StackOverflowOA
88.4
90.0
SyntheticTTxx22SLL
68.2
67.5
Average
63.0
64.3
Medical
CUREvI
61.2
57.2
CmedqaRetrieval
30.2
31.5
MedicalQARRtrievall
74.7
75.7
NFCorpus
38.8
38.5
PublicHealthQA
86.7
86.2
SciFact
77.5
77.6
SeiFact-PL.
73.6
74.0
TRECCOVID
83,4
82.6
TRECCOVID-PL
82.8
84.8
Average
67.7
67.6
Law
AlLACasedocs
39.0
41.1
AlLAStatutes
38.9
36.3
GerlaL.RRmall
34.7
35.8
LeCaRDv2
62.3
61.3
LegilBeencCConmsieeeCCCttttaaa
87.4
87.9
LegalBennciCoppprreeLoooiigg
95.1
94.9
LegalQuAD
62.2
60.8
LegalSummarization
66.9
67.1
Average
60.8
60.7
ChemTEB
ChemHHotpotQARetrivval
86.2
85.3
ChemNQRetrieval
70.0
69.6
Average
78.1
77.4
Table 9: Full results of SPLARE and SPLADE-Llama on MTEB domain specific datasets. Evalua- tion done with Top-K (40, 400).
24

---

Published as a conference paper at ICLR 2026
Task Name
SPLARE
SPLADE- -lama
MTEB (Indic,
v1)
BelebeleRetrieval
87.1
87.0
XQuADRetrieval
97.5
96.9
Average
92.3
91.9
MTEB (Scandinavian,
v1)
DanFeverRetrieval
42.8
41.7
NorQuadRetrieval
23.6
26.5
SNLRetrieval
98.0
98.0
SweragRetrieval
79.4
77.0
SwednRetrieval
84.1
82.5
TV2Nordretrieval
94.0
94.0
TwitterffemmRReriivval
73.2
73.4
Average
70.8
70.4
MTEB (deu,
v1)
GerDaLIR
18.9
19.3
GermanDPR
87.5
86.7
97.2
96.8
XMarket
24.8
26.3
Average
57.1
57.3
MTEB (fra,
, v1)
AlloprofRetrieva
56.4
58.4
BSARDRetrieval
64.4
59.9
MintakaRetrieval
46.2
57.7
SyntecRetrievas
90.7
87.4
XPQARetrieval
66.4
64.6
Average
64.8
65.6
MTEB (kor,
v1)
Ko-Strategy QA MIRACLRetrieval
82.7
83.1
69.4
66.5
Average
76.0
74.8
Table 10: Full results of SPLARE and SPLADE-LIame on MTEB language-specific benchmarks. Evaluation done with Top-K = (40, 400).
Model
bu
fa
fc
hi
ja
ko
SW
te
th
2h
de
yo!
Avg
Baselines (Prior
Work)
BM25
39.5
48.2
26.7
7.7
28.7
11.5
35.0
29.7
31.2
37.1
35.1
38.3
49.1
17.5
12.0
56.1
31.9
mDPR
49.9
44.3
48.0
43.5
38.3
43.9
29.9
35.6
51.2
41.8
mCContriever
$2.5
50.1
36.4
41.8
21.5
60.2
31.4
28.6
39.2
42.4
48.3
39.1
56.0
$2.8
51.7
41.0
40.8
41.5
43.1
59.0
56.0
ES 35mttral-77
73.3
70.3
57.3
52.2
52.1
74.7
55.2
52.1
52.7
66.8
61.8
67.7
68.4
73.9
74.0
54.0
$4.1
79.7
63.4
Gemini Embedding
78.3
79.0
58.7
57.0
60.9
78.0
55.6
65.4
54.3
75.1
68.9
7344
81.0
80.5
80.8
65.7
59.8
88.8
70.1
M3-Emb (Sparse)
67.1
68.9
43.8
38.6
45.1
35.3
48.2
56.1
61.5
44.5
57.9
79.1
36.1
32.5
53.9
M3. Emb (All)
80.2
81.5
59.6
59.7
63.4
80.4
61.2
63.3
59.0
75.2
72.1
71.7
79.6
88.1
83.7
64.9
59.8
83.5
71.5
SPLADE-Llama SPLARE-7B
78.0
77.5
56.0
59.8
58.5
61.7
75.3
80.1
82.4
61.8
69.9
79.7
79.9
60.9
58.5
62.1
81.4
60.5
65.5
57.6
75.9
69.5
74.1
81.8
83.2
83.
64.1
62.5
90.0
71.7
SPLARE-28
69.7
59.0
Table 11: Multilingual retrieval performance on MIRACL dev (nDCG@ 10). Baseline results are taken from Chen et al. (024b) and (Lee et al. 2025b). t denotes the two hidden test sets of MIR- ACL. Evaluation for SPLARE and SPLADE-Llame done with Top-K (40, 400).
25

---

Published as a conference paper at ICLR 2026
Model
MRR 10
SPLARE
58.6
SPLARE (Eng Only)
41.6
SPLADE-Llama
56.2
SPLADE-Llama (Eng Only)
30.5
Gemini Embedding
64.3
Gemini Embedding (Eng Only)
49.3
Gecko i18n Embedding
35,0
voyage-3- --ggee--aagg
39.2
Langg-Embed-Mistral
24.6
18.7
gte- Nre-QQen2-77B- -iiiitt--eisttt
17.4
text-cmbeddding---arg
18.8
Table 12: XTREME-UP dataset (MRR@ @ 10) Average Scores. Baselines taken from (Lee al.
2025b). Evaluation for SPLARE done with Top-K (40, 400).
Model
Avg.
bho
hi
hnc
mai
PM
Eng Only
SPLARE
44.4
40.9
57.0
52.6
47.6
20.1
49.8
26.3
43.5
45.6
48.6
SPLADE-Llama
30.5
31.7
46.8
10.7
45.8
30.7
24.2 54.7 44.5 19.0 48.7 21.8
20.0
45.2
48.2
23.3
2.2
38.5
27.8
19.5
Multá iillinnnlll
SPLADE-Llama
56.2
61.2
588
14.2
59.4
56.5
62.4
64.1
62.5
62.2
64.5
63.2
29.7
60.5
59.6
53.2
62.2
50.7
62.7
59.0
37.6
Table 13: XTREME- UP dataset (MRR 10) Full scores. Evaluation done with Top-K = (40, 400).
Parameter
Value
k
1000
query cut
30
heap factor
0.5
n_knn
0
sorted
False
num sum_hhr
1
Table 14: Seismic retrieval parameters used to measure latency.
SPLADE-Llama at intermediate layers
Layer No.
18
22
26
31
MTng
0.
43.6
44.5
52.9
Table 15: Training HELADE-Llama models at intermediate layers leads strong deterioration. At
layer < 22, models collapse during training.
26

---

Published as a conference paper at ICLR 2026
Figure 7: Retrieval example from BEIR/Scifan
Query: PPAR-RXRs are inhibited by PPAR ligands. Positive document:
Heterodimerization is common paradigm among eukaryotic transcrip- tion factors. The 9-cis retinoic acid receptor (RXR) serves as a common heterodimerization partner for several nuclear receptors, including the thyroid hormone receptor (T3R) and retinoic acid receptor (RAR). This raises the question as to whether these complexes possess dual hormonal responsiveness. We devised a strategy to examine
SPLARE
COD features (doc rank - 5)
SFLADE
top tokens (doc rank - 18)
Explanate Lon (Zron Nour cripedaa
Lin 2023
Token
torms rolated to gane transcrption rogulation
6.2
JueL
6.2
mathemetical variables and expressions
4.3
JFX
6.0
terms related to dopamine and receptor interactions in the context of medicine and psycho
4.2
5.2
abbrovittioss torms related to programming and data structures
4.2
5.0
references OR codles VR technologles information about medications used for trealing
4,2
4.9
3.8
-receptors
4,3
references
and their biological functions
nuclear
3.8
terminology related to
development
terms to cellular functione regulstory terms rolated to
.trancoriptioo
biological concepti. particularly hormonos
JAX
Figure 8: Retrieval example from
Ocean acidification the terrifying threat whereby all that man-made CO2 we've been pumping into the atmosphere may react with the to form sort of giant acid bath.
Query:
Positive document: greenhouse gas ( (abbrev. GHG is a gas an atmosphere that absorbs and emits radiation within the thermal infrared range This process the fundamental cause of the greenhouse effect The primary greenhouse gases in Earth 's atmosphere are water vapor, carbon dioxide, methane, nitrous oxide, and ozone. Without greenhouse gases the average temperature of Earth 's surface would be a
(doc
top
Explanatiion on Reusorpeda 2023
references
change associaled
statements dispussions regarding climate change terms related to olimate change and impacts
_greenhouse
7:3
content related envirenmental impacts, particularly concerning carbon cloxide and lood...
4.1
Allmatee
5.7
torms related to carbon omissions and onvironmental impacts mentions of carbon dioside and its related metrics or expressions references to carbon diovide and its implications various contexts
4.1
Jearbon
5.6
3.9
_ctoxis
4.7
3.3
anchrop
4,2
references to environmental impact and sustainability
3.2
3.6
references human activity and impact on the envionment mentions sustainabilty pnvironmental
atnespheric
3.6
3.5
Figure 9: Retrieval example from BEIR/ BBIR/Cliiatoo-fvee
Query: No state generates as much solar power as California, or has as many people whose jobs depend on it.
Positive document: California the most populous state in the United States and the third most extensive by area Located on the western ( Pacific Ocean coast of the U.S. California bordered by the other U.S. states of Oregon Nevada and Arizona and shares an international border with the Mexican state of Baja California. The state capital is Sacramento Los A
SPLARE
top features (doc
SPLADE
top
tokens (doc rank 6)
Explanno fon (Fron Neunooped poooodaaaalll Lin 20233
%
Taken
references to financil or budgetary topics relerences to California
7.0
_Califoriia
8.5
6.8
fornia
6.3
references mentions of cities references California and its locations inntitutions mentions political
5.7
6.1
territoiis
Lattt
refarences to political ligures and legislation related Califonia references to geographic locations and regions Calffornia, particularly related to agr
Californ
4,7
38
Cal Calitornia
46
relerences to governance, laws, and political contexts
33
iforni
3.6
positvve descriptions and (eferences to tavorable weather conditions Californi''s
32
3.5
regulatory bodies
27

---

Published as a conference paper at ICLR 2026
Figure 10: Retrieval example from BEIR BBIR/Hotpotqa
Query: The Death of Cook depicts the death of James Cook at bay what coast?
Positive document: Kealakekua Bay is located on the Kona coast of the island Hawaii
about 12 mi south of Kailua-Kona
Neuroopedii features (doc rank
SPLARE
SPLADE
tokens (doc rank 17)
Expl
referances
similar geographical
references a specific geagraphical
or name containing "Bay"
Jhay
geographical features and sale navigation routes
9.5
.áttaa
9.1
references to health and community support systems
9.5
-3ay
8.2
historical references and significant events references to coastal regions and their characteristics relerences to historical sites and landmarks
9.3
.20g
7,8
7.6
5.4
5.4
5.1
relerences specilic geographical locations information related to marine coastal consystems
significance
.awwalan
bay
4.5
relerences to sailing, ships, and booling experiences
4.2
Jlaw
4.0
Figure 11: Retrieval example from MIRACL/Swahi15
Query: Kiongozi chama cha Orange Democratic Movement ni nani?
Positive document: Orange Democratic Movement Katika uchaguzi wa rais Raila Odinga alitangazwa kuwa ameshindwa
rais Kibaki kwa kura 230,000. Lakini watazamaji wengi waliona kasoro katika hesabu ya kura na ODM ilidai kuwa Odinga mshindi halali. ODM ilifaulu vizuri upande viti vya bunge la Kenya. llipata karibu nusu ya wabunge wote yaani 99 kati ya 120 ikawa kubwa katika bunge baada ya uchaguzi wa Desemba 200
top
(doc
top rank
Explanat Son (From Neuronped
2023
mentions of "Orange" or related terms and concepts references to events or oocurrences in the future prominent political figures and their involvement elections references the aboreviation 'OD' and variations terms associated with politccll
.aall
9.3
6.4
.Orange
8.5
6.0
.Serys
7.1
typically related spectiic...
Bai
discussions
references lo business stralegies and company sperations
.0D
5.5
references to DMCA regulations and related legal terma
4.1
Orance
4.6
references places in Kenys
4,0
4,2
information related notable historical figures and their relationships
3.6
_leadar
4,0
political candidales their
Parly
.oaange
Figure 12: Retrieval example from MIRACL/ /Bengali
Query (translated from Bengali): What is the name of the first band in Bangladesh Positive document (translated from Bengali): Obscure (Bangla Band) Obscure is one of the notable bands in the history of Bangladeshi band music. In the 1980s, Sayed Hasan Tipu took initiative to establish this band. On March 15, 1985, Tipu founded Obscure in Khulna. During the 1980s, Obscure's first album was released from Sargam Studio. That first self-titled album, "Obscure Volume 1," released 1986, earned permanent place in the history of Bangla band music.
SPLARE top features (doc rank
SPLADE
top
tokens (dlc rank 20)
Explanate oo ("rom Natronpedlla Lin (023)
%
Token
%
references to specific individuals groups
or cultural
JRang1 adeah
references to musica bands or groups
6.2
.bardd
8.5
mentions of bands and musical groups
5.5
7.9
repeated emphasized mentions specific entilies concepts references to iconic rock bands and their legacy
Pang
4,7
Joang
5.5
occurrences of the country name: "Bangladesh:
4,7
.Rand
5.3
references to musical bands elements related to music and musicians
collaborations
band
4.8
3.4
bands
42
descriptors related music and performance quality
bang
3.5
proper names and the mention individuals in the text
-band
28

---

Published as a conference paper at ICLR 2026
Figure 13: Retrieval example from MIRACL/French
Query: Qui est le mathématicien le plus célèbre au monde?
Positive document: Nira Chamberlain 2017, il intervient dans l'atelier du New Scientist "Le monde mathématique*. En 2018, il est nommé "mathématicien le plus intéressant du monde" par 'Big Internet Math Off organisé par le site "Aperiodical". 2019, il donne une conférence la "Maxwell Society" sur "Les mathématiques qui peuvent arrêter une apocalypse de I'I"". fait des apparitions dans les médias brita
features (doc rank 5) ion Yauroaped 2023
SPLARE
SPLADE
tokene (doc rark 11)
quantliative analysla
.mathensticc
terms and phrases related mathematics
Jeathenstic
elements related scademic papers and research acknowledgments
8.5
.m2ths
10.3
refarences to the concept of "world' in various contexts
8.3
Jmath
10,2
references to mathematical concepts and theorems
6.6
.Nath
8.1
discussions about ertistio individuale terms
concept creetivity
8.6
7.8
to mathematics mathemallolans
relerences mathematiical concepts terms related academio prolessionals and researchers across various fields
Jionda
5.1
.aathenaticaa
4,4
relerences to notable individuals and their contrbutions or warnings the field of arti
5.0
ematiik
3.9
Figure 14: Retrieval example from XTREME-UP: Tamil - English
Query: wellgarn agmemluss ReLLugRRoss giñüs (b44
Translation: On average, how much temperature can human withstand?
Positive document: Cold and heat adaptations humans The human body always works to remain in homeostasis. One form of homeostasis is thermoregulation. Body temperature varies in every individual, but the average internal temperature is 37.0 "C (98.6 IF). Stress from extreme external temperature
cause the human body to shut down [...]
SPLARE top features (document rank 2)
top tokens (document rank
Explonation on (Eron Neurongesis)) Lin 2023
Token
reforences to "human baings' and rolated concopts terms related to tempersture varistions and environmental conditions
12.3
Jhunan
29.3
11.9
_Hunan
21.5
terms related to lever and its physiological effects
10.3
20.7
terms related biological concepts and interactions
9.6
20.3
everages average specilic guideines recammendatioos phrases related summer and heat conditions
hrithsttnd
Jendurani ando
-1iit
specilic tomperature values and their measurements references to bodily systems and their components quantitative data related spenting and financial metrics
6.8
6.7
5.9
29

---

Published as a conference paper at ICLR 2026
Figure 15: Retrieval example from CoderditSearcRReriievll
Query: Add totally untested pools
Positive document:
+++
@e -1,4 +1, 88
-import abc +from multiprocessing ng import Pool ProcessPool +from multiprocessing www.ooceesiinduuunmy... import Pool as ThreadPool +from multiprocessing import epu_count
def do fo.flow(Elow) result=None): @0 18,19 +10,41 80 return result +class PoolAFI (object):
def map (sclf, *args, ;
return self.pool. map (+args, **kw)
+class ThreadPool (Poo]APT):
de
anat io (from SPLARE
Featurea (document rank
SPIADE
takena (document rark
Lin 2025)
Token
%
refarences to "pool" and related concepts in various contexts aspects of vacation experiences related to comfort and ameniiies
22.4
pool
17.0
20.0
-pools
15.1
refaronpes in speciie e evants or actions assuminn in smaline or sequence topics rolated to punk culture and its Influenca on music and community refarences to the ferm "stream" or varations within contaxts ralated art and medi comments about changes and improvements, partcularyy nn processes, products, performa... relerences to swimming pools and related recrestional tacilles
7.1
_Pool
14.5
6.9
8.
5.8
/cel
88
5.5
pool ing
6.6
3.8
.000l
39
relerences programming refarences programming intoriaces and datanase managemant
softwre development
_ipprovenen:s
expressions sports parformnce compatttio
jaabchea
Figure 16: Retrieval example from Code@diiteearchheetiiealll
Query: Make sure that the interests register tables created Nose tries run the interests register tests, but they will fail unless the interest.registerr app is added to APSS,L because its ta
Positive document:
@0 -8,7 +8, 00 'pombola.plae placeddata,, ppobolla.ottemttee. sspecche', pombola. spinner' )
"pombola. spinner',
+
APP_RREQUTRRDLRY.PRRHHH
create the FEATURESS hash that is used to toggle features on and off.
SPLARE top features (document rank 14)
SPLADE
top tokens (document rank 1)
Exp ion (tron Seurenped:al Lhn 2025
%
"oxten
keywords related to the concept of registration in verious contexts
9.5
-apps
5.7
phrases relating to vario aspects of interest diferent contexts
9.2
.ewgister
5.3
montions applications or soltware- related ferminology
5.9
5.2
foatures related to softwere libraries and their instalation mentione intereet various conteds suojects
4.7
mntereet
5.1
4.0
5.0
word create'
variation, indicating focus creation and gen
4.8
tochnical enroliment
query parameters
phrases rolated importing structuring control keywords to system contiguration management
agge
4.5
laterest
Jinteeaa
30

---

Published as a conference paper at ICLR 2026
Figure 17: Retrieval example from zederditssearccieeeeielle
Query: Update variables names in exam tests
Positive document:
+++
@e 16 +17,16 00 def test.create wwwpsy. exam from biopsy.models import Biopsy apecific.exam create.specific.exam Biopsy') biopay_exam create_speci
oisy')
specific exam should | be kind of (Biopsy)
def exam I should be kind of (Biopsy) from necropsy mod
biopsy
necropsy exam (self):
SPLARE
top features (document rank - 19)
SPLADE Token
topp tokens (document rank 1)
Explanation (Fxom. Ne (i) Lin 2023
%
terms associated with analysis and examination in specialized medical context
19.8
12.4
the
occurrences
variations
_axam
relerences to exaiss and testing processes
15.2
Jeikan
7.4
references to lesling and lest cases programming contexts
10.2
exaa::
6.4
references to unit testing its sssociated concepts
9.6
6.2
figures technical terms keywords related to programming computer science
2.6
.exam
references to institutions organizations in a structured context numerical values and measurements
2.1
eexaiiattion
4,3
1.7
tests
4.0
31


Reviews:
[{"review_id": "CEqiKGgb96", "reviewer": "Reviewer_7hiw", "summary": "This paper studies sparse auto encoders based latent features effectiveness in representing documents and queries. Authors claim that sparse autoencoders based learned sparse retrievers outperform their vocabulary projection based counterparts. The main contributions  of this paper are:\n1. This paper introduces SPLARE a new sparse retrieval technique leveraging sparse auto encoders.\n2. Detailed investigations of benefits of using latent vocabulary vs standard LLM vocabulary.\n3. New 7B multilingual sparse retriever model.", "questions": "1. How does the proposed approach with sparse embed  paper https://research.google/pubs/sparseembed-learning-sparse-lexical-representations-with-contextual-embeddings-for-retrieval/?\n2. Is availability of Pre-trained SAE a constraint of this model? \n3. Any comparison of latency and training time when compared to spade and other dense models?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The main strengths of the paper are as follows:\n1. Idea of using latent features for document and query representation might have a lot of applications in text retrieval\n2. The SPLARE learned sparse retrieval framework is interesting.\n3. Detailed performance comparison with SOTA text retrieval models.", "weaknesses": "The main Weaknesses of the paper are as follows:\n1.  Details about SPLARE working is missing. It might need some more explanation.\n2. Terms like SAE width needs some more details.\n3. Limited novels of the overall retrieval model when compared to sparse embed and other LSR techniques.\n4. How SAEs are trained is not clear from the paper.\n5. Paper writing have a lot of scope of improvement.\n6. Limited performance improvement on MTEB retrieval benchmark when compared to SPLADE-LLama", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1HfnAhd7RZ", "reviewer": "Reviewer_EBaU", "summary": "This paper proposes SPLARE (SParse LAtent REtrieval), a novel Learned Sparse Retrieval (LSR) approach that leverages pre-trained Sparse Autoencoders (SAEs) to represent queries and documents as sparse vectors over a latent feature space rather than the traditional vocabulary space. By inserting SAEs into intermediate layers of large language models (LLMs), SPLARE produces semantically rich, multilingual, and domain-generalizable sparse embeddings. The authors demonstrate through extensive experiments on benchmarks like MMTEB, MIRACL, and XTREME-UP that SPLARE consistently outperforms vocabulary-based LSR models—especially in multilingual and out-of-domain settings—while maintaining high retrieval efficiency. A 7B-parameter multilingual variant of SPLARE achieves competitive results against state-of-the-art dense retrievers, establishing it as the top-performing LSR model on MMTEB at the time of submission.", "questions": "1.\tThe paper claims to propose \"Learned Sparse Retrieval\", but its core component SAE, is frozen throughout the training process. This means that the model cannot learn or optimize the \"latent vocabulary\" itself for retrieval purposes and can only be adapted to a fixed and non-designated feature space for the retrieval task through LoRA fine-tuning of the LLM's intermediate representations. This is fundamentally different from LSR methods such as SPLADE (whose LM head is learnable), and it weakens the claim of \"Learned\".\n2.\tThe paper not only employs DF-FLOPS type sparse regularization but also adds Top-K pooling during inference to forcibly control the number of activations. The authors also admit that training solely with Top-K would be worse, but still choose the \"moderately sparse\" + post-cropping approach during training. This inconsistency will introduce non-differentiable post-processing and distribution drift, potentially leading to instability near the critical threshold and sorting reversals.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThe paper is the first to systematically use the pre-trained sparse autoencoder as the \"implicit vocabulary\" of LSR, replacing the traditional methods based on the original tokenizer vocabulary.\n2.\tThe proposed method demonstrates outstanding performance in multilingual and cross-domain scenarios.", "weaknesses": "1.\tAlthough the authors find that the best performance was achieved at Layer 26 for different SAE widths, this conclusion is only drawn from the experiments conducted on the Llama-3.1-8B model. If readers wish to apply the SPLARE method to other models (such as Gemma), there is no guarantee that the selection of this layer will be successful. It is suggested that the author provide more experimental results on the models and the layer selection strategies to enhance the generalization ability of the method.\n2.\tCurrent state-of-the-art dense retrievers generally adopt contrastive learning, but SPLARE chose distillation. The paper merely refers to distillation as a \"common toolbox\", but does not provide an explanation as to why distillation is more suitable for the sparse latent space. This crucial design choice lacks an analysis of motivation.\n3.\tThe paper clearly states that only residual SAEs were used, but no ablation experiments or reasons are provided. SAEs using MLP or Attention might capture richer or different types of semantic information. Ignoring them could lead to information loss and limit the model's expressive power.\n4.\tTable 1 shows that in both the English and Multilingual benchmarks, SPLARE significantly lags behind SPLADE-Llama in the Code domain. The paper only mentions \"the advantage diminishes\" without providing any analysis (such as feature visualization, error cases). This may indicate that the SPLARE method has flaws when dealing with highly structured and symbolic text (such as code), and this needs to be emphasized.\n5.\tThe abstract and introduction have repeatedly emphasized that SPLARE is the first LSR model that can match the SOTA dense model on MMTEB. However, Table 3 shows that its score of 60.9 is much lower than that of top dense models such as Qwen-3-Embedding-8B (70.9), inf-retriever-v1 (66.5), etc. The discussion in the paper contains an exaggeration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lUb9XXDqwr", "reviewer": "Reviewer_xNQK", "summary": "This paper introduces SPLARE, a novel Learned Sparse Retrieval model that ingeniously integrates Sparse Autoencoders (SAEs) with existing LSR frameworks like SPLADE. SPLARE projects text representations into a \"latent feature space\" learned by a pre-trained SAE, which is designed to be more semantic and language-agnostic.Through extensive experiments, the authors demonstrate that SPLARE significantly outperforms its vocabulary-based counterparts on multilingual and out-of-domain retrieval tasks.", "questions": "1. Can you provide more information on the analysis and comparison of deployment costs?\n2. Regarding sparsity control, the paper notes that combining training-time regularization with inference-time Top-K pruning was superior to using Top-K alone. Could the authors elaborate on the distinct effects of these two methods on the final representations? For example, does the regularization loss encourage the model to learn a better distribution of important features, which Top-K then effectively truncates, whereas relying solely on Top-K might crudely discard features that are contextually important but have marginally lower activation scores?\n3. The model training relies on knowledge distillation from a cross-encoder. Was a contrastive learning approach, which is dominant in dense retriever training, considered or attempted?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.The paper originally combines the semantic feature decomposition of SAEs with learned sparse retrieval;\n2.The experimental results are impressive. SPLARE consistently outperforms SPLADE-Llama in multilingual and out-of-domain settings and achieves a level of performance comparable to SOTA dense models on comprehensive benchmarks like MMTEB.\n3.Excellent Generalization and Efficiency: SPLARE demonstrates strong multilingual generalization even when trained only on English data.", "weaknesses": "1. Lack of Detailed Cost Analysis: The introduction of the SAE module incurs additional computational and memory costs at inference time. While the paper mentions mitigating this by using intermediate LLM layers, it does not quantify the latency and memory overhead introduced by the SAE projection step itself. \n2. Insufficient Context for Sparsity: The paper controls document vectors to ~400 non-zero dimensions. While this is sparse in a >130k-dimensional space, its advantage is not immediately obvious when compared to dense vectors, which may only have a few hundred to a thousand dimensions in total. The paper lacks a direct, end-to-end comparison of latency, index size, and memory usage between SPLARE (with inverted indexes) and a top-tier dense retriever (with an ANN index like HNSW) at a similar effectiveness level.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)