CSRv2: Unlocking Ultra-Sparse Embeddings
Authors: Lixuan Guo
Yifei Wang
Tiansheng Wen
Yifan Wang
Aosong Feng
Bo Chen
Stefanie Jegelka
Chenyu You
Venue: ICLR 2026 Poster (2026)
Keywords: text embedding
sparse representation
contrastive learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Generating ultra-sparse representations with only 2 or 4 non-zero elements through CSRv2

Abstract: In the era of large foundation models, the quality of embeddings has become a central determinant of downstream task performance and overall system capability. 
Yet widely used dense embeddings are often extremely high-dimensional (e.g., 4096), incurring substantial costs in storage, memory, and inference latency. 
To address these, Contrastive Sparse Representation (CSR) is recently proposed as a promising direction, mapping dense embeddings into high-dimensional but $k$-sparse vectors, in contrast to compact dense embeddings such as Matryoshka Representation Learning (MRL). 
Despite its promise, CSR suffers severe degradation in the ultra-sparse regime (e.g., $k \leq 4$), where over 80\% of neurons remain inactive, leaving much of its efficiency potential unrealized.
In this paper, we introduce CSRv2, a principled training approach designed to make ultra-sparse embeddings viable. 
CSRv2 stabilizes sparsity learning through progressive $k$-annealing, enhances representational quality via supervised contrastive objectives, and ensures end-to-end adaptability with full backbone finetuning. 
CSRv2 reduces dead neurons from 80\% to 20\% and delivers a 14\% accuracy gain at $k=2$, bringing ultra-sparse embeddings on par with CSR at $k=8$ and MRL at 32 dimensions, all with only two active features. 
While maintaining comparable performance, CSRv2 delivers a 7$\times$ speedup over MRL, and yields up to 300$\times$ improvements in compute and memory efficiency relative to dense embeddings in e5-mistral-7b-instruct-based text representation.
Extensive experiments across text (MTEB, multiple state-of-the-art LLM embeddings (Qwen and e5-Mistral-7B), SPLADEv3, GraphRAG) and vision (ImageNet-1k) demonstrate that CSRv2 makes ultra-sparse embeddings practical without compromising performance, where CSRv2 achieves 7\%/4\% improvement over CSR when $k=4$ and further increases this gap to 14\%/6\% when $k=2$ in text/vision representation.
By making extreme sparsity viable, CSRv2 broadens the design space for large-scale, real-time, and edge-deployable AI systems where both embedding quality and efficiency are critical. 
Code is available at https://github.com/Y-Research-SBU/CSRv2.

Published as a conference paper at ICLR 2026
CSRV2: UNLOCKING ULTRA-SPARSE EMBEDDINGS
Lixuan 1.2 Yifei Wang 3
Tiansheng Wen 1.2
Yifan Wang!
Aosong Feng
Bo Chen?
Stefanie Jegelka 1,5
Chenyu You
Stony Brook University
Xidian University
Amazon AGI SF Lab
TUM
5 MIT Yale University
ABSTRACT
In the era of large foundation models, the quality of embeddings has become a central determinant of downstream task performance and overall system capa- bility. Yet widely used dense embeddings are often extremely high-dimensional (e.g., 4096), incurring substantial costs in storage, memory, and inference latency. To address these, Contrastive Sparse Representation (CSR) is recently proposed as promising direction, mapping dense embeddings into high-dimensional but k-sparse vectors, in contrast to compact dense embeddings such as Matryoshka Representation Learning (MRL). Despite its promise, CSR suffers severe degra- dation in the ultra-sparse regime (c.g.. S 4), where over 80% of neurons re- main inactive, leaving much of its efficiency potential unrealized. In this pa- per, introduce CSRv2, principled training approach designed to make ultra- sparse embeddings viable. CSRv2 stabilizes sparsity learning through progressive k-annealing, enhances representational quality via supervised contrastive objec- tives, and ensures end-to-end adaptability with full backbone finetuning. CSRv2 reduces dead neurons from 80% to 20% and delivers a 14% accuracy gain k 2. bringing ultra-sparse embeddings on par with CSR at k = 8 and MRL at 32 dimensions, all with only two active features. While maintaining compa- rable performance, CSRv2 delivers 7x speedup over MRL, and yields up to 300x improvements in compute and memory efficiency relative to dense em- beddings in E--miittal--b-iinttuut-bb ments across text (MTEB, multiple state-of-the-art LLM embeddings (Qwen and c5-Mistral-7BB, SPLADEv3, GraphRAG) and vision (ImageNet- 1k) demonstrate that CSRv2 makes ultra-sparse embeddings practical without compromising per- formance, where CSRv2 achieves 7%/4% improvement over CSR when k: 4 and further increases this gap to 14% when k = 2 in text/vision representation. By making extreme sparsity viable, CSRv2 broadens the design space for large- scale, real-time, and edge-deployable AI systems where both embedding quality and efficiency are critical. Code is available at https://github.comm com/Y-Research-- SBU/CSRv2.
text representation. Extensive experi-
INTRODUCTION
In the era of large foundation models, the quality of embeddings has become decisive factor shap- ing downstream performance across tasks such as retrieval, classification and recommendation. Yet the dominant practice still relies on dense representations with thousands of dimensions (e.g., 2048 8192). While highly expressive, such embeddings incur substantial costs in storage, memory, and inference latency. These inefficiencies are magnified in large-scale and real-time deployments, where embedding computation and serving often dominate system throughput. As models scale fur- ther, embedding efficiency emerges central bottleneck, which limits both web-scale applications and deployment on resource-constranne platforms such as mobile and edge devices.
Several methods have been proposed to improve embedding efficiency, but they face sharp trade-offs under extreme compression. Existing approaches improve efficiency but falter under extreme com-
* Equual Contribution.
corresponding author: chenyu. wwwuulssooyboooo..om edu
1 This work was done MIT prior to Yifei Wang joining Amazon.

---

Published as a conference paper at ICLR 2026
Full Rep. Dense
MRL Rep.
CSR Rep. Sparse
CSRv2 Rep.
MRL -C- CSR - CSRv2-linear
Dense Truncated
Ultra-Sparse
CSRv2 - Backbone CSRv2@K=2
Accuracy (%)
65
Festure Dim
50
35
Active Dim
(a) Different efficient embedding schemes.
(b) Average accuracy text embedding tasks.
Figure 1: Overview of our proposed method. (Left): An illustrative comparison between full emhedding, truncated MRL embedding, medium-sparse CSR embedding and ultra-sparse CSRv2 embedding. (Right): Comparison of average text embedding performance on 6 types of tasks in MTEB benchmark with 5-mmisral-7b-instrud backbone. To ensure fair comparison, all methods are trained on the same data. We refer to the <5-mistral-7b-instee model without task-specific finetuning as the 'Backbone' baseline.
pression. Matryoshka Representation Learning (MRL) (Kusupati al., 2022) trains embeddings to function at multiple truncation lengths, yet expressivity collapses and accuracy drops sharply below hundred dimensions. Contrastive Sparse Representation (CSR) (Wen et al., 2025) instead maps emheddings into high-dimensional sparse vectors, outperforming MRI and matching its quality with only one-quarter of the active dimensions. Despite this potential, CSR suffers severe degra- dation in the ultra-sparse regime (k = or 4). We refer to this regime as ultra-sparse embeddings, which in principle can deliver over 100x efficiency gains in large-scale retrieval. However, existing methods incur 20 40% accuracy losses in this regime, rendering such embeddings impractical in real-world scenarios. This raises a central question:
Are ultra-sparse embeddings inherently constrained, or can proper training mitigate this?
Driven by this question, we take closer look at ultra-sparse embeddings and identify three key challenges. First, they suffer from a "massive dead neuron" problem: even with modern mitiga- tion techniques, more than 85% of neurons remain permanently inactive when CSR activates only two neurons (A 2), severely limiting expressivity. Second, the mismatch between pretraining objectives and downstream tasks becomes amplified under ultra-sparsity, so CSR relying purely supe-super signals (e.g., image cropping) leads to pronounced degradation. Third, we observe that CSR also shows greater degradation when jointly trained on multiple datasets and domains, indicating that restricting it to a linear layer on top provides insufficient representational capacity.
To address the above challenges, develop CSRv2, improved training recipe for sparse em- beddings that is as simple and generic as CSR(v1) yet delivers substantial and consistent gains in ultra-sparse regimes. CSRv2 combines curriculum annealing schedule, which prevents early col- lapse when learning ultra-sparse embeddings, with natural supervision from labeled data, which replaces the noisy self-supervision of CSR and utilizes the few active dimensions more effectively. In addition, beyond training only linear layer (CSRv2-linear), we explore finetuning the entire backbone with our objectives, analogous to the MRL setting, and show that this further improves generalization across domains, establishing new state-of-the-ar results and outperforming MRI. by up to 25% under the same training conditions. Altogether, CSRv2 provides the first reliable recipe for shrinking modern embeddings to just two four active dimensions with only modest perfor- mance drops. This opens a new understanding of representational capacity and paves the way for extremely memory- and compute-efficien applications such edge devices, robotics, and real-time search engines. We discuss in detail the evolution of text embedding and adaptive embedding tech- niques in Appendix A, highlighting the correlations and limitations of existing methods that motivate CSRv2.
To summarize, contributions
2

---

Published as a conference paper at ICLR 2026
Table 1: Overview of the training paradigms, objectives. trainable parameters, and performance (cf. Figure 1b) of the four efficient embedding methods discussed in this paper.
Method
Training
Objectives
Trainable Params
MRL
Supervised Self-supervised Self-sup. + Sup. Self-sup. + Sup.
Multi-length Cross Entropy
Full Finetuning
CSR CSRv2-linear
SAE + Contrastive
Linear Head
k-anncaling SAE Sup. Contrastive
Linear Head
CSRv2
k-annealing SAE Sup. Contrastive
Full Finetuning
We systematically explore the regime of ultra-sparse embeddings and diagnose three main causes of failure in prior methods: dead neurons, lack of effective supervision, and limited model capacity.
We propose CSRv2, a simple and generic training recipe that addresses these issues through k-annealing for ultra-sparsity, supervised sparse contrastive learning, and optional full- model finetuning for multi-domain robustness.
We validate CSRv2 extensively on text (six MTEB tasks and two domains in GraphRAG- Benchmark) and image (ImageNet-Ik) show up to 4x efficiency gains over CSR and 16x over MRL at comparable performance, and attain 10% 30% accuracy improvements on state-of-ther Qwen3 Embedding models under short embedding lengths.
Our training data, code, and CShy2-enhanced versions of Qwen3 and e5-Mistral- 7B are available on ensuring compatibility with existing model configura- tions and readiness for production use. We are further committed to extending CSRv2 to a broader set of open-source models. By releasing these resources, we aim to encourage new research direc- tions and practical applications of ultra-sparse embeddings that have not yet been explored.
https://ittub..om// wetee/-eeeeeee-B//CCCCCC
BACKGROUND
The goal of representation learning is map high-dimensional inputs (such images or text) into low-dimensional embeddings that capture semantic similarity. Consider text embeddings LLM backbone encodes them into embedding pairs {(1 di).... (qN, dy }} where (9:, d,) denotes a query-document pair. The embeddings (Oord et al., 2018). However, standard embeddings typically remain high-dimensional (2k-8k), creating significant bottleneck for large-scale, real-time retrieval systems, including search, rec- ommendation, and address this by producing embeddings with adaptive dimensionality for efficient applications.
query- document pairs that share similar semantics, as an example: given a batch of query
then trained with contrastive loss such as InfoNCE
generation. Here, we review two representative approaches
Matryoshka Representation Learning (MRL). Instead of applying the loss solely on the full- size embeddings, MRL (Kusupati et al., 2022) truncates the first m E M dimensions of the text embeddings d[1 m] E "M and applies the same loss function on a set of truncated lengths M with relative importance scale Cy- Formally, the objective of MRL as follows:
log exp (s(g: 1:m, de 1.m)/7)
MMRL
Σ Cm MEM
(I)
N
RE[N]
where -(-,-) is the similarity function (cosine similarity in most cases), 7 is the temperature param- eter and Z; denotes the normalization factor that comes in different forms (Zhang et al., 2025a;b). Generally, the number of selected truncating lengths [M] will not be larger than [log(d) of the original embedding size d and all the relative importance scale Gm will be set to 1.
Contrastive Sparse Representation (CSR). Instead of training the whole model as in MRL, CSR (Wen et al., 2025) takes a pretrained encoding model (with frozen weights), and trains a simple sparse autoencoder (Cunningham et al., 2023) on top for mapping the pretrained dense embeddings x Idd into a sparse embedding z E dR with up to < non-zero elements (i.e., k-sparse):
z (op (x bpre) + bene)),
(2)
cc Wdee + bpre:
(3)
3

---

Published as a conference paper at ICLR 2026
CSR
cannall
CSR cannan
CSR
Cannannall
neuron ratio(%)
+0.30
10.08
60.0
60.0
Accuracy
40.0
40.0
20.0
Active Dim
Epoch
Active Dim
(a) Dead neuron active dim
(b) Dead neuron during training
(c) Accuracy active dim
Figure 2: K-annealing analysis on ImageNet-1k with FF2048 as backbone. (Left):: Comparison of dead neuron ratio before and after applying k-annealing in different sparsity levels. (Middle): Dead neuron trend during training before and after applying annealing when k = (Right): Eval- uation results on ImageNet-Ik IK with I-NN accuracy the main metric.
where the TopK operator keeps the top k largest values while setting the others to zero, RcLU(z) max(x; 0) keeps non-negative elements, and W enc and W de: the encoder and decoder matrices. The CSR model is jointly optimized via TopK sparse autoencoder (SAE) (Gao et al., 2024) and sparse contrastive learning (NCL) (Wang et al., 2024b). The overall training objective is,
LCSR [(k) ((4))/ 3Laxx + 7CSpCI.
(4)
The MSE loss L(k) || * caleulates the difference between original dense feature R
and reconstructed dense feature é E R2 from k-sparse embedding z. Training with the multi-TopK loss C(k) /(4k)/8 ensures that CSR could generalize to different ks time, The sparse contrastive loss LSpeL computes InfoNCE loss over sparse embeddings as Wang et al. (2024b):
[8]
exp 2, 2;
LspeL
log
(5)
[B
exp 2 2, Σ exp
Lastly, the auxiliary loss Laux |le é calculates the difference between the reconstruction error e x Ø and the reconstruction using the top-kaux dead latents W which is proposed by Gao et al. (2024) for reducing dead neuron's 's effect on performance degradation.
Computational Complexity. By exploiting short and sparse embeddings, both CSR and MRL sig- nificantly improve the memory and computational efficiency of embedding models. In particular, retrieval with a k--dimensional short embedding & requires only O(k) memory and compute to evalu- ate query- document similarity (instead of ((d) with æ). Likewise, storing k-sparse embedding in compressed formats (e.g.. CSR or CSC) incurs ((k) memory and enables O(k) compute via sparse matrix multiplication, which is natively supported in modern CPU/GPU libraries such PyTorch.
Wen et al. (2025) further show that CSR and MRL achieve comparable retrieval time at the same k. Hence, k serves as a convenient surrogate for both memory and computational cost.
CSRv2: TACKLING NEW CHALLENGES UNDER ULTRA-SPARSITE
Although CSR achieves impressive performance by closely matching the accuracy of full-size em- beddings at relatively high sparsity levels (k 8, 16, 32), we observe that its performance deterio- rates rapidly at extremely small values of k (e.g., In this section, we uncover several key reasons underlying CSR's 's failure in the ultra- regime and show that is actually largely fixable with several improved training techniques introduced here.
4). We refer to this regime as ultra-sparsity.
3.1 TACKLING MASSIVE DEAD NEURONS WITH K-ANNALLING
The Massive Dead Neuron Phenomenon. As discussed in Wen et al. (2025), critical advantage of CSR over MRL is that sparse embeddings z E I& can exploit a large number of hidden neu- rons d. > i for better feature expressivity, while only activating few (k) for retrieval efficiency.
4

---

Published as a conference paper at ICLR 2026
However, observe that as l + 1, dead neurons arise more severe problem. A dead neuron is a feature dimension that remains inactive on any data sample, indicating that it fails to represent anything useful. As shown in Figure 2a, the dead neuron ratio quickly increases as l decreases, rising to 70% at & = 4 and reaching 90% at k = 2. This means that these ultra-sparse embeddings can only utilize 10% to 30% hidden dimensions, which greatly limits their representation power.
Why Dead Neurons are more Severe under Ultra-sparsity. Although CSR already integrates common remedies for dead neurons, such auxiliary losses and multi-TopK strategies (Jermyn & Templeton, 2024; Gao et al., 2024), our experiments reveal that these approaches, effective at moderate sparsity (k- 32, 64), become largely ineffective when is extremely small. The difficulty is intrinsic: only the k selected dimensions in each sparse code receive non-zero gradients, leaving the majority of neurons untrained. Under ultra-sparsity with only a handful of active dimensions, this issue becomes particularly severe. Moreover, once neuron falls inactive, it receives no gradient signal and thus cannot recover, creating a self- relf-reinforcing loop that further increases dead neurons.
Alleviating dead neurons with k-annealing. To alleviate this problem, we instead adopt cur- riculum learning approach: we warm up the training with a sufficiently large initial sparsity level kmit (by default kintt = 64), which avoids severe neuron inactivity and allows the model to learn meaningful latent space in the early stage. As training proceeds, & is gradually annealed toward the target ultra-sparsity knmal (c.g., kanal = 2) using a linear schedule. Specifically, at epoch t we
k
pt) kimit + Pt kifinal,
Pt t/T,
where T' is the total number of annealing steps. In practice, we perform annealing for 70% of training, after which k is fixed at kfimal Analogous to simulated annealing, starting with a larger kinit promotes exploration and diverse neuron activations, while the gradual annealing hinit - kinal sharpens the representations and enables stable convergence in the ultra- ultra-sparse regime.
We find this approach effectively maintains a low dead-neuron rate during training. As shown in Figure 2b, although dead neurons rise slightly at target sparsity, their final proportion far lower than training directly with kinnal- This indicates that a curriculum schedule provides richer gradients and avoids collapse into the read-neuron regime. Similar to simulated annealing, a larger kintt pro- motes exploration and diverse activations, while annealing gradually sharpens embeddings toward the ultra-sparse regime. Figure 2c confirms this, as k-annealing yields consistent performance gains across sparsity levels.
Remark. It is worth noting that LlamaScope (He et al., 2024) also employs a k-aaneelling strategy, but with very different motivation and scope. Their annealing is applied only during the first 10% of training, reducing k from the full embedding dimension (kimi = d) to a moderate sparsity level (Kanal = 50) to accelerate convergence. In contrast, method anneals k over most of the train- ing process, specifically to mitigate the massive dead neuron problem that arises under ultra-sparsity. Moreover, LlamaScope restricts annealing SAEs, while we apply it efficient embeddings. Thus, our finding that progressive k-annealing is critical for overcoming dead neurons at beddings still constitutes a novel and valuable contribution to the literature.
em-
3.2 LEARNING DOWNSTREAM DALIINE FEATURES FROM NATURAL SUPERVISION
ALIGNED
For ultra-sparse embeddings that activate only a few dimensions, the model must prioritize infor- mative features and suppress noise. CSR, relying on self-supervised objectives like autoencoding and contrastive learning (Section 2), may be suboptimal. Its ingmentatioonbased positives (e.g.. cropping), though effective. transfer poorly when downstream tasks need properties ignored during training. (Ericsson et al., 2021). This weakness is exacerbated under ultra- ultra-sparsity, where noisy features are easily activated while informative ones are lost.
Sparse Supervised Contrastive Learning. To bridge this gap, we follow the setting of MRI. and adopt natural supervision, which is readily available in many retrieval tasks, to construct more accurate positive pairs. For example, in labeled datasets such as ImageNet, two random im- ages from the same class can be used as positive pair. In text retrieval datasets, qucry-document pairs naturally serve as positives. This supervision enables ultra-sparse embeddings to dedicate their limited active dimensions to encoding informative features that align with downstream applications, rather than wasting capacity on noisy features. Concretely, we replace CSR's sulf-suue con- trastive loss with supervised contrastive loss (Khosla et al., 2020) applied to the k-sparse embed-
Remedy:
5

---

Published as a conference paper at ICLR 2026
supervised - self selt-supee
Accuracy (%6
50
55
-50
75
50
25
25
50
75
-55
50
25
25
50
Active
(a) accuracy vs supervision
(b) sulf-supe t-SNE features
(c) supervised t-SNE features
Figure 3: (Left): Supervision leads to performance increase in ultra-sparse setting with e5-Mistral-
7B backbone. (Middle & Right): t-SNE visualization comparison MTOPDomain (Li et al., 2020) before/after adding supervision when
2.
SimgleDomain (linear)
MultiDomain (linear)
MultiDomain (finetune)
60.30
6o
nDCoo 10 19))
APN
80
V-measurg
%
40
20.01
60
20
50
Imdi
Banking77 TweetSenti
BioClu.525
TwenN
StaEx
30
NFCorpus
Arguana
SciFact
Classification
Clustering
Figure 4: Comparison of CSRv2-linear trained on single-domain dataset, CSRv2-linear trained multi-domain dataset and CSRv2 trained on multi-domain dataset in different tasks. c5-Mistral-7BB selected as backbone and training splits all tasks in the same task type are combined for multi- domain.
dings:
[8] Σ log
Eper(i) exf Σ EPEP() e-l + EmEN() e²!=="'
1
CspsCL (k)
[
where P(2) and Mii)) denote the sets of positive and negative samples derived from natural super- vision. Specifically. for classification and clustering tasks, samples with the same label are treated as positives, while others are negatives. For retrieval and reranking tasks, each query and its corre- sponding documents are positives. For semantie textual similarity, sentence pairs with similarity score above 3 are positives. For pair classification, sentence pairs with label 1, indicating strong correlation, are positives. A detailed description these tasks is provided Appendix B.
From Figure 3, observe that supervised training yields clear performance gain in ultra-sparse settings. Moreover, supervision produces sparse features that are far more discriminative across classes. It demonstrates that these supervision signals provide clearer signals for training ultra- sparse embeddings. More detailed ablation on applying natural supervision to CSR available in Section 4.3. The community has far curated abundant pretraining-scale paired text data for training retrieval models, such as 65M Q-A pairs (Lewis et al., 2021). Therefore, would quite useful be able leverage large-scale supervision.
3.3 MITIGATING MULTI-DOMAIN TRAINING GAPS VIA FINETUNING
notable property CSR is that can outperform MRL (which requires full finetuning) by training only simple encoder with single linear layer. However, this design also limits CSR's ability to fully exploit the potential of sparse embeddings, particularly when deploying single model across multiple downstream tasks. As shown in Figure 4, CSR with only linear layer experiences clear performance drop under joint training with multi molti-domain data in different task types, reflecting the limited capacity of such shallow adaptation.

---

Published as a conference paper at ICLR 2026
Table 2: Performance and retrieval efficiency six text embedding tasks with e5-Mistral- 7B. Since e5 does not natively support MRL or CSR, we enable fair comparison by training all methods on the same backbone, data, and configurations. For retrieval efficiency. experiments are conducted with IM database, and results are reported as retrieval time relative to CSRv2 at k 2.
Active Dim
Retrieval Time
Classifi. ACC 1
Clust. V-measure
Retrieval aDCU@101
STS
PairClassif. APT
Rerank.
Method
Spearman
MAP1 1
4096
BB-Mistrll7B
306.46x
80.67
51.55
49.35
84.11
91.77
69.52
69.99
MRL
301.86x
80.46
30.94
48.75
83.78
90.44
68.86
69:49
x
4096
CSRv2-linear
x96.04
80.55
51.19
49.07
84.02
91.48
69.02
CSRv2
x
80.49
51.34
49.16
83.94
91.70
69.18
69.80
MRL
17.30x
77.46
14.92x
64
CSRv2-linear
x
82.09
CSRv2
14.17x
79.98
82.90
90.46
67.34
MRL
7,77x
54.64
42.03
34.33
68.18
59.22
51.93
CSR
3,53x
77.30
84.28
62.83
16
CSRv2-linear
3.51
79.37
CSRv2
3.51
79.94
64.36
MRL
6.29
43.84
33.14
24.55
56.51
37.36
44.72
40.83
CSR
1.62x
67.22
39.25
23.54
70.13
74.44
48.57
52.94
CSRv2
73.31
CSRv2
1.63x
74.26
43.85
75.69
74.90
62.93
61.01
MRI.
6.20
34.84
26.13
16.63
52.14
26.67
40.30
33.81
62.93
2
CSRv2-linear
CSRv2
1,00x
73.82
62.46
To fully unlock the potential of sparse embeddings and push CSR its limit, we adopt the same setting as MRL: applying the TopK operator to the output embeddings of the backbone network and finetuning the entire model. Figure shows that full finetuning effectively mitigates the performance drop observed under the linear setting, recovering performance comparable to domain-specific CSR training (additional details are provided in Appendix C).
Building on all these findings above, propose the following improved sparse training objective:
LCSRV2 = C(ke) + C(4k)) + 3axxxxxx + (/Cspsc((ki)
(8)
where he is the annealed sparsity level step (Eq. 6) and LspscL denotes the sparse supervised contrastive loss (Eq. 7).
We designate the fully finetuned model CSRv2, and the variant that finetunes only linear layer on top (as CSR) as CSRv2-linear. TopK SAE (Gao al., 2024) finds that using L(k) + L(4k) is enough to obtain progressive representation over all k. We find similar phenomena for CSR and thus follow this common practice. The improved training recipe remains as simple and generic as the original CSR without introducing more training objectives. In the experiments that follow, we are able to show that both CSRv2 and CSRv2-linear deliver significant gains over CSR and MRL, particularly in the ultra-sparse regime. Furthermore, the fully finetuned CSRv2 sets performance -eficnncccceefcnnyyyyyyc frontier for adaptive embeddings, surpassing MRL by up to 25% absolute accuracy under the same setting. During inference, the embedding produced by the backbone first goes through an encoder which projects it onto a high dimensional vector (e.g. 16384). Afterwards, the TopK values in the vector are kept while others are 0, with normalization applied.
EXPERIMENTS
In this section, comprehensively evaluate the effectiveness of CSRv2. For language represen- tation, we evaluate on tasks in Appendix B. For visual representation, we conduct experiments on ImageNet- 1k (Deng al., 2009) and evaluate using I-NN accuracy (Kusupati et al., 2022). More-

---

Published as a conference paper at ICLR 2026
Table Performance comparison with Qwen3-Embeddding-4 (Zhang et al., 2025b), a state-of- art embedding model on MTEB that natively supports MRL. Backbone results are shown in the first line and first/second largest value on each active dimension bold underlined.
Active Dim
Classifi. ACC 1
Clust.
Retrieval nDCG@ 10
STS
PairCInssif.
Rerank. MAP
Method
Avg.
V-measure
Spearman
AP
Qwen3-Embed-4B
55.27
91.42
74.92
MRL
85.38
55.04
58.31
88.02
91.27
71.64
74.58
CSR
85,49
54.83
58.21
88.64
91.23
71.84
74.70
2560
CSRv2-lincar
55.43
58.74
89.05
91.03
72.25
74.99
CSRv2
88.47
91.39
74.91
MRL
83.42
53.73
44.13
86.60
88.08
69.61
70.54
CSR
83.94
52.36
51.51
85.33
90.54
70.
71.10
CSRv2-linear
84.03
53.19
53.22
85.88 85.88
90.72
72.31
CSRv2
54.57
86.32
90.90
72.79
MRL
75.22
47.24
20.40
79.21
73.29
60.82
58.89
CSR
78.60
49.08
35.66
82.08
85,80
65.00
64.66
CSRv2-linear
51.48
82.15
67.20
CSRv2
82.63
90.42
68.98
MRL
48.27
32.08
6.59
53.11
30.73
40.59
36.74
CSR
$7.39
36.03
16.27
64.13
64.29
50.26
46.66
CSRv2-lincar
43.24
72.11
77.74
56.76
CSRv2
80.20
48.27
77.94
82.28
62.41
MRL
26.47
24.20
5.23
30.22
18.46
32.43
22.84
CSR
41.83
30.02
9.37
51.27
54.60
44.20
36.29
CSRv2-linear
66.95
39.22
18:47
71.56
77.95
54.67
53.41
CSRv2
76.22
46.02
23.93
74.88
25.24
59.52
58.53
over, we conduct efficiency analysis and empirical analysis on ablation of each component and dead neurons. Case study of representation interpretability for a more detailed assessment of the advan- tages and potential of CSRv2 is proposed in Appendix F.
4.1 BENCHMARK PERFORMANCE
Evaluation under controlled setup. For fair comparison, we adopt e5-Mistral-7B (Wang et al., 2023) as backbone and finetune it on MTEB datasets to ensure MRL aligns with CSRv2 domains. Table 2 reports sask-type-ppecffic results on six task types commonly adopted in past works (Zhang al., 2024) (Li et al., 2024a) (Lee et al., 2024a) in MTEB (Muennighoff et al., 2022), where CSRv2 trained on all train splits of the same task type. Under equal activation dimensions, CSRv2 consis- tently outperforms CSR, with up to 14% gains in the ultra-sparse case i 2. Notably, CSRv2 also surpasses MRL: at k = 2, it exceeds MRL's dense 16-dim embedding and even outperforms 64-dim dense embeddings in text classification. Efficiency tests 1M database further show CSRv2's ultra-sparse embeddings achieve 300x retrieval speedup over the backbone and 7 faster retrieval than MRL's dense embeddings of similar accuracy. More detailed results and implementation details appear in Appendix C.
Evaluation on State-of-the-art Qwen3 Models. We further evaluate on Swen3-EEmbedding-4B whose series leads the MTEB leaderboard, with even the 0.6B model surpassing prior 7B re- sults. Unlike E5-Mistral-7B. Qwen3 integrates MRL into training, producing embeddings naturally aligned with it. As shown in Table 3, CSRv2 consistently outperforms both MRL and CSR at equal compression. In cross-level comparisons, CSRv2 at k = 16 rivals MRL = 64, and CSRv2 at k 2 rivals MRL at 16, highlighting its adaptability across backbones and sparsity levels.
Evaluation Comparison with SPLADE Sparse Retrieval Model. As a representative SOTA for learning-based sparse retrieval, SPLADE (Lassance et al., 2024) generally achieves high perfor- mance with ~ 3% activation. However, our MTEB comparison reveals its fragility high-sparsity settings. SPLADEv3 at K 16 clearly lags behind CSRv2, and remarkably, its retrieval quality at K 8 trails CSRv2 at K 2, highlighting our method's 's superior compression efficiency. (See Appendix C.5).
8

---

Published as a conference paper at ICLR 2026
Zero-shot Evaluation in GraphRAG System. We further assess CSRv2's zero-shot capability on medical and novel domains using the GraphRAG Benchmark (Xiang et al.. 2025), focusing on retrieval accuracy and generation quality. Despite being trained solely on MTEB (zero-shot setting), CSRv2 shows significantly less degradation than MRI., indicating superior robustness to unseen data distributions. Detailed results and setup provided in Appendix E.
Visual Embedding Evaluation on ImageNet-lk, Figure 5a demonstrates CSRv2's performance on ImageNet-Ik with pre-trained ResNet-50 noted as FF2048 in the MRL (Kusupati al., 2022) as backbone. We find that CSRv2 achieves continuous improvement in classification performance compared CSR and MRL. This phenomenon is particularly prominent in the extremely sparse case, where CSRv2 achieves a 6% I-NN accuracy increase over CSR and 20% over MRL. More detailed results and experiment setup are in Appendix D.
4.2 EFFICIENCY ANALYSIS
In Figure 5c, evaluate CSRv2 and MRL retrieval efficiency under hidden dimension R and ac- tive dimension on a IM-scale database. Retrieval time grows roughly linearly with as predicted by O(dk), though GPU architecture also influences performance. In ultra-sparse cases (k 2), CSRv2 leverages GPU sparse accelerators (e.g.. Sparse Tensor Core, cuSPARSH) to run over 6x faster than MRL. As sparsity decreases (k: 32), dense-optimized libraries (e.g., cuBLAS) reduce dense operators' overhead, shrinking CSRv2's advantage to 2.2x. Thus, CSRv2 excels in extreme sparsity while maintaining stable gains in general sparse settings. Experiment setups and more dis- cussions on encoding, indexing and retrieval are presented in Appendix G.1.
4.3 EMPIRICAL ANALYSIS
Ablation. Table reports ablations of CSRv2 components. Supervision proves most effective for compression, while anneal alone yields little gain. Yet combining them (CSRv2-linear) outperforms adding supervision alone, showing synergy: anneal promotes feature orthogonality and subspace expansion, while supervision directs semantic learning, where the two play different but comple- mentary roles. Finetuning further aligns backbone embeddings with sparse objectives, adding 5% improvement ls 2.
Dead Neurons. Figure 5b shows dead neuron fractions across components. While adding unsuper- vised contrastive loss in CSR yields more independent features and fewer dead neurons in sparse embedding (e.g. A = 32), CSR still suffers severe dead neuron issues in ultra-sparse cases (e.g. k 2). Anneal distributes semantic features into broader hidden subspace, reducing dead neu- rons by 70% at k = 2. Natural supervision further lowers them to about 20%. Finetuning brings little improvement, likely because the TopK strategy only aligns backbone embeddings with sparse objectives rather than fostering orthogonal representations.
K-Schedule Sensitivity Analysis. We test on k-annealing's 's sensitivity on three perspectives: k- schedule shape, length (i.e. ratio of steps before & reaches target sparsity level) and k's initialization. Results show that different k-schedule results in relatively stable increase performance improve- ment, while selected setting: initialized to 64, annealing target sparsity level at 70% step, and linear-annealing strategy achieves the best performance. More detailed results are Appendix G.2.
Further Discussions. Moreover, we have conducted several experiments, which provide potential directions for future cxploration. These discussions are analysis on unbalanced weightable settings for MRL and CSRv2 finetuning (Appendix G.3), emergence of superclass separability in sparsity representation (Appendix H. MRL-SAE exploration (Appendix H.5) and quantized comparison at fixed memory cost (Appendix H.2). Furthermore, CSRv2 can be potentially applied in vector quantization due to its sparse structure, with a brief discussion in Appendix H.3.
5 CONCLUDING REMARKS
Unlike prior methods (CSR, MRL) that fail once < 4. CSRv2 provides the first principled recipe that makes ultra-sparsity viable. The central insight is that ultra-sparsity is not merely a parame- ter regime but qualitatively different optimization problem: (i) standard sulf-super losses misalign with downstream semanties when only two four features remain, and (ii) dead neurons

---

Published as a conference paper at ICLR 2026
Table 4: Performance Ablation Comparison: We perform ablation study with c5-Mistral-7B as backbone through task-type-specifie evaluation and average performance of all task types is pre- sented. We mark improvement of different combinations relative to CSR with green, while perfor- mance gap between MRL and CSR with red.
Components auncalsupervisccinetung
Active Dimension
64
32
2
61.47
56.37
51.85
47.09
40.83
33.81
MRL
(-8:33)
(9.53)
(-11.0)
(42.11
CSR
x
x
66.68
64.60
61.38
58.19
52.94
44.17
67.35
65.24
61.91
58.79
54.55
45.33
anneal
X
X
(40,67)
(+0.64)
(+0.33)
(+0,00)
(+1.6)
(+1.1)
67.32
65,54
62.95
60.05
56.36
49.05
supervise
x
(+1.6)
(43422
(+4.88)
CSRv2-linear
x
67.58
65.83
63.73
61.53
58.62
53.25
(+0.90)
(+1.23)
(+2.35)
(+3.34)
(+5.68
(+9.08)
CSRv2
v
68.08
66.70
65.22
63.76
61.01
58.34
(+1.40)
(+2.10)
(+3.84)
(+5.57)
(+8.07)
(+14.17)
MAL -O- CSR CSRv2-linear
CSR
CSRv2-linar
-o- MRL (Dense Rep.) -o- hw4d (16384)
CSRv2
Backbone
CaR+anneal -o- CSRv2
h=8d (32768) -o- h=16d (C5536)
70
80.0
5.6
Accuracy (95)
80
40.0
2.8-
20.0
Retrieval
1.4
4
8
15
32
64
2
8
15
32
16
32
64
Active Dim
Active Dim
Active
(a) Visual embedding evaluation
(b) Dead neuron comparison
(c) Retrieval time per active dim
Figure 5: (Left)): Visual representation results on ImageNet- 1 with FF2048 as backbone. "Back- bone" serves as the evaluation results FF2048 without finetuning for consistent reference. (Mid- dle): Dead neuron trend with different components under varying compression levels. (Right): Efficiency analysis in IM database size with e5-Mistral-7B backbone.
accumulate irreversibly without curriculum. CSRv2 introduces two non-trivial modifications mo- tivated by this diagnosis: progressive k-annealing schedule that preserves gradient flow across neurons until late training, and supervised sparse contrastive objective that reallocates the few ac- tive features carry semantic signal. These mechanisms essential for surviving the ultra-sparse regime and go beyond "better tuning" of CSR's 's original objective.
A key open challenge is the and sharp degradation (Appendix H.4). Since k = 1 effectively reduces to clustering (mapping each input one-shot label), future work could explore prototype or vector quantization, balanced assigmment, entropy regularization, or optimal transport. Extending CSRv2 into this extreme setting remains an important direction, while the practically useful ultra-sparse range k E {2, 12,4,8, 8) already offers substantial efficiency gains with competitive accuracy.
regime, where CSRv2 still suffers from severe dead neurons
ceattering-inspiree approaches, such as
Ethics Statement. This work adheres to the ICLR Code of Ethics. Although our proposed methods are broadly applicable, their deployment in real-world scenarios may carry societal considerations, particularly regarding bias, fairness, and privacy. We advocate for responsible application of our techniques and disclose that we have no conflicts of interest.
Reproducibility. We provide comprehensive details of our methodology, datasets, model architec- tures, and evaluation protocols both the main text and Appendix. Full mathematical derivations and additional experimental results included in the Appendix. We publicly release the source code and scripts to facilitate complete reproduction of our experiments https//github.comm/ Research-SBU/CSRe
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
Yifei Wang and Stefanie Jegelka were supported in part by the NSF AI Institute TILOS (NSF CCF- 2112665), and an Alexander von Humboldt Professorship
REFERENCES
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. Semeval-2012 task 6: A pilot on semantic textual similarity. in sem 2012: The first joint conference on lexical and computational crauuntcs-volumee 1: Proceedings of the main conference and the shared task, and volume 2: Proceedings of the sixth international workshop semantic evaluation (semeval 2012). In Pro- ceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), Montréal, QC. Canada, pp. 7-8, 2012.
Eneko Agirre, Daniel Cer, Mona Diab, Aitor Goiralez-Agirre, and Weiwei Guo. sem 2013 shared task: Semantic textual similarity. In Second joint conference on lexical and computational se- mantics (* SEM), volume 1: proceedings of the Main conference and the shared task: semantic textual similarity, pp. 32 22-43,22 2013.
Parul Awasthy, Aashka Trivedi, Yulong Li, Mihaela Bornea, David Cox, Abraham Daniels, Martin Franz, Gabe Goodhart, Bhavani Iyer, Vishwajeet Kumar, et al. Granite embedding models. arXiv preprint arXiv:2502. 20204, 2025.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han. Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. 2023.
Anil Bandhakavi, Nirmalie Wiratunga, Stewart Massie, et al. Generating a word-emotion lexicon from# emotional tweets. In Proceedings the third joint conference on lexical and computational semantics (* SEM 2014), pp. 12-21, 2014.
Ergun Biçici. Rım-dcu: Predicting semantic similarity with referential translation machines. In Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015), pp. 56- 63, 2015.
Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. A full-text learning to rank dataset for medical information retrieval. In European Conference Information Retrieval, pp- 716-722. Springer, 2016.
Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda. Learning multi-level features with matryoshka sparse autoencoders. arXiv preprint (XXX::2503.155477 2025.
Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. Efficient intent detection with dual sentence encoders. arXiv preprint 24800 2020.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task and lingull focused evaluation. arXiv preprint
1: Semantic textual arXiv: 170..0005000 2017.
Xi Chen, Ali Zeynali, Chico Camargo, Fabian Flöck, Devin Gaffney, Przemyslaw Grabowicz, Scott Hale, David Jurgens, and Mattia Samory. SemEval-2009 task Multilingual news article sim- ilarity. In Guy Emerson, Natalie Schluter, Gabriel Stanovsky, Ritesh Kumar, Alexis Palmer, Nathan Schneider, Siddharth Singh, and Shyam Ratan (dds.), Proceedings of the 16th Interna- tional Workshop on Semantic Evaluation (S0mEval-2022), pp. 1094-1106 Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 18.18663/v11 1/tp:333///2222mememmss 1.155. URL https:ttpp.......... / /ccl//ccccccac anthll ooy.. .rgg wtgg//222 2022. semeval -..

---

Published as a conference paper at ICLR 2026
Qinyuan Cheng, Xiaogui Yang. Tianxiang Sun, Linyang Li, and Xipeng Qiu. Improving contrastive learning of sentence embeddings from ai feedback. arXiv preprint arXiv:2335.01198, 2023.
Chanyeol Choi, Junseong Kim, Seolhwa Lee, Jihoon Kwon, Sangmo Gu, Yejin Kim, Minkyung Cho, and Jy-yong Sohn. Ling-embed-mistre technical report. arXiv preprint arXiv:2412.03223, 2024.
CireleMind-Al. Fast-graphrag: Retrieval-augmented generation for graphs. https: //gith/b 2025. Software; commit (accessed 2025-11-11).
cjadams, Daniel Borkan, inversion, Jeffrey Sorensen, Lucas Dixon, Lucy Vasserman, and nithum. Jigsaw unintended bias in toxicity classification, https: com/ wwwm///mpoiitttooooooooomeoeeoee itnnss/
2019. Kaggle.
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. SPECTER:
Document-level representation learning using citation-informed transformers. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp- 2270-2282, Online, July 2020a. As- sociation for Computational Linguisties. doi: 10. 1202 //aclanthologg. httptchhhhoooooogg/eeeeo wwt.ooog///20 acl-main..mmmmmm
URL https:
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld. Specter: Document-level representation learning using citation-informed transformers. arXiv preprint arXiv:2004.07/88, 2020b.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.0866 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 218-24822555 255. leee, 2009.
Zhibin Duan, Tiansheng Wen, Muyao Wang, Bo Chen, and Mingyuan Zhou A non-negative vae: the generalized gamma belief network. arXiv preprint arXiv:2408. N.X0:22008.03388 2024a.
Zhibin Duan, Tiansheng Wen, Yifei Wang, Chen Zhu, Bo Chen, and Mingyuan Zhou. Contrastive factor analysis, 2024b. URL https: /ttpps://axxtee /ttp://wwwv.or////........o..oot 2btp///../2222////// 21740.
Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pp. 5414-5423, 2021.
Kion Fallah, Adam Willats, Ninghao Liu, and Christopher Rozell. Learning sparse codes from com- pressed representations with biologically plausible local wiring constraints. Advances in Neural Information Processing Systems, 1911399--1963,, 2020.
Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, et Massive: Im-example mul- tilingual natural language understanding dataset with 51 typologicully-divess languages. arXiv preprint arXiv:2204.085.5 2022.
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. Splade v2:
Sparse lexical and expansion model for information retrieval. arXiv preprint arXiv:2109.10088 2021a.
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. Splade: Sparse lexical and ex- pansion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2288-2292. 2021b.
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406. 04093, 2024.
12

---

Published as a conference paper at ICLR 2026
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821, 2021.
Tiezheng Ge, Kaiming He, Qifa Ke, and Jian Sun. Optimized product quantization for approximate nearest neighbor search. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 2946-2953, 2013.
Gregor Geigle, Nils Reimers, Andreas Rücklé, and Iryna Gurevych. Tweac: transformer with ex- tendable qa agent classifiers. arXiv preprint arXiv:2104.07081, 2021.
Yunchao Gong, Svetlana Lazebnik, Albert Gordo, and Florent Perronnin. Iterative quantization:
A procrustean approach to learning binary codes. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2011.
Ruiqi Guo, Philip Sun, Erik Lindgeen, Quan Geng, David Simcha, Felix Cher, and Sanjiv Ku- mar. Accelerating large-scale inference with anisotropic vector quantization. In International Conference on Machine Learning, pp. 3887-3896. PMLR, 2020.
Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang. Zuxuan Wu, et al. Llama scope: Extracting millions of features from 1lama-3.1-8 with sparse autoencoders. arXiv preprint arXiv:2410.20522 2024.
Doris Hoogeveen, Karin M Verspoor, and Timothy Baldwin. Cqadupstack: A benchmark data set for community question-ansse answering research. Proceedings of the 20th Australasian document computing symposium, pp. 1-8, 2015.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient ahigge--rithmmeticonn inference. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2704-2713, 2018.
Suhas Jayaram Subramanya, Fnu Devvrit, Harsha Vardhan Simhadri, Ravishankar Krishnawamy, and Rohan Kadekodi. Diskann: Fast accurate billion-point nearest neighbor search on a single node. Advances in neural information processing Systems, 32, 2019.
Herve Jegou. Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence, 33(1):117-128, 2010.
Hervé Jégou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1):117-128, 2011.
Adam Jermyn and Adly Templeton. Ghost grads: An improvement on resampling. Trans- former Circuits Thread, 2024. URL jan bannuppdate/ iindx...... htmll dedicttlearnnnneeeeagg.
pub/2024/
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 8661118661173 2020.
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation learning. Advances in Neural Information Processing Systems, 35:30233-3022 2022.
Michael Lan. Philip Torr, Austin Meek, Ashkan Khakzar, David Krueger, and Fazl Barez. Sparse autoencoders reveal universal feature spaces across large language models. arXiv preprint arXiv:2410.0698), 2024.
13

---

Published as a conference paper at ICLR 2026
Wuwei Lan, Siyu Qiu, Hua He. and Wei Xu. A continuously growing dataset of sentential paraphrases. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pp. 1224-1234, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/1/DD7711122 URL https:///cllanthhloogg. wwt.gyyoorg///1111/
Carlos Lassance, Hervé Déjean, Thibault Formal, and Stéphane Clinchant. Splade-v3: New base- lines for splade. arXiv preprint arXiv:2403.06789 2024.
Guillaume Leclere, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Alek- sander Madry. Ffev: Accelcrating training by removing data bottlenecks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12011-12021 2023.
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi. Bryan Catan- zaro, and Wei Ping. Nv-embed: Improved techniques for training Ilms as generalist embedding models. arXiv preprint arXiv:2405. /428,, 2024a.
Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer. Jeremy R Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, et al. Gecko: Versatile text embeddings distilled from large language models. arXiv preprint arXiv:2403.20327 2024b.
Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer, Madhuri Shanbhogue, Iftekhar Naim, Gus- tavo Hernández Ábrego, Zhe Li, Kaifeng Chen, Henrique Schechter Vera, et al. Gemini embed- ding: Generalizable embeddings from gemini. arXiv preprint arXiv:2503.077 07.9999, 2025.
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. Paq: 65 million probably asked questions and what you can do with them. Transactions of the Association for Computational Linguistics, 111098-1115. 2021.
Chaofan Li, MingHao Qin, Shitao Xiao, Jianlyu Chen, Kun Luo, Yingxia Shao, Defu Lian, and Zheng Liu. Making text embedders few-shot learners. arXiv preprint arXiv:2409.15700 2024a.
Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. Mtop: A comprehensive multilingual task-oriented semantic parsing benchmark. arXiv preprint arXiv:2008.099 09335, 2020.
Mingxin Li, Zhijie Nie, Yanzhao Zhang, Dingkun Long, Richong Zhang, and Pengjun Xie. Im- proving general text embedding model: Tackling task conflict and data imbalance through model merging. arXiv preprint arXiv:2410. 15035, 2024b.
Xianming Li, Zongxi Li, Jing Li, Haoran Xie, and Qing Li. 2d matryoshka sentence embeddings. arXiv preprint 147:72602..4776, 2024c.
Yuxiao Li, Eric J Michaud, David D Back, Joshua Engels, Xiaoqing Sun, and Max Tegmark. The geometry of concepts: Sparse autoencoder feature structure. Entropy, 27(4)3344, 2025.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches oul, pp. 74-81, 2004.
Xueqing Liu, Chi Wang. Yue Leng, and ChengXiang Zhai. Linkso: dataset for learning to retrieve similar question answer pairs software development forums. Proceedings of the 4th ACM SIGSOFT International Workshop on NLP for Software Engineering, 2018. URL https://
api semant cscholars org/ www.oo/ro///ooooo..ooo CoopppsDD:3311115
Xinyuan Lu, Liangming Pan, Yubo Ma, Preslav Nakov, and Min- Yen Kan. source tool-augmented framework for explainable table-based reasoning. arXiv preprint arXiv:2409. / 2024.
Tart: An open-
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguisties: Human language technologies, pp. 142-150, 2011.
14

---

Published as a conference paper at ICLR 2026
Maggie, Phil Culliton, and Wei Chen. Tweet sentiment extraction. https: ptttps://kaggle.oom com/ sent www.teinnneeettttiiiiiee 2020. Kaggle.
Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. Www' 18 open challenge: financial opinion mining and question answer- ing. In Companion proceedings of the the web conference 2018. pp. 1941-1942, 2018.
Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014), pp. 1-8, 2014.
Philip May. Machine translated multilingual sts benchmark dataset. 2021. URL https: wttsbmbtttimttt.....
// //githuub http//hub.ccm//hiiiiiie.ee...eee
Iman Mirzadeh, Keivan Alizadeh, Sachin Mehta, Carlo Del Mundo, Oncel Tuzel, Golnoosh
Samei, Mohammad Rastegari, and Mehrdad Farajtabar. Relu strikes back: Exploiting activation sparsity in large language models. arXiv preprint arXiv:2310.04566 2023.
Gabriel de Souza P Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge. Ny-retriever: Improving text embedding models with effective negatvee mining. arXiv preprint arXiv:2407.15831, 2024.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embed- ding benchmark. arXiv preprint arXiv:2210.07311 2022.
Niklas Muennighoff, SU Hongjin, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amampreet Singh, and Douwe Kiela. Generative representational instruction tuning. In The Thirteenth International Conference on Learning Representations, 2024.
Preslav Nakov, Alan Ritter, Sara Rosenthal, Fabrizio Sebastiani, and Veselin Stoyanov. Semeval- 2016 task 4: Sentiment analysis in twitter. arXiv preprint wwXiv:1912.01997 2019.
Thong Nguyen, Mariya Hendriksen, Andrew Yates, and Maarten de Rijke. Multimodal learned sparse retrieval with probabilistic expansion control. European Conference on Information Retrieval, pp. 448-464, Springer, 2024.
James O'Neill, Polina Rozenshtein, Ryuichi Kiryo, Motoko Kubota, and Danushka Bollegala. T wish i would have loved this one, but i didn t-a multilingual dataset for counterfactual detection in product reviews. arXiv preprint 2rXiv:2104.06893, 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint 1807.v11807.03.. 2018
Letian Peng, Yuwei Zhang, Zilong Wang. Jayanth Srinivasa, Gaowen Liu, Zihan Wang, and Jingho Shang. Answer is all you need: custruction-folloviii text embedding via answering the question. arXiv preprint arXiv:2402.0966 2024.
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoen- coders. arXiv preprint arXiv:2404.1660 16014, 2024.
Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang. Junlin Wu, and Yi-Shin Chen. Carer: Contex- tualized affect representations for emotion recognition. Proceedings of the 2018 conference on empirical methods in natural language processing, pp. 3687-3697, 2018
Darsh Shah, Tao Lei, Alessandro Moschitti, Salvatore Romeo, and Preslav Nakov. Adversar- ial domain adaptation for duplicate question detection. In Ellen Riloff, David Chiang, Ju- lia Hockenmaier, and Jun ichi Tsujii (eds.). Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pp. 1056-1063, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/ 1131. URL wwwethooogy.... ogy. ogg.org//1.-111111
15

---

Published as a conference paper at ICLR 2026
Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Q-BERT: Hessian based ultra low precision quantization of BERT. In Advances in Neural Information Processing Systems (NeurIPS), 2019
Gizem Soğancioğlu, Hakime Öztürk, and Arzucan Özgür. Biosses: a semantic sentence similarity estimation system for the biomedical domain. Bioinformatics, 33(14);14--i5, 2017.
Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang. Markus Krim- mel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, et al. jina-embeddings- v3: Multilingual embeddings with task lora. arXiv preprint arXiv:2409, MI11:3409.10173, 2024.
Nandan Thakur, Jianmo Ni, Gustavo Hernández Ábrego, John Wieting, Jimmy Lin, and Daniel Cer. Leveraging Ilms for synthesizing training data across many languages in multilingual dense retrieval. arXiv preprint arXiv:2311.05800 2023.
Cornell University, arXiv dataset (metadata for 1.7m+ scholarly papers). Kaggle dataset. httppp//wwww..kkggeeec/////////s inivesstiiniieeeeiyaaiirii, 2025. Accessed September 3, 2025.
Henning Wachsmuth, Shahbaz Syed, and Benno Stein. Retrieval of the best counterargument with- out prior topic knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 241-251, 2018.
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang. Madeleine van Zuylen, Arman Co- han, and Hannaneh Hajishirzi. Fact or fiction: Verifying scientific claims. arXiv preprint N0t::2000.149744 2020a.
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang. Madeleine van Zuylen, Arman Co- han, and Hannaneh Hajishirzi. Fact or fiction: Verifying scientific claims. arXiv preprint 2XXv:2004414977.. 2020b.
Hongyu Wang. Shuming Ma, Ruiping Wang, and Furu Wei. Q-sparse: All large language models can be fully sparsely-activated. arXiv preprint arXiv:2407. 10969, 2024a.
Kexin Wang, Nils Reimers, and Iryna Gurevych. Tsdae: Using transformer-based sequen- tial denoising auto-encoder for unsupervised sentence embedding learning. arXiv preprint arXiv:2104.06979, 2021. URL https: httpivaivvvv.////..//aaa wog.oorggo.........o............ 2204. 06979.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang. Rangan Majumder, and Furu Wei. Improv- ing text embeddings with large language models. arXiv preprint arXiv:2401.00333 2023.
Yifei Wang. Qi Zhang, Yaoyu Guo, and Yisen Wang. Non-negative contrastive learning. arXiv preprint arXiv:2403. 12459, 2024b.
Yair Weiss, Antonio Torralba, and Rob Fergus. Spectral hashing, In Advances Neural Information Processing Systems (NeurIPS), 2008.
Tiansheng Wen, Yifei Wang, Zequn Zeng, Zhong Peng, Yudi Su, Xinyang Liu, Bo Chen, Hong- wei Liu, Stefanie Jegelka, and Chenyu You. Beyond matryoshka: Revisiting sparse coding for adaptive representation. In International Conference on Machine Learning, 2025.
Zhishang Xiang, Chuanjie Wu, Qinggang Zhang, Shengyuan Chen, Zijin Hong, Xiao Huang, and Jinsong Su. When use graphs rag: comprehensive analysis for graph retrieval- -uggeeted generation. arXiv preprint 2506v:2506000 05660, 2025.
Hanqi Yan, Yulan He, and Yifei Wang. The multi-faceted monosemanticity in multimodal represen- tations. In Workshop on Responsibly Building the Next Generation of Multimodal Foundational Models, 2025.
Jinsung Yoon, Raj Sinha, Sercan O Arik, and Tomas Pfister. Marryoshka-adaptor:: Unsupervised and supervised tuning for smaller embedding dimensions. arXiv preprint arXiv:2407.20243 2024.
Chenyu You, Yifei Min, Weicheng Dai, Jasjeet S Sekhon, Lawrence Staib, and James S Duncan. Calibrating multi-modal representations: A pursuit of group robustness without annotations. In CVPR, 2024.
16

---

Published as a conference paper at ICLR 2026
Chenyu You, Haocheng Dai, Yifei Min, Jasjeet S Sekhon, Sarang Joshi, and James Duncan. Un- covering memorization effect in the presence spurious correlations. Nature Communications, 2025.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parametter-efficien fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199 2021.
Dun Zhang, Jiacheng Li, Ziyang Zeng, and Fulong Wang. Jasper and stella: distillation of sota embedding models. arXiv preprint arXiv:2412. 19048, 2024.
Xin Zhang, Yanzhao Zhang, Wen Xic, Dingkun Long, Mingxin Li, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. Phased training for llm-powered text retrieval models beyond data scaling. In Second Conference on Language Modeling, 2025a. URL hatttps://openreeett net/forum's 1ic::1GGNNNGG...
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advanc- ing text embedding and reranking through foundation models. arXiv preprint a7Xiv:2506.05176, 2025b.
Shengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin, and Guido Zuccon. Promptreps: Prompting large language models to generate dense and sparse representations for zero-shot doc- ument retrieval. arXiv preprint arXiv:2404. 18424, 2024a.
Shengyao Zhuang, Shuai Wang, Fabio Zheng, Bevan Koopman, and Guido Zuccon. Starbucks-v2: Improved training for 2d matryoshka embeddings, arXiv preprint (tti:22410.123.00 2024b.
17

---

Published as a conference paper at ICLR 2026
A ADDITIONAL RELATED WORK
LLM-based Text Embeddings. The integration of Large Language Model into text embedding generation has been a hot topic due to LLM's extraordinary capability of comprehensive semantic understanding. This integration has led to many embedding models that have demonstrated excellent performance in multiple domains, multiple tasks, and multiple languages, such GritLM (Muen- nighoff et al., 2024), 35-Mistral-7B-instree 2025), Qwen3 Embedding (Zhang et al., 2025b) and Linq- -istral (Choi et al., 2024).
(Wang et al., 2023), Gemini Embedding (Lee et al.,
Generally, the techniques utilized in training these models can mainly be classified into two cate- gories. One approach is utilizing L.LMs for text augmentation or synthetic data generation, therefore expanding the domain covered by model training. Jina-v3 (Sturua et al., 2024), Gecko (Lee et al., 2024b) and Tart (Lu et al., 2024) utilize LLM to generate synthetic examples enhance task-wise generation and expand targeted failure cases. NV-Embed-v2 (Moreira et al., 2024), E5-Mistral (Cheng et al., 2023) and SWIM-X (Thakur al., 2023) employ LLM to provide higher-quality supervision signals for existing embedding training.
Another approach directly adapting LLMs themselves to serve as text embedding models, there- fore transfer knowledge from large LLMs to more efficient embedding models. Generally, this approach takes one pretrained L.M as backhone such as Mistral 7B (Wang et al., 2023) and fine- tune with parameter-efficient finetuning strategies including BitFit (Zaken et al., 2021) and LoRA (Hu et al., 2022). Current innovation for LLM adaptation to embedding generation focus on three aspects: design of positive/negative pairs, multi-stage learning and instruction tuning. For design of positive/negative pairs, (Gao et al., 2021) proposes an unsupervised contrastive learning frame- positive
work for advancing sentence embeddings, where augmented unlabeled sentences are pairs. NV-Retriever (Moreira et al., 2024) filters out potential false negatives by comparing candi- date negatives against the positive relevance score. Granite Emhedding models (Awasthy et al., 2025) use additional bidirectional signal to expand negatives in retrieval set. Similarly, You et al. (2024) propose contrastive feature recalibration approach to mitigate spurious correlations and en- hance group robustness without relying on group annotations. For multi-stage training, NV-Embed (Lee al., 2024a) takes a two-stage contrastive trieval datasets with in batch and hard negatives, then blends in non-retrieval tasks without in bn-batch negatives, yielding strong improvements in both retrieval and general embedding tasks. Qwen3 Em- bedding model series (Zhang et al., 2025b) take a three-step pipeline that first performs large-scale weakly supervised pre-training on synthetic data, then finetunes with high-quality supervised and selected synthetic datasets, and finally applies model merging (Li et al., 2024b) to boost robust- ness and generalization. For instruction tuning, Inbedder (Peng et al., 2024) treats instructions as questions and derives embeddings from the expected answers rather than concatenating instructions with inputs. E5-Mistral (Cheng et al., 2023) employed an asymmetric instruction strategy that ini- tially applies instructions only to the query side which has been proven efficient in retrieval tasks by numerous subsequent works.
cntructiiontuningg approach that first trains on re-
Adaptive Representations Learning for Text Embedding Compression. Early work for text embedding sparsity focuses on directly mapping text to sparse vectors use loken-wise late interac- tion, with some recent work carried out following this approach. The SPLADE series (Formal et al., 2021b) (Formal et al., 2021a) introduced a BERT-based model for learning sparse, interpretable text representations via explicit sparsity regularization and log-saturation, enabling efficient inverted in- dex retrieval. PromptReps (Zhuang et al., 2024a) prompts LLM generate a single-wond word represen- tation of each text and sparsify the logits of that prediction by filtering to document tokens while applying ReLU and log-saturation. Mirzadeh et al. (2023) proposes a "relufication" sparsity strategy where non-ReLU activations in pretrained LLMs are replaced (and sometimes supplemented) with ReLU layers to induce high activation sparsity. Nguyen et al. (2024) uses probabilistic term ex- pansion control to transform dense text embeddings in multimodal retrieval into sparse, vocabulary- aligned vectors while preserving effectiveness. Wang et al. (2024a) introduces Q-Sparse, a method that achieves full activation sparsity in large language models by applying TopK sparsification to linear projections and using the traighh-througe estimator for training Moreover, You et al. (2025) reveal that spurious memorization where small set of neurons overfit to non-causal patterns can lead to biased representations and degraded generalization. Understanding and mitigating such effects provides complementary insight to sparsity- embedding learning.
e
18

---

Published as a conference paper at ICLR 2026
Matryoshka Representation Learning (Kusupati et al., 2022) (MRL) pioneers text embedding com- pression in recent years via training with truncated dimensions. Proposed in 2022, MRL demon- strates adaptive-length embeddings for large-scale retrieval and classification including NLP set- tings, leading to various works that focus on adapting MRI. to embedding model setrings. Li et al. (2024c) extends MRI. by introducing second scalability dimension, enabling embeddings to be truncated along both model layers and embedding sizes. Zhuang et al. (2024b) combines fixed-size sub-model finetuning with masked autoencoder pre-training introduces new structured training strategy for 2D Matryoshka embeddings. Yoon et al. (2024) transforms arbitrary embeddings gen- erated by embedding models or APIs into embeddings with Matryoshka properties in both unsu- pervised and supervised setups. Various open-sourced embedding models, such as Jina-v3 (Sturua et al., 2024) and Qwen3-Embedding series (Zhang et al., 2025b); and commercial APIs, such as Gemini (Lee et al., 2025), have supported MRI. dimension truncation.
Another promising direction for text embedding sparsification Sparse Autoencoder, which grows from sparse coding/dictionary learning to tackle polysemanticity by disentangling features, are now scaled to frontier LLMs and widely used for mechanistic interpretability (Cunningham et al., 2023) (Yan et al., 2025). Rajamanoharan al. (2024) introduces Gated SAE that solves the systematic underestimation of feature activations caused by L1 penalty and requires half as many firing features to achieve comparable reconstruction fidelity. Gao et al. (2024) utilizes k-sparse autoencoders replacement of traditional L.1-based sparsity, preventing activation shrinkage, reducing dead latents, and yielding cleaner scaling laws with more interpretable and effective features. Lan al. (2024) employs SAE to discover monosemantic features within language models, revealing high degree of similarity and potential universality in these learned sparse feature spaces across diverse LLM architectures. Duan et al. (2024a;b) explore sparse representations via principled Bayesian gamma priors in deep generative models. Wen et al. (2025) leverages contrastive objectives for preserving semantic quality, achieving results close to those of backbone embeddings in the downstream tasks when only 32 neurons are activated.
Orthogonal Efficiency Techniques. Quantization and hashing compress embedding values rather than reducing active dimensions. Product quantization and its optimized variants approximate dis- tances with compact codes (Jégou al., 2011; Ge et al., 2013), while binary hashing methods such as Spectral Hashing and ITQ yield extremely small codes with Hamming distance search (Weiss et al., 2008; Gong et al., 2011), 1). Model-side low-bit quantization of Transformer encoders further reduces memory and latency (Shen et al., 2019). These techniques are orthogonal and can be combined with sparse embeddings (c.g., PQ over nonzero coordinates or low-bit storage of sparse values), jointly improving storage and retrieval throughput.
coamming-disttnce
B TASKS
We cover types of tasks in this paper: classification, clustering, retrieval, pair classification, se- mantic textual similarity and reranking. They are taken from MTEB (Muennighoff et al., 2022) and include the vast majority of the tasks the MTEB English Leaderboard, as well as some multilin- gual tasks.
Classification: Classification involves 10 tasks, which are divided into general tasks and specialized tasks. General tasks include AmazonMassiveDomaann (FitZGerald al., 2022), AmazonMassiveSeenara, (FitzGerald et al., 2022) MTOPInteet (Li et al., 2020), and MTOPDomain (Li et al., 2020) for multilingual natural language understanding, IMDb (Maas et al., 20,1), TweetSentimendExtaaciion (Maggie et al., 2020) and Emotion (Sar- avia et al., 2018) for sentiment analysis. Specialized tasks include AmazonCounterfac- tual OO'Neill et al., 2021) for counterfactual detection in product reviews, ToxicConver- sation50k (ejadams et al., 2019) for detection of toxic speech and prejudice, Banking77 Casanueva et al. (2020) for financial intent recognition.
Clustering: Clustering involves 8 tasks. These tasks are BiorxivClustering222 Biorxiv- ClusteringS2S MeerxxivClusteringPea MedrxivClusteringS222 and Arxiv uxxiClusteringS25
'httss//api.bboorivvee. birrxiv.og//
https://apiimmexxiv.org
19

---

Published as a conference paper at ICLR 2026
(University, 2025) for research field clustering, TwentyNewsGroups for news topics iden- tification, StackExchangeP2P and StackExchange (Geigle et al., 2021) for clustering of titles from 121 stackexchanges.
Retrieval involves 8 tasks. These tasks are Arguana (Wachsmuth et al., 2018) and NFCorpus (Boteva et al., 2016) for medical information retrieval, CQADupstackGam- ing (Hoogeveen et al., 2015) and CQADupstack (Hoogeveen et al., 2015) for web community retrieval. SernatcFVERRaruNNNe change retrieval, FiQA2018 (Maia et al., 2018) for financial retrieval, SCIDOCS (Cohan et al., 2020b) and SciFact (Wadden et al., 2020b) for academic retrieval.
Retrieval:
(Wadden et al., 2020a) for climate-
Semantic Textual Similarity (STS): STS includes 10 tasks. These tasks include general- domain semantic comprehension tasks STS12 (Agirre et al., 2012), STS13 (Agirre et al., 2013), STS14 (Bandhakavi et al., 2).4), STSI5 (Biçici, 2015), STS16 (Nakov al., 2019), STSBenchmark (May, 2021), SICK-R (Marelli et al., 2014), STS17 (Cer et al., 2017) and STS22 (Chen et al., 2022) and medical domain semantic comprehension task BIOSSES (Sogancroglu et al., 2017).
Pair Classification: Pair Classification includes two tasks, with SprintDuplicateeuussioos (Shah al., 2018) for programming domain and TwitterURLCorpus (Lan al., 2017) for social media (Tweet) domain.
Reranking: Reranking includes 3 tasks, which are Ask LasuuuunuDuuQuustioons (Wang et al., 2021) and www.OOveeeloowwDuuusttttt (Liu et al., 2018) for reranking of related program- ming blogs and SciDocsRR (Cohan et al., 2020a) for reranking of scientific papers.
C EXPERIMENT DETAILS ON TEXT REPRESENTATIONS
C.I EVALUATION METRICS
We adopt the standardized evaluation protocols established by the Massive Text Embedding Bench- mark (MTEB) (Muennighoff et al., 2022). Specifically:
For classification tasks, train logistic regression classifier the embedded training split and report its accuracy on the test split.
For clustering tasks, we apply mini-batels k-means to the embedded training data and eval- uate performance on the test split using the V-measure.
For retrieval tasks, we compute normalized Discounted Cumulative Gain at rank 10 (nDCG@ 10), where document-query relevance scores are derived from cosine similarity between embeddings.
For semantic textual similarity (STS) tasks, we measure the Spearman rank correlation coefficient between the ground-truth similarity scores and the cosine similarities of the corresponding sentence embeddings.
For pair classification tasks, we evaluate using cosine-similarity-aasse average precision, with decision thresholds determined by optimizing over similarity scores the validation set.
For reranking tasks, we report Mean Average Precision (MAP), again using cosine similar- ity as the scoring function.
To assess retrieval efficiency, we construct a unified query set by aggregating all queries from the aforementioned retrieval and reranking datasets, and corresponding document database by merging their respective corpora. All efficiency metrics are computed over this consolidated benchmark setup.
C.2 EXPERIMENT SETUP
We select e5-Mistral-7B (Wang et al., 2023) and our backbone embedding models and evaluate their performance across six task categories defined
(Zhang et al., 2025b) as
hhtpss//sikkit-llmm..rg. wtw.../g...///aauuss///weeynn newsgroups.html
20

---

Published as a conference paper at ICLR 2026
in the Massive Text Embedding Benchmark (MTEB) (Muennighoff et al., 2022). For each task cate- gory, we restrict our evaluation to English-language datasets and the English subsets of multilingual datasets included in the MTEB leaderboard" This yields total of 10 classification (Classif.), 8 clustering (Clust.), retrieval (Retrieval), 10 semantic textual similarity (STS), 2 pair classification (PairClassifi.) and 3 reranking (Rerank.) datasets in our experimental suite.
evaluation pipeline: for each task type, we aggregate the training splits of all constituent datasets form unified training set, while preserving the original test split of each individual dataset for performance evaluation. This pipeline is applied consistently across all six task types using the aforementioned datasets.
We adopt a
All experiments are conducted on a server equipped with 8 NVIDIA A100-SXM4-40GB GPUs, except for backbone finetuning, which performed on separate server with 8 H20-NVLink GPUs (96 GB memory each).
C.3 IMPLEMENTATION DETAILS
To ensure fair comparison between MRL and CSRv2 on the MTEB benchmark, particularly with respect to domain alignment, we select backbone model that is not natively supported by MRL: c5-Mistral-7B (Wang et al., 2023). We then finetune this model on a carefully curated collection of multi-domain datasets. Specifically, the training data is drawn from three complementary sources: (i) datasets included in the MTEB benchmark (Muennighoff et al., 2022), (ii) the embedding training data collection curated by the Sentence Transformers team" and (iii) a suite of public retrieval datasets introduced in Zhang al. (2025a). During preprocessing, appear across multiple collections. Subsequently, following the natural supervision strategy outlined in Section 3.2, sample up to 20,000 sentence pairs per dataset, resulting in consolidated training corpus of approximately one million examples.
first deduplicate datasets that
We finetune c5-Mistral-7BB on this corpus using batch size of 2048, which is scale commonly adopted by existing MRL-compatible models. Full details of the hyperparameter configuration are model (Zhang al., 2025b) already
provided in Table 5. In contrast, the Owen3-Embeddiee incorporates native MRL support; thus, no additional finetuning is required for this backbone.
Table 5: Implementation details on MRL finetuning.
Backhone
Batch Size LoRA r LoRA a
Ir
epoch
warmup weight decay
MRL dim
MRL Cm
c5-Mistral-7B
2048
8
16 2e-5 10
1000
0.1
1,2,4...4096 (1, 1....,.1
For backbone finetuning, we adopt a methodology closely aligned with that of MRL (Kusupati et al., 2022), as detailed in Section 3.3. Specifically, we apply Topk operator with varying values of ie to the backbone's 's output embedding and finetune the model using LoRA (Hu et al., 2022). We restrict k powers of two (i.e., k {2' }), and assign uniform weight of 1 to each k-dimensional sub- embedding during training. The finetuning objective is the InfoNCE loss (Oord et al., 2018), and the selection of hyperparameters is provided in Table 6.
Table 6: Implementation details on backhone finetuning in text representation.
Backbone
Batch Size LoRA r LoRA a
Ir epoch warmup weight decay
Topk
c5-Mistral-7B
256
8
16
2c-5
10
1000
0.1
{1, 2, .... 2048, 4096)
Qwen3-Embeddiin-45
256
8
16 2e-5
10
1000
0.1
{1. 2, .... 2048, 2560)
In the training of CSRv2, we adopt the tied encoder- decoder architecture as proposed in CSR (Wen al., 2025). For the k-annealing schedule, the initial sparsity level kimi is set to 64 the current number of activated dimensions l is less than 64; otherwise set himi 4k. Positive and negative
httpps huttps://hugginggacce.oo spaces beracess/teb/aeeee wttee///aeebbbardd 'https://huget http:/::///hgggggf////c/ co/datasets //////ttteeeteeeeeeeeeeeeeeeeeee tetteeeneteeeeearrrreeee embeddi tereeddinngttaaiinn--deee
21

---

Published as a conference paper at ICLR 2026
MRL
CSR anneal
CSRV2-liner
MRL
CSR anneal
CSRV2-lnnar
MRL
CSR aannall
CSRV2-linear
CSp+sseeevvss
CSt+ssperviie
CSR+supervis
50
75
Perfomancc
Performance
Perforancc
40
55-
35
Embedding
30
Full Embedding
20
Embedding
32
16
32
Active
(a) Classification
(b) Clustering
(c) Retrieval
CSR+ aeneal
CSRv2-linen
MRL
CSR+ cSi+annnal
CSRv2 linear
CSR+anneal
CSRv2-lines
CSR
CSit+ Supersupervise vise
CSRv2
C5R
cuprssueeriiee
CSRv2
CSR
CSit supervie
CSRv2
Performenco
Performenca
Performenc
70
80
55
45
55
Full Embedding
Full Embedding ng
Full Embeddiing
32
16
32
Active Dim
Active Dim
Active Dim
(d) Semantic Textual Similarity
(c) Pair Classification
(f) Reranking
Figure 6: Task-type-specific ablation on varying components with e5-Mistral-7BB as backbone.
samples for supervision are constructed in accordance with the rule detailed in Section 3.2. We employ Adam as the optimizer and selection of other hyperparameters is in Table 7.
Table 7: Implementation details on CSRv2 training in text representation.
Backbone
d
Ir
epoch Batch size
kaux
3 7
K
weight decay
cS-Mistral-7B
4096
16384
4c-5
128
1024
0.1
2,4...,4096
lc-4
Swen33-Fmhedding-44 2560
10240
2c-5
128
256
1024
0.1
2,4,16,64,4096
lc-4
C.4 MORE DETAILED EXPERIMENT RESULTS ON TEXT REPRESENTATIONS
Building upon the e5-Mistral-7B backbone, we further validate the efficacy of CSRv2 through an extensive performance comparison across broad spectrum of active dimensions (ranging from to 4096 the full embedding dimensionality of the backbone) and through comprehensive ablation studies.
As shown in Table 8, we report lask-ty results for MRL, CSR. and CSRv2 across six distinct task categories.
Moreover, we systematically evaluate the impact of different component combinations on each task type and observe that the performance gains contributed by each individual component remain con- sistent across diverse domains. Detailed results are provided in Figure 6.
C.5 RETRIEVAL EVALUATION COMPARISON WITH SPLADE-BASER MODELS
Table demonstrates CSRv2's performance comparison with SPLADEv3 model. select e5- Mistral-7B (Wang et al., 2023) backbone, whose performance MTEB retrieval tasks is on par with SPLADEv3. Note that the Active Dim X Y for SPLADEv3 means that queries have X active dimensions and documents have Y active dimensions, which is common setting in LSR series model's 's evaluation. Results show CSRv2 is more suitable for ultra-sparse text representation generation in extreme application cases.
22

---

Published as a conference paper at ICLR 2026
Table 8: Performance in lask-type-specifte experiments across all dimensions.
Active
Classifi. ACC
Clust. V-measure
Retrieval nDCGe 10
STS Spearman
PairClassif. AP
Rerank. MAP
Avg.
Method
Dim
4096
c5-Mistral-7B
80.67
51.55
49.35
84.11
91.77
69.52
69.99
MRI.
34.84
26.13
16.63
52.14
26.67
40.30
33.81
2
CSR
52.50
33.20
16.14
62.93
52.95
46.77
44.17
CSRv2
71.59
41.29
37.48
73.82
62.46
60.91
58.34
MRL
43.84
33.14
24.55
56.51
37.36
44.72
40.83
CSR
67.22
39.25
23.54
70.13
74.55
48.57
52.94
CSRv2
74.26
43.85
39.04
75.69
74.90
62.93
61.01
MRI
CSR
73.77
41.68
31.61
74.00
77.95
55.32
58.19
CSRv2
76.60
46.49
42.67
78.67
79.29
63.15
63.76
MRL
54.64
68.18
61.38
CSRv2
77.79
47.97
79.94
86.50
65.22
MRI
59.37
45.31
39.68
72.33
68.80
58.86
56.37
32
CSR
77.11
43,21
85.48
64.59
C5Rv2
66.70
MRL
66.58
47.76
44.11
77.46
78.46
62.72
61.47
64
CSR
79.50
48.36
45.22
87.29
64.86
66.68
CSRv2
79.98
49.53
47.92
82.90
90.46
67.34
68.08
MRL
74.78
49.12
46.08
81.95
84.66
65.48
65.72
128
CSR
79.70
49.32
46.68
82.39
87.83
65.48
67.34
C5Rv2
80.14
49.83
48.27
83.12
90.75
67.44
68.32
MRL
76.52
49.21
46.64
82.07
85.25
66.04
66.37
256
CSR
80.00
49.64
47.47
82,
88.48
65 98
67.76
CSRv2
80.24
50.24
48.42
83.35
90.89
67.85
68.55
MRL
78.42
49.68
47.19
82.53
87.87
66.45
67,30
512
CSR
80.12
49.86
47.92
82.97
88.93
66.51
68.04
CSRv2
80.31
50.65
48.64
83.50
91.14
68.30
68.77
MRL
78.92
49.96
47.58
82.85
88.65
67.36
67.74
1024
CSR
80.26
50,36
48. 16
83.28
89.67
67.28
68.41
CSRv2
80.50
50.88
48.82
83.65
91.41
68.64
68.97
MRL
79.54
50.49
48.35
89.40
68.44
2048
CSR
80.38
50.79
48.62
83.63
90.42
68.36
68.81
CSRv2
80.51
51.27
48.93
83.88
91.63
68.87
69.
MRT.
90.99
CSRv2
80.49
$1.34
49.16
83.94
91.70
69.18
69.25
D EXPERIMENT DETAILS ON VISUAL REPRESENTATIONS
D.1 EVALUATION METRICS
Following the methodology established by Kusupati et al. (2022), we adopt 1-nearest neighbor (1- NN) accuracy as the primary metric for evaluating visual representations. This metric computed using FAISS (Jacob al., 2018) with exact L2 distance search. In contrast to classification accuracy, which depends on the specific architecture and training procedure of a downstream classifier, 1-NN accuracy provides direct assessment of whether semantically similar instances embedded in close proximity within the representation space. Consequently, it serves as a more model-agnostic and training-free probe of intrinsic representation quality.
23

---

Published as a conference paper at ICLR 2026
Table 9: CSRv2's Performance and Relative Retrieval Efficiency Comparison with SPLADEv3.
Active Dim
Retrieval Time
Method
Arguam
CQAGaming
CQAUnix
CF-HNN
Figa
Nicorpus
Scidogs
Seifact
Avg.
4096
c5-Mistral-7B
306.46x
62.73
64.13
47.99
30.71
56.93
39.67
18.09
74.53
49.35
40-400
SPLADES3
27.25 x
35.95
54.31
34.51
39.01
49.28
59.61
32.52
72.80
47.37
16-16
SPLADEv3
x
32.54
35,00
52.78
30.77
57.22
CSRv2
3.51
59.78
26.92
52.07
33.18
15.56
8-8
2.84
39.74
21.27
31,86
28.88
47.59
26.14
45.09
32.81
4-4
SPLADE>3
1,78
14.71
28.59
9.62
22.37
19.25
28.71
17.43
28.18
21.11
2-2
14.53
9.76
16.28
7.59
18.73
11.34
CSRv2
45.27
60.83
D.2 IMPLEMENTATION DETAILS
For fair comparison, we select the pretrained ResNet-50 weights, as noted in FF2048 in the MRL (Kusupati et al., 2022). Image preprocessing follows the identical pipeline employed in (Leclerc et al., 2023), (Kusupati et al., 2022) and (Wen et al., 2025). We utilize a tied encoder-decoder structure to build the CSRv2 framework and the implementation is based Wen et al. (2025). All experiments are conducted on a server with 8 NVIDIA 1100000M44000 For backbone (FF2048) finetuning. the selection of hyperparameters is in Table 10.
Table 10: Implementation details on FF2048 finetuning in visual representation.
Backbone Batch Size
Ir
epoch warmup Optimizer weight decay
Topk
FF2048
256
Se-6
10
1000
Adam
0.1
2.. 2048)
For CSRv2 training. we adopt the same settings as CSR (Wen et al., 2025). In the k-annealing schedule, we initialize kinit = 64 if the target activated dimension & is less than 64, otherwise we set kimit = 4k: For supervision, images belonging to the same semantic class are treated as positive pairs, while all others are considered negative samples. Adam is employed the training optimizer and selection other hyperparameters is Table 11.
Table 11: Implementation details on CSRv2 training in visual representation.
Backbone
epoch Batch size k
7
K
weight decay
FF2048 2048 8192 4e-5
10
4096
512 1/32 0.1 2,4....2048
le-4
D.3 I-NN CLASSIFICATION RESULTS
I-NN classification accuracy results on ImageNet- 1k are shown in Table 12.
E EXPERIMENT DETAILS ON GRAPHRAG EVALUATION
E. EVALUATION METRICS
We follow the evaluation design proposed in Xiang al. (2025). For retrieval, Context Relevance and Evidence Recall are adopted. For generation, Answer Accuracy, Faithfulness, Evidence Cover- age and ROUGE-L are adopted. Detailed explanation on each metric is follows:
Context Relevance<Relevaneee assesses how well the aggregate retrieved context satis- fies query's 's semantic requirements. Higher values indicate greater fidelity between the re- trieved material and the underlying informational intent of the user. Specifically, Context
24

---

Published as a conference paper at ICLR 2026
Table 12: I-NN accuracy of different methods on ImageNet- Ik classification.
Active Dim.
2
4
8
16
32
64
128
256
512
1024
2048
Full Rep.
71.19
MRL
47.81
55.65
62.19
67.91
69.46
70.17
70.52
70.62
70.82
70.89
70.97
CSR
61.05
65.33
67.78
69.17
70.15
70.94
70.99
71.31
71.29
71.30
71.18
CSRv2-linear
65.78
67.29
68.42
69.71
70.39
71.01
71.11
71.24
71.23
71.19
71.19
CSRv2
67.63
69.84
69.29
70.06
70.44
71.05
71.13
71.25
71.27
71.33
71.25
Relevance can be calculated as:
Relevance 1 Σ R(c, ER(c, 2,e) CEC
where is the of retrieved contents, Q the query, is the of evidence, and operator determines whether each context e relevant to the query Q and the evidence e.
Evidence Recall(Recal)) quantifies the completeness of evidence retrieval by measuring the proportion of critical reference claims that successfully covered by the system's output. It is defined as:
Recall = Σ (S(c,C) cER
(10)
|R|
where R is the set of reference claims, S is the operator to decide whether claim is supported by the retrieved content and the indicator function.
Answer Accuracy(ACC) comprehensively assesses answer quality through a combination of semantic alignment and factual precision. To be specific,
ACC (FC SS)
where FC qualifies generation correctness and SS = cos(f,, c,) calculates semantic simi- larity.
calcuates text similarity with n-gram overlap between generated and reference answers, capturing both syntactic and semantic alignment (Lin, 2004).
FaithfulnesssFFS) explicitly targets hallucination risks by quantifying the proportion of gen- erated claims that are grounded in the retrieved evidence, thereby serving as a direct mea- sure of factual consistency between the system's output and its supporting context. It is measured as follows:
Hc E C))
FS
[A]
where denotes the set of atomic claims the proposed response, C' the retrieved context and S(c, C)) denotes boolean function indicating whether claim is supported by C.
Evidence Coverage(Cov) quantifies the extent to which the generated response incorpo- rates all critical evidentiary elements required to construct comprehensive and factually complete answer. The formal computation as follows:
HE (IM G)H
Cov
E
where E is the set evidence, the generated answer and M(e, G) is boolean function indicating whether evidence appears in the generation G.
E.2 IMPLEMENTATION DETAILS
Our evaluation covers two domains proposed in Xiang et al. (2025): Medical and Novel. For fair comparison, select (wwnn3-Embedding-te Zhang et al. (2025b) the baseline embedding
25

---

Published as a conference paper at ICLR 2026
Table 13: CSRv2's Performance in GraphRAG-basee Retrieval. In GraphRAG-based retrieval is selected as backbone and two sparsity levels: 32 and are
evaluation, Owen3-Embedddig selected for comparison. No data in benchmark used in training for zero-shot evaluation.
Embedding
Active
Fact Retrieval
Complex Reasoning
Contextual Summarize
Creative Generation
Avg.
Recallf
Relevancef
Recallf
Relevancef
Recallf
Relevancef
Medleal
Qwen3-4B
2560
75.43
45.83
82,98
40.18
81.2
48.79
81.14
28.77
61.29
48.30
15.05
19.38
84.13
CSRv2-linear
67.24
38.73
76.35
36.49
72.8
43.53
82.87
24.62
CSRv2
71.75
40.81
78.74
38.48
79.63
46.03
84.55
26.02
58.25
8:42
82.64
CSRv2-linear
62.47
31.56
39.53
81.92
CSRv2
68.17
36.98
69.35
35.64
85.52
14.16
Novel
Qwen3-4B
2560
81.29
45.26
82,15
$1.39
83.41
49.03
80.29
36.94
63.72
MRT.
68.47
27.91
72.80
33.48
76.42
33.22
78.02
28.36
52.34
CSRv2-linear
32
75.23
36.62
76.47
39.31
81.75
39.07
69.17
30.18
55.98
CSRv2
79.08
41.40
78.88
43.85
83.37
44.82
74.10
29.10
59.33
MRL
63.20
19.39
69.71
22.58
22.44
80.82
20.52
CSRv2 linear
66.72
29.92
71.81
32.83
68.48
30.16
78.30
19.09
49.79
CSRv2
75.05
36.46
77.16
44.63
77.65
40.33
80.92
25.87
57.26
model and GPT-4o-mini for graph construction, answer generation and evaluation. Fast-graphrag (CircleMind-AI, 2025) is chosen the GraphRAG framework, with minor change following Xiang et al. (2025) for Hugging Face Embedding support. All hyperparameters are set according to the settings Xiang et al. (2025).
E.3 EVALUATION RESULTS
Table 13 and 14 demonstrate CSRv2's zero-shot capability: In retrieval performance evaluation, at the same level of dimension, CSRv2 achieves performance improvements of over 15% and 7% in medical and novel domains respectively compared to MRL, while in generation accuracy evaluation, CSRy2-basee systems achieve average improvements of over 10% and 3% in medical and novel domains.
F ADDITIONAL QUALITATIVE ANALYSIS
F.1 CASE STUDY OF FEATURE COMPARISON BETWEEN DIFFERENT METHODS
To facilitate more intuitive comparison the feature distributions induced by different represen- lation learning methods and to elucidate the factors underlying CSRv2's substantial performance gains over both CSR and MRL, we extract two-dimensional embeddings from the IMDb dataset (Maas et al., 201 1). Specifically, we obtain dense representations from MRL and ultra-sparse repre- sentations from CSR and CSRv2 under sparsity budget of k 2. The resulting embeddings are visualized via t-SNE in Figure 7, with positive and negative movie reviews rendered in green and purple respectively.
We observe that the MRL embedding demonstrates clear separation between the majority of pos- itive and negative reviews, reflecting its ability to capture dominant sentiment polarities. However, it exhibits notable limitations in handling compositional or contrastive sentiment expressions. For instance, in sentences such as "Although the plot of this movie is slow, the actors performed well and I really appreciated this movie", conflicting affective signals lead to ambiguous representations that cluster near the decision boundary. This suggests that dense, holistic representations may struggle disentangle nuanced mixed sentiment structures.
26

---

Published as a conference paper at ICLR 2026
Table 14: CSRv2's Performance GraphRAG-based Generation. In GraphRAG-based gener- ation evaluation, are selected for comparison. No data in benchmark is used in training for zero-shot evaluation.
is selected as backbone and two sparsity levels: 32 and 8
Embedding
Active
Fact Retrieval
Complex Reasoning
Contextual Summarize
Creative Generation
Avg.
Model
ACC1
ROUGE-LT
ACCT
ROUGE-LT
ACCT
Covt
ACCT
Medkeal
Qwen3-4B
2560
61.33
29.65
69.63
21.67
72.39
46.19
69.23
32.04
31.7
48.87
16.69
55.17
30.65
38.51
CSRv2-Linear
52.82
25.03
61.36
19.02
64.14
39.73
66.97
28.45
43.65
CSRv2
60.69
29.27
68.60
20.76
71,18
45.74
68.44
31.58
36.68
48.10
MRL
20.04
30.65
CSRv2-linear
16.49
28.
41.46
CSRv2
58.09
19.44
68.83
41.69
29.91
45.90
Novel
Qwen3-4P 4-B
2560
57.02
31,76
54.63
19.67
70.62
47.85
59.70
44.51
38.53
47.14
MRI.
45.72
25.65
45.06
18.23
65.85
43.78
57.38
31.28
36.82
41.09
CSRv2-linear
51.26
28.49
49.02
18:68
67.03
44.42
57.71
35.18
35.79
43.06
CSRv2
54.69
31.63
51.47
19.49
68.19
45.67
57.87
37.41
35.89
44.70
MRL
42.23
16.25
37.71
54.56
29.39
34.45
37.36
CSRv2 lineur
48.78
25.13
46.75
17.03
63.84
41.12
57.23
34.08
35.96
41.10
CSRv2
52.94
29.25
50.93
18.92
67.54
44.80
56.45
36.86
34.49
43.58
50
50
50
0
-50
-50
-50
50
(a) MRL.
(b) CSR
(c) CSRv2
Figure 7: t-SNE visualization of 2-dimensional features IMDb generated by MRL, CSR and CSRv2 with e5-Mistral-7BB backbone. The AP scores of MRL, CSR and CSRv2 are respectively 89.34%, 92.75% and 94.62%.
In contrast, CSR adopts compositional strategy by decomposing sentiment into fine-grained mantic primitives. This yields a highly fragmented latent space in the reduced two-dimensional projection, characterized by numerous small, localized clusters. While many of these clusters cor- respond to lexically precise phrases (e.g., "I really like" or "fail to"), the model faces ambiguity when encountering polysemous terms, such "strong" which appears in both positive and negative contexts. Consequently, representations involving such terms are scattered across disparate clusters, undermining feature consistency and increasing the risk of misclassification due to unstable feature binding-
CSRv2 mitigates this issue by jointly optimizing sparse feature learning with supervised signals that promote the emergence of both high-level sentiment abstractions (e.g., overall positivity or negativity) and the fine-grained semantic patterns preserved CSR. Crucially, we observe that individual neurons in CSRv2 consistently activate in response to emotionally salient yet lexically general terms, such as "awful" and "fantastic", which exhibit strong sentiment polarity while re- taining broad contextual applicability. This hybrid inductive bias enables CSRv2 to achieve more robust and interpretable separation of sentiment classes, effectively balancing semantic specificity with generalization capacity.
27

---

Published as a conference paper at ICLR 2026
F.2 ACTO-INTERPRETABILIT STUDY ON CSRv2 NEURONS UNDER DIFFERENT COMPRESSION SETTINGS
We analyze the semantic roles of individual neurons in the CSRv2 latent space under two sparsity regimes 64 (moderately sparse) and 2 (extremely sparse) using the IMDb dataset (Maas et al., 2011). I). For each neuron, we compute its activation values across all input sentences and iden- tify the top- 10 paragraphs that elicit the strongest responses. To interpret the semantic and affective patterns encoded by each neuron, we leverage @wen-7B-Chan (Bai et al., 2023) to generate concise summaries of the linguistic and emotional characteristics common to these maximally activating sentences.
Our analysis reveals that under the = 64 regime, while many neurons encode semantically coher- and sentiment-relevant concepts, a non-negligible subset predominantly activates in response to high-frequenne yet functionally neutral lexical items, such as "T" or "today", which carry little to no emotional polarity. In contrast, under extreme sparsity (k = 2), neuron activations exhibit markedly increased specialization: each active dimension consistently aligns with a distinct sentiment pole, either positive or negative. This indicates that ultra sparsity constraints exert strong pressure on the model to prioritize emotionally salient, task-relevant signals, thereby yielding representations that are not only more polarized but also more interpretable terms of their affective semantics.
G EMPIRICAL ANALYSIS
G.1 EFFICIENCY ANALYSIS DETAILS
Our efficiency analysis focuses on retrieval and storage, where computational cost meaningfully differs across methods. Even though end-to-end latency, encoder latency, and index construction cost could be relevant applied to downstream tasks, pre-caching is inevitable. That is, the corpus is encoded once, and embeddings are stored for repeated use. Typical examples include (1) RAG systems, where docu- ments change infrequently and their embeddings serve millions of queries, and (2) online services such as recommendation, where real-time encoding of large-scale text is infeasible. Therefore, en- coder and index construction costs are amortized and do not dominate real-world latency. To ensure fair comparison, we keep the encoder and indexing pipeline identical for all baselines (MRL, CSR, and CSRv2), so that any efficiency or performance variation arises strictly from the embedding representations.
fully online setting, in most practical scenarios where embeddings are
With Qwen3 Embeddiinbd (Zhang al., 2025b) as the backbone, we record encoding time on a 1M corpus sampled from MTEB retrieval and reranking datasets. Table 15 shows that CSRv2 introduces only negligible overhead compared to MRL (0.012% extra time, ~9..172 in total), which is insignificant relative to the hours required for large-scale corpus encoding.
Table 15: Encoding time comparison on a 1M corpus.
Method MRL
Encoding Time (s)
159854.091
CSR
159876.478
CSRv2
159873.263
In contrast, retrieval and storage costs differ dramatically. Under fixed encoder and index type, the dominant factor in retrieval time is effective embedding dimensionality (d for dense baselines A for CSRv2). As shown in Table 16, ultra-sparse vectors yield up to 7x faster retrieval than dense MRL and up x00 speedup times are averaged over 2000 rounds (batch size 512), excluding 100 warm-up iterations.
the uncompressed backbone
1M-scale corpus. Retrieval
These results reinforce main practical claim: CSRv2 offers substantial gains in the components that dominate real-world latency (retrieval throughput and embedding storage), while incurring neg- ligible overhead the encoder side.
28

---

Published as a conference paper at ICLR 2026
Table 16: Retrieval time per query under different active dimensions (ms).
Method
2
8
16
64
4096
MRL
1.402
1.428
1.571
1.748
3.972
68.522
CSRv2
0.227
0.370
0.633
0.797
3.217
45.722
D2
Dim-15
$1.0
Accuracy (95)
61.0
(96)
60.5
60.0
47.0
47.0
66.5
esparettiel
K-schedule
Length
Initialization
(a) k-schedulc sensitivity analysis
(b) length sensitivity analysis
(c) initialization sensitivity analysis
Figure 8: K-annealing sensitivity analysis. (Left): Sensitivity on three k-schedule strategies: lin- ear, exponential and cosine. (Middle): Sensitivity four annealing lengths. (Right): Sensitivity on four different initializations.
G.2 K-ANNEALING SENSITIVITY ANALYSIS
We evaluate k-annealing strategy's 's sensitivity from three perspectives: k-schedule, length and ini- tialization. For k-schedule, we adopt three settings: linear, exponential and cosine. For length, we take four settings: 0.3, 0.5, 0.7 (in the main paper) and 0.9, For initialization, we take four settings: 32, 64, 128 and 256. Evaluation are done in two MTEB task types (classification and retrieval) and two active dimensions are selected for each experiment for generalization. Results in Figure demonstrate that different k-schedule results in relatively stable increase in performance improve- ment, while our selected settings: & initialized to 64, annealing to target sparsity level 70% step, and linear- annealing strategy achieves the best performance.
G.3 ANALYSIS ON UNBALANCED WEIGHTABLE SETTINGS
MRI. (Kusupati et al., 2022) has explored the impact of different weightage settings for smaller representation sizes. They find that on ImageNet (Deng et al., 2009), setting larger weight on di- mensions and 16 result in 3% improvement on d = 8, with minor performance degradation on larger dimensions. Therefore, we further conduct additional studies into the impact of imbalanced weighting.
We propose MRL-reweight, where we follow MRL's settings and set the following weights {5, 4, 3, 2, 1,....1 1} for dimensions 2, 4, 8, 16, 32, and on, up to 4096. As shown in Table 17, applying larger weights at earlier stages does, to some extent, improve the performance of MRL on now-dimensional scales. However, while MRL-reweight offers some improvements, the perfor- mance does not quite reach the level of CSRv2. We hypothesize that this discrepancy arises because sparse vectors, which are more comprehensive in capturing feature combinations, are more difficult to achieve with truncated representation (e.g., retaining only the first 8 values).
G.4 DISCUSSION ON MULTI-SCALE LOSS TERMS
The core idea behind k-annealing curriculum is that setting larger kimi promotes exploration and diverse neuron activations. This raises question: can annealing in the curriculum be replaced by multiple terms that cover the range from kini to kinal, rather than simply setting reconstruction loss as L(k) L(4))? We conduct quantitative discussion on cases look deeper into this problem:
Multi-Toph loss over diverse ks: Since covering all ks (e.g., g., 64 loss terms) along the nealing would be too computationally prohibitive, we now consider a diverse representative subset that use for evaluation: & [2, 4, 8, 16, 32, 64).
29

---

Published as a conference paper at ICLR 2026
Table 17: Performance comparison between standard MRL and MRL-reweiihh.
Method
128
256
512
1024
2048
4096
Classification
43.84
54.64
59.37
66.58
74.78
76.52
78.42
78.92
79.54
80.46
MeL-reweight
45.08
52.42
55.18
61.97
65.73
70.82
76.61
77.93
77.18
77.56
79.28
80.42
CSRv2
37.48
39.04
42.67
43.38
46.58
47.92
48.27
48.42
48.64
48.82
48.93
49.16
Retriewol
MRL
16.63
24.55
29.65
34.33
39.68
44.11
46.08
46
47.19
47.58
48.35
48.75
MRL-reweight
24.48
29.75
33.25
37.56
42.12
46.17
47.20
46.96
46.82
47.24
48.17
48.37
CSRv2
CSRv2 with start/end multi-TopK. In this variant, keep only the boundary losses (i.e.
Kmm = 64 and Kinal" = 2).
We compare the performance of different methods the MTEB classification and retrieval subsets and also report their corresponding training costs. Results in Table 18 and Table 19 demonstrate that focusing start and end Ks help bit addressing ultra- aitra-sparsity but there is still a large gap to our k-annealing. Moreover, better & coverage with diverse multi-Topp delivers larger gains, but still underperforms k-annealing, while introducing significant training overhead. Therefore, believe that k-annealing is more preferable than these static multi-Tophe loss variants. would be interesting to look deeper into their interplay in future work.
Table 18: Performance comparison with static multi-scale loss terms on MTEB Classification and Retrieval using 7B-Mistral-7B.
Method
Active Dim
Classification
Retrieval
CSR
52.50
16.14
CSRV2-inear-StuttEddTook
57.46
23.65
2
CSRv2-linear- www---iinn--iieessMMttiiiioo
61.75
24.18
CSRv2-linear-anncal
66.43
31.58
Table 19: Training time comparison with static multi-scale loss terms MTEB Classification and Retrieval using c5-Mistral-7B
Time (s)
Classification
Retrieval
CSRv1
271.32
638.77
CSRV2-llnner-StartEndToop
285.94
653.13
CSRV2-DiierreMulltiigg
501.18
1183.36
CSRv2(anneal)
274.15
642.65
H FURTHER DISCUSSIONS
H.1 EMERGENCE OF SUPERCLASS SEPARABILITY UNDER ULTRAHIGH SPARSITY
Past works (Fallah et al., 2020) have shown that sparse codes argued to induce disentangled, semantically meaningful features. However, a key open question remains: when the sparsity is extremely high (i.e., very few active dimensions), do such representations still preserve higher- level semantic structure (such as superclasses or domains), or do they collapse into trivial, instance- specific separations?
We conduct a superclass-Ieves analysis on two multi-intent classification datasets, Banking77 (Casanueva al., 2020) and MTTOPintent (Li al., 2020). For Banking77, following the semantic
30

---

Published as a conference paper at ICLR 2026
structure commonly adopted in prior work, group its 77 types of bank-related queries into semantically coherent superclasses (e.g. account&identity. card management. For MTOPIntent, we adopted the original intent taxonomy in the paper and grouped these intents into 11 domains (e.g. alarm, music). These groupings allow us to evaluate whether ultrahigh sparsity induces representa- tions that align with higher-level semantic partitions.
We evaluated MRL, CSR, and CSRv2 under the ultra-sparse regimes of k: = 2 and k: =4, which correspond to ultra-sparse setting. Evaluation is done in accordance with MTEB benchmark, where a logistic regression is trained on the training set and evaluated on the test set. Results in Table 20 demonstrate a consistent and notable trend: CSRv2 produces significantly more structured sparse representations than CSR and MRL, even under extremely low k. Superclass clusters become more linearly separable under CSRv2, indicating that ultrahigh sparsity does not degrade semantic ab- straction.
Table 20: Performance comparison for superclass classification with Qwen3-Embedding- as backbone.
Banking77 (original-class)
MTOP (original-class
Banking77
MTOP
Method
Active Dim
(super-class)
(super-class)
MRL
2
3.57
5.16
28.96
37.08
CSR
2
11.75
18.08
77.43
83.26
CSRv2
2
17.03
23.52
88.44
93.16
MRL
4
6.93
11.51
31.04
45.24
CSR
4
19.02
24.39
82.91
86.79
CSRv2
4
23.16
28.51
94.43
97.56
H.2 QUANTIZED COMPARISON FIXED MEMORY CoST
To provide more holistic view of the efficiency-accuracy trade-off landscape, further evaluate CSRv2 of different levels precision under fixed bit size in three MTEB task types: classification, clustering and retrieval. We take two fixed bit sizes (64 and 128), and adopt three quantization (FP32, BF16, binary) settings under each bit size.
Table 21: Performance comparison on CSRv2 and dense MRL in fixed memory cost.
Method
Bit Size
Quantization
Active Dim
Classification
Clustering
Retrieval
FP32
2
71.59
41.29
37.48
BF16
4
73.05
42,46
38.19
CSRv2
64
binary
64
74.12
44.53
40.28
PQ
64
62.39
33.16
21.76
binary
64
64.48
40.04
27.61
MRL
64
PQ
64
58.37
37.18
22.04
FP32
74,26
43.85
39.04
BF16
8
75.02
44.76
40.98
CSRv2
128
binary
128
76.30
45.01
42.26
PQ
128
70.15
38.97
30.17
binary
128
72.54
44.37
29.15
MRL
128
PQ
128
68.42
41.58
25.61
Results in Table 21 demonstrate that CSRv2 remains highly competitive across a wide range of quantization strategies. The findings further indicate that (1) increasing the number of active di-
31

---

Published as a conference paper at ICLR 2026
mensions is often more advantageous than raising numerical precision, and (2) extremely compact binary variants of CSRv2 yield the strongest accuracy- secracy-mmmmory trade-offs. Notably, -inarry also substantially outperforms binary-quamtized dense embeddings, implying that structured spar- sity provides greater representational expressiveness than uniform quantization when bit budgets extremely constrained. Together, these observations underscore that CSRv2 constitutes flex- ible and efficient embedding mechanism capable of adapting to both moderate- and ultra-low-bii compression regimes.
In addition, the consistently strong performance of binary and higher-dimensional BF16 variants suggests that richer or more varied activation patterns can effectively compensate for the seman- tic degradation introduced by low numerical precision. This highlights a promising direction for CSR-style representations: exploiting larger or more structured sparse activation patterns further enhance expressiveness under increasingly aggressive quantization settings.
We also evaluate PQ on both baseline and CSRv2 embeddings with code budget 64. We use standard PQ settings with 256 codewords per subspace and 8 subvectors, and for 128-bit codes into 16 sub- vectors. For CSRv2, we apply PQ quantization on TopK=256's embedding. However, PQ does not outperform Binary Quantization (BQ) in this context. We attribute this performance gap to funda- mental structural mismatch. Standard PQ partitions vectors into independent subspaces with equal bit- bit-budests, implicitly assuming uniform distribution of semantic information. However, MRL and CSRv2 embeddings are strictly hierarchical, concentrating " "core" semantics in the early/sparse dimensions. Consequently, PQ's uniform allocation strategy disrupts this hierarchy by inefficiently assigning equal capacity to both the highly informative prefix dimensions and the fine-grained tail dimensions, resulting in suboptimal quantization. Conversely, Binary Quantization preserves the sign information of high-value dimensions directly, offering superior compatibility with hierarchi- cal representations.
H.3 POTENTIAL APPLICATIONS OF CSRv2 VECTOR QUANTIZATION
Vector quantization (VQ) methods, including Product Quantization (PQ) (Jegou et al., 2010), Op- timized Product Quantization (Ge et al., 2013), and more recent anisotropic schemes such as AVQ (Guo et al., 2020), are central to real-world large-scale vector search systems where memory foot- print, latency, and hardware efficiency are critical. While our main work focuses on the role of ultra-sparse representations in improving retrieval quality and compute efficiency, it is worth noting that CSRv2 is highly compatible with these widely- quantization techniques.
CSRv2's ultra-sparse structure, activating only k E {2, 4, 8} dimensions out of a large latent space, naturally complements vector quantization methods such as PQ and AVQ. As only a few coordinates are non-zero, quantization can be applied exclusively to these active values (or their indices), en- abling a two-stage compression pipeline of sparsity quantization that substantially reduces both memory and lookup cost. Unlike dense embeddings (e.g.. MRL), where quantization error spreads across all dimensions, CSRv2 concentrates signal in handful of features, making the quantization process more pienal-preserving and aligned with anisotropic quantization principles. This compati- bility also facilitates integration into practical ANN systems (c.g., DiskANN (Jayaram Subramanya et al., 2019)) that already combine traph-basce search with PQ, suggesting that CSRv2 can fur- ther lower sestem-level memory while maintaining high recall. However, as discussed in Appendix H.2, while PQ presents an interesting avenue, it necessitates adaptation to function effectively with MRL/CSRv2's hierarchical representations, which we leave for future work.
H.4 LIMITATIONS OF CSRv2 ON THE MOST EXTREME SETTING
Although CSRv2 achieves strong performance under ultra-sparse regimes, suffers notable degra- dation in the most extreme sparsity setting (k- = 1), which reduces the representation to hard clustering assignment. As shown in Table 22, activating only a single neuron still yiclds a signifi- cant improvement over baseline methods; however, CSRv2's performance drops by 27.56% relative to the dense backbone model. This decline is more than twice as severe as the degradation observed when & = 2 (1 (1..5%))
Apart from those discussed in Section 5, another hypothesis for this sharp performance drop stems from the complete absence of feature combination when only one latent dimension is active. With
32

---

Published as a conference paper at ICLR 2026
single activation, the model loses the capacity to compose multiple semantic cues capability that has been shown to be critical for robust representation learning under sparsity constraints (Gao et al., 2024). Potential remedies for this limitation may involve architectural innovations that enable richer single-feature representations, such as nonlinearly compositional encoders (Li et al., 2025) or hierarchical autoencoders that preserve multi-level semantic structure under extreme sparsity levels. Exploring such directions remains promising avenue for future work, leaving room for future improvement.
Table 22: Performance comparison the most extreme setting k 1.
Active
Classifi. f
Clust. V-measure 1
Retrieval aDCG@
STS
PairClassifi AP
Rerank.
Avg.
Method
Dim
Spearman
4096
c5-Mistral-78
80.67
51,55
49.35
84.11
91.77
69.52
69.99
MRL
17.52
14.70
3.81
37.93
12.95
23.98
19.52
CSR
28.54
24.14
6.54
48.93
37.96
28.16
28.79
CSRv2 linear
39.61
28.78
19.82
51.80
43.85
37.08
36.63
CSRv2
52.43
31.48
24.73
54.46
47.09
42.34
42.43
H.S ANALYSIS ON ONE PROMISING SAE-VARIANT
Recently there have been a variant SAE called MRL-SAE (Bussmann al., 2025) that combines MRL's core idea into SAE training. Specifically, standard SAE whose single encoder-decoder is trained act as many nested autoencoders at once. The encoder produces one sparse feature vector, but the decoder is forced reconstruct the input from multiple truncations of that vector (e.g.. first 256, 512, 4096 latents), and the loss the sum of these reconstruction errors plus sparsity. This simple change to the training objective induces hierarchy where early latents encode broad, reusable features and later latents add increasingly fine- rreigra detail, all within one overcomplete dictionary.
We compare MRL-SAE's performance in classification, clustering and retrieval tasks in MTEB with vanilla SAE and CSR. Results in Table 23 shows that MRL-SAE underperforms vanilla SAE and CSR for embedding tasks and also suffer from severe degradation in sparse representation genera- tion.
Table 23: Performance comparison on MRI-SAE, vanilla SAE and CSR.
Method
Active Dim
Classification
Clustering
Retrieval
vanilla SAE
76.74
46.85
42.09
MRL-SAE
32
76.49
46.45
41.57
CSR
77.11
47.38
43.21
vanilla SAE
72.95
40.27
30.43
MRL-SAE
8
72.33
39.19
29.74
CSR
73.77
41.68
31.61
LLM USAGE STATEMENT
In accordance with the ICLR policy, disclose the utilization of Large Language Models (LLMs) in the preparation of this manuscript. The application of these tools was strictly confined to lin- guistic and formatting support. Specifically, LLM was employed to proofread the text, correct grammatical errors, and enhance the clarity and readability of the prose. The LLM played no role any substantive scientific components of this work. including the conception of research ideas, the design of methodologies, the execution or analysis of experiments, or the generation of results and conclusions. All intellectual contributions and the essential content of this paper are exclusively attributable to the authors.
33


Reviews:
[{"review_id": "DVmtTm1bBL", "reviewer": "Reviewer_EiFR", "summary": "The paper proposes CSRv2 to create ultra sparse embeddings to address the challenges in Contrastive Sparse Representation of dead neurons, mismatch between the pretraining and downstream semantics and limited capacity when training a linear head. The authors improve the training recipe with progressive k-annealing, supervised sparse contrastive learning on the sparse codes, and full backbone fine-tuning instead of fine-tuning a linear head only. This creates ultra sparse embeddings for retrieval and downstream tasks as an alternative to matryoshka representation learning. Experimental results on MTEB with Mistral-7B and Qwen3-4B, and vision dataset ImageNet-1k with ResNet-50 show reduced dead neurons, accuracy gain at small k’s and matches the performance of CSR and MRL at relatively moderate k’s.", "questions": "Q. How sensitive are the results to the k-schedule shape, the length and initialization? \n\nAlso see weaknesses section above", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The analysis of failure modes of CSR in the ultra sparse regime exposing dead neurons with and without annealing is interesting. \n\n2. Evaluation setup is exhaustive and the accuracy improvements are compelling. The efficiency results on retrieval are also strong and indicate that the method should scale well to practical settings.", "weaknesses": "1. Novelty of the method is fairly limited. K-annealing and supervised contrastive objectives have been discussed (and experimented with a lot) before in prior work, as has been acknowledged by the authors. \n\n\n2. The results from annealing itself are relatively small compared to adding supervision and finetuning, which raises the concern to me whether the improvements are coming from the sparse learning principle, or mostly engineering training recipe. \n\n\n3. The efficiency cost focuses on retrieval but there are no end to end latency measurements, encoder latency, cost of index construction which are all relevant for any practical use.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9DalW2GGcf", "reviewer": "Reviewer_rU7X", "summary": "Contrastive Sparse Representation is a fairly recent paradigm that aims to improve the efficiency beyond Matryoshka Representation Learning. However, at ultrahigh sparse levels, the performance metrics drop quite a bit. In this work they show that it is due to dead neuron problem at that level, as well as usecase of SSL in CSR. CSRv2 applies a curriculum over the sparsity and replace SSL with SupCL, and show improvement over standard representation learning benchmarks along with timing analysis.", "questions": "Refer to the weakness.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "In general, improving any algorithm is great, and this work improves CSR, a fairly recent paradigm for ultrasparse representations, as well as providing improvements over the equivalent dimensional MRL representations.", "weaknesses": "I feel this work has several weaknesses, from novelty to an architectural point of view, leading me to reject this work. \n\n1. On curriculum: Do we always want a backbone that is only producing a sparse representation of specific sparsity? I believe that a more practical setup is where you've one single backbone, which is trained on multiple top-k values, from the largest desirable sparsity to the smallest sparsity. This current setup seems to be impractical, as I believe if there is a backbone that works on multiple sparsity all at once, it would not suffer from the dead neuron problems. \n2. For MRL, I believe one can use a different weightage for smaller representation sizes. Can authors point to where such a study has been done?\n3. There are more recent SAEs, called Matryoshka Sparse Autoencoders, and I think there can be a similar CSR that uses MRL-SAE. \n4. Supervised Contrastive learning seems quite a bit important for good performance at ultrahigh sparsity. However, for representation learning, we want the representations to be as general as possible (if they're to be used as vector databases), which may or may not have a supervised contrastive learning amenable dataset. Therefore, this requirement inherently limits the use case as a general-purpose representation for edge devices, say.\n5. While curriculum helps a little bit, it doesn't seem to solve the dead neuron problem, with a major fraction still dead, leading to a waste of resources. Therefore, are such ultrahigh sparsities really desirable, given that the performance dip is quite a bit, and may not be something a user is willing to sacrifice over? \n6. What are the ultrahigh sparse representations learning? Are they learning to separate hyperplanes for some superclasses? An analysis of this would be good!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dDUDg7xE4m", "reviewer": "Reviewer_tchZ", "summary": "This work proposes CSRv2, a training paradigm for learning contrastive sparse representations via progressive k-annealing and address the dead neuron problem of prior work. The authors compare to other popular adaptive representations like Matryoshka Embeddings and CSR and show promising empirical compute-accuracy tradeoff compared to these baselines on text (MTEB) and visual (ImageNet-1K) embedding tasks.", "questions": "I have a minor nitpick that can contextualize the future impact of this work, which I suggest adding in the next revision. You briefly touched on **vector quantization** in the concluding remarks (L484), which is extremely important for real-world applications (vector search). There are several works focusing on the compute-optimality of embeddings when popular quantization techniques [1 - 3] are involved, especially in context of the recent adaptive representations like MRL [4]. In the next revision, I suggest adding a small note (in the Appendix) commenting on the role of CSRv2 with quantization.\n\n### References\n[1] Jegou, Herve, Matthijs Douze, and Cordelia Schmid. \"Product quantization for nearest neighbor search.\" IEEE transactions on pattern analysis and machine intelligence 33.1 (2010): 117-128\n\n[2] Guo, Ruiqi, et al. \"Accelerating large-scale inference with anisotropic vector quantization.\" International Conference on Machine Learning. PMLR, 2020.\n\n[3] Jayaram Subramanya, Suhas, et al. \"Diskann: Fast accurate billion-point nearest neighbor search on a single node.\" Advances in neural information processing Systems 32 (2019).\n\n[3] Rege, Aniket, et al. \"Adanns: A framework for adaptive semantic search.\" Advances in Neural Information Processing Systems 36 (2023): 76311-76335.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* This paper addresses an important problem of learning sparse embeddings that are compute optimal for their memory and inference time, showing strong empirical performance on real-world embedding benchmarks like MTEB and ImageNet-1K.\n* The paper is written and presented very well, and is easy to follow. The figures and tables are clean, interpretable, and provide supporting evidence to the claims in the paper\n* Design choices made by the authors are empirically validated to show the benefits of these choices (Tab 4). For example, adding supervised training on top of CSR (Fig 3), using k-annealing SAE and its affect on dead neurons (Fig 2). The authors also provide details on the experimental setup (App. C, D), limitations (App. F), and additional ablations and qualitative analysis on sparsity (App. E) which all add to the overall quality of the work.\n* CSRv2 shows strong empirical accuracy and sparsity on both text and visual embedding tasks compared to strong baselines like MRL and CSR (Tab 2- 4, Fig 2, Fig 5)", "weaknesses": "I do not have any major concerns with this work. I will highlight several minor weaknesses that could be addressed to further improve its quality:\n\n* I suggest rearranging the abstract content to be more specific about empirical performance on each dataset/task:\n    * L22 - L26: *\"delivers a 14% accuracy gain at k = 2 ... a 7× speedup over MRL... 300× improvements in compute and memory efficiency\"* - gain /speedup with respect to what and on what data?\n    * For L27 - 31, how much (metric value) did CSRv2 beat the strongest baseline on MTEB (Tab 2, 3, 4) and ImageNet-1K (Fig 2, 5) respectively?\n\n* For MTEB experiments, please clarify these details explicitly in the main paper:\n    * you select \"six types of tasks\" (Fig 1b, Tab 2- 4). Why were these specific task types selected (Sec 4.1)?\n    * you mention in Tab 2 that you train all methods on same backbone and data (this is great!). Is this also the case for Fig 1b? \n    * Does the \"backbone\" line in Fig 1b and 5a represent zero-shot accuracy with base e5-mistral-7b and Resnet50 FF2048? \n    * In Tab 5, are all methods trained on MTEB from the base Qwen3-Embed-4B? If so, is the MRL number there zero-shot or is it finetuned on MTEB similar to what you do with e5 (L365)?\n\n* For Figure 1b, consider adding MR line for the same active dims as a baseline for better contextualization of CSR annealing\n\nOverall, I really like this work. If these minor weaknesses are addressed in the revision, I am willing to increase my score to a strong accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uFlmXrqoGi", "reviewer": "Reviewer_6A2r", "summary": "(I tend to write shorter reviews and the length of the review does not reflect the quality of paper or the time spend on reviewing it -- especially in this paper, I understand the background and technical details well enought to be able to write a to the point review). \n\nThis paper proposes an extension (fixing) of CSR (contrastive sparse representation) idea that was introduced to enable adaptive on-the-fly sparse vectors or desired complexity to enable faster retreival. This is an improvement on top of dense compact representations which order information implcitily like in MRL (matryoshka representation learning) where the shorter embeddings are obtained by truncation of dimensions. \n\nThis work investigates the case of ultra-sparsity (which is with active dims less than k=16) and find several underlying causes and fix the CSR training to be more supervised and matryoshka style to enable better ultra sparse retreival. The findings are supported by strong experimental results and discussion.", "questions": "Please make the 3 changes suggested above. the paper is strong and will have a lot of impact. all the best.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Jus a strong and useful paper overall. The study of each of the issues and natural fixes are commendable. \n\nI really like the k=1 experiments.", "weaknesses": "The Table 1 adds little value especially with calibrated use of stars -- it does not give reader any quantifiable takeaway and would recommend removing the use of stars. \n\nRe CSR and CSRv2: For a new reader it is not clear as to how one can vary \"k\" in CSR style models (unlike mrl where the truncation is defined). After chosing different \"k\" do we re-normalize. These are good things to go into the paper. \n\nWhile completly complementary, it would be good to show results of a fixed cost quanitzed embedding of same size. ie., if k=4 with bf16 you should have a result for Quanitzed embedding os size 64 bits (be it binary quant or PQ). It will give more completeness to the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)