Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation
Authors: Tiansheng Wen
Yifei Wang
Zequn Zeng
Zhong Peng
Yudi Su
Xinyang Liu
Bo Chen
Hongwei Liu
Stefanie Jegelka
Chenyu You
Venue: ICML 2025 oral (2025)
Keywords: Sparse Coding;Matryoshka representation learning;Adaptive Representation;Efficient Machine Learning; Sparse Autoencoder
Primary area: general_machine_learning->representation_learning
TL;DR: A novel framework that learns high-fidelity sparse embeddings for efficient representation

Abstract: Many large-scale systems rely on high-quality deep representations (embeddings) to facilitate tasks like retrieval, search, and generative modeling. Matryoshka Representation Learning (MRL) recently emerged as a solution for adaptive embedding lengths, but it requires full model retraining and suffers from noticeable performance degradations at short lengths. In this paper, we show that *sparse coding* offers a compelling alternative for achieving adaptive representation with minimal overhead and higher fidelity. We propose **Contrastive Sparse Representation** (**CSR**), a method that specifies pre-trained embeddings into a high-dimensional but  *selectively activated* feature space. By leveraging lightweight autoencoding and task-aware contrastive objectives, CSR preserves semantic quality while allowing flexible, cost-effective inference at different sparsity levels. Extensive experiments on image, text, and multimodal benchmarks demonstrate that CSR consistently outperforms MRL in terms of both accuracy and retrieval speed—often by large margins—while also cutting training time to a fraction of that required by MRL. Our results establish sparse coding as a powerful paradigm for adaptive representation learning in real-world applications where efficiency and fidelity are both paramount. Code is available at [this URL.](https://github.com/neilwen987/CSR_Adaptive_Rep)

Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation
Tiansheng Wen
Yifei Wang" *3 Zequn Zeng' Zhong Peng' Yudi Su
Xinyang Liu
Bo Chen'
Hongwei Liu'
Stefanie Jegelka 34 Chenyu You?
75
Full Rep
Feature Dim
AVg 1-NN Accuracy (9%)
CSR tours)
MAL-E
MRL
Full Rep-
Dense
IN Accuracyy (%)
70
Full
60
10.% artfomen pa1
MRL Rep.
Dense
8
X RNSO
SOTA, RN50
(N// SOTA RNS0!
40
SVD
CSR [ours]
MRI DA/ RNSG)
8
RNSO:
20
Rand.
MRL-E
Rep.
CSR (wl RN50)
Sparse
SOTA
Rand. PS
55
Retrieval Time Batch Query (ms)
GPU Hours
(a) of different representations
(b) Retrieval efticiency acroes different methods
Training
Figure 1. Overview of our proposed method. (a) Illustrative comparison between standard embeddings (dense, long) and different compression schemes: Matryoshka representations (MRL) (Kusupati et al., 2022) with short length and our Contrastive Sparse Represen- tation (CSR) based on sparsification. (b) Comparison of retrieval accuracy and time of different methods ImageNet with GPUs. For CSR, we present results with the SOTA RN50 backbone from Wightman (2019) as well as the same RN50 backbone from Kusupati et al. (2022) for a comparison. Compared to MRL and int8 quantification (Quant Int8) methods, our sparse embedding approach CSR attains the best retrieval accuracy (very close to full representations) while being much more efficient in retrieval time, using sparse matrix multiplication on GPU. (e) Training GPU hours CSR compared haseline methods, where we outperform MRL on average 1-NN accuracy with much less training time.
Abstract
into a high-dimensional but selectively activated feature space. By leveraging lightweight toencoding and task-aware contrastive objectives, CSR preserves semantic quality while allowing flexible, cost-effective inference at different spar- sity levels. Extensive experiments on image, text, and multimodal benchmarks demonstrate that CSR consistently outperforms MRL in terms of both accuracy and retrieval speed- large margins- wwille also cutting training time fraction of that required by MRL. Our results establish sparse coding as powerful paradigm for adaptive representation learning in real-world applications where efficiency and fidelity are both paramount. Code available at this https URL..
Many large-scale systems rely high-quality deep representations (embeddings) to facilitate tasks like retrieval, search, and generative model- ing. Matryoshka Representation Learning (MRL) recently emerged as a solution for adaptive embed- ding lengths, but it requires full model retraining and suffers from noticeable performance degra- dations short lengths. In this paper, we show that sparse coding offers compelling alterna- tive for achieving adaptive representation with minimal overhead and higher fidelity. We pro- pose Contrastive Sparse Representation (CSR), a method that sparsifies pre-traie pre-trained embeddings
by
'Equal contribution National Key Laboratory of Radar Sig- nal Processing, Xidian University, Xi' an, China "Stony Brook University, New York, USA 'MIT CSAIL. MA, USA 'TU Mu- nich. Correspondence Bo Chen <bchen @ Hongwei Liu <hwliu andiian.dd..c2..
1. Introduction
Representation learning is at the core of deep learning (Le- Cun et al., 2015) and high-quality representations inputs (e.g.. image, text) empower numerous large-scale systems, including not limited search engines, vector databases,
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation
and retrieval-augmeened generative AT (Lewis al., 2020). However, the rapid growth in data volume poses signifi- cant challenges for ailency-sensitive applications. It is thus desirable to develop representations of adaptive inference cost that can best trade-off between accuracy and inference speed.
(Lee al., 2024a), are pre-trained single representations massive Internet-scale data. Fine-tuning these models would be prohibitively expensive and would prevent lever- aging pre-trained open weights. Leveraging recent advances in training sparse autoencoders (SAEs) (Cunningham et al., 2023; Gao et al., 2024), we can train a lightweight 2-layer MLP module for sparsifying pre-trained embeddings within very short period of time (e.g., half of an hour on Ima- geNet with single GPU), which is of orders of magnitude faster than MRL, as shown in Figure I(c).
Recently, a class of methods called Matryoshka Represen- tation Learning (MRL) (Kusupati et al., 2022) has drawn lot of attention and is now officially supported in the latest OpenAl and Google's Gemini text embedding APIs (Ope- nAI, 2024; Lee et al., 2024b) with millions of users and applications. The idea if MRL is to train an ensemble of representations truncated at different lengths (e.g., from 8 to 2048) through joint multi-task training. However, MRI. deviates from standard representation learning and requires full parameter updates to the backbone; the joint training also inevitably sacrifices the quality of representations at noticeable margin (e.g., 5% drop of top-1 accuracy on Ima- geNet at full representation length). These limitations render MRL a costly and lossy method for adaptive representation.
These pieces of evidence on accuracy, retrieval time, and training time show that sparse features are strong alterna- tives to MRL methods for producing high-fidelity and com- putationally efficient representations with a lightweight mod- ule and training cost. Our proposed method, Contrastive Sparse Representation Learning (CSR), combines con- trastive retrieval and reconstructive autoencoding objectives to preserve the original feature semantics while better tailing it down to the retrieval tasks. We evaluate CSR on a range of standard embedding benchmarks, from image embedding, text embedding, to multimodal embeddings, and compare it against various state-of-the-art efficient embedding models. Extensive experiments show that CSR consistently outper- forms MRL and its variants by significant margins in terms of both accuracy and efficiency. Notably, under the same compute budget, CSR rivals MRL's performance by 9%, 15%, and 7% on ImageNet classification, MTEB text re- trieval, and MS COCO retrieval, respectively. Our main contributions are:
In this paper, we revisit sparse coding (Lee et al., 2006) as much faster, lightweight, and high-fidelity approach to achieve adaptive representation. As illustrated in Figure (a), instead of truncating the representation length as in MRL, we leverage sparse vectors and sparse matrix fac- torization to attain computational efficiency. Specifically, we sparsify a full representation at different levels (charac- terized by K, the number of activated neurons). We find that a few numbers of activated neurons (e.g.. to 16) can preserve the performance of a much longer dense represen- tation (e.g., 2048 dimensions). This is in sharp contrast to MRL embeddings whose quality deteriorates a lot at such extremely short lengths (> 10% drop). Therefore, sparse features using sparse vector formats can be stored efficiently with only a few activated neurons. With the help of sparse matrix factorization (with native GPU support in modern deep learning libraries such as PyTorch)', these sparse em- beddings can be used for retrieval tasks at a much higher speed with complexity order of O(K), where K is very small. In comparison, MRL requires a longer length of representation (e.g. 256) to attain similar accuracy (if possi- ble), leading to extra slower inference speed. As shown in Figure 1(b), MRL is inferior to our method in terms of both accuracy and retrieval time by a significant margin.
We propose sparse coding as an alternative approach to adaptive representation learning and demonstrate its numerous advantages over the MRL approach in terms of fidelity, retrieval cost, and training cost.
We introduce an effective learning method for sparse adaptive representation, Contrastive Sparse Rep- resentation (CSR) Learning. It combines a task- specific sparse contrastive learning loss with a recon- structive loss to maintain overall embedding quality. This generic design consistently improves performance across different tasks like classification and retrieval.
We conduct a detailed analysis of CSR, examining var- ious factors and providing a fair comparison with MRL terms of retrieval time and accuracy. We further val- idate CSR's effectiveness across real-world domains and benchmarks, where it achieves competitive per- formance against heavily trained state-of-the-art MRL. models with significantly lower computational costs. On the inference side, CSR delivers a 69x speedup on ImageNet 1-NN tasks without compromising perfor- mance compared to quantization- based approaches.
Another key advantage of sparse features is that they elim- inate the need to retrain the entire network. In contrast, MRL-Kusupati et al. (2022) noted-performs poorly un- less full-parameter tuning. However, many existing founda- tion models, such as the multimodal representations in CLIP (Radford al., 2021) and the text embeddings in NV-Embed
Pp Torch's 's native sparse vector library can he found at https: // "ppytrrhh.or /ocs/ boocs/sabll stable e/ rse. html.
2

---

Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation
2. Related Work
coding - we investigate how SAEs can be further devel- oped to learn adaptive representations with high efficiency, expanding their applicability to wider range of tasks.
Adaptive Representation Learning. Recent research has increasingly focused on learning adaptive representations that cater to multiple downstream tasks with diverse com- putational requirements. Early efforts explored context- based architectural adaptations (Kim & Cho, 2020), dy- namic widths and depths in BERT (Hou et al., 2020), and random layer dropping during training to improve pruning robustness (Fan et al., 2019). More recently, Matryoshka Representation Learning (Kusupati et al., 2022) introduced novel technique for creating flexible, nested substructures within embeddings, enabling fine tineggriied control over the trade-off between latency and accuracy. This concept has since been extended to various modalities and applications, including large language models (OpenAl, 2024; Nussbaum et al., 2024; Yu et al., 2024), diffusion models (Gu et al., 2023), and multimodal models (Cai et al., 2024; Hu al., 2024). Other works have further explored token reduction in image and video processing (Yan al., 2024b; Duggal et al., 2024).
3. Method
Our proposed framework, Contrastive Sparse Representa- tion learning (CSR), is illustrated in Figure 2. Starting from pre-trained embedding representation space R", selectively activating the most rel- evant dimensions for adaptive representation learning. We then regularize this hidden space using a reconstruction- based sparse compression loss (Section 3.2.1). Addition- ally, with theoretical motivations and guarantees provided by (Wang et al., 2024), we introduce non-negative con- trastive loss to expand model capacity and feature identifia- bility. (Section 3.2.2)
R we project it into a sparse
3.1. Preliminaries
Problem Formulation. For simplicity, we first introduce our framework in the context of a classification task. Let DM = v: (x } be training dataset of size N, where 2S E X' are an input sample and Vi E y are corre- sponding labels with L classes, We obtain an embedding v = f(x;0)) X + Rd. We can apply exact lz-based k- nearest neighbor (KNN) search for classification, which has (dN) complexity. In practice, KNN often employs high- dimensional embeddings (i.e. d 4096) to achieve stronger performance, but at the cost of increased computational la- tency. Our goal is to learn a more compact representation o E RT (where m & d) that balances accuracy and query latency. This shortened embedding can also benefit other downstream tasks such as retrieval and clustering.
Despite these advances, existing methods often do not fully harness the capabilities of large foundation models, high- lighting the need for more effective compression strategies. Our proposed sparse compression methodology addresses this gap by providing a lightweight, plug-and and--lay solution that be readily applied on top of any foundation model significantly reducing computational overhead while pre- serving representational quality.
Sparse Coding. Sparse coding serves as a powerful tech- nique for compressing nigh-dimensional signals and extract- ing salient features (Wright et al., 2010; Zhang al., 2015), with learned sparse representations often providing addi- tional computational benefits and robustness (You et al., 2024; 2025). Prior work has induced sparsity through mod- ifications to model design or training protocols, including modifications to attention mechanisms (Correia et al., 2019), applying Bayesian standard Gamma priors (Duan et al., 2024a;b; Hu et al., 2025), incorporating discrete sparse con- cept layers (Koh al., 2020; Xie et al., 2025), and promot- ing sparse activations in large language models (Mirzadeh et al., 2023; Zhang et al., 2024). However, training state-of- the-art foundation models from scratch under these sparsity constraints has proven challenging (Elhage et al., 2022), limiting their current applicability.
Matryoshka Representation Learning (MRL). MRL (Kusupati et al., 2022) simultaneously optimizes embed- dings at multiple dimensions, as illustrated in Figure 2, to produce representations of variable size. Specifically, let M be set of target embedding sizes. For each m E M, MRL applies an additional linear classifier to the first m dimensions of the embedding vector, U1:m E IR. This design ensures each truncated representation is explicitly trained via the final loss. Formally, the MRL objective is
CMRL= = E CmLCE (w(m). f(xx;;(()))) Bi), (I)
MEM
Meanwhile, Sparse Autoencoders have achieved notable success in improving the interpretability of foundation mod- els (Cunningham et al., 2023; Yan et al., 2024a), primar- ily because they uncover semantic information by map- ping high-dimensional data onto power-dimensional sub- spaces (Cunningham et al.. 2023). Building on these in- sights - and harnessing the inherent advantages of sparse
is the linear classifier weights corre- sponding V1:m- Each loss term is scaled by a nen-negative coefficient {cm 2 0) meM- The multi-granularityy arises from selecting dimensions in M, whose size is constrained to at most log(d), that is, IM < [log(d)) For example, Kusupati et al. (2022) choose M = {8, 16.... 1024) as the nesting dimensions.
where (m)) E 1R
3

---

Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation
3.2. Contrastive Sparse Representation
Trainable modules
Frozen modules
As discussed in Section MRL (Equation 1) faces two key constraints: it requires (full) training of the backbone param- eters y and its performance often deteriorates lot under small hidden dimensions.
Zero-out dimension
E B
To overcome these limitations, we propose a new methodology that relies on the computa- tional efficiency of sparse vectors for efficient retrieval. The method, named Contrastive Sparse Representation (CSR), learns a simple one-layer sparse module on top of frozen pretrained embedding models (with full representation size, e.g., 2048) that maps dense embeddings to highly sparse embeddings with small number of active (i.e., non-zero) dimensions (e.g., 32). As result, CSR not only saves a lot training effort, but also allow using sparse matrix multipli- cation at inference time to accelerate retrieval significantly. Below, we outline how we train the CSR module through combination of sparse autoencoding (Section 3.2.1) and sparse contrastive learning (Section 3.2.2).
Backbone
EcE
EMRL
MRL
+
Backbane
Arect
CSR
Lyason
IR
Figure 2. Overview of our proposed CSR framework. As post- training approach, CSR differs fundamentally from MRL by pro- jecting embeddings into higher- dimeesioddiniiaall space and dynam- ically activating only the TopK dimensions for a compact repre- sentation. The hidden space is constrained hy both reconstruction and contrastive losses, which together enhance the capacity of the sparse representation while preserving computational efficiency.
3.2.1. SPARSE AUTOENCODING
Autoencoding is long-standing unsupervised objective that extract salient features that could preserve the original data the most a reconstruction objective. In CSR, we aim at com- pressing dense embeddings to sparse vectors for efficient sparse retrieval while retaining most of the useful informa- tion. To achieve this goal, we adopt sparse autoencoders due to their ability to scale with large data and restore feature semantics (Cunningham et al., 2023; Yan et al., 2024a).
Moreover, as the hidden dimension h increases, we empiri- cally observe that an increasing number of latent dimensions remain inactive during training phenomenon referred to as "dead latents". : A large proportion of dead latents reduces the model's 's capacity and leads to performance degrada- tion (Lu et al., 2019; Templeton et al., 2024). To mitigate this issue, an auxiliary loss Laux and Multi-TopK losses are proposed to mitigate this problem. The overall reconstruc- tion loss is
Sparse Autoencoders (SAEs). SAEs (Makhzani & Frey, 2013; Cunningham et al., 2023; Gao et al., 2024; Yan et al., 2024a) aim to extract a sparse representation zı by learning to reconstruct the dense feature from 24. Specifically, given a pretrained dense embedding v == f(x) E Rd as the input, we apply TopK SAE (Gao et al., 2024) with the following autoencoding process:
Crecon = =((k) + C(4k)/8 + 3Lauxi
(5)
(2) where Laux = -1e-lle00 e= ((x)
and è Wasc is the reconstruction using the top-kaux dead latents. By default, we set Kaux 512 and / = 1/32, following the setting in Gao et al. (2024). We also offer dynamic sparsity selection, with k ranging from 8 to 256, to accommodate different tasks across various modalities.
2k = a (TopK ( (f(r) bpre) ) + bene));
f(x) = 2k + bpre:
(3)
where Wone& E Rdd and Were& E Rdxh are the encoder and decoder weight matrices, respectively; bene R and bpee E Rdd are bias terms. The function o (-) = max(0.)) -) denotes the ReLU activation, and TopK(-) selects the top & largest elements of the input, zeroing out the rest (as in Gao et al. (2024)). As a result, the latent 24 is always a sparse non-negative vector with k active dimensions. This en- ables direct control over the accuracy- trade-off in downstream tasks, particularly under resource-constrained conditions. We formulate the loss function follows:
3.2.2. SPARSE CONTRASTIVE LEARNING
Furthermore, we consider to incorporate an additional sparse contrastive loss to the representations discriminative power. Most state-of-the tott----rttt--ttteee embedding models today, e.g., CLIP (Radford et al., 2021), follow a contrastive learning paradigm, which that learns to use the embeddings to dis- tinguish between positive and negative pairs. And it applies to both supervised and unsupervised settings (Huang et al., 2024).
C(k) = ((x) - M
(4)
4

---

Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation
The loss objective can be formulated as:
bi Bd (:-3-4) Fim Fen 132//00)
natrooaa Time par Batoo Quenny PaP
1FM
B log Ld = B 1=1
exp(zf 2.) exp(= 2)) + Ejis crp(2][a) 2j)
(6)
-10
Natriwall M
By leveraging the non-negative nature of latent variables 2 in sparse autoencoders, Equation 6 can be viewed as a variant of the Non-negative Contrastive Loss (NCL) pro- posed in Wang et al. (2024). This interpretation enables us to draw on the theoretical guarantees of NCL, as stated in the following theorem:
32
Activate Dim
Activate Dim
(a) Effeect of hidden dim (b) Effect of database size
Theorem 5 (Wang et al. (2024)). Under mild conditions, the solution (xx) is the unique solution to the NCL. objectivv. As a result, NCL features are identifiable and disentangled.
Figure 3. Comparision of retrieval time based different faccoor.. (a) Fixed-scale scenario (IM database): Both methods achieve performance sweet spots at 1opK=16, with CSR exhibiting 2x speedup over dense embeddings when sparsity exceeds 80%. (b) Scaling scenario (h 8192): CSR exhibits increasingly efficient scalability from 0,5M to 10M, with performance gains accelerating at larger scales. This makes it highly practical for real-world applications involving millions of entries.
Theoretically guaranteed by Theorem 5, the model is en- couraged to utilize larger number of latent dimensions to reconstruct the input data. This behavior is empirically demonstrated in Figure 6, where we observe reduction in "dead" dimensions compared to vanilla SAE approaches.
3.2.3. OVERALL TRAINING OBJECTIVE
In Section 4.1, we further carefully benchmark them in prac- tice and find that the two indeed have similar retrieval time, and sparse ones can be even slightly faster under small k.
At last, we optimize the sparse module through a combina- tion of sparse autoencoding Lrecon and sparse contrastive learning Cncl. The former incentivizes the model preserve original semantic information in the original representation, while the latter shapes the sparse representation to be better at discriminative tasks. The final training objective of our Contrastive Sparse Representation (CSR) method is formu- lated as:
To account for variations in retrieval time due to sample size, we establish standardized benchmarking protocol (denoted as T) to measure retrieval latency by default. Specifically, to simulate large-scale retrieval scenarios, report the average retrieval time for 512 queries over ImageNet- scale database containing 1.3 million entries (equivalent to the size of the ImageNet training set). For CSR, we use a default hidden dimension of h = 16,384 and an active dimension of k = 32. All experiments are conducted in a consistent GPU environment using PyTorch (Paszke et al., 2019). To facilitate comparison, we also report the relative retrieval time of each method by normalizing it against the retrieval time of CSR under the default setup. Additional implementation details can be found in Section E.3.
LeSR Lrecon + mmel-
(7)
Here, y is hyperparameter that balances the two loss com- ponents and is set to I by default.
4. Empirical Analysis
In this section, we conduct a careful study on the empir- ical performance of the proposed CSR. All experiments in this section are conducted on ImageNet, using 1-NN accuracy (Johnson al., 2019) as the evaluation metric. By default, we set the hidden dimension h of CSR to be h = 4d, where d is the dimension of the pretrained dense embeddings, and set the default active dimension to = 32.
4.1. Retrieval Time Comparison with MRI.
In this section, we benchmark the retrieval time of MRL and CSR under the same active dimension & and analyze the impact of hidden dimension R.A,, database size N and sparsity k.
For a fair and intuitive comparison of MRL and CSR, First, we adopt the notion of active dimension as a surrogate metric to benchmark the retrieval time under dense (MRL- type) and sparse (CSR-type) embeddings. For example, "Active Dim 8" denotes either length-8 dense embed- ding (MRL) or a sparse embedding with TopK (k = 8) activation (CSR). Notably, we choose it because dense and sparse matrix multiplication have the same computation complexity under the same active dimension k iee. (k))
(i) Active dimension. Figure 3(a) shows retrieval time under varying hidden dimensions, with database size fixed. We can see that the retrieval time of CSR (i.e., sparse multipli- cation) and MRL (i.e., dense multiplication) both grow with large k and remain relatively on the same level. And for smaller k. CSR shows a clearer advantage over MRL. AI- though CSR and MRL have similar theoretical complexity (dk)), their actual runtimes are affected by backend im-
5

---

Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation
83
256
76
256
75
Futt Repneeetttttn Bestietsc
1-NN Accuracy (%)
Fdll Degresentatonn on of VITA
1-NN Accuracy (%)
1-NN Accuracy (%)
Full Represenatiion
1-NN Accuracy (%)
74
Full Repress entaatio
80
70
664
81-
Topk
Topk
72
Fall Regresenttton VUTS
65
S.U Bestect
80
16
16
75
70
60
79-
78
8
68
8
VIT-S/16
55
ResNet18 ResNet50
20
4d
16d
2d
8d 160
VIT-1/26
Hidden Dimension
Hidden Dimension
8
16
64
256
16
64 TopK
256
(a) ViT-L/16 model.
(b) ResNet50 model.
TopK
(a) ViT-based Model.
(b) ResNeet-base Model.
Figure 5. Performance of CSR under different hidden dimen- sions and different types of backbone models (Res.Net-50 (con- volution) and VIT-L (Transformers)) CSR exhibits a reverse U-shape across different models and hidden dimensions. CSR's performance peaks at h = 4d (d is the input dimension size) but degrades beyond this, especially with higher sparsity.
Figure 4. Performance of CSR under different sparsity levels with different sizes of backbone models. CSR achieves higher fidelity greater sparsity levels when applied larger backhone models (which provide hetter hase performance), observed consis- tently in both ViT and ResNet architectures.
0.8
SAE w/o Aause Multi-Topk
plementations. For instance, cuBLAS (used for dense ops) is highly optimized but has high launch overhead, while cuSPARSE (used for CSR) is lighter but less optimized for small k. Interestingly, we can observe that for sparse em- beddings, retrieval time decreases as hidden dimension h increases. This suggests notable benefit of CSR that it can use higher latent dimensions for better expressivity while attaining faster retrieval. On the contrary. MRL with higher dense dimensions always has slower retrieval. We elaborate potential reasons on this distinction at Appendix E.4.
Fraction of Dead Latents
Caon Multi-Topk CSR (ours)
0.4
0.0
16
32
128
1024
TopK
(ii) Database size. Figure 3(b) shows that CSR demonstrates superior scalability as the database size N increases from 0.5M to 10M. The relative efficiency gain becomes more pronounced with larger datasets, underscoring the practical- ity of sparse embeddings in real-world retrieval scenarios.
Figure 6. Comparison of dead latent fractions across loss combina- tions under varying sparsity constraints. Results show that even equipped with Lavck and Multiple-Topp at extreme sparsity levels (i.e., = 8, 16, 32). CSR further alleviates this issue, outperform- ing baselines and demonstrating its robustness.
4.2. Effect of Backbone Size
4.3. Effect of Hidden Representation Dimension g
Experiment Setup. We examine fidelity versus backbone size (with different input dimension ) and sparsity, using fixed hidden dimension Ra across architectures. For ViT, we use VIT-S/16 (d 384) and ViT-L/16 (d = 1024) with h = 4096. For ResNet, we test RNI8 (d = 512) and RN50 (d = 2048) with h- = 8192. A more detailed experiment setup is provided in Section E.
Experiment Setup. We explore how hidden dimension R effects on our model, we use ViT-Large and ResNet50 as pre-trained backbones, sweeping h from d to 16d while keeping all other parameters at their default values. Addi- tional implementation details are provided in Section E.2.
Analysis. Figure 5 compares model performance across different hidden dimensions under varying sparsity con- straints. Notably, shift in the performance trend occurs at h 4d. When h 4d, performance gradually im- proves with increasing hidden dimension, reaching its peak at h = 4d. However, beyond this point, further increases in h lead to performance degradation, particularly under higher sparsity constraints. This trend aligns with the observations of Gao et al. (2024), which suggest that excessively large
Analysis. Figure demonstrates that larger backbone with higher input embedding dimensions improves model fi- delity equal sparsity levels. This insight is particularly sig- nificant, as larger embedding sizes generally encode richer information, thereby achieving better downstream perfor- mance. By leveraging these high-dii nigh-dimensional embeddings, our approach more effectively retains essential features and relationships within the data.
6

---

Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation
Pur Rep.
T0p1 Accuracy (95)
Average Performance
40-
00
1AN Accuracs accraacy
55
65
MLL
45
HiLE
60
MAL MALE
60
HNL-P
59
Rand,
Rand. F5
55-
SAE
54D
35-
$10
SAE
CSA (nurd)
toun)
50
8
20
32
54
125
256
512 1024 2045
50
15
32
04
120
255
512 2014 2043
Active Dim
Active Dim
Classrication
Gluterfing
Retrieval
(a) Comparison of ImageNetlk Topk-1 Ac-
(b) Comparison of ImageNetlk I-NN Accu- racy under the same pretrained backbone
(c) Comparison of natural language tasks
curacy under the same pretrained backbone
across models with similar retrieval time
Figure 7. (Left & Middle): Results of ImageNet Top-1 accuracy (a) and I-NN accuracy (b) across active dimensions under the same pretrained ResNet-50 backbone used in Kusupati et al. (2022). We can see that while MRL trains the whole network and CSR only uses frozen embeddings, CSR still performs consistently better across all embedding sizes has significant margins beyond 20% at lower active dimensions (the region that yields the largest efficiency gains). (Right): Comparison of text embedding methods similar retrieval cost. For CSR, we use = 32 by default. For each task, the model is trained on three datasets and evaluated on three unseen datasets. The text embeddings learned by CSR outperformed other MRI--hased baselines by significant margins across different natural language tasks at much lower training cost.
hidden dimensions may not be fully utilized, ultimately di- minishing model performance. A similar pattern is observed in ResNet. Based on these findings, we set h = 4d as the default configuration for all subsequent experiments unless otherwise specified.
representation (see Section 5.2), we focus on three primary tasks: text classification, text clustering, and text retrieval on the MTEB benchmark (Muennighoff et al., 2022). For multimodal representation (see Section 5.3), we report both in-distribution and zero-shot cross-modal retrieval perfor- mance two widely-used datasets: MS COCO (Lin et al., 2014) and Flickr30K (Young et al., 2014). Through these experiments, we aim to provide a holistic understanding of the capabilities of our proposed framework.
4.4. Effect of Different Losses
Experiment Setup. We investigate how different loss functions affect model capacity, particularly in addressing the dead latent problem discussed in Section 3.2.1, using RN50 backbone with h = 4d. Other parameters are set at their default values.
5.1. Vision Representation Comparision
Baselines We compare our proposed method with the fol- lowing baseline approaches. 1) MRL/MRL-E (Kusupati et al., 2022): RN50 model where the fully connected layer is replaced by multiple (MRL) or single (MRL-E) classi- fication head(s) that take truncated input dimensions (e.g.. only the first 8 of the original 2048 dimensions). 2) SVD: We performed a low-rank approximation of the 1000-way classification layer of RN50, with rank 1000. 3) Rand-LP: We compared against a linear classifier fit on randomly se- lected features (He et al., 2020). 4) Rand-FS: We randomly selected features extracted from RN50 for 1-NN classifica- tion.
Analysis. Figure 6 illustrates the impact of different loss functions on model capacity. The naïve SAE suffers from severe dead latents, while the inclusion of an auxiliary loss Laux and the multi-TopK loss partially mitigates this issue. Introducing non-negative contrastive loss (NCL) further alleviates the problem. particularly at extreme sparsity levels (e.g., k 8, 16, 32). Empirical results validate the effec- tiveness of Theorem 5, demonstrating that representation learning with NCL promotes more orthogonal and disentan- gled features. This, in turn, increases the number of active dimensions and enhances overall model performance.
Experiment Setup. We evaluate 1-NN accuracy and Top- accuracy on ImageNetlk classification, following Kusu- pati et al. (2022). For fair comparison, we used the same RN50 backbone weights as MRL (denoted as FF2048 in the original work) and trained CSR on its ImageNet1k encoded embeddings. For further implementation details, please refer to Section B.
5. Benchmark Results and Analysis
We evaluated the effectiveness of our proposed CSR frame- work across three mainstream representation modalities: vision, language, and vision+language. For vision repre- sentation (see Section 5.1). we conduct image classification on ImageNet-1K and evaluate performance using 1-NN accuracy, following Kusupati et al. (2022). For language
7

---

Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation
Table Performance and efficiency of text embeddings on three natural language tasks: classification, clustering, and retrieval. We use NV-Embed- V2 as our pre-trained model, and present its performance in the first line of the table in gray. We analyze Dataser- Speecfffi Evaluation results along two key dimensions: (1) Relative Retrieval Time under matched performance and ii) performance under matched retrieval efficiency. Under matched performance, CSR achieves a remarkable 61x speedup, while under matched retrieval efficiency, it improves performance by 15%, demonstrating its superior balance between speed and accuracy. The maximum values are indicated in bold, while the second second-highest values underlined. Relative Retrieval Time is calculated follows the definition in Scction 4.
Text Chasalication Top-1 Acc (8)
Text Clusterig
Text Retrieval
Active
Rerrieall
Top-I Ace (8)
NDCG (9)
Category
Model
Time
MTOPIoiest
Runking77
TweetSeniment
BioreivP2P
BlorcivS25
Twonyyoos
FIQAZ018
NFCrepus
SSFACC
Full Rep
NV Embedd-V V2
4096
37.5
93.55
92.20
79.73
53.61
49.50
64.82
62.65
43.97
77.93
Stella-1 -BB-v5
2.6
90.45
76.75
50.81
46,42
600.07
55.59
36.97
77.48
Jinu-V3
256
2.8
78.81
$4.08
73.81
38. 14
34,39
51.96
55.73
36.63
66.63
Nomic-Fibott Vi1
256
2.7
72.47
83.69
59.30
38.
31,83
48.55
35,00
32.54
68.
Gocko Embbed- GoGugle))))Gogllee)
2.4
77.82
72.97
36.28
37.81
70.86
MRL
Text-Embed 3-L (OpenAI)
Aretic-Fmbed-LL-V -V2
256
256
2.8
2.6
70.45
67.00
80.99
$3.19
53.98
59.06
34.25
35,43
33.86
34,077
54,24
30,06
50.33
44.69
35.02
37.94
23.10
69.51
M2V-Base Glove
2.4
59:26
72.39
50.02
52.26
25.38
23:15
50.66
Jinn-V3
64
1.2
66.12
67.98
71.18
36.89
33.57
50.22
44.18
33.66
68. 84
Nomie Emiii---bb---15
1.6
62.77
$0,63
55.23
34,81
44.61
48.06
10.22
18.96
36,55
Potion Baso-2M
1.4
42.50
65.17
52.52
25.78
14.94
27.07
32.08
30.72
64.28
Sparse
SAE (w/ NV-Eibbed-V2)
32
1.0
87:45
88:11
75.19
$1.02
48.68
58.63
49.18
35.14
66.04
CSR NV-Embedd-22)
$9.86
91,02
78.55
53,49
63.05
57.54
38.06
71.17
Analysis. Figure 7(a) and (b) illustrate the comparison of learned representation quality through the 1op-1 and 1- NN classification accuracy of RN50 models trained and evaluated on ImageNet- IK. For linear probing results (Fig- ure 7(a)), reconstruction-busee sparse compression methods (CSR& & SAE) outperform MRL-LP (both linear probing methods) by a large margin and also surpass MRL/MRL-E (train from scratch) in lower active dim (k < 128). Further- more, Figure 7(a) demonstrates the superior representation quality learned by CSR, which consistently outperforms MRL across various active dimensions. CSR also surpass traditional post-hoe compression techniques (e.g., SVD) and linear probes on random features by increasing the overall model total capacity while keeping active dimensions for each sample unchanged, as discussed Section 1 and Sec- tion 3.2.1. This enhanced capability allows CSR to maintain remarkable robustness, even under extrem sparsity where k= k= 2, 4, 8. These results highlight that the proposed CSR design can effectively compress pre-trained embeddings while leveraging the natural benefits of sparse matrix multi- plication. More detailed experimental results can be found in Section 4.
For further experimental details, please refer to Section C. To improve readability, we refer to CSR-K model with the TopK activations and so as SAE.
Analysis Table 1 demonstrates the performance of CSR and baseline models across multiple tasks and datasets. CSR not only maintains the strong performance of the pre-trained model but also surpasses baselines under varying resource constraints. Taking text classification as an example, CSR achieves 15% accuracy improvement matched compu- tational cost (i.e., with retrieval times comparable to Jina- V3-64 and
Vi.odd-V1.--6-- while attaining a 61x speedup when matched for performance (i.e., compared to NV-Embed-V2). The results underscore CSR 's exceptional ability to maintain an optimal speed-accuracy trade-off a critical requirement for practical deployment in large-scale retrieval systems. We further evaluate the generalization capability of CSR (with k = 32) on three unseen datasets per task, as shown in Figure 7(c). The results demonstrate that sparse representations yield more robust performance compared to dense alternatives at same activation dimen- sions. These results underscore the efficacy and versatility of CSR, demonstrating its strong potential for real-world applications.
5.2. Text Representation Comparision
Experiment Setup. We assessed CSR on three key tasks from the MTEB benchmark, testing it across six datasets for each task. In detail, we conduct evaluations in two distinct settings: Sppeiic-Specific Evaluation, where CSR is trained and tested on different splits of the same dataset to ensure consistency, and Task-Specific Evaluation, where CSR is trained on one dataset and evaluated on unseen datasets within the same task to rigorously assess its generalization capabilities. We choose NV-Embed- V2 (Lee al., 2024a) as our pre-trained model and present its performance in gray.
5.3. MultiModal Representation Comparision
Experiment Setup. We evaluated our methods on multi- modal retrieval tasks using the ViT-B-16 backbone, testing both in-distribution and zero-shot cross-modal retrieval on MS COCO (Lin et al., 2014) and Flickr30K (Young et al., 2014) datasets. For basclines, we fine-tuned MRL on these datasets (using CC3M (Changpinyo et al., 2021) for zero- shot training). following standard MRL training protocols (Kusupati et al., 2022). The performance of our backbone,
8

---

Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation
Table Comparison of diffferet methods on multi-modal retrieval tasks using two benchmark datasets, MS COCO and Flickr30k, evaluated under both in-distribution and zero-shot settings, with @ (%) as the performance metric. We use VIT-B/I6 as our pre-trained model, and present its performance in the first line of the table in gray. For Zero- setting, CSR first trained on a large-scale scale, dataset-CC3M and evaluated on downstream tasks. CSR (plug-and-play) consistently outperforms ViT-B-I6-MRL. (fully fine-tuned) in various tasks with significant training efficiency.
In- In-Distribution
Zero-Shot
Active
Trainable
MS COCO
Flickr30K
MS COCO
Flickr30K
Method
Dim
Parms
12T
T21
12T
T21
12T
T2I
12T
T2I
VIT-B-16
512
86M
74.42
86.47
91.92
97.79
69.23
83.03
89.82
97.70
VIT-B-16-MRL
86M
67.12
77.53
80,41
89.89
56.90
65.82
80.94
89.20
SAE
256
1.1M
71.21
82.58
87.76
95.59
58.22
67.40
82.44
86.19
CSR
1.1M
71.41
83.49
87.98
96.79
61.85
70.14
85.22
91.10
VIT-B-I6-MRL
86M
64.19
73.02
77.56
87.80
53.63
61.16
77.67
85.10
SAE
128
1.1M
64.67
76.70
81,40
91.20
53.20
63.02
77.54
85.19
CSR
1.1M
69.34
81.04
84.05
93.00
54.37
68.04
78.08
88.09
VIT-B-16-MRI
86M
62.61
72.43
74.22
84.79
47.47
54.42
71.16
79.00
SAE
64
1.IM
56.30
69.45
70.58
81.30
44.48
53.56
69.58
82.29
CSR
1.1M
62.75
78.10
76.44
88.50
48.61
61.90
73.04
84.10
using the same fine-tuning procedure, shown in gray. Dur- ing training, both SAE and CSR leverage a shared sparse embedding layer for images and text. Additional experi- mental setup and implementation details are provided in Section D.
6. Conclusion & Discussion
In this paper, we introduce Contrastive Sparse Representa- tion Learning (CSR), a generic learning framework offering a high-fidelity and flexible approach to compress embed- ding, surpassing existing methods like MRL in various tasks and modalities. We believe CSR paves the way for more efficient and flexible representation learning, especially in scenarios constrained by memory, latency or other computa- tional considerations.
Analysis. Table 2 presents the multimodal retrieval task results across different methods and settings. In general, iiconstruction-basel methods exhibit relatively low perfor- mance degradation on both datasets. Compared the MRL method, CSR achieves average performance gains of 4.6% and 6.8% on 12T retrieval, and 10.3% and 6.5% on T2I retrieval across the two datasets in in-Distribution Evalua- tion. Besides, under zero-shot scenario, CSR also surpasses MRL by 3.2% and 3.3% on 12T, and 9.2% and 3.9% on T2I, respectively. Notably, these results demonstrate CSR's potential handle large-scale datasets (e.g., CC3M-3M im- ages, compared to ImageNet's IM and MS COCO's 0.3M), confirming CSR's 's consistent superiority across various ac- tive dimensions and its scalability. SAE experiences more severe performance degradation compared to CSR, which underlines the efficacy our design in image-text alignment. However, as the sparsity constraint becomes more stringent, the performance gap between CSR and MRL narrows. Upon further investigation, find that CSR still suffers from the "dead latents" problem even when equipped with advanced mechanisms. Addressing the mitigation of dead latents in the alignment space remains an open challenge, leaving room for future work and study. For a detailed analysis, please refer Section D.4.
Our method, CSR, is orthogonal to existing acceleration techniques such as pruning (He et al., 2017), quantiza- tion (Jacob al., 2018), and distillation (Hinton al., 2015), which primarily target embedding generation. In contrast, CSR optimizes the post-processing stage, enabling comple- mentary speedups with minimal performance trade-off. A current limitation of CSR, shared by other sparsity- based ap- proaches, is the emergence of dead neurons under high spar- sity, especially in multimodal settings. While techniques like contrastive loss partially mitigate this (see Figure 6), fully resolving the issue remains an open challenge and direction for future work.
A Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant U21B2006; in part by the Fundamental Research Funds for the Central Universities QTZX24003 and QTZX23018. in part by the 111 Project under Grant B B18039; and in part by Shaanxi Youth Innovation Team Project.

---

Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation
Impact Statement
Duan, Z., Wen, T., Wang, Y., Zhu, C., Chen, B., and
Zhou, M. Contrastive factor analysis. arXiv preprint arXiv:2407.217400 2024b.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
Duggal, S., Isola, P., Torralba, A., and Freeman, W. T. Adap- tive length image tokenization via recurrent allocation. arXiv preprint arXiv:241 1104411.0299. 2024.
References
Elhage, N., Hume, T., Olsson, C., Nanda, N., Henighan, T., Johnston, S., ElShowk, S., Joseph, N., DasSarma, N., Mann, B., Hernandez, D., Askell, A., Ndousse, K., Jones, A., Drain, D., Chen, A., Bai, Y., Ganguli, D., Lovitt, L., Hatfield-Dodds, Z. Kermion, J., Conerly, T., Kravec, S., Fort, S., Kadavath, S., Jacobson, J., Tran- Johnson, E., Kaplan, J., Clark, J., Brown, T., McCan- dlish, S., Amodei, D., and Olah, C. Softmax linear units. Transformer Circuits Thread, 2022. https:/trrasffommrr hhtmm.
Model2vec: Turn any sentence transformer into small fast model, 2024.
Boteva, V., Gholipour, D., Sokolov, A., and Riezler, S. A full-text learning to rank dataset for med- ical information retrieval. 2016. URL http: //www. cl. uni www- de/dddeeeeeeeeeeeeeeeeeeeieeeeee heeae publications http:aaaitinnnaaaassssseeeeeeeee http://ppppss/III///// pdf.
Cai, M., Yang, J., Gao, J. and Lee, Y. J. Matryoshka multimodal models. arXiv preprint 1rX4v 2024.
Fan, A., Grave, E., and Joulin, A. Reducing transformer depth on demand with structured dropout. arXiv preprint arXiv: 11556, 2019.
Casanueva, I., Temčinas, T., Gerz, D., Henderson, M., and Vulić, I. Efficient intent detection with dual sentence encoders. arXiv preprint arXiv:2003.04807 2020.
FitzGerald, J., Hench, C., Peris, C., Mackie, S., Rottmann, K., Sanchez, A., Nash, A., Urbach, L., Kakarala, V.. Singh, R., et al. Massive: A Im-example mul- tilingual natural language understanding dataset with typologically-divers languages. arXiv preprint arXiv:2204.0m 085822, 2022.
Changpinyo, S., Sharma, P., Ding, N., and Soricut, R. Con- ceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2558-3568, 2021.
Gao, L., la Tour, T. D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J. Scal- ing and evaluating sparse autoencoders. arXiv preprint arXiv:2406.0440 04093, 2024.
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, J. Reproducible scaling laws for con- trastive language-image learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2818-2829, 2023.
Geigle, G., Reimers, N., Rücklé, A., and Gurevych, I. Tweac: transformer with extendable qa agent classifiers. arXiv preprint arXiv:210.00088/ 2021.
Gu, J., Zhai, S., Zhang, Y., Susskind, J. M., and Jaitly, N. Matryoshka diffusion models. In The Twelfth Interna- tional Conference on Learning Representations, 2023.
Correia, G. M., Niculae, V., and Martins, A. F. Adaptively sparse transformers. arXiv preprint arXiv: 2019.
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Mo- mentum contrast for unsupervised visual representatton learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse autoencoders find highly inter- pretable features in language models. arXiv preprint arXiv:2309.086000 2023.
Deng, J., Dong, W., Socher, R., L...J, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
He, Y., Zhang, X., and Sun, J. Channel pruning for acceler- ating very deep neural networks. In Proceedings of the IEEE international conference computer vision, pp. 1389-1397, 2017.
Duan, Z. Wen, T., Wang, M., Chen, B., and Zhou, M. A non-negative vae: the generalized gamma belief network. arXiv preprint 2408 03388, 2024a.
Hinton, G., Vinyals, O., and Dean, Distilling the knowledge in a neural network. arXiv preprint arXiv: /5V3:/5003.225 503.02531,, 2015.
10

---

Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation
Hoogeveen, D., Verspoor, K. M., and Baldwin, T. Cqadup- stack: A benchmark data set for community question- answering research. In Proceedings of the 20th Aus- tralasian Document Computing Symposium (ADCS), ADCS '15, pp. 3:1-3:8, New York, NY, USA, 2015. ACM. ISBN 978- -78-1-4503-4440-3, doi: 10.1145/2838931.
Lectere, G., Ilyas, A., Engstrom, L., Park, S. M., Salman, H., and Madry, A. Ffev: Accelerating training by remov- ing data bottlenecks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2011--2020, 2023.
2838934. URL http:////// /tt////iii.o//////// acm. /ttg/11.11554 2838931. 2838934.
LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. nature, 521(7553):436 444, 2015.
Lee, C., Roy, R., Xu, M., Raiman, J., Shoeybi, M., Catan- zaro, B., and Ping, W. Ny-embed: Improved techniques for training Ilms as generalist embedding models. arXiv preprint arXiv:2405. 7428, 2024a.
Hou, L., Huang, Z., Shang, L., Jiang, X., Chen, X., and Liu, Q. Dynabert: Dynamic bert with adaptive width and depth. Advances in Neural Information Processing Systems, 33:9782-9799 2020.
Lee, H., Battle, A., Raina, R., and Ng. A. Efficient sparse coding algorithms. Advances in neural information pro- cessing systems, 19, 2006.
Hu, W. Dou, Z.-Y., Li, L. H., Kamath, A., Peng, N., and Chang, K.-W. Matryoshka query transformer for large vision-language models. arXiv preprint WrXiv:2405.19315 2024.
Lee, J., Dai, Z., Ren, X., Chen, B., Cer, D., Cole, R., Hui, K., Boratko, M., Kapadia, R., Ding, W., et al. Gecko: Versatile text embeddings distilled from large language models. arXiv preprint arXiv:2403.20322 2024b.
Hu, X., Duan, Z. Chen, B., and Zhou, M. Enhancing un- certainty estimation and interpretability with bayesian non-negative decision layer. In The Thirteenth In- ternational Conference on Learning Representations, 2025, URL https: httttss//ppeneeviee..etttt 1d=xJXq6FkQEw.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., al. Retrievallaugmmened generation for knowledge- intensive nlp tasks. Advances in Neural Information Pro- cessing Systems, $3:9459-94777 2020.
Huang, J., Hu, Z., Jing, Z., Gao, M., and Wu, Y. Piccolo2: General text embedding with multi-task hybrid loss train- ing. arXiv preprint arXiv:2405.066 06932. 2024.
Li, H., Arora, A., Chen, S., Gupta, A., Gupta, S., and Mehdad, Y. Mtop: A comprehensive multilingual task- oriented semantic parsing benchmark. arXiv preprint arXiv:2008. 00X3:25008.09335, 2020.
Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization and training of neural networks for efficient integer- arithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2704-2713 2018.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Johnson, J., Douze, M., and Jégou, H. Billion-scale similar- ity search with gpus. IEEE Transactions on Big Data, (3):535-547, 2019.
Lu, L., Shin, Y., Su, Y., and Karniadakis, G. E. Dying relu and initialization: Theory and numerical examples. arXiv preprint arXiv: 1903. 2019.
Kim, G. and Cho, K. Length-adaptive transformer: Train once with length drop, use anytime with search. arXiv preprint arXiv:2010.07003, 2020.
Maggie, Culliton, P., and Chen, W. Tweet sentiment extrac- com compet t ons/ tion. htter-eeetteeeeeeeettettttieeeee extractionnn.....e. 2020. Kaggle.
Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., and Liang. P. Concept bottleneck models. In International conference on machine learning, pp. 5338- 5348. PMLR, 2020.
Maia, M., Handschuh, S., Freitas, A., Davis, B., McDer- mott, R., Zarrouk, M., and Balahur, A. Www' 18 open challenge: financial opinion mining and question answer- ing. In Companion proceedings of the the web conference 2018, pp. 1941-1942, 2018
Kusupati, A., Bhatt, G., Rege, A., Wallingford, M., Sinha, A., Ramanujan, V., Howard-Snyder, W. Chen, K., Kakade, S., Jain, P., et al. Matryoshka representation learning. Advances in Neural Information Processing Systems, 35:30233-30243 2022.
Makhzani, A. and Frey, B. K-sparse autoencoders. arXiv preprint arXiv: 2013.

---

Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation
McAuley, J. and Leskovec, J. Hidden factors and hidden top- ies: understanding rating dimensions with review text. In Proceedings of the 7th ACM conference on Recommender systems. pp. 165-172, 2013.
Wachsmuth, H., Stede, M., El Baff, R., Al Khatib, K., Skeppstedt, M., and Stein, B. Argumentation synthe- sis following rhetorical strategies. In Proceedings of the 27th International Conference on Computational Linguistics, pp. 3753-3765. Association for Compu- tational Linguistics, 2018a. URL http://aclven org/anthol 1//CC/111111
Mirzadeh, I., Alizadeh, K., Mehta, S., Del Mundo, C. C., Tuzel, O., Samei, G., Rastegari, M., and Farajtabar, M. Relu strikes back: Exploiting activation sparsity in large language models. arXiv preprint 10X0:42/00 2023.
Wachsmuth, H.,, Syed, S., and Stein, B. Retrieval of the best counterargument without prior topic knowledge. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Pupers), pp. 241-251, 2018b.
Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316, 2022.
Wadden, D.,, Lin, S., Lo. K.. Wang. L. L.,, van Zuylen, M., Cohan, A., and Hajishirzi, H. Fact fiction: Verifying scientific claims. arXiv preprint arXiv:2004. 14974, 2020.
Nussbaum, Z., Morris, J. X., Duderstadt, B., and Mulyar, A. Nomic embed: Training a reproducible long context text embedder. arXiv preprint arXiv:2402.016613 2024.
OpenAL. New embedding models and api up- dates.
Wang, Y., Zhang, Q., Guo, Y., and Wang, Y. Non-negative contrastive learning. In The Twelfth International Confer- ence on Learning Representations, 2024.
https: /ttp://ppppaaaai/emeeee com/ index/ new-embedting-modee http:/mmodooaaaaapm----------ee- 2024.
Wightman, R. Pytorch image models. https: /gittbb. gith... com comm/hugginggfacc//pttoobb image- models, 2019.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, et.etal Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Wright, J., Ma, Y., Mairal, J., Sapiro, G., Huang, T. S., and Yan, S. Sparse representation for computer vision and pattern recognition. Proceedings of the IEEE, 98(6): 1031-1044, 2010.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., ct al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Xie, Y., Zeng. Z., Zhang, H., Ding, Y., Wang, Y., Wang, Z., Chen, B., and Liu, H. Discovering fine-grained visual-concept relations by disentangled optimal trans- port concept bottleneck models, 2025. URL https: // //arxiv. org/ abs 25005 07209.
Saravia, E., Liu, H.-C. T., Huang. Y.-H., Wu, J., and Chen, Y.SS. Carer: Contextualized affect representations for emotion recognition. In Proceedings of the 2018 confer- ence on empirical methods in natural language process- ing. pp. 3687-3697, 2018.
Yan, H., Hc, Y., and Wang. Y. The multi-faccted monose- manticity in multimodal representations. In Workshop on Responsibly Building the Next Generation of Mul- timodal Foundational Models, 2024a. URL https: / //openreviww.. net/ forum?idd Torumpid--NLLRoEELLn 9NLRowELnI.
Sturua, S., Mohr, I., Akram, M. K., Günther, M., Wang, B., Krimmel, M., Wang, F., Mastrapas, G., Koukounas, A., Wang, N., et al. jina-embeddings-'3 Multilingual embed- dings with task lora. arXiv preprint (..i:22409.1172.. 2024.
Yan, W., Zaharia, M., Mnih, V., Abbeel, P., Faust, A., and Liu, H. Elastictok: Adaptive tokenization for image and video. arXiv preprint arXiv: 2024b.
Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Freeman, C. D., Sumers, T. R., Rees, E., Batson, J., Jermyn, A., Carter, S., Olah, C., and Henighan, T. Scaling monosemanticity: Ex- tracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. URL https:
You, C., Mint, Y., Dai, W., Sekhon, J. S., Staib, I.., and Duncan, J. S. Calibrating multi multi-modal representations: A pursuit of group robustness without annotations. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 26140-26150. IEEE, 2024.
You, C., Dai, H., Min, Y., Sekhon, J. S., Joshi, S., and Dun- can, S. The silent majority: Demystifying memoriza- tion effect in the presence of spurious correlations. 2025. URL hwtps:ttp///....... /awxivivvv///////////////////ava wwtpiiv.orgabb/////eee 20500.0011...1.11.... 00961.
scal ing mono html.
12

---

Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation
Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Lin- guistics, 2:67-78, 2014.
Yu, P., Merrick, I., Nuti, G., and Campos, D. Arctic-embed 2.0: Multilingual retrieval without compromise. arXiv preprint 2rX1v:2412.04566, 2024.
Zhang, D., Li, J., Zeng, Z., and Wang, Jasper and stella: distillation of sota embedding models, 2025. URL https: / /arxivv..v/////////////////avav/ org/ abs //os/24122 2212.412222 19048.
Zhang, Z., Xu, Y., Yang, J., Li, X., and Zhang, D. A survey of sparse representation: algorithms and applications. IEEE access, 3:490-530, 2015.
Zhang. Z., Song, Y., Yu, G., Han, X., Lin, Y., Xiao, C.,, Song, C., Liu, Z., Mi, Z., and Sun, M. Relu wins: Discovering efficient activation functions for sparse llms. arXiv preprint arXiv:2402.03880 2024.
13

---

Beyond Matryoshka; Revisiting Sparse Coding for Adaptive Representation
A. Datasets
For Image embedding Experiment:
ImageNet-1K (Deng et al., 2009): ImageNet-Il IK is a large-scale visual database designed to provide researchers with comprehensive resource for developing and evaluating computer vision models. It contains 1,000 categories, each with a diverse set of images. Specifically, the dataset includes 1,281,167 training images, 50,000 validation images, and 100,000 test images.
For Text embedding Experiment: Note that, all datasets mentioned below can be found MTEB (Muennighoff et al., 2022).
MTOPIntent (Li et al., 2020): MТOР is a multilingual dataset introduced in 2021. comprises 100,000 annotated dialogue sentences across six languages and eleven domains. Designed serve as benchmark for multilingual task-oriented semantic parsing, this dataset plays crucial role in advancing technology in this field.
Banking77 (Casanueva et al., 2020): Dataset composed of online banking queries annotated with their corresponding intents, consisting of 13,083 customer service queries labeled with 77 intents.
TweetSentimentxxtracee (Maggie et al., 2020): Dataset from Kag gle competition. Sentiment classification of tweets as neutral, positive or negative.
MassiveScenario (FitzGerald al., 2022): collection of Amazon Alexa virtual assistant utterances annotated with the associated intent. For each user utterance the label is a theme among 60 scenarios like "music", 'weather', etc. This is multilingual dataset with 51 available languages.
AmazonReviews (McAuley & Leskovec, 2013): A collection of Amazonreviews designed to aid research in multilin- gual text classification. For each review the label is the score given by their view between and 4 (1-5 stars). This is a multilingual dataset with available languages.
Emotion (Saravia et al., 2018): The dataset consists of English Twitter messages categorized into basic emotions, including anger, fear, joy, love, sadness, and surprise.
ArxivClusteringgSS,, BiorxivClusteringe dataset is created using public APIs from bioRxiv. For S2S datasets, the input text is simply the title of the paper, while for P2P the input text is the concatenation of the title and the abstract.
(Muennighoff et al., 2022): The BioxivS2S
SreentyyewsgrousSCllsereiien Clustering of the 20 Newsgroups dataset, given titles of article the goal is to find the newsgroup (20 in total). Contains 10 splits, each with 20 classes, with each split containing between 1,000 and 10,000 titles.
ReddiiClustering22. (Muennighoff' et al., 2022): created for MTEB using available data from Reddit posts". The task consists of clustering the concatenation of title+post according to their subreddit. It contains 10 splits, with 10 and 100 clusters per split and 1,000 to 100,000 posts.
StackExchangeChhstering (Geigle al., 2021): Clustering of titles from 121 stack exchanges. Clustering of 25 splits, each with 10-50 classes, and each class with 100-1000 sentences.
FiQA2018 (Maia et al., 2018): A dataset for aspect-based sentiment analysis and opinion -bsed question answering in finance.
NFCorpus (Boteva et al., 2016): NFCorpus a full-text English retrieval data set for Medical Information Retrieval. It contains a total of 3,244 natural language queries, with 169,756 automatically extracted relevance judgments for 9,964 medical documents.
SciFACT (Wadden et al., 2020): A dataset of 1.4K expert-written claims, paired with evidence-containing abstracts annotated with veracity labels and rationales.
html
14

---

Beyond Matryoshka; Revisiting Sparse Coding for Adaptive Representation
Arguana (Wachsmuth al., 2018b): The dataset consists of debates from idebate.org. collected as of January 30, 2018. Each debate includes the thesis, introductory text, all points and counters, bibliography, and metadata.
CQaDupStack (Hoogeveen al., 2015): A benchmark dataset for community question-ansse answering research. contains threads from twelve StackExchange subforums, annotated with duplicate question information.
Quora Question Pairs*: A dataset consists of over 400,000 question pairs, and each question pair annotated with binary value indicating whether the two questions paraphrase of each other.
For Multimodal embedding Experiment:
MS COCO (Lin al., 2014): The MS COCO dataset a large-scale object detection, segmentation, and captioning dataset. It contains images with complex scenes involving multiple objects, each annotated with labels, bounding boxes, and segmentation masks.
Flickr30K (Young al., 2014): The Flickr30k dataset a collection of images with corresponding textual descriptions. Each image is annotated with multiple captions that describe the scene, objects, and actions depicted.
B. Experiment Detail on Vision Representation.
B.1. Evaluation Metric
We adopt 1-NN as our evaluation metric, implemented using FAISS (Johnson et al., 2019) with exact L.2 search, following setup (Kusupati et al., 2022). This approach provides an efficient and cost-effective way to evaluate the utility of learned representations for downstream tasks, as I-NN accuracy requires additional training. In detail, we use the training with 1.3M samples as the database and the validation set with 50K samples as the query set. We also report linear probing and few-shot results using Top-1 aceuracy. For a holistic evaluation, different methods, Figure I (c) presents the average 1 I-NN performance (active dimensions 64).
B.2. Baselines
We select MRL and MRL-E from (Kusupati et al., 2022) baselines. This work introduces novel training paradigm that learns representations of varying lengths. MRL-E is an efficient version of MRL, also proposed in (Kusupati et al., 2022).
B.3. Implementation Detail
For fair comparison, selected the pre-trained ResNet50 weights, noted as FF2048 in the MRL (Kusupati et al., 2022). Additionaly, select the ResNet50 model as SOTA backbone from Wightman (2019). For image preprocessing, we adopt the same procedure as described in Kusupati et al. (2022); Leclere et al. (2023). Consistent with Gao et al. (2024), we utilize a tied encoder-decoder structure to build the CSR framework. The implementation of CSR is based on the codebase provided by OpenAL. All experiments are conducted on server equipped with 4 RTX4090 GPUs. The selection of hyperparameters are:
Table 3. Implementation details Image experiment.
Backbone
h
Ir
epoch 10
Batch Size kaux
8
K
Optimizer weight decay
eps 6.25 1e-10 10
ResNet50 2048 8192
4e-5
4096
512
1/32
0.1 8,16,32...2048
Adam
1e-4
B.4. I-NN Classification Results
I-NN classification and Top-1 linear probing results are shown in Table 4 and Table 5.
inlk
"https://githun thub. com..ccm//ppoom http:b/ccoooopppaaaarrr.eeee
15

---

Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation
Table
1-NN accuracy of different methods ImageNetlk classification.
Active Dim
Full Rep.
MRL
MRL-E
SVD
Rand. FS
SAE
CSR
SOTA Full Rep.
CSR (w/ SOTA RN50)
8
62.19
57.45
19.14
2.36
67.14
67.78
73.84
16
67.91
67.05
46.02
12.06
68.14
69.17
74.39
32
69.46
68.60
60.78
32.91
68,91
70.15
74.53
64
70.17
69.61
67.04
49.91
69.69
70.94
74.62
128
70.52
70.12
69.63
60.91
69.74
70.99
74.65
256
70.62
70.36
70.67
65.75
70.35
71.31
74.73
512
70.82
70.74
71.06
68.77
71.21
71.29
74.88
1024
70.89
71.07
71.22
70.41
71.20
71.30
74.90
2048
71.19
70.97
71.21
71.21
71.19
71.24
71.20
75.19
74.91
Table 5. Top-1 classification accuracy results of different methods on ImageNetlk classification.
Active Dim
Full Rep.
MRL
MRL-E
MRL-LP
SVD
Rand. LP
SAE
CSR
SOTA Full Rep.
CSR (w/ SOTA RN50)
8
66.63
56.66
5.15
2.34
4.56
73.46
73.62
79.17
73.53
71.94
13.79
7.17
11.29
74.60
74.75
79.72
32
75.03
74.48
32.52
20.46
27.21
75.28
75.44
79.96
64
75.82
75.35
52.66
48.10
49.47
75.81
75.88
80.16
128
76.30
75.80
67.24
65.70
75.91
76.24
80.24
256
76.47
76.22
69.29
74.59
72.43
76.27
76.25
80.31
512
76.65
76.36
70.51
76.78
74.94
76.43
76.34
80.33
1024
76.76
76.48
70.19
76.87
76.10
76.59
76.54
80.32
2048
76.87
76.80
76.51
69.72
76.87
76.66
76.52
80.59
80.35
C. Experiment Detail Text Representation
C.1. Evaluation Metric
We adopt the universal evaluation metrics used the MTEB benchmark (Muennighoff et al., 2022). For text classification and clustering, we use Top-1 accuracy to assess model performance. For the text retrieval task, we use NDCG@ 10 (Normalized Discounted Cumulative Gain at 10), metric that evaluates the quality of ranked list of items, commonly used in information retrieval and recommendation systems.
C.2. Experiment Setup
We choose three main tasks on MTEB benchmark and randomly select six datasets(for each task) measure our methods. We also design two experiment settings evaluate the effectiveness and generalization ability of our methods.
Firstly, we introduce Dpteset-Specific Evaluation, where CSR are trained and tested on different splits of the same dataset. We use MTOPIntent (Li et al., 2020), Banking77 (Casanueva et al., 2020) and TweetSeniimenExtraattion (Maggie et al., 2020) for text classification task. We use BiorxivClusternngSSS, BiorxivClusteringP2P (Muennighoff et al., 2022) and Twen- tyNewsgroupdClustriin for text clustering. For text retrieval, we select FiQA2018 (Maia et al., 2018), NFCorpus (Boteva al., 2016) and SciFACT (Wadden al., 2020).
Furthermore, we introduce Task-Specific Evaluation, where CSR are trained and tested different datasets within the same task to evaluate the generalization ability of proposed method. We construct training dataset using the training splits of the aforementioned datasets and test on the corresponding task datasets. For classification: MassivScenario (FitzGerrad et al., 2022), ReazonRevies (McAuley & Leskovee, 2013) and Emotion (Saravia et al., 2018). For clustering: ArxivCluster- ingS2S, ReddiiClusteringP2P (Muennighoff et al., 2022) and StackExchangeClusterim (Geigle et al., 2021). For retrieval: Arguana (Wachsmuth et al., 2018a), CQADupStack (Hoogeveen et al., 2015) and Quora.
C.3. Baselines
We choose several models that provide MRL embeddings on MTEB benchmark (Muennighoff et al., 2022). These models are Beella-en-1.5B..o (Zhang et al., 2025), Jina-V V3 (Sturua et al., 2024), Homic-Embed V1.5 (Nussbaum et al., 2024), Gecko- Text-Embeeding-0044222 (Lee al., 2024b),
2--2 (OpenAI, 2024), Aretic-Embed-L-VV (Yu
16

---

Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation
al., 2024) and Potion-Base-2M (min, 2024).
C.4. Implementation Detail
We select NV-Embed-V. V2 (Lee et al., 2024a) as our pre-trained model. We utilize a tied encoder-decoder structure to build the CSR framework. For text classification and clustering tasks, we use data from the same class as positive samples while the other as negative samples to calculate Equation 6. The hyperparameters are set follows:
Table 6. Implementation details Text experiment.
Backbone d
h
Ir
epoch
Batch Size
kaux
B
IK
Optimizer
weight decay
eps 6.25
NV-Embed-V2 V2 4096
16384
4c-5
10
128
1024
0.1
1.0 32,64,256
Adam
1c-4
le-10
D. Experiment Detail on MultiModal Representation
D.1. Evaluation Metric
We adopt the universal evaluation metric Recall@5 to measure performance in the MultiModal Retrieval task. This metric evaluates a model's ability to retrieve relevant items within its top predictions. Calculated as the fraction of relevant items appearing in the top 5 results out of the total relevant items, higher Recall indicates better performance in capturing relevant content early in the ranked list, making useful for recommendation systems and retrieval tasks.
D.2. Experiment Setup
We selected ViT-B-16, trained on the DFN2B dataset", as our pre-trained model. For the in-distribution cross-mode retrieval experiment, implemented MRL in the pre-trained ViT model following Kusupati al. (2022), and fine-tuned it for 50 epochs the MSCOCO (Lin al., 2014) and Flickr30K (Young et al., 2014) datasets, respectively. For fair comparison, we also fine-tuned the backbone on both datasets for 50 epochs using the same hyperparameters, which were then used for the backbone of CSR. The hyperparameters used for fine-tuning are as follows:
Table 7. Hyperparameters for fine-tuning ViT-B/16 backhone.
Dataset
Ir
epoch
Batch Size
warmup
Optimizer weight decay
MS COCO
5e-6
50
10000
Adam
0.1
Flickr30k
5e-6
50
64
10000
Adam
0.1
For zero-shot cross-modal retrieval, we employed the same MRL fine-tuning procedure as in our in-distribution experiment, maintaining identical hyperparameters while training for epochs with 2208 batch size on CC3M (Changpinyo et al., 2021).
D.3. Implementation Detail
We select the ViT-B-16' as our backbone from Wightman (2019). Consistent with Gao et al. (2024), we utilize a tied encoder-decoded structure to build the CSR framework. The encoder and decoder structure share between image space and text space. The implementation of CSR is based on the codebase and OpenCLIP (Cherti et al., 2023). The metric is evaluated through CI.IP-benchmark following standard procedure. All experiments are conducted on server equipped with 4 RTX4090 GPUs. We present detailed training parameters for the multimodal experiment in Table 8.
D.4. Discussion On Dead Latents
Addressing the mitigation of dead latents the alignment space remains an open challenge, leaving room for future work and study. Table presents the performance comparison between CSR and MRL, revealing that the gap between the two
hattes///hogginnaaac...////////p hhttps://githubs http://ttttuu..oooommooeee http:///w//oggaaagaaaaaaaattoona
17

---

Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation
Table Implementation details
MultiModal experiment.
Dataset
d
h
Ir
epoch
Batch Size
kaux
B
IK
Optimizer weight decay
cps
MS COCO
512 2048
4e-4
5
256
512
1/32
1.0
64,128,256
Adam
le-4
6.25 1e-10
Flickr30k
512 2048
4e-4
5
64
1024
1.0
1.0
64,128,256
Adam
le-4
6.25 * 1e-10
CC3M
512 4096
4e-4
1
1024
1024
1/32 1.0
64,128,256
Adam
0.0
6.25 1e-10
methods diminishes sparsity constraints become more stringent. Further analysis indicates that CSR continues to face the "dead latents" issue despite incorporating advanced mechanisms. As shown in Figure 8, CSR exhibits a significant performance drop, corresponding to sharp rise in dead latent dimensions. We attribute this to a technical challenge. as CSR has demonstrated robust performance in both image and text domains under similar sparsity constraints. This suggests that representations in alignment spaces may require more specialized design, presenting an opportunity for future research and improvement.
80
Fraction of dead latent
0.8
Recall@5 (%)
60
Image-to-Text
0.6
40
Text-to-mmage Image
Dead Latents
0.4
20
0.2
0
16.32
64
128
256
TopK
Figure 8. Dead latents still exits in image-text alignment space.
E. Empirical Analysis
E.1. Effect on Input Embedding Dimension IRd
The implementation details are shown in Table 9. To avoid other unknown factors, we choose ViT-based and ResNet-based models following same pre-training procedure respectively. To ensure generalizability, we train the model using three different random seeds and report the mean performance
main paper.
Table Implementation details on empirical study input embedding dimension Rd
Backbone
Ir
epoch
Batch Size
kaux
7
KK
Optimizer
weight decay
eps
VIT-L/16
512
4096
4e-5
10
1024
512
1/32
1.0
8,16,64,256
Adam
lc-4
6.25 le-10 10
ViT-L/16
1024
4096
4e-5
10
1024
512
1/32
1.0
8,16,64,256
Adam
le4
6.25 1e-10
ResNet18
512
8192
4e-5
10
1024
512
1/32
1.0
8,16,64,256
Adam
le-4
6.25 * le-10
ResNet50
2048
8192
4e-5
10
1024
512
1/32
1.0
8,16,64,256
Adam
lc-4
6.25 * lc-10
inlk
18

---

Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation
Table 10. Implementation details empirical study of hidden dimension R
Backbone
Ir
epoch
Batch Size
kaux
g
?
Optimizer
weight decay
eps
1024
1024
4e-5
10
1024
512
1/32
1.0
8,16,6,,256
Adam
le-4
6.25 * le-10
1024
2048
4e-5
10
1024
512
1/32
1.0
8,16,64.256
Adam
le-4
6.25 * le-10
ViT-L/16
1024
4096
4e-5
10
1024
512
1/32
1.0 8,16,64,256
Adam
lc-4
6.25 * le-10
1024
8192
4e-5
10
1024
512
1/32
1.0
8,16,642256
Adam
le-4
6,25 1e-10
1024
16384
4e-5
10
1024
512
1/32
1.0
8,16,64225
Adam
le-4
6.25 1e-10
2048
2048
4e-5
10
1024
512
1/32
1.0
8,16,64226
Adam
le-4
6,25 le-10
2048
4096
4e-5
10
1024
512
1/32
1.0
8,16,64,256
Adam
le-4
6,25 * le-10
ResNet50
2048
8192
4e-5
10
1024
512
1/32
1.0
8,16,64,256
Adam
le-4
6.25 1e-10
2048
16384
4e-5
10
1024
512
1/32
1.0
8,16,64,256
Adam
le-4
6.25 lc-10
2048
32768
4e-5
10
1024
512
1/32
1.0
8,16,64,256
Adam
le44
6.25 1e-10
E.2. Effect on Hidden Representation Dimension I
Implementation details are shown in Table 10. The pre-trained ViT-L/16 and ResNet50 models can be found at timm (Wightman, 2019). To ensure generalizability, we train the model using three different random seeds and report the mean performance the main paper.
E.3. Retrieval Time Evaluation
We employ PyTorch (Paszke et al., 2019) to measure retrieval time on ImageNetlk. The average retrieval time is computed over 2000 rounds with a batch size of 512 queries, excluding an initial 100 warm-up rounds. For the learned CSR representation, both query and key embeddings are stored esr format, and sparse product operations are utilized for similarity computation while maintaining identical experimental settings for fair comparison.
E.4. Understanding Retrieval Time Difference between Dense and Sparse Embeddings
Although CSR and MRL have similar theoretical complexity Okk), their actual runtimes are affected by backend implemen- tations. For instance, cuBLAS (used for dense ops) highly optimized but has high launch overhead, while cuSPARSE (used for CSR) is lighter but less optimized for small k. Here, can be faster than dense embeddings and why it can get faster with larger hidden dimension h.
share preliminary insight into why sparse embeddings
Sparse matrix multiplication benefits from zero-skipping: only overlapping non-zero entries are used. For each query, computing the i-th output only involves comparing indices of non-zero entries-an an integer operation much cheaper than foating-point multiplication. As increases and & stays small, overlap likelihood drops, reducing the number of multiplications required. Table We empirically verify this by counting the number of multiplications under various
Table 11, Comparison on the number of multiplication operation between MRL (dense) and CSR (embeddings) the default setup.
Active Dim
MRL
CSR (h = 8192)
CSR (h 16384)
CSR (h 32768)
1.3 x 109
3.2 x 105
1.7 x 10
8.4 x 10a
2.6 x 109
1.3 108
6.7 10t
3.4 10a
The number of operations in CSR several orders magnitude smaller than in MRL, and decreases with larger h. This counterintuitive yet practical effect highlights the appeal of using sparse high-dimensional embeddings: they allow richer representations while improving runtime.
2_224__ugreeg_iii.... in21k_11kk 1na fo2kk_f__nn ft_in'k
"https://hugginggfee
19


Reviews:
[{"review_id": "seV8oVTnbo", "reviewer": "Reviewer_11DH", "summary": "This paper focuses on the problem of creating adaptive representations from foundation models, focusing on contrastive sparse coding (CSR) as a novel method applied after pre-training to produce efficient representations for a range of downstream tasks. CSR is compared with Matryoshka Representation Learning (MRL), which creates dense representations at multiple scales by truncating dense feature vectors. In contrast to MRL, CSR creates sparse representations where a target number of dimensions activates per input. The paper demonstrates several advantages of CSR, including higher fidelity, faster retrieval, and lower training cost compared to MLR and other simple baselines. The experiments focus on image, text, and text-image tasks, showing consistent gains over MRL in all cases. Ablations analyze the design choices, including sparsity level, input and hidden dimensionality, and data scaling. The method seems practically useful as a simple post-training method for which.one can tradeoff computational efficiency with accuracy reasonably in some relevant downstream tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive comments and suggestions, which are helpful for us to improve the quality of our paper further. The concerns have been addressed as below:\n\n---\n**Q1** May be useful to add a discussion on pruning, quantization, and distillation methods.\n\n**A1.** Indeed, CSR, pruning, quantization, and distillation all enable acceleration.\nHowever, while the other three methods accelerate the backbone and inference embedding generation, our approach (CSR) focuses on post-processing optimization from embedding to retrieval. Because of this, CSR is **orthogonal** to those other methods. \n\nFor example, combining CSR with Int8 quantization, as shown in *Table 1* below, provides additional speed-up beyond quantization alone, with minimal loss in the compressed model’s performance. This benefit arises because CSR utilizes a significantly smaller number of active dimensions (e.g., 8) while maintaining overall effectiveness. Compared to other methods, applying CSR to the original approach results in the highest retention of performance. Thanks for bringing this up, and we will include this discussion in the revised version.\n\n*Table 1: 1-NN Acc Comparison on Different Methods*\n\n|Method| Active Dim | Vanilla | Int8 Quant | Retrieval Time |\n|-|-|-|-|-|\n|Resnet50| 2048|75.19|73.48|5.17|\n|+CSR|8|73.84|72.32|0.24|\n|Resnet50|2048|70.97|69.11|5.16|\n|+MRL|8|62.19|59.38|0.42|\n\n---\n**Q2** Ablation study on fidelity at lower sparsity levels.\n\n**A2.** Please see A6. \n\n---\n**Q3** Whether wider hyperparameter searches across design factors may help reduce approximation error?\n\n**A3.** Thank you for raising this point. CSR's primary influential parameters are the activation dimension $k$ and the hidden dimension $h$, both of which we analyze extensively in **Figure 5**. \nAs for other standard training hyperparameters (learning rate, loss coefficients, etc.), Gao et al. [1] conducted extensive ablations in the context of SAEs, and we adopted their default configuration. \nWe will include this clarification in the revised version.\n\nRef: \n\n[1] Gao, Leo, et al. \"Scaling and evaluating sparse autoencoders.\" arXiv preprint arXiv:2406.04093 (2024).\n\n---\n**Q4** Further evaluation in linear probes and few-shot scenarios.\n\n**A4.** Good question! Following your suggestions, we have conducted additional experiments evaluating CSR with linear probing (*Table 2*) and few-shot learning (*Table 3*). \n\n*Table 2* shows the Top-1 accuracy of CSR under linear probing on ImageNet1K, while *Table 3* reports 1000-way {3-, 5-, 7-}-shot average performance across three test sets on ImageNetV2.\nWe utilize the same backbone architecture as MRL [2] in the few-shot scenario for a fair comparison.\n\n*Table 2* compares the linear probing performance of CSR and MRL across varying active dimensions, demonstrating that CSR exhibits minimal performance degradation relative to MRL. \nA similar trend appears in *Table 3*, where CSR shows stronger robustness in few-shot classification. \nThese additional results underscore CSR’s consistently high performance across different downstream tasks beyond 1-NN. We hope this addresses your concerns, and we welcome any further discussion.\n\n*Table 2: Linear probing performance comparison between different methods.*\n\n|Methods |Active Dim|Top-1 Acc|\n|-|-|-|\n|ResNet50|2048|80.59|\n|+CSR|128|79.76|\n|+CSR|32|78.94|\n|+CSR|8|78.60|\n| ResNet50|2048|76.80|\n|+MRL|128| 76.30|\n|+MRL|32| 75.03|\n|+MRL|8|66.63|\n\n*Table 3: Few-shot performance comparison between different methods.*\n\n| Methods  | Active Dim | 3-Shot | 5-Shot | 7-Shot |\n|-|-|-|-|-|\n|ResNet50|2048|0.57|0.62|0.65|\n|+MRL|8|0.52|0.55|0.57|\n|+CSR|8|0.56| 0.61|0.63|\n\nRef:\n\n[2] Kusupati, Aditya, et al. \"Matryoshka representation learning.\" NIPS,2022.\n\n---\n\n**Q5** Referation to Wang et al's Theorem 5.\n\n**A5.** Thanks for the reminder. We will revise the statement and include proper reference to Wang et al.\n\n---\n\n**Q6** What makes TopK=16 a “sweet spot”?\n\n**A6.** Thank you for highlighting this point. Here, we consider $k=16$ as the ''sweet spot'' because it attains the optimal balance between accuracy and efficiency. As demonstrated in Figures 4 and 5, $k=16$ outperforms smaller $k$ values (e.g., 8) while maintaining higher efficiency than MRL (Figure 3(b)). *Table 4* further confirms that $k=16$ is a strong trade-off point.\nNevertheless, using a smaller $k$ remains an option for those who prioritize speed over performance.\n\n*Table 4: Performance vs Efficiency under Different Sparsity*\n\n|Top K||2|4|8|16|32|\n|-|-|-|-|-|-|-|\n|1-NN Acc|| 66.17|69.97|73.84|74.39|74.53|\n|Relative Retrieval Time||0.2|0.2|0.3|0.5|1.4|\n\n---\n\nWe are sincerely grateful for your valuable feedback, which has been instrumental in refining our manuscript. Should you have any additional comments or questions, please do not hesitate to contact us.", "claims_and_evidence": "The paper does a good job of comparing CSR to popular adaptive representation baselines within the MRL family along with some simpler baselines, though it may be useful to also compare with or at least discuss other approaches that focus on pruning, quantization, and/or distillation. The method shows clear gains over MRL in most cases examined across the sparsity spectrum both in terms of fidelity and runtime, though there is some loss in fidelity — the degree to which this matters is probably out of scope of this contribution, but is important to acknowledge.", "methods_and_evaluation_criteria": "Methods and evaluation criteria appear comprehensive, and not designed to favor the CSR method from what I can tell. That being said, ablation studies could go deeper, e.g., to try to understand the limitations e.g. in fidelity at lower sparsity levels of the sparse coding approach, and whether wider hyperparameter searches across design factors may help reduce approximation error further.", "theoretical_claims": "The paper did not introduce any new theoretical claims, though they did make use of Wang et al, Theorem 5. Given that the empirical evidence backed up the theoretical claim, I do not have specific concerns here. Of course, additional theory characterizing the limits of this CSR method would be welcome!", "experimental_designs_or_analyses": "No specific concerns here. It could be useful to examine whether CSR holds in other representation learning settings beyond 1-KNN probes. What about linear probes and/or few shot scenarios?", "relation_to_broader_scientific_literature": "The paper relates to recent sparse autoencoder findings on interpretability that gained some interest in the past year, and may serve to interest researchers beyond that particular use. There’s also a potential connection to other recent works on sparsity including sparse mixtures-of-experts. In addition, there is a wide literature on post-training methods including pruning, distillation, quantization, etc., that could be discussed given the practical importance to efficiency, especially on device!", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The application of modern sparse coding (with contrastive task regularization) to adaptive feature representations is not something I’ve seen, which could be of good practical relevance in efficiency research. That being said, there is not much in the way of theoretical or empirical insights from this paper.", "other_comments_or_suggestions": "Since Theorem 5 is taken from Wang et al and is not a conribution of this paper, make sure to refer to it as such in the text anywhere it’s mentioned to minimize confusion.", "questions_for_authors": "From Figure 3, it’s not clear to me what makes TopK=16 a “sweet spot”. In terms of retrieval time, it is the highest, though it is indeed the case that benefits over MRL baseline are maintained. Can you explain why this is considered a sweet spot?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L8WvPvKW0P", "reviewer": "Reviewer_cLDh", "summary": "The authors propose a method for converting pretrained dense embedding vectors into sparse embedding vectors and show that it often outperforms standard approaches such as Matryoshka Representation Learning (MRL) in terms of both accuracy, training time and retrieval speed.\n\nTheir CSR method is inspired by Sparse Autoencoders and projects fixed embeddings into a higher-dimensional space and activating only the TopK dimensions for a compact representation. CSR uses an SAE style reconstruction loss as well as a non-negative contrastive loss.\n\nThey compare across various domains, architectures and tasks. This is an exciting contribution to the field and has the potential to become the new standard for efficient vector representations.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and for appreciating the quality of our work. The concerns have been addressed as below:\n\n---\n\n**Q1** Typos in Figure 1 caption, Section C.3, and Table 5.\n\n**A1.** Thank you for pointing out! Following your great suggestions, we will fix them in the revised manuscript.\n\n---\n\n**Q2** It would be helpful to state that embedding models in Table 1 use MRL.\n\n**A2.** Thank you for your great suggestion! We will clarify this distinction in Table 1 in the revised manuscript.\n\n---\n**Q3** Releasing Codebase.\n\n**A3.** We will certainly open-source the entire project upon acceptance. For now, we have released the [codebase](https://anonymous.4open.science/r/ICML_rebuttal-78D1/) of the ImageNet results as a reference implementation. \n\n---\n**Q4** How many active dims in Figure 1(c)?\n\n**A4.** Here, we computed the average 1-NN accuracy across active dimensions 8 to 128 for each method to have a holistic evaluation. We will add this explanation to the experiment setup in Appendix B for further clarification.\n\n---\n**Q5** In Figure 7(c), are the results for active dimensions 32?\n\n**A5.** Thank you for pointing out! Indeed, the results in Figure 7(c) are for active dimensions 32. We will explicitly clarify this detail in the figure caption in the revised manuscript.\n\n---\n\n**Q6** In Figure 7(c), it would be helpful to see the full performance of each model (e.g., NV Embed, Nomic-v1.5 etc.) with the full vector representation.\n\n**A6.** Thank you for pointing this out! We will include the full performance of each model for better reference. One major advantage of CSR compared to these MRL models is that it is a very lightweight method and can be easily built upon the latest SOTA embedding models, while the others have to rely on their own heavy pretraining, which leads to inferior full representation performance as well. \n\nTherefore, the advantages of CSR are essentially two folds: 1) we can use off-the-shelf SOTA embedding models for the best full representation performance, and 2) we have the slightest degradation when converted to efficient embeddings in a lightweight way.\n\n---\n**Q7** Briefly discuss MRL-E, SVD and Rand-LP in the main text.\n\n**A7.** Thank you for this valuable suggestion! We will elaborate on their setups in the main text as well in the revision.\n\n---\n**Q8** It would be helpful to elaborate on Section E.3 with regard to retrieval time evaluation.\n\n**A8.** Thanks for your valuable suggestion! Here is a more detailed breakdown of the evaluation protocol for the retrieval time:\n1. We first precompute embeddings for the entire ImageNet training set, storing them in a standard CSR (compressed sparse row) format in GPU memory as the retrieval database.\n2. We compute the retrieval time as the average over 2,000 retrieval rounds, following an initial warm-up period of 100 rounds. In each retrieval round, the query set consists of 512 samples randomly drawn from the database. The warm-up phase is standard practice when benchmarking GPU computations, as it effectively eliminates initialization bias.\n\n---\n**Q9** It is slightly confusing that the method is called CSR (contrastive sparse representation), while the embeddings are stored in the CSR (compressed sparse row) format. I assume this was intentional - if so, it could be worth explicitly mentioning this by saying something like “the training method is called CSR, and the vector is stored in the standard CSR format…”\n\n**A9.** Indeed, we intentionally call our method CSR (contrastive sparse representation) to be the \"ML version\" of CSR, i.e., how to learn a model that converts dense embeddings to highly sparse ones. And thanks for your suggestions -- we will add the explicit discussions on this terminology connection and distinction to be clearer.\n\n---\n\nWe are genuinely grateful for your thoughtful feedback, which has been instrumental in helping us refine the manuscript. Please do not hesitate to reach out if you have any additional comments or questions.", "claims_and_evidence": "The authors claim that Contrastive Sparse Representation (CSR) \n1. Has higher accuracy than MRL when controlling for the number of active dimensions\n* Figure 7(a)-(b) on ImageNet 1-NN accuracy\n* Table 2 on MS COCO and Flickr30k (image to text and text to image)\n* Figure 7(c) on a subset of the MTEB retrieval dataset\n* Table 1 on a subset of data from MTEB\n2. Has shorter training time than MRL\n* Figure 1(c)\n3. Has faster retrieval time than MRL\n* Figure 1(b) \n* Table 1 on subset of data from MTEB", "methods_and_evaluation_criteria": "The proposed methods and benchmark datasets make sense for the problem at hand. They do a very thorough comparison against Matryoshka Representation Learning (MRL) by evaluating across similar domains and benchmarks used in the original MRL paper.", "theoretical_claims": "The authors reference one theoretical proof for motivation from Wang et al. 2024, but do not rely on this for their experimental results. I did not check the correctness of this theorem.", "experimental_designs_or_analyses": "The authors have done a good job of benchmarking their approach across multiple modalities and benchmarks. They also benchmark against prior work (MRL by Kusupati et al. 2022)", "relation_to_broader_scientific_literature": "The results in this paper are related to the broader literature around efficient vector representations. The main target of their approach, MRL, is widely used by modern embedding models including OpenAI’s text-embedding-3-large. Their approach has the potential to become the new standard for efficient representations.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "This work is potentially quite significant for the field of retrieval, text embedding and image embeddings.", "other_comments_or_suggestions": "Typos:\nFigure 1 caption “Compared to MLR” should be “MRL”\nFigure 1 caption “we outperform MSR on 1-NN accuracy” should be “we outperform MRL”?\nSection C.3 “MTEB benchmar” should be “benchmark”\nTable 5 “NV-EmbeV2” should be “NV-EmbedV2”\n\nThe embedding models in Table 1 all use MRL; while this is stated clearly in the supplementary material, it would be helpful to state this explicitly in the main text.\n\nMy review is contingent on the authors releasing the code for CSR so that others can replicate their results.", "questions_for_authors": "Figure 1 (c) how many active dims for the data points in this plot? Is this the same data as in Table 1? If so please state this explicitly.\n\nIn Figure 7(c), are the results for active dimensions 32? If so, please state this explicitly in the caption. Currently the caption hints at this by stating “the results of CSR-32…”.\n\nIn Figure 7(c), it would be helpful to see the full performance of each model (e.g. NV Embed, Nomic-v1.5 etc.) with the full vector representation. This would remind the reader that sparse representations with low active dimensions usually have lower accuracy than the full embedding.\n\nIt would be helpful to briefly discuss MRL-E, SVD and Rand-LP in the main text (and not just in the supplementary materials), as these appear in multiple figures and tables.\n\nIt would be helpful to elaborate on the Section E.3 with regards to retrieval time evaluation. \n\nIt is slightly confusing that the method is called CSR (contrastive sparse representation), while the embeddings are stored in the CSR (compressed sparse row) format. I assume this was intentional - if so, it could be worth explicitly mentioning this by saying something like “the training method is called CSR, and the vector is stored in the standard CSR format…”", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jE5YWFNNNF", "reviewer": "Reviewer_cpsE", "summary": "The paper presents Contrastive Sparse Representation (CSR) as a novel approach to adaptive representation learning, addressing the limitations of Matryoshka Representation Learning (MRL), which requires extensive retraining and suffers from performance degradation at shorter representation lengths.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your thoughtful assessment of our paper. We appreciate the recognition of our work's contributions, particularly noting the clear improvements of CSR over MRL by ``Outperforms MRL in Accuracy and Speed Evidence``, ``Reduces Training Time Significantly``. \n\nMeanwhile, thank you for acknowledging that our ``methods and evaluation criteria are well-aligned with the problem of adaptive representation learning`` and provide ``comprehensive assessment of its performance across multiple dimensions``.\n\nNext,  we will address your comments on the aspects for further improving this work.\n\n---\n\n**Q1** Documentation for reproducibility.\n\n**A1.** Experimental details with key hyperparameter settings are provided in Appendix B.3, C.4, and D.3. We have included preliminary ([code](https://anonymous.4open.science/r/ICML_rebuttal-78D1/)). and will definitely complete the resources available for reproducibility.\n\n---\n**Q2**. Broader comparisons with other methods.\n\n**A2.** Thanks for your suggestions! We acknowledge that several other methods (e.g., pruning, quantization, and distillation) also enable acceleration. \nHowever, it is important to clarify that while these methods primarily focus on accelerating the backbone model and embedding generation, CSR distinctively focuses on optimizing the post-processing phase, specifically the transition from embedding to retrieval. This distinction positions CSR as fundamentally **orthogonal** to existing acceleration approaches.\n\nTo illustrate, we combine CSR with Int8 quantization, as demonstrated in the *Table 1* below. This combination achieves additional acceleration beyond quantization alone while incurring only minimal degradation in the compressed model’s performance. \nThe effectiveness of CSR arises from its capability to maintain high performance even when employing significantly reduced active dimensions (e.g., 8). Compared to alternative approaches, applying CSR to the original model consistently results in the highest performance retention. \nWe will incorporate this detailed discussion into the revised manuscript.\n\n*Table 1: 1-NN Acc Comparison on Different Methods*\n\n| Method    | Active Dim | Vanilla | Int8 Quant | Retrieval Time |\n|-----------|------------|---------|------------|----------------|\n| Resnet50  | 2048       | 75.19   | 73.48      | 5.17           |\n| +CSR      | 8          | 73.84   | 72.32      | 0.28           |\n| Resnet50  | 2048       | 70.97   | 69.11      | 5.16           |\n| +MRL      | 8          | 62.19   | 59.38      | 0.42           |\n\n---\n**Q3** Statistical significance testing.\n\n**A3.** Thank you for your suggestion! Due to time constraints, we firstly report the standard division results in *Table 2* below for the comparison under 8 active dimensions, calculated with 5 independent runs. We can see that the stdev is only around 0.5, while the gap between MRL and CSR can be as large as 15.9 (for MRL, we adopted the original paper's results that did not report stdev). Consequently, our method consistently outperforms MRL by a significant margin.\n\n*Table 2: CSR statistical results on ImageNet1k*\n\n| Methods | Active Dim | 1-NN Acc     |\n| ------- | ---------- | ------------ |\n| MRL     | 8          | 62.19        |\n| MRL-E   | 8          | 57.45        |\n| CSR     | 8          | 73.39 ± 0.51 |", "claims_and_evidence": "Claim 1: CSR Outperforms MRL in Accuracy and Speed\nEvidence: Compared to the MRL method, CSR achieves average performance gains of 4.6% and 6.8% on image-to-text retrieval, and 9.1% and 6.5% on text-to-image retrieval across the two datasets. This indicates that CSR provides higher accuracy while maintaining efficient retrieval times.\nClaim 2: CSR Reduces Training Time Significantly\nEvidence: The training time for CSR is reported to be a fraction of that required by MRL. For instance, CSR can be trained on ImageNet in about half an hour with a single GPU, compared to the extensive retraining needed for MRL.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-aligned with the problem of adaptive representation learning. CSR effectively addresses the limitations of previous approaches, and the chosen metrics provide a comprehensive assessment of its performance across multiple dimensions. This makes the methodology robust and applicable to real-world scenarios, thereby contributing valuable insights to the field.", "theoretical_claims": "Several theoretical claims are made regarding the effectiveness and efficiency of Contrastive Sparse Representation Learning (CSR). However, the paper does not provide formal proofs for these claims.", "experimental_designs_or_analyses": "The experimental designs and analyses in the paper are largely valid and appropriate for evaluating CSR. \nHowever, improvements could be made in terms of documentation for reproducibility, broader comparisons with other methods, and the inclusion of statistical significance testing.", "relation_to_broader_scientific_literature": "no", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "no", "other_comments_or_suggestions": "no", "questions_for_authors": "no", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iGGSUjLIaZ", "reviewer": "Reviewer_PD5c", "summary": "In this paper, the authors propose Contrastive Sparse Representation (CSR) as an alternative to Matryoshka Representation Learning (MRL) for adaptive embeddings. MRL requires retraining models and suffers from performance drops at shorter embedding lengths, while CSR achieves adaptive representation through sparse coding, preserving high-dimensional semantic quality. CSR combines reconstruction-based sparse autoencoding with a contrastive loss to maintain accuracy and retrieval efficiency at various sparsity levels. Experiments on image, text, and multimodal benchmarks show that CSR outperforms MRL in accuracy and retrieval speed while requiring significantly less training time (up to 69× faster). CSR maintains the semantic integrity of the original embeddings and achieves strong generalization across downstream tasks with fewer computational resources, and it is a more efficient and scalable approach for adaptive representation learning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful reading and critical review. Following your suggestions, we have added more discussions on complex multimodal generation ability and the scalability of CSR. We further address each of your concerns below and hope you find them satisfactory.\n\n---\n\n**Q1** Move extensive technical details (especially Section 3.2.2) to the Preliminaries or Related Work section to highlight the paper's core innovations.\n\n**A1.** We sincerely appreciate this suggestion. As you suggested, we will move the background part of Sec 3.2.2 to Preliminaries for better readability. In this way, Sec 3.2 would be mostly devoted to the design of CSR and highlight the novelty of our design.\n\n---\n**Q2** CSR's ability to handle complex multimodal tasks(e.g., cross-modal generation).\n\n**A2.** In this work, our primary goal is to develop a more efficient approach to representation learning. To evaluate the quality of learned representations, we follow the standard evaluation protocol in the efficient representation learning literature, such as MRL [1], focused mainly on well-known image (e.g., ImageNet) and text embedding (MTEB [2]) benchmarks.\n\nWe further evaluate CSR on multimodel embeddings as well in Table 2 (in the main text) and *Table 1* below (zero-shot retrieval performance) and follow common evaluation protocols of CLIP embeddings (e.g.,  CLIP\\_benchmark [3]) with standard metrics like 1-NN accuracy, NDCG@10, and Recall@5.\nAlthough CLIP embeddings can be used for multiple downstream tasks (including generation), our focus here is not to explore these alternatives, and thus we follow the standard protocol of CLIP evaluation for our experiments. Please let us know if you need any further clarification.\n\nRef:\n\n[1] Kusupati, Aditya, et al. \"Matryoshka representation learning.\" NIPS,2022.\n\n[2] Muennighoff, Niklas, et al. \"MTEB: Massive text embedding benchmark.\" arXiv preprint arXiv:2210.07316 (2022).\n\n[3] https://github.com/LAION-AI/CLIP_benchmark\n\n---\n**Q3** The scalability of CSR on extremely large datasets (beyond ImageNet or MS COCO scale) is not tested, which limits its generalizability to real-world large-scale applications. \n\n**A3.** Our evaluation on ImageNet-1k follows the standard setup in MRL, which also did not have results that far beyond this scale as well. Indeed, **with constraints of our academic compute**, ImageNet-1k is already a large-scale dataset for us and it is not really feasible for us to conduct industry-level \"extremely large datasets\" like JFT-300M. While MRL only evaluates image and text scenarios, we are the first to include the evaluation on multimodal embeddings as well, showing consistent gains as the other domains. Therefore, we believe that our results (marked as ``comprehensive`` by Reviewers 11DH and cpsE) do support the generality of our approach.\n\nFollowing your suggestion, we evaluated CSR against MRL on the larger CC3M dataset (3M images, compared to ImageNet's 1M and MS COCO's 0.3M). Results in Table 1 demonstrate CSR's consistent superiority across various active dimensions, confirming its scalability.\n\n*Table 1: Zero-Shot Retrieval Performance on MS COCO*\n|Model|Active Dim|I2T@5|T2I@5|\n|-|-|-|-|\n|ViT-B/16(Pre-trained)|512|69.23|83.03|\n|+MRL|256|54.46|61.06|\n|+CSR|256|**57.75**|**70.34**|\n|+MRL|128|48.96|55.86|\n|+CSR|128|**49.97**|**63.12**|\n|+MRL|64|38.71| 45.72|\n|+CSR|64|**40.19**|**52.39**|\n\n---\n**Q4** The impact of different backbone architectures (e.g., transformer vs. convolutional models) on CSR’s performance is not fully examined.\n\n**A4.** In fact, we have included the results of CSR under both transformer (ViT) and convolutional models (ResNet-50) in both Figure 4 and Figure 5, where CSR behaves quite similarly under different backbone architectures, indicating that CSR is rather general and agnostic of the underlying backbone architecture.\n\n---\n**Q5** Providing a brief discussion on potential limitations or failure cases of CSR.\n\n**A5.** CSR faces dead latent issues under extreme sparsity, particularly in multimodal settings(Sec. D.4). Based on our extensive experiments in other domains and ablation studies, we identify this as a technical challenge requiring solutions such as alternative loss designs or deeper architectures.\n\n---\n**Q6** Can you clarify how CSR performs under extreme sparsity constraints (e.g., TopK = 4 or 8)? While the paper shows strong performance at moderate sparsity levels, it would be helpful to see a more detailed analysis of how CSR handles extreme sparsity.\n\n**A6.** Good question! Our additional experiments (Table 2 below) demonstrate CSR's robust performance even under extreme sparsity constraints (TopK = 2 or 4), consistently outperforming MRL \n\n*Table 2: 1-NN results on ImageNet 1k*\n|Active Dim|2|4|8|16|\n|-|-|-|-|-|\n|CSR|66.17|69.97|73.84|74.39|\n|MRL|-|-|62.19|67.91|\n\nThank you for your constructive feedback. We hope our responses have addressed your concerns, and we welcome any further questions or discussion.", "claims_and_evidence": "CSR is shown to achieve higher performance and speed than MRL, supported by extensive experiments on ImageNet, MTEB, and MS COCO. The reduction in training costs is also well-supported, with experiment results showing that CSR requires significantly less training time than MRL, achieving up to a 69× speedup on ImageNet1k tasks. Furthermore, the paper provides evidence that CSR preserves semantic quality while improving efficiency by using a reconstruction-based sparse coding approach with contrastive loss, as shown through consistent accuracy at different sparsity levels. \nThe generalization claim across modalities is overstated since the multimodal experiments are limited to MS COCO and Flickr30K, which may not represent broader multimodal challenges", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are clearly defined and appropriate for the problem. The authors present a detailed explanation of the CSR framework and evaluate it on a range of benchmarks (ImageNet, MTEB, MS COCO) using relevant metrics like retrieval accuracy and inference time. The comparison with MRL and other baselines is fair and well-structured.  \n\nHowever, I believe the authors can do a better job at highlighting the novelty of the work. A lot of material in Section 3.2, especially Section 3.2.2, would fit better in the Preliminaries or Related Work section. This would help keep the focus on the core contributions of the paper, preventing the audience from being distracted by too much technical detail before understanding the main idea.", "theoretical_claims": "N/A - there is no theoretical claims.", "experimental_designs_or_analyses": "Yes - the experiment designs and analyses are sound and well-constructed. The authors evaluate CSR across multiple benchmarks (ImageNet, MTEB, and MS COCO) and provide thorough comparisons with MRL and other baselines using consistent and relevant metrics like retrieval accuracy, inference time, and training costs. The analysis includes ablation studies, scaling experiments, and tests on different sparsity levels. The inclusion of both vision and text tasks, along with multimodal settings, adds further credibility to the evaluation.", "relation_to_broader_scientific_literature": "The paper builds on prior work in adaptive representation learning, particularly MRL, but proposes a more efficient sparse coding approach using reconstruction and contrastive learning. It also draws from earlier approaches of sparse autoencoders and contrastive learning to improve retrieval accuracy and efficiency while reducing training costs.", "essential_references_not_discussed": "No, the paper discusses the key related works thoroughly.", "other_strengths_and_weaknesses": "The paper does not explore how CSR handles complex multimodal tasks beyond simple retrieval tasks (e.g., cross-modal generation or reasoning).\nThe scalability of CSR on extremely large datasets (beyond ImageNet or MS COCO scale) is not tested, which limits its generalizability to real-world large-scale applications.\nThe impact of different backbone architectures (e.g., transformer vs. convolutional models) on CSR’s performance is not fully examined.", "other_comments_or_suggestions": "Providing a brief discussion on potential limitations or failure cases of CSR would further strengthen the paper.", "questions_for_authors": "Can you clarify how CSR performs under extreme sparsity constraints (e.g., TopK = 4 or 8)? While the paper shows strong performance at moderate sparsity levels, it would be helpful to see a more detailed analysis of how CSR handles extreme sparsity, especially in comparison to MRL.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pCkKjxlMdO", "reviewer": "Reviewer_BiSs", "summary": "This paper presents a novel approach to the problem of learning adaptive-length representations. While previous methods, particularly MRL, have shown good performance, this work carefully studies the utility of high-dimensional but sparse representations, as opposed to lower dimensional but dense representation, for the adaptive-length setting. To accomplish this, the authors use sparse autoencoders and introduce a contrastive-based loss for training. Their method has significantly reduced training time, and the authors claim better performance over MRL and relevant baselines.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed reading and valuable comments. We will address your concerns as follows.\n\n---\n\n**Q1** Typos in line 69,272 and 297\n\n**A1.** Thanks for your suggestions and we will fix the typos in the revision.\n\n---\n**Q2** The experiment that requires careful attention is the one proposed in Section 4.1 and shown in Figure 3. Namely, I'm not convinced of this paper's timing results, and the authors lack a clear explanation as to why their method is so much faster than MRL.\n> To explain why I'm skeptical, a >2x speedup is achieved when the number of active dimensions is 2. In this case, the comparison in speed is between a length-2 dense representation and a length-(8192,16384,32768) representation with only 2 active dimensions.\n\n**A2.** We believe there may be some misunderstandings, and we’d like to clarify a key point. \nMost importantly, the speedup shown in Figure 3 (comparing sparse over dense computation) is **not the main reason** for CSR’s efficiency gains. \nThe main goal of Figure 3 is to simply show that sparse MM can have similar (sometimes faster) computation to dense MM under the same active dimension (x-axis). \nThe main gain of MRL is that it can use a smaller active dimension to better preserve the model accuracy. \nAs shown in the table below (quoted from Table 4 (in Sec B.4.)), CSR only degrades 1.8\\% accuracy with 8 active dimensions under MRL degrades 12.4\\% accuracy. Therefore, to attain the same accuracy level, CSR models can utilize a **much smaller active dimension**. \nAs a result, **even if the ``retrieval time per dim`` is similar** among sparse and dense MM, CSR can still attain significant speedup. And this part is the main reason for CSR's significant improvement in efficiency as we demonstrated in Figure 1(b).\n\n*Table 1: 1-NN performance comparison between MRL and CSR across various activation dim*\n\n| Active Dim| MRL   | CSR   |\n|-|-|-|\n|2|-| 66.17 |\n| 4 | -| 69.97 |\n| 8 | 62.19 | 73.84 |\n|16| 67.91 | 74.39 |\n| 32| 69.46 | 74.53 |\n| 2048 (Full rep with ResNet50) | 70.97 | 75.19 |\n\n---\n**Q3** Why $\\mathcal{T}$ is needed?\n\n**A3.** The datasets used in our paper vary by orders of magnitude. For example, FiQA-2018 has 57,638 entries, while ImageNet-1k contains 1.3M in its database. Considering this, we give an ablation study on database size $N$ in Figure 3(c). Our study demonstrates that the efficiency advantage of sparse methods scales with database size(e.g., 1M, 10M entries), making them ideal for real-world applications. Thus,\nto simulate real-world scenarios (e.g., massive-scale entities like in RAG),  we standardize all datasets to ImageNet-1k's scale (1.3M entries) and use a time-based counter $\\mathcal{T}$ to eliminate the effect of $N$ across different datasets.  \n\n---\n**Q4** How $\\mathcal{T}$ is measured? It sounds like the authors used CSR with \n$h$=16384,$k$=32, and then normalized all timings by this value?\n\n**A4.** Indeed, we calculated the average time $\\mathcal{T}$ for performing 2000 matrix multiplications of two sparse matrices using ``torch.sparse.mm()``. Here is a more detailed breakdown of the evaluation protocol for the retrieval time:\n1. We precompute the embeddings of all ImageNet training data and store them in standard CSR (compressed sparse row) on GPU memory as the database for retrieval. \n2. We compute the retrieval time as the average over 2,000 rounds of retrieval, after a warm-up period of 100 rounds. Each time, the query consists of 512 samples randomly drawn from the database, \nThe warm-up procedure is common in measuring GPU computation time, cause it eliminates initialization bias. \n\n---\n**Q5** How does the base time metric affect the timings over the raw timings?\n\n**A5.** It does not affect raw timings, as the relative retrieval time is computed by $(\\text{raw timing})/\\mathcal{T}$, as shown in Figure 3 and Table 1. We also present the raw timing comparison for MRL in Figure 1(b) to demonstrate our improvements. We will revise a more detailed explanation of $\\mathcal{T}$ calculation in Sec. 4 and E.3 to clarify our motivation.\n\n---\n**Q6** Are timings improvements strictly a result of sparse matrix operations as opposed to some architecture changes over MRL? Is there anything the authors don't currently consider that may explain the speedup?\n\n**A6.** As detailed in A2, we would like to reiterate that the core reason CSR is more efficient than MRL lies in its ability to maintain high fidelity even with very small active dimensions. The primary advantage of CSR is not simply that sparse matrix multiplication outperforms dense computation (as shown in Figure 3), but rather that CSR can achieve comparable model accuracy with significantly fewer active dimensions.\n\n---\n\nThank you for your thoughtful questions. We hope the responses provided adequately address your concerns. Please don’t hesitate to reach out if any further clarification is needed.", "claims_and_evidence": "The claims are clear, although I have concerns on the experimental evidence (see my concerns on experimental designs or analyses below).", "methods_and_evaluation_criteria": "Proposed methods and evaluation criteria are relevant", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiments are well-designed, and most are sound. The experiment that requires careful attention is the one proposed in Section 4.1 and shown in Figure 3. Namely, I'm not convinced of this paper's timing results, and the authors lack a clear explanation as to why their method is so much faster than MRL. This is especially clear from the language; around line 246 in the second column, the authors say that the decrease in retrieval time is $\\textit{likely}$ due to efficient sparse matrix multiplications, and that their results $\\textit{suggest}$ that \"higher sparsity enables more effective utilization of sparse matrix operations.\"\n\nTo explain why I'm skeptical, a >2x speedup is achieved when the number of active dimensions is 2. In this case, the comparison in speed is between a length-2 dense representation and a length-(8192,16384,32768) representation with only 2 active dimensions. According to Section E.3, csr format is used along with sparse matrix operations. However, there is still an overhead associated with using csr format, and, considering that both CSR and MRL have only two active dimensions in this case, I find it very counterintuitive that CSR can be over 2x faster.\n\nI've noted that the authors reported the normalized retrieval time, and I'm wondering if this is may be a reason for this discrepancy. The authors introduced a base-time metric $\\mathcal{T}$, the utility of which is unclear to me. How $\\mathcal{T}$ is recorded is also unclear to me. It sounds like the authors used CSR with $h=16384, k=32$, and then normalized all timings by this value? First, I hope the authors can clarify if my understanding is correct or not. Second, I hope the authors can elaborate on why $\\mathcal{T}$ is needed. They say \"This metric enables a more realistic simulation of large-scale retrieval scenarios for fair computation comparison.\" Still, they do not elaborate on why normalizing $\\mathcal{T}$ makes the comparison more fair.\n\nIndeed, the timing results are somewhat plausible, but I hope the authors can clarify how this speedup is achieved. Is it strictly a result of sparse matrix operations as opposed to some architecture changes over MRL? How does the base time metric affect the timings over the raw timings?", "relation_to_broader_scientific_literature": "This work presents an interesting and useful alternative to MRL, the leading method for length-adaptive representations. The use of sparse autoencoders in this setting is novel.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Some strengths include:\n- Clear presentation of ideas, experiments, and results\n- Paper is written well, aside from a few typos\n- Thoughtful and extensive experiment design", "other_comments_or_suggestions": "Some typos that I found:\n- first column, line 69: \"more fast\" --> \"faster\"\n- second column, line 272: this sentence seems incomplete/incorrect\n- first column, line 297: \"from high-dimensional to high-dimensional\" --> \"from high-dimensional to low-dimensional\" (?)", "questions_for_authors": "Please see the questions in the Experimental Designs section. Answering these questions is important and may warrant additional clarification in the paper. I'll repeat the questions here to make them available in a list format:\n1. How $\\mathcal{T}$ is measured is also unclear to me. It sounds like the authors used CSR with $h=16384, k=32$, and then normalized all timings by this value?\n2. Why is $\\mathcal{T}$ needed? Why does it make the timing comparisons more fair?\n3. How does the base time metric affect the timings over the raw timings?\n4. Are timings improvements strictly a result of sparse matrix operations as opposed to some architecture changes over MRL? Is there anything the authors don't currently consider that may explain the speedup?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this work, the authors present a new optimization method for learning adaptive embedding representations in foundational models. Their new method termed Contrastive Sparse Coding (CSR) is compared to strong baseline methods such as Matryoshka Representation Learning (MRL) and demonstrates gains in terms of overall fidelity, and lower training cost compared to MLR and other simple baselines. The experiments focus on image, text, and text-image domains. The reviewers were generally impressed with the strong empirical results and connection to previous lines of work in machine learning. The reviewers did note some concerns about not testing on complex multimodal settings, and scalability on large-scale datasets, however these concerns were largely resolved during discussion. Given the strong empirical results and wide potential application, this paper will be accepted for this conference.
Decision: Accept (oral)