Matched Data, Better Models: Target Aligned Data Filtering with Sparse Autoencoders
Authors: Arnav Mohanty Das
Gantavya Bhatt
Sahil Verma
Yiping Wang
Viswa Virinchi Muppirala
Jeff Bilmes
Venue: ICLR 2026 Poster (2026)
Keywords: data filtering
submodular
sparse autoencoders
Primary area: datasets and benchmarks
TL;DR: We use a submodular function instantiated on sparse autoencoder features to curate high quality + diverse datasets

Abstract: Data filtering plays a central role in improving model performance, particularly for vision language models that are pretrained on large, noisy, and redundant image-caption datasets. Existing filtering techniques assess every sample individually and retain those that exceed a certain quality threshold, but such strategies fail to capture higher-order interactions. In this work, we propose a novel submodular framework for data selection that addresses this limitation. Our method, Submodular Distribution Matching (SDM), selects a subset by: (1) training a type of sparse autoencoder to learn disentangled and \emph{monotone} features; (2) estimating a target feature distribution from a target dataset; and (3) selecting a subset of samples whose feature distribution closely matches the target via submodular maximization. Given the DataComp-medium training set and no external models, SDM achieves state-of-the-art accuracy on both ImageNet-1K and average performance across 38 downstream tasks. On the full DataComp-medium benchmark, SDM delivers performance within 1\% of the state-of-the-art results while using over \textbf{\emph{5×}} fewer GPU hours than the leading approach.

Published as a conference paper at ICLR 2026
MATCHED DATA, BETTER MODELS: TARGET ALIGNED DATA FILTERING WITH SPARSE AUTOENCODERS
Arnav M. Dayf Gantavya Bhatff Sahil Verma, Yiping Wang Viswa Virinchi Muppirala, Jeff Bilmes
University Washington
ABSTRACT
Data filtering plays central role in improving model performance, particularly for vision language models that are pretrained on large, noisy, and redundant image- caption datasets. Existing filtering techniques assess every sample individually and retain those that exceed a certain quality threshold, but such strategies fail to capture higher-order interactions. In this work, we propose novel submodular framework for data selection that addresses this limitation. Our method, Submodu- lar Distribution Matching (SDM), selects subset by: (1) training type of sparse autoencoder to learn disentangled and monotone features; (2) estimating a target feature distribution from target dataset; and (3) selecting subset of samples whose feature distribution closely matches the target via submodular maximization. Given the PataComp-medinn training set and no external models, SDM achieves state-of-the-ar accuracy both ImageNet-Ii IK and average performance across 38 downstream tasks. On the full DataComp-mediinn
benchmark, SDM delivers performance within 1 of the state-of-the-e therart results while using over 5x fewer GPU hours than the leading approach.
INTRODUCTION
Web-scale image-caption datasets have been critical to recent advances multimodal learning, enabling capabilities such zero-shot image classification (Jia al. [022] Radford al. (222]) text-guided generation (Kim et al.) 2022] Ramesh et al. 2021 Zhang et al. (002)). multimodal retrieval (Radford al., (021), and range of other applications (Jiang
2023). However,
because these datasets are typically scraped from the web, they contain noisy and redundant samples
that can degrade model performance (Elazar et al., 2024 Webster et al. 2023). Given their scale, manual curation is infeasible, making algorithmic data selection an increasingly important area of research.
Most data selection methods applied at web- scale follow a common strategy: estimate the quality of each sample and retain those above threshold. Some methods use sim- ple heuristics, such image resolution cap- tion length (Gadre et al., 2023); others lever- semantic align- ment (Gadre et al.] 2023 Wang et al.) 2024b)
"Gnen backgronf CUP-Soom bottom 8%
"Cef CLI-Scorm top 20%
in synd CUP-Soom top 20%
Figure I: Quality Score Limitations. Low-quality samples still be useful for learning broader concepts. For example, model may have enough samples learn the concept "cat," but without the low-scoring "green background" image, it lacks examples needed to learn the concept of "green."
age CLIP embeddings to
proximity to reference dataset like Ima- geNet (Gadre et al., 2023 Wang et al. (2024b).
More recent approaches train specialized mod-
els on curated external datasets to estimate sample quality (Fang et al., 2024 Kim et al. 2024 Shechter & Carmon, 2025b). In general, these methods treat each sample independently when evaluating their utility.
While evaluating samples independently can be computationally efficient, overlooks properties that only emerge at the dataset level. For example, individually high-quality samples may be redundant
Equul contribution. Correspondence (arnavmd2, (arnavv(,,gbhattt2)))e gbhatt2]

---

Published as a conference paper at ICLR 2026
1. Train SAE
2. Compute Target Distribution
3. Subset Selection
nource
SAE Decodar
theeeeer atenbuatinn
swiches begget
esbaidiege for imagen iager
tile KL (p PCAA)
SAC Encoder
Vtar
quase eSANCCAA bY
relected echellige St eutturr Inbord deetuutcce
Pretrained Brooddrr
P(A)
Figure 2: Overview. Our pipeline consists of three steps: (1) Train sparse autoencoder (SAE) to disentangle pretrained neural features into meaningful sparse representations, (2) estimate an empirical distribution of target dataset over sparse features (3) use submodular distribution matching objective to select A S V that matches the target distribution.
with each other when selected together, while samples that seem low-quality in isolation may capture
rare concepts that enhance diversity when included (see Figure (1. Ignoring distributional properties during selection can lead to datasets that are imbalanced, ultimately limiting the generalization ability of models trained on them (Aghabagherloo et al., (025)..
We argue that effective data selection must also account for distributional properties, particularly by ensuring balanced representation of underlying concepts at the sel level. This goal requires us to (1) identify and reason about the presence of individual concepts at set level and (2) select a subset of data such that the distribution over these concepts exhibits desirable properties.
Standard neural representations, such as CLIP embeddings, are ill-suited for this purpose. These representations tend to be entangled (Olah et al. 2020), blending multiple concepts in ways that make it difficult to quantify or control how much of given concept is present in a set. To address this, we sparse autoencoders (SAEs) to disentangle compact neural representations into sparse features that correspond to disentangled concepts (Makhzani & Frey (014)). We further require these features to be monotone, meaning that their values increase more of the corresponding concept is present (Gupta et al. 2016). This encourages the feature values behave more like (fractional) counts: when aggregating over set, the total contribution of a concept can be obtained simply by summing its values across individual items. To this end, we introduce a novel loss term inspired by the "peripteral" loss introduced in Bhatt et al. (2024) to promote learning monotone features. Intuitively, these disentangled and monotone features provide set of "knobs" we can adjust to control which concepts-and and in what proportions-are included in the final dataset.
To balance concepts in desirable way, we frame data selection as a distribution matching prob- lem: our goal to match the distribution over concepts in the selected set to a target distribution derived from downstream task. We then connect this distribution matching objective submodular maximization: specifically, we show that maximizing certain instances of fcature-based functions (FB functions) class of monotone, non-decreasing submodular functions correspond to mini- mizing the KL divergence between the concept distribution of the selected set and the target. This connection allows us use scalable algorithms for approximate maximization with constant factor guarantees (Neemauser et al., 1978). Finally, we demonstrate that this objective can be combined with existing quality-based filtering strategies, enabling a unified approach that considers both distributional properties and individual sample quality.
Our contributions can be summarized as follows: (1) We introduce a data selection framework that disentangles dense features into disentangled and monotone concepts and selects a subset of data that maxmimizes a FB function based objective. (2) We offer a theoretical interpretation of the FB objective by connecting it to distribution matching. (3) We demonstrate strong empirical results on Data DataComp-medium dataset of 128M image-caption pairs, showing that models trained on our filtered dataset achieve significant improvements compared to existing data selection methods.
METHOD
In this section, we begin with brief introduction to feature-based functions (FB functions) ( 22.1 describe the process we use to construct a disentangled and monotone set of concepts ( 2$.. $2.3). and finally derive the proposed subset selection objective ($2.4, $2.5). Our full workflow is shown in Figure We also provide a brief introduction to submodularity and a scalable strategy commonly used approximately maximize these functions in Appendix C
2

---

Published as a conference paper at ICLR 2026
CLIP Representations
SAE Representations
Neuron
Nouron
Neuron 2
Neuron 2
G
Images w/ Highest Activations
Images w/ Highest Activations
Figure 3: SAE Visualization. Each row displays the top-activating images from DataComp for two randomly selected neurons of the CLIP image encoder (left) and an SAE (right). CLIP features tend to activate for mix of unrelated concepts. while SAE features represent more coherent concepts. See Appendix [or more examples.
Notations. Unless mentioned otherwise, V == V-source = [n] 21,2.... n} denotes the ground set indexing the data points to select a subset from, and Vtar denotes the ground set indexing the
target data points. We denote the space of latent representation of images as X' C R and the
space of representation of sparse representations output 2 C T Aapause Zaource and Ztar the design matrices for source and target respectively, such that Zgouce & hene(" (xjourcey) (similarly for the target), where hame; : X + Z denotes the encoder of sparse autoencoder. A represents a probability simplex d-dimensionss and [[-] is an indicator function. Lastly, we use Φ to denote the concave function, and (()) represents modular function defined on any A S V.
2.1 PRIMER ON FEATURE- BASED FUNCTIONS
This work focuses on a specific class of submodular function referred to as feature-based func- tions (Stobbe & Krause 2010 Bilmes & Bai 2017) or FB functions. Given set of indices
V 1,..., n/, a design matrix = [21,-.., Zn E IR non-negative entries, a list of
monotone, non-decreasing concave functions (=:) and a vector of non-negative weights E R
we define feature-based functions for any A C V as follows:
f(A) Σ wi 0i (m,(
where m²(A) A
Zjá+
(I)
JEA
For FB-functions, each feature must be monotone, meaning that larger 2js values correspond to a stronger presence of concept i in item j. and larger mx(A) indicates greater total presence of concept i in set A. For instance, bag-of-words or TF-IDF-like representations, Zyi the frequency (or weighted frequency) of word in document j. so (A) naturally increases more documents in A contain that word. The concavity of each Φ; enforces diminishing returns: once a concept is already well represented in set A, adding more instances with similar concepts contributes less marginal gain. Finally, the weights w; allow assigning relative importance to different features.
2.2 OBTAINING SPARSE FEATURRS
We begin by extracting information-ree representations from a pretrained image encoder; however. these features typically respond to several, potentially unrelated concepts (Olah et al.) 2020). To address this, we disentangle these representations using Sparse Autoencoders (SAEs), drawing inspiration from early sparse coding methods (Lee et al., 2006) and recent advances in interpretabil-
ity (Makhzani & Frey 2014 Gao et al. 2025 Huben et al. 2024 Pach et al. 2025).
Formally, given an input vr already in some dense representation space X C Rd we define a k-SAE (Gao al. 2025 Makhzani & Frey 2014) with autoencoding deparse dimensions using an encoder heme and an affine mapping based decoder, respectively, with encoder projecting input into sparse, high dimensional non-negative space (daparse >> d,n).
A ene 2 = Topk (ReLUKRReWWWeen
buse) + bene)); T = Waec h (x) + baec
(2)
where W ene deppuee x dis, bene Relaam Waec x Xdapasee bdec Rd. and TopK(-) only retaining top-k entries of a vector and setting anything else as O. In other words, every input vector is transformed by the encoder into sparse representation that has k nonnegative entries
3

---

Published as a conference paper at ICLR 2026
and reconstructs h, using affine transformation. k-SAE's are typically trained with simple reconstruction loss Crecona (z, Ø) = | - 21|2 and we include an additional activity regularizer ractrey Eres (x))( / to keep activations from being too large (Merity et al. 2017). The resulting sparse representations tend to exhibit the property that individual features respond to images
containing concept which can often be interpretable by humans shown in Figure [3 this property
is frequently referred to as monosemanticity (Gao et al. 2025 Huben et al. 2024 Pach et al. (2225).
We note that the property of monosemanticity is ill-defined, since it depends entirely on how one chooses to define a "concept." For instance, a hidden neuron that activates in the presence of any bird in an imput image can be considered monosemantic with respect to the bird concept, but polysemantic with respect to individual bird species. Nevertheless, we use the shorthand "monosemanticitt" to mean so with respect to fixed set of concepts.
Monosemanticity is also distinct from the notion of monotonicity. For example, a hidden neuron that only activates in the presence of bird may still be considered monosemantic even if its activation value is higher when more non-bird species are present in the image. Despite being monosemantic, the magnitude of the activation does not correspond to the "strength" or prevalence the concept's presence. Conversely, feature may be monotone and polysemantic-for instance, if higher values indicate a larger presence of both birds and cats. While k-SAEs disentangle dense representations into useful concepts, we additionally introduce novel loss term to explicitly encourage monotonicity.
2.3 MONOTONICITY LOss
To encourage our k-SAE to learn monotone features, we introduce an additional contrastive "peripteral" loss term inspired by Bhatt et al. (2024). Concretely, we sample hEterogeneous and hoMogeneous sets E, M C V, define a margin function ((E[M) which measures how much more diverse E is than M, and instantiate unweighted FB-function (A)
Eie [deparaa]
log(1 [g(11m,,1 (A)).
The monotonicity loss defined as
Lionn ( Er, M) = (A(E)M) log 1 (1+exp exp (1
((E) A(E (--10D-1(M))
((M)
This loss aligns the sign and magnitude of f(E) - f(M) with the margin (EE Intuitively, heterogeneous sets E (sets with more concepts) should achieve higher values of f than homogeneous sets M (sets with few concepts). Because f(A) E; log(1 + mA)) is concave, repeatedly increasing already-active features contributes little additional gain; across many sampled pairs, the most consistent way to reduce the loss is for new concepts in L' to activate additional features. This biases the SAE toward representations that behave monotonically with respect to concept presence.
In practice, the true concept set is unknown, so we approximate A(E) M) by defining A M) = Ee,.")eMr x²) ((.r)E (2, 2²), the difference summed pairwise similarities computed in the original dense representation space. We sample E uniformly random, and M is obtained by taking a single element of E and finding its [E| 1 nearest neighbors in V. The final SAE trained with combination of Crecons. and Lmnon- Refer to Algorithm I for pseudocode and additional implementation details, and to Appendix D for further analysis of how low monotonicity loss encourages monotone features.
2.4 SUBMODULAR DISTRIBUTION MATCHING
Now that we have provided framework to learn sparse and monotone features, we turn to the problem of subset selection. Specifically, we seek to maximize a feature-based (FB) function to ensure that the final subset maintains a balanced representation of concepts Bilmes & Bai (017). This section details our subset selection objective.
Definition 2.1 (Empirical Distribution). Given a ground set of indexing V [m], any subset A C and design matrix 2 E B. meddprre we define histogram based empirical distribution m.(A)
((A) A deparen-1 such that p(A), Exe(hopanel my(A)' where m²(A) & EjeA 2zz-
4

---

Published as a conference paper at ICLR 2026
Corollary 2.2 (Target Empirical Distribution). Given ground set of indexing V and target design matrix Zrar, we define target empirical distribution p E A Aspuasee such that p!
m,(V [((ppprree) my(Vtor) where (A) _ ΣSEA 2ss
Given the target empirical distribution plar E Rarear our goal is to match it as closely as we can, by selecting examples from source distribution. That is, aim to find C Visouree which minimizes (p a || p (A)), (A)), where prouree (A) denotes the empirical distribution of source for given A S V, per definition 2.1 In the following theorem, show that this problem is an instance of maximizing a Difference of Submodular (DS) functions (Iyer & Bilmes] 2012).
Theorem 2.3 (Distribution Matching as DS Maximization). Given the target and the source empirical
distribution, minimizing DKL(D" (p || prouece (A)) is an instance of optimizing Difference of 25uuree
Submodular Functions, where ma(A) = -EjEA 2j
A argmin DKL (p thr (
(4)
AgV
argmax
pj tar log(m;(A)) log
Σ
my(A)
- argmax g(A) AgV
(5)
ACV
sE(deparmn]
jEldeparan
Proof. Please refer to the Appendix
As eq. (5) is an instance of DS-maximization, there exists no polynomial time approximate algorithm, unless P -NP. Since direct optimization is intractable, we instead maximize a lower bound on
g(A) obtained by upper-bounding log [Eje(Aaprae]] m, (A) While tight submodular lower bounds follow from the existence of modular upper bounds (Nemhauser et al. 1978), computing them is prohibitive for large ground sets (n ~ millions). With budget b in subset selection, we thus maximization and employ an approximation that, when optimized via accelerated greedy methods (Minoux 2005 Mirzasoleiman al. 2014), matches the performance of modular upper-bounds-basee algorithms (Iyer & Bilmes) 2012).
consider cardinality-constraiine
Lemma 2.4. Assuming sparse feature map : X' + Z is such that ||All S 3 and given that h is an instance of k-SAE, if ğ(A) A Eie(d P. log(m;(A) log (k:) then ((A) < g(A). In many scenarios where subset selection is subject to cardinality constraint, Le.. A < for given budget b, the quantity g(A) satisfies the submodular lower bound ((A) ExE(daparo)) pi log(m;(A)) log(k:
Proof. Please refer to the Appendix E In Appendix F we validate this approach on Gaussian mixture dataset, showing that the omission of the term log (k3b) does not degrade performance.
To encourage 33 to be low, we use an activity regularization term as discussed in Section 2.2 To guarantee that the submodular function remains monotone, non-decreasing, and normalized (i.e., it evaluates to 0 on the empty set), we replace log(.) with log(1 + in the lemma above. This modification has negligible effect in practice because m²(A) >>> each m²(A) aggregates millions of samples.
2.5 COMBINING DISTRIBUTION MATCHING OBJECTIVE WITH QUALITY MEASURES
While maximizing the above objective helps align the feature distribution of the selected subset with that of the target, it only considers image features. However, effective data filtering must also account for the semantic alignment between each image and its corresponding caption. Prior work has proposed various ways to quantify this alignment typically by computing the dot product between the image and text embeddings (Gadre et al. 2023 Wang et al.; 2024b) Kim et al.] 20244] Fang et al. 2024 Maini et al. 2024) and aim incorporate this into our distribution matching framework.
key advantage of using a submodular objective is that it remains submodular even when combined with modular function (Bilmes) 2022). Therefore, a simple objective that combines distribution matching with metric that measures quality would be f(A) - ğ(A) + Eae q(a) where ğ( (A
5

---

Published as a conference paper at ICLR 2026
is the previously defined distribution matching objective and ((a) measures how well caption corresponds to image a. However, this solution has a few shortcomings. First of all, the gain of the overall objective can be expressed f(o|AA) ((][A) + q(b) for t V \ A but only ( decreases as |A] gets larger. Therefore, g(v) may dominate the value of [(v] A) which diminishes the overall objective's ability jointly consider distributional similarity and quality. Furthermore, existing measures of quality are noisy and may not be useful in expressing finegrained preferences. In other words, qi)) > q(j) does not necessarily imply that i preferable to J when q(i) - {()) is small. Therefore, we devise a novel technique to combine a quality score with our distribution matching objective that (I) improves the overall objective's ability to consider both functions during selection and (2) only uses coarse-grained preferences of the quality score.
Instead of using directly, we create feature-based function based on a quantized version of which can be expressed as follows:
(A)
Σ SEld
u, log
1+ E [g(()) [b)-1,
(6)
SEN
where {bo, .i...., b denote bin edges, such that bin i corresponds to the interval [b,-1, b.) and U, is weight that expresses the degree of preference we have for bin i. Note that since o' has form similar to g. is easy to show that maximizing g'(A) is minimizing the KL-divergence to a distribution u if Eier U4 =
Overall Objective Since the optimization cardinality constrained, for distribution matching, we can use Eie [dsparuec] p! log(m; as a proxy, since log (k3) Al) is constant. We can now combine the distribution matching with a quantized version of the quality score, which we call SDM (Submodular Distribution Matching).
Definition 2.5 (SDM), Given ground sets and design matrices Vsource, Z source, modular
function m (A) & Ejen z for any subset A C Vsource and E [dsparso] (similarly for the target). Let the larget empirical distribution be defined as plar A deparea where
tar (Vtar)
tax
El(dsparo 17 (Vtar)
Furthermore, given e bins {bo. bi..... be} over quality metric along with bin weight vector u, a trade-of off parameter X [o, 1), and a selection budget SDM optimizes:
arg A
Σ [dupppeee]
Pt log( m, (ource(A)
source (A))
AS P sounce
distribution mutcring
+ (1
Σ u; log + Σ 1((lt) E (b)11
JEA
quality weighting
EXPERIMENTS
our experiments, our primary goal is to determine the most effective subset selection strategy for training a CLyP-style model in the DataComp-medium benchmark (Gadre et al. 2023).
3.1 SETUP
Datasets We use the image-caption dataset associated with the DataComp-medium benchmark (128M image-caption pairs) for all training. All subset selection strategies in this section are designed to identify the best subset from DataComp-medie models are evaluated based zero-shot performance
for training a CLIP-style model. The resulting
suite of 38 zero-shot tasks proposed by Gadre et al. (2023) which includes ImageNet- 1K (Deng et al. 2009), several classification datasets from VTAB (Zhai et al. 2019), and retrieval benchmarks such MS-COCO (Chen et al. 2015).

---

Published as a conference paper at ICLR 2026
CLIP Training We adopt the DataComp-medde medium training configuration for all experiments. Fol- lowing the selection a subset via a given filtering approach, training is conducted using a CLIP-B/32 model under standardized compute regime where the total number of training steps is fixed to 128M, regardless of dataset size, Each training run requires 40 A A100 hours.
SAE Configuration We train an SAE on the CLIP ViT-L/14 image embeddings (di, = 768), with deparse- 98, 304. The SAE is trained on the unfiltered DataComp-medium dataset; no external data sources are used. Note that the image encoder parameters are frozen, so we are able to train the SAE on precomputed image embeddings.
Submodular Objective and Optimization By default, our submodular objective uses negCLI- PLoss (Wang et al., 2024b) as a measure of quality. We quantize this into three equally sized bins, and assign weights of O, 0.01, 0.99 to the low, medium, and high values respectively. We use the training set of ImageNet- I K as the target distribution unless otherwise stated. To maximize this objective, we use the stochastic greedy algorithm |Mizzaoleeiman al. 2014) (shown in A Algorithm with e = 0.001. We run stochastic greedy 5 times to select subsets and compute the intersection between them to get the final subset (see Appendix K for ablation this). Selecting subset of 25M samples from a pool
128M takes approximately hour on a CPU using stochastic greedy with Bilmes i the total time required to compute the final summary aggregated over stochastic greedy results is hours though this can be fully parallelized across separate CPU's.
Computational Cost Overall, our method requires 60 A100 hours to compute the initial features, 5 A100 hours to compute the quality scores, 15 A 100 hours to train the SAE, and 5 CPU hours to run stochastic greedy. Importantly, both feature extraction (a shared bottleneck across all methods) and NegCLIPLos, scoring scale linearly with the size of the candidate pool. In contrast, the SAE training cost can remain fully independent of the pool size. Training CLIP on DataComp-mediut requires 40 A100 hours.
3.2 MAIN RESULTS
Table 1:
Main Results. Performance of filtering strategies on medium- seale evaluation tasks. Bold values indicate the best each column, and underlined values indicate the second best. For all methods, we report results at their best-performing data fraction.
Filtering Strategy
Size
INIK
INIK Shifts
VTAB
Retrieval
Avg
No Filter
128M
17.6%
15.2%
25.9%
21.9%
25.8%
Basic Filtering Gadre al.
2023)
30M
22.6%
19.3%
28.4%
25.1%
28.5%
Text-Based Gadre et al. 2023
31M
25.5%
21.5%
32.8%
24.9%
30.7%
Image-Basee Based (Gadre et
2023
29M
26.8%
21.3%
31.9%
25.6%
31.2%
CL.PP-Score Gadre
2023
38M
27.3%
23.0%
33.8%
25.1%
32.8%
Image- Image-Based (7 Cc.re Gadre ct al.
2023
14M
29.7%
23.9%
34.6%
23.1%
32.8%
D2 Pruning Maharana el al., 2024
26M
24.1%
20.6%
30.6%
19.6%
29.8%
negCLIPLoss (NCL) WWagg al. 2024b
33M
28.8%
23.8%
35.4%
25.3%
34.4%
NCL n NormSim (INIK) (Wang et al.
2024b
22M
32.8%
26.8%
36.2%
26.5%
35.3%
NCL. n NormSim (Target)
Wang et al. 2024b
22M
32,7%
26.5%
37.5%
26.5%
35.7%
SDM (ours)
35.2%
27.1%
38.6% %
%
36.4%
In this section, we focus on the setting where only a single model (specifically CLIP ViT-i/14) is available for generating embeddings. We argue that this setting is the most general because (1) training new dedicated model for data selection for a spe- cific domain is often prohibitively expen- sive- sometimes costing more than train- ing the primary model itself (Fang et al.]
Imagelett Accuracy
Accuracy
Accoracy
26.0
34
SDM (ours)
35.5
SDM (ours) farsh faxeline
Rast Baseline
34
10
22
34
1.6
18
70
Dataset Size (millions)
Detaset (militone)
Figure 4: Different Sizes. We test SDM different sizes the best baseline (NCL NormSim) which uses 22M samples.
7

---

Published as a conference paper at ICLR 2026
2024 Kim et al. 2024), and (2) approaches
that score data using multiple embedding sources or models (Maini et al. 2024 Yu et al. 2023) are not always feasible. especially in domains where such models are unavailable, such as biomedical image/caption datasets (Ikezogwo et al.
2023 Lozano et al. 2025).
In Table 1 evaluate several baselines. CLIP-Score (Gadre et al. 2023 selects the image- caption pairs whose corresponding image and text embeddings have the highest dot product. Image- Based (Gadre al. 2023) performs k-means clustering on the image embeddings and selects samples which belong to clusters that are close to embeddings of training images from ImageNet. Image-Based O CLIP-Score Gadre et al.] (2023) simply takes the set intersection between the sets selected by the two methods. D2 Pruning (Maharana et al. 2024) constructs an undirected graph initialized with CLIIP scores and selects data by jointly optimizing for sample difficulty and diversity. NCL. (Wang et al. 2024b) is similar to CLIP-Score but assesses sample quality based the CLIP loss (Radford et al. 2021)- batch wise measure that considers semantic alignment and specificity. NormSim measures how similar image is to target dataset in terms of p-norm. By default, we always consider p- = 3o which was the highest performing variant in Wang et al. 2024b), and consider two target datasets (1) the Imagenet- 1k training set or (2) Target which is the training sets of 24 different downstream tasks. NCL n NormSim combines the two prior approaches by taking& set intersection of the resulting subsets. This method was previously the state of the art data selection technique among those that only use CLIP ViT-L/14 (Wang et al. (024b)) We find that SDM consistently delivers substantial performance improvements over state-of-the-ar data selection methods. On ImageNet-IKK SDM outperforms both variants of NCL n NormSim by 2.5% on ImageNet-IK,, IK, and achieves 0.7% gain in average performance.
Notably, in Figure 4 we find that our method is especially effective at smaller dataset scales: with 33% fewer samples, SDM still outperforms NCL NormSim by nearly 2% on ImageNet. We attribute this advantage to SDM's ability to reduce redundancy in the selected subsets.
3.3 ALTERNATE BACKBONES AND QUALITY SCORES
NormSim (Wagg et al. 2024b) and SDM
are both targef-aware data selection meth- ods that can be applied with different fea- ture extractors and quality scoring fune- tions. Figure S compares their perfor- mance across several backbone architec- tures (CLIP ViT-L/14, ViT-B/32, and DFN- P) and quality scores (CLIPScore (Gadre ett al. 2023) and NegCLIPLoss).
Zero Shot Acc (%)
16
Imagelte Performance
20.6%
Zero Shot Act 1990
Performance
35.95
In each case, we assume access sin- gle embedding model, as in the previous section. Across all backbones and scoring functions, SDM consistently outperforms NormSim, both on ImageNet and in the average performance. This demonstrates that distribution aware selection is more effective than evaluating samples individually.
CLIPScone NegttPLoO CLIPäcar RASCUPLUCA
Figure 5: SDM vs. NormSim. We test both SDM Norm- Sim with different quality scores and backbones, and find that SDM consistently achieves better zero-shot performance.
3.4 IMPACT OF MONOTONICITY LOss
We study the effect of adding the loss term Cmono to the standard reconstruction loss Crecons. when training a K-SAE. We introduce two metrics to evaluate SAE quality: the Monosemanticity Score (MS Score) and the Monotonicity Score (MT Score). MS Score captures semantic consistency of neurons (Zhang et al. 2025). For each neuron E [dsparse]. we compute the average pairwise cosine similarity among images activating that neuron, and then average over neurons. higher score indicates that images activating the same neuron are semantically aligned. MT Score measures how well features behave like "counts." " For each neuron i, we sort images by activations Zi: i), yielding permutation o; with o.[1] denoting the strongest activation. If is the dataset size, the neuron-level score is |
T-T Σj=1 < To.b)) [o+1++1] >. We average the neuron-level score across all neurons, to report the MT Score. Intuitively, adjacent images with similar activations for feature should also
8

---

Published as a conference paper at ICLR 2026
exhibit similar concept counts, resulting in higher similarity. Finally, we evaluate the impact of Luono within SDM, following the same procedure as before but performing distribution matching with a different set of features. We report both zero-shot ImageNet- IK accuracy and the average zero-shot accuracy across all 38 evaluation tasks. Please refer to Appendix for more details.
Table 2: Impact Lmono on SAE training; MS: Monosemanticity Score (t), MT: Monotonicity Score (f).
DutaComp Small
ImageNet-IK
Accuracy
MS Score
MT Score
MS Score
MT Score
ImageNet-IK
Avg
Liecons. 0.55
0.60
0.58
0.60
44.8%%
35.00%
Crcccn..
Luccso
0.62
0.66
0.63
0.65
35.20%
36.40%
In Table 2
show that adding mmno yields consistent improvements across all metries. Although our study focuses on applying SAES to data selection, the substantial gains in both MS and MT Scores highlight the potential relevance of this approach to the broader interpretability community.
3.5 COMPONENT-VE WISE ABLATION
In this section, we disentangle the contributions of two components in our approach: (1) using SAE-derived sparse features, and (2) using submodular objective. To isolate the effect of using sparse feataures, we construct the feature-based objective directly on dense CLIP embeddings (after applying ReLU ensure nonnegativity). To isolate the effect of submodularity, we remove the log() function from both components in Theorem 2.5 which eliminates diminishing returns. We report results on ImageNet-IK well as the average performance across 38 downstream datasets in Table 3 The results demonstrate the using sparse features is critical to downstream performance using dense CLIP features results 12% drop in ImageNet performance and 9% overall. We also find that removing the concave function substantially diminishes overall performance, by 1.3%.
Table 3: Effect of Sparse Features and Submodularity. Performance on ImageNet- IK (left) and the average over 38 datasets (right) under two ablations: (i) using dense CLIP features instcad of SAE-derived sparse features, and (ii) removing the concave log term that induces submodularity.
ImageNet-1K (%)
Average Over 38 Datasets (%)
Submodular
No Submodular
Submodular
No Submodular
Sparse
35.2
34.6
Sparse
36.4
35.1
Dense
24.1
23.7
Dense
29.1
28.9
3.6 COMBINATION WITH OTHER SELECTION APPROACHES
We also integrate SDM with the highest-pefforming method on DataComp-medium that publicly releases its indices, namely the NCL + DFN HYPE ensemble proposed by Wang et al. (2024b). Their approach constructs the dataset by taking the disjoint union of three independently selected subsets; any sample that appears in & subsets is upweighted by being duplicated times in the final collection. To incor- porate SDM, we define a submodular objective that encourages quality in a manner analogous to our earlier formulation and apply the same upweighting scheme. To further boost performance, instantiate SDM objectives using SAE features derived from two embedding models CLIP ViT-L/14 and DINOv2. Full details of this procedure provided in Appendix
Table 4: Results w/ External Models. Performance of filtering strategies on ImageNet and overall average when using external data and models. Bold best, underline second best.
Filtering Strategy
INIK
Avg
TMARS + SSFT Maini et al. 2024
33.8%
36.2%
HYPE (Kim et al. 2024
34.6%
37.3%
DEN (Fng
20241
37.1%
37.3%
NCL DFN + HYPE Wang al. 2024b
38.2%
38.8%
M-FLYT + SCS (Shechter & Carmon, 2025b
40.1%
37.7%
Metagrudient Descent (Ennstrrum al. 2025
27.0%
40.2%
SDM Wang et al.,
2024b
(ours)
39.21%
39.2%

---

Published as a conference paper at ICLR 2026
As shown in Table 4 this hybrid approach achieves substantial gains over the base method and ranking 2nd based on average performance out of all methods on the DataComp- Dataaomp--medum leaderboard. The only method outperforming SDM is Metagradient Descent (Engstrom et al., (025). which is prohibitively expensive and attains 12.2% lower zero-shot accuracy on ImageNet- IK. Please refer to Appendix [ approach.
which discusses how SDM requires over 5x less GPU hours than the Metagradients
RELATED WORK
Targeted Data Selection. Targeted data selection is broader field that has been explored in other settings as well. Wallingford et al. (2023); Udandarao et al. (2023) propose similarity based data selection methods for few-shot adaptation of VLMs, which also evaluate the utility of each sample independently. Some recent work also explores using submodularity to select samples that are diverse and relevant to target (Das et al. 2025 Kothawade et al.) 2021; Kumari et al. 2024: Agarwal et al., 2024) in various contexts. However, these methods are difficult to scale due to their reliance on pairwise similarity matrices. Finally, there has also work that uses the regularized optimal transport objective (Liu et al., 2024) in addition to kernel density- However, these methods cannot include quality score trade-off and can still be O(n log log n)), even ignoring the subset selection cost, where is the size of the dataset to select from. Our work approximates the distribution matching objective with a tractable submodular function, and can easily handle additional quality score(s).
estimation to promote diversity.
SAE's in Data Selection. Two recent works have explored the use of SAE's for data selection for supervised finetuning of LLMs. Yang et al. (2025) introduced the first approach in this direction, proposing LAE-GreedSeless which greedily builds subset by repeatedly selecting the sample that covers the largest number of previously uncovered concepts. However, this strategy is ill-suited to our setting, where due is much smaller than the target summary size, causing the pool of concepts to be exhausted prematurely. Moreover, Yang et al. (2025) do not provide mechanisms to (1) combine diversity and
when working with highly uncurated multimodal datasets- or (2) assign relative importance different concepts, both of which are addressed by SDM. More recently, Ma et al. (2055) proposed a method similar to NormSim (Wang et al.) (024b), for task-aware selection method that uses SAB-derived representations to construct more robust similarity metric.
CONCLUSION AND FUTURE WORK
Overall, we propose SDM, a novel submodular framework for filtering multimodal datasets using iisstibution-matchinn objective based on SAE features, combined with quality-based objective. When filtering the DataComp-medium pool, and only using CLIP embeddings, SDM achieves SOTA accuracy on both ImageNet-Iie IK and average performance across 38 downstream tasks. On the full benchmark, SDM delivers performance within 1 1% of the SOTA while using over 5x fewer GPU hours than the leading approach. There are several promising directions for future work. First, our framework is modality/dooaiinagnoss explore in the future. Second, while our main experiments fixed the target dataset to ImageNet, we include in Appendix K an ablation in which we vary the target dataset. This analysis shows that performance can be sensitive to the choice of target, suggesting that careful selection of the target distribution is important. Exploring principled strategies for target selection, such using strategies inspired by data mixture optimization (Xie et al. 2023), remains an exciting direction for future work.
and can be readily applied to other tasks which we plan to
6 ACKNOWLEDGEMENT
This work is supported by NSF Grant Nos. IIS-2106937 and IIS I11-274..
REFERENCES
Ishika Agarwal, Krishnateja Killamsetty, Lucian Popa. and Marina Danilevksy. Delift: Data efficient language model instruction fine tuning. arXiv preprint 2:Xiv:2411.04422 2024.
10

---

Published as a conference paper at ICLR 2026
Alireza Aghabagherloo, Aydin Abadi, Sumanta Sarkar, Vishnu Asutosh Dasu, and Bart Preneel. Impact of data duplication on deep neural based image classifiers: Robust vs. standard models, 2025. URL httlhhts///// httpaaxiii..o wtt/ htapgg/abs//aa/ 2504 00638
Ashwinkumar Badanidiyuru and Jan Vondrák. Fast algorithms for maximizing submodular functions. In Proceedings of the twenty-fifth annual ACM-SIAM symposium on Discrete algorithms, pp. 1497-1514. SIAM, 2014.
Gantavya Bhatt, Arnav Das, and Jeff Bilmes. Deep submodular peripteral networks. In The Thirty- eighth Annual Conference Neural Information Processing Systems, 2024. URL https
//openreview.uo wtt.:e/eoooommmddttttetto.
Jeff Bilmes. Submodularity in machine learning and artificial intelligence. arXiv preprint
arXiv:2202.00132. 2022.
Jeff Bilmes. Submarine: SUBModularity for ARtificial INtelligencE. and machine learning. Online Software System, 2026.
page
Jeffrey Bilmes and Wenruo Bai. Deep submodular functions. arXiv preprint arXiv: 08993,, 2017.
John Binder, Kevin Murphy, and Stuart Russell. Space-efficient inference in dynamic probabilistic networks. Belr, 1:t1, 1997.
Xinlei Chen, Hao Fang. Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco captions: Data collection and evaluation server, arXiv preprint 1wXi::1504.00325, 2015.
Arnav Das, Gantavya Bhatt, Lilly Kumari, Sahil Verma, and Jeff Bilmes. COBRA: COmBinatorial Retrieval Augmentation for Few-Shot Adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2025.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Yanai Flazar, Akshita Bhagia, Ian Helgi Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hannanch Hajishirzi, Noah A. Smith, and Jesse Dodge. What's in my big data? In The Twelfth International Confer- Learning Representations, 2024. URL https:/ http:/ss///ppeneeeeweeeeeee wtw.://ueteeeeeeeeeeeeem RvfPnOkPV4]
Logan Engstrom, Andrew Ilyas, Benjamin Chen, Axel Feldmann, William Moses, and Aleksander Madry. Optimizing ml training with metagradient descent, 2025. URL
2bs/2503.13755
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander T Toshev, and Vaishaal Shankar. Data filtering networks. The Twelfth International Conference on Learning Represen- tations, 2024. htthttss::// htttttss://opeeeeeiieeee
net/ Eorum22d-KAA6ng200
Uriel Feige. A threshold of In n for approximating set cover. Journal of the ACM (JACM), 45(4): 634 652, 1998.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang. Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah M Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. In Thieth- Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: /pwpreeieeeeeeee// net. bet/fforumt d=dVaWCDMRof
11

---

Published as a conference paper at ICLR 2026
Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In The Thirteenth International Conference on Learning Representations, 2025. URL httpttt///ppeeeeeeeee
i cs 9t 9ZNKD
Maya Gupta, Andrew Cotter, Jan Pfeifer, Konstantin Voevodski, Kevin Canini, Alexander Mangylov, Wojciech Moczydlowski, and Alexander van Esbroeck. Monotonic calibrated interpolated look-up tables. Joumnal of Machine Learning Research, 17(109):1-47, 2016. URL http: /ttp:/mmmmrrooooooo org
papers/v17/1s- 200055-243hhtm.
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: / http///ppeeevvieee ew net/ forum?
1ddF76bbRSLeK
Wisdom Oluchi Ikezogwo, Mehmet Saygin Seyfioglu, Fatemeh Ghezloo, Dylan Stefan Chan Geva, Fatwir Sheikh Mohammed, Pavan Kumar Anand, Ranjay Krishna, and Linda Shapiro. Quilt-Im: One million image-text pairs for histopathology. In Thirly-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL net htt.://weeetooooooooooooooooo000
ftttps://cpenevviiie
Rishabh lyer and Jeff Bilmes. Algorithms for approximate minimization of the difference between submodular functions, with applications. In Proceedings of the Twenty-Eighe Eighth Conference on Uncertainty in Artificial Intelligence, 12 pp. 407-417, Arlington, Virginia, USA, 2012. AUAI Press. ISBN 9780974903989.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference machine learning, pp. 4904-4911111 PMLR, 2021.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts, 2023. httphhtsss///axii..ooo/o
/ttb://21100 03094
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVE conference on computer vision and pattemn recognition, pp. 2426-2435. 2022.
Wonjae Kim, Sanghyuk Chun, Taekyung Kim, Dongyoon Han, and Sangdoo Yun. Hype: Hyperbolic entailment filtering for underspecified images and texts. European Conference on Computer Vision, pp. 247-265. Springer, 2024.
Suraj Nandkishor Kothawade, Nathan Alexander Beck, Krishnateja Killamsetty, and Rishabh K Iyer. SIMILAR: Submodular information measures based active learning in realistic scenarios. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL httpps://pperreiieeee wwt.:/eee.eeeeeeee formm? iid=
VGDF
Lilly Kumari, Shengjie Wang, Amay Das, Tianyi Zhou, and Jeff A. Bilmes. An end-to-end submodular ( (Findings), pp. 3293-3308, 2024. findings-naacl.
framework for data-efficient in-context learning. In NAACL-HLT
URL httts::///oo..... /tw.://........................0
Honglak Lee, Alexis Battle, Rajat Raina, and Andrew Ng. Efficient sparse coding algorithms.
Advances in neural information processing systems, 19, 2006.
Zifan Liu, Amin Karbasi, and Theodoros Rekatsinas. Tsds: Data selection for task-specific model finetuning. In Advances in Neural Information Processing Systems, 2024.
Alejandro Lozano, Min Woo Sun, James Burgess, Liangyu Chen, Jeffrey J. Nirschl, Jeffrey Gu, Ivan Lopez, Josiah Aklilu, Anita Rau, Austin Wolfgang Katzer, Yuhui Zhang, Collin Chiu, Xiaohan Wang, Alfred Seunghoon Song, Robert Tibshirani, and Serena Yeung-Levy. Biomedica: An open biomedical image-caption archive, dataset, and vasion-language models derived from
12

---

Published as a conference paper at ICLR 2026
scientific literature. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern June 2025.
Recognition (CVPR), Pp.
Da Ma, Gonghu Shang, Zhi Chen, Libo Qin, Yijie Luo, Lei Pan, Shuai Fan, Lu Chen, and Kai Yu. Task-specific data selection for instruction tuning via monosemantic neuronal activations, 2025. [RLtts:::
org/ abs /2003.2503..22 15573
Adyasha Maharana, Prateek Yadav, and Mohit Bansal. ma pruning: Message pass- ing for balancing diversity & difficulty in data pruning. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https://penneview..eee net forntt/fformmdd t.btt.AkCe9
Pratyush Maini, Sachin Goyal, Zachary Chase Lipton, J. Zico Kolter, and Aditi Raghunathan. T-mars: Improving visual representations by circumventing text feature learning. In ICLR, 2024. URL
htttps://openrett net
Alireza Makhzani and Brendan Frey. k-sparse autoencoders, 2014. URL https://axxiv..rrg
abs/ 1312.5663
Stephen Merity, Bryan McCann, and Richard Socher. Revisiting activation regularization for language mnss. arXiv preprint 1rXiv:1708.011 010099 2017.
Michel Minoux. Accelerated greedy algorithms for maximizing submodular set functions. In Optimization Techniques: Proceedings of the 8th IFIP Conference on Optimization Techniques Würrzburg September 5-9, 1977, pp. 234-243. Springer, 2005.
Baharan Mirzasoleiman, Ashwinkumar Badanidiyuru, Amin Karbasi, Jan Vondrak, and Andreas Krause. Lazier than lazy greedy, 2014. URL http:: http//xxxvvvi..........////.. http://///////..../////////////e 7938
George Nemhauser, Laurence Wolsey, and Marshall I. Fisher. An analysis of approximations for maximizing submodular set functions-i. Mathematical programming, 14:265-294. 1978.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi:
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khali- dov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features with- out supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL Featured Certification.
htttps://openrewe new forum wid-mmisutuuusseeemm 6zFt
Mateusz Pach, Shyamgopal Karthik, Quentin Bouniot, Serge Belongie, and Zeynep Akata. Sparse autoencoders learn monosemantie features in vision-language models, 2025. URL https?
//axxiv.org/abs///222 02880
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. CoRR, abs/2102. 2021. URL Sttps://arrxer org/ abs 2102. 12092
Mikey Shechter and Yair Carmon. Filter like you test: Data-driven data filtering for clip pretraining. arXiv preprint arXiv:2503.08805, 2025a.
Mikey Shechter and Yair Carmon. Filter like you test: Data-driven data filtering for clip pretraining, 2025b. URL htRl:ttts:// ptttpps://axiie www.wvoorrgggoooo abs 255/ 08805
13

---

Published as a conference paper at ICLR 2026
Peter Stobbe and Andreas Krause. Efficient minimization of decomposable submodular functions. Advances in Neural Information Processing Systems, 23, 2010.
Vishaal Udandarao, Ankush Gupta, and Samuel Albanie. Sus-x: Training-free name-only transfer of vision-language models, 2023. URL http::ttt///ccm httptttsttt.ii..tttot. oww.:/iiiirrrrreeoe.e abs http:/bbb/22..111119
Matthew Wallingford, Vivek Ramanujan, Alex Fang, Aditya Kusupati, Roozbeh Mottaghi, Aniruddha Kembhavi, Ludwig Schmidt, and Ali Farhadi. Neural priming for sample-efficient adaptation, 2023.
org abss//a//// 2306. 1019111911111
Yifci Wang, Qi Zhang, Yaoyu Guo, and Yisen Wang. Non-negative contrastive leanning. In The Twelfth International Conference Learning Representations, 2024a. URL https::/oppeeeeeeett.
Yiping Wang, Yifang Chen, Wendan Yan, Alex Fang, Wenjing Zhou, Kevin Jamieson, and Simon Shaolei Du. Cliploss and norm-based data selection methods for mul- timodal contrastive learning. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances Neural Information Pro- Curran Associates, Inc., 2024b. URL cessing Systems, volume 37, pp. http:rrwiwi.cccppppaaaaa//a//a/r
httpp::/pproceeddngssseeet wwtp:/rp.pppppppppprpppppp c2d4
Ryan Webster, Julien Rabin, Loic Simon, and Frederic Jurie. On the de-duplication of laion-2b, 2023.
URL httthttss///xxiiii
Sang Michael Xie, Hicu Pham, Xuanyi Dong, Nan Du Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 69798 69818. Curran Asso- ciates, Inc., 2023. URL URL.httpss:/pprceediiii
rwwrreeesppppp..e. cc fess filesppprrreeeepppe/reepp 2023/flle/ccaaee1111333333333332 Papee
Conferenceeccocco Core pdff
Xianjun Yang. Shaoliang Nie, Lijuan Liu, Suchin Gururangan, Ujjwal Kar,, Rui Hou, Madian Khabsa, and Yuning Mao. Divereity-driven data selection for language model tuning through sparse autoencoder, 2025.
abs 2aaba/25022 2502.
Haichao Yu, Yu Tian, Sateesh Kumar, Linjie Yang, and Heng Wang. The devil is in the details: A deep dive into the rabbit hole of data filtering, 2023. BtLphttss////xxit org/ abs /2009 15954
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04800 2019.
Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon. Text-to-image diffusion models in generative ai: survey, arXiv preprint arXiv:2303.0700 079999999 2023.
Qi Zhang, Yifei Wang, Jingyi Cui, Xiang Pan, Qi Lei, Stefanie Jegelka, and Yisen Wang. Beyond interpretability: The gains of feature monosemanticity on model robustness. In The Thirteenth International Conference on Learning Representations, 2025. URL /tttps://opent//ee net htte//Erumniid-g000333000
14

---

Published as a conference paper at ICLR 2026
A TABLE OF NOTATION
Here we provide list of notations we used throughout this work.
Notation
Deseription
V Vsouree
[n]]
Source data indices (ground set)
Vtar
Target data indices
C Rdd
Input image feature space
Z C Rupaarre
Sparse representation space
7
source
Source design matrix
Z tar
Target design matrix
A hene (200 (ourre
Encoded rep. of j-th source input Sparse autoencoder encoder d-dimensional simplex
nmc
X +
Ad-1
[[]]
Indicator function
Ø
Concave function
m(A)
Modular function on A V
I
Image space
T
Caption space
Duouree
Source image-caption pairs Quality score for index h
(k)
Diar Dirain
Target image set
Selected training subset
Table 5: Summary of Notations
B ADDITIONAL RELATED WORK
DataComp The DataComp paper introduces benchmark that fixes the training and evaluation procedures for VLMs to measure the effect of data filtering techniques on model performance Gadre et al.] (2023). Since then, many new data filtering techniques have emerged and can broadly be categorized into two types: (1) approaches that rely exclusively on off-the-shelf CLIP embeddings while refining how those embeddings are employed Gadre et al, (2023); Maharana et al. (2024); Wang et al. (2024b); (2) methods that use curated external data sources to either enhance the CLIP model Fang et al. (2024) itself incorporate wholly different external models to guide the data- selection process Kim et al. (2024): Yu et al.] (2023); Maini et al. (2024). Generally, the latter outperform the former but demand substantially more development resources often exceeding those required to train the VLM itself. Our work falls in the former category, but can be easily combined with methods in the latter.
Comparison with Metagradients Engstrom et al. (2025) frames data-subset selection as gradient- based hyperparameter tuning, where each datapoint's 's weight the loss is treated a hyperparameter. However, this approach requires "backpropagation through the entire training sequence", ", which naively incurs a prohibitive memory cost of (T)), where is the number of training steps. To mitigate this, the paper introduces a Ok log T) space algorithm using lazy k-ary tree, akin to the island algorithm (Binder et al.) 1997). Beyond memory, the method still necessitates substantial computation: training on the DataComp dataset for 40 steps (Fig. 6 from their paper, assuming they require a similar number of steps for DataComp Medium, since they only report experiments on the DataComp Small dataset) translates 40 5 200 A100-hours on NVIDIA A100 GPUs, assuming 5 A100-hours per metagradient step. contrast, our approach trains at most 2 SAEs, requiring only 40 hours. Thus, SDM is approximately 5x faster than the Metagradients-basel approach.
15

---

Published as a conference paper at ICLR 2026
C PRIMER ON SUBMODULAR FUNCTIONS
In this section, we formally describe submodular functions. For a given ground set = [n] & {1...., mn), set function f 2 + R is submodular if and only if it satisfies fAA (o)) - f(A) ((BU(u) - (B) for subsets A B V and v E V B, where [(vl & (AU often referred to as gain of adding a new element v. In case when referred as modular function, and in such case for any A V. can be decomposed as f(A) = je f((i))+ f0). Diminishing returns allows submodular functions to effectively model notions of diversity and coverage Bilmes (2022). When set functions are monotone non-decreasing (f(A) I f(B), AA B V) and normalized (f(0) 0). they referred as polymatroid functions. In general, maximizing submodular functions is NP-Hard, however, polymatroid functions can be approximately maximizee subject variety of constraints from set cardinality, to knapsack, to bigger and theoretically interesting class of matroid rank constraints. For cardinality constraints, a greedy algorithm achieves an approximation guarantee of - (Feige] 1998. Minoux 2005 Nemhauser et al. 1978); for other constraints, please refer to Badanidiyuru & Vondrak (2014).
[(f)),, then the function
D ANALYSIS OF MONOTONICITY LOSS
The goal of this section is to show that minimizing the monotonicity loss Lmnno on carefully constructed (E, M) pairs, together with the margin function A(E) M), results learned features that are monotone with respect to a set of concepts.
Definition D.1 (Monotonicity). Let 6 be a finite set of concepts. For each item
associate concept vector (v)) [0, 1] Given nonnegative features a E R IV|xd we say that z is monotone with respect 6 for every a & there exists nonempty index Ge S [d] and a constant nj 2 such that, for any two items v, o' V with (v( > Ve(v"),
Σ (Ees z011 ((bc(1)) bc(')) EG
Remark D.2 (Monotonicity vs. Monosemanticity). is important to distinguish monotonicity from monosemanticity, Monotonicity requires that some coordinates consistently increase as the presence of concept grows, but it does not require one-to-one mapping between concepts and features. A single feature may still respond to multiple concepts, as long as activation is nondecreasing in each relevant direction. By contrast, monosemanticity typically implies that each feature should be specialized to a single concept.
Definition D.3 (Feature Sums and Aggregator). For E
<R!Vixd and subset A S V, define
d
(A))) Σ
2/0
Vi [d],
f(A) Σ log((1 1+ mm()) ==1
JEA
Definition D.4 (Concept Margin), For subsets E, C V of equal size, define
(E M) = [bb(E) - ((Mil,,
which measures the total excess of concepts in E compared to M. Definition D.5 (Monotonicity Loss). For pair M), the contrastive loss
Luonn (F, M) A(E (M) log exp [(E)-I(M) Given a dataset = ((E,, M,) the total loss
A(EEM)
T(=) Σ Cmonoo (E, M,).
1ET
Definition D.6 (Per-Cone (Per-Concept Partition). For each concept E C, define To = ((E,,M)) E T Er, M, differ only in concept c).
Let
Te Σ A(E Mr),
Jo(z)
Σ TET
Emono (E, M,).
TT
16

---

Published as a conference paper at ICLR 2026
Theorem D.7 (Low Loss Implies Average-Case Concept Monotonicity). Suppose for every $6 the per-concept average loss is small:
1 Jelz) S e. re
Then for each there exists nonempty Ge S [d] such that
1 ΣΣ (m((E.) - mM(MM)
In particular, average across dataset pairs differing only in c, features in Ge grow proportionally with t/ with margin 1) = {11 - E).
Proof. Fix c. E and write A, A(E,) M,). Define
pc(z) f(E)) f(M))
A, 0.00. 0.
A.
Then
Jelz) Σ
A. logl (1 + exp(1 p.)).
TE
Since log(1 + e²) 2 taking = 1 - gives
Je(2)
Σ
A.(1 E.((1 p.) = = Σ
Aipt.
LET.
LET.
Dividing by Te and applying the loss bound,
1
Σ
Dept 2 1-5.
(I)
To TE
By definition,
Pt [(E.)-f(M-)
D.
Thus (1) implies
(f(E)) f(Mi)) 2 -
(2)
re TE
Now, note that for any x,y 2 0,
log(1 + x) log(1 + v) S x y.
Applying this coordinatewise,
f(M)) s Σ (mi(E:) (M)))..
(3)
Combining (2) and (3),
(E,) - m²(M)) 2 1 1- E.
(4)
r. TET :=1
Equation (4) states that, averaged over pairs differing only in concept c, the total feature mass increases in line with Ve. Averaging across coordinates,
Σ ==1
Σ LE
(m:(E))-o - m.(Mi))) 2 [ 61-
Hence at least one coordinate (and thus some nonempty G.) satisfies the same bound, proving the claim.
17

---

Published as a conference paper at ICLR 2026
E PROOFS FROM THE MAIN PAPER
For completeness, we re-state the statement of the theorem
Theorem E.1 (Distribution Matching as DS Maximization). Given the target and the source empirical distribution, minimizing DKL(p prource (A)) is an instance of optimizing Difference of Submodular Functions.
A" argmin DKL. (p'a AgV
I poocce (A))
(8)
arginax
Σ
pl log(m:(()) - log
Σ
my(A)
AgV
iE iEldsanee
argmax ACV
(AA)
(9)
where mA(A) jEA
oj
Proof. We hegin by expanding the definition of the KI. divergence.
plat psouree (A);
(p || =
source (A))
Σ
log
(10)
KE [e]deparen
-1
pi log (p source (A)))))(AA)))
(11)
diparee
Where H.)) refers to the shannon entropy function. Since entropy of target distribution is not function of A, we can focus on the second term, and aim to maximize it.
argmin (KL(p tar
source (A)) = argmax
Σ
P;
log (p source (A)s)
(12)
ACV
ACV
&E [E(deparen]
Now we use the fact that
(A(
pr source A
jE [deparr)) M2(A)
Plugging above yields
argmax ACV
Σ
Pt tar
log (p source (A):) argmax
Σ
P4 tar log(m,(()) - log
Σ jE
m2(A)
my (A)
ACV
V [dapassc]
iE [dspusee
Lemma E.2. Assuming sparse feature map h X such that |1 S 3 and given that an instance of k-SAE, if (AA) & Eie P. log(m;(A)) log (k:] then g(A) S 9(A).
Moreover, ((A) submodular for any A V..
Proof. To arrive this lower bound, first observe that
mA(A)=
Σ
Zij
(13)
jE[dsparoe]
j jE(dsparno] iEA
Σ
Zij
(14)
SEA jEA
Since the design matrix is generated using a TopK sparse autoencoder features, for every example
i E A, define & & {l: l E [dsparsa] 2ik Z > 0}: note that &&l < k. Since ||h|] < 3, therefore, Zuk 3 for all E A and & &. Therefore, we have the following
18

---

Published as a conference paper at ICLR 2026
Σ
my(A) Σ
Σ
24j
(15)
jE depareal
iE
JEE:
Σ
k:
(16)
iEA
||| k
(17)
Plugging the above relation in the definition of ((A) yields the desired inequality.
F ILLUSTRATION OF DISTRIBUTION MATCHING
We generate a synthetic dataset by sampling from mixture of 30 two-dimensional Gaussian com- ponents with diverse means and covariances, which defines the source distribution. The target distribution is constructed ing probability mass on a restricted subset of the mixture. To represent the data, the 2D domain is partitioned into fixed 50 50 grid, yielding 2,500 bins. Each sample activates only the bin it falls into, and by aggregating counts, we form an empirical distribution over the bins. Figure 6 illustrates both the source and target datasets.
selecting 4 of these Gaussians and oversampling from them, concentrat-
In this setup, the design matrix Z has one-hot entries along the feature axis. It follows that
mA( =
Σ
25
(18)
jE j((dpparoo]
j [daparm
iEA
(19)
SEA jE [Aspurse]
Σ
-|||
(20)
SEA
Consequently, the objective function corresponds to Lemma 2.4 with the overall objective:
argmax ACV [E(daparue]
p log(m,(A) log (IA1),
where deparse = 2500. In this setup, the target distribution has an entropy of 5.467. Optimizing only the first term (excluding log()-A)) match the target size results in sharp decline in KL divergence, achieving 3.07. In contrast, random subsets of the same size yield much higher KL divergence, averaging 23.46 = 1.90 across 1000 trials.
Finally, we observe that the supergradient-boe based approach that optimizes the full difference, including the Al) term as in Iyer & Bilmes (2012), performs similarly optimizing only the first term. This demonstrates the robustness of our method, which is also orders of magnitude faster than optimizing the full objective.
G ALGORITHM BLOCKS
Here, include pseudocode for the SAE training procedure used in all experiments (Algorithm) I and the stochastic greedy procedure employed to maximize the final submodular objective (Algorithm
2.. To train the SAE (Algorithm) [p. we use [M] = 50, 000, 128. A1 1, A 0.002 and
Ag = 10 10 In Algorithm we use c 10-3
H COMPLEXITY ANALYSIS
Let N = |/sonrce |, .M = | J, b is our budget, and dspars the dimensionality our sparse encodings with time only At entries being non-zero. Our algorithm has two main components:
19

---

Published as a conference paper at ICLR 2026
hearc Infiod Beget outbet
beavo bufrist Srget pittee
Figure 6: Source dataset (left) and target subset (right) derived from a mixture of Gaussians. We optimize only the first term, as suggested in Lemma 2.4 to obtain subset that effectively covers the target components.
62.02
Distribution Matched Subset
$3.58
Avg. Randem Subset S, [S]=2000
45.13
36.68
28.24
11.34
2.90
10
10a
10a
[A]: Cardinalioy
distribution matching
Figure 7: KL divergence reduction over optimization steps, compared to random subsets of size 2000.
(1) generating and storing sparse features, and (2) performing submodular maximization. Sparse feature generation involves simple matrix multiplication. Since each example has only k non-zero entries (128 our case), the space required to store sparse features is O(N) and O(M) for the source and the target, respectively. Our submodular objective combines binned quality function and feature-based diversity terms. Because the quality function is discretized, can be stored efficiently using fewer bits than standard floating-pont representations. The overall evaluation cost of the objective is O(N + d=parse). For the maximization step, we employ the stochastic greedy algorithm Mirzasoleiman et al. (2014) (shown in Algorithm [ which offers near-optimal approximation guarantee 1 1/e in O(N log 2), instead of (bbN).
I MORE ON MONOTONICITY AND MONOSEMANTICITY SCORES
To evaluate the effect of Lmono: compare SAEs trained with and without this loss on two datasets: ImageNet- IK and DataComp-Small (a 10% subset of (aaaCom--Meii))) These datasets provide diverse and noisy web-scale images, allowing to assess robustness.
Monosemanticity Score measures the semantic consistency of neurons (Zhang et al. 2025 Wang al. 2024a). For each neuron [daparse], we compute the average pairwise cosine similarity among images that activate it. A higher score indicates that the neuron consistently responds to semantically aligned images. We plot histograms of these scores for all neurons, comparing the two SAES.
Monotonicity Score quantifies how well features behave like "counts." " For neuron é, we sort images by activation values 7[; yielding permutation ds with o;(1] denoting the strongest
20

---

Published as a conference paper at ICLR 2026
Algorithm Train Sparse Autoencoder (SAE)
Require: Batch list B, set size buffer size B, weights À = [11, A2, Aa), learning rate 7
Ensure: Trained SAE he
1: // Instantiate SAE and buffer for reservoir sampling.
2: M [[
3: he < INSTANTIATESAE
4: for each batch b B do
5: M
b, B)
6: Compute TopK SAE loss according eq.
7: CSAB f TOPKSAELOSS ha)
8:
Lact-reg + EzzE (x))l
9:
10:
Use a random subset as hEterogeneous set. b such that [E] Choose e E E at random
Sample
12:
Get 1-1 nearest neighbor of from M I/ Use nearest neighbors as hoMogeneous set.
14:
M < {e}U NEARESTNEIGHBOR(E, M, k
16:
1/ Compute set contrastive loss according to
17:
Lmoo f- SETCONTRASTIVELOOSSSS M, ha)
18:
19:
Liotal + A1 LSAE + X2 Lmono + X3 Lact-reg Update model using gradient descent. hg <- GRRADIENTIESSEENN/nn Liotals 77)
20:
21:
end for
23:
return hs
Algorithm 2 SStcchsttc--Greed)
Require: /:2V + R4, A E {1,...,,i,
Ensure: A set C V satisfying |A| S
1: A +-0 0
2: for f-
do
3:
R <- random subset of size
n log 21
obtained by sampling s random elements from
V\A
4:
a; < arg m A(a A)
5: A A-AUKE {a+}
6: end for
7: return A
activation. If is the dataset size, the score is
Σ (Fobl Tollti))
Intuitively, adjacent images in this ordering should exhibit similar concept counts, leading higher similarity. We again plot histograms of these scores across neurons.
In the main paper, aggregate these histograms by taking the mean, as reported in Table 2
J SDM WITH DATACOMP METHODS THAT USE EXTERNAL MODELS
In this section, describe how combine SDM with the ensemble method proposed by Wang et al. (2024b). Overall, we adapt the quality function to incorporate we include per-sample weights, and finally add another SAE trained DINOv2 (Oquab et al. 2024) features.
Their approach constructs three distinct subsets of DataComp. each selected by a different strategy. Let Dj. denote the set of samples that appear in exactly i of the three subsets, where 0, 1,1,2,3.
21

---

Published as a conference paper at ICLR 2026
4003
2580
2003
2200
1003
02
0.4
0.6
38
2.0
02
8,4
0.
8.
3.0
Figure 8: MS Score for DataComp- Small
Figure MS Score for ImageNet- IK
Figure 10: Impact of mmno on Monosemanticity Score (higher is better). On both datasets, Lmono improves semantic consistency, enhancing interpretability.
Coom
4003
Loom
Coom Craen
Foom Ceat
2008
2009
2003
208
Monstreichty Score
Monstrrlcily Share
Figure 11: MT Score for DataComp-Snin Small
Figure 12: MT Score for ImageNet- 1K
Figure 13: Impact of Lmono on Monotonicity Score (higher is better). Across both datasets, Luono improves the count-like interpretability of features, making them better suited for feature-based submodular functions.
That is, Dj: contains all samples jointly selected by l of the methods. We then define a new quality function, analogous to the one introduced in Section 2.5
3 Σ ui, log 1+ 1+E
TD ED.
(AA) A
(21)
JEA
Here, the weights u, determine the relative preference for samples selected by multiple methods. In particular, we set i which encourages the selection procedure to favor samples that are chosen by a larger number of the ensemble methods.
To further strengthen SDM, instantiate this quality function using features from two independently trained SAEs: one based on CLIP ViT-L/14 embeddings and another based on DINOv2 embeddings. The SAE trained DINOv2 uses the same hyperparameters as the original SAE.
Finally, we note that all competitive methods DataComp use some form of sample reweighting, where samples may appear multiple times in the final dataset. Since SDM by design selects subsets rather than multisets, we adopt the reweighting scheme of Wang et al. (2024b), repeating each sample in D₂ times.
22

---

Published as a conference paper at ICLR 2026
K ADDITIONAL ABLATIONS
Removing Intersection over 5 rums For each step of the stochastic greedy algorithm, we randomly sample a subset of V \ A (where the size of the subset controlled by the parameter and evaluate the gain only on these. Occasionally, we may get unlucky and random sampling will select a very poor candidate set, leading to the addition of bad sample. To mitigate this, we take the intersection of multiple stochastic greedy runs, which effectively filters out such suboptimal selections. Moreover, this procedure is parallelizable across cores, allowing us perform several in parallel with minimal computational overhead. In T we show the comparison between a single run and the intersection over runs downstream performance.
Table 6: Effect of Intersecting Multiple Runs Comparison of single run versus intersection over 5 independent stochastic greedy runs of maximizing the SDM objective.
Method
INIK (%)
Avg. (%)
Single run
33.5
36.0
Intersection over runs
35.2
36.4
Modifying the Target The choice of target distribution plays central role in the effectiveness of SDM. Following prior work (Wang et al,, 2024b] Shechter & Carmon| 2025a; Gadre et al.) 2023). main experiments
the target distribution for subset selection. However, we also tested a broader target distribution constructed by concatenating the training splits of 24 downstream datasets (including ImageNet), as done WWan et al. (2024b). Interestingly, this reduced overall performance as shown in Table 7 We hypothesize that this decrease stems from (1)severe concept imbalance when merging datasets of different sizes and concept distributions, and (2) inclusion of low-quality datasets (e.g., (FFAR-10). whose samples degrade performance on others. Developing more principled strategies for constructing larger target distributions (e.g., via data mixture optimization (Xie et al. 2023)) remains an interesting direction for future work.
ImageNet-1K
Table 7: Modifying the Target Comparison of performance when changing the target from ImageNet- IK to a pool of 24 downstream datasets done in Wang et al. (2024b).
Target
INIK
INIK Shifts
VTAB
Retrieval
Avg
ImageNet-1K
35.2
27.1
38.6
26.8
36.4
24 datasets
33.3
26.7
37.1
26.7
35.8
23

---

Published as a conference paper at ICLR 2026
L ADDITIONAL SAE VISUALIZATIONS
In this section, we present visualizations several additional neurons randomly selected from the SAE trained on CLIP embeddings. For each neuron, we display the top 5, middle 5, and bottom images with nonzero activations, ordered in descending activation value.
LOUCE
2000
4..0
A000
US.02
VOEE
.012
10.00
A000
401
Figure 14: Visualization of different k-SAE features. For each row, we display the top 5, middle and bottom 5 images with nonzero activations for the neuron corresponding to the row, ordered in descending activation value.
24


Reviews:
[{"review_id": "tTPdvCVbBZ", "reviewer": "Reviewer_hCWT", "summary": "The paper proposes SDM (Submodular Distribution Matching), a data filtering framework for vision-language models that combines sparse autoencoders (SAEs) with submodular optimization. The method learns disentangled and monotone features through SAEs with a novel monotonicity loss, then selects data subsets that match a target distribution while considering sample quality. The authors claim state-of-the-art results on DataComp-medium benchmark.", "questions": "1. How sensitive are results to ε-smoothing value? Please provide ablation.\n2. Can you guarantee the β bound in practice? What's the actual max activation value observed?\n3. Why take intersection of 5 greedy runs rather than union or single run?\n4. What's the breakdown of improvements from SAE features vs submodular selection?\n5. How does performance vary with different target distributions?\n\nThis paper makes a solid contribution to data selection for large-scale training. The idea of using SAEs to obtain interpretable features for submodular selection is clever and well-executed. While there are technical details to clarify, the core contribution is valuable and the experimental results support the claims.\n\nThe 0.7% average improvement may seem modest, but in the context of large-scale training where compute costs are substantial, even small improvements are valuable. The framework is also general and could be applied to other domains beyond vision-language models.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Novel framework integration: First to combine SAEs with submodular optimization for data selection, providing both interpretability and theoretical guarantees\n2. Theoretical contribution: Establishes connection between KL divergence minimization and submodular maximization (Theorem 2.3), enabling efficient algorithms\n3. Practical impact: Achieves competitive performance on DataComp-medium with reasonable computational budget compared to alternatives\n4. Comprehensive evaluation: Tests across 38 downstream tasks, showing consistent (if modest) improvements\n5. Monotonicity loss innovation: Novel loss term (Eq. 3) for encouraging monotone features in SAEs could be valuable for interpretability community", "weaknesses": "Major\n1 Mathematical Soundness: \n- The objective \\log m_i(A) is undefined when mi​(A)=0. Add explicit ε-smoothing: log(mi​(A)+ϵ) with sensitivity analysis.\n- Unverified bound: The proof of Lemma 2.4 relies on∥h∥∞​≤β which the SAE architecture doesn't guarantee.\n2. Statistical Validation: All results lack error bars. Re-run with ≥3 seeds, report mean±std for all tables, and provide significance tests.\n3. Computational reporting: Should clarify total pipeline costs including encoding time\n\n\nMinor\n\n1. No ablation separating component contributions\n2. Provide more details on Algorithm 1 (distance metrics, buffer size)\n3. Explain the 5-run intersection choice", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6caUuq1PxS", "reviewer": "Reviewer_7pgs", "summary": "This paper addresses data filtering for vision-language models pretrained on large, noisy datasets. The authors propose Submodular Distribution Matching (SDM), a submodular framework that selects samples by matching feature distributions learned via sparse autoencoders. Experiments on DataComp-medium demonstrate that SDM achieves state-of-the-art or near state-of-the-art accuracy across ImageNet-1K and 38 downstream tasks, while significantly reducing computational cost.", "questions": "1. My main concern is that selecting a data subset that exactly matches or aligns with the target distribution may critically affect the model’s out-of-distribution generalization. How to balance the goal of distribution matching with maintaining data diversity during filtering?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed Submodular Distribution Matching (SDM) presents a novel and promising approach to data filtering. Extensive experiments and superior performance compared with baseline methods convincingly demonstrate its effectiveness.\n\n2. The theoretical analysis linking the designed submodular maximization objective to the distribution-matching target strengthens the rationale behind the proposed method.\n\n3. The paper is well organized and clearly written, making it easy to follow and understand.", "weaknesses": "From a model training perspective, selecting a data subset that precisely matches or aligns with the target distribution may critically influence the model’s out-of-distribution (OOD) generalization capability. Providing additional empirical evaluation of OOD performance using the filtered data would help clarify the practical impact of the proposed method and further highlight its contribution to improving generalization beyond the training distribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hb0vOn2mBQ", "reviewer": "Reviewer_aMFk", "summary": "This paper tackles the problem of filtering large-scale image-text datasets (like those scraped from the web) to improve model training. Such datasets are often noisy (containing irrelevant or low-quality data) and redundant (many very similar examples), which can hurt model performance. Traditional filtering methods usually score each data sample for quality and drop the low-scoring ones, but fail to capture the 'high-level feature. This paper proposed SDM to capture the high-level features and estimate the target distribution from ImageNet as a standard. Follow the standard, SDM can select high-quality samples from new datasets that follow the standard distribution at a high conceptual level.", "questions": "1.  In line 154, why the sparse dimension $d_{sparse} >> d_{in}$. For the learned sparse features, how can we verify what the features are? Why are they informative to represent the information shown in the Image or text?\n2. For estimating the target distribution of 'high-level' features, this paper uses ImageNet as the standard. If all datasets share the same 'high-level' features? \n3. Although ImageNet is comprehensive in images, different datasets may have different features. This paper uses the same sparse autoencoder to align the learned high-level features. What if there are new important features in new datasets? \n4. Moreover, the proportion of different classes of images may affect the target distribution. If the gained performance is because the autoencoder is pre-trained on ImageNet, the data selection process uses ImageNet as a standard, which makes the selected data fit the model. \n5. For the comparison of different model sizes, does the best baseline have the same results across the different sizes?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper proposed SDM to capture high-level interactions (features), which go beyond single-sample selection. \n2. SDM uses high-quality datasets as standard to estimate the target distribution of high-level features\n3. This data selection strategy is verified in rich downstream tasks.", "weaknesses": "1. It highly relies on the high-quality dataset, the high-level features learned from it are treated as the target distribution.\n2. The interpretability of the high-level features is a big concern as I listed in the questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)