CoDA: From Text-to-Image Diffusion Models to Training-Free Dataset Distillation
Authors: Letian Zhou
Songhua Liu
Xinchao Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Dataset Distillation
Text-to-Image Diffusion Model
Core Distribution Alignment
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Prevailing Dataset Distillation (DD) methods leveraging generative models confront two fundamental limitations. First, despite pioneering the use of diffusion models in DD and delivering impressive performance, the vast majority of approaches paradoxically require a diffusion model pre-trained on the full target dataset, undermining the very purpose of DD and incurring prohibitive training costs. Second, although some methods turn to general text-to-image models without relying on such target-specific training, they suffer from a significant distributional mismatch, as the web-scale priors encapsulated in these foundation models fail to faithfully capture the target-specific semantics, leading to suboptimal performance. To tackle these challenges, we propose Core Distribution Alignment (CoDA), a framework that enables effective DD using only an off-the-shelf text-to-image model. Our key idea is to first identify the ``intrinsic core distribution'' of the target dataset using a robust density-based discovery mechanism. We then steer the generative process to align the generated samples with this core distribution. By doing so, CoDA effectively bridges the gap between general-purpose generative priors and target semantics, yielding highly representative distilled datasets. Extensive experiments suggest that, without relying on a generative model specifically trained on the target dataset, CoDA achieves performance on par with or even superior to previous methods with such reliance across all benchmarks, including ImageNet-1K and its subsets. Notably, it establishes a new state-of-the-art accuracy of 60.4\% at the 50-images-per-class (IPC) setup on ImageNet-1K. Our code is available on the project webpage: https://github.com/zzzlt422/CoDA

Published as a conference paper at ICLR 2026
CoDA: FROM TEXT-TO-IMAGE DIFFUSION MODELS TO TRAINING-FREE DATASET DISTILLATION
Letian Zhou
Songhua Liu"
Xinchao Wang"
National University of Singapore 1.54699 6594 ndu
Shanghai Jiao Tong University 1 usonghuals jau edu
National University of Singapore
nchacênus. eeteenusseuuu..... sg
ABSTRACT
Prevailing Dataset Distillation (DD) methods leveraging generative models con- front two fundamental limitations. First, despite pioneering the use of diffusion models in DD and delivering impressive performance, the vast majority of ap- proaches paradoxically require a diffusion model pre-trained on the full target dataset, undermining the very purpose of DD and incurring prohibitive training costs. Second, although some methods turn to general text-to-image models with- out relying on such target-specific training, they suffer from a significant dis- tributional mismatch, as the web-scale priors encapsulated in these foundation models fail to faithfully capture the target-specific semanties, leading subop- timal performance. To tackle these challenges, we propose Core Distribution Alignment (CoDA), a framework that enables effective DD using only an off- the-shelf text-to-image model. Our key idea is to first identify the "intrinsic core distribution" of the target dataset using a robust density- discovery mecha- nism. We then steer the generative process to align the generated samples with this core distribution. By doing so, CoDA effectively bridges the gap between general-purpose generative priors and target semantics, yielding highly represen- tative distilled datasets. Extensive experiments suggest that, without relying on generative model specifically trained the target dataset, CoDA achieves per- formance on par with or even superior to previous methods with such reliance across all benchmarks, including ImageNet-IK and its subsets. Notably, it es- tablishes new state-of-the ahr accuracy of 60.4% at the 50-images- persss--er-sssss (IPC) setup on ImageNet- IK. IK. Our code is available on the project webpage: https::/ / com /tl//zzzlttt 222lt422/CoDA
INTRODUCTION
Dataset Distillation (DD) Yu et al. 2023b); Xiao et al. (2025) has gained significant attention for its core promise: to synthesize a compact dataset, allowing drastically more efficient model training with highly competitive performance. Addressing the limitations of early DD methods Yin et al. (202bb); Yu et al. (2024) 2025), which often faced prohibitive computational demands and struggled to scale to large, high regouuesolution datasets, a recent wave of research has turned to leveraging powerful generative models like GANs (Goodfellow et al. (2020)) and Diffusion Models (Ho et al. (2020); Song et al. )..00a)). However, this emerging paradigm confronts two fundamental limitations.
First, although pioneering and highly effective, the vast majority of these methods (Minimax Gu jet al. (2024); IGD Chen et al. (005): ( et al. (2025); Moser et al. (2024): /Zhao et al. (2025) employ diffusion model pre-trained or fine-tuned on the target dataset itself, conceptually flawed process that incurs prohibitive computational costs. This approach puts the cart before the horse: the core motivation of DD is to create an efficient dataset substitute to avoid the prohibitive cost of training on the full dataset, yet this approach inherently requires first training a diffusion model on the full dataset, only then use it to create the distilled dataset. For instance, the diffusion trans-
former (DiT) model (Peebles & Xie )2023) used by Minimax (uu et al.] (2024) to distill ImageNet first requires a pre-training phase the full ImageNet dataset, consuming approximately 3,000 TPU-days.
Second, although some methods (D'M Su al. (2024)) turn to general text-to-image models without relying on such target-specific training, they suffer from a significant distributional mis- match, as the web-scale priors encapsulated in these foundation models fail faithfully capture "Corresponding authors.

---

Published as a conference paper at ICLR 2026
Original Data
Consral LOM
Imagekeet LDM
the target-specific semantics, leading to suboptimal per- formance. This mismatch is visually evident in Figure Ia The general LDM's samples are confined to a limited re- gion, failing to capture the data distribution and achiev- ing poor 52%. In contrast, the target-pretrained LDM achieves a strong accuracy of 60% precisely because it inherently capable of generating dataset that effec- tively covers the data distribution. This suggests that the remarkable success of these target-trained methods stems largely from this pre-existing alignment. A more detailed analysis in Appendix AA.12
Ours General LDM
(a) Distribution and performance of samples text-to-image LDM, LDM pre-trained ImageNet, and text-to-image LDM with our method.
In this work, we tackle these challenges by proposing Core Distribution Alignment (CoDA), a novel framework that enables effective DD using only single, off-the- shelf text-to-image diffusion model. The foundation of CoDA is our hypothesis that dataset's "intrinsie core distribution" is constituted by its most representative sam- ples (S,), which typically reside in high-density regions of the data manifold. Therefore, our approach consists of stages: first, in Distribution Discovery, we intro- duce a novel density-based clustering mechanism to au- tomatically and effectively identify these S,; second, in Distribution Alignment, we subsequently utilize S, to directly guide the generative process, aligning the gener- ated samples with core distribution.
(c) VAE space Figure (a): Performance compari- of different LDMs ImageNette (IPC=10). (b-c): ImageNette original features in ResNet18 (b) and VAE (c) latent spaces. Each color indicates a dif- ferent class.
(b) ResNet18 space
By doing so, CoDA effectively bridges the gap between general-purpose generative priors and target- specific semantics, yielding distilled datasets that are both compact and highly representative, as demonstrated by extensive visualization and quantitative evaluations. In summary, our contributions are follows:
We are the first to confront the conceptual flaws of existing generative DD methods, en- abling off-the-shelf text-to-image model to achieve sfate-of-the-arr performance.
We introduce the concept of an "intrinsic core distribution" and novel, density- -ased strategy to identify its representative samples, which are then used to guide the generative process and effectively bridge the model-dataset distributional mismatch.
Our method establishes new state-of-the-a performance across all seven major ImageNet subsets. Notably, it establishes new state-of-the-e of-the-art accuracy of 60.4% at the 50-images- per-class (IPC) setup on ImageNet-IK.
2 PRELIMINARIES
2.1 BACKGROUND ON DATASET DISTILLATION
[T We consider target dataset T = (1,, y: m where each sample x; IRd is drawn i.i.d. from a distribution ( with a ground-truth label ys E (1...., C). The goal of DD is to learn much
smaller synthetic dataset = [(u,, y/i) (5) where |S| < (T). This process can be formulated as an optimization problem, where the synthetic set S is learned to serve as an effective proxy for T. Specifically, the goal is to find an optimal set S* that minimizes the expected loss over the true data distribution when model is trained on S:
S* argmin ((z.))=((x)) [[[ v)]
(1)
where 0g represents the model parameters after training to convergence on the synthetic set S, fes the corresponding trained model, and & the loss function.
The degree of compression is measured by Images Per Class (IPC), while the quality of the synthetic set S is measured by the accuracy of a model trained on it against the original test dataset.
2

---

Published as a conference paper at ICLR 2026
2.2 LATENT DIFFUSION AND SCORE-BASED GUIDANCE
In this work, we employ Latent Diffusion Model (LDM) Rombach et al. (2022) that operates in compressed latent space 2 E(x:). encoded by VAE encoder E from images Diffusion Models (DMs) (Ho et al. (2020); Song et al. (2020a) train a network to predict the noise added during the forward noising process:
20 + v1 āye
(2)
The core principle of DMs the direct proportionality between the optimal noise prediction and the model's 's score function (Ho et al. 2020); Song et al. (2020b)):
logPe(2))
(3)
Vi- Oz
This equivalence implies that modifying the noise prediction is functionally equivalent to modi- fying the gradient field that the sampling process follows. This enables Guided Diffusion (Dhariwal & Nichol (202T): Bansal et al. (2023]; Yu et al. (2023a)). By introducing an external energy func- tion (EBM) (2)) the gradient of this function during sampling =c((((2) is superimposed onto the original prediction. This creates new, guided gradient field, forcing the generative process towards states that satisfy condition C. Our methodology builds upon this principle, adding an additional energy guidance field derived from the target dataset's ' core distribution S,.
3 METHOD
Our approach comprises two main stages: Distribution Discovery and Distribution Aligument. In the first stage, we identify the representative samples (8,) using a rigorous density-based clustering mechanism, primarily based on the HDBSCAN algorithm. Aside from the initial VAE encoding, the remainder of Distribution Discovery is executed on CPUs. We then employ a novel SplitCluster mechanism to ensure the number of samples in S, meets our target, Images Per Class (IPC). Sub- sequently, in the alignment stage, we introduce a new guidance term derived from S, to modify the predicted noise, thereby steering the sampling process of the generative model.
3.1 DISTRIBUTION DISCOVERY
The objective of the Distribution Discovery stage is to locate high-density regions within the target dataset's s manifold and, from these regions, to extract set of representative samples S,, This process begins by mapping the images into a suitable feature space where the manifold's structure can be analyzed. To adhere to our principle of relying on an off the--hhehhl generative model, we eschew any components trained on the target dataset. which includes avoiding the external feature extractors common in prior work (Zhou et al. (2022): Yin et al. (2023a); [Shao et al. (2024)) such ResNet without its classifier. Instead, we leverage the VAE encoder native to the LDM to map the target dataset into its latent space for the subsequent search for S,.
The VAE Latent Space Challenge. However, significant challenge arises from the intrinsic properties of the VAE encoder. The encoder's latent space is regularized by a standard Gaussian prior (O,, I). which compels all encoded data points into unified hyperspherical latent distribution (Kingma & Welling (2013)). While this regularization is essential for maintaining a smooth and continuous latent manifold suitable for the diffusion model's generative process, it inherently mixes features from different classes, thereby severely degrading inter-class separability and intra-class compactness. As shown in Figure Tbland and Figure Ic these factors collectively result in latent space that is not well-structured for direct clustering, exhibiting poor separation between classes compared features from target-tratt trained classifier.
Limitations of Prior K-Means-Based Approaches. Previous works, such as D'M (Su et al. (2024)) and MGD3 (Santiago et al. (2025)), have also attempted to identify representative proto- types in VAE latent space. However, unaware of failing to account for the aforementioned issues, they typically resort to a direct application of K-Means clustering for each class, forcing the num- ber of clusters, k, to be equal to the IPC. This one-step approach is brittle in the face of the latent space challenges for two primary reasons. First, K-Means implicitly assumes clusters are convex and isotropic, requirements that are fundamentally violated by the VAE latent space, which contains
3

---

Published as a conference paper at ICLR 2026
S the IPC
Run Kmeans with all outtiers
"O" Into Forced Split order of size
largest clusters
HOBSCAN
*sufficien?
Outliers sufficienti
Pass O into
Split crder
N
space after UMAP
Initial Clusers
Samples and outliers'
Step2
StepN
Figure 2: The workflow our density- pipeline. Red dots are samples found by HDBSCAN, while Blue dots are found by K-Means outliers (Black dots). Yellow regions highlight two new clusters split from a "mother-cluster..
arbitrarily shaped class structures within single Gaussian hypersphere. Second, K-Means is highly sensitive to outliers, as it assigns every point to a cluster. When the IPC is large, the algorithm' ob- jective of minimizing Within-Cluster Sum of Squares (WCSS) forces it to establish spurious clusters for outliers, thereby failing to identify the true high-density regions. However, we aim to create a compressed dataset utilizing information all original data points. Therefore, we deliberately avoid data purification techniques like LOF (Breunig et al. (2000)) that would discard outliers.
All these limitations and constraints motivate the need for more advanced mechanism capable of finding meaningful cluster centers without strong distributional assumptions, and one that can even make effective use of the outliers themselves.
Our density-based pipeline. Our Distribution Discovery stage employs a more robust, multi-step pipeline, which is applied independently to each class of the target dataset. We select HDBSCAN (Hicrarchical Density- -Baed Spatial Clustering of Applications with Noise) (Mcinnes et al. (0177) algorithm to automatically identify the high-density regions, tions about cluster shape, robustly handles outliers, and allows us to control the granularity of the discovered clusters via its min cluster size parameter. However, before applying HDBSCAN, first leverage UMAP (Uniform Manifold Approximation and Projection) (McInnes et al. (2088) to preprocess the latent embeddings from the VAE. This step only reduces dimensionality, enabling our density-based HDBSCAN to operate reliably, but also allows us to tune the local compactness of the VAE latent space to yield a more distinct clustering structure.
it makes no assump-
Following the initial HDBSCAN, we identify M valid high-density clusters from the data, which de- noted as the collection Co = f(i C2. ... Cas From each cluster e, E Cg. we then select the single point with the highest membership probability to form our initial set of representative samples, So = [8), {81,82-- .2...... ..)) A key challenge, however, that the size M data-driven and therefore highly unlikely to equal the target IPC. We devise post-processing scheme to resolve this discrep- ancy. The overall pipeline is shown in Figure 2 and the details of the SplitCluster and ForcedSplit The full pseudocode is provided in Appendix A.4
functions are provided in Algorithm [
M Z IPC, we select the representative samples corresponding to the IPC largest clusters to form the final set S,.. Conversely, if M < IPC, we bridge the gap by iteratively splitting clusters from Co to identify new representative samples.
Strategy 1: SplitCluster. We iteratively apply HDBSCAN the largest viable "mother- cluster" to identify its internal sub-clusters. We use the representative points (s;) from each sub-cluster as candidate initial centroids for K-Means, selecting the optimal pair that yields the lowest inertia for 2-way partition. This process successively replaces the single orig- inal sample with two new, well-placed ones. the number of samples is still insufficient and Strategy 1 fails to split any remaining clusters, we turn to Strategy 2.
2. Strategy 2: Clustering Outliers. In this stage, we utilize the unassigned outliers as secondary source. Applying K-Means exclusively to this population serves a dual purpose: it is complementary, as it finds samples in low-density regions that HDBSCAN ignored, and it is non-disruptive, as it prevents K-Means from altering the high-quality clusters already identified. If sufficient outliers exist, we apply K-Means directly to this population to identify the exact number of remaining samples needed, efficiently closing the gap in a single step. If there insufficient outliers, we turn to Strategy 3.
4

---

Published as a conference paper at ICLR 2026
Algorithm Post-processing Scheme for IPC Matching
1: Global variables: C, current clusters; S, corresponding representative samples.
2: procedure TLLTCLUSTERRCmoll min ciin.clusser_sizz
3: Run HDBSCAN on Cmother with win minn_cluseer-sia to find & sub-clusters as Seand
4: A <
return False
5:
S' S {mmother) and C C C. {Cmother
6:
Iterate all pairs (8,, 8j) from Seand as initial seeds for K-Means (k: = 2) on Cmothers select the
pair yielding the minimum resulting inertia, denoted {Cnewl, Cnew? } and { (Snewl Snew2
7: C' CU [nnewl Crew2 } and S & SU (Snwwl, Snew2
8:
return True
9:
end procedure
10: procedure FORCEDSPLIT(Cuolar min_size)
11: while SPLITCLUSTER((mm) min_size) False & min_size do min size f- max(2, min size x 0.75J)
12:
b Rellax the constraint
13: end while
14:
end procedure
3. Strategy 3: ForcedSplit. This stage works by gradually lowering the crin_cluster sice threshold until the cluster can be split into two. This entire process is repeated until the number of samples finally aligns with the IPC.
3.2 DISTRIBUTION ALIGNMENT
In the Distribution Alignment stage, we leverage the obtained set of representative samples, S, "is....,,SpP]) to steer the generation process. We achieve this by introducing a dynamic, data- driven guidance term derived from S,.. This term works by modifying the predicted noise. thereby augmenting the standard Classifier-Free Guidance (CFG) framework (Ho & Salimans (2022)
At each timestep t of the reverse process, we start with the noisy latent 24. This latent is processed by the U-Net eø to produce two noise estimates for CFG: the unconditional noise ep (2,, 0), which is intentionally left unmodified to preserve the backbone model's inherent stability, and the original text-conditional noise (2 1, c), which serves as the basis for our guidance.
First, we can estimate the predicted clean latent in from co(ze,t, c). using the following closed-form solution derived from Equation (2)
20(2)) = Vāa" (zt v1 Cn (()
Next, synthesize the final IPC images (per class), we employ a sequential assignment strategy: the generation of the j-th synthetic image (where j E {1,..., IPC)) is exclusively and consistently guided by the j-th representative sample (s, S,). Thus, for the entire generation process of the j-th image, the guidance vector at every timestep is defined as:
g(2z) sj 20((2)
(5)
This guidance is then scaled by a hyperparameter controlling the alignment strength, yielding our
required correction vector, A = Y g Applying this correction to the original prediction equivalent to linear interpolation: 2o,new (0(2) Aăo = (1-)))o(2a)) +2))) +ysy. To apply this Aêo during the denoising process, we must translate into an equivalent correction the noise space, Aeg. From Equation
4), we can derive the relationship between AEo and Aeo as follows:
(z+ /I ār(Es + Aee)) = 80(2)) - V
d
80,new
Aee
(6)
Va.
This leads our final formulation for the correction term:
(~>Yor))
Vāa
Aco = 7 )(2))
Finally, this correction term Acp is added to the original c0 to obtain ca c0,,,, Acg- This ea is then processed through the standard CFG framework, and the resulting final noise estimate used by the scheduler for the next denoising step.
5

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
4.1 EXPERIMENT SETUP
Datasets and Evaluation. Our method extensively evaluated on the challenging ImageNet-IK benchmark (224x 224) (Deng et al. (2009)) and its eight 10-class subsets 226),, including the commonly used ImageNette (Howard (2019), ImagelDC (Kim et al. (2022)), ImageWoof and the ImageNet-A through ImageNet-E sets from Cazenavette et al. (2023). The scope of our evalua- tion extends further in two key aspects. First, we include additional benchmarks like Places365 (256x 256) (Zhou et al. (2017) to demonstrate the versatility of the proposed method. Second, we perform multi-resolution experiments, testing at 128x 128 and establishing the first high-resolution baseline for dataset distillation at 1024 x 1024.
We evaluate our distilled datasets using two standard protocols. For the hard-label protocol, we follow the established setup of Gf et al. (2024). For our ImageNet-IK evaluation, we employ a soft- label knowledge distillation framework from Sun et al. (2024), where both the teacher and student models are ResNet-18 architectures.
Baselines. Our diffusion- based baselines are categorized into two groups: (1) methods that uti- lize generative models pre-trained or fine-tuned on the target dataset, including Minimax (Gu et al. (2024)) LD3M (Moser et al. (2024)). IGD (Chen et al. (2025)), MGD (Santiago et al. (2025)) p), as well as the standard LDM (Rombach al. (2022) and DiT (Peebles Xie (2023)); and (2) meth-
ods that employ truly off- offlt models, D'M (Su et al. (2024)) and standard SDXL (Podell et al.
(2023)) Furthermore, for a more extensive verification, we also include several key non-diffusion
paradigms: Methods based on Generative Priors (GLaD Cazenavette et al. (2023); H-PD Zhong
cct al.] (2025)); Statistical Matching (SRe2L Yin et al. (2023a); G-VBSM Shao et al. (024 )) and
Patch Assembly (RDED Sun et al. (2024)).
Implementation details. For the Distribution Diseovery stage, we configure UMAP with an output dimension of 50 and minndiss = 0.0 maximize cluster density. For HDBSCAN, we fix min_samples = 3 to preserve the data's hierarchical structure. The key hyperparameters for this stage are UMAP's n_neighbors and HDBSCAN's min Alignment stage, we employ SDXL as our base generative model with a DPM++ Karras sampler over 50 inference steps (Lu al. 2025): Karras et al. (2022), ) The text prompt for each class the first part of its official label, while the negative prompt is left empty and the CFG scale is fixed at 5.0. The primary hyperparameters are our alignment guidance strength (? in Equation 7 and Prior Injection Steps (PIS). We define PIS as the final number of steps that revert to the standard CFG, allowing the model's prior to naturally refine image details and enhance photorealism.
For the Distribution
4.2 COMPARISON WITH STATE-OF-TIEE-AA ART METHODS
ImagelDC and IeageNette. The primary results of our method against state-of-the-ar (SOTA) approaches on ImagelDC and ImageNette are presented in Table For fair comparison, the results for D'M are based on the hard-label protocol.
Our pipeline establishes a new and decisive SOTA across all IPC configurations on both datasets. Specifically, it outperforms the prior SOTA by 2.6% %, 2.7%, and 5.5% on ImageIDC for IPC=10, 20, and 50 respectively, and by 2.4% 0.9%, and 3.6% on ImageNette.
These results also reveal the impact of the distributional mismatch. The text-to-image SDXL shows a significant performance drop compared to the LDM and DiT pre-trained on ImageNet. Meanwhile, MGD ' performance also collapses when transplanted the SDXL, suggesting effectiveness not from its algorithm alone, but largely dependent on the pre-existing distributional alignment.
Ioof.Woof. We also provide a comprehensive comparison on ImageWoof, a challenging 10-class subset characterized by extremely high inter-class similarity. As shown in Table 2 while previous SOTA methods maintain a marginal advantage in low storage budget settings (IPC < 20), our method establishes new SOTA across all configurations with IPC > 50, demonstrating its robustness on challenging. fine-grained datasets. For example, CoDA achieves 71.4% on ResNet- 18 at IPC=100, significantly outperforming the prior SOTA of 68.8%

---

Published as a conference paper at ICLR 2026
Table I: Comparison with state-of-the-art methods on ImageIDC and ImageNette across various IPC settings. All methods are evaluated by training ResNet10-AP at 256x256 resolution. The best results are in bold, and the second best are in bold.
IDC
Nette
Type
IPC
10
20
50
10
20
50
Random
48.1+-0.8
52.5+0.9
68.1+0.7
54.2+
63.5++0.5
76.1+1.1
Minimax
53.1=:0.2
59.0=1.44
69.6+0.2
62.0+0.2
66.8+0.4
76.6:0.22
Based on Model Pretrained With
LDM
50.81
$5.112.0
63.810.4
60.31:..6
62.0:1..6
71.011.44
LDM+MGD*
53.2.10.2
58.11..7
6..211.
61.91:411
65.311.3
74.210..
ImageNet
DiT
54.1100.4
58.910.2
64.3200.6
59.110.7
64.811..
73.310.99
MGG+
55.91:2.1
61.910.9
72.1-0.8
66.4122
71.2:10.5
79.51:13
SDXL
40.1
41.8+022
4..1
51.7=0.22
56,1
67.3+0022
Based on General text-to-image Model
SDXL+D'M
41.1110
45.110.7
56.610.2
58.4114
66,6119
73.410,77
SDXL+MGD*
47.2110.6
52,01100
61.41.
59.110.55
63.910.8
75,510,9
SDXL+Ours
58.5.0.99
64.6-1..5
77.610.6
68.810.11
72.110.2
83.1
Table 2: Performance comparison with pre-trained diffusion models and other SOTA methods on ImageWoof. All the results are reproduced by us for the 256x256 resolution. The missing results are due to out-of-memory. Results shown for the previous works are from MGD3 (Santiago et al. (025 Best result bold and second best bold. The arrow annotations indicate the performance difference between CoDA and the previous SOTA.
JPC
Test Model
Randon
Herding
DM
IDC-1
GLaD)
Minimax
MGD
CoDA(Ours)
Full
C
2..3
26.7.0.5
34,2=111
26.9=12
11
33.820.9
37.0::10
34.7.1111
((1.))
06.430.2
(0.8)
10
29.4+0.8
3204-03
34.7-0.
30.3-1.2
39.1+0.5
32.+009
39.2-1.3
40.4-1.9
39.2+0.7
875++0.
ResNet-18
27.7+0.9
30.241.
34.7=0.4
33.4-0.7
37.34022
31.7408
37.6=0.9
38.5-2.5
38.8
(10.3)
89.3
ConvNet-6
29.1100.7
29.5203
36.1208
29.9=1.0
35.5208
37.6:009
39.0:3.5
38.1:1.3
(10.9)
86420.2
(1.66)
ResNetAP-10
32.710.4
34.10..1
41.120.8
35.2-00.6
43.41.3
45.8=0.5
43.0.1..6
42.510.
87.510.5
ResNet-18
29.7+0..
32.240.6
40.5-0.5
29,8-1.7
38.6+0.2
42.5-0.6
41.9+2.11
42.0+0.
(10.5)
89.3+1.2
ConvNe-66
41.320.6
40.320.7
46.520.8
411=10
13.9112
$3.920.6
54.5-:166
$6.8.0099
(12.3)
86/110.2
(3.8%)
ResNetAP-10
47.2s11.3
49.3:0077
49.3=002
47,1=111
48.3:10
$6.3=1.0
56.5::199
59.4:
(12.9)
87.520.5
ResNet-18
48.31:12
50.120..5
46.220.6
57.120.6
58.311.4
61.210.9
(12.9)
89.3
ConvNet-6
46.3+0066
46-2+016
50.1+1.2
47.5-008
48.9+077
58.7-0.9
55.1+2.5
56.4+1.2
(11.3)
864+0022
(5.4%)
ResNetAP-10
50.8110.6
5341144
$4.3:009
51.7=0.8
528118
58.3=0.2
60.2:12.4
61.2::0.7
(11.0)
87.510.5
ResNet-18
52.1:1.0
49.720.
51,5=110
51.9:088
588:00.
59.7:2.77
62.4:114
(12.7)
89331.2
ConvNet-6
52.2:10.4
54.411.1
53.420.33
55.0=113
53.2.009
61.1-0.7
60.1
62.41:0.7
(11.3)
86.420.2
(7.7%)
ResNetAP.10 10
59.4+1.0
61.7409
58.3+08
564-0.8
56.1+09
64.5-0.2
66.5-1.0
67.6++0.8
(11.1)
87.5+0.3
ResNet-18
61.51.3
39.320.7
58.9213
60.2:110
58.3:1.2
65.7=0.4
(12.6)
89.321.2
ImageNet-A to ImageNet-E. We evaluate the cross-architecture performance of our method against several generative baselines with four distinct downstream architectures: AlexNet (Krizhevsky et al. (2012)). VGGII (Simonyan & Zisserman (2014)). ResNet18 (He et al. (2016). and VIT (Dosovitskiy et al. (0200). Specific data are presented in Table 3 Our method demon- strates superior arcsttarch generalization, outperforming the prior SOTA by 9.6%, 8.4%, 7.5%, 4.8%, 6.6 % on ImageNet-A to ImageNet-E, respectively. We provide the results other resolutions and more detailed breakdown of the cross-architecture performance in Appendix A.8
ImageNet-1KK We compare our method against various categories of SOTA baselines on ImageNet-1K 1K using the soft-label protocol, which detailed Table
At IPC=10, while IGD, which relies on an ImageNet-trained diffusion models, holds the top posi- tion, our method achieves a massive lead over our truly off-the-shelf competitor D (+16.4% and performs par with the excellent ImageNet-traing method Minimax. At IPC=50, we not only de- cisively outperform D (+5.2%), ), but also surpass the strongest baseline IGD, establishing new SOTA accuracy of 60.4%.
Baselines on 1024 x 1024 Resolution Images As shown in Table 5 we provide baseline re- sults on seven common subsets at resolution of 1024x 1024 at IPC=10. We present a broad cross-architecture performance comparison, testing ResNet18, ResNet50, ResNet101, ResNet10 with average pooling and ViT architectures for ImagelDC, ImageNette, and ImageNet-A through ImageNet-E. All experiments use hard labels and adhere to the setup in Gu et al. (2044). and all results the mean and standard deviation of three independent runs.

---

Published as a conference paper at ICLR 2026
Table 3: Comparison with SOTA methods ImageNet-A ImageNet-E with IPC=10. All results are the average accuracy across all four architectures and five independent runs.
Table 4: Comparison with SOTA on ImageNet- IK using the soft- label protocol.
ImNet-A
ImNet-B
ImNet- C
ImNer-D ImNer-
Type
Method
IPC=10
IPC=50
52.3:1077
45.11833
40.11:.66
36.11004
38.1:0.4
SRc21.
21.3
46.8.10.2
DC
GLaD
H-PD
54.1=112 52.0-111
13.1:1144
50. 1..1
49.5+088
4.99
39,8+0.7
38.9 1.0 18.4-0.7
40.1=0.7
Non-diffusion
G-VBSM
31.4=0.5
51840.4
55
51.811
49.911.
39.51.0
39.0=1.3
RDED
42.0::011 56.51.0.1
Pixel
44.4=00.5 52.6+044
50.6+05
47.5+0.7
35.4=0.4
Minimax
44.3=0.5 58.6
DM
H-PD
GLaD
55.1=00.5 54,2++0.5
5288=10 51,3+006
50.80044
49.74004
37.6
36.4004
39.9=0.7
38.6=0.7
Model Trained
On ImageNet
IGD
DIT
46.2=0.6 60.3+00.4
39.6=0.4 52.9
LD3M
1. 52.3.1.1
48.2.449
3.5
3.
MGD
63.4-008
66.31-111
5.
4688:0.8
General Model
D'M
27.9=0.2 55.200011
Ous
73.0=007
74.7=112 66.1+11
51.60088
57.7=0.7
44.3=001 6..4
Table 5: Cross-architecturee comparison on the ImagelICC, ImageNette and ImageNetA-F datasets under the IPC10 setting 1024x 1024 resolution. The evaluated architectures include ResNet18, ResNet50, ResNet101, Res and ViT. Note that each result represents the mean and stan- dard deviation obtained from three independent runs of the architecture in the current row.
IDC
NetteNette
ImageA
ImageB
ImageC
ImageD
ImageE
ResNet10-AP
54.9 = 1.7
57.1
1.1
79.2 2 0.2
61.100.
72.2 0.4
52.3 0.1
62.7 1 0.5
ResNet18
5..5
56.9 200
79.7 +
0.5
71.4 +
52.7 0.6
64.9
ResNet50
46.4
52.9 1.3
77.1 0.1
73.2 1.4
70.6
49.7
2.2
61.7
ResNet101
45.8 0.5
52.0
+ 0.5
72.0
71.8
0.8
1.1
50.6 1.3
61.0 0.9
VIT
47.5 1.0
49.0
+ 1.2
72.1
0.8
70.5 1.2
61.8
1.8::00 0.7
47.6 1.2
56.6 1.6
Table 6: Comparing Ours(R) and Ours(G) against the SOTA baseline on ImagelDC and ImageNette across different IPC. The best results are in bold, and the second best are in bold.
Table 7: ResNet18 accuracy on the first 10 classes of Places365 at IPC=10.
Method
ImngeIDC 20
50
10
ImageNette 20
Method
Accuracy (%)
41.7
DIT-MGD
25.9-2.1 1 61 9-:09
72.11-88
6.44224
71.2:105
79 5.133
SDXL
33.9 4:
Ours (7C)
36.3-199
6288196
78.711
6611111
70.7206
81.924.1
SDXL KMeans
34.4 4 1.6
Ours (6)
58.5
64.6220.5
77.6:20.6
6881811
72.120022
K3.124.3
SDXI. + Ours
47.0 L.6
ABLATION STUDY
Ablation of Method Components. Each stage of our method yields a distinct dataset. The first stage, Distribution Discovery, produces R: set of selected and resized real images from the original dataset. This set represents the output of our efficient, CPU-based process alone. The subsequent Distribution Alignment stage produces G: the final set of generated images.
The results in Table 6 are striking. Our CPU-derived set R is sufficient to outperform the SOTA baseline across nearly all settings. Specifically, it surpasses the prior SOTA by 0.4%, 0.9%. and 3.6% on ImageIDC for IPC=10, 20, and 50, and by 2.4% on ImageNette for IPC=50. Leveraging this strong foundation, generated set G improves performance even more significantly.
Generalizability and Computational Efficiency. Our framework's generalizability and effi- ciency provide key advantage. Crucially, our framework achieves genuine zero-shot generaliza- tion, performing well on new domains like Places365 without any modification, as validated in Table As the results show, our method significantly improves on the base SDXL model's perfor- mance and is substantially more effective than a direct K-Means. This overcomes the fundamental limitation of target-trate trained methods, which are locked into their original training domain and require another tholi-ttouuand TPU-day re-training effort to adapt to new datasets. A further case study on domain change is provided in Appendix A.9
Moreover, the efficiency is evident across both stages. For example, on ImageNet- IK, the Distri- bution Discovery stage completes in 2.2 hours for all 1,000 classes. The subsequent Distribution Alignment stage only adds few arithmetic operations per step, which introduces negligible com- putational overhead compared to the U-Net forward pass, generating a 224 image within 2.7s.
A more detailed computational analysis is provided in Appendix A.10
Efficacy of Distribution Discovery Components. To demonstrate the importance of the "intrinsic core distribution" and our discovery stage's components, we conduct step-by-step ablation study.
8

---

Published as a conference paper at ICLR 2026
MGDHNN
Accutayy
Mengettd
56.5
61.3
Dotautt
inagelleta IPC 10
Oelost Imagelati ICC
Daland magpiiett INC 50
723
d0.1
min size
Figure 3: Step-by-step ablation on the efficacy of our Distribution Discovery stage components. (G)*(a) is the ImageNet-trained SOTA baseline; MGD*(b) the direct application of MGD to SDXL; MGD²(c) adds UMAP before K-Means; MGD (d) uses our full discovery pipeline.
Figure 4: Analysis of min_size sensitiv- ity on ImagelDC at IPC=50. The solid line tracks the accuracy of a model trained on the real-image set (R). The stacked area chart illustrates the proportional source of the final IPC samples.
Table 8: Performance across different y and PIS val- on ImagelIDC (IPC=10). PIS=50 is equivalent to = 0. Cells highlighted in yellow indicate perfor- mance greater than the SOTA of 55.9.
Table 9: Performance with only 25 infer- ence steps. Cells highlighted in yellow in- dicate performance greater than the SOTA of 55,9.
7.
5-
1=0.03
7=0.05
1=0.08
:=0.10
45.7a3.
5472112
57.720.
16.1200
35.92113
57.1:15
359213
PIS=0
56.911.7
58.510.6
55.410.7
55.411.8
M65-5
45.5:10
$4.100
SS8:02
$6.:05
55.5309
570:111
969214
PIS=2
56.1+0.5
59.3440.6
56.0+111
55.9-006
PIS-10
46.91.
54.3110
500:10
S7.1503
57.34133
967:10
555504
PIS=5
56.510.9
59.111.2
55.8.066
55.310.3
PI5-20
PI5:75
45.1205
453:10
45.0004
51.340.5
54.2.0.7
54.2007
56.5-15
$72:111
57.1:03
36761.3
169106
$67100
57.1810
57.1303
$65-0077
566210
56.5-1
58.1:08
563-18
$64209
559911
P1S=10
PIS=8
55,910.4
56.5+1.5
59.110.8
59.54144
55.5+1.2
56.510.7
55.611.3
56.5-0.6
43.908
52.1408
55.5
547:16
$25511
155=15
55.54007
60.141.0
56.34-1.0
55.4+14
PIS-40
423::08
50.3210
52.3:133
56.211.
$1.5110
47.7:12
40.7813
P1S=20
50.1
56.3+0.6
59.140.4
15.0=14
P1S:30
401--0
401108
40.1-08
10.1
401103
40.1-08
401-088
25
39.7+110
39.7+1.0
39.7411.0
39.7+-1.0
We validate our approach by progressively enhancing the MGD (Santiago et al. (2025)) baseline on SDXL, with full results in Figure For fair comparison. and because UMAP embeddings cannot be inverted, the virtual cluster centers in all MGD3 variants (b, c, d) are replaced with the nearest real data points, consistent with our main method's protocol.
The results show a clear, cumulative benefit. While MGD*(b) improves on the base SDXL, it still lags far behind the ImageNet-trained SOTA (a). By introducing UMAP for manifold pre- processing (M)D we observe substantial performance boost. Crucially, substituting the prototype selection process with our complete Distribution Discovery pipeline (MGD*(d) dramati- cally increases accuracy, surpassing the SOTA (MGD*(a) in most scenarios. This demonstrates the superior ability of our method to identify the "intrinsic core distribution".
Hyperparameter Analysis for Distribution Discovery. The performance of the Distribution Dis- covery stage is influenced by several hyperparameters, with the HDBSCAN min size being the most critical. This parameter dominantly controls the behavior of our multi-strategy post-processing design (detailed in Section 3.I. , as it directly governs the source of the final samples: whether they originate from initial HDBSCAN clusters, subsequent splits, or K-Means applied to outliers. As illustrated in Figure 4 this mechanism reveals an optimal setting (min_size E [10, 20] for IPC=50) where these strategies act in synergy. While the accuracy gracefully degrades outside this range, method remains robust, consistently outperforming the SOTA baseline. A detailed region-by-reginn analysis is provided in Appendix A.5 The analysis of other hyperparameters for this stage, such as the nenneghoorss of UMAP, can be found in Appendix A.6
Hy Hlyperaaraeeter Analysis for Distribution Alignment. The Distribution Alignment stage is governed by two hyperparameters: the alignment guidance strength y (from Equation 7 ) and Prior Injection Steps (PIS). Jointly, they regulate the critical trade-off between the explicit guidance from S, and the model's 's inherent generative prior.
The results in Table 8 yield several key insights. First, even minimal guidance (y 0.03) dramat- ically outperforms the base SDXL by 14.6%. Second, our method exhibits remarkable robustness, consistently surpassing the SOTA baseline across wide y range of [0.05, 0.20), provided appro-

---

Published as a conference paper at ICLR 2026
Table 10: Ablation study on feature spaces Table 11: Ablation study on transferring CoDA to an and clustering methods. All results are for ImageNet- trained DiT model. All results are the mean ImagelDC (IPC=10) and are the mean of of three runs on ResNetAP10. Best results are in bold, three runs on ResNetAP10. Best result is second best are in bold.
in bold, second best is in bold.
Dataset
IPC
DIT
DiT-MGD3
DiT-CoDA
SDXL+CoDA
Dim
Pre
Space
Kmeans
CoDA
14.1
559+21
59,4+1.5
58.5+0.9
3136
None
VAE
1..0=111
45.8=0..9
ImagelDC
20 58.910.2
619110.9
65.821.2
64.6:10.5
None
CLIP
$1.811.2
52,410.7
50
64.310.6
72.140.8
74.6:166
77.6:10.6
768
PCA
VAE
46.620.6
53.0=1.0
10
59. 1:0.7
66.4:224
65.211.5
68.8=0.1
PCA
CLIP
VAE
49.8=0.5
$1.7=0.8
54.6+t0.6
53.610.9
ImageNoute
73.3::09 50
20 64.82:12
71 21:55
79.5+:1.3
81,4
73.0::1.2
72.1:10.2
83.1::0.3
50
34.7.0.5
4.
39.2411.3
39.2
UMAP
CLIP
54.410.77
56.01:1.2
ImageWoof
20
41.1+088
43.6+116
44.2+00.9
42.5++0.6
VAE
54.8=007
58.5::0.9
50
19.3::022
56,54:9
59,2:10.7
59.411.0
priate PIS (PIS < 25). Third, allowing the model to freely inject its own prior knowledge during the final generation steps is crucial, as peak accuracy consistently achieved when PIS >
We also find that reducing the inference steps to 25 can even yield higher peak accuracy than 50 steps (as shown in Table D This increased efficiency, cutting the 1024 1024 generation time to just 3 seconds (1.3 seconds for 256x 256), comes with reasonable trade-off in hyperparameter robustness. Further analysis is provided in Appendix A.7
Transferring CoDA to Different Encoders. To validate CoDA's versatility, we transferred the Discovery stage to the CLIP (ViT-L/14) feature space. As shown in Table the CoDA framework effectively adapts to diverse feature spaces. Notably, its optimal result in the CLIP space (56.0%) on par with the previous SOTA MGD* (55.9% from Table D. However, this peak performance is still surpassed by our VAE-space result (58.5%). We attribute this to "Domain Gap": the CLIP feature space (Zeli)) is optimized for tixt-visual alignment, whereas our downstream models, being purely visual, prefer prototypes selected in space that prioritizes the visual manifold, as further analyzed in Appendix IT Furthermore, to guide SDXL generation, is necessary to retrieve the source images corresponding to Zetipp and re-encode them with the VAE to get Z (8z). This discrepancy arising from the indirect encoding process degrades peak performance.
Transferring CoDA to Traiie- Models. As shown in Table I1 transferring CoDA to DiT significantly improves DiT's baseline capability. At the IPC=50 setting, surpasses the original DiT by 10.3%, 8.1 %, and 9.9 % on ImagelDC., ImageNette, and Image Woof, respectively. This combi- nation outperforms the previous SOTA (DiT+MGD*) in most configurations, and even exceeds our "SDXL+CoDA" pipeline in several cases. This demonstrates that CoDA can be seamlessly adapted to various generative architectures, provided a mechanism for external guidance exists.
We further analyze the optimal hyperparameters and observe shift between low and high IPC settings. For low IPCs (5 20). the optimal parameters (7 = 1.0, PIS = 15) allow greater gener- ative freedom. However, at IPC=50. peak performance demands tighter alignment with the real dataset R (7 = 0.8, PIS = 8). This suggests that while DIT's inherent knowledge is beneficial, it is diveretty-limited:
the demand increases to IPC=50, the model saturates and necessitates stronger guidance from the R to achieve peak performance. Additional visualizations are provided in Section A.14 To further demonstrate CoDA's cross-architecture applicability, we also apply its alignment stage to the Stable Diffusion 3 (SD3) framework, as detailed in Appendix A.
CONCLUSION
In this work, we identify the conceptual flaws in the prevailing dataset distillation paradigm that relies on diffusion models pre-trained on the full target dataset. We introduce CoDA, training- free framework that enables effective DD using only an off-the-shelf text-to-image model to address these drawbacks. Our core strategy involves two stages: first, discovering the dataset's "intrinsic core distribution" " using a novel, density-based mechanism, and second, aligning the generated samples with this discovered distribution. Extensive experiments demonstrate that by directly addressing the distributional mismatch, the proposed CoDA achieves performance on par with or even superior to previous methods relying target-specific pre-training.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We have adhered to the ICLR Code of Ethics throughout this research. Our work built upon publicly accessible research assets, including standard academic datasets (ImageNet, Places365) and the open-source model SDXL, all used in accordance with their respective licenses.
A significant ethical advantage of our dataset distillation approach lies in the generation of entirely synthetic data. By creating a distilled set that contains no real-world images, our method inherently provides strong protections for the privacy of individuals and the copyright of original creators.
While our method offers these robust privacy and copyright advantages, we recognize the broader challenges within the field of generative AI. Foundational models like SDXL,, due their training on web-scale data, may inadvertently reflect societal biases. Our current framework focuses on faithfully reproducing the source dataset's distribution and does not include an explicit de-biasing mechanism. Therefore, addressing the potential for bias propagation from the source data to the distilled remains an important consideration and valuable direction for future research.
In summary, we believe our method represents positive step towards creating more efficient, secure, and privacy-conscious machine learning workflows.
REPRODUCIBILITY STATEMENT
To support the reproducibility of our research, we have provided key materials in the supplementary file attached to this submission. This includes a representative subset of the datasets distilled by our CoDA framework for several key benchmarks, along with the complete evaluation code required to validate their performance.
ACKNOWLEDGEEMENTS
This project supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (Award Number: MOE-T2EP20122-0066
REFERENCES
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 843 852, 2023.
Markus Breunig, Hans-Peter Kriegel, Raymond T Ng. and Jörg Sander. Lof: identifying density- based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference Management of data, pp. 93-104, 2000.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4750-4759, 2022.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Gener- alizing dataset distillation via deep generative prior. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3739-3748, 2023.
Mingyang Chen, Jiawei Du, Bo Huang, Yi Wang, Xiaobo Zhang, and Wei Wang. Influence-guided diffusion for dataset distillation. In The Thirteenth International Conference on Learning Repre- sentations, 2025.
Justin Cui, Ruochen Wang. Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation imagenet- Ik with constant memory. In International Conference on Machine Learning, pp. 6565-6590. PMLR, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-24 255.255. leee, 2009.

---

Published as a conference paper at ICLR 2026
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 11188800-7994 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1135. PMLR, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commun, ACM, 63(11); 139-144, October 2020. ISSN 0001-0782. doi: 10.1145/3422622. URL [tttattss///o.ooo
10. 10..1145200 001114/3422222 3422622
Jianyang Gu, Saeed Vahidian, Vyacheslav Kungurtsev, Haonan Wang. Wei Jiang. Yang You, and Yiran Chen. Efficient dataset distillation via minimax diffusion. In Proceedings of IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 15793-15803, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Jonathan Ho and Tim Salimans. urXiv:2207.12590 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851. 2020.
Jeremy Howard. A smaller subset of 10 easily classified classes from Imagenet, and a little more Accessed: 2025-09-
French, 2019. 08.
Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 35:26565-26577 2022.
Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameteriza- tion. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 11102-11111 11118. PMLR, 17-23 Jul 2022. URL httpp:://proceeeeee
mlr. http://poeeeciaoo....rrrrereeeaa
html
Diederik P Kingma and Max Welling, Auto-encoding variational bayes.
arXiv preprint
arXiv:1312.61 14, 2013.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. Machine Intelligence Research, pp. 1-22, 2025.
Leland Mclnnes. John Healy, Steve Astels, et al. hdbscan: Hierarchical density based clustering, J. Open Source Softw, 2(1 1):205, 2017.
Leland MeInnes, John Healy, and James Melville. projection for dimension reduction. arXiv preprint
Umap: Uniform manifold approximation and 2018.
Brian B Moser, Federico Raue, Sebastian Palacio, Stanislay Frolov, and Andreas Dengel. Latent dataset distillation with diffusion models. arXiv preprint arXiv:2403.0388) 2024.
12

---

Published as a conference paper at ICLR 2026
Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kemel ridge- regression. arXiv preprint arXiv:2011.00050 2020.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4195 4205, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint urXiv:2307.01952, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Jeffrey A Chan Santiago, praveen tirupattur, Gaurav Kumar Nayak, Gaowen Liu, and Mubarak Shah. MGD$ Mode-guides dataset distillation using diffusion models. In Forty-second International Conference on Machine Learning, 2025. URL htttps:///ppereeeee
net./
Forum?id
Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. Generalized large- scale data condensation via various backbone and statistical matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16709-16718, 2024.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint vrXiv:1409.1556 2014.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020a.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint ar.iv:2001.134555 2020b.
Staal, M.D. Abramoff, M. Niemeijer, M.A. Viergever, and B. van Ginneken. Ridge-based vessel segmentation in color images of the retina. IEEE Transactions on Medical Imaging, 23(5: 509, 2004. doi: 10.1 109/TMM.2004.825627
Duo Su, Junjie Hou, Weizhi Gao, Yingjie Tian, and Bowen Tang. D4: Dataset distillation via disentangled diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5818, 2024.
Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9390-9399, 2024.
Tongzhou Wang, Lun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv
preprint arXiv: 10959, 2018
Lingao Xiao, Songhua Liu, Yang He, and Xinchao Wang. Rethinking large-scale dataset compres- sion: Shifting focus from labels images. arXiv preprint N.Xiv:25002004344, 2025.
Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective. Advances Neural Information Processing Systems, 36: 73582-73603, 2023a.
Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from new perspective. Advances in Neural Information Processing Systems, 36: 73582-73603, 2023b.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training-fie free energy-guided conditional diffusion model. In Proceedings s of the IEEE/CVF International Con- ference Computer Vision, pp.
2023a.
13

---

Published as a conference paper at ICLR 2026
Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: A comprehensive review. IEEE transactions on pattern analysis and machine intelligence, 46(1):150-170, 2023b.
Ruonan Yu, Songhua Liu, Jingwen Ye, and Xinchao Wang. Teddy: Efficient large-scale dataset distillation via laylor-aprrxxmmated matching. European Conference on Computer Vision, pp- 1-17. Springer, 2024.
Ruonan Yu, Songhua Liu, Zigeng Chen, Jingwen Ye, and Xinchao Wang. Heavy labels out! dataset distillation with label space lightening. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5017-5026, 2025.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint (tth:20066059229 2020.
Lin Zhao, Yushu Wu, Xinru Jiang, Jianyang Gu, Yanzhi Wang, Xiaolin Xu, Pu Zhao, and Xue Lin. Taming diffusion for dataset distillation with high representativeness. arXiv preprint arXiv:2505.18390 2025.
Xinhao Zhong, Hao Fang, Bin Chen, Xulin Gu, Meikang Qiu, Shuhan Qi. and Shu-Tao Xia. Hier- archical features matter: A deep exploration of progressive parameterization method for dataset distillation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 30462-30471, 2025.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence,
2017.
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regres- sion. Advances in Neural Information Processing Systems, 35:9813-9827, 2022.
A APPENDIX
A.I THE USE OF LARGE LANGUAGE MODELS (LLMS)
In line with ICLR 2026 guidelines, we report the use of a large language model in preparing this paper. Its function was strictly as writing aid, assisting in the refinement of language and en- hancement of textual clarity. We confirm that the LLM did not contribute to the conceptualization of the research, the design of the methodology, or the analysis of the results. The authors are solely responsible for all scientific contributions presented.
A.2 RELATED WORK
Traditional Methods. Early paradigms in Dataset Distillation (DD) were often hampered by pro- hibitive computational demands. This limitation largely stemmed from their reliance on complex bi-level optimization frameworks Finn et al.] (2017). Mainstream approaches, including Perfor- mance Matching (Wang al. (2018): Nguyen et al. (2020); /hhou et al. (2022)) and Parameter Matching (Zhao al. (2020); Cazenavette et al. (2022); Cui al. (2023)), emblematic of this dependency. Such frameworks necessitate backpropagation through long, unrolled optimization trajectories, imposing substantial computational and memory overheads that severely restrict their scalability. Consequently, the application of these methods has been predominantly confined to small-scale, low-resolution datasets such CIFAR-10/100 (32 32 pixels) and MNIST (28 x 28 pixels).
Methods Based on Target-Trained Diffusion Models. To overcome the scalability issues of tra- ditional methods, recent wave of research has shifted focus towards leveraging powerful genera- tive models for dataset distillation. However, many prominent methods within this paradigm, such LosMr
Minimax Gu et al. 22024), LD3M Moser et al. 2024), IGD Chen et al. (2025) and MGD* San- (iago et al.] (2025), paradoxically require a diffusion model pre-trained on the full target dataset, undermining the very purpose of DD and incurring prohibitive costs.
14

---

Published as a conference paper at ICLR 2026
MGD3 serves case in point. It first encodes the target dataset into VAE latent space and then applies K-Means clustering to identify virtual 'mode' prototypes. Guidance is then applied during denoising by directly adding a heuristic correction the predicted clean sample (2)), steering towards these prototypes. This approach fundamentally differs from our CoDA framework, which paarericalll-grounnde
instead modifies the predicted noise (ca). CoDA achieves this through mapping that translates a desired interpolation in the data space into a self-consistent correction in the noise space.
Methods Based on General Text-to- Image Diffusson Models. A few "logically correct" meth- ods, such as the notable D'M Su et al. (024)) turn to general text-to-image models like Stable Diffusion. The process begins by encoding the target dataset into VAE latent space and then ap- plying Mini-batch K-Means to identify representative prototypes from the real data points closest to the cluster centers. These selected prototypes are then noised to serve the initial latents for the reverse diffusion process.
Crucially, no further guidance is applied after this initialization. This hands-off approach gives the model's powerful, general-purposs priors free rein, which in turn can cause the generation to drift from the target distribution, overriding the valuable information in the initial prototypes and ultimately leading to suboptimal performance. This stands in stark contrast to our CoDA framework, which begins from pure Gaussian noise but applies continuous, targeted guidance throughout the entire generation process to ensure alignment with the core distribution.
A.3 LIMITATIONS AND FUTURE WORK
While the CoDA framework exhibits strong robustness across wide range of parameters, in pursuit optimal performance, we observed subtle drift its key Distribution Discovery hyperparameters: UMAP's n_neighbors and HDBSCAN's cein_cluuter_siee The root cause is that each dataset has its unique "intrinsic core distribution", which inevitably influences our density-based discovery mechanism. For instance, under the IPC=10 setting, the optimal config- uration for the ImageNet-1K K and ImageNet-A through E subsets was min wtt.:cuustesssisseeee and n. In contrast, the optimal configuration for ImageIDC, ImageNette, and Places365 shifted to min_clusee www.stutsssieee533. and n..nighhor==90, Although this difference is minimal and the Distribution Discovery stage can be fully optimized via grid search on a CPU, this approach is ul- timately not the most elegant or efficient solution and represents a key area we aim to improve. A promising future direction is develop meta- meta-learning model that automatically selects optimal hyperparameters by analyzing the intrinsic variance and density of the target dataset's distribution.
A.4 FULL PSEUDOCODE FOR THE POST-PROCESSING SCHEME
This section provides the full pseudocode for our post-procest proccssccsing scheme, detailed in Algorithm 2 This algorithm designed to address the gap between the initial number of representative samples (M) and the target number IPC. It resolves this discrepancy by iteratively identifying new samples, either by splitting existing "mother-clusters" or by applying K-Means to the population outliers until the sample count matches the IPC.
A.5 ANALYSIS OF POST-PROCESSING COMPONENTS
This section provides the detailed, region-by-region analysis of the post-processing components, as summarized in the main paper. We examine how the core HDBSCAN hyperparameter, min size, governs the source distribution of the final distilled samples, which in turn dictates the final model performance. The analysis is supported by the data presented in Figure 4
Region 1 (min_size < 5). When the threshold too low, HDBSCAN produces multitude of trivial, fine-grained clusters, causing the initial count M to vastly exceed the / IPC budget. This forces the post-processin discards the contributions of all original data points belonging the remaining M IPC clusters, leading to significant information loss. Consequently, performance in this region gradually rises min size increases, as this higher threshold prevents the fragmentation of the data, effectively
scheme to only perform simple truncation. This approach effectively
15

---

Published as a conference paper at ICLR 2026
Algorithm 2 Post-processing Scheme for IPC Matching
Input: Initial representative samples Sy {S1,....s ............. sM); Clusters Co {ci...., CM Parameters: IPC, min cluster size.
Ensure: Final representative set with |S| IPC.
S So. C Co
2: if [S| 2 IPC then
3:
S Samples corresponding to the IPC largest clusters in
4: else
5:
- fc |c| > meinclusstr.ssize
6:
while |S| < IPC & W + 0 do
& Strategy 1: Normally Split the valid clusters Cmother <- Select and remove the largest cluster from W SPLITCLUSTER(Gmal)
7:
8:
min cluster size)
9:
end while
if |S| IPC then
11:
N + the set of all outliers
Kinedddd IPC
13:
if [W] 2 Kneeded then
D Strategy 2: Cluster the outliers if available
Run K-Means on A get kneeded new clusters Coutiers and samples Soutiers-
15:
C-CU- Coutliers and S +- S Soutiers.
else
b Strategy Force split the viable clusters
17:
W+ + CcE > min.cluster_sizz
18:
while [S IPC do
19:
Cmother Select and remove the largest cluster from W FORCEDSPLLIT(Cuuber chin.clustrr....
20:
21:
end while
22:
end
23: end if
24:
end if
return S
26: procedure SPLITCLUSTER( (mooters man_cluster_size)
27:
Run HDBSCAN on Gmother with min cluster size to find & sub-clusters as Seand
28:
k <
return False
29:
S (-5
and C {Cmother]
Iterate all pairs (8i, Sj) from Seand initial seeds for K-Means (k on Gmothers select the pair yielding the minimum resulting inertia, denoted as {Cnewl, Cnew2] } and { (Sewwl, Snew2
C-CU {Cnowl Cnewz) and S < S {Snew1, Snew2)
32:
W + W U {c E (Cnew1, Cnew2] } |cl 2 x min cluster_size
33:
return True
34:
end procedure
35: procedure FORCEDSPLIT(Cmaeer min_size)
while SPLITCLUSTER( min_size) False & min_size do min_size + max(2, miin_szz x 0.75J) end while
37:
b> Relax the constraint
38:
39:
end procedure
consolidating the data points from trivial (and previously discarded) clusters into the larger, stable clusters that are ultimately retained.
Region 2 (min_size E [10, 20)). This is the optimal operational range for our method. In this region, M < IPC, which activates our complete post-processing pipeline. Here, each strategy fulfills its designated role: The first HDBSCAN (Init Clusters) is responsible for capturing the primary, high-density features; Strategies 1 and 3 (Splits) are responsible for refining these "large features" into more granular details; and Strategy 2 (K-Means) serves the perfect complement, efficiently extracting the remaining feature details from the outliers that HDBSCAN intentionally ignores. The synergy of all components achieves peak performance in this specific region.
16

---

Published as a conference paper at ICLR 2026
Region 3 (min_size > 25). As the threshold continues to increase, HDBSCAN becomes overly conservative and the number of initial clusters, M, drops sharply. In the [30,50] range, this often causes Strategy 1 to fail, forcing the pipeline to rely heavily on Strategy 3, which partially salvages performance. However, as M approaches zero, Strategy 3 also fails due to the lack of any "mother- clusters" to operate on. This causes the pipeline to degenerate entirely into solely activating Strategy 2 2-running a single-step K-Means on all data leading to sharp performance collapse.
is critical to note, however, that while performance degrades relative to its peak min_size 15, our method's accuracy across this vast operational range, min_size up to 100, consistently and significantly outperforms the SOTA (Santiago al. 2025
A.6 IMPACT OF KEY HYPERPARAMETER IN DISTRIBUTION DISCOVERY
The quality of R is primarily governed by two key hyperparameters: n_neighbors in UMAP and min_cluster.se _size in HDBSCAN. The n.neighbors controls the balance between preserving lo- cal versus global data structures while the mnincclsterrstt _sic dictates the granularity of the clus- tering. As illustrated by the performance heatmaps for ImagelDC in Figure 5 the optimal space for these two parameters shifts with IPC. For IPC=10, peak performance is achieved with large min cluster _sze (>50) and a large n. w.neighbors (>85). In contrast, for IPC=50, it shifts to a small min cluster size (< 20) and small n. neighbors (< 50).
This behavior is intuitive. At a low IPC, a large min cluster size prevents the fragmentation of major clusters, ensuring that the selected representative samples encapsulate features common across a broad spectrum of the data. This, in turn, requires large n_neighhors to accurately model the global structure. Conversely, at high IPC, the goal is obtain more fine-grained clusters so that each representative can embody more specialized, local features. This necessitates small n. neighbors, which effectively narrows the "field of view" for each point and compels UMAP to preserve distinct local structures.
Heatmap of Neighbours vs. Min size
Heatmap Neighbours vs.
15
48.5
19.2
49.1
49.9
49.4
49.9
49.6
10-
70.8
72.8
75.7
74.5
72.5
72.1
70.0
50
51.4
54.1
54.5
50.8
51.2
51.0
50.6
20
73.4
74.6
14.7
71.9
73.2
73.5
72.0
55
52.3
52.9
48.9
48.4
45.8
50.1
51.2
MM
30
73.9
74.3
74.9
72.6
73.1
74.2
74.1
Neighbours
60-
49.6
47.5
47.4
49.6
49.7
50.8
53.3
Neighbours
40
75.2
70.0
73.9
72.9
72.6
73.3
73.1
52.9
52.6
51.9
49.4
49.4
50.5
49.7
50-
73.0
70.0
73.2
73.4
72.5
74.0
73.1
51.3
53.1
52.2
50.7
49.1
60
71.4
75.2
72.6
73.7
73.4
69.9
71.9
51.5
52.1
70
72.6
72.4
73.2
72.7
73.0
72.2
68.9
52.5
50.5
50.1
50.5
49.2
50.7
80
72.9
74.0
72.8
71.5
72.1
73.9
73.5
50.6
50.7
55.0
53.8
49.9
49.9
90
72.0
72.2
71.7
72.9
73.0
74.1
71.8
90
48.4
50.2
52.6
56.3
55.9
55.3
54.2
100
74.3
72.0
73.3
72.5
70.5
72.2
73.5
40
45
50
55
60
65
70
10
15
20
30
Min_size
Min
Figure 5: Performance heatmaps for hyperparameter sensitivity on ImagelDC. The optimal region shifts from large neighbors values and min cluster size for IPC=10 (left) to low values for IPC=50 (right).
A.7 IMPACT OF KEY HYPERPARAMETERS IN DISTRIBUTION ALIGNMENT
Full Results for y and PIS Sensitivity Analysis In this section, we provide the complete data table used for the sensitivity analysis of the guidance strength and Prior Injection Steps (PIS). The results are presented in Table 12 Furthermore, we test whether the optimal hyperparameter space identified at IPC=10 is transferable to a high-IPC setting (IPC=50). We select a core range of effective parameters, with guidance strength 7 E 0.05, 0.08, 0.10, 0.13, 0.15 and Prior Injection Steps PIS E 0, 10, 15, 20, 25. As presented in Table 13 and Figure 6 The performance in all tested cases decisively surpasses the SOTA baseline of 72.1 %, with the optimal parameter setting achieving peak performance of 77.6% (+5.5% over SOTA).
17

---

Published as a conference paper at ICLR 2026
Table 12: Performance across different y and PIS values. Cells highlighted in yellow indicate performance greater than the SOTA of 55.9.
7 0.01
7
0.03
7 0.05
7 0.08
0.10
7 0.13
7= 0.15
y= 0.18
7 0.20
PIS=0
45.741.11
54.71122
$7.710.6
56.11100
55.91155
57.1110
57.11:.5
56.110.5
55.911.3
PIS=5
45.5+10
54.620.4
55.820.2
56.010.8
58.520.9
57.3204
57.01111
56.3+1.2
56.9=14
PIS=10
44.9.110
54.3.100
58.0110
$7.110.3
$7.3.1.3
$7.1.0.8
$6.7.10.6
$6.0.0.4
S5.510.4
PIS=15
45.1+0.5
54.0+0.7
57.1+0.3
56.7+13
57.1+100
56.5+133
56.6+1.0
56.4+0.9
56.4=0.9
PIS=20
45.3+1.0
54.2+0.7
57.2+1.1
56.7+0.9
57.1+088
56.6+0.4
58.1+0.8
56.1+0.6
55.9+1.6
PIS=25
45.0=0.4
53.3:0.5
56.5#1.5
56.920.6
56,5#0.77
56,2:00.7
56,5
55.7=2.0
56,3=1.8
PIS=30
43,91008
$2.1.108
$6.0.10.5
$6.0.10.7
$5.5.1100
$54.10.9
$4.7.11.6
$2.11122
52.5112
PIS=35
43.11:0.6
50.91133
54.110.3
56.61112
52,9114
53.9121
53.51100
52.511.5
52.8:2.00
PTS=40
42.3+088
50.3+1.0
1..3+1.3
58.2+
$1.5+100
50.4+066
47.7+112
49.7+
49.7+1.3
PIS=45
38
44.8=0.9
48.1+13
54.1=15
514+13
49,4=099
50.3+0.8
47.1=006
47.7=16
PIS=50
40.1+0.8
40.1+088
40.1+0.8
40.1+0.8
40.1+008
40.1+0.8
40.1+0.8
40.1+0.8
40.1
Table 13: Robustness of key hyperparameters (7 and PIS) on ImageIDC at IPC=50. All results reported as the mean + standard deviation over three independent runs. Notably, the performance in all tested cases decisively surpasses the SOTA baseline of 72.1 %, with the optimal parameter setting achieving a peak performance of 77.6% (+5.5 over SOTA).
70005
-=0,08
7=0.10
7=0.13
7=0.15
75.1
74.62
PIS=5
0.9
7.97
7..0
75.9
75.5 0.6
PIS-10
74.9202
7.55
7.
7..1 7..4
PIS=15
0.2
0.2
76.5 1.2
7.40
PIS=20
+
0.
76.5 1.2
0.8
75.3 0.8
P1S-25
74.5
1.2
7.6
0.9
7
Sampler Configuration. In this section, we analyze the impact of different sampler configura- tions. We use the DPM++ Karras sampler and investigate its performance inference steps. Specifically, we set the total number of steps to 25 and evaluate performance on the ImagelDC (IPC=10) dataset across range of y and PIS values. The results presented in Table
low number of
The results demonstrate that our method remains highly effective with significantly reduced number of inference steps. At just 25 steps, our method maintains a decisive advantage over the SOTA baseline. Notably, peak performance of 60.1 is achieved at ==0.05 and PIS=15, which not only surpasses the SOTA by 4.2% but also exceeds the optimal result we obtained with 50 inference steps.
However, this efficiency comes with trade-off in robustness. As a comparison between Table [4 (25 steps) and Table & (50 steps) reveals, the range of hyperparameter settings that outperform the SOTA is considerably narrower at 25 steps than the broad robust region observed at 50 steps. Given this trade-off, we prioritized robustness and therefore chose use 50 inference steps for our main experiments.
Table 14: Performance across different and PIS values when using the DPM++ Karras sampler with only 25 inference steps. Cells highlighted in yellow indicate performance greater than the SOTA of 55.9.
7=0,01
70003
7=0.05
7=0.08
0=0.10
7=0.13
0=0.5
7=0.18
7=0,20
PIS=0
41.8+0.9
56.9+1.7
58.5+00.6
55.4-0.7
55.4+188
55.4+0.2
55.1+1.0
$4.7+1.5
53.6+0.4
PIS=2
42.1
56.11..55
59.3:10.6
56.0=1.1 1.1
55.9
55.7:11.2
54.9+144
55.0:
5..0
PIS=5
41.8+0.4
56.5+-0.9
59.1+112
55.8-0.6
55.340.3
55.6+2.2
56.2+1.6
54.3-2.1
55.340.6
PIS-8
41.8+0.8
16.5+115
19.5+114
55.5-1.2
56.5+0066
55.5++1.2
54.5+009
$4.9+0.5
56.1+0.6
P1S=10
42.241.9
55.920.4
59.1-0.88
56.5-0.7
55.611..
56.110.3
54.710.8
55.3-1.0
55.211.1
P1S=15
40.5+1.3
55.5=0.7
60.1+1.0
56.3=1.0
554+14
54.5+113
53.400.5
51.5=110
50.9+100
PIS=20
39.8+0.2
50.1+0.4
56.3++0.6
59.1-0.4
55.0+14
54,2+0.7
524+2.1
49,7+0.6
45.7400
PIS=25
39.7+1.0
39.721.0
39.741.
39.7-1.0
39.7410
39.74100
39.74100
39.7-1.0
39.74110
18

---

Published as a conference paper at ICLR 2026
79
SOTA (72.15)
78
y=0.02
27
76-
Accuracy
75-
74-
73-
72
71
15
Figure 6: Performance on ImagelDC at IPC=50 across varying y and PIS settings.
A.8 PERFORMANCE EVALUATION ACROSS DIFFERENT RESOLUTIONS
This section provides detailed cross-architecture performance evaluation of our method on the ImageNet-A through ImageNet-E subsets across three different resolutions: 128x128, 256x256, and 1024x1024. The arohttcccieeeeture performance is evaluated with four distinct downstream archi- tectures: AlexNet (Krizhevsky et al. (2012), VGG11 (Simonyan Zisserman (2014), ResNet18 (He et al. (20T6)) and ViT (Dosovitskiy al. (2020)). These results demonstrate the scalability and consistent superiority of our approach.
Results at 128x128 Resolution. As shown in Table 15 method's performance 128x128
resolution decisively surpasses previous SOTA methods. Specifically, on the ImageNet-A through E subsets, our method achieves an average improvement of 12.7% %. 17.5%. %, 14.6%. 9.9%, and 14.5%. respectively. The detailed performance breakdown for each downstream architecture is provided in Table
Table 15: Comparing our method against SOTA baselines on ImageNet-A to ImageNet- E at 128x128 resolution with IPC=10. The best results in bold red and the second best in bold.
Method
Alg.
ImNetA
ImNetB
ImNetCC
InNetD
ImNetE
DC
52.340.7
45.1
40.117..6
36.140.4
18.04:0.4 36.0::0.5
Pixel
52.6::.4
50.6:20.5
47.5=:0.7
35.4:20.4
53.11114
50.12:06
48.911111
38.9+1.0
38440.77
GLaD
DM
52.8+100
51.3::066
49.7=0.4
36.420.4
38.620.77
55.241100
11.82::44
4.39
19.5:1
3..0
LDSM
DM
57.0-11.3
1..3=1111
48.2:4499
39,5=115
39.41:18
Ours
69.7+:1.1
6..8:1077
64.5
49.4
$3.91113
Table 16: Detailed performance of our method across different architectures. Results reported for IPC=10 at 128x128 resolution.
Architecture
ImNettA
ImNet-B
ImNet-C
InNettD
ImNet-E
ResNet18
7.
78.710..8
73.94:44
54.0::066
58.5+11.8
AlexNet
70.3:1.6
70.5
6..911.4
50.0:09
58.910.7
VGGH
68.6-00.5
69.61:0.6
61.31114
49.611.66
53.811111
VIT
65.7+112
60.3+007
56.74:06
44.1=111
44.2+177
Mean
6..7.1
69.810.77
6..5:1:55
49.41.11
$39113
Results at 256x256 Resolution. The comparison of our method's mean performance against SOTA baselines this resolution is presented in the main paper (Table B}. This section provides the detailed breakdown our method's performance on each of the four downstream architectures, with the full results available in Table 17
19

---

Published as a conference paper at ICLR 2026
Table 17: Detailed cross-architecture accuracies (%). The results for each architecture reported as the mean + standard deviation over five independent runs, using each distilled dataset at 256x256 resolution.
Architecture
ImNet-A
ImNet-B
ImNet-C
ImNet-D
ImNet-E
ResNer18
78.1
80.6 0.7
72.0 1.3
55.0 18
62.2
AlexNet
74.4
74.5 + 1.5
66.9 + 0.6
52.1 +.2
VGGII
0.5
65.1 2.0
5..
0.5
VIT
70.0 1.1
68.1 0.5
60.3 1.7
48.1
50.6 + 0.9
Mean
73.0 0.7
74.7 + 1.2
66.1
51.6 0.8
57.7 + 0.7
Results at 1024x1024 Resolution. We are the first to establish a dataset distillation baseline at high resolution of 1024x1024, opening new avenue for research in this area. The detailed results for this track are presented in the main paper (Table
A.9 CASE STUDY ON ZERO-COST DOMAIN CHANGE: DRIVE
To further demonstrate the zero-cost domain change capability of CoDA, we present a case study transferring the framework the DRIVE (Digital Retinal Images for Vessel Extraction) dataset (taal et al. (2004)). Qualitative results are presented in Figure 7 where the top row displays the real representative samples (R) identified by our "Discovery" stage, and the bottom showcases the final synthesized images (G) from the "Alignment" stage.
This visually confirms that CoDA can seamlessly adapt new and highly specialized domain without any additional manual intervention. This example powerfully illustrates the core limitation of target-trained methods. To our knowledge, no publicly available diffusion model pre-trained on DRIVE exists. Therefore, target-trained approach would be forced to first train or fine-tune generative model on DRIVE before the distillation could even begin.
Qualitative results of CoDA's zero-cost domain transfer the DRIVE dataset. Top row: The set of real representative samples (R) identified by the Distribution Discovery stage. Bottom row: The final set of guided images (9) generated by the Distribution Alignment stage.
Figure 7:
A. FURTHER COMPUTATIONAL EFFICIENCY ANALYSIS
We provide a comprehensive analysis of CoDA's efficiency, including end-to-end runtime, resource utilization, and comparison against training-free free (D*M) and target-trained (MGD") SOTA meth- ods, as shown in Table 18
The "Encode" stage utilizes two different VAE models. The Target-Trained DiT MGD3 employs "sd--ae-ft-mse". while others utilize "sdxl-vae". However, our practical tests, using a single RTX 4090, show no significant difference in runtime efficiency for this one-time Lanczos resizing and VAE encoding or VRAM usage for this stage.
For the "Discovery" stage, we implement all algorithms exclusively on a 16-VCPU Intel Xeon Gold 6430. The specific performance breakdown for CoDA is detailed Table 19 As observed, the computational cost of CoDA's "Discovery" stage is dominated by UMAP preprocessing, which is proven to be indispensable for performance in Appendix AT As shown in Table 18 if CoDA's UMAP is replaced with simple PCA, there is significant difference in the discovery stage speed compared to SOTA methods. This indicates that CoDA's entire clustering logic, while complex, is highly efficient in low-dimensional space and does consume significantly resources than
20

---

Published as a conference paper at ICLR 2026
Table 18: End-to-end efficiency and resource comparison (ImageNet-IKs IPC=50). "Change Do- main" is the cost to adapt to a new dataset (e.g., Places365).
Type
Step
Eucode (VRAM)
Diseosary
(R))
Generate (VRAM)
End-to-End Time
Change Domati
D4M
1.50 (155TMBB)
0.22 (1,996 MIB)
75/ing (12325381B)
39,4h
25..:0.1
MCDD1
1.5h (1551MB8)
0:4th (1 1996 MiR)
2. Fwing (12893M6i8)
19.4h
570-00 07.2
50
SDNL
(aDX/PCA)
1.5h (1553MBB)
0.42: (2,019 MIB)
55.40022
T.iing (135253I8B
39.4h
57.8.002
OJDA(UMAP)
(1553M68)
2.17 (2.144 MiB)
58.26002
2.rwiig (13303Mi8)
60:4002
25 sap
CsDA/UMAP)
(1553MBB)
2.17% MIB)
58.24902
1.siing (12323MiB)
21.7h
60:1:02
Targee
step
(1553MIB)
0:42 MIB)
1.3xing (2796MIB)
602:02
1PU-day
Table 19: Breakdown of the Distribution Discovery stage (224x224, on ImageIDC IPC=50).
Type
Preprocess-lime
HDBSCAN
Strategy SpittCluster
Strategy 2: Clustering Outliers
Strategy ForcedSplit
UMAP
72.275
0.969 0,740
1.156
0,083 s
0.200
PCA
1,592
0,002
0,157 s
0.000
direct K-Means. Notably, only CoDA produces a usable set of representative real images, R, at this stage, which already possesses high performance. As shown in the main paper (Table (6) in some cases, CoDA(R) already surpasses the final generated datasets of some SOTA methods.
In the "Generate" stage, the variance in speed stems from the generative model architectures. While DiT benefits from faster generation, we demonstrate in the main paper (Table (9) and Appendix A.7 that configuring our sampler to steps maintains or even improves optimal performance. With this setting, CoDA's efficiency approaches that of the fastest MGD3 without performance degradation. Crucially, it must be emphasized that DiT's speed advantage merely amortizes the massive pre- training cost required for each domain. In contrast, as demonstrated in the main paper (Table 7 and Appendix A.9 CoDA is completely general and its cost to change domains is zero.
A.11 VISUALIZATION AND ANALYSIS OF FEATURE SPACES AND THE ROLE OF UMAP
To further investigate the distinctions between the CLIP and VAE spaces, and to underscore the indispensability of UMAP preprocessing the CoDA pipeline, we present comprehensive visu- alizations alongside quantitative metrics. We employ t-SNE (configured with "n commponents=2"" "perplexit==40", "random =tatee")ttte=42") to project high-dimensional feature spaces into 2D for visual analysis.
To quantify the "compactness" of the resulting manifolds visualized in Figure S we report the Mean (the average kernel bandwidth from t-SNE) in Table 20 As shown in Figure & a), the original VAE space is highly diffuse, exhibiting a large Mean of 12.57. This confirms our hypothesis (see Figure Ic) that VAE spaces are ill-suited for density- clustering. Comparing the 768-dimensional VAE (c) and CLIP (b) spaces, both the visualization and the Mean (8.06) are remarkably similar. Although inter-class separation is marginally improved, data points remain heavily intermixed, and intra-class compactness is insufficient. This observation challenges the assumption that CLIP is inherently superior solely due to its semantic alignment. Furthermore, as illustrated in (d), simply applying linear dimensionality reduction (PCA) to CLIP (50d) fails to yield a structured manifold; the Mean a drops to 6.33 primarily due to dimensionality reduction, yet the visualization reveals even severe class overlap.
In sharp contrast, applying UMAP (with "min dist=0.0") in (e) dramatically transforms the mani- fold. The Mean plummets to 0.29. This quantitatively validates that our choice of UMAP is both necessary and effective. UMAP successfully disentangles the mixed feature space into distinct is- lands, significantly maximizing inter-class distance while enhancing intra-class compactness. This transformation establishes a robust foundation for the subsequent clustering task, explaining why the UMAP+CoDA combination achieves SOTA performance (see Table 10
Furthermore, we demonstrate UMAP's robustness on Image a dataset characterized by ex- tremely high inter-class similarity (containing breeds such Australian terrier, Samoyed, Beagle, etc.). As shown in Figure 9 the original VAE space (a) is highly diffuse (Mean σ: 10.84), and stan- dard PCA (b) fails to induce meaningful separation (Mean 7.92). In stark contrast, the UMAP
21

---

Published as a conference paper at ICLR 2026
(a).
(b)
(c)
(d)
(e)
Figure 8: I-SNE visualization of different feature spaces for ImagelDC. (a) Original VAE (3136d).
(b) Original CLIP (768d). (c) VAE (PCA to 768d). (d) CLIP (PCA to 50d), (e) CLIP (UMAP to 50d, using our pipeline's "min.dist=O.0"" setting).
Table 20: Manifold compactness metrics for the spaces visualized in Figure
Space
Dimension
Mean
Original VAE
3136
12.57
Original CLIP
768
8.06
VAE (PCA)
768
8.06
CLIP (PCA)
50
6.33
CLIP (UMAP, "min_dist=000"
50
0.29
used by CoDA (c) successfully reshapes this challenging manifold, creating dense, compact, and well-separated clusters (Mean σ: 0.30). This confirms that our UMAP-based preprocessing is a critical component that remains effecttve even for fine-grained datasets.
A.12 LIMITATIONS OF GENERAL-PURPOSS MODELS IN DD TASKS
As illustrated in Figure 10 using only the class name prompt for general-purpose model like SDXL leads to highly inconsistent results. For classes with unambiguous names, such as 'garden spider' "gyromitra', the generated images like 'bonnet', the model's generic, web-scale priors cause it to completely deviate from the target dataset's context. For instance, in the figure, we can see that out of ten generated images for the 'bonnet' class, only eight depict the required 'hat'. The other two, showing a bird and a car (referring to other meanings of the word), would introduce significant noise and confusion during subsequent model training. This highlights the necessity of our additional guidance to correct these semantic misunderstandings and align the model with the target dataset's specific distribution.
often acceptable. However, for polysemous words
To provide more direct comparison,
visualize the "tench" class from ImageNet in Figure II As shown, the images generated by the General.LDM (b) using the prompt "tench" are visually inconsistent with the original data (a), the target-trained ImageNet LDM (c), and our CoDA (d). Although they are all technically images of "tench", the General LDM's prior generates a style that is completely misaligned with the target dataset's 's distribution.
We also compute the FID score using these four sets of images, with the results presented in Table 21 Although the absolute values high due the small sample size, the relative relationship is clear:
22

---

Published as a conference paper at ICLR 2026
(a)
(b)
(c)
Figure 9: t-SNE visualization of the VAE feature space for ImageWoof. (a) Original VAE space
(Mean a: 10.84). (b) VAE (PCA to 50d) (Mean a: 7.92). (c) VAE (UMAP to 50d) (Mean o: 0.30).
Figure 10: Qualitative comparison of SDXX's generation consistency ImagelDC (IPC=10). The top two rows show successful generations for unambiguous classes ('garden spider' and 'gyromi- tra'). In contrast, the bottom row demonstrates failure case for the polysemous class 'bonnet'. Despite the prompt intending a type of hat, the model's general prior also generates incorrect images related other meanings of the word (a car and bird), highlighting the distributional mismatch.
Table 21: FID-10 comparison for the "tench" class. Lower is better.
Method
FID Score (Lower is Better)
Original Data General LDM
81.42
104.14
ImageNeLLLDM
71.05
CoDA (Ours)
60.09
The Original Data (randomly sampled) serves as a baseline FID of 81.42. An effective DD method must perform better (lower FID) than this random selection.
The ImageNet LDM (71.05) achieves a better score, showing its inherent alignment the data it was trained on, which naturally facilitates a simple DD process.
The General LDM (104.14) fails this task completely. Its severe distribution mismatch results an FID far worse than random sampling.
CoDA achieves the lowest FID (60.09) by a significant margin, demonstrating that our alignment strategy successfully bridges the gap and matches the target dataset's core distri- bution features.
23

---

Published as a conference paper at ICLR 2026
(a) Original Data
General_LDM
ImageNet_LDM
(d) CoDA
Figure 11:
Qualitative comparison of the "tench" class. (a) 10 random samples from the original
dataset. (b) samples from General_LDM (SDXL) using the prompt "tench". (c) 10 samples from the target-trained ImageNet_LDM, (d) Our final CoDA (IPC=10) dataset. The General_LDD (b)
produces completely different style from the other three groups.
Figure 12: Visualizing the impact of guidance strength on generated images from ImagelDC (IPC=10). Rows (top to bottom): The generated images with ? (Original SDXI.), 7 0.01, = 0.03, 7 0.05, = 0.1, 7 0.2 and the ground-truth real image. Columns (left to right): The 10 classes of the dataset: 'green mamba', 'garden spider', 'saluki', 'doberman', 'langur', 'bonnet', 'cocktail shaker', 'vacuum', "window screen', and 'gyromitra'.
24

---

Published as a conference paper at ICLR 2026
Table 22: Performance comparison of distilled datasets under different guidance strengths 7, with PIS fixed 5.
7=00.00
0=0.01
7=0.03
7=0.05
7=0.10
7=0.20
Real Image Set (R)
40.1 # 0.8 45.5 # 1.0
54.6 0.4
55.8 # 0.2
58.5 0.9
56.9 + 1.4
56.3 0.9
Figure 13: detailed comparison between original real images (left in each pair) and their corre- sponding generated images (right in each pair) under our optimal guidance strength (? 0.1). The classes shown are "green mamba' (top-left), 'garden spider' (top-right), 'saluki' (bottom-left), and 'langur' (bottom-right).
13 VISUALIZING THE IMPACT OF GUIDANCE STRENGTH
This section visualizes the impact of the guidance strength on the final image quality. As shown in Figure [2] fix PIS=5 and select an image for each the 10 classes from the ImagelDC dataset, comparing the results for 7 E {0, 0.01, 0.03, 0.05, 0.1, 0.2) and the ground-truth real images. The corresponding test accuracy for each setting is presented in Table 22 can observed, guidance strength of = 0.03 is already sufficient to substantially correct most generated images. As the strength increases, the resulting dataset's 's performance peaks at 7 = 0.1. As the guidance is further increased to y 0.2, the generated images become nearly indistinguishable from the real images, and their corresponding test accuracy converges to that of the real-image set.
A more detailed qualitative comparison, presented in Figure 1 reveals the source of this perfor- mance gain. The 2.1% accuracy improvement at = 0.1 over the real-image set stems from the generated images' ahility enhance details while preserving the structure of the real images. This core structure, identified by our Distribution Discovery stage, provides a solid foundation for the generation process, ensuring that the synthesized images align with the dataset's intrinsic core distribution. The richer details are injected by the model's prior, facilitated by two key mechanisms: (1) preserving the unmodified unconditional noise (z , & (), and (2) allowing the model freely refine the output during the final PIS steps.
Specifically, the generated images for 'green mamba' and 'garden spider' exhibit significantly im- proved brightness, contrast, and overall clarity compared to the real images. For 'saluki', the gen- erated image avoids potentially confusing pose in the real image- where the dog's 's legs are folded during sprint-and instead depicts the limbs in a more canonical and recognizable state, which is less likely to introduce ambiguous features for a downstream classifier. For 'langur', the generation process intelligently removes distracting element from the real image (the face of baby monkey held in its arms) and seamlessly replaces it with the arm of the parent monkey. This refinement results in cleaner composition where the langur's 's key features These seemingly intelligent refinements serve as compelling evidence for the crucial role of the diffusion model's prior knowledge in creating a superior distilled dataset.
more distinct and less occluded.
25

---

Published as a conference paper at ICLR 2026
0.00
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0.10
0.12
Y 0.15
0.20
0.25
Y 0.30
0.35
real
Figure 14: Visualization of the generation process collapsing as increases. The guidance (PIS=5) applied to a "tench" class sample from ImageNette. A = 0.35, the generation fails completely, resulting in black image.
However, the guidance strength cannot be set arbitrarily high. As visualized for "tench" class sample in Figure 14 the generation process collapses when y becomes too large. Once the value reaches 0.35 or higher, the conditional guidance becomes overpowering, pulling the latent vector into a sparse region of the manifold. This causes the subsequent denoising steps to fail, resulting in pure black image. This highlights the necessity of setting reasonable parameter in the Distribution Alignment stage to balance fidelity and generative stability.
A.14 VISUALIZATION FOR CODA TRANSFER TO DIT
To support our analysis in Table II
we provide a qualitative comparison for the "doberman" class (IPC=10) in Figure 15 This visualization supports our hypothesis regarding the optimal hyperpa- rameter settings for DIT.
As shown in (c), the optimal low-IPC configuration (7 1.0, PIS 15) allows DiT more generative freedom, leveraging its target-trained prior to produce diverse images. In contrast, the optimal high- IPC configuration (b) (y 0.8, PIS 8) applies stronger guidance, pulling the generated images closer in style to the real images (e). This reinforces our conclusion: DiT's inherent knowledge is beneficial but limited. At low IPCs (s 20), allowing more freedom is effective. However, when a larger, more diverse set is required (IPC=50), DiT's prior is saturated, and it requires stronger guidance from the real dataset R to achieve peak performance.
A.15 ROBUSTNESS TO NOISY AND INCOMPLETE LABELS
In this section, address the robustness of CoDA's Distribution Discovery stage when facing noisy or incomplete labels.
Robustness to Noisy Labels. CoDA is inherently designed to handle noisy labels. This was a pri- mary motivation for building pipeline based on HDBSCAN rather than directly using K-Means. K-Means lacks intrinsic outlier detection capabilities and forces every data point into a cluster, of- ten requiring additional, external outlier removal steps such LOF. In contrast. HDBSCAN is fundamentally dansity-based, making it inherently robust to noise. For instance, 'dog' image er- roneously labeled 'cat' will typically reside in extremely low-density region within the 'cat'
26

---

Published as a conference paper at ICLR 2026
(a) DIT
0.80.8 PIS-B)
CoDA (v-1.0, PIS-15)
(d) SDXL_CODE CODA
(e) Real
Figure 15: Qualitative comparison of CoDA transferred to DiT for the "doberman" class (IPC=10).
(a) Original ImageNet-Trained DIT. (b) DiT+CoDA with high-IPC optimal parameters (y
0.8, PIS 8). forcing closer alignment. (c) DiT+CoDA with low-IPC optimal parameters (7 =
1.0, PIS 15), allowing more generative freedom. (d) Our SDXL+CoDA method. (e) Real images
from the dataset.
VAE feature space, distinct from the dense core of true 'cat' samples. HDBSCAN automatically identifies such points as noise/outliers and excludes them from the representative sample set (S,).
To validate this, we conducted visualization experiment on the class "n02102040 English springer", where we introduced noise by mixing in 10 randomly sampled images from the class "n02123045 Tabby cat". We applied our standard pipeline: VAE encoding, followed by UMAP dimensionality reduction (with min_dist = 0.0). Figure 16 visualizes the resulting manifold us- ing t-SNE. Crucially, as observed the figure, none of the noisy samples (Tabby cats, marked in red) are located in absolute high-density regions. Since our selection mechanism strictly targets the highest-density cores to form S,,, these noisy data points are naturally bypassed without requiring manual intervention.
Robustness to Incomplete Labels (Semi-Supervisedd Learning). The robustness of CoDA also enables handle incomplete labels efficiently, extending its applicability to semi-supervised dataset distillation. To illustrate this, we simulate semi-supervised setting on the ImagelDC dataset and construct three configurations: (1) Full Supervision: The standard setting using 100% of the labeled data. (2) Limited Labels (10%): We randomly select only 10% of images (130 images per class) the labeled set, treating the remaining 90% as unseen. (3) Semi-Supervised (10% Pseudo): We utilize the 10% labeled set to guide the utilization of the 90% unlabeled data via lebeliga Specifically, we calculate the class centers of the 10% labeled data in the CLIP feature space. We then perform global similarity search for the unlabeled samples against these centers. An unlabeled sample is assigned a pseudo-label and added to the pool if its cosine similarity to class center exceeds confidence threshold of 0.85.
The results are presented in Table 23 While using only 10% of the labels leads to performance drop (70.6%), employing our labeligga strategy significantly recovers performance to 76.0%, closely approaching the fully supervised upper bound 77.6% and significantly surpassing the previous SOTA MGD3 (72.1% by 3.9 This indicates that CoDA effectively discover the
27

---

Published as a conference paper at ICLR 2026
Engien spttoge Tatey
Figure 16: t-SNE visualization of the Distribution Discovery stage under noisy labels. The dataset consists of all images from the 'English springer' class (Black dots) mixed with 10 noisy images from the "Tabby cat' class (Red dots). The noisy samples are naturally isolated in low-density regions, ensuring they are not selected by our density-based discovery algorithm.
core distribution even when the majority of the data lacks ground-truth labels, provided small seed set of labeled data is available.
Performance comparison ImagelICC (IPC=50) under different label availability set- tings. The test model is ResNe1AP-10., Results are the mean and standard deviation of three inde- pendent runs. The best results are in bold, and the second best are bold. CoDA's discovery stage operates efficiently under both fully supervised and semi-supervised settings, consistently surpass- ing the SOTA.
Table 23:
Method
Label Availability
Results
Random
68.1+0.7
D'M
63.8+00.4
Full Supervision
MGD3
72.1+00.8
CoDA
77.6:10.6
CoDA
Limited Labels (10%)
70.6++0.8
Semi-Supervised (10% + Pseudo)
CoDA
76.0
A.16 TRANSFERRING CODA TO FLOW-MATCHING MODELS SD3
To further demonstrate the transferability of our framework, we extended the CoDA alignment stage to the Stable Diffusion 3 (SD3) architecture, which employs Rectified Flow matching objective. The results on ImagelDC, ImageNette, and Image Woof are presented in Table 24
We draw several key observations from these results:
28

---

Published as a conference paper at ICLR 2026
1. Similar to SDXL, SD3 is general-purpose model and thus is unable to independently perform the DD task.
2. CoDA demonstrates remarkable transferability, seamlessly adapting SD3's flow- matching architecture. The integration of CoDA (SD3+CoDA) leads to a substantial per- formance improvement over the base SD3 model.
3. Furthermore, across all configurations on ImageIDC and ImageNette, the SD3+CoDA combination matches or even surpasses the previous SOTA method (DiT+MGD*).
Overall, these results further validate CoDA's high adaptability and its effectiveness in bridging the distribution gap across different generative frameworks, including those based flow matching.
Table 24: Comparison of transferring CoDA to the SD3 architecture against other baselines. All results evaluated using ResNetAP-10. Best results bold, and second best bold.
Dataset
IPC
SD3
DIT
DIT+MGD'
SD3+CoDA
DIT+CoDA
SDXL+CoDA
10
1.3
54.1+0.4
55.9+2.1
56.2+0.6
59.4+1.55
58,5+0.9
ImageIDC
20
42.410.7
58.910.2
61.910.9
65.41
65.8:11.2
64.6-10.5
50
5..411.6 64.3:10.6
72.1
7..01:100
7..6:1..6
77.6=1..6
10
53.4+1.1
59.1+0.7
66.4+2244
66.6++0.5
65.2+11.5
68.8+0.1
ImageNette
20
59.6.0.9
6..8111.
71.2:10.5
72.2:00.9
73.0
72.1-0.2
50
6.44
73.3:0.9
7..
80.1+13
81.41.3
83.1-1033
32.2+0.8
34.7+0.5
40.4++1.9
38.24-0.8
39.2++1.3
39,2-0077
ImogeWoof
20
37.611.2
41.1+0.88
43.61:1.6
43.41:6
44.2:1099
42.5-0.6
41.8
49.3+0.2
56.5
55.41111
59.211.7
59.4-10
A.17
VISUALIZATION OF FINAL RESULTS
This section showcases some of the state-of-the-art (SOTA) images generated by our method.
29

---

Published as a conference paper at ICLR 2026
Figure 17:
The complete generated dataset for the first classes of Places365 at IPC=10. The rows, from top to bottom, correspond to the classes: 'airfield', sarplane_cabin' "airport_terminal" 'alcove', 'alley', 'amphitheater', 'amusement arcade', camusement_parks "apartment.building: outdoor', and 'aquarium'.
30

---

Published as a conference paper at ICLR 2026
Figure 18: The complete generated dataset for ImagelDC at IPC=10.
Figure 19: The complete generated dataset for ImageNette at IPC=10.
31

---

Published as a conference paper at ICLR 2026
Figure 20: The complete generated dataset for ImageNet-A at IPC=10.
Figure 21: The complete generated dataset for ImageNet-B at IPC=10.
32

---

Published as a conference paper at ICLR 2026
Figure 22: The complete generated dataset for ImageNet-C at IPC=10.
Figure 23: The complete generated dataset for ImageNet-D at IPC=10.
33

---

Published as a conference paper at ICLR 2026
Figure 24: The complete generated dataset for ImageNet-E at IPC=10.
34


Reviews:
[{"review_id": "WKwBGWVD44", "reviewer": "Reviewer_NgTi", "summary": "This paper presents CoDA, a training-free dataset distillation framework that leverages an off-the-shelf text-to-image diffusion model to generate distilled data that is better aligned with the target dataset. The key idea is to first discover an intrinsic, high-density “core” distribution in the latent space of the target data and then guide the diffusion process so that the synthesized samples follow this core distribution. Experiments on ImageNet-1K and its subsets indicate that this distribution-aware generation can match or outperform several existing, diffusion-based dataset distillation methods that require training or fine-tuning on the target data.", "questions": "1. Can you report end-to-end runtime (discovery + generation) per dataset and per IPC, and compare it with both target-trained diffusion distillation methods and recent training-free approaches?\n\n2. How robust is the distribution discovery stage when the off-the-shelf model’s latent space is not well aligned with the target domain (e.g., domain shift, fine-grained categories)?\n\n3. Is it possible to reuse or amortize the discovered core distribution across related datasets or class subsets to reduce the discovery cost?\n\n4. How does the method behave when labels are noisy or incomplete during the discovery stage?\n\n5. For higher-resolution generation, do you require per-class retuning of guidance strength, or is there a single configuration that works across classes?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper clearly identifies a practical gap in current diffusion-based dataset distillation: many approaches rely on a diffusion model trained on the very dataset they aim to compress. CoDA directly addresses this by using only an off-the-shelf model.\n\n2. The two-stage pipeline—(i) discovering a core distribution in latent space and (ii) aligning the diffusion generation to that distribution—matches the problem formulation and makes the contribution transparent.\n\n3. The use of latent embedding, dimensionality reduction, and density-based clustering allows the method to pick representative, high-density samples instead of relying on naïve per-class sampling.\n\n4. The method is tested on multiple ImageNet variants, different IPC regimes, and different evaluation settings, and it consistently shows advantages over other training-free or general-model baselines.\n\n5. The ablation studies demonstrate that most of the gains come from closing the distribution gap rather than from task-specific training, which supports the main claim of the paper.", "weaknesses": "1. Although the approach does not train a diffusion model, the discovery stage (encoding, dimensionality reduction, clustering) plus guided sampling still incurs noticeable computational cost. A more explicit comparison of wall-clock time and memory with key baselines would make the “training-free” claim more convincing.\n\n2. The quality of the discovered core distribution may depend on settings for the dimensionality reduction and clustering steps. The paper suggests that different datasets may require slightly different configurations, which could limit robustness.\n\n3. The method is closely tied to the latent geometry of the chosen text-to-image model. It is not yet clear how portable the approach is to other diffusion architectures or to future models with different encoders.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QHOMwNUFi2", "reviewer": "Reviewer_ugRR", "summary": "This paper proposes Core Distribution Alignment (CoDA), a \"training-free\" framework for Dataset Distillation (DD) that solves a key problem: prior methods either paradoxically require models pre-trained on the target dataset or fail due to a \"distributional mismatch\" when using general models. CoDA works in two stages: first, a Distribution Discovery pipeline uses VAE, UMAP, and HDBSCAN to find the \"intrinsic core distribution\" of the target data. Second, a Distribution Alignment stage guides an off-the-shelf text-to-image model (like SDXL) to generate samples matching this core distribution. This approach achieves state-of-the-art results, surpassing even target-trained methods on benchmarks like ImageNet-1K.", "questions": "1.⁠ ⁠The \"intelligent refinement\" shown in Figure 9 is a key finding. But does this always work? You use the polysemous word 'bonnet' as an example of T2I failure in Figure 7. What happens in a high-disagreement scenario where the guiding sample s_j is a hat, but the SDXL prior (activated during PIS) is strongly biased towards cars? Does your guidance successfully correct the model, or does the \"refinement\" step cause the image to revert to a car, making the generated dataset worse than the real one?\n\n2.⁠ ⁠Why did you choose to build this highly complex, heuristic-driven pipeline on a feature space, rather than using a different off-the-shelf feature extractor that is also \"training-free,\" such as a CLIP encoder, which is known to provide a much more semantically separate latent space? Did you experiment with this?\n\n3.⁠ ⁠What analysis did you perform to measure the stability of this Distribution Discovery stage? How do we know this cascade of heuristics reliably converges to a high-quality set of samples, rather than just being a fragile mechanism that happened to work for your tested datasets?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.⁠ ⁠The paper tackles a fundamental, conceptual flaw in a popular research area and provides a complete, working solution. This \"truly training-free\" framework is what generative DD should have been from the start.\n\n2.⁠ ⁠The 2-stage design is a key strength. The paper identifies that both discovering the right distribution and aligning to it are essential. The Distribution Discovery pipeline is a major contribution on its own.\n\n3.⁠ ⁠⁠The discovery that the generated set Ours (G) can outperform the real set Ours (R)  is a highly significant finding. It suggests that the generative prior is not just a source of bias to be overcome, but a powerful tool that can be harnessed to \"denoise\" and \"improve\" real-world data for a specific task.", "weaknesses": "1.⁠ ⁠The authors disclose that the key hyperparameters for the Distribution Discovery stage (n_neighbors for UMAP and min_cluster_size for HDBSCAN) exhibit \"drift\" across different datasets. While the method remains robustly above the baseline, achieving peak performance requires a new, dataset-specific grid search. This somewhat undermines the \"plug-and-play\" nature of the \"truly training-free\" framework.\n\n2.⁠ ⁠This entire chain is extremely brittle. A small, insignificant change in the initial UMAP embedding could cause a cluster to be \"un-splittable\" by Strategy 1, triggering a completely different logic path (e.g., Strategy 3 vs. Strategy 2), leading to a radically different set of representative samples. The paper does not analyze the stability or variance of the Distribution Discovery stage itself, which is a significant methodological gap given its complexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dycKI919fn", "reviewer": "Reviewer_ptsN", "summary": "The paper proposes Core Distribution Alignment (CoDA) — a dataset distillation framework that enables large-scale dataset distillation, relying solely on an off-the-shelf text-to-image diffusion model (SDXL).\n\nCoDA introduces two phases:\n- Distribution Discovery: Identifies the core distribution via UMAP + HDBSCAN clustering in the VAE latent space, selecting representative samples.\n- Distribution Alignment: Guides diffusion sampling using a core-distribution-based energy term, aligning generated images with selected samples.\n\nExperiments on ImageNet-1K and its subsets show CoDA achieves SOTA performance. The method is also computationally efficient and generalizes to new domains like Places365.", "questions": "- **distribution mismatch**: Figure 1a is confusing. The authors claim that the general T2I LDM “generates samples confined to a limited region of the data distribution,” implying an intrinsic coverage limitation. Which algorithm do you use to generate the visualization? T-SNE or UMAP? Do you have quantitative results to support your observation? Since the pre-trained DMs have better IS-FID score among generative models, they generate high fidelity and diversity samples. I know the dataset domain is different, but another reason for raising such distribuition mismatch may be the hyper parameters you used in your visualization algorithm. Are the results come from the same settings?\n- **Cross-architecture results**: What's the settings of table4? The results are come from a single eval network or the avg results of several eval networks? Are the eval networks be the same?\n- **Lack of experiments**: How about apply coda to the DiT or other DMs pre-trained on the target (such as Imagenet) dataset? Will it augment the SOTA results? It is also important to explore the ultimate capacity of coda.\n- **Lack of datasets**: Since the authors claim that the intrinsic core distribution is important in DD, I think the authors should provide the results on ImageWoof, which is a common dataset used in DD with more inter-class similarity.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **Clear formulation**: the paper presents a clean DD framework that combines the clustering-based representative discovery (such as D$^4$M and MGD$^3$) and energy-based diffusion guidance (such as MGD$^3$ and IGD). Even though both components exist in prior works, CoDA articulates them under a consistent probabilistic formulation (Eq. 4–7), which makes it easy to follow and reproduce.\n\n- **Strong empirical performance and generalization**: CoDA achieves SOTA accuracy on multiple benchmarks, and the results generalize across both architectures and datasets. The experiments are extensive and well-controlled.\n\n- **Efficient implementation**: CoDA’s distribution discovery phase runs entirely on CPU using VAE features, and the guidance process uses a well pre-trained T2I SDXL with no further training or SFT.", "weaknesses": "- **Overstated “truely training-free DD” claim and misleading title**: The title “From Text-to-Image Diffusion Models to Truly Training-Free Dataset Distillation” is somewhat exaggerated and conceptually inconsistent. The authors argue that prior generative DD methods are paradoxical because they depend on diffusion models pretrained on the target dataset like imagenet-1k, whereas CoDA avoids this by using a general text-to-image diffusion model (SDXL). However, this reasoning is misleading: using a pretrained generative model regardless of its training data is a standard and acceptable practice in DD research, not a paradox. In fact, CoDA still relies on a large diffusion model that was pretrained on an even larger and more expensive dataset (LAION) than ImageNet. Thus, the method is not “training-free” in any absolute sense. It merely shifts reliance from a domain-specific model to a more general pretrained one. The paper’s framing unfairly overemphasizes this distinction with a whole para and risks confusing readers about what “training-free” actually means in the context of DD. I do agree with your distributional mismatch claim, but not the paradoxical pretrained model claim that you described in both abstract and introduction.\n\n- **Incremental method**: The authors a) replace the K-means algorithm with HDBSCAN and design a IPC Matching algorithm to obtain sufficient samples (similar to the prototype conception in D4M). Then, they b) use these samples to guide the denosing process (similar to the guidance mechanism in MGD3). Although the authors claim that the proposed method is distinct from D4M and MGD3,  it is still somewhat incremental based on the prior works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RRhAgIqHl4", "reviewer": "Reviewer_cWUx", "summary": "This paper proposes a two-stage approach. First, the Distribution Discovery stage employs HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) to identify all valid high-density clusters and subsequently select a set of representative samples. Second, the Distribution Alignment stage then utilizes the guidance of these representative samples to generate synthetic images, aligning the synthetic data distribution with the real data distribution.", "questions": "1. In Table 5, why does the performance achieved by the \"Ours (R)\" configuration (utilizing real samples selected by HDBSCAN) yield lower accuracy compared to the \"Ours (G)\" configuration (which are synthetic images)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The authors discover that the VAE latent space features exhibit poor separation due to the mixing of inter-class features, which severely degrades inter-class separability and intra-class compactness. To mitigate this issue, they leverage UMAP to preprocess the latent embeddings from the VAE.\n2. The proposed HDBSCAN method, combined with several split strategies, successfully identifies and selects well-distributed, representative samples from the original ImageNet dataset.\n3. The proposed approach demonstrates consistent performance improvements across the ImageNet-1k dataset and its 7 subsets.", "weaknesses": "1. Insufficient visual validation of UMAP: While the paper employs UMAP to preprocess VAE latent embeddings, the effectiveness of this dimensionality reduction for improving feature separation is not visually demonstrated.\n2. Ambiguity in distribution alignment and scope violation: The Distribution Alignment stage involves a mechanism that appears to directly compensate for the difference between generated images and real samples. The strong dependence on a lambda hyperparameter suggests this step might be manually compensating for the difference, or potentially regressing to the exact real samples if lambda=1. This approach contradicts the principle of dataset distillation, which requires the synthesis of a small, artificial dataset, as its reliance on near-real samples aligns more closely with coreset selection rather than true distillation, making the paper's claimed scope questionable.\n3. The claimed achievement of a new state-of-the-art accuracy of 60.4% in the 50-images-per-class (IPC) setup on ImageNet-1K in the abstract or contribution section is only a 0.1% improvement over the baseline of 60.3%. This minimal gain is highly likely to be within the margin of experimental randomness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)